Pick your Poison: Undetectability versus Robustness in Data Poisoning Attacks against Deep Image Classification

Nils Lukas

University of Waterloo

Florian Kerschbaum

University of Waterloo

Deep image classification models trained on large amounts of web-scraped data are vulnerable to data poisoning, a mechanism for backdooring models.

Even a few poisoned samples seen during training can entirely undermine the model's integrity during inference.

While it is known that poisoning more samples enhances an attack's effectiveness and robustness, it is unknown whether poisoning too many samples weakens an attack by making it more detectable.

We observe a fundamental detectability/robustness trade-off in data poisoning attacks:
Poisoning too few samples renders an attack ineffective and not robust, but poisoning too many samples makes it detectable.

This raises the bar for data poisoning attackers who have to balance this trade-off to remain robust and undetectable.

Our work proposes two defenses designed to (i) detect and (ii) repair poisoned models as a post-processing step after training using a limited amount of trusted image-label pairs.

We show that our defenses mitigate all surveyed attacks and outperform existing defenses using less trusted data to repair a model.

Our defense scales to joint vision-language models, such as CLIP, and interestingly, we find that attacks on larger models are more easily detectable but also more robust than those on smaller models.

Lastly, we propose two adaptive attacks demonstrating that while our work raises the bar for data poisoning attacks, it cannot mitigate all forms of backdooring.

<concept_desc>Computing methodologies Supervised learning</concept_desc>
<concept_desc>Security and privacy Software and application security</concept_desc>
<concept_desc>Computing methodologies Machine learning</concept_desc>

[500]Security and privacy Software and application security
[500]Computing methodologies Machine learning

§ INTRODUCTION

The integrity of deep image classification models is essential for their safe and effective use in real-world applications <cit.>.

A model with integrity should make accurate, transparent, and robust decisions that are resistant to manipulation.

Training models with integrity is challenging, which limits the applicability of models in domains with impactful decision-making like healthcare <cit.>, credit scoring <cit.>, and access control systems <cit.>.

One form of manipulation that can undermine a model's integrity is backdooring - an attack during training enabling an attacker to influence the model's predictions during inference by stamping a secret trigger pattern on an image <cit.>.
Backdooring can be accomplished through data poisoning, a technique in which the attacker injects a limited number of poisoned samples into the model's training data, thereby creating spurious correlations <cit.> between the trigger and an attacker-chosen target class.
Given that modern deep learning models rely on vast amounts of web-scraped data for optimal accuracy <cit.>, poisoning attacks are considered among the most worrisome threats to a model's integrity in practice <cit.>.

Problem. Consider a provider who trains a deep image classifier using web-scraped data and deploys their model for a downstream task such as content moderation, like OpenAI with CLIP <cit.>.
The provider, being trustworthy, must balance the trade-off between high model accuracy and integrity but needs to train on large, web-scraped datasets to achieve the former.
The threat is an attacker who poisons a small number of samples during training to evade content moderation during inference.
With CLIP, it has been demonstrated that backdooring succeeds by poisoning only $400$ out of $400$ million image-label pairs <cit.>.
The provider needs a method to mitigate data poisoning while training an accurate model.

To counter backdooring, the provider can (i) sanitize the training dataset <cit.> or (ii) use algorithmic defenses during <cit.> or after training <cit.> or during inference <cit.>.
However, none of these methods can withstand all attacks in practice <cit.>.
Existing data sanitation methods have been broken by more covert attacks <cit.>, and existing algorithmic defenses during or after training substantially deteriorate the model's accuracy on clean data <cit.>.
While poisoning attacks have been tested against large models such as CLIP <cit.>, the evaluation of defenses is lagging behind.
No existing defense shows effectiveness against all existing data poisoning attacks, which has called into question whether web-scraped training is at all desirable <cit.>.

Post-Training Defenses. Existing post-training defenses <cit.> are designed to (i) detect whether the model contains a backdoor <cit.> or (ii) disable the backdoor <cit.> by repairing the model using a limited set of trusted (non-poisoned) data.
However, these defenses have studied undetectability and robustness in isolation <cit.>, whereas an attacker wants to design an attack that is both undetectable and robust.
We make fundamental observations on the undetectability/robustness trade-off for a data poisoning attack and show that attackers who over-poison by injecting too many samples become detectable, whereas attackers who under-poison are not robust.
Our findings raise the bar for all data poisoning attackers who need precise control over the number of samples injected during training to maximize both goals.

During training, the model learns a lower-dimensional, latent representation of images known as the latent space.
The latent space encodes salient characteristics relevant to the classification task and can be visualized to interpret the model <cit.>.
While there are methods to enhance latent representations through regularization <cit.> or latent space alignment <cit.>, no method exists to maximize the dissimilarity between two latent spaces to achieve model repair.
Our idea is to find a repaired model with latent representations that are orthogonal to the poisoned model's representations in order to weaken spurious correlations between the trigger and target class using a limited set of clean, trusted data.

Our Defenses. We design two defenses for (i) model repair and (ii) backdoored model detection.
Our model repair works by maximizing latent dissimilarity, which we optimize using a known regularization method called Pivotal Tuning <cit.>.
Pivotal Tuning clones the pre-trained model, referred to as the Pivot and regularizes the poisoned model to (i) maximize the dissimilarity between both latent spaces using a loss function we propose and (ii) minimize the deviation of the repaired model from the Pivot in the parameter space.
Since a data poisoning attacker has no control over the model's parameters, we hypothesize that repaired parameters exist with high proximity to the Pivot's parameters.

Our backdoor detection method reverse-engineers the secret trigger pattern, similar to Neural Cleanse <cit.>, through iterative optimization and, as a result, outputs anomaly scores for each class.
We observe that Neural Cleanse easily gets stuck in local minima due to the model's vulnerability to universal adversarial examples <cit.> and has a high chance of failing at faithfully reconstructing the trigger even for patch-based triggers <cit.> it was designed to defend against.
Our method uses the repaired model to guide the optimization and avoids getting stuck in local minima.
We demonstrate improved detection rates of our method over Neural Cleanse.

Results. Our empirical evaluation shows that Pivotal Tuning is effective and repairs all surveyed backdoored models on CIFAR-10 <cit.> using only 1% of trusted data and 2.5% on ImageNet <cit.> with a maximum deterioration in test accuracy of $2\%$,  outperforming all other surveyed post-training defenses.
By ablating over the number of poisoned samples, we observe a fundamental trade-off between robustness and undetectability that allows crafting more potent defenses.
Interestingly, we find that detection-based defenses such as Neural Cleanse <cit.> are successful at detecting backdoors in larger models but often fail to reconstruct the exact trigger needed to repair the model.
In contrast, on smaller models, Neural Cleanse often fails to detect an attack but can reconstruct the trigger with high fidelity when it is successful (which is useful for repairing the model).
We expand our evaluation to state-of-the-art joint vision-language models <cit.>, and demonstrate the effectiveness and data efficiency of our post-training defenses.

Finally, we propose two adaptive attacks that reveal the limitations of our defenses.
The first attack, called Parameter-Controlled Backdoor (PCB), assumes the attacker has control over the entire training process and embeds the malicious backdoor functionality into a small subset of the model's parameters.
In the second data poisoning attack, the attacker uses a technique we call trigger scattering to maximize robustness (at the expense of higher detectability).

§.§ Contributions

Our key contributions can be summarized as follows.

* We raise the bar for all data poisoning attacks by demonstrating an undetectability/robustness trade-off in the number of poisoned samples.
Attacks without control over the number of poisoned samples are substantially less threatening.

* We propose a method to classify whether a model contains a backdoor using at most $5\%$ of clean, trusted data.

* We propose a backdoor repair method via Pivotal Tuning <cit.> and latent space orthogonalization.
Our defense is the most data efficient and repairs backdoored models on CIFAR-10 and ImageNet using only 1% and 2.5% of the training data with a deterioration of less than 2% test accuracy.

* We are the first to experiment with post-training backdoor defenses on large pre-trained models such as OpenAI's CLIP model <cit.>. Our results indicate that attacks on large models are easier to detect but more robust.

* We propose rigorous, game-based definitions of robustness and detectability for data poisoning.

* We propose two adaptive attacks showing that data poisoning attackers can balance undetectability and robustness using precise control over the number of injected samples.

* We provide source code to reproduce all our experiments, including all hyper-parameters and ablation studies.

§ BACKGROUND

An exemplary training pipeline for deep image classification models from data collection to deployment. Our detection and model repairing defenses are applied post-training.

In this section, we review related work on backdooring deep image classifiers.
We consider a dataset represented by $D \subseteq \mathcal{X}$, where $\mathcal{X}$ denotes the set of images, and a labeling function $\mathcal{O}: \mathcal{X} \rightarrow \mathcal{Y}$ that maps images to labels in the set $\mathcal{Y}$.
The goal of the model is to replicate the labeling function for any image dataset accurately.

§.§ Data Poisoning Attacks

Data poisoning attacks manipulate a subset of the training data to undermine a model's integrity during inference and can be divided into two categories: poison label and clean label attacks.
Poison label attacks give the attacker limited control over images and the labeling function <cit.>, while clean label attacks assume an attacker can only control the images <cit.>.

Poison Label. Badnets <cit.> uses a visible trigger during training and inference and assigns the target class label to all poisoned images.
Adaptive Blend (A-Blend) and Adaptive Patch (A-Patch) <cit.> assign the target label only to a subset of all poisoned images with the goal of increasing the undetectability of their attack.

Clean Label. Shafahi et al. <cit.> propose a clean label attack that stamps poisoned images with an adversarially crafted trigger, assuming access to the attacked model's parameters.
Turner et al.<cit.> present an attack that does not require knowledge of the attacked model's parameters by applying adversarially crafted triggers using a different model trained for the same task.
Refool <cit.> uses naturally occurring trigger patterns, such as reflections, while WaNet <cit.> uses imperceptible warping of the image.
The surveyed clean label attacks differ in only the generation of the trigger pattern but are based on the same principle of poisoning only samples from the target class.

In total, we survey six data poisoning attacks from related work <cit.> (<cit.> propose two backdoors).
For brevity, we moved detailed descriptions of all surveyed attacks, their hyper-parameters, and illustrations of the triggers to <Ref>.

§.§ Data Poisoning Defenses

<Ref> illustrates the training pipeline for a deep image classification model using web-scraped data.
The model's training data is scraped from public sources and can contain a small number of poisoned images <cit.>.
A dataset sanitation process aims to filter or augment poisoned images <cit.> before training the model using a robust training approach <cit.>.
After training, the defender can choose to inspect or repair the model using a limited amount of trustworthy data <cit.> before deployment when they suspect backdooring.
The last line of protection includes defenses implemented during inference, such as anomaly detection systems <cit.>, input purification <cit.> or certifiable defenses <cit.>.

Post-Training Defenses.
We summarize four existing post-training defenses for deep image classification models.
Weight Decay <cit.> is a known regularization method that penalizes the model for learning weights with a large magnitude.
Fine-Pruning <cit.> is a defense against backdoors in convolutional neural networks that uses channel-wise pruning of convolutional filters if the absolute activation of the filter on clean data is low.
Neural Attention Distillation <cit.> (NAD) fine-tunes a teacher model using the poisoned model as a starting point and then distills <cit.> its activations to a student model.
Neural Cleanse <cit.> reconstructs the trigger pattern by backpropagating through the model and optimizing for a mask that behaves like a malicious trigger.
The backdoor can be removed by fine-tuning the model on the reconstructed trigger.

§ THREAT MODEL

This section describes our threat model and provides rigorous security games for the robustness and detectability of an attack.
To this end, we introduce the following functions resembling the training pipeline of a deep classification model depicted in <Ref>.

* Sanitize($D, Y$): Returns a filtered and augmented set of images $\hat{D}$ and labels $\hat{Y}$ given a set of images $D$ and labels $Y$.

* Train($D, Y; \theta_{\text{init}}$): A stochastic training algorithm that tunes a model on image-label pairs. This function optionally accepts pre-trained weights $\theta_{\text{init}}$.

* Post-Training($D, Y, \theta$): Given a model $\theta$ and labelled samples, this method returns modified model parameters $\hat{\theta}$.

* Deploy($\theta$): Given a model $\theta$, this method modifies the inference function of the model.

* Detect($\theta, D, Y$): Outputs $0$ for backdoored models and $1$ for clean models, given a labelled dataset $D, Y$ and model $\theta$.

* $\mathcal{A}(D, \mathcal{O})$: Given images $D$ and a ground-truth labeling function $\mathcal{O}$, this function returns poisoned images and the attack's target labels for each poisoned image.

* Acc($D, Y;\theta$): Given a set of image-label pairs $(D,Y)$ and a labelling function on $D$ parameterized by $\theta$, this function computes the accuracy of the predicted labels against $Y$.

<Ref> summarizes our notation.

Summary of Notation

Notation                             Description

$\theta$                                 Parameters of a deep image classification model

$\mathcal{D}$                            A distribution over images

$D \sim \mathcal{D}^n$                   Draw $n$ independent images $D$ from $\mathcal{D}$

$\mathcal{O}$                            A ground-truth labelling function for images

$\mathcal{A}$                            A procedure denoting an adversary

$\nicefrac{m}{n}$                        Ratio between poisoned and clean samples

$r$                                      The number of trusted, clean image-label pairs

$\eta$                                  The Attack Success Rate (ASR)

$y \gets \mathcal{P}(\vec{x})$           Call $\mathcal{P}$ with arguments $\vec{x}$ and assign result to $y$

Adversary's Capabilities and Goals.
We consider an attacker capable of poisoning a limited number of $m$ samples in the defender's training dataset.
The attacker's goal is to provoke a misclassification of a sample during inference into a pre-defined target class by stamping the secret trigger pattern on the image.
The adversary succeeds when the model predicts the target class deviating from the ground truth (backdoor functionality) while correctly predicting clean images not containing the trigger (benign functionality).
In such a targeted attack, the adversary has the incentive to preserve the model's accuracy to (i) remain stealthy and (ii) maximize the chance that the defender deploys this backdoored classifier.

Defender's Capabilities and Goals.
Our defender controls the entire training pipeline of their model from data collection to deployment as depicted in <Ref>.
We also assume the defender has access to a limited set of trustworthy data with ground-truth labels, which is however insufficient to train an accurate model from scratch.
The availability of limited trustworthy data is a realistic and often-made <cit.> assumption in the context of training large machine learning models on web-scraped data where a subset of sources are more trustworthy than others (e.g., by restricting who can post content to these platforms).
The defender wants to train models with high (i) accuracy and (ii) integrity.
A primary goal of the defender is to (i) repair models suspected of backdooring while limiting the impact on its accuracy and (ii) reliably detect a backdoored model.
A secondary goal of the defender is to minimize the amount of trusted data used in the defense.

§.§ Robustness

The robustness of a data poisoning attack describes its integrity/accuracy trade-off curve: How much loss in utility does repairing a backdoor incur to the model?
In the context of deep image classification, we consider the model's accuracy on an unseen, clean test set and the model's integrity as its accuracy on a set of poisoned images with attacker-chosen target labels.
A defender can mitigate backdooring by modifying multiple steps of the model training pipeline illustrated in   <Ref>, such as the (i) dataset sanitation, (ii) model training, (iii) post-training processes, and (iv) model deployment.
<Ref> encodes the data poisoning robustness game for a given data distribution $\mathcal{D}$, an attack procedure $\mathcal{A}$ and some ratio of poisoned and clean samples $\nicefrac{m}{n}$.

Data Poisoning Robustness Game
[1]
Collect$\mathcal{D}, \mathcal{A}, n, m$ (e.g., web-scraping)
$D_A \sim \mathcal{D}^m$

$\Tilde{D}_A, \Tilde{Y}_A \gets \mathcal{A}(D_A, \mathcal{O})$ Poisoned samples and target labels

$D \sim \mathcal{D}^n$

$\Tilde{Y} \gets \mathcal{O}(D \cup \tilde{D_A}) \text{ if \cleanlabel{clean label} else } \mathcal{O}(D) \cup \tilde{Y}_A$
$D \cup \Tilde{D_A}, \Tilde{Y}$
Robustness$\mathcal{D}, \mathcal{A}, n, m$

$\tilde{D}_{train}, \tilde{Y}_{\text{train}} \gets \textsc{Collect}(\mathcal{D}, \mathcal{A}, n, m)$

$\hat{D}_{\text{train}}, \hat{Y}_{\text{train}} \gets \textsc{Sanitize}(\tilde{D}_{\text{train}}, \tilde{Y}_{\text{train}})$ Filter/Augment

$\theta_0 \gets \textsc{Train}(\hat{D}_{\text{train}}, \hat{Y}_{\text{train}})$
$D_{\text{trust}} \sim \mathcal{D}^r$  Limited trustworthy data

$\theta_1 \gets \textsc{Post-Training}(D_{\text{trust}}, \mathcal{O}(D_{\text{trust}}), \theta_0$) Model repair

$\theta_2 \gets \textsc{Deploy}(\theta_1)$ Modify inference algorithm
$D_{\text{test}} \sim \mathcal{D}$
$\Tilde{D}_{\text{test}}, \Tilde{Y}_{\text{test}} \gets \mathcal{A}(D_{\text{test}}, \mathcal{O})$ Poisoned test set

$A_{CDA} \gets \textsc{Acc}(D_{\text{test}}, \mathcal{O}(D_{\text{test}}); \theta_2)$ Clean data accuracy

$A_{ASR} \gets \textsc{Acc}(\Tilde{D}_{\text{test}}, \Tilde{Y}_{\text{test}}; \theta_2) - \textsc{Acc}(\Tilde{D}_{\text{test}}, \Tilde{Y}_{\text{test}}; \mathcal{O})$
$A_{\text{CDA}}, A_{\text{ASR}}$

The robustness game can be described as follows.
First, the defender collects web-scraped data by sampling $n$ clean images and $m$ poisoned images independently at random (lines 1-4).
We assume the defender collects ground-truth labels for all images in the case of a clean label attack; otherwise, the attacker injects their poisoned target labels into the defender's training dataset (line 5).
After collection, the defender can sanitize the dataset (line 9) and train a model (line 10).
Once trained, the defender post-processes the model using a limited number of $r$ independently sampled, trustworthy data elements with ground-truth labels (lines 11-12) before deploying the model (line 13).
Finally, the game evaluates and returns the utility $A_{\text{CDA}}$ and integrity $A_{\text{ASR}}$ of the model.

Defense Effectiveness. The effectiveness of a defense against a data poisoning attack is the expected trade-off between the attack success rate $A_{\text{ASR}}$ and the model's accuracy on clean data $A_{\text{CDA}}$ when playing the game encoded by <Ref>.

\begin{align*}
\textrm{Succ}_{\text{Repair}} = \mathbb{E} \left [ A_{\text{CDA}} - A_{\text{ASR}}  \right ]
\end{align*}

Consider that a defender can always trivially repair a backdoor in their model by resetting the model weights, which also nullifies the model's utility.
We consider the entire trade-off curve and do not make assumptions for the defender about which point of this trade-off curve is optimal for their use case.

§.§ Detectability

The detectability of a data poisoning attack describes the ability of a defender to predict whether a model has been trained on poisoned data.
Detectability substantially weakens a data poisoning attack because (i) it allows the defender to reject and re-train a model deemed to contain a backdoor, and (ii) it can help a defender in localizing poisoned samples or repair malicious behavior in the model.
The backdoor detection game is formalized by <Ref>.
Note that <Ref> considers post-training detectability of a data poisoning attack which only has access to the pre-trained model weights and a limited amount of trustworthy data (but not the model's training data).

Data Poisoning Detectability Game
[1]
Detectability$\mathcal{D}, \theta_{\text{init}}, n, m, r$
$\tilde{D}_{\text{train}}, \tilde{Y}_{\text{train}} \gets \textsc{Collect}(\mathcal{D}, \mathcal{A}, n, m)$
Poisoned Data
$D_{\text{train}}, Y_{\text{train}} \gets \textsc{Collect}(\mathcal{D}, \mathcal{A}, n+m, 0)$
Clean Data

$\theta_0 \gets \textsc{Train}(\Tilde{D}_{\text{train}}, \Tilde{Y}_{\text{train}}; \theta_{\text{init}})$
$\theta_1 \gets \textsc{Train}(D_{\text{train}}, Y_{\text{train}}; \theta_{\text{init}})$
$b \sim \{0, 1\}$ Unbiased coin flip
$D_{\text{trust}} \sim \mathcal{D}^r$
$p \gets \textsc{Detect}(D_{\text{trust}}, \mathcal{O}(D_{\text{trust}}), \theta_b)$
$A_{\text{detect}} \gets \text{ 1 if } p=b \text{ else 0} $

The data poisoning detectability game can be described as follows.
First, we collect a set of poisoned images with poisoned labels (line 2) and a set of clean images with ground-truth labels (line 3).
Then, we train a poisoned and a clean model $\theta_0, \theta_1$ (lines 4-5), flip an unbiased coin $b$ (line 6), and instantiate the defender's detection algorithm given $r$ clean samples with ground-truth labels (lines 7-8).
We measure and return the correctness of the prediction as $A_{\text{detect}}$ (lines 9-10).
The success of the detectability game is its expected classification accuracy $\mathbb{E} \left [ A_{\text{detect}} \right ]$ when repeating the game.

§ CONCEPTUAL APPROACH

This section first describes our Soft Latent Orthogonalization Loss (SLOL) to regularize a model's latent space and then presents a method based on Pivotal Tuning <cit.> to optimize a regularization constraint with minimal loss in model accuracy.

§.§ Model Repair through the Latent Space

Properties of a Latent Space.

During training, deep image classifiers learn a low-dimensional latent space from high-dimensional images that capture salient image features relevant for the classification task <cit.>.
Latent representations (or short latents) have many useful properties, such as the ability to measure perceptual image similarity <cit.> or guiding image generation models <cit.>.
We focus on two properties of the latent space that can be summarized as follows.

* Images from similar classes have similar latents <cit.>.

* The model learns latent directions that control one or more salient features.
For example, CLIP <cit.> learned a latent direction to interpolate the 'age' attribute for any image (e.g., by turning a lion cub into an adult lion) <cit.>.

It has been observed that images containing a backdoor trigger form a dense cluster in the latent space, which has been used to detect poisoned samples <cit.>.
However, no defense exists that considers latent directions learned by the model to defend against backdoors.
We hypothesize that we can repair the model by enforcing a regularization constraint on the model's latent space by modifying its latents and latent directions.

Objective 1 - Parameter Space. Our objective is to find a repaired model, $\theta^*$, given a (possibly) poisoned model, $\Tilde{\theta}$, and limited trusted data, $D_{\text{trust}}$.
The repaired model should retain its functionality on clean data while behaving differently on poisoned data.
We assume that since a data poisoning attacker cannot control the poisoned model's parameters, $\tilde{\theta}$, there likely exists a set of repaired parameters, $\theta^*$, for a repaired model that is close to the poisoned model's parameters, such that $||\theta^*-\tilde{\theta}||_2 < \varepsilon$ for a small positive $\varepsilon$.
To preserve functionality on clean data, our first objective is to minimize the perturbation in the parameter space applied to the repaired model.

Objective 2 - Latent Space. Our second objective is to regularize the model's latent space to disable functionality not present in the trusted data.
Regularizing the latent space is a known and effective method to improve a model's latent representations <cit.>.
The idea is to maximize the dissimilarity between the latent spaces of the (poisoned) model before tuning and the repaired model.
Since studies have shown that large models such as CLIP encode meaningful functionalities through latent directions <cit.>, we impose two regularization constraints on the latent space.

* We orthogonalize the clusters of the latent space within the repaired model by using a pairwise cosine similarity loss.

* We orthogonalize latent directions across both models. More precisely, given two centroids in the latent space of the poisoned model $\vec{\Tilde{Z_1}}, \vec{\Tilde{Z_2}}$, and the same two centroids (derived using the same images) in the repaired model $\vec{Z_1}, \vec{Z_2}$, we (i) first compute the pairwise latent directions $\vec{\Tilde{A}} = \vec{\Tilde{Z_1}} - \vec{\Tilde{Z_2}}$ and $\vec{A} = \vec{Z_1} - \vec{Z_2}$ and then (ii) assign a high loss if they are non-orthogonal by measuring their cosine similarity.

<Ref> conceptually illustrates our two constraints for the latent space of a poisoned model with parameters $\Tilde{\theta}$.
We initialize the repaired model's parameters $\theta^*$ by cloning $\Tilde{\theta}$ and regularize its latent space to find new latents for each class that are orthogonal to the latents from the poisoned model.

A conceptual illustration of our defense in the latent space. Black and yellow colors refer to the model before and after repair. The repaired model's latent cluster for the class Boar is orthogonalized and pushed away from its origin.

SLOL. We refer to this latent regularization method as the Soft Latent Orthogonalization Loss (SLOL) which is encoded by <Ref>.
<Ref> defines the following three procedures.
The procedure Centroid groups images by their ground-truth label and computes their latents, denoted by $E(\cdot;\theta)$ (lines 2-3).
Then, it returns the centroids for each class (line 4).
Orthogonality computes the aforementioned regularization over all class pairs (lines 7-8) by orthogonalizing latent directions across both models (line 10) and orthogonalizing the centroids within the repaired model (line 11).
Finally, SLOL takes two models $\Tilde{\theta}, \theta^*$ and trustworthy image-label pairs and returns the orthogonality loss (lines 14-16).
We include PyTorch code outlining the matrix operations needed to compute both terms in the Orthogonality procedure efficiently.

Soft Latent Orthogonalization Loss / Model Repair
[1]
Centroids$\theta, D, \mathcal{O}$
$i \gets 1 \textbf{ to } |\mathcal{Y}|$
$H_i \gets \{E(x;\theta) |~x \in D \text{ and } \mathcal{O}(x) = i\}$ Group Latents
$\{\frac{1}{|H_i|} \sum_{h \in H_i} h |~ i \in 1 \textbf{ to } |\mathcal{Y}|$} Class Centroids
Orthogonality$C, \hat{C}$
$l \gets 0$
$i \gets 1 \textbf{ to } |\mathcal{Y}|$
$j \gets 1 \textbf{ to } |\mathcal{Y}|$
$i < j$
$l \gets l + \text{cos}(C_i-C_j, \hat{C}_i-\hat{C}_j)$
$l \gets l + \text{cos}(\hat{C}_i, \hat{C}_j)$

SLOL$\Tilde{\theta}, \theta^*, D, \mathcal{O}$

$C \gets \textsc{Centroids}(\Tilde{\theta}, D, \mathcal{O})$ Frozen model
$\hat{C} \gets \textsc{Centroids}(\theta^*, D, \mathcal{O})$ Trainable model
$\textsc{Orthogonality}(C, \hat{C})$

Pivotal Tuning. We optimize both objectives (in the parameter and latent space) by using a known regularization method for pre-trained models called Pivotal Tuning <cit.>.
Pivotal Tuning has been shown to enforce a regularization constraint in generative models while limiting the deterioration of the model's utility <cit.>.
The frozen model before tuning is also referred to as the Pivot around which the tunable model is optimized.
<Ref> illustrates the optimization procedure given the Pivot $\Tilde{\theta}$, the tunable model $\theta^*$ and one image-label pair $(x, y) \in D$.
We compute the SLOL loss, a task-specific loss (we use the cross-entropy loss for image classification), and the distance in the parameter space between the frozen pivot and the tunable model.
The combined loss can be written as follows for a set of frozen parameters $\Tilde{\theta}$ (the Pivot), a set of tunable parameters $\theta^*$, and a set of trusted images $D$.

\begin{align*}
\mathcal{L} = \text{CE}(M(E(D; \theta^*);\theta^*) + \lambda_{\text{S}} \textsc{SLOL}(\Tilde{\theta}, \theta^*, D, \mathcal{O}(D)) + \lambda_{\text{P}} ||\Tilde{\theta}-\theta^*||_2
\end{align*}

We refer to <Ref> for the hyper-parameters $\lambda_\text{S}, \lambda_{\text{P}}$ that we used for our defense in the evaluation.

An overview of our Pivotal Tuning method for model repair. See <Ref> for a definition of the orthogonality loss. Grey, green, and yellow boxes, respectively, indicate frozen modules, tunable modules, and loss terms.

Contrasting with Existing Work. We highlight that our work is different from previous latent-space repair methods because our method maximizes the dissimilarity of the latent space before and after repairing the model.
Existing work, specifically Neural Attention Distillation (NAD) <cit.>, is based on the idea of maximizing similarity between a student and a (potentially poisoned) teacher.
They create a teacher model by cloning the poisoned model and fine-tuning it for a limited number of epochs on trustworthy data.
Then, they fine-tune the poisoned model with a regularization loss to align (i.e., maximize the similarity) the teacher's and student's latent representations.
However, it has been shown that a teacher containing a backdoor can transfer that backdoor to the student through its latent representations <cit.>.
In summary, while NAD maximizes similarity in the latent space and minimizes similarity in the parameter space, we do the opposite.

§.§ Backdoor Model Detection

Neural Cleanse <cit.> is a method to detect backdoors in deep image classification models.
This method works by attempting to reverse-engineer a trigger pattern and calculating an anomaly score for each class based on the trigger's norm.
The trigger, represented as $T^*$, is added to an image using an operation such as element-wise addition, represented by $\oplus$.
The goal of Neural Cleanse is to update the trigger through an optimization loop and make the model misclassify any image into a target class when adding a trigger.
A high anomaly score indicates that the reverse-engineered trigger has a low L1 norm.
To determine whether an anomaly is detected, the authors suggest using a constant value to threshold the trigger's L1-norm.
We refer to <Ref> in the Appendix for an implementation of Neural Cleanse using our notations.

Issues with Neural Cleanse.
Using the L1-norm as a measure of anomaly can be exploited by attackers who design triggers with L1-norms above the threshold.
For instance, natural reflections <cit.> create triggers with low perceptual distance from the original image but a large L1-norm distance, allowing the attack to remain undetected even when the trigger is successfully reconstructed.
Another issue is that Neural Cleanse can get stuck in local minima, rendering it unable to detect patch-based backdoor attacks <cit.> it was designed to defend against <cit.>.
Interestingly, this can be linked to the vulnerability of deep models to universal adversarial perturbations <cit.>, a type of adversarial attack optimizing the same loss as Neural Cleanse.
<Ref> illustrates this failure case for Neural Cleanse on CIFAR-10 and ImageNet, where it is able to find a trigger with a high success rate for any class, including non-backdoored ones.

Issues with Neural Cleanse related to universal adversarial perturbations <cit.> where a reverse-engineered trigger achieves near-perfect success for a non-poisoned class.

Adding Calibration. We propose a simple improvement over Neural Cleanse to remedy both aforementioned issues.
Our idea is first to repair a model, for instance, using Pivotal Tuning presented in the previous section, and then use the repaired model to guide the optimization procedure used to reverse-engineer a trigger.
We compute the anomaly score as the difference in the success rates of the reverse-engineered trigger causing a misclassification in the model before repair and preserving the correct classification in the model after repair.
Since the models before and after repair are highly similar, we expect that they are vulnerable to many similar universal adversarial perturbations, decreasing the probability that the optimization gets stuck at a local minimum.
Furthermore, we believe that representing an anomaly score as a difference in success rates is more readily interpretable for practitioners and makes it easier to choose a threshold for flagging anomalies.

Calibrated Trigger Inversion / Backdoor Detection
[1]
CNC$\Tilde{\theta}, D_\text{trust}, Y_\text{trust}, D_{\text{test}}, N, \alpha$
$\theta^* \gets \textsc{PivotalTuning}(\Tilde{\theta}, D_{\text{trust}}, Y_{\text{trust}})$ Model Repair
$S \gets \{\}$
$c \gets 1 \textbf{ to } |\mathcal{Y}|$
$T^* \gets \text{Random Initialization}$ Shape of an image
$1 \textbf{ to } N$
$(x,y) \sim D_{\text{trust}} \times Y_{\text{trust}}$ Sample image-label pair
$\Tilde{y} \gets M(E(x \oplus T^*;\Tilde{\theta});\Tilde{\theta})$ Pivotal model
$y^* \gets M(E(x \oplus T^*;\theta^*);\theta^*)$ Repaired model

$g_{T} \gets \nabla_{T^*} (\text{CE}(\Tilde{y}, c) + \text{CE}(y^*, y))$
$T^* \gets T^* - \alpha \cdot \text{Adam}(T^*, g_{T^*})$ Update Trigger
$D_{\text{psn}} \gets \{x \oplus T^*|x \in D_{\text{test}} \}$
$S \gets S \cup \{\textsc{Acc}(D_{\text{psn}}, c; \Tilde{\theta}) - \textsc{Acc}(D_{\text{psn}}, Y; \theta^*) \}$
$\max(S)$  Return highest anomaly score

<Ref> implements our Calibrated Trigger Inversion method to detect backdoored models.
First, we repair the model (line 2), and then, for each class (line 4), we randomly initialize a trigger in the shape of an image (line 5).
We optimize the trigger for $N$ steps by sampling an image-label pair (line 7) and computing the predictions from the model before (the Pivot) and after repair (lines 8-9).
Then we compute the gradient with respect to the trigger to optimize for misclassification in the Pivot and a correct classification in the repaired model and update the trigger using this gradient (lines 10-11).
After $N$ steps, we compute the difference in success rate as the anomaly score (lines 12-13) and return the largest anomaly score to predict backdoored models (line 14).

§.§ Adaptive Attacks

We design two adaptive attacks against our defenses called the (i) Trigger-Scattering Backdoor (TSB) and (ii) Parameter-Controlled Backdoor (PCB).
Both attacks are designed with the goal of maximizing robustness while remaining undetectable.

Trigger-Scattering Backdoor (TSB). Our first adaptive attack uses a technique we refer to as trigger scattering.
The technique involves breaking a single, large trigger into multiple smaller segments and then poisoning each image with only one of these segments during training but utilizing multiple segments during inference.
The trigger segments should be pairwise, perceptually distinct to make the model learn a separate filter for each segment.
This technique does not require the model to learn sample-specific triggers <cit.> and is different from other work <cit.>
in that, we require the model to recognize each individual segment by itself as opposed to showing a large, complex trigger pattern.
For brevity, we move <Ref> implementing TSB into the Appendix.

Parameter-Controlled Backdoor (PCB): This attack assumes an elevated threat model compared to data poisoning.
Here, the attacker has control over the entire training process, not just injecting poisoned samples, which allows them to make arbitrary modifications to any model parameter.
These attacks are sometimes referred to as code poisoning <cit.> or, in certain cases, supply-chain attacks <cit.>.
We investigate this elevated threat model with the goal of evaluating whether an attacker who controls the training process can execute more powerful attacks to evade our defenses.

Parameter-Controlled Backdoor / Adaptive Attacks
[1]
PCB$\theta, D_\text{train}, Y_\text{train}, y_\text{target}, p, N$ Code Posioning
$\Tilde{\theta} \gets \textsc{SampleParams}(\theta, p)$ Randomly sample parameters
$T \gets \text{Random Initialization}$
$i \gets 1 \textbf{ to } N$
$(x, y) \sim D_{\text{train}} \times Y_{\text{train}}$
$\theta^* \gets \Tilde{\theta} \text{ if }i\text{ is even else } \theta$
$\hat{x} \gets (x \oplus T)\text{ if i is even else } x$ Poison if i is even
$\hat{y} \gets y_\text{target} \text{ if }i\text{ is even else } y$
$y_{\text{pred}} \gets M(E(\hat{x}; \theta); \theta)$ Classify image
$g \gets \nabla_{\theta^*}~\text{CE}(y_{\text{pred}}, \hat{y})$ Gradients w.r.t parameters
$\theta^* \gets \theta^* - \alpha \cdot \text{Adam}(\theta^*, g)$
$\theta$ Return poisoned model

The core idea behind PCB is to embed malicious backdoor functionality only within a smaller, randomly sampled subset of the poisoned model's parameters.
PCB exploits the first assumption made by our Pivotal Tuning-based defense that there exists a set of parameters for a repaired model with a low distance to the poisoned model's parameters.
By embedding malicious functionality only in a subset of parameters, we expect that this results in a stronger perturbation to these parameters (because fewer parameters need to encode the same malicious functionality).
To repair such a backdoor, our Pivotal Tuning-based defense would have to target specifically these parameters and apply stronger perturbations (on average) to them than to other parameters.

<Ref> encodes our PCB attack.
The input is a set of model parameters $\theta$ (can be randomly initialized), image-label pairs $D_{\text{train}}$, $Y_{\text{train}}$, a target class $y_{\text{target}}\in \mathcal{Y}$, a fraction for the sampled parameters $p$ and the number of optimization steps $N$.
First, we randomly sample a fraction of $p$ poisoned parameters from the model (line 2) and a trigger pattern (line 3).
Then, we use alternating optimization on the model's parameters.
In a for-loop (line 4), we update the poisoned parameters on poisoned data when the iteration counter is even, and otherwise, we update all model parameters (including the poisoned ones) on clean, non-poisoned data (lines 5-9).
Finally, we return the poisoned model's parameters.

§ EXPERIMENTS

We begin by describing the experimental setup, including datasets, model architectures we use to conduct our experiments, and measured quantities.
Then we present results on the detectability of backdoored models by (i) showing how Neural Cleanse <cit.> compares with our Calibrated Trigger Inversion and (ii) demonstrating a detectability/attack effectiveness trade-off.
Then we show results on the robustness of existing attacks by (i) contrasting our defense with other defenses and (ii) demonstrating a robustness/attack effectiveness trade-off.
Finally, we evaluate two adaptive data poisoning attacks against our defenses.

§.§ Experimental Setup

Datasets. We experiment with two image classification datasets: CIFAR-10 <cit.> and ImageNet <cit.>.
CIFAR-10 consists of 50k images and 10 class labels with a resolution of $32^2$ pixels.
ImageNet consists of 1.23m images and 1k class labels, which we resize and then center-crop to a resolution of $224^2$ pixels.

Model Architectures. We experiment with two model architectures: ResNet-18 <cit.> and a pre-trained CLIP <cit.> model checkpoint released by OpenAI.
We train all models from scratch on CIFAR-10.
Due to the high computational demands for re-training even the smallest versions of CLIP (tens of hours on hundreds of GPUs <cit.>) from scratch, we instead re-use pre-trained checkpoints and perform backdooring by fine-tuning the models on poisoned data.
Torchvision provides checkpoints for ResNet-18[<https://pytorch.org/vision/stable/models.html>] and Huggingface provides checkpoints for CLIP[<https://huggingface.co/openai/clip-vit-base-patch32>].
All models achieve a high test accuracy: $95\%$ on CIFAR-10, $70\%$ on ImageNet with a ResNet-18, and $73\%$ test accuracy by CLIP.

We re-implement all surveyed attacks and defenses from scratch in PyTorch 1.13 and make our framework and pre-trained, backdoored models publicly available to foster reproducibility.
All experiments are evaluated on A100 GPUs with 80Gybte VRAM on a server with 1TByte of RAM.

§.§ Measured Quantities.

Like related work <cit.>, we compute the following metrics.

* Clean Data Accuracy (CDA): The CDA measures a model's utility in terms of its accuracy on clean, unseen data.

* Attack Success Rate (ASR): The ASR measures the model's integrity in terms of its accuracy on poisoned data with attacker-chosen labels. Models with integrity have low ASR.

Attack Quantities. We measure the following quantities of a data poisoning attack in relation to the number of poisoned samples an attacker injected into the poisoned model's training data.

* Attack Effectiveness: To measure the effectiveness of an attack, denoted by $\eta_m$, we estimate the expected ASR $\eta$ when the attacker can poison at most $m$ samples.

* Detectability: For detectability, denoted by $\rho_m$, we calculate the expected ROC AUC $\rho$ of predicting backdoored/non-backdoored models from a balanced set when the attacker can poison at most $m$ samples.

* Robustness: We measure the robustness $\eta$ of an attack-defense pair by its ASR after using a defense.

Defense Quantities. We study a defender who is limited in the availability of trustworthy data because – as outlined in <Ref>, a secondary goal of the defender is to minimize the use of this data.
We measure effectiveness and efficiency for a defender as follows.

* Defense Effectivenes: We measure the effectiveness $\eta$ of an attack-defense pair by the lowest expected ASR when limiting the maximum allowed degradation of the model's accuracy to $\Delta$.
Throughout our paper, we treat effectiveness as a utility/integrity trade-off and refer to a model as repaired when $\eta\leq 5\%$, likely preventing an attacker from executing their attack successfully in practice.

* Data Efficiency: The efficiency of a defense against an attack, denoted $\upsilon_{m}$ is the attack's robustness $\eta$ when limiting the defender's data to at most $m$ clean, labeled images.

Throughout our paper, we limit the maximum acceptable degradation in the CDA that a defense incurs to $\Delta=2\%$.
This means that if the model has an accuracy of $95\%$ of CIFAR-10, we terminate the defense if it deteriorates the model's test accuracy below $93\%$.

§.§ Hyper-Parameter Optimization

To ensure a fair comparison between all defenses, we perform a systematic grid search to determine optimal hyper-parameters for all defenses, simulating how defenders would identify them in practice.
To this end, we use the limited trusted data available to the defender to poison their own model with a BadNets <cit.> attack.
By knowing the trigger, the defenders can measure ASR/CDA pairs while using their defense.
We define a space of valid hyper-parameters, run a grid search and select hyper-parameters yielding the highest defense effectiveness (i.e., lowest remaining ASR $\eta$ with $\Delta=2\%$).
We run a separate grid search for each (i) dataset, (ii) model architecture, and (iii) number of trusted data $r$ available to the defender.
From this point forward, we fix the same optimal defense hyper-parameters for every experiment.
<Ref> provides detailed descriptions of all hyper-parameters and their configurations.
To the best of our knowledge, our approach is the first to systematically determine defense hyper-parameters for evaluating the robustness and detectability of data poisoning attacks.

§.§ Detectability

We begin by studying the detectability of data poisoning attacks relative to the number of injected, poisoned samples.
Our hypothesis is that attackers who increase their attack effectiveness by over-poisoning also increase the detectability of their attack.
To study detectability, we compare Neural Cleanse <cit.> and our Calibrated Trigger Inversion (see <Ref>) against a clean label and poison label attack.
For brevity, we moved <Ref> describing Neural Cleanse into the Appendix.

§.§.§ Ablation Study for Patch-based Attacks.

We first study the detectability of patch-based poisoning attacks, which Neural Cleanse and our method were designed to protect against.
Patch-based attacks stamp a trigger within a restricted region of the image, such as a little square on the image's corner <cit.>.
We create a variant of BadNets that only stamps the trigger on samples from the target class, meaning that the label remains unmodified, and refer to this attack as C-BadNets.
Both attack variants use the same trigger.

Setup. Our goal is to ablate over the number of poisoned samples $m$ to evaluate detectability.
For each $m$, we train $20$ separate models on clean and poisoned data for CIFAR-10 and $3$ models for ImageNet.
We train fewer models on ImageNet due to the higher computational intensity for model training.
Since clean label attacks are less effective <cit.> and each ImageNet class is limited to about 1k samples, we boost poisoned samples with a factor $b$ by artificially repeating them $b$ times during training.
We only have to boost on ImageNet due to the low effectiveness of all clean label attacks in order to meaningfully study the robustness/detectability trade-off.
The trigger is a patch of a downscaled apple () covering less than 1% of the image's total area (e.g., $3^2$ pixels for CIFAR-10).

[CIFAR-10 / BadNets ]
[CIFAR-10 / C-BadNets ]
[ImageNet / BadNets ]
The effectiveness and detectability of a poison label and clean label attack on CIFAR-10 and ImageNet. The shaded area denotes 1 standard deviation. We train 10 separate, poisoned models for each $m$ to measure the attack's detectability in ROC AUC and repeat the experiment three times. On ImageNet, due to high computational intensity, we only train three models and repeat the experiment three times. C-BadNets is a clean label variant of the BadNets attack.

Result Overview. <Ref> shows the number of poisoned samples ($m$) on the x-axis and the attack success rate (ASR) and detectability in ROC AUC on the y-axis.
The green, solid line shows the attack's ASR, whereas the blue and orange dotted lines show the detectability of each attack using (i) our Calibrated Trigger Inversion and (ii) Neural Cleanse <cit.>.
We evaluate detectability against a ResNet-18 model <cit.>.
All plots show that more effective attacks are more easily detectable and that our Calibrated Trigger Inversion strictly outperforms Neural Cleanse in the ability to detect backdoors for every $m$.
We observe that Neural Cleanse performs substantially worse than our approach because it often reverse-engineers a generic trigger that barely resembles the real trigger.
Since many unrelated trigger patterns can already perfectly satisfy the optimization criterion (to achieve a high ASR), Neural Cleanse often gets stuck in local minima.
Our detection strictly improves over Neural Cleanse, and thus when referring to detectability, we only refer to an attack's detectability using our method.

CIFAR-10. <Ref> shows that BadNets achieves a high ASR of $\eta_{5}=0.69$ with a low detectability $\rho_{5}=0.56$ on CIFAR-10.
As the attacker poisons more samples, the ASR increases to a near-perfect value $\eta_{20}=0.99$, but also the attack's detectability  $\rho_{20}=0.80$ increases.
Interestingly, the attack's detectability does not substantially increase once it achieves a near-perfect ASR.
We observe the same phenomenon for C-BadNets, which requires injecting more poisoned samples to achieve a similar effectiveness as its poison label counterpart.
When equalizing for effectiveness, we observe a slightly higher detectability for clean label attacks than for poison label attacks.
For instance, in <Ref>, we measure an effectiveness/detectability pair of $\eta_{750}=0.60/\rho_{750}=0.81$ for C-BadNets, but only $\eta_{5}=0.69/\rho_{5}=0.56$ for BadNets on CIFAR-10.

ImageNet. <Ref> shows the effectiveness and detectability of BadNets on ImageNet.
For BadNets, we measure an effectiveness $\eta_{4000}=0.93$ and a detectability of $\rho_{4000}=0.96$ (i.e., a near-perfect attack detectability).
Even for relatively low effectiveness $\eta_{100}=0.03$, we observe a detectability of $\rho_{100}=0.57$.
Recall that on CIFAR-10, we measure a similar detectability of $\rho_5=0.56$ - but at a much higher attack effectiveness of $\eta_5=0.69$.
We conclude that the same attacks are substantially easier to detect on ImageNet than on CIFAR-10 when equalizing for effectiveness.

§.§.§ Detectability of Complex Triggers.

<Ref> shows the clean data accuracy (CDA), effectiveness, and detectability on CIFAR-10 and ImageNet for all six surveyed data poisoning attacks.
As described in <ref>, these attacks differ from BadNets and C-BadNets in the complexity of the trigger.
By complexity, we refer to the function used to apply the trigger (e.g., computing reflections <cit.> or warping <cit.>).
Since both Neural Cleanse and our Calibrated Trigger Inversion search for a constant trigger that, when added to any image, produces a misclassification, we expect that complex triggers evade detection.
Surprisingly, this is not the case against many complex triggers, and our method yields high detection rates against these types of attacks.

$\mathcal{D}$     Attack     $m$     $b$     CDA     ASR     Detectability

5*valign=m,rotate=90,margin=0cm .4cmCIFAR-10 <cit.>     No Backdoor     0     0    $0.953$    $0.00$    0.50

BadNets <cit.>     50     1    $0.952$     $1.00$     0.76

A-Blend <cit.>     50     1     0.952     0.94     0.65

A-Patch <cit.>     50    1      0.951     1.00     0.73

Adv <cit.>     500    1      0.947     0.98     0.68

Refool <cit.>     500     1     0.954     0.38     0.70

WaNet <cit.>     500     1     0.954     0.29     0.65

5*valign=m,rotate=90,margin=0cm .5cmImageNet <cit.>     No Backdoor     0     0     0.702     0.00     0.50

BadNets <cit.>     $4\,000$     1     0.702     0.97      0.97

A-Blend <cit.>     $4\,000$     1     0.702     0.56     0.95

A-Patch <cit.>     $4\,000$     1     0.701     0.75     1.00

Adv <cit.>     $500$    50     0.705     0.46     0.55

Refool <cit.>     $500$     50     0.694     0.74     1.00

WaNet <cit.>     $500$     50     0.699     0.20     0.63

Listed are all surveyed poison label and clean label attacks, the number of poisoned samples $m$, the boosting factor $b$, the CDA/ASR pair, and the detectability in ROC AUC measured using our Calibrated Trigger Inversion method.

[CIFAR-10 / BadNets ]
[CIFAR-10 / C-BadNets]
[ImageNet / BadNets]
[ImageNet / C-BadNets]
The integrity/utility trade-offs on CIFAR-10 <cit.> and ImageNet <cit.> against a BadNets <cit.> attacker who injected a varying number of poisoned samples.
We record measurements using Pivotal Tuning as a model repair method with $r=1\%$ clean data while measuring the ASR/CDA pair every fixed number of iterations.
For ImageNet, we boost the number of poisoned samples by repeating them $b$ times to obtain a higher attack effectiveness. The shaded region represents one standard deviation.

From <Ref>, we see that all poison label attacks achieve a high, near-perfect ASR on CIFAR-10, whereas the clean label attacks achieve a much lower ASR, despite poisoning $10\times$ more samples.
A notable exception among the clean label attacks is the Adv <cit.> attack, which uses adversarially crafted triggers and achieves an ASR of 0.96 on CIFAR-10.
Our findings corroborate with findings by Fowl et al. <cit.> that adversarial examples <cit.> are effective triggers to use in data poisoning.
We find that our Calibrated Trigger Inversion method has improved detection rates against all attacks on CIFAR-10 of at least $0.65$.
This detectability is even higher on ImageNet, where we observe a near-perfect detectability against all attacks except against WaNet <cit.> and Adv <cit.>.

The lower detectability against WaNet can be explained by its relatively low effectiveness of only $0.20$.
In contrast, Adv achieves a much higher ASR of 0.46 but has a substantially lower detectability of only 0.55.
We conclude that even complex triggers such as natural reflections <cit.> or warping <cit.> are detectable by approaches that reverse-engineer a constant trigger.
We observe this is because images from the poisoned class are more sensitive to perturbations, enabling reverse-engineered triggers with a high success rate.

§.§ Robustness

We evaluate the data efficiency of the five surveyed model repair defenses by limiting the number of trusted image-label pairs available to the defender to $r\in \{1\%, 2.5\%, 5\%\}$ relative to the training dataset.
We study the robustness of the same poisoned model checkpoints used to evaluate detectability in <Ref>.
Since every surveyed post-training defense involves an iterative fine-tuning procedure, we record the ASR/CDA pairs every $50^{\text{th}}$ and $200^{\text{th}}$ iteration on CIFAR-10 and ImageNet to plot the trade-off curves between
the model's integrity and its test accuracy.
We terminate a defense if the CDA deteriorates by more than $\Delta=2\%$.

§.§.§ Ablation Study.
<Ref> shows the integrity/accuracy curves on CIFAR-10 and ImageNet for the BadNets and C-BadNets attacks introduced in <Ref>.
The x- and y-axes show the ASR and CDA.
We show the results of our Pivotal Tuning-based defense against three attacks that vary only in the number of poisoned samples $m$, and the shaded region represents one standard deviation.

CIFAR-10. <Ref> shows that an BadNets attacker who injected more poisoned samples will enhance their attack's robustness.
This means that an attacker who injected $m=2\,000$ instead of only $100$ samples can expect a higher remaining ASR in the repaired model for the same tolerable deterioration of CDA chosen by the defender.
For instance, <Ref> shows that repairing a model that has been poisoned with only $10$ samples can be repaired with a drop of only about $1\%$ CDA, whereas a model poisoned with $2\,000$ samples experience a deterioration in CDA of nearly $2\%$.
We observe a similar phenomenon for C-BadNets in <Ref>, where attackers can increase their attack's robustness by over-poisoning.

ImageNet. <Ref> show the robustness of attacks on ImageNet using $m\in \{1\,000, 2\,000, 10\,000\}$ when the defender is limited to $r=1\%$ trusted image-label pairs.
As for CIFAR-10, we observe that over-poisoning increases the attack's robustness.
Interestingly, <Ref> shows that the defender fails to repair the C-BadNets attack using the most poisoned samples ($m=500, b=50$), but can repair two attacks that use fewer poisoned samples.

§.§.§ Robustness of Complex Triggers.
<Ref> shows the robustness against attacks using more complex triggers, such as natural reflections or image warping, and compares all surveyed defenses by their data efficiency.
We evaluate three settings where the defender is limited to $r\in \{1\%, 2.5\%, 5\%\}$ clean data.
On CIFAR-10, we observe that our defense can repair all models using only $1\%$ of the training data while limiting the decline in test accuracy to at most $\Delta=2\%$.
In comparison, all other defenses fail to repair these models using the same amount of data.
Notably, we observe that simple fine-tuning with weight decay outperforms other defenses.
Surprisingly, we observe that Neural Cleanse often fails to repair a model, even if it reconstructs a trigger with a low L1 norm because the reconstructed triggers are too dissimilar from the real trigger.

On ImageNet, we observe that our defense requires at least $2.5\%$ of clean training data to repair backdoored models.
We observe that weight decay closely matches our defense's data efficiency for $r=1\%$ and occasionally even outperforms our defense (e.g., for the Adv <cit.> attack, where we measure $\eta=0.07$ using our defense whereas weight decay yields $\eta=0.01$).
Our defense is able to repair all poisoned models with the availability of $r=2.5\%$ trusted data.

3c|Ours     3c|Weight Decay <cit.>     3c|Fine-Pruning <cit.>     3c|NAD <cit.>     3c|Neural Cleanse <cit.>

$\mathcal{D}$     Attack     $\upsilon_{1\%}$     $\upsilon_{2.5\%}$     $\upsilon_{5\%}$     $\upsilon_{1\%}$     $\upsilon_{2.5\%}$     $\upsilon_{5\%}$     $\upsilon_{1\%}$     $\upsilon_{2.5\%}$     $\upsilon_{5\%}$     $\upsilon_{1\%}$     $\upsilon_{2.5\%}$     $\upsilon_{5\%}$     $\upsilon_{1\%}$     $\upsilon_{2.5\%}$     $\upsilon_{5\%}$

5*valign=m,rotate=90,margin=0cm .3cmCIFAR-10 <cit.>
BadNets <cit.>     0.03     0.02     0.03     0.13     0.08      0.09     0.69     0.64     0.60     0.98     0.95     0.98     0.13     0.12     0.10

A-Blend <cit.>     0.00     0.00     0.01     0.45     0.13     0.09     1.00     1.00     1.00     1.00     0.24     0.12     0.72     0.84      0.79

A-Patch <cit.>      0.00     0.00     0.00     0.12     0.04     0.08     1.00     1.00     1.00     0.76     0.65     0.54     0.53     0.21     0.11

Refool <cit.>     0.03     0.00     0.02     0.06     0.04     0.06     0.29     0.35     0.38     0.26     0.23     0.18     0.26     0.34     0.33

Adv <cit.>     0.01     0.03     0.01     0.64     0.12     0.13     0.95     0.95     0.95     0.95     0.95     0.95     0.95     0.95     0.95

WaNet <cit.>     0.04     0.02     0.03     0.14     0.06     0.08     0.14     0.08     0.16     0.28     0.26     0.25     0.28     0.28     0.28

5*valign=m,rotate=90,margin=0cm .4cmImageNet <cit.>
BadNets <cit.>     0.20     0.04     0.03     0.23     0.23     0.10     0.94     0.93     0.91     0.69     0.71      0.63     0.83      0.77     0.62

A-Blend <cit.>     0.44     0.05     0.02     0.44     0.03     0.04     0.79     0.79     0.76     0.38     0.36     0.32     0.37     0.50     0.46

A-Patch <cit.>     0.48     0.03     0.01     0.66     0.24     0.23     1.00     0.99     0.98     0.86     0.86     0.84     0.29     0.33     0.15

Adv <cit.>     0.07     0.02     0.01     0.01     0.02     0.01     0.42     0.40     0.31     0.16     0.14     0.10     0.31     0.23     0.22

Refool <cit.>     0.03     0.01     0.00     0.01     0.01     0.00     0.13     0.13     0.13     0.03     0.02     0.02     0.08     0.09     0.08

WaNet <cit.>     0.02     0.01     0.02     0.02     0.02     0.02     0.18     0.17     0.14     0.04     0.04     0.04     0.10      0.08     0.07

The data efficiency of attack-defense pairs on CIFAR-10 and ImageNet.
A value represents the remaining Attack Success Rate (ASR) after implementing a defense, with lower values signifying a more effective defense.
We allow for a maximum tolerable decline in Clean Data Accuracy (CDA) of up to $\Delta=2\%$.
The data efficiency of an attack-defense pair is measured by $\upsilon_r$, which describes the expected robustness $\eta_\Delta$ of an attack against a defense when the defender has $r$ trustworthy images (as a percentage of the original training dataset size).
We evaluate the efficiency of all surveyed post-training defenses against poison label and clean label attacks on a ResNet-18 model.
The attacked model's hyper-parameters are outlined in <Ref>. Due to the high computational demands, we present each data point with just a single repetition.

§.§.§ Adaptive Attacks

A summary of the robustness and detectability of all surveyed attacks, including our adaptive attacks PCB and TSB for different models on CIFAR-10 and ImageNet.

We train ResNet-18 models using our adaptive attacks PCB, described in <Ref> with an adversarially crafted trigger pattern, and the TSB attack described in the Appendix in <Ref>.
For TCB, we use $k=8$ different triggers and inject $m=1000$ and $m=4000$ poisoned image-label pairs on CIFAR-10 and ImageNet, respectively.
We observe that both attacks successfully evade our model repair defense on CIFAR-10 and ImageNet (using $r=5\%$ trusted data) when limiting the maximum tolerable deterioration in test accuracy to $\Delta=2\%$.

<Ref> shows the remaining robustness $\eta$ measured in ASR on the x-axis after applying our Pivotal Tuning-based defense.
The y-axis shows the detectability using our Calibrated Trigger Inversion method.
The TSB attack has a high robustness ($\eta \geq 0.35$) on CIFAR-10 and ImageNet, but also a high detectability.
We observe that TSB remains effective because our defense does not fully repair the model from each trigger but still weakly correlates the trigger with the target class.
Hence, applying many triggers during inference substantially increases the ASR.
We observe high robustness ($\eta \geq 0.75$) at a relatively low detectability ($\rho \leq 0.66$) for the PCB attack.
This is likely for two reasons: (1) The adversarial trigger (the same used in the Adv <cit.> attack) is hard to detect by our detection method.
(2) The PCB attack poisons only a small subset of the model's parameters, exploiting an assumption made by our Pivotal Tuning-based defense that there exist repaired model parameters that are highly similar to the poisoned model's parameters.

§.§.§ CLIP
<Ref> also shows our results for CLIP <cit.> using $r=1\%$ trusted data.
We find substantially higher robustness of all attacks using CLIP when limiting the tolerable decline in CDA to $\Delta=2\%$.
For many attacks, we observe a larger degradation in CDA  (up to $8\%$) to fully repair a CLIP model compared to a smaller ResNet-18 model.
However, even though our defense cannot repair CLIP from most backdoors using $r=1\%$ trusted image-label pairs, it is still able to detect reliably that the model has been backdoored.

§.§ Summary of Results

We enumerate our key results below:

* Our model repair defense is the most data-efficient defense on ImageNet.
Using only 2.5% of clean training data, we can repair all ImageNet models while limiting the test accuracy degradation to at most $\Delta=2\%$.
This makes our defense approximately $2\times$ more data efficient than existing methods.

* Our calibrated trigger inversion (see <Ref>) detects backdooring at a higher rate than Neural Cleanse <cit.>.

* An attack that needs to optimize both undetectability and robustness can be repaired with little loss in test accuracy.

* Detecting whether an attack has occurred is more successful than a complete and faithful reconstruction of the trigger.

* Model repair with the reverse-engineered trigger is often insufficient because, based on our observations, the reversed trigger is often too dissimilar from the real trigger.

* Comparing attacks by their effectiveness against different models reveals that attacks against larger models are more robust (i.e., harder to repair), but also more easily detectable.

§ DISCUSSION AND LIMITATIONS

Below, we discuss extensions and limitations of our work and identify further research motivated by our findings.

General Applicability. We observe a fundamental trade-off between an attack's detectability and its robustness which an attacker can leverage to enhance their defense's effectiveness.
Our work shows that attackers who have to balance both ends of this trade-off are substantially weaker because lowering the detectability of an attack also decreases its robustness.
We bring attention to the point that the effectiveness of defenses against data poisoning attacks may have been underestimated and call for future attacks and defenses to evaluate undetectability and robustness jointly.

Ablation Studies.
Our work focuses primarily on demonstrating a robustness/detectability trade-off in the number of poisoned samples injected by the attack.
While our work features ablations over different model architectures, model sizes, and different trigger patterns, we acknowledge that our study offers only limited insight into the influence of other attack parameters, such as the trigger's complexity, size, or location or whether it is easier to poison certain target classes over others.
For example, <cit.> find that the poisoning ratio $\nicefrac{m}{n}$ can yield mixed effectiveness when varying the number of clean samples $n$.
We study poisoning rates by fixing $n$ and varying only $m$.
Wu et al. <cit.> find that the attacked model's architecture  can strongly influence an attack's effectiveness and robustness.
Our study considers ResNet-18 <cit.> and CLIP <cit.>.
Another line of work <cit.> has studied the connection between the trigger's properties, such as its complexity or size, and an attack's effectiveness.
We present results for multiple triggers but do not ablate over different triggers for each attack.

Need for Models Poisoned From Scratch. We obtain our results attacking ResNet-18 and CLIP on ImageNet by fine-tuning pre-trained, clean model checkpoints on poisoned data (as opposed to poisoning from scratch on CIFAR-10).
Training large models on ImageNet from scratch requires orders of magnitude more computational capacities, specifically around 100 GPU hours <cit.> for ResNet-18 and 10k GPU hours <cit.> for CLIP.
Our study would not have been possible if we had to poison models from scratch due to the prohibitively high training time.
Further research is necessary to explore the effects on an attack's robustness and detectability when poisoning large models trained from scratch.

Advanced Attacks. We do not evaluate all types of targeted attacks, such as subpopulation attacks <cit.>, triggerless poisoning attacks <cit.> that do not require modifying the image during inference, other code poisoning attacks <cit.> or against natural backdoors <cit.>.
Further research is necessary to validate the effectiveness of the surveyed defenses against these attacks.

Using one A100 GPU, it takes 0.5 to 5 hours to poison a ResNet-18 model on CIFAR-10 and ImageNet, and 15 hours for CLIP.
Our Pivotal Tuning defense needs 0.04 to 1 hour on CIFAR-10 and ImageNet, and 5 hours for CLIP.

§ RELATED WORK

We discuss related work on defending from poisoning attacks at various stages in the model training pipeline depicted in <Ref>.

(Dataset Sanitation.) Existing methods use the poisoned, pre-trained model to detect poisoned samples in its training dataset.
<cit.> use Spectral Signatures, <cit.> propose Activation Clustering and more recent detection methods claim improved detection rates by further decomposing latents <cit.>.

(During Training)
<cit.> use differentially private (DP) training <cit.> to shape gradients, DP-InstaHide <cit.> couples DP with data augmentation and <cit.> and RAB <cit.> use a variant of adversarial training to enhance the model's robustness.

(Post-Training.) Apart from the surveyed defenses <cit.>, TABOR <cit.> and <cit.> reverse-engineer triggers using new optimization constraints, DeepSweep <cit.> fine-tunes with augmented data and ANP <cit.> uses adversarial pruning.

(During Deployment.) STRIP <cit.> modifies an image at test time to detect anomalous behavior and other work uses ensembling <cit.> to achieve provable guarantees.
Februus <cit.> removes malicious pixels using a saliency map, and PatchCleanser <cit.> uses input occlusions to defend against patch-based triggers.

Related work studies either undetectability or robustness in isolation.
Instead, our work shows that an attacker needs to balance both objectives, which can enhance a defense's effectiveness.

§ CONCLUSION

Our research reveals a trade-off between the detectability and robustness of a data poisoning attack: Over-poisoning enhances robustness but makes the attack more detectable, while under-poisoning reduces detectability but also weakens the attack's effectiveness and robustness.

Attackers must carefully balance this trade-off, which considerably diminishes their attack's potency, enabling model repair using (i) fewer trusted image-label pairs with (ii) a smaller impact on the model's test accuracy.

We propose two defenses that effectively (i) repair and (ii) detect backdoored models using $1\%$ and $2.5\%$ of clean data on CIFAR-10 and ImageNet, respectively while limiting the decline in test accuracy to at most $2\%$.

We expand our findings to vision-language models, such as CLIP, and observe increased robustness but also a higher detectability.

We propose two adaptive attacks showing our defense's limitations against an attacker who controls the entire training process.

Our work demonstrates that the effectiveness of data poisoning defenses has likely been underestimated by studying undetectability and robustness in isolation instead of treating them as a trade-off the attacker needs to balance.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Abadi et al(2016)]
authorpersonMartin Abadi, personAndy Chu,
personIan Goodfellow, personH Brendan McMahan,
personIlya Mironov, personKunal Talwar, and
personLi Zhang. year2016.
Deep learning with differential privacy. In
booktitleProceedings of the 2016 ACM SIGSAC conference on
computer and communications security. pages308–318.

[Aghakhani et al(2021)]
authorpersonHojjat Aghakhani, personDongyu
Meng, personYu-Xiang Wang, personChristopher
Kruegel, and personGiovanni Vigna.
Bullseye polytope: A scalable clean-label poisoning
attack with improved transferability. In booktitle2021 IEEE
European Symposium on Security and Privacy (EuroS&P). IEEE,

[Biggio et al(2012)]
authorpersonBattista Biggio, personGiorgio
Fumera, personFabio Roli, and personLuca Didaci.
Poisoning adaptive biometric systems. In
booktitleJoint IAPR International Workshops on Statistical
Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern
Recognition (SSPR). Springer, pages417–425.

[Borgnia et al(2021)]
authorpersonEitan Borgnia, personJonas
Geiping, personValeriia Cherepanova, personLiam
Fowl, personArjun Gupta, personAmin Ghiasi,
personFurong Huang, personMicah Goldblum, and
personTom Goldstein. year2021.
Dp-instahide: Provably defusing poisoning and
backdoor attacks with differentially private data augmentations.
journalarXiv preprint arXiv:2103.02079

[Bücker et al(2022)]
authorpersonMichael Bücker, personGero
Szepannek, personAlicja Gosiewska, and
personPrzemyslaw Biecek. year2022.
Transparency, auditability, and explainability of
machine learning models in credit scoring.
journalJournal of the Operational Research Society
volume73, number1 (year2022),

[Carlini et al(2023)]
authorpersonNicholas Carlini, personMatthew
Jagielski, personChristopher A Choquette-Choo,
personDaniel Paleka, personWill Pearce,
personHyrum Anderson, personAndreas Terzis,
personKurt Thomas, and personFlorian Tramèr.
Poisoning Web-Scale Training Datasets is
journalarXiv preprint arXiv:2302.10149

[Carlini and Terzis(2022)]
authorpersonNicholas Carlini and
personAndreas Terzis. year2022.
Poisoning and Backdooring Contrastive Learning. In
booktitleInternational Conference on Learning

[Chen et al(2018)]
authorpersonBryant Chen, personWilka
Carvalho, personNathalie Baracaldo, personHeiko
Ludwig, personBenjamin Edwards, personTaesung Lee,
personIan Molloy, and personBiplav Srivastava.
Detecting backdoor attacks on deep neural networks
by activation clustering.
journalarXiv preprint arXiv:1811.03728

[Chen et al(2022)]
authorpersonRuoxin Chen, personZenan Li,
personJie Li, personJunchi Yan, and
personChentao Wu. year2022.
On Collective Robustness of Bagging Against Data
Poisoning. In booktitleInternational Conference on Machine
Learning. PMLR, pages3299–3319.

[Chen et al(2017)]
authorpersonXinyun Chen, personChang Liu,
personBo Li, personKimberly Lu, and
personDawn Song. year2017.
Targeted backdoor attacks on deep learning systems
using data poisoning.
journalarXiv preprint arXiv:1712.05526

[Deng et al(2009)]
authorpersonJia Deng, personWei Dong,
personRichard Socher, personLi-Jia Li,
personKai Li, and personLi Fei-Fei.
Imagenet: A large-scale hierarchical image
database. In booktitle2009 IEEE conference on computer
vision and pattern recognition. Ieee, pages248–255.

[Doan et al(2020)]
authorpersonBao Gia Doan, personEhsan
Abbasnejad, and personDamith C Ranasinghe.
Februus: Input purification defense against trojan
attacks on deep neural network systems. In booktitleAnnual
Computer Security Applications Conference. pages897–912.

[Doan et al(2021a)]
authorpersonKhoa Doan, personYingjie Lao,
and personPing Li. year2021a.
Backdoor attack with imperceptible input and latent
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Doan et al(2021b)]
authorpersonKhoa Doan, personYingjie Lao,
personWeijie Zhao, and personPing Li.
Lira: Learnable, imperceptible and robust backdoor
attacks. In booktitleProceedings of the IEEE/CVF
International Conference on Computer Vision. pages11966–11976.

[Dolatabadi et al(2022)]
authorpersonHadi Mohaghegh Dolatabadi,
personSarah Erfani, and personChristopher Leckie.
COLLIDER: A Robust Training Framework for Backdoor
Data. In booktitleProceedings of the Asian Conference on
Computer Vision. pages3893–3910.

[Du et al(2020)]
authorpersonMin Du, personRuoxi Jia, and
personDawn Song. year2020.
Robust anomaly detection and backdoor attack
detection via differential privacy. In
booktitleInternational Conference on Learning

[Fowl et al(2021)]
authorpersonLiam Fowl, personMicah
Goldblum, personPing-yeh Chiang, personJonas
Geiping, personWojciech Czaja, and personTom
Goldstein. year2021.
Adversarial examples make strong poisons.
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Frosst et al(2019)]
authorpersonNicholas Frosst, personNicolas
Papernot, and personGeoffrey Hinton.
Analyzing and improving representations with the
soft nearest neighbor loss. In booktitleInternational
conference on machine learning. PMLR, pages2012–2020.

[Gal et al(2022)]
authorpersonRinon Gal, personOr Patashnik,
personHaggai Maron, personAmit H Bermano,
personGal Chechik, and personDaniel Cohen-Or.
StyleGAN-NADA: CLIP-guided domain adaptation of
image generators.
journalACM Transactions on Graphics (TOG)
volume41, number4 (year2022),

[Gao et al(2019)]
authorpersonYansong Gao, personChange Xu,
personDerui Wang, personShiping Chen,
personDamith C Ranasinghe, and personSurya Nepal.
Strip: A defence against trojan attacks on deep
neural networks. In booktitleProceedings of the 35th Annual
Computer Security Applications Conference. pages113–125.

[Ge et al(2021)]
authorpersonYunjie Ge, personQian Wang,
personBaolin Zheng, personXinlu Zhuang,
personQi Li, personChao Shen, and
personCong Wang. year2021.
Anti-distillation backdoor attacks: Backdoors can
really survive in knowledge distillation. In
booktitleProceedings of the 29th ACM International
Conference on Multimedia. pages826–834.

[Geiping et al(2021)]
authorpersonJonas Geiping, personLiam Fowl,
personGowthami Somepalli, personMicah Goldblum,
personMichael Moeller, and personTom Goldstein.
What Doesn't Kill You Makes You Robust (er): How to
Adversarially Train against Data Poisoning.
journalarXiv preprint arXiv:2102.13624

[Goodfellow et al(2014)]
authorpersonIan J Goodfellow, personJonathon
Shlens, and personChristian Szegedy.
Explaining and harnessing adversarial examples.
journalarXiv preprint arXiv:1412.6572

[Gu et al(2017)]
authorpersonTianyu Gu, personBrendan
Dolan-Gavitt, and personSiddharth Garg.
Badnets: Identifying vulnerabilities in the machine
learning model supply chain.
journalarXiv preprint arXiv:1708.06733

[Guo et al(2019)]
authorpersonWenbo Guo, personLun Wang,
personXinyu Xing, personMin Du, and
personDawn Song. year2019.
Tabor: A highly accurate approach to inspecting and
restoring trojan backdoors in ai systems.
journalarXiv preprint arXiv:1908.01763

[Hayase et al(2021)]
authorpersonJonathan Hayase, personWeihao
Kong, personRaghav Somani, and personSewoong Oh.
Spectre: Defending against backdoor attacks using
robust statistics. In booktitleInternational Conference on
Machine Learning. PMLR, pages4129–4139.

[He et al(2016)]
authorpersonKaiming He, personXiangyu
Zhang, personShaoqing Ren, and personJian Sun.
Deep residual learning for image recognition. In
booktitleProceedings of the IEEE conference on computer
vision and pattern recognition. pages770–778.

[Hinton et al(2015)]
authorpersonGeoffrey Hinton, personOriol
Vinyals, personJeff Dean, et al
Distilling the knowledge in a neural network.
journalarXiv preprint arXiv:1503.02531
volume2, number7 (year2015).

[Hong et al(2021)]
authorpersonSanghyun Hong, personNicholas
Carlini, and personAlexey Kurakin.
Handcrafted Backdoors in Deep Neural Networks.
journalarXiv preprint arXiv:2106.04690

[Hong et al(2020)]
authorpersonSanghyun Hong, personVarun
Chandrasekaran, personYiğitcan Kaya,
personTudor Dumitraş, and personNicolas
Papernot. year2020.
On the effectiveness of mitigating data poisoning
attacks with gradient shaping.
journalarXiv preprint arXiv:2002.11497

[Huang et al(2022)]
authorpersonKunzhe Huang, personYiming Li,
personBaoyuan Wu, personZhan Qin, and
personKui Ren. year2022.
Backdoor defense via decoupling the training
journalInternational Conference on Learning
Representations (ICLR) (year2022).

[Huang et al(2020)]
authorpersonW Ronny Huang, personJonas
Geiping, personLiam Fowl, personGavin Taylor, and
personTom Goldstein. year2020.
Metapoison: Practical general-purpose clean-label
data poisoning.
journalAdvances in Neural Information Processing
Systems  volume33 (year2020),

[Ilharco et al(2021)]
authorpersonGabriel Ilharco, personMitchell
Wortsman, personRoss Wightman, personCade Gordon,
personNicholas Carlini, personRohan Taori,
personAchal Dave, personVaishaal Shankar,
personHongseok Namkoong, personJohn Miller,
personHannaneh Hajishirzi, personAli Farhadi, and
personLudwig Schmidt. year2021.

If you use this software, please cite it as below..

[Jagielski et al(2021)]
authorpersonMatthew Jagielski, personGiorgio
Severi, personNiklas Pousette Harger, and
personAlina Oprea. year2021.
Subpopulation data poisoning attacks. In
booktitleProceedings of the 2021 ACM SIGSAC Conference on
Computer and Communications Security. pages3104–3122.

[Jia et al(2021)]
authorpersonJinyuan Jia, personXiaoyu Cao,
and personNeil Zhenqiang Gong.
Intrinsic certified robustness of bagging against
data poisoning attacks. In booktitleProceedings of the AAAI
Conference on Artificial Intelligence, Vol. volume35.

[Jia et al(2022)]
authorpersonJinyuan Jia, personYupei Liu,
personXiaoyu Cao, and personNeil Zhenqiang Gong.
Certified robustness of nearest neighbors against
data poisoning and backdoor attacks. In
booktitleProceedings of the AAAI Conference on Artificial
Intelligence, Vol. volume36. pages9575–9583.

[Khandelwal et al(2022)]
authorpersonApoorv Khandelwal, personLuca
Weihs, personRoozbeh Mottaghi, and personAniruddha
Kembhavi. year2022.
Simple but effective: Clip embeddings for embodied
ai. In booktitleProceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition. pages14829–14838.

[Koh et al(2022)]
authorpersonPang Wei Koh, personJacob
Steinhardt, and personPercy Liang.
Stronger data poisoning attacks break data
sanitization defenses.
journalMachine Learning (year2022),

[Krizhevsky et al(2009)]
authorpersonAlex Krizhevsky, personVinod
Nair, and personGeoffrey Hinton.
CIFAR-10 (Canadian Institute for Advanced

<http://www.cs.toronto.edu/ kriz/cifar.html>

[Kumar et al(2020)]
authorpersonRam Shankar Siva Kumar,
personMagnus Nyström, personJohn Lambert,
personAndrew Marshall, personMario Goertzel,
personAndi Comissoneru, personMatt Swann, and
personSharon Xia. year2020.
Adversarial machine learning-industry
perspectives. In booktitle2020 IEEE security and privacy
workshops (SPW). IEEE, pages69–75.

[Levine and Feizi(2020)]
authorpersonAlexander Levine and
personSoheil Feizi. year2020.
Deep partition aggregation: Provable defense
against general poisoning attacks.
journalarXiv preprint arXiv:2006.14768

[Li et al(2021a)]
authorpersonYuezun Li, personYiming Li,
personBaoyuan Wu, personLongkang Li,
personRan He, and personSiwei Lyu.
Invisible backdoor attack with sample-specific
triggers. In booktitleProceedings of the IEEE/CVF
International Conference on Computer Vision. pages16463–16472.

[Li et al(2021b)]
authorpersonYige Li, personXixiang Lyu,
personNodens Koren, personLingjuan Lyu,
personBo Li, and personXingjun Ma.
Anti-backdoor learning: Training clean models on
poisoned data.
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Li et al(2021c)]
authorpersonYige Li, personXixiang Lyu,
personNodens Koren, personLingjuan Lyu,
personBo Li, and personXingjun Ma.
Neural Attention Distillation: Erasing Backdoor
Triggers from Deep Neural Networks. In
booktitleInternational Conference on Learning

[Li et al(2021d)]
authorpersonYige Li, personXixiang Lyu,
personNodens Koren, personLingjuan Lyu,
personBo Li, and personXingjun Ma.
Neural attention distillation: Erasing backdoor
triggers from deep neural networks.
journalarXiv preprint arXiv:2101.05930

[Liu et al(2018)]
authorpersonKang Liu, personBrendan
Dolan-Gavitt, and personSiddharth Garg.
Fine-pruning: Defending against backdooring attacks
on deep neural networks. In booktitleResearch in Attacks,
Intrusions, and Defenses: 21st International Symposium, RAID 2018, Heraklion,
Crete, Greece, September 10-12, 2018, Proceedings 21. Springer,

[Liu et al(2020)]
authorpersonYunfei Liu, personXingjun Ma,
personJames Bailey, and personFeng Lu.
Reflection backdoor: A natural backdoor attack on
deep neural networks. In booktitleEuropean Conference on
Computer Vision. Springer, pages182–199.

[Loshchilov and Hutter(2017)]
authorpersonIlya Loshchilov and
personFrank Hutter. year2017.
Decoupled weight decay regularization.
journalarXiv preprint arXiv:1711.05101

[Lukas et al(2022)]
authorpersonNils Lukas, personEdward Jiang,
personXinda Li, and personFlorian Kerschbaum.
Sok: How robust is image classification deep neural
network watermarking?. In booktitle2022 IEEE Symposium on
Security and Privacy (SP). IEEE, pages787–804.

[Lukas and Kerschbaum(2023)]
authorpersonNils Lukas and personFlorian
Kerschbaum. year2023.
PTW: Pivotal Tuning Watermarking for Pre-Trained
Image Generators.
journalarXiv preprint arXiv:2304.07361

[Madry et al(2018)]
authorpersonAleksander Madry,
personAleksandar Makelov, personLudwig Schmidt,
personDimitris Tsipras, and personAdrian Vladu.
Towards Deep Learning Models Resistant to
Adversarial Attacks. In booktitleInternational Conference
on Learning Representations.

[May et al(2023)]
authorpersonBrandon B May, personN Joseph
Tatro, personPiyush Kumar, and personNathan
Shnidman. year2023.
Salient Conditional Diffusion for Defending Against
Backdoor Attacks.
journalarXiv preprint arXiv:2301.13862

[Moosavi-Dezfooli et al(2017)]
authorpersonSeyed-Mohsen Moosavi-Dezfooli,
personAlhussein Fawzi, personOmar Fawzi, and
personPascal Frossard. year2017.
Universal adversarial perturbations. In
booktitleProceedings of the IEEE conference on computer
vision and pattern recognition. pages1765–1773.

[Morcos et al(2018)]
authorpersonAri Morcos, personMaithra
Raghu, and personSamy Bengio.
Insights on representational similarity in neural
networks with canonical correlation.
journalAdvances in Neural Information Processing
Systems  volume31 (year2018).

[Nguyen and Tran(2021)]
authorpersonAnh Nguyen and personAnh
Tran. year2021.
WaNet–Imperceptible Warping-based Backdoor
journalarXiv preprint arXiv:2102.10369

[Pang et al(2022)]
authorpersonRen Pang, personZheng Zhang,
personXiangshan Gao, personZhaohan Xi,
personShouling Ji, personPeng Cheng, and
personTing Wang. year2022.
TrojanZoo: Towards Unified, Holistic, and Practical
Evaluation of Neural Backdoors. In booktitleProceedings of
IEEE European Symposium on Security and Privacy (Euro S&P).

[Papernot et al(2018)]
authorpersonNicolas Papernot, personPatrick
McDaniel, personArunesh Sinha, and personMichael P
Wellman. year2018.
Sok: Security and privacy in machine learning. In
booktitle2018 IEEE European Symposium on Security and
Privacy (EuroS&P). IEEE, pages399–414.

[Qi et al(2023)]
authorpersonXiangyu Qi, personTinghao Xie,
personYiming Li, personSaeed Mahloujifar, and
personPrateek Mittal. year2023.
Revisiting the assumption of latent separability
for backdoor defenses. In booktitleInternational Conference
on Learning Representations.

[Qiu et al(2021)]
authorpersonHan Qiu, personYi Zeng,
personShangwei Guo, personTianwei Zhang,
personMeikang Qiu, and personBhavani
Thuraisingham. year2021.
Deepsweep: An evaluation framework for mitigating
DNN backdoor attacks using data augmentation. In
booktitleProceedings of the 2021 ACM Asia Conference on
Computer and Communications Security. pages363–377.

[Radford et al(2021)]
authorpersonAlec Radford, personJong Wook
Kim, personChris Hallacy, personAditya Ramesh,
personGabriel Goh, personSandhini Agarwal,
personGirish Sastry, personAmanda Askell,
personPamela Mishkin, personJack Clark,
et al year2021.
Learning transferable visual models from natural
language supervision. In booktitleInternational conference
on machine learning. PMLR, pages8748–8763.

[Ramesh et al(2022)]
authorpersonAditya Ramesh, personPrafulla
Dhariwal, personAlex Nichol, personCasey Chu, and
personMark Chen. year2022.
Hierarchical text-conditional image generation with
clip latents.
journalarXiv preprint arXiv:2204.06125

[Rasheed et al(2022)]
authorpersonKhansa Rasheed, personAdnan
Qayyum, personMohammed Ghaly, personAla Al-Fuqaha,
personAdeel Razi, and personJunaid Qadir.
Explainable, trustworthy, and ethical machine
learning for healthcare: A survey.
journalComputers in Biology and Medicine
(year2022), pages106043.

[Roich et al(2022)]
authorpersonDaniel Roich, personRon Mokady,
personAmit H Bermano, and personDaniel Cohen-Or.
Pivotal tuning for latent-based editing of real
journalACM Transactions on Graphics (TOG)
volume42, number1 (year2022),

[Sagawa* et al(2020)]
authorpersonShiori Sagawa*, personPang Wei
Koh*, personTatsunori B. Hashimoto, and personPercy
Liang. year2020.
Distributionally Robust Neural Networks. In
booktitleInternational Conference on Learning

[Salman et al(2022)]
authorpersonHadi Salman, personSaachi Jain,
personAndrew Ilyas, personLogan Engstrom,
personEric Wong, and personAleksander Madry.
When does Bias Transfer in Transfer Learning?
journalarXiv preprint arXiv:2207.02842

[Sandoval-Segura et al(2022)]
authorpersonPedro Sandoval-Segura,
personVasu Singla, personLiam Fowl,
personJonas Geiping, personMicah Goldblum,
personDavid Jacobs, and personTom Goldstein.
Poisons that are learned faster are more
effective. In booktitleProceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition.

[Schulth et al(2022)]
authorpersonLukas Schulth, personChristian
Berghoff, and personMatthias Neu.
Detecting Backdoor Poisoning Attacks on Deep Neural
Networks by Heatmap Clustering.
journalarXiv preprint arXiv:2204.12848

[Schwarzschild et al(2021)]
authorpersonAvi Schwarzschild, personMicah
Goldblum, personArjun Gupta, personJohn P Dickerson,
and personTom Goldstein. year2021.
Just how toxic is data poisoning? a unified
benchmark for backdoor and data poisoning attacks. In
booktitleInternational Conference on Machine Learning.
PMLR, pages9389–9398.

[Shafahi et al(2018)]
authorpersonAli Shafahi, personW Ronny
Huang, personMahyar Najibi, personOctavian Suciu,
personChristoph Studer, personTudor Dumitras, and
personTom Goldstein. year2018.
Poison frogs! targeted clean-label poisoning
attacks on neural networks.
journalAdvances in neural information processing
systems  volume31 (year2018).

[Shan et al(2022)]
authorpersonShawn Shan, personArjun Nitin
Bhagoji, personHaitao Zheng, and personBen Y
Zhao. year2022.
Poison forensics: Traceback of data poisoning
attacks in neural networks. In booktitle31st USENIX
Security Symposium (USENIX Security 22). pages3575–3592.

[Steinhardt et al(2017)]
authorpersonJacob Steinhardt, personPang
Wei W Koh, and personPercy S Liang.
Certified defenses for data poisoning attacks.
journalAdvances in neural information processing
systems  volume30 (year2017).

[Tang et al(2021)]
authorpersonDi Tang, personXiaoFeng Wang,
personHaixu Tang, and personKehuan Zhang.
Demon in the Variant: Statistical Analysis of DNNs
for Robust Backdoor Contamination Detection.. In
booktitleUSENIX Security Symposium.

[Tao et al(2022)]
authorpersonGuanhong Tao, personGuangyu
Shen, personYingqi Liu, personShengwei An,
personQiuling Xu, personShiqing Ma,
personPan Li, and personXiangyu Zhang.
Better trigger inversion optimization in backdoor
scanning. In booktitleProceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition.

[Tran et al(2018a)]
authorpersonBrandon Tran, personJerry Li,
and personAleksander Madry. year2018a.
Spectral Signatures in Backdoor Attacks. In
booktitleNeural Information Processing Systems.

[Tran et al(2018b)]
authorpersonBrandon Tran, personJerry Li,
and personAleksander Madry. year2018b.
Spectral signatures in backdoor attacks.
journalAdvances in neural information processing
systems  volume31 (year2018).

[Turner et al(2018)]
authorpersonAlexander Turner, personDimitris
Tsipras, and personAleksander Madry.
Clean-label backdoor attacks.

[Udeshi et al(2022)]
authorpersonSakshi Udeshi, personShanshan
Peng, personGerald Woo, personLionell Loh,
personLouth Rawshan, and personSudipta
Chattopadhyay. year2022.
Model agnostic defence against backdoor attacks in
machine learning.
journalIEEE Transactions on Reliability
volume71, number2 (year2022),

[Van der Maaten and Hinton(2008)]
authorpersonLaurens Van der Maaten and
personGeoffrey Hinton. year2008.
Visualizing data using t-SNE.
journalJournal of machine learning research
volume9, number11 (year2008).

[Wang et al(2019)]
authorpersonBolun Wang, personYuanshun Yao,
personShawn Shan, personHuiying Li,
personBimal Viswanath, personHaitao Zheng, and
personBen Y Zhao. year2019.
Neural cleanse: Identifying and mitigating backdoor
attacks in neural networks. In booktitle2019 IEEE Symposium
on Security and Privacy (SP). IEEE, pages707–723.

[Wang et al([n. d.])]
authorpersonZhenting Wang, personHailun
Ding, personJuan Zhai, and personShiqing Ma.
year[n. d.].
Training with More Confidence: Mitigating Injected
and Natural Backdoors During Training. In booktitleAdvances
in Neural Information Processing Systems.

[Wang et al(2022)]
authorpersonZhenting Wang, personHailun
Ding, personJuan Zhai, and personShiqing Ma.
Training with more confidence: Mitigating injected
and natural backdoors during training.
journalAdvances in Neural Information Processing
Systems  volume35 (year2022),

[Weber et al(2020)]
authorpersonMaurice Weber, personXiaojun
Xu, personBojan Karlas, personCe Zhang, and
personBo Li. year2020.
RAB: Provable Robustness Against Backdoor Attacks.
journalArXiv  volumeabs/2003.08904

[Wenger et al(2022)]
authorpersonEmily Wenger, personRoma
Bhattacharjee, personArjun Nitin Bhagoji,
personJosephine Passananti, personEmilio Andere,
personHeather Zheng, and personBen Zhao.
Finding Naturally Occurring Physical Backdoors in
Image Datasets.
journalAdvances in Neural Information Processing
Systems  volume35 (year2022),

[Wu et al(2022)]
authorpersonBaoyuan Wu, personHongrui Chen,
personMingda Zhang, personZihao Zhu,
personShaokui Wei, personDanni Yuan, and
personChao Shen. year2022.
Backdoorbench: A comprehensive benchmark of
backdoor learning.
journalAdvances in Neural Information Processing
Systems  volume35 (year2022),

[Wu and Wang(2021)]
authorpersonDongxian Wu and personYisen
Wang. year2021.
Adversarial neuron pruning purifies backdoored deep
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Xiang et al(2022)]
authorpersonChong Xiang, personSaeed
Mahloujifar, and personPrateek Mittal.
$\{$PatchCleanser$\}$: Certifiably Robust Defense
against Adversarial Patches for Any Image Classifier. In
booktitle31st USENIX Security Symposium (USENIX Security
22). pages2065–2082.

[Xue et al(2022)]
authorpersonMingfu Xue, personCan He,
personYinghao Wu, personShichang Sun,
personYushu Zhang, personJian Wang, and
personWeiqiang Liu. year2022.
PTB: Robust physical backdoor attacks against deep
neural networks in real world.
journalComputers & Security
volume118 (year2022), pages102726.

[Yao et al(2019)]
authorpersonYuanshun Yao, personHuiying Li,
personHaitao Zheng, and personBen Y Zhao.
Latent backdoor attacks on deep neural networks.
In booktitleProceedings of the 2019 ACM SIGSAC Conference on
Computer and Communications Security. pages2041–2055.

[Zhang et al(2018)]
authorpersonRichard Zhang, personPhillip
Isola, personAlexei A Efros, personEli Shechtman,
and personOliver Wang. year2018.
The unreasonable effectiveness of deep features as
a perceptual metric. In booktitleProceedings of the IEEE
conference on computer vision and pattern recognition.

[Zhang et al(2022)]
authorpersonYuhao Zhang, personAws
Albarghouthi, and personLoris D'Antoni.
BagFlip: A Certified Defense against Data
journalarXiv preprint arXiv:2205.13634

[Zhu et al(2019)]
authorpersonChen Zhu, personW Ronny Huang,
personHengduo Li, personGavin Taylor,
personChristoph Studer, and personTom Goldstein.
Transferable clean-label poisoning attacks on deep
neural nets. In booktitleInternational Conference on
Machine Learning. PMLR, pages7614–7623.

§ APPENDIX

§.§ Model Architectures

We use a ResNet-18 <cit.> architecture with a clean data accuracy of 95% on CIFAR-10.
For ImageNet, we use two models: a pre-trained ResNet-18 model with a clean data accuracy of 70% and OpenAI's pre-trained [<https://huggingface.co/openai/clip-vit-base-patch32>] model checkpoint.

§.§ Summary of the Surveyed Attacks

Three rows showing clean and poisoned samples in the top two rows and a heatmap of their differences on the bottom row. We highlight poison label and clean label attacks and summarize the hyper-parameters in <Ref>.

We re-implement all surveyed data poisoning attacks from scratch and distinguish between clean label and poison label attacks.
This section first describes all surveyed data poisoning attacks and then provides details on the hyper-parameters used for studying the data efficiency of our defense in <Ref>.

BadNets <cit.>: This attack uses a visible, patch-shaped trigger pattern that is stamped on the image.
In our experiments, we downscale the image of a black apple on a white background to the size of the trigger and stamp this trigger on the top left corner of the image.
Throughout our paper, we use a trigger pattern with size $\lfloor \nicefrac{3}{32} \cdot w \rfloor$ pixels relative to the image size $w$, covering less than $1\%$ of the image's total area.

Adaptive Patch <cit.>: This attack is an extension of BadNets that (i) places a grid on top of the trigger with $s\times s$ squares and (ii) randomly occludes squares for each poisoned image.
We use a grid size of $s=4$ with an occlusion probability per square of $50\%$.
The authors distinguish between payload and regularization samples.
A payload sample will be assigned the target label, whereas a regularization sample will receive the ground-truth label (despite containing the trigger).
The authors argue this is done to ensure that the poisoned and target samples are not separable in the poisoned model's latent space.
They use a conservatism ratio to control the ratio between payload and regularization samples.
We use a conservatism ratio of $0.5$.

Adaptive Blend <cit.>: This is a variation of the Adaptive Patch attack, but instead of placing a patch on the top left corner of the image, the trigger is equivalent to the size of the image, but it is superimposed by blending it using an opacity $\alpha\in [0, 1]$.
The authors use the same trick as before and partially occlude the trigger.
We use $s=4$, an occlusion rate of $50\%$, and an opacity $\alpha=0.3$.
This attack uses the same trick as the Adaptive Patch attack to prevent latent separability.
We also use a conservatism ratio of $0.5$ for this attack.

AdvClean <cit.>: This attack assumes access to a similar base model trained on the same task.
For both CIFAR-10 and ImageNet, we use a base model provided by the torchvision package[<https://pytorch.org/vision/stable/models.html>] that has been trained on ImageNet and has a ResNet-18 architecture (the same architecture as the attacked model).
To create a trigger, the attacker generates targeted adversarial examples using the base model against some class (we always use the target class 421 - banister) using the Projected Gradient Descent (PGD) attack  <cit.>.
We iterate PGD for 4 steps using $\epsilon=\nicefrac{8}{255}$.

Refool <cit.>: The idea behind this attack is to use naturally occurring phenomenons as the trigger pattern, such as reflections.
We attack the target class swing and create the ghosting effect as implemented by the  library <cit.>.
In our experiments, we find that using small values for the ghosting intensity does not lead to effective attacks.
For this reason, we use larger values for creating the ghosting effect, specifically $\alpha=1$, which creates some blurry artifacts as illustrated in <Ref>.
We verified that a non-poisoned model still correctly predicts a majority of these images, but note that our chosen hyper-parameters could make the generated samples more easily detectable during dataset sanitation.
The authors do not present results on ImageNet.

WaNet <cit.>: This attack uses a warping effect on the image, which preserves the image's perceptual quality (as opposed to Refool).
We re-use the original implementation by the authors[<https://github.com/VinAIResearch/Warping-based_Backdoor_Attack-release>] but again have to use stronger perturbations as those proposed by the authors.
We use about three times as much noise as the authors use in their paper.
The authors do not show results on ImageNet.

§.§.§ Training Poisoned Models.

On each dataset, we use the same hyper-parameters to train the models on poisoned data for all attacks, mirroring an attack scenario in practice.
We always train on 100% of the training data and poison without replacement, i.e., a clean label attack does not modify the class distribution in the training dataset (i.e., the number of samples per class remains constant).
The hyper-parameters for the trainer are listed below in <Ref>.
As outlined in the paper, we only fine-tune models on ImageNet as opposed to training from scratch on CIFAR-10.

Training Parameters for CIFAR-10 and ImageNet

Parameter                         CIFAR-10     ImageNet

Epochs                            120          10

Momentum                          0.9          0.9

Learning rate (lr)                0.1          0.0001

Weight decay                      0.0005       0.0001

Cosine annealing scheduler        True         False

T_max                            120          -

Optimizer                         SGD          SGD

From Scratch                      True         False

Batch size                        128          128

§.§ Summary of the Surveyed Defenses

Weight Decay <cit.>: We implement fine-tuning with weight decay, which is a well-known regularization technique for deep neural networks that adds a loss-term scaled by $\lambda_w$ to each weight proportional to its Euclidean norm.

Fine-Pruning <cit.>: Fine-Pruning consists of three steps: (1) Measure the absolute activation of every convolutional layer on clean data, (2) select the $\rho\in [0,1]$ channels with the lowest mean absolute activations and perform a channel-wise resetting of all parameters (we reset these weights to zero).
Finally, (3) fine-tune the model on clean data to regain the test accuracy lost to pruning.

Neural Cleanse <cit.>: Neural Cleanse can be instantiated as a backdoor detection or model repair defense.
It consists of three steps that are encoded by <Ref>: (1) reverse-engineer a trigger pattern for each class, (2) compute the L1-norm for each trigger and return the lowest norm as an indication of whether the model has been backdoored and (3) if the objective is to repair the model, fine-tune the model on a mix of clean and poisoned data (using the reverse-engineered trigger).

We calibrate Neural Cleanse by computing the trigger norm for each class, depicted in <Ref>.
As opposed to the authors, who only provide constant bounds for detection which are dataset-specific and cannot be applied to our use-case, whenever we do not use ROC AUC, we apply a statistical test to estimate the probability that this measurement has not been sampled from this normal distribution.

Neural Attention Distillation (NAD) <cit.>: NAD consists of two steps: (1) Clone the poisoned model, which we refer to as a teacher model, and fine-tune it on the trusted data for a limited number of steps.
(2) Fine-tune the poisoned, student model using the attention loss presented in the author's paper on each layer of the neural network to align the latent spaces between the student and teacher models on the trusted data.

§.§ Adaptive Attacks

<Ref> encodes our adaptive Trigger-Scattering Backdoor (TSB) attack.
The data poisoning attack receives an image $x$, a target label $y_{\text{target}}\in \mathcal{Y}$ and $k$ perceptually distinct, secret trigger patterns as input and returns a poisoned image.
We distinguish between two cases: The attack adds a single trigger to poisoned samples that are injected before the training of the model (lines 2-4) and multiple triggers on samples used to exploit the backdoor during inference (lines 5-8).

Neural Cleanse <cit.>
[1]
Neural-Cleanse$\theta, D_{\text{trust}}, Y_{\text{trust}}, N_1, N_2, \alpha$
$S \gets \{\}$ Used to store reverse-engineered triggers
$y_{\text{target}} \gets 1 \textbf{ to } |\mathcal{Y}|$
$T^* \gets \text{Random Initialization}$
$1 \textbf{ to } N_1$
$x \sim D_{\text{trust}}$
$y_{\text{pred}} \gets M(E(x \oplus T^*;\theta);\theta)$
$g_{T^*} \gets \nabla_{T^*} \text{CE}(y_{\text{pred}}, y_{\text{target}})$ Classify
$T^* \gets T^* - \alpha \cdot \text{Adam}(T^*, g_{T^*})$ Update trigger
$S \gets S \cup \{T^*\}$
$i \gets \operatorname{argmin} \{\left\lVert S_i \right\rVert_1|S_i \in S\}$ Find lowest norm
isDetection Detection case
$\left\lVert S_i \right\rVert_1$
$j\gets 1 \textbf{ to } N_2$ Model repair case
$x, y_{\text{target}} \sim D_{\text{trust}} \times Y_{\text{trust}}$
$\hat{x} \gets (x \oplus S_i) \text{ if j is even else } x$
$y_{\text{pred}} \gets M(E(\hat{x};\theta);\theta)$ Classify
$g_{\theta} \gets \nabla_{\theta} \text{CE}(y_{\text{pred}}, y_{\text{target}})$
$\theta \gets \theta - \alpha \cdot \text{Adam}(\theta, g_{\theta})$ Update model

Trigger-Scattering Backdoor (TSB) / Adaptive Attack
[1]
$\mathcal{A}_{\poisonlabel{TSB}}$$x, y_\text{target}, T_1,..,T_k,$
injectSamples Data poisoning / Before training
$i \sim \{1,.., |D_\text{clean}|\}$
$x \oplus T_i, y_\text{target} $Add trigger at a random location
$\Tilde{x} \gets x$
$j \gets 1 \textbf{ to } k$
$\Tilde{x} \gets \Tilde{x} \oplus T_j$ Add all triggers to exploit the backdoor
$\Tilde{x}, y_\text{target}$

§.§ Defense Hyper-Parameter Search

Results of hyper-parameter search for all surveyed defenses on CIFAR-10 and ImageNet using $r=1\%$.

Parameter     CIFAR-10     ImageNet
3cPivotal Tuning

n_steps / $N$     2,000     10,000

opt     sgd     sgd

lr / $\alpha$     0.01     5e-4

slol_lambda / $\lambda_S$     0.05     1e-5

param_lambda / $\lambda_R$     20,000     150,000

weight_decay     0     0

batch_size     128     128

3cNeural Attention Distillation

n_steps / $N$     2,000     8,000

opt     sgd     sgd

lr / $\alpha$     0.01     5e-4

teacher_steps     1,000     1,000

power / $p$     2     2

at_lambda / $\lambda_{\text{at}}$     1,000     1,000

weight_decay     0     0

batch_size     128     128

3cNeural Cleanse

n_steps / $N$     1,000     3,000

opt     sgd     sgd

lr / $\alpha$     0.001     5e-4

steps_per_class / $N_1$           100             200

norm_lambda / $\lambda_N$           2e-2            1e-5

weight_decay       0                    0

batch_size       128                    128

n_steps / $N$         1,000               5,000

opt                sgd                  sgd

lr / $\alpha$          0.001              5e-4

prune_rate / $\rho$        $96\%$             $73\%$

sampled_batches         10             10

weight_decay       0                    0

batch_size       128                    128

n_steps / $N$         1,000               5,000

opt                sgd                  sgd

lr / $\alpha$          0.001              5e-4

weight_decay       0.01                    0.001

batch_size       128                    128

This section summarizes the parameters and our notation used in the hyper-parameter grid search for the five surveyed post-training defenses described in <Ref>.
As described in the main part of the paper, we find optimal hyper-parameters by poisoning the model with a BadNets <cit.> backdoor through fine-tuning with the trusted data.
This mirrors a realistic scenario, where the defender has access to the pre-trained model weights and limited trusted data, meaning they are capable of poisoning their own model.

We search the following space of hyper-parameter for each defense.
For all defenses, we experiment with an SGD and Adam optimizer, a learning rate of $\alpha \in [0.1,..,1\text{e-3}]$ on CIFAR-10 and $\alpha \in [1\text{e-3},..,1\text{e-6}]$ on ImageNet.
For each defense, we ablate over the following hyper-parameters.
For brevity, we only provide the ranges that we checked instead of providing every combination of each parameter.
In total, we evaluate over 20 configurations for each defense on each dataset.

* Pivotal Tuning (Ours): On CIFAR-10, we search over $\lambda_S \in [0.1,..,0.001]$ and on ImageNet we use $\lambda_S \in [1\text{e-2},..,1\text{e-6}]$.
For the parameter regularization $\lambda_R$, we search over $\lambda_R \in [10^6,..,2\cdot 10^7]$ for both datasets.

* Weight Decay: We search over the strength of the weight decay $\lambda_{w}\in \{0.01,..,1\text{e-5}\}$.

* NAD <cit.>:
We search over the steps used to fine-tune the teacher $N_0 \in \{1000,.., 2000\}$, the exponent $p\in \{2,..,4\}$ and the strength for the feature alignment $\lambda_{\text{at}}\in \{100,..,10\,000\}$.

* Neural Cleanse <cit.>: We search over the penalty for the trigger norm $\lambda_N \in \{2\text{e-2},..,1\text{e-6}\}$.

* Fine-Pruning <cit.>: We apply channel-wise pruning with a ratio $\rho \in \{0.5, .., 0.99\}$.

The anomaly scores on ImageNet of reverse-engineered trigger for Neural Cleanse <cit.> (top) and our Calibrated Trigger Inversion method (bottom), measured by the difference in success rate for the model before and after repair using our Pivotal Tuning-based defense with $r=5\%$. We iterate over all classes and store the anomaly score for the reversed trigger for each class. This statistic is useful for detecting anomalies (i.e., backdoored classes).

We reduce the latent space of a poisoned CIFAR-10 model to two dimensions using t-SNE <cit.> while using our Pivotal Tuning-based model repair.
We show the progression of the latent space before, during, and after applying our defense using the Latent Space Orthogonalization Loss (LSOL). Red crosses are poisoned samples, orange squares ('Airplane') are the target class, and we show results for a model poisoned using $m=2\,000$ samples with a BadNets attack.