# Really Unlearned? Verifying Machine Unlearning via Influential Sample Pairs

Heng Xu, Tianqing Zhu*, Lefeng Zhang, Wanlei Zhou  *Tianqing Zhu is the
corresponding author. Heng Xu and Tianqing Zhu are with the Center for Cyber
Security and Privacy and the School of Computer Science, University of
Technology Sydney, Ultimo, NSW 2007, Australia (e-mail:
<EMAIL_ADDRESS>tianqing.zhu@uts.edu.au). Lefeng Zhang and
Wanlei Zhou are with the City University of Macau, Macau (e-mail:
<EMAIL_ADDRESS>wlzhou@cityu.edu.mo).

###### Abstract

Machine unlearning enables pre-trained models to eliminate the effects of
partial training samples. Previous research has mainly focused on proposing
efficient unlearning strategies. However, the verification of machine
unlearning, or in other words, how to guarantee that a sample has been
successfully unlearned, has been overlooked for a long time. Existing
verification schemes typically rely on machine learning attack techniques,
such as backdoor or membership inference attacks. As these techniques are not
formally designed for verification, they are easily bypassed when an
untrustworthy MLaaS undergoes rapid fine-tuning to merely meet the
verification conditions, rather than executing _real_ unlearning. In this
paper, we propose a formal verification scheme, IndirectVerify, to determine
whether unlearning requests have been successfully executed. We design
influential sample pairs: one referred to as _trigger samples_ and the other
as _reaction samples_. Users send unlearning requests regarding trigger
samples and use reaction samples to verify if the unlearning operation has
been successfully carried out. We propose a perturbation-based scheme to
generate those influential sample pairs. The objective is to perturb only a
small fraction of trigger samples, leading to the reclassification of reaction
samples. This indirect influence will be used for our verification purposes.
In contrast to existing schemes that employ the same samples for all
processes, our scheme, IndirectVerify, provides enhanced robustness, making it
less susceptible to bypassing processes.

###### Index Terms:

Machine unlearning, unlearning verification, machine learning as a service,
influence function.

##  I Introduction

Machine learning as a service (MLaaS) is gaining increasing popularity in
machine learning and cloud computing [1]. It enables users to delegate complex
training tasks to a model provider, accessing the model in a black-box manner,
thereby enhancing accessibility and efficiency in machine learning deployment.
However, similar to traditional machine learning models, MLaaS have also
raised various privacy and security concerns, as they can potentially memorize
sensitive information from the training dataset. [2, 3]. Machine unlearning
has attracted attention recently as a solution. It refers to the process of
removing the effect of partial training samples from models [4, 5]. This
technological advancement has received significant attention recently due to
various factors. For example, legislators around the world have introduced
laws and regulations that grant users  the right to be forgotten, which
requires companies to remove user data from their systems after receiving
unlearning requests [6].

Previous machine unlearning studies have primarily focused on efficiently
unlearning samples from models [7, 8, 9, 10, 11]. For example, Bourtoule et
al. [8] proposed a “sharded, isolated, sliced and aggregated”
($\mathbf{SISA}$) unlearning framework, inspired by the data segmentation
technique. Guo et al. [12] calculated the effect of samples that need to be
unlearned and directly adjusted model parameters to compensate for those
effects. However, those schemes usually assume that model providers in MLaaS
are trustworthy and will honestly unlearn samples based on received unlearning
requests [13, 14, 15, 16]. In practice, model providers may not be entirely
trustworthy due to various reasons. For example, they may not comply with
unlearning requests due to the time-consuming nature of unlearning operations
[17, 18]. In this situation, an efficient scheme to verify unlearning is
urgently needed to confirm that model providers indeed unlearn the samples
requested by users. In addition to this, reasonable verification schemes can
also provide model providers with self-verification that they have actually
performed the unlearning operation.

Only a limited number of studies have been proposed to address these issues.
Weng et al.[19] suggested employing trusted hardware to enforce proof of
unlearning but relied on the trusted execution environments on MLaaS. David et
al.[18] and Yu et al. [17] proposed verification schemes based on backdoor
attacks, in order to assess if the model provider successfully performs
unlearning by evaluating the success rate of the backdoor before and after the
unlearning request. Other schemes are based on the difference between the
outputs of training and test samples, and combined with membership inference
attacks (MIAs) for verification [20].

However, those verification schemes did not consider the possibility that
model providers may bypass those verification schemes through a rapid fine-
tuning process. In the case of backdoor-based verification schemes, model
providers can quickly fine-tune a model using unlearning samples with true
labels. This process alters the output of those samples, thereby affecting the
results of the verification. Similarly, for MIAs-based verification schemes,
model providers can bypass it by aligning the outputs of unlearning samples
with those of corresponding test samples. Therefore, there is an urgent need
for an effective and robust formal verification solution.

In this paper, we propose a novel sample-pair verification scheme,
IndirectVerify, to validate the machine unlearning process in MLaaS. First, to
show the existing bypass problems, we propose two methods to bypass existing
backdoor and MIA-based verification schemes. This highlights the unreliability
of current verification methods when faced with untrustworthy MLaaS. Second,
we propose IndirectVerify, which mainly involves constructing influential
sample pairs. This pair includes trigger samples and a reaction sample, where
the presence or absence of trigger samples within the training process will
affect the classification results of the reaction sample. Users can inject
trigger samples into the training dataset to induce misclassification of
reaction samples. When users request the unlearning of trigger samples; if
executed successfully, the model will alter the classification result of the
reaction samples. This change in classification result will be used for our
verification.

Specifically, we propose an optimization-based method for generating trigger
samples that lead to the misclassification of selected reaction samples. To
maintain model performance after embedding trigger samples for verification,
we transform our perturbation objective to a gradient matching problem [21].
The objective is to modify those trigger samples and align their gradients
with those of the targeted reaction samples. This alignment is crucial for our
verification purpose, as minimizing the loss on modified trigger samples
simultaneously reduces the loss on the targeted reaction samples. This will
serve our purpose of classifying reaction samples as our target label when
trigger samples are in the training set. We provide theoretical proofs and
analyses of IndirectVerify based on influence functions, which illustrate how
unlearning each training sample affects reaction samples, providing a
theoretical foundation for IndirectVerify. We also briefly emphasize the
robustness of IndirectVerify by explaining how it resists bypassing methods.

In summary, we make the following contributions.

  * •

We take the first step in addressing the machine unlearning verification
problem when confronted with a potent, untrusted MLaaS. To prove that, we
propose fine-tuning-based methods to bypass existing verification schemes.

  * •

We propose a perturbation-based verification scheme named IndirectVerify. It
involves two samples: trigger samples and a reaction sample. Trigger samples
serve as the sample in the unlearning request, and reaction samples are used
for subsequent verification. The presence or absence of trigger samples can
affect reaction samples.

  * •

We provide theoretical proofs and analyses of IndirectVerify based on
influence functions to illustrate the effectiveness of our scheme. We also
explain its robustness by explaining how it resists fine-tuning-based
bypassing methods.

  * •

We comprehensively evaluate IndirectVerify, considering various perspectives,
datasets, and models. The results demonstrate that IndirectVerify maintains
model utility while achieving robust verification effectiveness.

##  II Preliminary

###  II-A Related Works

####  II-A1 Machine Unlearning

The machine learning community has proposed numerous unlearning schemes in
response to the right to be forgotten. In our previously published survey
paper [4], we conducted a comprehensive survey encompassing recent studies on
machine unlearning. This survey thoroughly summarized several crucial facets,
including: (i) the motivation behind machine unlearning; (ii) the objectives
and desired outcomes associated with the unlearning process; (iii) a novel
taxonomy for systematically categorizing existing machine unlearning schemes
according to their rationale and strategies; and (iv) the characteristics as
well as the advantages and disadvantages of existing verification schemes.

In general, existing unlearning schemes are mainly based on data
reorganization and model manipulation techniques [4]. Data reorganization
refers to the technique by which model providers reorganize the training
dataset to accelerate the unlearning process [7, 8, 9, 10, 11]. Bourtoule et
al. [8] presented a $\mathbf{SISA}$ architecture, similar to the existing
ensemble training methodologies, to achieve the unlearning purpose. Other
similar schemes are used in recommendation tasks [9] and graph data
classification tasks [10]. Graves et al. [11] presented a framework that
leverages random relabeling and fine-tuning strategies to facilitate class-
level unlearning. For the unlearning schemes based on model manipulation, the
model provider aims to realize unlearning operations by directly adjusting the
model’s parameters. Guo et al. [12] proposed a certified removal unlearning
scheme based on influence function [22] and differential privacy [23]. In [13,
14, 15, 16], similar approaches were proposed, which typically involve
directly removing the influence of the corresponding samples from the model.
[24, 25] considered the unlearning requests in the federated learning setting
and proposed model pruning-based methods. Wu et al. [26] and Brophy et al.
[27] proposed parameter replacement-based unlearning schemes that involve
precomputing and subsequently replacing operations.

####  II-A2 Unlearning Verification

Currently, there are no specialized schemes designed for verifying machine
unlearning. The methods employed for this purpose are usually drawn from
various other fields. Generally, existing methods can be broadly classified
into the following two categories: empirical evaluation and theoretical
analysis.

Empirical evaluation schemes usually use attack methods to evaluate how much
information about samples that needs to be unlearned remains within the model.
Model inversion attack [28] has been used in [11], while membership inference
attacks (MIAs) has been used in [11, 24, 13, 14]. Cao et al. [7] performed
data pollution attacks to verify whether the model’s performance, post-
unlearning process, was restored to its initial state. Similar schemes are
adopted in [17, 18] based on the backdoor attack in MLaaS. However, those
schemes, such as model inversion attack [11], MIAs [11, 24, 13, 14], backdoor
[17, 18] can be bypassed by malicious MLaaS. This is because the samples used
in the unlearning request are the same as those used for verification.
Malicious MLaaS can bypass the verification by modifying the output of the
samples directly to meet the verification condition, rather than indeed
executing the unlearning process. We will discuss this in detail in Section
III-B.

In addition to attack-based methods, Brophy et al. [27], Wang et al. [25], Liu
et al. [16], Wu et al. [26] and Golatkar et al. [13, 15, 14] used accuracy-
based verification schemes. However, the essential purpose of machine
unlearning is to remove the effects of partial samples from the model. This
process is similar to removing those samples from the training dataset to the
test dataset. Verification schemes based on accuracy cannot reflect whether
samples have been unlearned. There is no direct relationship between the
unlearned samples’ accuracy and the unlearning scheme’s effectiveness [27, 25,
16, 26, 13, 15, 14]. Baumhauer et al. [8], Golatkar et al. [15, 14] and Liu et
al. [24] measured the similarity of the resulting distributions of pre-softmax
outputs. There are also some efforts to measure the similarity between the
parameter distributions of the model after unlearning and the model after
retraining from scratch [16]. However, those methods have been proven to be
ineffective [29]. Theoretical analysis schemes typically ensure that the
unlearning operation can indeed unlearn sample information [12]. However,
these approaches often come with specific unlearning strategies and are
ineffective when dealing with complex models and large datasets [30].

####  II-A3 Discussion of Related Works

The limitations of existing machine unlearning verification schemes can be
summarized as follows: (1). Attack-based schemes, such as backdoor attacks
[17, 18] and membership inference attacks [11, 24, 13, 14], are susceptible to
malicious MLaaS. MLaaS can bypass those verification schemes based on fine-
tuning processes; (2). Accuracy-based verification schemes usually fail to
reflect the effectiveness of unlearning in removing sample effects [27, 25,
16, 26, 13, 15, 14]. (3). Distribution similarity-based verification schemes
[8, 15, 14, 24], like output distributions and parameter distributions, have
been demonstrated to be ineffective [29]. (4). Theoretical analysis schemes
[12, 30], while ensuring the unlearning results, are often limited by specific
unlearning strategies and show inefficacy with complex models and large
datasets. Consequently, there is a need for mechanisms through which users can
verify whether model providers comply with their unlearning requests.

TABLE I: Notations Notations | Explanation
---|---
$\mathcal{D}$ | The training dataset
$\left(\mathbf{x},y\right)$ | One instance in $\mathcal{D}$
$M$ | The original trained model
$\mathcal{A}(\cdot)$ | The machine learning process
$\mathcal{D}_{u}$ | The unlearning dataset for unlearning process
$\mathcal{D}_{r}$ | The remaining dataset after unlearning process
$\mathcal{U}(\cdot)$ | The machine unlearning process
$M_{u}$ | The model after machine unlearning process
$\mathcal{V}(\cdot)$ | The verification process for machine unlearning
$O(\cdot)$ | The outputs of the model
$\ell\left(M\left(\mathbf{x}\right),y\right)$ | The loss of one instance
$R(\theta)$ | The empirical risk
$\hat{\theta}$ | The empirical risk minimizer
$\mathbf{x}_{t}$ | The test sample used for verification
$\Delta\ell$ | The change of model loss
$\eta(\cdot)$ | The query request
$\upsilon(\cdot)$ | The unlearning request

###  II-B Machine Unlearning and MLaaS

MLaaS is a comprehensive framework that typically comprises two key entities:
data provider and model provider. Data provider uploads their own data to
model providers for model training. We use
$\mathcal{D}=\left\\{\left(\mathbf{x}_{1},y_{1}\right),\left(\mathbf{x}_{2},y_{2}\right),...,\left(\mathbf{x}_{n},y_{n}\right)\right\\}\subseteq\mathbb{R}^{d}\times\mathbb{R}$
to represent the dataset of a data provider, in which each sample
$\mathbf{x}_{i}\in\mathcal{X}$ is a $d$-dimensional vector,
$y_{i}\in\mathcal{Y}$ is the corresponding label, and $n$ is the size of
$\mathcal{D}$. Let $\mathcal{A}$ be a (randomized) learning algorithm that
trains on $\mathcal{D}$ and outputs a model $M$. $M=\mathcal{A}(\mathcal{D})$,
$M\in\mathcal{H}$, where $\mathcal{H}$ is the hypothesis space. Data providers
sometimes wish to remove their partial samples from the trained model and send
an unlearning request to the model provider. Let
$\mathcal{D}_{u}\subset\mathcal{D}$ be a subset of the training dataset, whose
influence one data provider wants to remove. Let its complement
$\mathcal{D}_{r}=\mathcal{D}_{u}^{\complement}=\mathcal{D}/\mathcal{D}_{u}$ be
the remaining dataset. Now, we give the definition of machine unlearning.

######  Definition 1 (Machine Unlearning [7])

Consider a cluster of samples that one data provider wants to remove from the
already-trained model, denoted as $\mathcal{D}_{u}$. An unlearning process
$\mathcal{U}(M,\mathcal{D},\mathcal{D}_{u})$ is defined as a function from an
already-trained model $M=\mathcal{A}(\mathcal{D})$, a training dataset
$\mathcal{D}$, and an unlearning dataset $\mathcal{D}_{u}$ to a model $M_{u}$,
which ensures that the unlearned model $M_{u}$ performs as though it had never
seen $\mathcal{D}_{u}$.

After machine unlearning process, a verification process $\mathcal{V}(\cdot)$
can make a distinguishable check, that is,
$\mathcal{V}\left(M,\mathcal{D}_{u}\right)\neq\mathcal{V}\left(M_{u},\mathcal{D}_{u}\right)$.
$\mathcal{V}(\cdot)$ can be used to measure whether a model provider has
indeed unlearned the requested samples $\mathcal{D}_{u}$. Other symbols that
appear in this paper and their corresponding descriptions are listed in TABLE
I.

##  III Problem Defination

###  III-A Existence Verification Scheme Process

Existing verification schemes in MLaaS follow the process outlined in Figure
1. It mainly consists of a series of steps. Firstly, trigger samples
$\mathcal{D}_{u}$ are selected and incorporated into the training dataset.
Those trigger samples $\mathcal{D}_{u}$ are either generated by the data
provider or obtained from a trusted third party. The model is then trained
based on this aggregated dataset. In the second step, the data provider
queries the output of $\mathcal{D}_{u}$, and the model provider returns the
corresponding outputs. A pre-prepared verification scheme is employed by the
data provider to determine if $\mathcal{D}_{u}$ with the received outputs is
present in the model. Following this verification, the data provider submits
an unlearning request for the $\mathcal{D}_{u}$. Then, the data provider
checks the model’s outputs again, using the same verification scheme to assess
the presence of $\mathcal{D}_{u}$. By comparing the results before and after
unlearning, the data provider can ascertain whether the model has indeed
undergone the unlearning operation.

Figure 1: Existing Verification Scheme Process.

Taking MIAs-based verification schemes [11, 24, 13, 14] and backdoor-based
verification schemes [17, 18] as examples, if $\mathcal{D}_{u}$ that need to
be unlearned are verified as existing in $M$ but not $M_{u}$, that is
$\mathcal{V}\left(O(\mathcal{D}_{u})\right)=true$ and
$\mathcal{V}(O(\mathcal{D}_{u})^{{}^{\prime}})=false$, the data provider can
conclude that the model provider has performed the unlearning operation.

###  III-B Case Studies: the bypass of current verification schemes

Currently, there are two main techniques used for verifying the unlearning
process in MLaaS, namely membership inference attacks (MIAs) [20, 11, 24, 13,
14] and backdoor attacks [17, 18]. Membership inference attacks is a type of
privacy attack that targets machine learning models. It usually requires pre-
training an attack model based on the output of one dataset that closely
resembles the training dataset as much as possible. However, in MLaaS,
fulfilling this condition becomes challenging as it is impractical for data
providers to obtain the output of the entire training dataset. Moreover, since
the performance of MIAs is usually significantly influenced by the type of
dataset and the configuration of the attack model, they usually achieve less-
than-ideal success rates. These characteristics make MIAs not entirely
suitable for verifying the unlearning process. Backdoor attacks involve
manipulating a model during the training phase. Verification schemes based on
backdoor attacks also inherit the drawbacks of the backdoor itself. Backdoors
pose security risks to trained models, which will hinder the widespread
adoption for verification in MLaaS.

In addition to the above issues, the samples used for verification and those
used for unlearning requests are the same in the above two schemes. This gives
the model provider a chance to know what the model output will be after the
unlearning operation. Based on this, they can specify the output of the
samples that are used for verification, thus bypassing the above schemes.
Here, we illustrate how to bypass the MIAs-based verification schemes [11, 24,
13, 14] and backdoor-based verification schemes mentioned in [17, 18].

####  III-B1 Bypassing MIAs-based Schemes

MIAs-based verification schemes [11, 24, 13, 14] determine whether given
samples have been unlearned based on the model’s output. If the model’s output
matches the output from the training dataset, MIAs conclude that those samples
have not been unlearned. On the other hand, if the sample’s output matches
that of the test dataset, MIAs conclude that the sample has been unlearned.
Based on this, after receiving the unlearning request, the model provider can
make the output $\mathcal{O}(\mathcal{D}_{u})$ of received samples consistent
with a sample from the test dataset $\mathcal{O}(\mathcal{D}_{a})$, thus
bypassing MIAs-based schemes. Specifically, they can fine-tune the model using
the following loss rather than executing the unlearning process:

$L=\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)+\lambda||O(\mathcal{D}_{u})-O(\mathcal{D}_{a})||_{2}$
(1)

The front part of Equation 1 is the original training loss, and the latter
part uses $l_{2}$ norm to encourage the similarity between the two outputs.
$O(\mathcal{D}_{u})$ represents the outputs of samples that need to be
unlearned, while $O(\mathcal{D}_{a})$ denotes the outputs of one sample
$\mathcal{D}_{a}$ from the test dataset with the same label as
$\mathcal{D}_{u}$.

####  III-B2 Bypassing Backdoor-based Schemes

For the backdoor-based verification schemes mentioned in [17, 18], the model
provider can use existing backdoor removal schemes [31] to bypass the
verification. Here, we employ a simple yet effective approach that is based on
assigning original true labels and fine-tuning model:

$L=\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
(2)

where $y_{i}$ are the true labels for received unlearning samples
$\mathbf{x}_{i}$. Given that the model provider has access to all the samples
from the data provider, it is straightforward for the model provider to obtain
any label from the dataset and assign true labels to any received samples.
Therefore, the above task is not challenging for the model provider.

Both of the above bypassing schemes are simpler than methods involving
unlearning, offering the model provider a way to reduce resource costs and
meet verification requirements. Detailed evaluation results will be presented
in Section VI-D.

###  III-C Objectives

In this paper, our objective is to address three primary issues associated
with machine unlearning verification. First, we aim to propose an efficient
verification mechanism that enables the data provider to confirm if the model
provider has indeed executed the unlearning process. Second, we need to
consider situations in which the model provider may be untrustworthy,
potentially bypassing the verification process through easily implementable
fine-tuning procedures. For instance, this could involve output alignment or
fine-tuning based on the original true labels, as discussed in Section III-B.
Third and finally, the scheme we propose must minimize any adverse impact on
the model. Formal definitions of these problems are provided below, with key
notations summarized in Table I.

Specifically, the data provider sends its own dataset $\mathcal{D}$ to the
model provider in MLaaS, where the model provider trains a model, $M$, using
one (randomized) learning algorithm $\mathcal{A}(\cdot)$. Upon completing the
training process, and in addition to using the trained model for regular
prediction requests, the data provider can also submit unlearning requests,
aiming to remove the effect of partial samples from the model. While the model
provider faithfully follows the MLaaS specification for its service rewards;
however, it might not fully comply with the unlearning requests to save cost
[17, 18] or aims to serve other commercial purposes [20, 32]. Consequently, we
aim to provide a method for data providers to verify if their samples have
been unlearned from the trained model.

For the purposes of this paper, we assume that data provider can only access
the trained model in a black-box manner, with access to only some of the
samples for verification. Additionally, data provider also has knowledge of
the model architecture employed by its model provider. During both pre- and
post-verification stages, the data provider should perform verification
operations, either independently or through a trusted third party. This
assumption is made without loss of generality, as in many MLaaS scenarios,
normal data providers have limited ability to do such a verification process.
We also assume that the model provider knows who sent the requests to unlearn
which samples and aims to bypass the data provider’s verification based on
fewer computational resources rather than complying with the unlearning
request.

##  IV Methodology

###  IV-A Overview

Model providers can bypass existing verification schemes, since these methods
usually use the same samples for both unlearning requests and verification
process. This allows the model provider to fulfill the verification conditions
based on the samples in the unlearning request by directly altering the output
of those samples. In this paper, we propose a novel scheme, called
IndirectVerify, that consists of influential sample pairs: one referred to as
trigger samples and the other as reaction samples. Users send unlearning
requests concerning trigger samples and use reaction samples to verify the
success of machine unlearning.

Figure 2: Our Verification Scheme

Theoretically, the presence or absence of any training sample in the training
set can influence a given test sample [22]. Some training samples, when
present, may increase the loss of this test sample, while others may decrease
the loss. We refer to the former as harmful samples and the latter as helpful
samples. The fundamental idea of our scheme, IndirectVerify, is to generate
those harmful samples (trigger samples $\mathcal{D}_{u}$ in our scheme) for a
given test sample (reaction sample $\mathbf{x}_{t}$ in our scheme), causing
the loss of this test sample to become sufficiently large, leading to
misclassification. Upon performing the unlearning operation of those harmful
samples, the loss of the test sample decreases, allowing it to be reclassified
correctly. This before-and-after change serves as the basis of our
verification.

As shown in Figure 2, IndirectVerify differs from the one mentioned in Figure
1 mainly in that the samples in unlearning requests and those used for
verification are not the same. In Figure 1, the data provider infers whether
the model provider has performed unlearning based on the outputs of the
trigger samples. In IndirectVerify, we use the reaction samples, rather than
trigger samples. To this end, our scheme, IndirectVerify mainly involves the
following two steps: influential sample pair generation and unlearning process
verification.

###  IV-B Influential Sample Pair Generation

This section will introduce how to generate the influential sample pairs
$(\mathcal{D}_{u},\mathbf{x}_{t})$. We assume that the data provider knows the
model architecture used by the model provider. However, it has no knowledge of
model training process employed by model provider. This includes how the model
is initialized and the hyperparameter settings during training process, such
as learning rate and batch size. The data provider can access limited samples
and seek to modify these samples. The objective is to induce misclassification
of a specific test sample when these changed samples are involved in the
training process. Once these changed images are generated, the data provider
can upload them to model provider for model training, along with the original
training dataset.

Formally, the task of the data provider is as follows:

$\displaystyle\min\ell\left(M_{\theta}\left(\mathbf{x}_{t}\right),y_{\text{target}}\right)\quad$
(3) $\displaystyle\text{ s.t.
}\theta\in\underset{\theta}{\arg\min}(\frac{1}{p}\sum_{i=1}^{p}\ell\left(M\left(\mathbf{x}_{i}+\Delta_{i}\right),y_{i}\right)+$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\frac{1}{n-p}\sum_{i=1}^{n-p}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right))$

Input: One text sample $\mathbf{x}_{t}$ and partial training set
$\mathcal{Q}$, where
$\mathcal{Q}=\left\\{\left(\mathbf{x}_{1},y_{1}\right),\left(\mathbf{x}_{2},y_{2}\right),...,\left(\mathbf{x}_{p},y_{p}\right)\right\\}$,
$p\ll n$.

Output: The final model $M$ and the training completion response.

1 Data provider executes:

2 // Constructing influential sample pairs

3 Modifying $\mathcal{Q}$ to $\mathcal{D}_{u}$ based on
$\mathcal{B}(\Delta,\theta)=1-\frac{\left\langle\Phi,\Psi\right\rangle}{\left\|\Phi\right\|\cdot\left\|\Psi\right\|}$
until results in a misclassification of $\mathbf{x}_{t}$.

4 Send dataset $\mathcal{D}_{u}$ with own dataset to model provider.

5 Query the classification results $O(\mathbf{x}_{t})$ of $\mathbf{x}_{t}$.

6 Verify if the sample with output $O(\mathbf{x}_{t})$ is in the model, and
return $\mathcal{V}\left(O(\mathbf{x_{t}})\right)$.

7 Model provider executes:

8 Model provider train model based on the received dataset.

9 Send a training completion response to the data provider.

10 Upon receiving a prediction request for $\mathbf{x}$

11 return _$O(\mathbf{x})$_.

Algorithm 1 Generation and Injection

Input: Influential sample pairs $(\mathcal{D}_{u},\mathbf{x}_{t})$.

Output: Whether model provider has unlearned samples $\mathcal{D}_{u}$

1 Data provider executes:

2 // Verification Process.

3 Sending the unlearning request regarding samples $\mathcal{D}_{u}$.

4 Re-query the classification results $O(\mathbf{x}_{t})^{{}^{\prime}}$ of
$\mathbf{x}_{t}$.

5 Re-query the result of output $O(\mathbf{x}_{t})^{{}^{\prime}}$, and return
$\mathcal{V}\left(O(\mathbf{x}_{t})^{{}^{\prime}})\right)$.

6 if  _$\mathcal{V}(O(\mathbf{x}_{t}))$ == false and
$\mathcal{V}(O(\mathbf{x}_{t})^{{}^{\prime}})$ == true_ then

7 return _Unlearning process has executed._

8

9 if  _$\mathcal{V}(O(\mathbf{x}_{t}))$ == false and
$\mathcal{V}(O(\mathbf{x}_{t})^{{}^{\prime}})$ == false_ then

10 return _Unlearning process has not executed._

11

12Model provider executes:

13 Upon receiving a prediction request for $\mathbf{x}$

14 return _$O(\mathbf{x})$_.

Algorithm 2 Re-query and Verification

where, $n$ denotes the total number of training samples, and $p$ denotes the
number of samples we intend to modify. For notation simplicity, we assume that
the first $p$ training images are modified by introducing a perturbation
$\Delta_{i}$. The objective is to optimize all $\Delta_{i}$ that the one test
sample $\left(\mathbf{x}_{t},y_{t}\right)$ is reclassified with the newly
assigned target label $y_{target}$.

Inspired by [21], we can transform the above optimization problem to a
gradient matching problem, ensuring that the training gradient aligns with the
target gradient:

$\displaystyle\Phi=$
$\displaystyle\nabla_{\theta}\ell\left(M\left(\mathbf{x}_{t}\right),y_{\text{target}}\right)\approx$
(4)
$\displaystyle\frac{1}{p}\sum_{i=1}^{p}\nabla_{\theta}\ell\left(M\left(\mathbf{x}_{i}+\Delta_{i}\right),y_{i}\right)=\Psi$

This process ensures that the model’s gradient steps, minimizing the training
loss on the modified data, $\Psi$, will also minimize the target loss, $\Phi$,
which will lead to the misclassification of the test sample $\mathbf{x}_{t}$.
Similarly, we also further choose to minimize their negative cosine
similarity, since gradient magnitudes vary significantly across different
training stages:

$\mathcal{B}(\Delta,\theta)=1-\frac{\left\langle\Phi,\Psi\right\rangle}{\left\|\Phi\right\|\cdot\left\|\Psi\right\|}$
(5)

We do this by optimizing $\Delta$ to achieve the misclassification of one
given test sample. Note that if the data provider faces limitations in
computational capacity, any trustworthy third party can undertake this task.
This configuration is quite common in existing verification schemes [17, 18].

###  IV-C Unlearning Process Verification

Section IV-B mainly explains how to generate
$(\mathcal{D}_{u},\mathbf{x}_{t})$ such that the presence of samples
$\mathcal{D}_{u}$ will affect $\mathbf{x}_{t}$, leading to the
misclassification of $\mathbf{x}_{t}$. In this section, we will describe how
to implement the verification of machine unlearning based on generated
$(\mathcal{D}_{u},\mathbf{x}_{t})$.

####  IV-C1 Generation and Injection

For the purpose of unlearning verification, the data provider needs to
generate the influential sample pairs, and then send the trigger samples along
with their own dataset to MLaaS for model training. Then, the data provider
sends reaction samples $\mathbf{x_{t}}$ and asks the model provider what the
output of those samples is. The model provider returns the outputs
$O(\mathbf{x_{t}})$ of those samples. After that, the data provider uses a
pre-prepared verification scheme to check whether the sample with the output
$O(\mathbf{x_{t}})$ is in the model. The verification scheme will output a
result of $\mathcal{V}\left(O(\mathbf{x_{t}})\right)$. In Algorithm 1, we
provide a detailed process for sample generation and injection.

In Algorithm 1, we begin by selecting a set of samples $\mathcal{Q}$.
Following that, we modify those samples $\mathcal{Q}$ based on the Equation 5,
resulting in the misclassification of another test sample $\mathbf{x}_{t}$
(lines 3). It is important to note that these optimization processes do not
alter the labels of $\mathcal{Q}$; they merely introduce minor perturbations
to each sample. This minimizes the impact on the model’s performance as much
as possible. The data provider then sends the modified samples, along with the
original training data, to the server, where the model is trained based on
this combined dataset (line 8).

####  IV-C2 Sending and Executing Unlearning

To request unlearning, the data provider sends an unlearning request regarding
samples $\mathcal{D}_{u}$. The model provider locates and deletes samples
$\mathcal{D}_{u}$ from the dataset and executes the unlearning process.

####  IV-C3 Re-query and verification

After sending the unlearning request, the data provider asks again what the
output of $\mathbf{x_{t}}$ is. The model provider returns the outputs
$O(\mathbf{x_{t}})^{{}^{\prime}}$. The data provider, once again, uses the
verification scheme to check whether the sample under the output
$O(\mathbf{x_{t}})^{{}^{\prime}}$ is in the model. Normally, the verification
scheme will output a result of
$\mathcal{V}\left(O(\mathbf{x_{t}})^{{}^{\prime}}\right)$. Data provider
compares the relationship between $\mathcal{V}\left(O(\mathbf{x_{t}})\right)$
and $\mathcal{V}\left(O(\mathbf{x_{t}})^{{}^{\prime}}\right)$, thereby
determining whether the model provider has executed the unlearning operation.
In Algorithm 2, we provide a detailed process for this process.

In lines 5–6 in Algorithm 1, the data provider initiates the verification
process by querying the output for $\mathbf{x}_{t}$. Normally, the output
results a misclassification result, $\mathcal{V}(O(\mathbf{x}_{t}))$ == false.
Subsequently, in line 3 in Algorithm 2, the data provider will send an
unlearning request, followed by another query for the output related to
$\mathbf{x}_{t}$ (lines 4-5). If $\mathbf{x}_{t}$ is classified correctly
($\mathcal{V}\left(O(\mathbf{x_{t}})^{{}^{\prime}}\right)$ == True), it
confirms that the model provider has indeed performed the unlearning operation
(lines 6-7); otherwise, it suggests that the model provider has not executed
the unlearning operation (lines 8-9). IndirectVerify can be further combined
with methods, such as hypothesis testing [18], to enhance its robustness and
verification effectiveness. This will be a focus of our future research.

##  V Theoretical Analysis

###  V-A Effectiveness Analysis

IndirectVerify is mainly based on the idea that the presence or absence of
trigger samples affects the classification results of reaction samples. In
this section, we would like to provide theoretical analyses to illustrate why
IndirectVerify is effective. We formalize the change of classification result
of reaction samples as the change of loss: when given a test sample, how can
we calculate the effect of each training sample’s presence on the change of
model’s loss regarding this test sample? We will give a closed-form expression
for this. Based on this expression, we will illustrate how IndirectVerify
minimizes this effect to achieve verification purposes.

Formally, we use
$\mathcal{D}=\left\\{\left(\mathbf{x}_{1},y_{1}\right),\left(\mathbf{x}_{2},y_{2}\right),...,\left(\mathbf{x}_{n},y_{n}\right)\right\\}$
to represent the training dataset. Let
$\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$ be the loss, and define
$R(\theta)\stackrel{{\scriptstyle\text{ def
}}}{{=}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
as the empirical risk. Assuming the empirical risk minimizer is given by:

$\hat{\theta}\stackrel{{\scriptstyle\text{ def
}}}{{=}}\arg\min\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
(6)

Considering how the model’s loss of a test sample would change if we were to
remove samples $\mathcal{D}_{u}$ from the training dataset, Formally, this
change is

$\Delta\ell=\ell\left(M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}\left(\mathbf{x}_{t}\right),y_{t}\right)-\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)$
(7)

where

$\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}\stackrel{{\scriptstyle\text{
def
}}}{{=}}\arg\min_{\mathcal{D}_{u}\notin\mathcal{D}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
(8)

Directly calculating the above is also prohibitively slow, since retraining
the model each time after removing one training sample and observing how the
loss for specific samples changes is not feasible. Inspired by the work of
influence function [22], we can compute the above change if $\mathcal{D}_{u}$
were upweighted by some small $\epsilon$, giving us new parameters

$\displaystyle\hat{\theta}_{\epsilon,\mathcal{D}_{u}}\stackrel{{\scriptstyle\text{
def }}}{{=}}$ $\displaystyle\arg\min(R(\theta)+\epsilon
L(M\left(\mathcal{D}_{u}\right),y_{i}))$ (9)
$\displaystyle\stackrel{{\scriptstyle\text{ def }}}{{=}}$
$\displaystyle\arg\min(\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
$\displaystyle+\epsilon\sum_{\mathbf{x}_{i}\in\mathcal{D}_{u}}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right))$

Then the change $\Delta\ell$ will be

$\Delta\ell=\ell\left(M_{\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}\left(\mathbf{x}_{t}\right),y_{t}\right)-\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right),$
(10)

when we consider that $\mathcal{D}_{u}$ is upweighted by some small
$\epsilon$. Note that, as $\hat{\theta}$ doesn’t depend on $\epsilon$,
$\Delta\ell$ can be expressed as the impact of changes in $\epsilon$ on the
loss, that is:

$\displaystyle\Delta\ell$
$\displaystyle=\left.\frac{\mathrm{d\ell}\left(M_{\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}\left(\mathbf{x}_{t}\right),y_{t}\right)}{\mathrm{d}\epsilon}\right|_{\epsilon=0}$
(11)
$\displaystyle=\left.\frac{\partial\ell\left(M_{\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}\left(\mathbf{x}_{t}\right),y_{t}\right)}{\partial\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}\frac{\partial\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}{\partial\epsilon}\right|_{\epsilon=0}$
$\displaystyle=\left.\nabla_{\theta}\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)^{\top}\frac{d\hat{\theta}_{\epsilon,\mathcal{D}_{u}}}{d\epsilon}\right|_{\epsilon=0}$

Since $\hat{\theta}_{\epsilon,\mathcal{D}_{u}}$ is a minimizer of empirical
risk, that is:

$\nabla
R_{\mathcal{D}}\left(\hat{\theta}_{\epsilon,\mathcal{D}_{u}}\right)+\epsilon\nabla
L_{\mathcal{D}_{u}}\left(\hat{\theta}_{\epsilon,\mathcal{D}_{u}}\right)=0$
(12)

Meanwhile, since
$\hat{\theta}_{\epsilon,\mathcal{D}_{u}}\rightarrow\hat{\theta}$ as
$\epsilon\rightarrow 0$, we can conduct a Taylor expansion of the left-hand
side:

$\displaystyle{[\nabla R_{\mathcal{D}}(\hat{\theta})+\epsilon\nabla
L_{\mathcal{D}_{u}}(\hat{\theta})]+}$ (13)
$\displaystyle{\left[\nabla^{2}R_{\mathcal{D}}(\hat{\theta})+\epsilon\nabla^{2}L_{\mathcal{D}_{u}}(\hat{\theta})\right]\Delta_{\epsilon}}\approx
0,$

where we have omitted terms of order
$o\left(\left\|\Delta_{\epsilon}\right\|\right)$.

Finding the solution for $\Delta_{\epsilon}$ yields:

$\begin{gathered}\Delta_{\epsilon}\approx-\left[\nabla^{2}R_{\mathcal{D}}(\hat{\theta})+\epsilon\nabla^{2}L_{\mathcal{D}_{u}}(\hat{\theta})\right]^{-1}\\\
{[\nabla R_{\mathcal{D}}(\hat{\theta})+\epsilon\nabla
L_{\mathcal{D}_{u}}(\hat{\theta})].}\end{gathered}$ (14)

As $\hat{\theta}$ minimizes $R$, we have $\nabla R(\hat{\theta})=0$.
Neglecting terms of order $o(\epsilon)$, we obtain:

$\Delta_{\epsilon}\approx-\nabla^{2}R(\hat{\theta})^{-1}\nabla
L_{\mathcal{D}_{u}}(\mathbf{x}_{i},\hat{\theta})\epsilon.$ (15)

We further make the assumption that the empirical risk is both twice-
differentiable and strongly convex, and the following

$H_{\hat{\theta}}\stackrel{{\scriptstyle\text{ def
}}}{{=}}\nabla^{2}R(\hat{\theta})=\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}^{2}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$
(16)

exists and is positive-definite. Therefore, we conclude that:

$\left.\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}\right|_{\epsilon=0}=-H_{\hat{\theta}}^{-1}\nabla
L_{\mathcal{D}_{u}}(\mathbf{x}_{i},\hat{\theta})$ (17)

Given that removing a sample is equivalent to upweighting it by
$\epsilon=-\frac{1}{n}$, we can linearly estimate the change in loss resulting
from the removal of $\mathcal{D}_{u}$ without retraining the model by
computing:

$\Delta\ell\approx-\frac{1}{n}\nabla_{\theta}\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)^{\top}H_{\hat{\theta}}^{-1}\nabla
L_{\mathcal{D}_{u}}(\mathbf{x}_{i},\hat{\theta}).$ (18)

$\Delta\ell$ describes how, when a test sample $\mathbf{x}_{t}$ is provided,
removing a training sample $\mathbf{x}_{i}$ influences the loss of
$\mathbf{x}_{t}$. Consider any training dataset, with no external manipulation
involved; if we want to find influential sample pairs, we can first identify
misclassified test samples, satisfying the criteria:

$\displaystyle M_{\hat{\theta}}(\mathbf{x}_{t})$
$\displaystyle=\hat{y_{t}}\neq y_{t},$ (19) s.t.
$\displaystyle\hat{\theta}=\arg\min_{\mathcal{D}_{u}\in\mathcal{D}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$

Subsequently, we can index the training dataset to find the sample whose
removal minimizes the loss of that test sample, that is, $\min(\Delta\ell)$,
and thereby:

$\displaystyle M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}$
$\displaystyle(\mathbf{x}_{t})=\hat{y_{t}}=y_{t},$ (20) s.t.
$\displaystyle\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}=\arg\min_{\mathcal{D}_{u}\notin\mathcal{D}}\frac{1}{n}\sum_{i=1}^{n}\ell\left(M\left(\mathbf{x}_{i}\right),y_{i}\right)$

However, directly finding the influential sample pairs based on the above
equation is very difficult, especially when faced with complex models and
enormous training data (see the experimental results in Figure 6c).

To ensure the persistent existence of such influential sample pairs, we would
like to directly modify specific samples to $\min(\Delta\ell)$. That is, for a
given reaction sample, we want to modify a set of trigger samples such that
removing those trigger samples will:

$\displaystyle\min\left(\ell\left(M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}\left(\mathbf{x}_{t}\right),y_{t}\right)-\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)\right)$
(21)
$\displaystyle\text{s.t.}M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}(\mathbf{x}_{t})==y_{t},M_{\hat{\theta}}(\mathbf{x}_{t})\neq
y_{t}$

In general, minimizing $\Delta\ell$ can contain two methods: maximizing
$\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)$ or
minimizing
$\ell\left(M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}\left(\mathbf{x}_{t}\right),y_{t}\right)$.
Therefore, we first choose a test sample that is classified correctly, which
ensures minimizing
$\ell\left(M_{\hat{\theta}_{\mathcal{D}_{u}\notin\mathcal{D}}}\left(\mathbf{x}_{t}\right),y_{t}\right)$.
Then, perturb some training samples, which will maximize
$\ell\left(M_{\hat{\theta}}\left(\mathbf{x}_{t}\right),y_{t}\right)$. This
perturbation is designed to induce a high loss for the associated test sample
and result in misclassification. Subsequently, upon removing these modified
samples from the model, we will achieve $\min(\Delta\ell)$, facilitating the
recovery of the original labels for the given test sample.

Figure 3: The Illustration of Determinism for Existing Verification Schemes.
Figure 4: The Illustration of Determinism for Our Verification Schemes.

###  V-B Robustness Analysis

Our analysis mainly focuses on how to ensure the determinism of the output of
the test sample $\mathbf{x}_{t}$ used for verification, as well as the
confidentiality of the relationship between $\mathcal{D}_{u}$ and
$\mathbf{x}_{t}$. We respectively use $\eta(\cdot)$ and $\upsilon(\cdot)$ to
represent prediction and unlearning requests.

  * •

The Determinism of the Output of the Verification Test Sample $\mathbf{x}_{t}$
is Achieved. Existing verification schemes, such as [11, 24, 13, 14, 17, 18],
typically involve sending $\upsilon(\mathcal{D}_{u})$ and then observing
$\eta(\mathcal{D}_{u})$. In this situation, as shown in Figure 3, the model
provider can introduce various fine-tuning strategies to ensure that the fine-
tuned model’s outputs bypass the verification process without any effect on
other query requests.

IndirectVerify involves sending $\upsilon(\mathcal{D}_{u})$ and then observing
$\eta(\mathbf{x}_{t})$. In this case, model provider cannot directly alter the
output of sample $\mathcal{D}_{u}$ in any received unlearning request to
bypass the verification. As shown in Figure 4, any arbitrary adjustment may
impact the model’s performance during the querying process, resulting in the
misclassification of samples that were originally classified incorrectly. In
addition, changing the output of $\mathcal{D}_{u}$ does not necessarily alter
the output of $\mathbf{x}_{t}$. Therefore, for the verification scheme itself,
it remains effective.

Therefore, the model provider must indeed execute the unlearning process to
meet the verification request. Any attempt to bypass this process could result
in inaccuracies in the model’s precision, affecting its overall usability.

  * •

The Confidentiality of the Relationship between $\mathcal{D}_{u}$ and
$\mathbf{x}_{t}$ is Achieved. IndirectVerify is to generate corresponding
$\mathcal{D}_{u}$ based on the selected sample $\mathbf{x}_{t}$. Model
providers may aim to find the corresponding reaction samples $\mathbf{x}_{t}$
by analyzing the $\mathcal{D}_{u}$. This allows the model provider only to
alter the output of the corresponding verification sample $\mathbf{x}_{t}$,
thereby bypassing the verification without affecting other normal queries.
However, for the model provider, it is challenging to find $\mathbf{x}_{t}$
based on the corresponding $\mathcal{D}_{u}$.

We use $\varsigma(\cdot)$ to denote the process of generating
$\mathcal{D}_{u}$ through $\mathbf{x}_{t}$. For any given $\mathbf{x}_{t}$,
generating different $\mathcal{D}_{u}$ based on the function
$\varsigma(\cdot)$ is straightforward. The mapping from $\mathbf{x}_{t}$ to
$\mathcal{D}_{u}$ is injective, as different $\mathbf{x}_{t}$ values
correspond to different $\mathcal{D}_{u}$. That is,

$\text{if }\mathbf{x}_{t,1}\neq\mathbf{x}_{t,2},\text{ then
}\varsigma(\mathbf{x}_{t,1})\neq\varsigma(\mathbf{x}_{t,2}).$

However, this mapping is not surjective because not all
$\mathcal{D}_{u}\in\mathcal{D}$ have corresponding $\mathbf{x}_{t}$. That is,

$\exists\mathcal{D}_{u}\in\mathcal{D},\text{for which},\text{for any
}\mathbf{x}_{t},\varsigma(\mathbf{x}_{t})\neq\mathcal{D}_{u}\text{ holds
True}.$

Therefore, for model provider, since $\varsigma(\cdot)$ is injective and not
surjective, this prevents the model provider from deducing $\mathbf{x}_{t}$
based on $\mathcal{D}_{u}$, thereby ensuring that the model provider cannot
bypass the verification process.

##  VI Performance Evaluation

###  VI-A Experiment Setup

####  VI-A1 Model and Dataset.

IndirectVerify can be applied to all types of classification models. To show
its effectiveness, we choose two powerful and popular machine learning models:
VGG and ResNet. We adopt five real-world image datasets for evaluation: MNIST
111http://yann.lecun.com/exdb/mnist/, Fashion MNIST 222http://fashion-
mnist.s3-website.eu-central-1.amazonaws.com/, SVHN
333http://ufldl.stanford.edu/housenumbers/, CIFAR-10 and CIFAR-100
444https://www.cs.toronto.edu/ kriz/cifar.html. The datasets cover different
attributes, dimensions, and numbers of categories, allowing us to explore the
verification results of the proposed scheme effectively.

####  VI-A2 Baseline Methods.

To demonstrate the effectiveness of IndirectVerify, we consider the following
baseline schemes:

  * •

Backdoor-based verification: Backdoor-based verification scheme has been
proposed for verifying the unlearning process in MLaaS [17, 18]. These schemes
usually involve training models in advance using samples with embedded
backdoors, $\mathcal{D}_{u}$ and subsequently using samples with the same
backdoor pattern to detect whether $\mathcal{D}_{u}$ have been successfully
unlearned.

  * •

MIAs-Based verification: MIAs have already been widely used as one of the
methods to measure the effectiveness of machine unlearning scheme [11, 24, 13,
14]. Here, we also include MIAs as one of the baseline methods

  * •

Random removing sample: Our verification scheme is based on generating
$(\mathcal{D}_{u},\mathbf{x}_{t})$, where after unlearning $\mathcal{D}_{u}$
from the model, the model’s loss for $\mathbf{x}_{t}$ will decrease. To
compare, in Section VI-C, we consider the impact on $\mathbf{x}_{t}$ after
unlearning randomly selected $\mathcal{D}_{u}$ from the model.

(a) ResNet CIFAR-10

(b) ResNet CIFAR-100

(c) ResNet SVHN

(d) VGG CIFAR-10

(e) VGG CIFAR-100

(f) VGG SVHN

(g) Fashion MNIST

(h) MNIST

Figure 5: Verification results of different schemes. The Y-axis represents
ACC, INA and accuracy on the selected test sample, respectively. In the
backdoor scheme, embedding initially yields high accuracy, reduced to near 0%
after unlearning, confirming successful removal. MIAs-based verification
effectively identifies training samples but fails post-unlearning.
IndirectVerify, using influential sample pairs, can verify the unlearning
process effectively.

####  VI-A3 Metrics

For backdoor-based verification schemes, we measure whether the model
correctly classifies samples with the backdoor pattern as the target label,
that is, $ACC=\frac{TP}{TP+FN}$, where $TP$ denotes the number of backdoored
samples predicted as the target label and $TP+FN$ represents the total test
backdoored samples. Ideally, before unlearning, $ACC$ should be close to
$100\%$. And after unlearning, $ACC$ should approach $0\%$. For MIAs-based
verification schemes, we use the success rate of attacking unlearning samples
as our metric, that is, $INA=\frac{TP}{TP+FN}$, where $TP$ denotes the number
of unlearning samples predicted to be in the training set and $TP+FN$
represents the total unlearning samples. Ideally, $INA$ on these unlearning
samples should be close to 100% before unlearning. After unlearning, $INA$
should be close to $0\%$. For our scheme, we evaluate based on whether the
test samples are classified correctly. Ideally, before unlearning, the
classification accuracy of the test sample should be close to $0\%$. After
unlearning, the accuracy should be close to $100\%$.

###  VI-B Verification Results

Setup. To evaluate IndirectVerify, we will thoroughly compare IndirectVerify
with backdoor-based and MIAs-based verification schemes. We adopt retraining
from scratch as our unlearning scheme, aligning with experiments conducted in
existing works [17, 18].

  * •

For the backdoor-based verification scheme [17, 18], we use the same method
described in [17] and insert backdoor into $10\%$ of the samples during the
training process. Following this, we execute the unlearning process and record
the $ACC$ before and after the unlearning process.

  * •

For the MIAs-based verification scheme, we do our evaluation following the
approach outlined in [33]. In the case of the CIFAR100 dataset, we choose the
top $10$ outputs as inputs for the attack model. We record the $INA$ before
and after the unlearning process.

  * •

For our scheme, we begin by choosing one specific test sample. Subsequently,
we perturb $5\%$ of the training dataset using Equation 5. We then measure the
model’s classification accuracy on the selected test sample, both with and
without these perturbed training samples.

The results are shown in Figure 5. Each sub-figure illustrates the results of
various verification schemes applied to various models and datasets.

Results. For the backdoor-based scheme, after embedding the backdoor, the
accuracy of correctly classifying samples with the backdoor as the target is
very high. This indicates that the model is indeed trained using the backdoor
samples. After performing the unlearning operation, the classification results
of samples with the backdoor are close to $0\%$. This suggests that the
backdoor-based verification scheme successfully confirms that the model has
indeed undergone an unlearning process, resulting in the removal of backdoor
samples from the model and causing a decrease in the accuracy of samples with
the backdoor. For MIAs-Based verification schemes, before the unlearning
process, all success rates are relatively high, indicating that MIAs
successfully identify samples used for model training. However, after
unlearning, the attack accuracy does not show a significant decrease. This
illustrates that MIAs cannot identify samples that are not in the training
set. This result suggests that, given the limited effectiveness, MIAs may not
be suitable for verifying the unlearning process [2].

(a) Removing Harmful Samples (MNIST)

(b) Random Removing Samples(MNIST)

(c) Removing Harmful Samples (CIFAR10)

Figure 6: The loss of each sample before and after removing different types of
samples. The x-axis denotes various test samples, while the y-axis represents
the corresponding loss. For MNIST, removing the most harmful samples
significantly reduces loss, while random removal has no effect. For CIFAR-10,
removing selected harmful training samples doesn’t decrease test sample loss.
Possible reasons include a lack of abnormal samples or the ineffectiveness of
Equation 18 assumptions in complex models. To address the consistent presence
of harmful samples in the dataset, we propose a pre-embedding method in
Section IV-B. It can be applied to more complex models and dataset scenarios.
Figure 7: Harmful and helpful samples. we present a misclassified test sample
(labeled 5 but classified as 6) alongside its harmful samples (all labeled 5)
and helpful samples (all labeled 6). The harmful samples resemble 6 more than
5, contributing to the misclassification of the test sample. Removing these
harmful samples eliminates their impact, allowing the helpful samples to
better differentiate the test sample and result in the correct classification.

For our scheme IndirectVerify, before unlearning, all test samples are
misclassified due to trigger samples in the training set. At the same time,
upon unlearning these trigger samples from the model, the test samples are all
classified correctly. This indicates that IndirectVerify is effective for
verifying the unlearning. It is noteworthy that our trigger samples are
different from backdoored samples. Firstly, we did not alter the labels of
those trigger samples, effectively minimizing the impact on the model
performance. Secondly, in practical scenarios, model providers do not allow
data providers to add backdoors, as it poses a serious threat to the model’s
security. IndirectVerify, being without backdoors, allows for verification of
the unlearning operation while ensuring model security.

###  VI-C Feasibility Analysis

(a) Outputs before Fine-tuing (MIAs)

(b) Outputs after Fine-tuing (MIAs)

(c) Bypassing Results for MIAs

(d) Bypassing Results for Backdoor

Figure 8: The results of our proposed bypassing schemes for MIAs-based and
backdoor-based verification schemes. In Figure 8a, before fine-tuning, the
model’s outputs for member and non-member samples differ, enabling MIAs to
verify the unlearning process based on the different outputs. However, in
Figure 8b, the bypass method based on fine-tuning makes the outputs of member
and non-member samples indistinguishable, rendering MIAs ineffective (see
Figure 8c). For backdoor-based verification schemes (Figure 8d), post-
training, the model effectively classifies samples with a backdoor. After
fine-tuning, the model fails to correctly classify samples with backdoors,
making the backdoor-based verification scheme ineffective.

(a) Outputs before Fine-tuing

(b) Outputs after Fine-tuing

(c) Bypassing Results (Same setting with MVS)

(d) Bypassing Results (Same setting with BVS)

Figure 9: The results of our proposed bypassing schemes for our verification
scheme. MVS represents the MIAs-based verification scheme, and BVS denotes the
backdoor-based verification scheme. Despite closely aligning with one test
sample’s output after fine-tuning (Figures 9a and 9b), our verification
results (Figure 9c) show no change, indicating effective resistance against
fine-tuning bypass schemes based on Equation 1. Similarly, IndirectVerify also
remains effective against bypass schemes based on Equation 2 (Figure 9d).
Overall, IndirectVerify robustly resists known bypass schemes.

Setup. IndirectVerify aims at generating a pair of samples
$(\mathcal{D}_{u},\mathbf{x}_{t})$. Samples $\mathcal{D}_{u}$ can influence
the accuracy of another sample $\mathbf{x}_{t}$. Removing $\mathcal{D}_{u}$
corrects the misclassification of sample $\mathbf{x}_{t}$. In this section, we
validate the above points by directly searching samples $\mathcal{D}_{u}$
based on Equation 19 and 20 within the dataset, rather than generating them.
We first construct one simple model and train it based on the MNIST dataset.
Then, we select five misclassified samples and calculate the impact of each
training sample on each test sample based on Equation 18. We sort the values
of the results of Equation 18 in ascending order, and then select the
corresponding training samples. If the value is significant, we refer to the
corresponding sample as helpful; conversely, if the value is small, we label
the corresponding sample as harmful. Subsequently, we remove the topmost
harmful sample corresponding to each misclassified test sample from the
dataset and execute the unlearning process. Once each model is retrained, we
retest these misclassified samples using the corresponding unlearned model. As
a comparison, we also retrain the model when randomly removing one training
sample. In addition, we choose the ResNet18 architecture for the CIFAR10
dataset to do the same process as above. We record the loss of different test
samples before and after the unlearning process; the results are shown in
Figure 6.

Results. In Figure 6, the x-axis represents different test samples, while the
y-axis denotes the corresponding loss of each test sample. Figures 6a and 6c
respectively illustrate the results of removing the most harmful samples from
the MNIST and CIFAR10 datasets. Figure 6b shows the results when randomly
removing one sample from the MNIST dataset. As shown in Figure 6a, the loss
for each misclassified test sample is relatively high before retraining the
model. However, upon removing the selected most harmful sample, the test
sample’s loss significantly decreases, enabling correct classification.
Meanwhile, as shown in Figure 6b, when we randomly remove a sample, the
unlearned model does not show a decrease in the loss of each test sample. The
above results demonstrate that when dealing with a simple model and dataset,
given a misclassified test sample, we can identify the most harmful training
sample for this misclassified sample. Removing the harmful training sample
will reduce the loss for the corresponding misclassified test sample, leading
to a correct classification. Those results illustrate the feasibility of our
verification scheme.

In Figure 7, we show a misclassified test sample (labeled 5, but classified as
$6$) along with its corresponding harmful samples (all labeled $5$) and
helpful samples (all labeled $6$). These harmful samples appear dissimilar to
$5$ but are closely similar to $6$. The misclassified sample shares
significant similarities with these harmful samples. In contrast, the most
helpful samples exhibit true characteristics of $6$, aiming to differentiate
the misclassified sample as much as possible. The reason for the
misclassification of the test sample may be attributed to these harmful
training samples, which causes the test sample to be closer to the data space
of $6$, resulting in the misclassification. When we remove those harmful
samples, the impact of these harmful samples is eliminated, and those helpful
samples strive to differentiate the test sample as much as possible, leading
to the correct classification of the original test sample.

As shown in Figure 6c, when removing the selected harmful training samples for
CIFAR-10, the loss of the test sample does not decrease. We think there are
several possible reasons for this. On one hand, the dataset may lack abnormal
samples, resulting in the absence of harmful samples. On the other hand, when
we derive Equation 18, we make the assumption that the empirical risk is both
twice-differentiable and strongly convex. But this may not hold in complex
models with large datasets, making Equation 18 ineffective. Therefore, in
Section IV-B, we have proposed an optimization-based method that is applicable
to any complex model and dataset.

(a) ResNet CIFAR-10

(b) VGG CIFAR-10

(c) ResNet SVHN

(d) VGG SVHN

Figure 10: Model Accuracy with different perturbed trigger samples. All
figures show that embedding trigger samples has a negligible impact on model
performance. For instance, with the ResNet model and CIFAR-10 dataset, the
difference before and after embedding is less than $0.01\%$, likely due to
training process randomness. The minimal effect stems from modifying pixel
values without changing samples’ labels.

(a) With Modification

(b) Without Modification

Figure 11: Original samples and the corresponding trigger samples. It is noted
that the samples, before and after perturbation, are almost the same, with no
significant difference.

###  VI-D Bypassing

Setup. We proposed two methods that can bypass the current verification
schemes in Section III-B. In this Section, we will evaluate whether our
verification scheme remains effective in the presence of these bypass methods.
For comparison, we also evaluate both backdoor-based and MIAs-based
verification schemes [17, 18]:

  * •

For MIAs-based verification schemes, we attempt to bypass the scheme in [33].
We select three training samples as member samples to mimic those requiring
unlearning. Then, we choose one nonmember sample from the test dataset with
the same label as these three samples. Following this, we fine-tune the model
using Equation 1, where $\lambda$ is set to $0.01$. The model outputs of the
samples before and after fine-tuning are shown in Figure 8a and Figure 8b,
respectively. The x-axis denotes the index of each posterior probability, and
the y-axis indicates the corresponding value. The results of the MIAs are
presented in Figure 8c.

  * •

For backdoor-based verification schemes, we conduct our experiments based on
the scheme in [17]. Then, we fine-tuned the model based on Equation 2. We
record both the model’s overall accuracy and the accuracy of the backdoor. The
results are shown in Figure 8d.

  * •

For our scheme, for Equation 1, we employ the same settings as MIAs-based
verification schemes. Also, in Figures 9a and 9b, the x-axis denotes the index
of each posterior probability, and the y-axis indicates the corresponding
value. For Equation 2, since we maintain the original labels for all samples
without alteration, we use these original labels in the fine-tuning procedure.
In both processes, we record the same metrics as MIAs-based and backdoor-based
verification schemes. The results are shown in Figure 9.

Results. As shown in Figure 8a, before fine-tuning the model, the outputs for
member samples and nonmember samples are different, enabling MIAs to verify
that samples have not been unlearned based on the model’s output. In Figure
8b, the fine-tuning process makes the output of member samples nearly
indistinguishable from that of nonmember samples. This makes MIAs unable to
determine whether member samples are used to train the model. As illustrated
in Figure 8c, as fine-tuning progresses, the accuracy of MIAs in identifying
member samples gradually decreases until they can no longer identify them.
This will result in verification schemes based on MIAs being unable to verify
whether these samples have been unlearned. Meanwhile, the model’s accuracy
remains nearly unchanged, indicating that the model provider has successfully
bypassed the verification schemes.

As shown in Figure 8d for backdoor-based verification schemes, after the
training process, if a sample contains a backdoor, the model can correctly
classify that sample as a backdoored sample. In this case, a backdoor-based
verification scheme is effective, meaning that once the backdoor samples are
unlearned, the model will no longer be able to classify backdoored samples
into a specific target. However, when fine-tuning the model, it cannot
correctly classify backdoored samples. This results in the backdoor-based
verification scheme becoming ineffective as, even if the model provider has
not unlearned the backdoor samples, the model may still fail to correctly
classify samples containing a backdoor.

For IndirectVerify, as shown in Figure 9a and Figure 9b, even if the outputs
align closely with the output of one test sample after fine-tuning, our
verification results show no change (as shown in Figure 9c), which indicates
that IndirectVerify can effectively resist fine-tuning bypass schemes based on
Equation 1. This is because IndirectVerify incorporates two samples: one used
for unlearning requests and the other for verification. Changing one sample’s
output does not affect the classification result of another sample. Similarly,
for the bypass scheme based on Equation 2, as shown in Figure 9d,
IndirectVerify also remains effective. Overall, IndirectVerify demonstrates
effective resistance against currently known bypass schemes and exhibits a
certain level of robustness.

###  VI-E Evaluating the Harmlessness

Setup. To evaluate if the embedded trigger samples affect the model’s
performance, we evaluate our schemes using the VGG and ResNet models on two
datasets: CIFAR-10 and SVHN, respectively. We perturb $1\%$, $2\%$, $3\%$, and
$4\%$ of the training samples to trigger samples, respectively, for later
verification purposes. We record the model’s accuracy before and after
embedding the trigger samples. The experimental results are shown in Figure
10.

Results. As can be seen from all the figures, the embedding of the trigger
samples has an almost negligible effect on the model performance. For example,
when considering the ResNet model and the CIFAR-10 dataset, the difference
before and after embedding is less than $0.01\%$. This small difference can be
attributed to the randomness inherent in the training process. The reason why
the impact on the model is minimal is that we did not change the labels of the
samples used for verification; we only modified the pixel values of those
samples. Furthermore, these modifications are almost imperceptible. Figure 11
illustrates a few of the original images and the corresponding trigger
samples. It is noted that the two sets of images are almost the same, with no
significant difference.

##  VII Conclusion and Future Work

In this paper, we take the first step in addressing machine unlearning
verification problems when faced with untrustworthy model providers in MLaaS.
We first propose two distinct bypass schemes to illustrate the shortcomings of
current membership inference and backdoor attack-based verification methods.
Then, we propose a perturbation-based verification scheme to generate
influential sample pairs, including trigger samples and a reaction sample.
Trigger samples are utilized in the unlearning request, while reaction samples
are used for subsequent verification, with the presence or absence of trigger
samples influencing reaction samples. We also provide theoretical proofs and
analyses of the effectiveness of the proposed scheme, and briefly emphasize
its robustness by explaining how our scheme resists current bypassing methods.
The experimental results provide evidence that our scheme effectively
maintains model utility while achieving robust verification effectiveness.

## References

  * [1] J. Weng, J. Weng, C. Cai, H. Huang, and C. Wang, “Golden grain: Building a secure and decentralized model marketplace for mlaas,” _IEEE Trans. Dependable Secur. Comput._ , vol. 19, no. 5, pp. 3149–3167, 2022.
  * [2] H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, “Membership inference attacks on machine learning: A survey,” _ACM Comput. Surv._ , vol. 54, no. 11s, pp. 235:1–235:37, 2022.
  * [3] T. Zhu, D. Ye, S. Zhou, B. Liu, and W. Zhou, “Label-only model inversion attacks: Attack with the least information,” _IEEE Trans. Inf. Forensics Secur._ , vol. 18, pp. 991–1005, 2023.
  * [4] H. Xu, T. Zhu, L. Zhang, W. Zhou, and P. S. Yu, “Machine unlearning: A survey,” _ACM Comput. Surv._ , vol. 56, no. 1, pp. 9:1–9:36, 2024.
  * [5] L. Zhang, T. Zhu, H. Zhang, P. Xiong, and W. Zhou, “Fedrecovery: Differentially private machine unlearning for federated learning frameworks,” _IEEE Trans. Inf. Forensics Secur._ , vol. 18, pp. 4732–4746, 2023.
  * [6] “General Data Protection Regulation (GDPR),” 2018, [Online; Retrieved in December 19, 2023 from https://data.stats.gov.cn.
  * [7] Y. Cao and J. Yang, “Towards making systems forget with machine unlearning,” in _2015 IEEE SP 2015, San Jose, CA, USA, May 17-21, 2015_. IEEE Computer Society, 2015, pp. 463–480.
  * [8] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,” in _42nd IEEE SP, San Francisco, USA, 24-27 May_ , 2021, pp. 141–159.
  * [9] C. Chen, F. Sun, M. Zhang, and B. Ding, “Recommendation unlearning,” in _Proceedings of the ACM Web Conference 2022_ , ser. WWW ’22, New York, NY, USA, 2022, p. 2768–2777.
  * [10] M. Chen, Z. Zhang, T. Wang, M. Backes, M. Humbert, and Y. Zhang, “Graph unlearning,” in _CCS, Los Angeles, USA, November 7-11, 2022_ , H. Yin, A. Stavrou, C. Cremers, and E. Shi, Eds., 2022, pp. 499–513.
  * [11] L. Graves, V. Nagisetty, and V. Ganesh, “Amnesiac machine learning,” in _AAAI 2021_. AAAI Press, 2021, pp. 11 516–11 524.
  * [12] C. Guo, T. Goldstein, A. Y. Hannun, and L. van der Maaten, “Certified data removal from machine learning models,” in _ICML 2020, 13-18 July 2020, Virtual Event_ , vol. 119. PMLR, 2020, pp. 3832–3842.
  * [13] A. Golatkar, A. Achille, and S. Soatto, “Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations,” in _ECCV UK, August 23-28, 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12374. Springer, 2020, pp. 383–398.
  * [14] A. Golatkar, A. Achille, A. Ravichandran, M. Polito, and S. Soatto, “Mixed-privacy forgetting in deep networks,” in _IEEE CVPR, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 792–801.
  * [15] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in _CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , 2020, pp. 9301–9309.
  * [16] Y. Liu, L. Xu, X. Yuan, C. Wang, and B. Li, “The right to be forgotten in federated learning: An efficient realization with rapid retraining,” in _IEEE INFOCOM, London, May 2-5_ , 2022, pp. 1749–1758.
  * [17] Y. Guo, Y. Zhao, S. Hou, C. Wang, and X. Jia, “Verifying in the dark: Verifiable machine unlearning by using invisible backdoor triggers,” _IEEE TIFS_ , vol. 19, pp. 708–721, 2024.
  * [18] D. M. Sommer, L. Song, S. Wagh, and P. Mittal, “Athena: Probabilistic verification of machine unlearning,” _Proc. Priv. Enhancing Technol._ , vol. 2022, no. 3, pp. 268–290, 2022.
  * [19] J. Weng, S. Yao, Y. Du, J. Huang, and C. Wang, “Proof of unlearning: Definitions and instantiation,” _CoRR_ , vol. abs/2210.11334, 2022.
  * [20] G. Liu, T. Xu, X. Ma, and C. Wang, “Your model trains on my data? protecting intellectual property of training data via membership fingerprint authentication,” _IEEE TIFS._ , vol. 17, pp. 1024–1037, 2022.
  * [21] J. Geiping, L. H. Fowl, W. R. Huang, W. Czaja, G. Taylor, M. Moeller, and T. Goldstein, “Witches’ brew: Industrial scale data poisoning via gradient matching,” in _ICLR 2021, Austria, May 3-7, 2021_ , 2021.
  * [22] P. W. Koh and P. Liang, “Understanding black-box predictions via influence functions,” in _Proceedings of the 34th ICML, Sydney, 6-11 August_ , D. Precup and Y. W. Teh, Eds., vol. 70, 2017, pp. 1885–1894.
  * [23] T. Zhu, G. Li, W. Zhou, and P. S. Yu, “Differentially private data publishing and analysis: A survey,” _IEEE Trans. Knowl. Data Eng._ , vol. 29, no. 8, pp. 1619–1638, 2017.
  * [24] G. Liu, X. Ma, Y. Yang, C. Wang, and J. Liu, “Federaser: Enabling efficient client-level data removal from federated learning models,” in _29th IEEE/ACM IWQOS, Tokyo, June 25-28, 2021_ , 2021, pp. 1–10.
  * [25] J. Wang, S. Guo, X. Xie, and H. Qi, “Federated unlearning via class-discriminative pruning,” in _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29_. ACM, 2022, pp. 622–632.
  * [26] Y. Wu, E. Dobriban, and S. B. Davidson, “Deltagrad: Rapid retraining of machine learning models,” in _ICML 2020, 13-18 July 2020, Virtual Event_ , vol. 119. PMLR, 2020, pp. 10 355–10 366.
  * [27] J. Brophy and D. Lowd, “Machine unlearning for random forests,” in _ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 1092–1104.
  * [28] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _Proceedings of the 22nd ACM SIGSAC_. ACM, 2015, pp. 1322–1333.
  * [29] A. Thudi, H. Jia, I. Shumailov, and N. Papernot, “On the necessity of auditable algorithmic definitions for machine unlearning,” in _31st USENIX Security, Boston, USA, August 10-12_ , 2022, pp. 4007–4022.
  * [30] A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh, “Remember what you want to forget: Algorithms for machine unlearning,” in _NeurIPS 2021, December 6-14, 2021, virtual_ , 2021, pp. 18 075–18 086.
  * [31] B. Mu, Z. Niu, L. Wang, X. Wang, Q. Mia, R. Jin, and G. Hua, “Progressive backdoor erasing via connecting backdoor and adversarial attacks,” in _IEEE/CVF CVPR_. IEEE, 2023, pp. 20 495–20 503.
  * [32] Y. Li, M. Zhu, X. Yang, Y. Jiang, T. Wei, and S. Xia, “Black-box dataset ownership verification via backdoor watermarking,” _IEEE Trans. Inf. Forensics Secur._ , vol. 18, pp. 2318–2332, 2023.
  * [33] Y. Liu, R. Wen, X. He, A. Salem, Z. Zhang, M. Backes, E. D. Cristofaro, M. Fritz, and Y. Zhang, “Ml-doctor: Holistic risk assessment of inference attacks against machine learning models,” in _31st USENIX, Boston, MA, USA, August 10-12, 2022_ , 2022, pp. 4525–4542.