# Single Layer Single Gradient Unlearning

Zikui Cai Yaoteng Tan M. Salman Asif
University of California Riverside
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Machine unlearning methods seek to revise pretrained models such that effects
of certain training samples can be removed. In addition to effective erasure,
low computational cost and general utility retention are also highly
desirable. Existing unlearning methods usually involve iterative updates over
the model parameters, which incurs a high computational cost. In this work, we
propose an efficient method that only requires a one-time gradient
computation, with which we modify only a single layer of model parameters.
Specifically, we first identify a small number of model layers that lie on the
Pareto front of high forget importance and low retain influence as critical
layers. Then we search for a suitable step size and take a step along the
gradient direction of a single critical layer while keeping other layers
frozen. This method is highly modular and can be used to unlearn multiple
concepts simultaneously in a controllable manner. We demonstrate the
effectiveness and efficiency of this method on various models including CLIP,
stable diffusion, and VLMs, surpassing other state-of-the-art methods.

##  1 Introduction

Figure 1: Overview of our framework. Given an unlearning query, such as
removing an identity like Elon Musk, we first curate or generate a forget set
containing relevant data and a retain set with data points we want to
preserve. Using these datasets, we calculate and store the model gradients.
Based on these gradients, we identify the important layers to update for
unlearning. We then take a step along the forget gradients of a single layer
and evaluate the model’s unlearning performance. To determine a suitable step
size $\lambda$, we employ a binary search. After unlearning, the specified
concepts are effectively erased while retaining the model’s overall utility.

Modern machine learning models, including large language models (LLMs) [37,
38, 4, 3, 24], Stable Diffusion (SD) [39, 40, 35, 47], and vision language
mdoels (VLMs) [50, 28, 27, 26] leverage vast amounts of data for training.
Large-scale web scrapers, such as Common Crawl, archive billions of webpages
monthly, providing massive data sources to train such revolutionary AI models.
While these large unsupervised datasets significantly enhance performance
under scaling law [20], they also raise serious data privacy and legal
compliance [1] concerns. For instance, researchers [44] discovered hundreds of
URLs linked to child sexual abuse material in LAION-5B [41], a popular dataset
used to train stable diffusion. Consequently, the dataset was taken offline,
and associated models are now subject to safety investigations. Completely
abandoning previously trained model weights and retraining large models from
scratch using scrutinized dataset is prohibitively expensive, highlighting the
need for efficient data removal methods.

Machine unlearning [5, 32, 2, 17, 14, 7, 16, 15, 9, 23, 19, 43, 10, 11, 48,
29, 25], a technique designed to reverse the learning process, aims to
efficiently remove specific data points from a trained model without the need
of retraining from scratch, has been gaining increasing attention as a
solution to the challenges posed by data privacy and legal compliance issues
in modern AI training processes. Its objective is threefold. First, low
computational cost, as the naïve approach of training from scratch achieves
the best result (exact unlearning), but is expensive. Second, effective
unlearning, ensuring that the model forgets the intended data completely.
Third, utility retention, maintaining its performance on tasks unrelated to
the forgotten data. However, current unlearning methods often fall short. They
usually involve iterative updates over the model parameters, resulting in high
computational costs. Additionally, achieving a balance between effective
unlearning and utility retention remains a challenge.

In this paper, we push the boundaries of efficiency with an algorithm that
requires only a single gradient calculation and updates a single layer to
achieve effective unlearning without affecting the model’s general utility. We
first calculate gradients of forget and retain losses with respect to model
weights using a designated or curated forget and retain set. Based on these
gradients, we propose two metrics, namely layer importance and gradient
alignment, to select the appropriate layers for unlearning. To determine a
suitable step size for weight updates, our method employs an efficient binary
search along the direction of forget gradients. We demonstrate that our method
outperforms state-of-the-art methods in unlearning tasks involving CLIP
models, stable diffusion and VLMs, across various tasks and architectures. In
addition to its efficiency and effectiveness, our methods offers higher
modularity and better interpretability. Specifically, we can calculate the
gradients of different unlearning tasks independently and unlearn them
simultaneously. By assigning different layers for different unlearning tasks,
we avoid interference. Our layer selection approach provides insights into the
features learned by different layers and their functionalities, offering
generalized guidance for new tasks and model architectures without additional
computation.

##  2 Background

In this section, we present background knowledge on machine unlearning and
vision-language alignment using CLIP, which is the primary focus of this
paper.

###  2.1 Machine unlearning preliminaries

One of the core objectives of machine unlearning is to remove the influence of
a specific subset of training data $D$, on a pre-trained model, represented as
$F_{\theta}(D)$ with model parameters represented as $\theta$. Let us assume
$D_{\texttt{f}}\subset D$ denotes the subset selected for unlearning
(forgetting). The primary challenge is to ensure that the unlearning process
is computationally efficient, significantly reducing the time and resources
compared to retraining the model from scratch with the remaining dataset
$D_{\texttt{r}}=D\setminus D_{\texttt{f}}$. The ideal unlearning algorithm,
which we refer to as $U$, should yield an unlearned model
$U(F_{\theta}(D),D,D_{\texttt{f}})$ with model parameters
$\theta_{\texttt{f}}$ that is functionally indistinguishable from a model
retrained exclusively on the retain set
$F_{\theta_{\texttt{r}}}(D_{\texttt{r}})$. The main objective is captured in
Eqn. 1, where $N$ denotes the number of elements in the dataset $D$, $\alpha$
is the balancing factor, $(x,y)$ is an element from the dataset, and $\ell$ is
the loss function, usually the training loss.

$\min_{\theta}\underbrace{\frac{1}{N_{\texttt{r}}}\sum_{(x_{\texttt{r}},y_{\texttt{r}})\in
D_{\texttt{r}}}\ell(F_{\theta}(x_{\texttt{r}}),y_{\texttt{r}})}_{\mathcal{L}_{\text{retain}}}-\underbrace{\frac{\alpha}{N_{\texttt{f}}}\sum_{(x_{\texttt{f}},y_{\texttt{f}})\in
D_{\texttt{f}}}\ell(F_{\theta}(x_{\texttt{f}}),y_{\texttt{f}})}_{\mathcal{L}_{\text{forget}}}$
(1)

Naïve gradient ascent (GA) solely on the forget set (minimizing the second
term in Eqn. 1) is effective at increasing forget loss but is prone to over-
unlearning, which can significantly reduce test accuracy if iterations exceed
an appropriate limit. Fine tuning (FT), which only uses the retain set
(minimizing the first term in Eqn. 1), is ineffective at forgetting. However
it can be combined with GA in a two-stage training paradigm [10] to mitigate
over-unlearning. We refer to this two-stage framework, involving GA and FT, as
GAFT (Eqn. 1).

Recent works[10, 11] have focused on identifying salient parameters via
gradient calculation to stabilize unlearning by updating only these parameters
while keeping the rest of the model intact. Specifically, SalUn [10] uses hard
thresholds on the gradient of forgetting loss with respect to model weights,
updating only those weights with gradient magnitudes higher than a certain
threshold (e.g., the median gradient magnitude across the model). SSD[11]
proposes dampening model weights that are disproportionately important to the
forget and retain sets, scaling down the values of such weights. While these
methods achieve localized editing and improve overall unlearning performance,
the significant weights are spread throughout the model, which does not offer
understanding about the structure of the model. They also require careful
hyperparameter tuning, including learning rate, the number of iterations, and
hyperparameters for selecting suitable masks for saliency approaches. All
these limitations motivate us to develop a method that is hyperparameter-free
and easily interpretable.

###  2.2 Vision language alignment

Existing methods in machine unlearning often suffer from high computational
costs and limited scalability, primarily focusing on small-scale image
classification models [19, 11]. In contrast, our method demonstrates superior
scalability and versatility, making it applicable to large multi-modal
foundation models such as CLIP[36], stable diffusion [39], and visual language
models (VLMs) [28]. CLIP models, in particular, play a central role in driving
the development of various multi-modal models by aligning visual and textual
representations through a contrastive loss function [8].

$\ell=\frac{1}{2N}\sum_{i=1}^{N}\left(\ell_{i2t}(i)+\ell_{t2i}(i)\right),$ (2)
$\ell_{i2t}(i)=-\log\frac{\exp(\cos(\mathbf{v}_{i},\mathbf{t}_{i})/\tau)}{\sum_{j=1}^{N}\exp(\cos(\mathbf{v}_{i},\mathbf{t}_{j})/\tau)},\leavevmode\nobreak\
\leavevmode\nobreak\
\ell_{t2i}(i)=-\log\frac{\exp(\cos(\mathbf{t}_{i},\mathbf{v}_{i})/\tau)}{\sum_{j=1}^{N}\exp(\cos(\mathbf{t}_{i},\mathbf{v}_{j})/\tau)}.$
(3)

Here $\mathbf{v}_{i}$ is the normalized embedding of image $x_{i}$ through the
vision model $f_{\texttt{v}}$ and $\mathbf{t}_{i}$ is the normalized embedding
of text $y_{i}$ through the text model $f_{\texttt{t}}$. The temperature
$\tau$ controls the sharpness of the probability distribution after softmax.
Cosine similarity between $\mathbf{v}_{i}$ and $\mathbf{t}_{j}$ is defined as
$\cos(\mathbf{v}_{i},\mathbf{t}_{j})=\mathbf{v}_{i}\cdot\mathbf{t}_{j}$.

By minimizing the contrastive loss in Eqn. 2, the model learns to align vision
and language information of the same pair in the embedding space. In
unlearning, we break such alignments.

##  3 Unlearning via weight arithmetic

Our approach improves the state-of-the-art along three axes: (1) low
computational cost (achieved through a single gradient calculation followed by
weight arithmetic), (2) effective unlearning (by identifying layers with high
importance to the forget set), and (3) high retention of general utility (via
pinpointing layers with low gradient alignment to the retain set). The
framework is illustrated in Figure 1.

###  3.1 Loss functions for gradient calculation

It is important to select appropriate loss functions to perform unlearning.
For image classification models, cross-entropy loss can be directly used as
both retain loss and forget loss. However, the scenario is different for
contrastive learning. For the retain set, we can still use the original
contrastive loss as in Eqn. 2.

$\mathcal{L}_{\text{retain}}=\frac{1}{2N}\sum_{i=1}^{N}\left(\ell_{i2t}(i)+\ell_{t2i}(i)\right),$
(4)

However, for the forget set, we employ the cosine embedding loss (Eqn. 5)
instead since it directly pushes the embeddings of positive pairs away while
not tampering with the embeddings of negative pairs.

$\mathcal{L}_{\text{forget}}(\mathbf{v}_{i},\mathbf{t}_{j})=1-\text{cos}(\mathbf{v}_{i},\mathbf{t}_{j})$
(5)

Using the original contrastive loss as forget loss will result in ineffective
unlearning.

###  3.2 Layer identification

In this work, we are inspired by the distinct feature representations learned
by different layers within deep networks [49, 34, 13]. For example, early
layers in vision models generally extract basic features such as edges and
textures, while later layers discern more specific details. To target
unlearning effectively, we aim to modify only those layers that hold
information directly relevant to the unlearning objectives, rather than those
that process abstract features which could be learned independently of the
examples to be forgotten. Our primary goal is to pinpoint the layers most
critical to the specific unlearning task, ensuring that adjustments lead to
desired outcomes without needless alterations to unrelated functionalities.
Moreover, maintaining the overall utility of the model is paramount; hence, we
endeavor to perform unlearning within the “nullspace" of the retain set. This
strategy involves identifying layers that have minimal impact on the
performance of retained data while effectively facilitating the unlearning of
selected features. This method not only enhances the precision of the
unlearning process but also provides greater interpretability and a clearer
insight into how unlearning and data retention are managed within the network.

Similar to [11], we leverage Fisher information matrix [21, 18, 22] to measure
the effect of each parameter on the objective. Calculating the exact Fisher
information matrix requires obtaining the second-order derivative of the model
output with respect to its parameters, which is computationally prohibitive.
To make the computation more tractable, we use the diagonal approximation:

$\mathcal{I}_{D}(\theta)=-\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta^{2}}\log
L(\theta;D)\right]=\mathbb{E}\left[\left(\frac{\partial}{\partial\theta}\log
L(\theta;D)\right)\left(\frac{\partial}{\partial\theta}\log
L(\theta;D)\right)^{\text{T}}\right].$ (6)

The diagonal elements of the Fisher Information Matrix quantify the
sensitivity of the log-likelihood function to changes in individual
parameters. To extend this to layers, we aggregate these sensitivities at the
layer level. For a layer $l$, the importance of the layer is measured by
aggregating the ratio of the $\ell_{2}$ norm of the forget loss gradients to
the $\ell_{2}$ norm of the parameters within the layer as

$\texttt{Importance of layer l:
}\text{Importance}(l)=\frac{\sqrt{\mathcal{I}_{D_{\texttt{f}}}(\theta_{l})}}{\|\theta_{l}\|_{2}}=\frac{\|\nabla_{\theta_{l}}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}})\|_{2}}{\|\theta_{l}\|_{2}}.$
(7)

Using the importance criteria alone is not sufficient, as it only indicates
which layer can influence the forget set. For layer selection, we prefer
layers for which the forget gradients are orthogonal to the retain gradients
by computing

$\texttt{Gradient alignment:
}\text{Alignment}(l)=\text{cos}\bigl{(}\nabla_{\theta_{l}}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}}),\nabla_{\theta_{l}}\mathcal{L}_{\text{retain}}(\theta,D_{\texttt{r}})\bigr{)}.$
(8)

This ensures that updates for unlearning the forget set do not adversely
affect the retain set.

To identify important layers, we use the ratio of the $\ell_{2}$ norm of the
forget loss gradients to the $\ell_{2}$ norm of the layer parameters as the
importance measure, and ensure orthogonality of the forget gradients to the
retain gradients to maintain performance on the retain set. To balance these
objectives, we utilize the concept of Pareto optimal set [31], which
represents a set of optimal solutions with a trade-off in two objectives. In
our context, the two objectives are maximizing the importance measure and
ensuring the orthogonality condition. We present an experimental result in
Figure 2 to illustrate the Pareto front for unlearning a person identify from
CLIP ViT-B-32. The layers on the top left are the ones that unlearn the forget
set without affecting the retain set.

(a) Layers of vision model

(b) Layers of language model

(c) Unlearn single vision layer

(d) Unlearn single language layer

(e) GA on whole model

(f) GAFT on whole model

Figure 2: The first row shows scatter plots of layers within (a) vision and
(b) language models from CLIP ViT-B-32 trained on Laion-400M dataset. Layers
on Pareto front for unlearning an identity (e.g. Elon Musk) are marked as dots
in different colors, while less significant layers are marked as crosses in
black. The second row demonstrates the unlearning performance when moving
along the direction of forget gradient (with step size $\lambda$) on a single
(c) vision layer 11.attn.out_proj and (d) language layer 0.attn.out_proj. Our
method precisely identifies a small number of significant layers within the
entire model and efficiently unlearns without significantly compromising
utility. Additional examples of different identities, models, and layers are
provided in Figures 11, 15, and 16 respectively. The third row shows the
performance of (e) GA and (f) GAFT on the whole model, illustrating that
iterative gradient calculations are not superior to a single gradient step.

###  3.3 Linearizing unlearning trajectory

Existing approaches for unlearning calculate gradients at each iteration for
updating the model parameters, which adds to computational complexity. In this
work, we observe that calculating gradients repeatedly is unnecessary.
Instead, we only perform one gradient calculation for the initialized model
and update the parameters, $\theta_{l}$ for any layer $l$, in a weight
arithmetic manner. That is, update the weights along a fixed gradient
direction at every iteration as

$\theta_{l}\leftarrow\theta_{l}-\lambda\nabla_{\theta_{l}^{0}}\mathcal{L}_{\text{forget}},$
(9)

where $\nabla_{\theta_{l}^{0}}\mathcal{L}_{\text{forget}}$ represents the
gradient computed at the first iteration. This can be regraded as linearizing
the unlearning trajectory. Combined with our previous discussion on important
layer identification, this method is sufficient in unlearning while retaining
general performance. For selecting the step size, we perform a binary search
along the linearlized trajectory, and stop whenever appropriate as suggested
by the evaluation metric.

###  3.4 Generalization to stable diffusion and VLMs

By harnessing the capability of effective unlearning in CLIP models, we can
extend this technique to edit larger and more powerful models that rely on
CLIP, including image generation models such as stable diffusion [39, 40]) and
VLMs like LLaVA [28, 27]).

Unlearn stable diffusion. Diffusion models are popularized by their strong
ability to generate high quality images based on text descriptions [33, 39].
The text prompt is first embedded into a high-dimensional vector space using a
pre-trained language model, for example the latest Stable Diffusion v2 model
uses CLIP ViT-H/14 text encoder. Then the text conditioning is achieved
through a cross-attention mechanism, where the text embedding guides each step
of the denoising process. Starting from the initial noise $\mathbf{x}_{T}$,
the iterative updates are performed as follows:

$\mathbf{x}_{t-1}=\sqrt{\alpha_{t}}\left(\mathbf{x}_{t}-\gamma_{t}\nabla_{\mathbf{x}}\log
p(\mathbf{x}_{t}|\mathbf{e})\right)+\sqrt{1-\alpha_{t}}\mathbf{z}_{t},$ (10)

where $\mathbf{x}_{t}$ is the noisy image at step $t$, $\mathbf{z}_{t}$ is the
noise added at step $t$, $\alpha_{t}$ is a time-dependent parameter
controlling the noise balance, $\gamma_{t}$ is the learning rate,
$\mathbf{e}=f_{\mathbf{t}}(\texttt{txt})$ is the text embedding, and
$\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}|\mathbf{e})$ is the gradient of the
log-probability of the noisy image given the text embedding, guiding the
denoising process. We can keep the CLIP vision model frozen and only update
the language model to achieve unlearning.

Unlearn VLMs. Vision-Language Models (VLMs) are at the forefront of AI
research, enabling LLMs to process and understand multi-modal information.
Recent work in LLaVA-1.5 [27] uses a pretrained CLIP visual encoder
ViT-L/14-336px to extract the visual features
$\mathbf{e}=f_{\mathbf{v}}(\texttt{img})$. These visual features are then
projected as a sequence of visual tokens into the word embedding space using
an Multi-Layer Perceptron (MLP) denoted as $\mathbf{W}$. The visual tokens,
$\mathbf{H_{v}}=\mathbf{W}\cdot\mathbf{e}$, are then concatenated with
language tokens $\mathbf{H_{q}}$ as input
$\mathbf{H}=[\mathbf{H_{v}};\mathbf{H_{q}}]$ to the language model to generate
language response. The vision capability of VLMs hinges on the visual encoder.
Therefore, by unlearning certain concepts from the CLIP vision model, we can
influence the language model’s understanding and generation of responses.

##  4 Experiments

###  4.1 Experiment setup

Unlearning scenarios. Our experiments addressed two main types of unlearning
scenarios for practical and generalizable impact: identity unlearning for
privacy concerns and the unlearning of copyrighted content for compliance with
legal standards. The focus was primarily on large scale multimodal models,
including CLIP, stable diffusion, and VLMs.

Models and datasets. We employed various models in our experiments to
demonstrate the broad applicability of our unlearning method. For CLIP models,
we utilized architectures ranging from ViT-B-32 to EVA01-g-14, trained on
LAION-400M dataset [42], and model weights sourced from the OpenCLIP
repository [6]. For stable diffusion models, we opted for the latest version
from StabilityAI, which incorporates the CLIP-ViT-H-14 trained on the LAION-5B
dataset. For VLMs, we tested the improved LLaVA-v1.5 model from HuggingFace,
which employs a CLIP ViT-L/14-336px model, trained by OpenAI, as the visual
extractor. To unlearn an identity, we curate the forget set by filtering the
LAION-400M dataset to isolate image-text pairs specific to each identity. In
our experiments, the number of filtered image-text pairs for each identity
ranged from around 1k to 6k. For the retain set, we use a single shard of
LAION-400M dataset, which typically consists of 10K images. However, due to
expiring URL inks, we were only able to download approximately 7900 images
from each shard.To assess the effectiveness of our unlearning method on
various identities, we utilize the CelebA dataset [30], which features over
10K celebrities from diverse backgrounds. We sample 100 celebrities with high
frequency in Laion-400M for evaluation. For evaluating the general utility of
the models post-unlearning, we employ the ImageNet dataset.

Evaluation. We evaluate the unlearning performance using forget accuracy, i.e.
zero-shot classification accuracy of unlearned contents. We follow the common
zero-shot paradigm [36], where the prediction label of any given image is the
text string whose text embedding is closest to the image embedding. For
utility performance, we use zero-shot classification accuracy on ImageNet and
CelebA dataset. Besides quantitative results, we also provide qualitative
results of stable diffusion generated images before and after unlearning,
question answering results of VLMs before and after unlearning.

Comparing methods. We compare with state-of-the-art methods along with
classical methods. Specifically, we consider fine tuning (FT) [46], gradient
ascent (GA) / negative gradient (NG) [45], saliency unlearning (SalUn) [10],
selective synaptic dampening (SSD) [11] for CLIP zero-shot classification. For
unlearning stable diffusion models, we compare with SalUn [10] and ESD [12].

###  4.2 Localizing layers and unlearning for CLIP

Our approach effectively pinpoints critical layers for updates, narrowing the
search space from tens or hundreds of layers to just a few, as illustrated in
Figures 2, 11 and 15. This strategy provides clear guidelines on which parts
of the model to modify, revealing the distinct roles and mechanisms different
layers serve in processing and integrating information within their respective
architectures. We observe that, across different identities (see Figure 11)
and architectures (see Figure 15), the final projection layers of both vision
and language models are selected, due to their roles of transforming complex
features extracted by the previous layers for making final predictions. We
also observe that the late attention layers in vision models and early
attention layers in language models. Vision transformers utilize attention
mechanisms to focus on different parts of an image and aggregate contextual
information from various spatial regions. The late attention layers in these
models are closer to the output, thus more specialized in refining and
utilizing contextually rich, high-level features. In contrast, language models
often employ attention mechanisms right from the early layers to capture and
process the sequential and contextual dependencies inherent in textual data.
Early attention layers are crucial for establishing a foundational
understanding of the language structure, including syntax and semantics. By
focusing on early layers, modifications can influence the foundational
processing of input text, effectively guiding the subsequent layers’
interpretation and response to the content.

We demonstrate that modifying a single layer suffices to unlearn a concept
while preserving the model’s overall utility, as depicted in Figures 2, 3 and
Table 1. It is noteworthy that the performance on unrelated tasks, like
ImageNet for common object recognition, remains intact. However, for closely
related tasks such as CelebA, which focuses on face recognition, there is a
slight impact on performance. As shown in Table 1, our method outperforms
other comparing methods on different metrics. In contrast, other methods
require careful tuning of hyper-parameters, such as learning rate, to achieve
a balance between unlearning effectiveness and utility preservation.

(a) Original cosine similarity matrix

(b) Cosine similarity matrix after unlearning

Figure 3: Cosine similarity matrix of image and text pairs before (a) and after (b) unlearning an identity - Elon Musk as an example. The original model can associate image and text of the same person with high cosine similarity. After unlearning, the image and text pair of Elon Musk are no longer matched, while other persons are only slightly affected. Here the $11_{\text{th}}$ attention out projection layer of the vision model ViT-B-32 is unlearned. Table 1: Performance comparison of different unlearning methods on CLIP model. FA@1 and FA@5 represents the top-1 and top-5 forget accuracy (%), i.e. zero-shot classification accuracy of unlearned identity (average of 5 identities from Figure 3). TA_IN@1 and TA_CA@1 stands for the top-1 zero-shot test accuracy (%) on ImageNet val set and CelebA dataset respectively. In time complexity analyses, $k$ denotes the number of iterations for unlearning, which we set to $10$ in our experiments. $K$ is the number of epochs for training, which is $32$. $N$ is the size of whole training set, which is much larger than our sampled forget and retain set $N_{\texttt{f}}$ and $N_{\texttt{r}}$. Method | FA$@1$ $\downarrow$ | FA$@5$ $\downarrow$ | TA_IN$@1$ $\uparrow$ | TA_CA$@1$ $\uparrow$ | Compute Time ($\mathcal{O}$)
---|---|---|---|---|---
Original | $73.05$ | $92.22$ | $60.12$ | $61.38$ | $\mathcal{O}(K\cdot N)$
lr = $10^{-6}$
FT [46] | $66.08$ | $90.10$ | $60.36$ | $60.70$ | $\mathcal{O}(k\cdot N_{\texttt{r}})$
GA [45] | $0.00$ | $0.00$ | $35.88$ | $24.92$ | $\mathcal{O}(k\cdot N_{\texttt{f}})$
GAFT | $0.00$ | $0.00$ | $55.52$ | $25.71$ | $\mathcal{O}(k\cdot(N_{\texttt{f}}+N_{\texttt{r}}))$
SalUn [10] | $0.00$ | $0.00$ | $55.45$ | $26.11$ | $\mathcal{O}(N_{\texttt{f}})+\mathcal{O}(k\cdot(N_{\texttt{f}}+N_{\texttt{r}}))$
lr = $10^{-7}$
FT [46] | $70.50$ | $92.22$ | $60.26$ | $61.35$ | $\mathcal{O}(k\cdot N_{\texttt{r}})$
GA [45] | $0.00$ | $4.91$ | $60.03$ | $53.86$ | $\mathcal{O}(k\cdot N_{\texttt{f}})$
GAFT | $2.67$ | $15.89$ | $60.13$ | $55.22$ | $\mathcal{O}(k\cdot(N_{\texttt{f}}+N_{\texttt{r}}))$
SalUn [10] | $3.33$ | $15.69$ | $60.26$ | $55.81$ | $\mathcal{O}(N_{\texttt{f}})+\mathcal{O}(k\cdot(N_{\texttt{f}}+N_{\texttt{r}}))$
SSD [11] | $0.00$ | $0.00$ | $51.84$ | $35.96$ | $\mathcal{O}(N_{\texttt{f}}+N_{\texttt{r}})$
Ours | $0.00$ | $0.00$ | $59.96$ | $58.32$ | $\mathcal{O}(N_{\texttt{f}}+N_{\texttt{r}})$

Further, we explore the simultaneous unlearning across multiple layers for
various tasks in Section 4.5. Additional examples of unlearning, extending
beyond identities, are detailed in Section 4.6.

###  4.3 Unlearning stable diffusion

Stable diffusion models exhibit remarkable capabilities in text comprehension
and the generation and manipulation of personal images. For instance, when
prompted with "A portrait of Elon Musk," these models can produce a high-
fidelity portrait. Moreover, by altering the prompt, one can generate
imaginative content, such as a vivid depiction of "Elon Musk on Mars."
However, the potential misuse of such powerful tools raises significant
concerns regarding the harm they can cause to the data provider. In this
study, we demonstrate a method to effectively erase personal information from
the image generation model, ensuring that prompts related to the erased
individual fail to produce accurate results. As shown in Figure 4, our
approach to unlearning Elon Musk interestingly results in representations of
electronic circuits, consistently across various prompts, without compromising
the model’s general capability to generate a diverse range of other objects.
Contrarily, alternative methods not only struggle to accurately render
portraits of other individuals but also degrade the image quality of unrelated
objects.

Figure 4: Unlearn performance on stable diffusion. First row shows the
generated images from the original pretrained model, the second row shows the
output of unlearned model, where Elon Musk is unlearned. We can see that Elon
Musk is effectively unlearned, whereas other objects are unaffected. Comparing
methods in the bottom two rows suffer from hurting general utility.

In addition to identity removal for privacy protection, we address copyright
concerns that increasingly challenge generative models. We successfully apply
our method to remove copyright-protected content, specifically targeting well-
known characters such as Marvel’s “Iron Man" and Walt Disney’s “Mickey Mouse."
Figure 5 illustrates that our technique precisely unlearns the targeted
concepts, effectively disabling the generation of images associated with these
copyrighted entities while preserving the model’s capability to produce images
of other concepts. Further examples are detailed in Section D.

Figure 5: Qualitative evaluation on unlearning copyright characters “Iron man”
and “Mickey Mouse” from the Stable Diffusion. Our method precisely unlearned
copyright protected concepts from SD, while the image generation quality on
other concepts is highly preserved.

###  4.4 Unlearning VLMs

VLMs demonstrate impressive ability in visual understanding and question
answering. For example, when provided with an image of a person, these models
can accurately identify and name the individual depicted. Figure 6
demonstrates this by showing that when given an image of Elon Musk and asked,
“What’s the name of the person in this image?”, the model correctly identifies
him. Our method effectively makes the model "forget" certain subjects; for
instance, it erroneously identifies Elon Musk as Michael Jackson. This
alteration does not significantly impact the model’s overall utility.
Additional examples of this phenomenon are discussed in Section E.

Figure 6: Unlearn performance on LLaVA 1.5. First row shows the generated
images from the original pretrained model, the second row shows the output of
unlearned model, where Elon Musk is unlearned. We can see that Elon Musk is
effectively unlearned, whereas other subjects are unaffected.

###  4.5 Joint update for unlearning multiple identities

We study the composite effect of our approach where we unlearn multiple tasks
simultaneously. For instance, in the task of unlearning multiple identities,
we take the gradients calculated for each identity on the original model, and
then perform layer updates to simultaneously forget all of them. We
demonstrate our results in Figure. 7, where we successfully unlearn (a) {Elon
Musk, Mark Zuckerberg} and (b) {Elon Musk, Taylor Swift}.

(a) Cosine similarity matrix after unlearning
Elon Musk and Mark Zuckerberg

(b) Cosine similarity matrix after unlearning Elon Musk and Taylor Swift

Figure 7: Cosine similarity matrix of image and text pairs after unlearning
multiple name identities (see Figure. 12(a) for cosine similarity matrix on
original model). After unlearning, the image and text pair of both selected
names are not matched, while other persons are only slightly affected. Here
the $9_{\text{th}}$ vision attention out projection layer and the
$11_{\text{th}}$ vision attention out projection layer are unlearned for Elon
Musk and the other identity respectively. CLIP model: ViT-B-32.

Furthermore, we investigate how the unlearning performance varies as the
number of identities to be forgotten increases. The cosine similarity matrices
presented in Figure 8 demonstrate our effectiveness after identifying
different layers associated with different names (see Figure 11), and
performing joint updating on different layers.

(a) $N=1$

(b) $N=2$

(c) $N=3$

(d) $N=4$

(e) $N=5$

(f) $N=6$

Figure 8: Cosine similarity matrices after unlearning $N$ identities, where
$N\in\\{1,2,...,6\\}$.

###  4.6 Unlearn different concepts

In addition to unlearning personal identities from CLIP, we also sample $7$
classes {Basketball, Beach, Castle, Revolver, Rifle, School bus, Sunglasses}
from ImageNet to evaluate the unlearning performance of our method on object
concepts. For this experiment, we use $10$k ImageNet validation images. The
results, presented in Table. 2, show the CLIP zero-shot accuracy evaluations
for both the forgetting of sampled classes and the retention of other ImageNet
classes after unlearning. Our findings indicate that our method effectively
reduces the CLIP zero-shot accuracy for the targeted classes to $0.0\%$, while
the accuracy for remaining classes stays high, experiencing only minimal
degradation (ranging from $0.03\%$ to $2.03\%$) compared to the original pre-
trained model..

Table 2: Unlearning performance of our method on common object concepts. FA@1 and FA@5 represents the top-1 and top-5 forget accuracy (%) of each class, i.e. zero-shot classification accuracy of unlearned class. FA@1 and FA@5 represents the top-1 and top-5 forget accuracy (%) of all classes of ImageNet. Each row shows the forget class accuracy and average accuracy over all classes of ImageNet before and after unlearning a class. Our method can reduce the accuracy of desired classes to $0.0\%$ while keeping the accuracy of retaining classes close to original model (within $0.06-2.03\%$ difference). CLIP model: ViT-B-32. Forget class | Original | Unlearned
---|---|---
FA$@1$ | FA$@5$ | TA$@1$ | TA$@5$ | FA$@1$ $\downarrow$ | FA$@5$ $\downarrow$ | TA$@1$ $\uparrow$ | TA$@5$ $\uparrow$
Basketball | 100.0 | 100.0 | 60.16 | 85.52 | 0.0 | 0.0 | 59.18 | 84.48
Beach | 54.55 | 72.73 | 0.0 | 0.0 | 59.54 | 84.78
Castle | 87.50 | 100.0 | 0.0 | 0.0 | 58.13 | 83.87
Revolver | 100.0 | 100.0 | 0.0 | 0.0 | 59.94 | 85.43
Rifle | 42.86 | 57.14 | 0.0 | 0.0 | 60.08 | 85.49
School bus | 76.92 | 100.0 | 0.0 | 0.0 | 59.50 | 89.18
Sunglasses | 44.44 | 55.56 | 0.0 | 0.0 | 60.13 | 85.23

##  5 Conclusion

In this study, we introduced a new machine unlearning method characterized by
its computational efficiency and efficacy. Our approach, which requires only a
single computation of gradients and updates a single layer of the model, not
only enhances the feasibility of unlearning large models but also retains the
general utility of the models involved. Through rigorous testing on various
models, including CLIP, stable diffusion, and VLMs, our method has been shown
superior to other state-of-the-art techniques, particularly in its ability to
maintain performance on tasks unrelated to the data being unlearned. Moreover,
this method’s modular design allows for the simultaneous unlearning of
multiple concepts, thereby broadening its applicability and utility in
practical scenarios. Future research could expand upon the foundational
strategies proposed here to further refine the balance between unlearning
effectiveness and utility retention, potentially transforming practices in
data privacy and compliance within machine learning.

## References

  * [1] Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation) (text with eea relevance).  Official Journal of the European Union, vol. 119, pp. 1–88, 2016\.  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv%3AOJ.L_.2016.119.01.0001.01.ENG.
  * [2] Evaluation for the neurips machine unlearning competition.  August 2023.  https://www.kaggle.com/competitions/neurips-2023-machine-unlearning/data?select=Machine_Unlearning_Notion_Metric.pdf.
  * [3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.  https://arxiv.org/abs/2303.08774.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.  https://arxiv.org/abs/2005.14165.
  * [5] Yinzhi Cao and Junfeng Yang.  Towards making systems forget with machine unlearning.  In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015.  https://ieeexplore.ieee.org/document/7163042.
  * [6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023.  https://arxiv.org/abs/2212.07143.
  * [7] Eli Chien, Chao Pan, and Olgica Milenkovic.  Certified graph unlearning.  arXiv preprint arXiv:2206.09140, 2022.  https://arxiv.org/abs/2206.09140.
  * [8] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), volume 1, pages 539–546. IEEE, 2005.  https://ieeexplore.ieee.org/document/1467314.
  * [9] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli.  Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217, 2023.  https://arxiv.org/abs/2205.08096.
  * [10] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  ICLR, 2024.  https://arxiv.org/abs/2310.12508.
  * [11] Jack Foster, Stefan Schoepf, and Alexandra Brintrup.  Fast machine unlearning without retraining through selective synaptic dampening.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12043–12051, 2024.  https://arxiv.org/abs/2308.07707.
  * [12] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau.  Erasing concepts from diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436, 2023.
  * [13] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein.  What do vision transformers learn? a visual exploration.  arXiv preprint arXiv:2212.06727, 2022.  https://arxiv.org/abs/2212.06727.
  * [14] Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru.  Towards adversarial evaluations for inexact machine unlearning.  arXiv preprint arXiv:2201.06640, 2022.  https://arxiv.org/abs/2201.06640.
  * [15] Aditya Golatkar, Alessandro Achille, and Stefano Soatto.  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304–9312, 2020.  https://arxiv.org/abs/1911.04933.
  * [16] Aditya Golatkar, Alessandro Achille, and Stefano Soatto.  Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pages 383–398. Springer, 2020.  https://arxiv.org/abs/2003.02960.
  * [17] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten.  Certified data removal from machine learning models.  ICML, 2020.  https://arxiv.org/abs/1911.03030.
  * [18] Babak Hassibi, David G Stork, and Gregory J Wolff.  Optimal brain surgeon and general network pruning.  In IEEE international conference on neural networks, pages 293–299. IEEE, 1993.  https://ieeexplore.ieee.org/document/298572.
  * [19] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu.  Model sparsification can simplify machine unlearning.  NeurIPS, 2023.  https://arxiv.org/abs/2304.04934.
  * [20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.  https://arxiv.org/abs/2001.08361.
  * [21] Steven M Kay.  Fundamentals of statistical signal processing: estimation theory.  Prentice-Hall, Inc., 1993.  https://dl.acm.org/doi/abs/10.5555/151045.
  * [22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.  https://arxiv.org/abs/1612.00796.
  * [23] Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafillou.  Towards unbounded machine unlearning.  arXiv preprint arXiv:2302.09880, 2023.  https://arxiv.org/abs/2302.09880.
  * [24] Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, and Steffen Eger.  Chatgpt: A meta-analysis after 2.5 months.  Machine Learning with Applications, 16:100541, 2024.  https://arxiv.org/abs/2302.13795.
  * [25] Guihong Li, Hsiang Hsu, Radu Marculescu, et al.  Machine unlearning for image-to-image generative models.  ICLR, 2024.  https://arxiv.org/abs/2402.00351.
  * [26] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia.  Mini-gemini: Mining the potential of multi-modality vision language models.  arXiv preprint arXiv:2403.18814, 2024.  https://arxiv.org/abs/2403.18814.
  * [27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.  https://arxiv.org/abs/2310.03744.
  * [28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.  https://arxiv.org/abs/2304.08485.
  * [29] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al.  Rethinking machine unlearning for large language models.  arXiv preprint arXiv:2402.08787, 2024.  https://arxiv.org/abs/2402.08787.
  * [30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.  https://arxiv.org/abs/1411.7766.
  * [31] R Timothy Marler and Jasbir S Arora.  The weighted sum method for multi-objective optimization: new insights.  Structural and multidisciplinary optimization, 41:853–862, 2010\.  https://link.springer.com/article/10.1007/s00158-009-0460-7.
  * [32] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen.  A survey of machine unlearning.  arXiv preprint arXiv:2209.02299, 2022.  https://arxiv.org/abs/2209.02299.
  * [33] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  ICML, 2022.  https://arxiv.org/abs/2112.10741.
  * [34] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2(11):e7, 2017.  https://distill.pub/2017/feature-visualization/.
  * [35] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.  https://arxiv.org/abs/2307.01952.
  * [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.  https://arxiv.org/abs/2103.00020.
  * [37] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.  https://openai.com/index/language-unsupervised/.
  * [38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.  https://openai.com/index/better-language-models/.
  * [39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.  https://arxiv.org/abs/2112.10752.
  * [40] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  ICLR, 2022.  https://arxiv.org/abs/2202.00512.
  * [41] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.  https://arxiv.org/abs/2210.08402.
  * [42] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.  https://arxiv.org/abs/2111.02114.
  * [43] Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li.  Exploring the landscape of machine unlearning: A survey and taxonomy.  arXiv preprint arXiv:2305.06360, 2023.  https://arxiv.org/abs/2305.06360.
  * [44] David Thiel.  Identifying and eliminating csam in generative ml training data and models.  Technical report, Technical report, Stanford University, Palo Alto, CA, 2023. URL https://purl.stanford.edu/kh752sm9123, 2023.
  * [45] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot.  Unrolling sgd: Understanding factors influencing machine unlearning.  In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022.  https://arxiv.org/abs/2109.13398.
  * [46] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck.  Machine unlearning of features and labels.  Network and Distributed System Security Symposium (NDSS), 2023.  https://arxiv.org/abs/2108.11577.
  * [47] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  ACM Computing Surveys, 56(4):1–39, 2023.  https://arxiv.org/abs/2209.00796.
  * [48] Yuanshun Yao, Xiaojun Xu, and Yang Liu.  Large language model unlearning.  ICLR, 2024.  https://arxiv.org/pdf/2310.10683.
  * [49] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014.  https://arxiv.org/abs/1311.2901.
  * [50] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu.  Vision-language models for vision tasks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024\.  https://arxiv.org/abs/2304.00685.

##  Appendix A More named identities

In addition to the experiment of unlearning name “Elon Mask” from CLIP model
discussed in Sec. 4.2 of the main text, we have expanded our study to include
a broader set of identities: {Kanye West, Barack Obama, Bruce Lee, Fan
Bingbing, Lady Gaga}. These names were selected from the CelebA dataset to
represent a diverse cross-section of ethnicities and genders. Our methodology
effectively pinpointed the crucial layers associated with each name. These
layers can then be specifically targeted to efficiently unlearn the
corresponding identity from the CLIP model.

Figure 11 illustrates the Pareto-front plots that identify important layers
selected by our method for unlearning various name identities. Furthermore, as
demonstrated in Figure 12, our approach successfully removed the desired names
from the CLIP model.

(a) Vision layer Pareto - Mark Zuckerberg

(b) Language layer Pareto - Mark Zuckerberg

(c) Vision layer Pareto - Jeff Bezos

(d) Language layer Pareto - Jeff Bezos

(e) Vision layer Pareto - Taylor Swift

(f) Language layer Pareto - Taylor Swift

(a) Vision layer Pareto - Kim Kardashian

(b) Language layer Pareto - Kim Kardashian

(c) Vision layer Pareto - Kanye West

(d) Language layer Pareto - Kanye West

(e) Vision layer Pareto - Barack Obama

(f) Language layer Pareto - Barack Obama

(a) Vision layer Pareto - Bruce Lee

(b) Language layer Pareto - Bruce Lee

(c) Vision layer Pareto - Fan Bingbing

(d) Language layer Pareto - Fan Bingbing

(e) Vision layer Pareto - Lady Gaga

(f) Language layer Pareto - Lady Gaga

Figure 11: More identities, in addition to Figure 2. CLIP variant: ViT-B-32

(a) Original cosine similarity matrix

(b) Cosine similarity matrix after unlearning

Figure 12: Cosine similarity matrix of image and text pairs before and after
unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk
are not matched, while other persons are only slightly affected. Here the
vision attention out projection layer at the 9th resblock is unlearned. CLIP
model: ViT-B-16

##  Appendix B More CLIP models

Here we show experiments using an expanded set of model architectures:
{ViT-B-16, ViT-L-14, EVA01-g-14}. As illustrated in Figure 15, our method
demonstrates scalability and effectiveness across a range of model sizes, from
$149.62$ million parameters (ViT-B-16) to $1.136$ billion parameters
(EVA01-g-14). This underscores the flexibility of our approach to accommodate
models of different scales.

(a) Original cosine similarity matrix

(b) Cosine similarity matrix after unlearning

Figure 13: Cosine similarity matrix of image and text pairs before and after
unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk
are not matched, while other persons are only slightly affected. Here the
vision projection layer within the MLP of the 23rd resblock is unlearned. CLIP
variant: ViT-L-14

(a) Original cosine similarity matrix

(b) Cosine similarity matrix after unlearning

Figure 14: Cosine similarity matrix of image and text pairs before and after
unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk
are not matched, while other persons are only affected. Here the language
attention out projection layer at the 11th resblock is unlearned. CLIP model:
EVA01-g-14.

(a) Vision layer Pareto - ViT-B-16

(b) Language layer Pareto - ViT-B-16

(c) Vision layer Pareto - ViT-L-14

(d) Language layer Pareto - ViT-L-14

(e) Vision layer Pareto - EVA01-g-14

(f) Language layer Pareto - EVA01-g-14

Figure 15: More CLIP models, in addition to Sec 4.2. Unlearning name Elon Musk
from CLIP variant: {ViT-B-16, ViT-L-14, and EVA01-g-14}

##  Appendix C Linearity of unlearning trajectory of different layers

In addition to the layers presented in Figure 2 (c) and (d), we show in Figure
16 that different layers show similar monotonic behaviors however their
utility performance may vary, thus it is important to select the best layer
from the pareto set for best performance.

(a) Vision layer visual.proj

(b) Language layer text_projection

(c) Vision layer 11.mlp.c_fc

(d) Language layer 11.attn.in_proj

Figure 16: More examples of unlearning different layers. Correspond to Figure
2. The performance changes monotonically with the step size $\lambda$.

##  Appendix D More examples on SD

To demonstrate the performance and practical utility of our method, we further
consider unlearning more celebrity names and more scenarios including
unlearning copyright characters and novel concepts from Stable Diffusion.

More celebrity name. Beyond unlearning “Elon Musk” from Stable Diffusion,
which is presented in the Figure 4, here we also provide additional
qualitative evaluations on unlearning other celebrity names {Taylor Swift,
Jeff Bezos} with our method in Figure 17.

Figure 17: Qualitative evaluation on unlearning celebrity names Taylor Swift
and Jeff Bezos from the Stable Diffusion.

Novel concept. One of the intriguing properties of the Stable Diffusion is its
ability to generalize image generation to novel concepts that are infrequently
or never observed in the real world. In this experiment, we explore the
unlearning of a unique concept, “Avocado chair” from Stable Diffusion. We
first generate $500$ image using SD with the prompt “An avocado chair” to
create the forget set. In Figure 18, we show that our method successfully
unlearn the concept “Avocado chair” from SD, resulting in the model’s
inability to generate images corresponding to this specific concept.

However, it is noteworthy that the model’s capability to generate images
related to the constituent atomic concepts—namely “Avocado" and “Chair"—is
also compromised. We hypothesize that this occurs due to the model’s treatment
of novel concepts as compositions of atomic concepts. For example, the concept
"Avocado chair" is interpreted by the model as “Avocado" plus “Chair."
Consequently, when a novel concept is unlearned, the associated atomic
concepts are inadvertently affected as well. This highlights a challenge in
the model’s approach to handling the interoperability of novel and atomic
concepts.

Figure 18: Qualitative evaluation on unlearning a novel concept “Avocado
chair” from the Stable Diffusion.

##  Appendix E More examples on VLM

In addition to results presented in the main text Figure 6, we also present
additional results on unlearning a different name “Taylor Swift” from VLM in
Figure 19. We demonstrate that our method can anonymizes celebrity names from
the pretrained Vision-language models, and simultaneously preserves the
model’s ability on image understanding, reasoning and distribution shift
robustness on art work, cartoon style images.

Figure 19: Qualitative evaluation on unlearning name “Taylor Swift” from LLaVA
1.5. While “Taylor Swift” is mapped to “woman” after the unlearning, the other
female celebrity identification remain unaffected. Besides, model’s robustness
against style distribution shift is also preserved.

##  Appendix F Algorithm pseudo code

Algorithm 1 SLSGU: Single Layer Single Gradient Unlearning

0: Forget set $D_{\texttt{f}}$ and retain set $D_{\texttt{r}}$ ; Original
model $F_{\theta}$ with model weights $\theta$; The set of all layers in the
model, as $L$;Forget loss function $\mathcal{L}_{\text{forget}}$ and retain
loss function $\mathcal{L}_{\text{retain}}$; Evaluation metrics forget
accuracy FA and test accuracy TA.

0: Unlearned model parameters $\theta_{\texttt{f}}$

1: Calculate and store gradients
$\nabla_{\theta}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}}),\nabla_{\theta}\mathcal{L}_{\text{retain}}(\theta,D_{\texttt{r}})$
$\triangleright$ Single gradient calculation

2: for each layer $l$ in $L$ do

3:
$\text{Importance}(l)=\|\nabla_{\theta_{l}}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}})\|_{2}/\|\theta_{l}\|_{2}$
$\triangleright$ Calculate layer importance

4:
$\text{Alignment}(l)=\text{cos}\bigl{(}\nabla_{\theta_{l}}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}}),\nabla_{\theta_{l}}\mathcal{L}_{\text{retain}}(\theta,D_{\texttt{r}})\bigr{)}$
$\triangleright$ Calculate layer alignment

5: end for

6: $P=\textbf{PO}(L,\text{Importance},\text{Alignment})$ $\triangleright$
Pareto optimal algorithm 2

7: $Q\leftarrow\emptyset$ $\triangleright$ Set of layers and their
performances

8: for each layer $l$ in $P$ do

9: $\lambda_{0}=\text{Importance}(l)/10$ $\triangleright$ Initialize step size

10: $(\lambda,\texttt{FA},\texttt{TA})=\textbf{BS}(\lambda_{0},l)$
$\triangleright$ Binary search algorithm 3

11: $Q\leftarrow Q\cup\\{(l,\lambda,\texttt{FA},\texttt{TA})\\}$

12: end for

13: $\texttt{FA}_{\min}=\min_{(l,\lambda,\texttt{FA},\texttt{TA})\in
Q}\texttt{FA}$ $\triangleright$ Identify minimum FA

14: $Q_{\min}=\\{(l,\lambda,\texttt{FA},\texttt{TA})\in Q\
|\texttt{FA}=\texttt{FA}_{\min}\\}$ $\triangleright$ Filter sets with minimum
FA

15:
$(l^{*},\lambda^{*},\texttt{FA}^{*},\texttt{TA}^{*})=\arg\max_{(\lambda,\texttt{FA},\texttt{TA})\in
Q_{\min}}(\texttt{TA})$ $\triangleright$ Select set with highest TA

16: return
$\theta_{\texttt{f}}=\theta-\lambda^{*}\nabla_{\theta_{l^{*}}}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}})$

Algorithm 2 Pareto Optimal:
$P=\textbf{PO}(L,\text{Importance},\text{Alignment})$

0: The set of all layers in the model, as $L$;Layer importance and gradient
alignment of all layers

0: The set of Pareto optimal layers

1: Initialize $P\leftarrow\emptyset$ $\triangleright$ Set of layers on the
Pareto front is empty

2: for each layer $l$ in $L$ do

3: ParetoDominant $\leftarrow$ true

4: for each layer $l^{\prime}$ in $L\setminus l$ do

5: if ($\text{Importance}(l^{\prime})>\text{Importance}(l)$ and
$\text{Alignment}(l^{\prime})<\text{Alignment}(l)$) then

6: ParetoDominant $\leftarrow$ false

7: break

8: end if

9: end for

10: if ParetoDominant then

11: $P\leftarrow P\cup\\{l\\}$ $\triangleright$ Identified a Pareto optimal
layer

12: end if

13: end for

14: return $P$ $\triangleright$ Return the set of Pareto optimal layers

Algorithm 3 Binary Search for Optimal Step Size:
$(\lambda^{*},\texttt{FA}^{*},\texttt{TA}^{*})=\textbf{BS}(\lambda_{0},l)$

0: Initial step size $\lambda_{0}$; Maximum number of steps $K$; Model
parameters $\theta$; Forget gradient of layer l:
$G_{l}=\nabla_{\theta}\mathcal{L}_{\text{forget}}(\theta,D_{\texttt{f}})$

0: Optimal $\lambda^{*}$, forget accuracy FA, test accuracy TA

1: $\lambda_{\text{low}}\leftarrow 0$

2: $\lambda_{\text{high}}\leftarrow\infty$

3: $\lambda\leftarrow\lambda_{0}$

4: $k\leftarrow 0$

5: Initialize $P\leftarrow\emptyset$ $\triangleright$ Performance set

6: while $k<K$ do

7: $\texttt{FA},\texttt{TA}=\texttt{eval}(\theta-\lambda G_{l})$

8: $P\leftarrow P\cup\\{(\lambda,\texttt{FA},\texttt{TA})\\}$ $\triangleright$
Store results

9: if $\texttt{FA}>0$ then

10: $\lambda_{\text{low}}\leftarrow\lambda$ $\triangleright$ Should increase
step size to unlearn

11: else

12: $\lambda_{\text{high}}\leftarrow\lambda$ $\triangleright$ Should reduce
step size to avoid over-unlearning

13: end if

14: if $\lambda_{\text{high}}==\infty$ then

15: $\lambda\leftarrow 2\lambda$

16: else

17: $\lambda\leftarrow(\lambda_{\text{low}}+\lambda_{\text{high}})/2$

18: end if

19: $k\leftarrow k+1$

20: end while

21: $\texttt{FA}_{\min}=\min_{(\lambda,\texttt{FA},\texttt{TA})\in
P}\texttt{FA}$ $\triangleright$ Identify minimum FA

22: $P_{\min}=\\{(\lambda,\texttt{FA},\texttt{TA})\in P\
|\texttt{FA}=\texttt{FA}_{\min}\\}$ $\triangleright$ Filter sets with minimum
FA

23:
$(\lambda^{*},\texttt{FA}^{*},\texttt{TA}^{*})=\arg\max_{(\lambda,\texttt{FA},\texttt{TA})\in
P_{\min}}(\texttt{TA})$ $\triangleright$ Select set with highest TA

24: return $\lambda^{*},\texttt{FA}^{*},\texttt{TA}^{*}$ $\triangleright$
Select the set with lowest FA which has the highest TA


-g\end{bmatrix}+\begin{bmatrix}\nabla_{\boldsymbol{\phi}}g^{T}g\\\
\nabla_{\boldsymbol{\theta}}f^{T}f\end{bmatrix}=\begin{bmatrix}\nabla_{\boldsymbol{\phi}}E\\\
-\nabla_{\boldsymbol{\theta}}E\end{bmatrix}+\frac{1}{2}\begin{bmatrix}\nabla_{\boldsymbol{\phi}}\left\lVert\nabla_{\boldsymbol{\theta}}E\right\rVert^{2}\\\
\nabla_{\boldsymbol{\theta}}\left\lVert\nabla_{\boldsymbol{\phi}}E\right\rVert^{2}\end{bmatrix}=\begin{bmatrix}\nabla_{\boldsymbol{\phi}}(E+\frac{1}{2}\left\lVert\nabla_{\boldsymbol{\theta}}E\right\rVert^{2})\\\
\nabla_{\boldsymbol{\theta}}(-E+\frac{1}{2}\left\lVert\nabla_{\boldsymbol{\phi}}E\right\rVert^{2})\end{bmatrix}$
(A.61)

Therefore, since $\hat{\boldsymbol{\xi}}$ defines the negative of the vector
field followed by the system, the modified losses for the two players can be
written respectively as:

$\displaystyle\widetilde{L}_{1}$
$\displaystyle=E+\frac{1}{2}\left\lVert\nabla_{\boldsymbol{\theta}}E\right\rVert^{2}$
(A.62) $\displaystyle\widetilde{L}_{2}$
$\displaystyle=-E+\frac{1}{2}\left\lVert\nabla_{\boldsymbol{\phi}}E\right\rVert^{2}$
(A.63)

The functional form of the modified losses given by SGA is the same used to
cancel the interaction terms of DD in the case of simultaneous gradient
descent updates in zero sum games. We do however highlight a few differences
in our approach compared to SGA: our approach extends to alternating updates
and provides the optimal regularization coefficients; canceling the
interaction terms of the drift is different compared to SGA for general games
(see Equation A.60 for SGA and Theorem 3.1 for the interaction terms of DD).

##  Appendix G DiracGAN - an illustrative example

In their work assessing the convergence of GAN training Mescheder et al.
(2018) introduce the example of the DiracGAN, where the GAN is trying to learn
a delta distribution with mass at zero. More specifically, the generator
$G_{\theta}(z)=\theta$ with parameter $\theta$ parametrizes constant functions
whose images $\\{\theta\\}$ correspond to the support of the delta
distribution $\delta_{\theta}$. The discriminator is a linear model
$D_{\phi}(x)=\phi\cdot x$ with parameter $\phi$.

The loss function is given by:

$\displaystyle E(\theta,\phi)=l(\theta\phi)+l(0)$ (A.64)

where $f$ depends on the GAN used - for the standard GAN it is
$l=-\log(1+e^{-t})$. As in Mescheder et al. (2018), we assume $l$ is
continuously differentiable with $l^{\prime}(x)\neq 0$ for all
$x\in\mathbb{R}$. The partial derivatives of the loss function

$\displaystyle\frac{\partial
E}{\partial\phi}=l^{\prime}(\theta\phi)\theta,\hskip 30.00005pt\frac{\partial
E}{\partial\theta}=l^{\prime}(\theta\phi)\phi,$

lead to the underlying continuous dynamics:

$\displaystyle\dot{\phi}=f(\theta,\phi)=l^{\prime}(\theta\phi)\theta,\hskip
30.00005pt\dot{\theta}=g(\theta,\phi)=-l^{\prime}(\theta\phi)\phi.$ (A.65)

Thus the only equilibrium of the game is $\theta=0$ and $\phi=0$.

###  G.1 Reconciling discrete and continuous updates in Dirac-GAN

Mescheder et al. (2018) observed a discrepancy between the continuous and
discrete dynamics. They show that, for the problem in Equation (A.64), the
continuous dynamics preserve $\theta^{2}+\phi^{2}$, and thus cannot converge
(Lemma 2.3 in Mescheder et al. (2018)), since:

$\displaystyle\frac{d\left(\theta^{2}+\phi^{2}\right)}{dt}=2\theta\frac{d\theta}{dt}+2\phi\frac{d\phi}{dt}=-2\theta
l^{\prime}(\theta\phi)\phi+2\phi l^{\prime}(\theta\phi)\theta=0.$

They also observe that with the discrete dynamics of simultaneous gradient
descent that $\theta^{2}+\phi^{2}$ increases in time (Lemma 2.4 in Mescheder
et al. (2018)). We resolve this discrepancy here, by showing that the modified
continuous dynamics given by discretization drift result in behavior
consistent with that of the discrete updates.

######  Proposition G.1.

The continuous vector field given by discretization drift for simultaneous
Euler updates in DiracGAN increases $\theta^{2}+\phi^{2}$ in time.

###### Proof.

We assume both the generator and the discriminator use learning rates $h$, as
in (Mescheder et al., 2018). We first compute terms used by the zero-sum
Colloraries in the main paper.

$\displaystyle\left\lVert\nabla_{\theta}E\right\rVert^{2}=l^{\prime}(\theta\phi)^{2}\phi^{2}$
$\displaystyle\left\lVert\nabla_{\phi}E\right\rVert^{2}=l^{\prime}(\theta\phi)^{2}\theta^{2}$

and

$\displaystyle\nabla_{\theta}\left\lVert\nabla_{\theta}E\right\rVert^{2}$
$\displaystyle=2\phi^{3}l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)$
$\displaystyle\nabla_{\theta}\left\lVert\nabla_{\phi}E\right\rVert^{2}$
$\displaystyle=2\theta l^{\prime}(\theta\phi)^{2}+2\theta^{2}\phi
l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)$
$\displaystyle\nabla_{\phi}\left\lVert\nabla_{\theta}E\right\rVert^{2}$
$\displaystyle=2\phi l^{\prime}(\theta\phi)^{2}+2\phi^{2}\theta
l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)$
$\displaystyle\nabla_{\phi}\left\lVert\nabla_{\phi}E\right\rVert^{2}$
$\displaystyle=2\theta^{3}l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)$

Thus, the modified ODEs are given by:

$\displaystyle\dot{\phi}$
$\displaystyle=l^{\prime}(\theta\phi)\theta+\frac{h}{2}\left[-\theta^{3}l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)+\phi
l^{\prime}(\theta\phi)^{2}+\phi^{2}\theta
l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)\right]$
$\displaystyle\dot{\theta}$
$\displaystyle=-l^{\prime}(\theta\phi)\phi-\frac{h}{2}\left[-\theta
l^{\prime}(\theta\phi)^{2}-\theta^{2}\phi
l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)+\phi^{3}l^{\prime}(\theta\phi)l^{\prime\prime}(\theta\phi)\right]$

By denoting $l^{\prime}(\theta\phi)$ by $l^{\prime}$ and
$l^{\prime\prime}(\theta\phi)$ by $l^{\prime\prime}$, then we have:

$\displaystyle\frac{d\left(\theta^{2}+\phi^{2}\right)}{dt}$
$\displaystyle=2\theta\frac{d\theta}{dt}+2\phi\frac{d\phi}{dt}$
$\displaystyle=2\theta\left(-l^{\prime}\phi-\frac{h}{2}\left[-\theta l^{\prime
2}-\theta^{2}\phi
l^{\prime}l^{\prime\prime}+\phi^{3}l^{\prime}l^{\prime\prime}\right]\right)+2\phi\left(l^{\prime}\theta+\frac{h}{2}\left[-\theta^{3}l^{\prime}l^{\prime\prime}+\phi
l^{\prime 2}+\phi^{2}\theta l^{\prime}l^{\prime\prime}\right]\right)$
$\displaystyle=2\theta\left(-\frac{h}{2}\left[-\theta l^{\prime
2}-\theta^{2}\phi
l^{\prime}l^{\prime\prime}+\phi^{3}l^{\prime}l^{\prime\prime}\right]\right)+2\phi\left(\frac{h}{2}\left[-\theta^{3}l^{\prime}l^{\prime\prime}+\phi
l^{\prime 2}+\phi^{2}\theta l^{\prime}l^{\prime\prime}\right]\right)$
$\displaystyle=h\theta^{2}l^{\prime 2}+h\theta^{3}\phi
l^{\prime}l^{\prime\prime}-h\phi^{3}\theta
l^{\prime}l^{\prime\prime}-h\phi\theta^{3}l^{\prime}l^{\prime\prime}+h\phi^{2}l^{\prime
2}+h\phi^{3}\theta l^{\prime}l^{\prime\prime}$
$\displaystyle=h\theta^{2}l^{\prime 2}+h\phi^{2}l^{\prime 2}>0$

for all $\phi,\theta\neq 0$, which shows that $\theta^{2}+\phi^{2}$ is not
preserved and it will strictly increase for all values away from the
equilibrium (we have used the assumption that $l^{\prime}(x)\neq 0,\forall
x\in\mathbb{R}$).

We have thus identified a continuous system which exhibits the same behavior
as described by Lemma 2.4 in Mescheder et al. (2018), where a discrete system
is analysed. Figure A.2 illustrates that the divergent behavior of
simultaneous gradient descent in this case can be predicted from the dynamics
of the modified continuous system given by backward error analysis.

Figure A.2: DiracGAN. Left: The dynamics of simultaneous gradient descent
updates match the continuous dynamics given by backward error analysis for
DiracGAN. Right: Explicit regularization canceling the interaction terms of DD
stabilizes the DiracGAN trained with simultaneous gradient descent.

###  G.2 DD changes the convergence behavior of Dirac-GAN

The Jacobian of the unmodified Dirac-GAN evaluated at the equilibrium solution
given by $\phi=0$, $\theta=0$ is given by

$\displaystyle J=\begin{bmatrix}\nabla_{\phi,\phi}E&\nabla_{\theta,\phi}E\\\
-\nabla_{\phi,\theta}E&-\nabla_{\theta,\theta}E\end{bmatrix}=\begin{bmatrix}0&l^{\prime}(0)\\\
-l^{\prime}(0)&0\end{bmatrix}$ (A.66)

We see that $\operatorname{Tr}(J)=0$ and the determinant
$|J|=l^{\prime}(0)^{2}$. Therefore, the eigenvalues of this Jacobian are
$\lambda_{\pm}=\operatorname{Tr}(J)/2\pm\sqrt{\operatorname{Tr}(J)^{2}-4|J|}/2=\pm
il^{\prime}(0)$ (Reproduced from Mescheder et al. (2018)). This is an example
of a stable _center equilibrium_.

Next we calculate the Jacobian of the modified ODEs given by DiracGAN,
evaluated at an equilibrium solution and find $\widetilde{J}=J-h\Delta/2$,
where

$\displaystyle\Delta=$
$\displaystyle\begin{bmatrix}\alpha(\nabla_{\phi,\phi}E)^{2}-\alpha\nabla_{\phi,\theta}E\nabla_{\theta,\phi}E&\alpha\nabla_{\theta,\phi}E\nabla_{\phi,\phi}E-\alpha\nabla_{\theta,\theta}E\nabla_{\theta,\phi}E\\\
\lambda\nabla_{\phi,\theta}E\nabla_{\theta,\theta}E-\lambda\nabla_{\phi,\phi}E\nabla_{\phi,\theta}E&\lambda(\nabla_{\theta,\theta}E)^{2}-\lambda\nabla_{\theta,\phi}E\nabla_{\phi,\theta}E\end{bmatrix}$
$\displaystyle=$
$\displaystyle\begin{bmatrix}-\alpha\nabla_{\phi,\theta}E\nabla_{\theta,\phi}E&0\\\
0&-\lambda\nabla_{\theta,\phi}E\nabla_{\phi,\theta}E\end{bmatrix}$
$\displaystyle=$ $\displaystyle\begin{bmatrix}-\alpha l^{\prime}(0)^{2}&0\\\
0&-\lambda l^{\prime}(0)^{2}\end{bmatrix}$

so

$\displaystyle\widetilde{J}=\begin{bmatrix}h\alpha
l^{\prime}(0)^{2}/2&l^{\prime}(0)\\\ -l^{\prime}(0)&h\lambda
l^{\prime}(0)^{2}/2\end{bmatrix}$ (A.67)

Now, we see that the trace of the modified Jacobian for the Dirac-GAN is
$\operatorname{Tr}(\widetilde{J})=(h/2)(\alpha+\lambda)l^{\prime}(0)^{2}>0$,
so the modified ODEs induced by gradient descent in DiracGAN are unstable.

###  G.3 Explicit regularization stabilizes Dirac-GAN

Here, we show that we can use our stability analysis to identify forms of
explicit regularization that can counteract the destabilizing impact of DD.
Consider the Dirac-GAN with explicit regularization of the following form:
$L_{1}=-E+u{\|\nabla_{\theta}E\|}^{2}$ and
$L_{2}=E+\nu{\|\nabla_{\phi}E\|}^{2}$ where $\phi_{t}=\phi_{t-1}-\alpha
h\nabla_{\phi}L_{1}$ and $\theta_{t}=\theta_{t-1}-\lambda
h\nabla_{\theta}L_{2}$ and with $u,\nu\sim\mathcal{O}(h)$. The modified
Jacobian for this system is given by

$\displaystyle\widetilde{J}=$
$\displaystyle\begin{bmatrix}h\alpha/2\nabla_{\phi,\theta}E\nabla_{\theta,\phi}E-u\nabla_{\phi,\phi}\left\lVert\nabla_{\theta}E\right\rVert^{2}&\nabla_{\theta,\phi}E\\\
-\nabla_{\phi,\theta}E&h\lambda/2\nabla_{\theta,\phi}E\nabla_{\phi,\theta}E-\nu\nabla_{\theta,\theta}\left\lVert\nabla_{\phi}E\right\rVert^{2}\end{bmatrix}$
$\displaystyle=$
$\displaystyle\begin{bmatrix}(h\alpha/2-2u)l^{\prime}(0)^{2}&l^{\prime}(0)\\\
-l^{\prime}(0)&(h\lambda/2-2\nu)l^{\prime}(0)^{2}\end{bmatrix}$

The determinant of the modified Jacobian is
$|\widetilde{J}|=(h\alpha/2-2u)(h\lambda/2-2\nu)l^{\prime}(0)^{4}+l^{\prime}(0)^{2}$
and the trace is
$\operatorname{Tr}(\widetilde{J})=(h\alpha/2-2u)l^{\prime}(0)^{2}+(h\lambda/2-2\nu)l^{\prime}(0)^{2}$.
A necessary and sufficient condition for asymptotic stability is
$|\widetilde{J}|>0$ and $\operatorname{Tr}(\widetilde{J})<0$ (since this
guarantees that the eigenvalues of the modified Jacobian have negative real
part). Therefore, if $u>h\alpha/4$ and $\nu>h\lambda/4$, the system is
asymptotically stable. We note however that in practice, when using discrete
updates, the exact threshold for stability will have a $\mathcal{O}(h^{3})$
correction, arising from the $\mathcal{O}(h^{3})$ error in our backward error
analysis. Also, we see that when $u=h\alpha/4$ and $\nu=h\lambda/4$, the
contribution of the cross-terms is cancelled out (up to an
$\mathcal{O}(h^{3})$ correction). In Figure A.2, we see an example where this
explicit regularization stabilizes the DiracGAN, so that it converges toward
the equilibrium solution.

##  Appendix H Additional experimental results

###  H.1 Additional results using zero-sum GANs

We show additional results showcasing the effect of DD on zero-sum games in
Figure A.3. We see that not only do simultaneous updates perform worse than
alternating updates when using the best hyperparameters, but that simultaneous
updating is much more sensitive to hyperparameter choice. We also see that
multiple updates can improve the stability of a GAN trained using zero-sum
losses, but this strongly depends on the choice of learning rate.

Figure A.3: The effect of discretization drift on zero-sum games.

####  H.1.1 Least squares GANs

In order to assess the robustness of our results independent of the GAN loss
used, we perform additional experimental results using GANs trained with a
least square loss (LS-GAN (Mao et al., 2017)). We show results in Figure A.4,
where we see that for the least square loss too, the learning rate ratios for
which the generator drift does not maximize the discriminator norm (learning
rate ratios above or equal to 0.5) perform best and exhibit less variance.

Figure A.4: Least squares GAN: The effect of discretization drift on zero-sum
games.

###  H.2 GANs using the non-saturating loss

Next, we explore how the strength of the DD depends on the game dynamics. We
do this by comparing the relative effect that numerical integration schemes
have across different games. To this end, we consider the non-saturating loss
introduced in the original GAN paper ($-\log
D_{\boldsymbol{\phi}}(G_{\boldsymbol{\theta}}(\mathbf{z}))$). This loss has
been extensively used since it helps to avoid problematic gradients early in
training. When using this loss, we see that there is little difference between
simultaneous and alternating updates (Figure A.5), unlike the saturating loss
case. These results demonstrate that since DD depends on the underlying
dynamics, it is difficult to make general game-independent predictions about
which numerical integrator will perform best.

Figure A.5: The effect of discretization drift depends on the game: with the
non saturating loss, the relative performance of different numerical
estimators is different compared to the saturating loss, and the effect of
explicit regularization is also vastly different.

###  H.3 Explicit regularization in zero-sum games trained using simultaneous
gradient descent

We now show additional experimental results and visualizations obtained using
explicit regularization obtained using the original zero sum GAN objective, as
presented in Goodfellow et al. (2014).

We show the improvement that can be obtained compared to gradient descent with
simultaneous updates by canceling the interaction terms in Figure A.6. We
additionally show results obtained from strengthening the self terms in Figure
A.8.

In Figure A.7, we show that by cancelling interaction terms, SGD becomes a
competitive optimization algorithm when training the original GAN. We also
note that in the case of Adam, while convergence is substantially faster than
with SGD, we notice a degrade in performance later in training. This is
something that has been observed in other works as well (e.g. (Qin et al.,
2020)).

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.6: Using explicit regularization to cancel the effect of the
interaction components of drift eads to a substantial improvement compared to
SGD without explicit regularization.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.7: Using explicit regularization to cancel the effect of the
interaction components of drift allows us to obtain the same peak performance
as Adam using SGD without momentum.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.8: Using explicit regularization to cancel the effect of the drift
eads to a substantial improvement compared to SGD without explicit
regularization. Strengthening the self terms – the terms which minimize the
player’s own norm – does not lead to a substantial improvement; this is
somewhat expected since while the modified ODEs give us the exact coefficients
required to cancel the drift, they do not tell us how to strengthen it, and
our choice of exact coefficients from the drift might not be optimal.

####  H.3.1 More percentiles

Throughout the main paper, we displayed the best $10\%$ performing models for
each optimization algorithm used. We now expand that to show performance
results across the best $20\%$ and $30\%$ of models in Figure A.9. We observe
a consistent increase in performance obtained by canceling the interaction
terms.

(a) Top 10% models. (b) Top 20% models. (c) Top 30% models.

Figure A.9: Performance across the top performing models for vanilla SGD and
with canceling the interaction terms. Across all percentages, canceling the
interaction terms improves performance.

####  H.3.2 Batch size comparison

We now show that the results which show the efficacy of canceling the
interaction terms are resiliant to changes in batch size in Figure A.10.

(a) Batch size 64. (b) Batch size 128. (c) Batch size 256.

Figure A.10: Performance when changing the batch size. We consistently see
that canceling te interaction terms improves performance.

####  H.3.3 Comparison with Symplectic Gradient Adjustment

We show results comparing with Symplectic Gradient Adjustment (SGA) (Balduzzi
et al., 2018) in Figure A.11 (best performing models) and Figure A.12
(quantiles showing performance across all hyperparameters and seeds). We
observe that despite having the same functional form as SGA, canceling the
interaction terms of DD performs better; this is due to the choice of
regularization coefficients, which in the case of canceling the interaction
terms is provided by Corollary 6.1.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.11: Comparison with Symplectic Gradient Adjustment (SGA). Canceling
the interaction terms results in similar performance, but with less variance.
The performance of SGA heavily depends on the strength of regularization,
adding another hyperparmeter to the hyperparameter sweep, while canceling the
interaction terms of the drift requires no other hyperparameters, since the
explicit regularization coefficients strictly depend on learning rates. Figure
A.12: Comparison with Symplectic Gradient Adjustment (SGA), obtained from all
models in the sweep. Without requiring an additional sweep over the
regularization coefficient, canceling the interaction terms results in better
performance across the learning rate sweep and less sensitivity to
hyperparameters.

####  H.3.4 Comparison with Consensus Optimization

We show results comparing with Consensus Optimization (CO) (Mescheder et al.,
2017) in Figure A.13 (best performing models) and Figure A.14 (quantiles
showing performance across all hyperparameters and seeds). We observe that
canceling the interaction terms performs best, and that additionally
strengthening the self terms does not provide a performance benefit.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.13: Comparison with Consensus Optimization (CO). Despite not
requiring additional hyperparameters compared to the standard SGD learning
rate sweep, canceling the interaction terms of the drift performs better than
consensus optimization. Using consensus optimization with a fixed coefficient
can perform better than using the drift coefficients when we use them to the
strengthen the norms – this is somewhat expected since while the modified ODEs
give us the exact coefficients required to cancel the drift, they do not tell
us how to strengthen it, and our choice of exact coefficients from the drift
might not be optimal. Figure A.14: Comparison with Consensus Optimization
(CO), obtained from all models in the sweep. Without requiring an additional
sweep over the regularization coefficient, canceling the interaction terms
results in better performance across the learning rate sweep and less
sensitivity to hyperparameters.

####  H.3.5 Variance across seeds

We have mentioned in the main text the challenge with variance across seeds
observed when training GANs with SGD, especially in the case of simultaneous
updates in zero sum games. We first notice that performance of simultaneous
updates depends strongly on learning rates, with most models not learning. We
also notice variance across seeds, both in vanilla SGD and when using explicit
regularization to cancel interaction terms. In order to investigate this
effect, we ran a sweep of 50 seeds for the best learning rates we obtain when
canceling interaction terms in simultaneous updates, namely a discriminator
learning rate of $0.01$ and a generator learning rate of $0.005$, we obtain
Inception Score results with mean 5.61, but a very large standard deviation of
2.34. Indeed, as shown in Figure A.15, more than 50% of the seeds converge to
an IS grater than 7. To investigate the reason for the variability, we repeat
the same experiment, but clip the gradient value for each parameter to be in
$[-0.1,0.1]$, and show results in Figure A.16. We notice that another $10\%$
of jobs converge to an IS score grater than 7, and $10\%$ drop in the number
of jobs that do not manage to learn. This makes us postulate that the reason
for the variability is due to large gradients, perhaps early in training. We
contrast this variability across seeds with the consistent performance we
obtain by looking at the best performing models across learning rates, where
as we have shown in the main paper and throughout the Supplementary Material,
we obtain consistent performance which consistently leads to a substantial
improvement compared to SGD without explicit regularization, and obtains
performance comparable with Adam.

(a) Learning curves. (b) Performance Histogram.

Figure A.15: Variability across seeds for the best performing hyperparameters,
when canceling interaction terms in simultaneous updates for the original GAN,
with a zero sum loss.

(a) Learning curves. (b) Performance Histogram.

Figure A.16: With gradient clipping. Gradient clipping can reduce variability.
This suggests that the instabilities observed in gradient descent are caused
by large gradient updates.

###  H.4 Explicit regularization in zero-sum games trained using alternating
gradient descent

We perform the same experiments as done for simultaneous updates also with
alternating updates. To do so, we cancel the effect of DD using the same
explicit regularization functional form, but updating the coefficients to be
those of alternating updates. We show results in Figure A.17, where we see
very little difference in the results compared to vanilla SGD, perhaps apart
from less instability early on in training. We postulate that this could be
due the effect of DD in alternating updates can can be beneficial, especially
for learning rate ratios for which the second player also minimizes the
gradient norm of the first player. We additionally show results obtained from
strengthening the self terms in Figure A.18.

####  H.4.1 More percentiles

Throughout the main paper, we displayed the best $10\%$ performing models for
each optimization algorithm used. We now expand that to show performance
results across the $20\%$ and $30\%$ jobs in Figure A.19. We observe a
consistent increase in performance obtained by canceling the interaction
terms.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.17: In alternating updates, using explicit regularization to cancel
the effect of the interaction components of drift does not substantially
improve performance compared to SGD, but can reduce variance. This is
expected, given that the interaction terms for the second player in the case
of alternating updates can have a beneficial regularization effect.

(a) Inception Score ($\uparrow$). (b) Frechet Inception Distance
($\downarrow$).

Figure A.18: In alternating updates, using explicit regularization to cancel
the effect of the interaction components of drift does not substantially
improve performance compared to SGD, but can reduce variance. This is
expected, given that the interaction terms for the second player in the case
of alternating updates can have a beneficial regularization effect.
Strengthening the self terms – the terms which minimize the player’s own norm
– does not lead to a substantial improvement; this is somewhat expected since
while the modified ODEs give us the exact coefficients required to cancel the
drift, they do not tell us how to strengthen it, and our choice of exact
coefficients from the drift might not be optimal.

(a) Top 10% models. (b) Top 20% models. (c) Top 30% models.

Figure A.19: Performance across the top performing models for vanilla SGD and
with canceling the interaction terms and only cancelling the discriminator
interaction terms. We notice that canceling only the discriminator interaction
terms can result in higher performance across more models, and that the
interaction term of the generator can play a positive role, likely due to the
smaller strength of the generator interaction term compared to simultaneous
updates.

##  Appendix I Experimental details

###  I.1 Classification experiments

For the MNIST classification results with alternating updates, we use an MLP
with layers of size $[100,100,100,10]$ and a learning rate of $0.08$. The
batch size used is $50$ and models are trained for 1000 iterations. Error bars
are obtained from 5 different seeds.

###  I.2 GAN experiments

SGD results: All results are obtained using sweep over
$\\{0.01,0.005,0.001,0.0005\\}$ for the discriminator and for the generator
learning rates. We restrict the ratio between the two learning rates to be in
the interval $[0.1,10]$ to ensure the validity of our approximations. All
experiments use a batch size of 128.

Learning rate ratios: In the learning rate ratios experiments, we control for
the number of experiments which have the same learning rate ratio. To do so,
we obtain 5 learning rates uniformly sampled from the interval $[0.001,0.01]$
which we use for the discriminator, we fix the learning rate ratios to be in
$\\{0.1,0.2,0.5,1.,2.,5.\\}$ and we obtain the generator learning rate from
the discriminator learning rate and the learning rate ratio.

Adam results: For Adam, we use the learning rate sweep $\\{10^{-4},2\times
10^{-4},3\times 10^{-4},4\times 10^{-4}\\}$ for the discriminator and the same
for the generator, with $\beta_{1}=0.5$ and $\beta_{2}=0.99$.

Explicit regularization coefficients: For all the experiments where we cancel
the interaction terms of the drift, we use the coefficients given by DD. For
SGA and consensus optimization, we do a sweep over coefficients in
$\\{0.01,0.001,0.0001\\}$.

Model Architectures: All GAN experiments use the CIFAR-10 convolutional SN-GAN
architectures – see Table 3 in Section B.4 in Miyato et al. (2018).

Libraries: We use JAX (Bradbury et al., 2018) to implement our models, with
Haiku (Hennigan et al., 2020) as the neural network library, and Optax (Hessel
et al., 2020) for optimization.

Computer Architectures: All models are trained on NVIDIA V100 GPUs. Each model
is trained on 4 devices.

###  I.3 Implementing explicit regularization

The loss functions we are interested are of the form

$\displaystyle
L=\mathbb{E}_{p(\mathbf{x})}f_{\boldsymbol{\theta}}(\mathbf{x})$

We then have

$\displaystyle\nabla_{\boldsymbol{\theta}_{i}}L$
$\displaystyle=\mathbb{E}_{p(\mathbf{x})}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\mathbf{x})$
$\displaystyle\left\lVert\nabla_{\boldsymbol{\theta}}L\right\rVert^{2}$
$\displaystyle=\sum_{i=1}^{i=|\boldsymbol{\theta}|}\left(\nabla_{\boldsymbol{\theta}_{i}}L\right)^{2}=\sum_{i=1}^{i=|\boldsymbol{\theta}|}\left(\mathbb{E}_{p(\mathbf{x})}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\mathbf{x})\right)^{2}$

to obtain an unbiased estimate of the above using samples, we have that:

$\displaystyle\left\lVert\nabla_{\boldsymbol{\theta}}L\right\rVert^{2}$
$\displaystyle=\sum_{i=1}^{i=|\boldsymbol{\theta}|}\left(\mathbb{E}_{p(\mathbf{x})}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\mathbf{x})\right)^{2}$
$\displaystyle=\sum_{i=1}^{i=|\boldsymbol{\theta}|}(\mathbb{E}_{p(\mathbf{x})}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\mathbf{x}))(\mathbb{E}_{p(\mathbf{x})}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\mathbf{x}))$
$\displaystyle\approx\sum_{i=1}^{i=|\boldsymbol{\theta}|}\left(\frac{1}{N}\sum_{k=1}^{N}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\widehat{\mathbf{x}_{1,k}})\right)\left(\frac{1}{N}\sum_{j=1}^{N}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\widehat{\mathbf{x}_{2,j}})\right)$

so we have to use two sets of samples $\mathbf{x}_{1,k}\sim p(\mathbf{x})$ and
$\mathbf{x}_{2,j}\sim p(\mathbf{x})$ from the true distribution (by splitting
the batch into two or using a separate batch) to obtain the correct norm. To
compute an estimator for
$\nabla_{\boldsymbol{\phi}}\left\lVert\nabla_{\boldsymbol{\theta}}L\right\rVert^{2}$,
we can compute the gradient of the above unbiased estimator of
$\left\lVert\nabla_{\boldsymbol{\theta}}L\right\rVert^{2}$. However, to avoid
computing gradients for two sets of samples, we derive another unbiased
gradient estimator, which we use in all our experiments:

$\displaystyle\frac{2}{N}\sum_{i=1}^{i=|\boldsymbol{\theta}|}\sum_{k=1}^{N}\nabla_{\boldsymbol{\phi}}\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\widehat{\mathbf{x}_{1,j}})\nabla_{\boldsymbol{\theta}_{i}}f_{\boldsymbol{\theta}}(\widehat{\mathbf{x}_{2,j}})$
(A.68)