# E-LPIPS: Robust Perceptual Image Similarity via Random Transformation
Ensembles

Markus Kettunen
Aalto University &Erik Härkönen
Aalto University &Jaakko Lehtinen
Aalto University
NVIDIA

###### Abstract

It has been recently shown that the hidden variables of convolutional neural
networks make for an efficient perceptual similarity metric that accurately
predicts human judgment on relative image similarity assessment. First, we
show that such learned perceptual similarity metrics (LPIPS) are susceptible
to adversarial attacks that dramatically contradict human visual similarity
judgment. While this is not surprising in light of neural networks’ well-known
weakness to adversarial perturbations, we proceed to show that self-ensembling
with an infinite family of random transformations of the input — a technique
known _not_ to render classification networks robust — is enough to turn the
metric robust against attack, while retaining predictive power on human
judgments. Finally, we study the geometry imposed by our our novel self-
ensembled metric (E-LPIPS) on the space of natural images. We find evidence of
“perceptual convexity” by showing that convex combinations of similar-looking
images retain appearance, and that discrete geodesics yield meaningful frame
interpolation and texture morphing, all without explicit correspondences.

##  1 Introduction

Computational assessment of perceptual image similarity — how close is an
image to another image — is a fundamental question in vision, graphics, and
imaging. For an image similarity metric $d(a,b)$ to be perceptually equivalent
to human observations, it clearly suffices that

  1. (C1)

$d(a,b)$ is large when human observers perceive a large dissimilarity between
$a$ and $b$, and

  2. (C2)

$d(a,b)$ is small when human observers consider the images $a$ and $b$
similar.

Perceptually motivated image similarity metrics have a long history, with
well-known challenges due to dependence on context and high-level image
structure [27, 28, 20]. Standard vector norms applied to images pixelwise,
such as the $L_{2}$ or $L_{1}$ distance, are brittle in the sense that many
transformations that leave images visually indistinguishable — shifting by one
pixel is a classic example — can yield arbitrary jumps in distance, indicating
a violation of Condition (C2). Other transformations exhibiting similar issues
include intensity variations, small rotations, and slight blur.

In recent years, several authors have made the observation that the hidden
layer features in a convolutional neural network (CNN) trained for image
classification [24, 16, 12] yields a perceptually meaningful space in which to
measure image distance [3], with applications in, e.g., image generation and
restoration [13, 6, 14] and performance metrics for generative models [11].
Inspired by this, Zhang et al. [30] recently showed that _hidden CNN
activations are indeed a space where distance can strongly correlate with
human judgment_ , much more so than per-pixel metrics or other prior
similarity scores. Their Learned Perceptual Image Path Similarity (lpips)
metric is calibrated to human judgment using data from two-alternative forced
choice (2AFC) tests.

Our message is threefold. First, we will show that lpips-style image
similarity metrics computed as Euclidean distances between the hidden
variables of a classifier CNN are brittle: they are easy to break by standard
adversarial attack techniques. This is not at all surprising given neural
networks’ well-known weakness to adversarial perturbations. More precisely, we
show that these metrics fulfill neither (C1) nor (C2) by showing how to easily
craft image pairs that have a small lpips distance albeit human observers see
the images as completely different, violating (C1), or that have a large lpips
distance, although a human observer is hard pressed to see the difference,
violating (C2).

Second, we show that computing the CNN feature distance as an average under an
effectively infinite ensemble of random image transformations makes the metric
significantly more robust against adversarial attacks, even with the attack
crafted with full knowledge of the ensemble defense.We call the novel metric
the Ensembled Learned Perceptual Image Patch Similarity metric, e-lpips. That
self-ensembling through random transformations yields a robust similarity
metric might seem surprising at first: similar random transformation ensembles
have been previously shown to _not_ be sufficient to make classifier networks
robust [1, 2]. We attribute the qualitative difference to the fact that
attacking lpips-style metrics requires modification of _all_ hidden variables
of a CNN, while attacking a classifier network only requires the attacker to
change the output layer.

Finally, we observe that the geometry on the space of natural images induced
by the e-lpips metric has several intriguing properties we call “perceptual
convexity”. In particular, we show by several examples that when computed
under our e-lpips metric, averages (barycenters) of similar-looking images
retain a similar appearance even when pixel-wise averages appear very
different, and that discrete geodesics between two images yields reasonable
frame interpolation and texture morphing — all without explicit correspondence
or optical flow. Furthermore, as a practical consequence of a more
perceptually convex metric, we show that e-lpips yields consistently better
results than non-ensembled feature losses when used as a loss function in
training image restoration models.

##  2 Ensembled Perceptual Similarity

We build on lpips [30] that computes image differences in the space of hidden
unit activations resulting from the input images when run through the
convolutional part of an image classification network such as VGG [24],
SqueezeNet [12] or AlexNet [16]. Other, prior uses of “VGG feature distances”
[13, 6, 14] differ from lpips in minor details only.

lpips first normalizes the feature dimension in all pixels and layers to unit
length, scales each feature by a feature-specific weight, and evaluates the
square of the $L_{2}$ distance between these weighted activations. These
squared distances are then averaged over the image dimensions and summed over
the layers, and this provides the final image distance metric:

$d_{\textsc{lpips}}(x,y)=\sum_{l}{\frac{1}{H_{l}W_{l}}}\sum_{i,j}\|w_{l}\odot(\hat{x}^{l}_{ij}-\hat{y}^{l}_{ij})\|^{2}_{2}.$
(1)

Here, $\hat{x}^{l}_{ij}$ and $\hat{y}^{l}_{ij}$ denote the normalized feature
vectors at layer $l$ and pixel $(i,j)$, $w_{l}$ contains weights for each of
the features in layer $l$, and $\odot$ multiplies the feature vectors at each
pixel by the feature weights. The weights $w_{l}$ are optimized such that the
metric best agrees with human judgment derived from two-alternative forced
choice (2AFC) test results.

###  2.1 E-LPIPS: Ensembled LPIPS

Below (Section 3.1), we prove that lpips fulfills neither (C1) nor (C2) by
describing simple adversarial attacks that violate both conditions. The reader
may consult Figures 1 and 2 before proceeding. With hopes of increasing its
robustness, we alter lpips in four ways:

  1. 1.

We apply a randomized geometric and color transformation to the input images
before feeding them to the network, and take the expected lpips distance over
realizations. As the model is not limited to a single glimpse of the input,
but has access to an essentially infinite number of different versions, it is
harder to fool.

  2. 2.

While lpips only considers the features in the last layer of each resolution,
we compute distances over _all_ convolutional layers. In particular, while
lpips is blind to the input layer, considering all layers means Equation (1)
includes the $L_{2}$ distance of the input color tones, with a learned weight
$w_{0}$.

  3. 3.

We make the network stochastic by applying dropout with keep probability
$0.99$ on all layers. Matching intuition, this makes the metric consistently
more robust.

  4. 4.

We replace the image classification networks’ max pooling layers with average
pooling. As observed by several prior authors [8, 10], max pooling distracts
gradient flow, making optimization harder, and leaves more blind spots to be
exploited since many of the network activations can be freely altered without
the change propagating forward.

For a single evaluation of our method, we first transform both input images
with the same random combination of simple transformations. Our transform set
includes translations, flipping, mirroring, permutation of color channels,
scalar multiplication (brightness change), and downscaling, all with randomly
chosen parameters. Furthermore, the simple random transformations are combined
randomly, making the effective size of the ensemble very large. This is in
stark contrast to an ensemble over a fixed, small number of separate
transformations. Algorithms 1 and 2 detail the construction and application of
the transformations.

We now define e-lpips as the expectation

$d_{\textsc{e-lpips}}(x,y)=\mathbb{E}[d_{\textsc{lpips}}(T(x),T(y))]$ (2)

where the expectation is taken over the stochastic combinations of
transformations $T$ described above, and the classical $d_{\textsc{lpips}}$
distance evaluated with our network with dropout, average pooling, and weights
applied to all layers, including the input layer.

For numerical optimization using the metric, e.g., training neural networks,
we find that a single random sample per iteration is generally sufficient, but
multiple samples may sometimes result in faster convergence. For more precise
distance evaluations we recommend averaging $N\geq 300$ evaluations, depending
on the required precision. For comparing distances of two images from a third
image, we recommend using the same transformations and dropout variables for
all images. A single evaluation of e-lpips without gradients takes on average
about 10% longer than lpips-vgg, and 20% longer with gradients.

1

Output : Random transformation parameters

2 dx, dy = random.int$(0\ldots 7)$;

3 flip_x, flip_y, swap_xy = random.int() % 2;

4 col_perm = random.permute$(0,1,2)$;

5 sr, sg, sb = random.uniform$(0.2,1.0)$;

6 scale = random.int$(1\ldots 8)$ with $p(i)\propto\frac{1}{i^{2}}$;

scale_dx, scale_dy = random.int$(0\ \ldots\ \text{scale - 1})$

Algorithm 1 Sampling a transformation

1

Input : Image $X$, transformation parameters

Output : Transformed image

2 X = mirrorPad(X, scale_dx, scale_dy, 0, 0);

3 X = downscaleBox(X, 1/scale);

4 X = mirrorPad(X, dx, dy, $7-\text{dx}$, $7-\text{dy}$);

5 if  _flip_x_ then X = flipX(X);

6 if  _flip_y_ then X = flipY(X);

7 if  _swap_xy_ then X = swapXY(X);

8 X = permuteColorChannels(X, col_perm);

9 X = X * (sr, sg, sb);

10

Algorithm 2 Applying a transformation

##  3 Experiments

Source $a$ Anchor Target $b$ Attack on lpips-sqz Attack on lpips-vgg Attack on
e-lpips (our)

Rel. $L_{2}$ dist.: 1 16.0 14.0 14.5 3.7

Rel. $L_{2}$ dist.: 1 13.9 9.2 10.2 1.5

Figure 1: Attack (A1). Both lpips metrics allow to pull far away from the
source towards the target while remaining at the same lpips distance from the
source as the anchor. With the same constraint, attacks on e-lpips lie much
closer to the source image both visually and by relative $L_{2}$ distance.

This section studies the properties of e-lpips in comparison to lpips and
simpler metrics in various scenarios. We use the VGG-16 version as basis for
e-lpips in all results, but compare to both VGG and Squeezenet -based lpips,
the best models reported by Zhang et al. [30]. All models are implemented in
TensorFlow, and all training and other optimization is performed on NVIDIA
V100 GPUs. We perform all optimizations with Adam [15], propagating gradients
the random input transformations. When attacking the metrics, this directly
implements the _Expectation over Transformations_ (EoT) attack previously used
to break self-ensembling in image classifiers [5, 1].

###  3.1 Adversarial Examples

#### Attack (A1), violates (C1).

We construct image pairs that humans perceive dissimilar but lpips considers
close. To attack metric $d(\cdot,\cdot)$, we select a source image $a$,
constrain $d(a,x)$ to be small, and optimize $x$ towards a distant target
image $b$ in the $L_{2}$ sense by solving

$\operatorname*{\text{arg min}}_{x}\|x-b\|^{2}_{2}\quad\text{subject to}\quad
d(a,x)\leq\varepsilon.$ (3)

with the anchor distance $\varepsilon$ chosen as the distance between $a$ and
a slightly noisy version of $a$. We perform the attack on 30 images from the
OpenImages [17] dataset. Figure 1 shows representative results. Both lpips
metrics succumb badly: the result shows that the $\varepsilon$-lpips-ball
around the source image contains images that are wildly different from it. The
e-lpips images consistently stay much closer to the input image, as desirable.
On average, lpips lets the attack pull the images five times farther from the
input image in $L_{2}$ distance than e-lpips: $16.2\pm 0.8$ times
$\varepsilon$ for lpips-vgg, $15.2\pm 0.7$ for lpips-sqz, and $3.0\pm 0.2$ for
e-lpips. See the supplemental material for more examples.

#### Attack (A2), violates (C2).

In the opposite direction, we construct pairs of images that are similar to
the human eye (proxied by a small $L_{2}$ distance), but far apart in the
lpips sense by solving

$\operatorname*{\text{arg max}}_{x}d(a,x)\quad\text{subject
to}\quad\|a-x\|^{2}_{2}\leq\varepsilon.$ (4)

The other details of the attack are the same as in (A1). Figure 2 shows
representative results. Both lpips metrics allow a very large increase
distance while staying within a small $L_{2}$-ball around the input: $1.74\pm
0.07$ times the mean distance between the images in the dataset for lpips-sqz,
on average, and $3.12\pm 0.08$ for lpips-vgg. In contrast, e-lpips gives in
much less: $0.63\pm 0.06$.

For lpips, the attack images are visually highly similar to the source,
despite the large increase in the distance, indicating violation of (C2); in
contrast, the image of maximal e-lpips distance within the
$\varepsilon$-$L_{2}$-ball is consistently visibly more different from the
source image. This is desirable. The supplemental material contains more
examples.

Source $a$ Attack on lpips-sqz Attack on lpips-vgg Attack on e-lpips (ours)
Source $a$ Attack on lpips-sqz Attack on lpips-vgg Attack on e-lpips (ours)

(e-)lpips: 1.36 2.83 0.35 (e-)lpips: 1.59 2.95 0.58

Figure 2: Attack (A2). Both lpips metrics allow the image to be pushed far
away in distance by modifications that are small both visually and in $L_{2}$
sense. In contrast, the attack is unable to increase the e-lpips distance
nearly as much; furthermore the visual change is much more clearly visible at
the same $L_{2}$ distance, which is desirable. The relative distance reported
below the images is normalized such that 1 is the mean distance between the
different images in the dataset.

###  3.2 Analysis

Together, the attack results suggest that e-lpips fulfills conditions (C1) and
(C2) better than the non-ensembled feature metrics. We now proceed to study
the reasons for this result.

#### Are all the transformations necessary?

We study the necessity of the various random transformations in our ensemble
by an ablation. We repeat Attack (A1) on an image pair, adding transformations
to the ensemble one-by-one, with dropout being added last. Figure 3 shows
representative results: clearly, each individual transformation “plugs more
holes”, making the attack less and less successful. On a test set of 30
images, the mean distance of lpips-vgg from the clean image is $16.3\pm 0.8$
times the anchor distance. The effectiveness of the adversarial attack drops
as we add all layers and average pooling ($14.5\pm 0.7$), translations,
rotations and swaps ($10.8\pm 0.7$), color transformations ($6.3\pm 0.5$),
multiple scale levels ($4.6\pm 0.4$), and finally dropout ($3.0\pm 0.2$),
reaching a five-fold robustness improvement in terms of the $L_{2}$ distance.

The supplemental material contains another ablation study that highlights the
necessity of using an effectively infinite ensemble of random transformations,
as opposed to a fixed ensemble.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Source Target lpips-vgg +all layers +geom. +color +scaling +dropout

Figure 3: Success of Attack (A1) against increasingly powerful variants of
e-lpips. The increasing robustness resulting from a richer transformation
ensemble is visible as the increasing visual similarity between (a) and attack
results (c)-(h). Image (h) corresponds to the full e-lpips metric.

#### Sensitivity analysis.

Above, we found large $L_{2}$ perturbations that increased the lpips distance
only little, and small $L_{2}$ perturbations that greatly increased the lpips
distance. The results are systematic: attacks succeed easily against lpips,
and much less so against e-lpips. This suggests severe ill-conditioning of the
lpips metric. We now probe the metrics’ local behavior in order to see if it
is consistent with this hypothesis. Specifically, we study the behavior of
$f_{x_{0}}(v)=d(x_{0},x_{0}+v)$ around a fixed image $x_{0}$ for small $v$
through its 2nd-order Taylor expansion. As $v=0$ is the minimum of $f(v)$, the
first-order term vanishes, and the expansion equals $v^{T}Hv/2$, where
$H=\partial^{2}f/\partial v^{2}$ is the Hessian. Note that $\dim v=W\times
H\times 3$ for an RGB image of size $W\times H$.

| e-lpips | lpips |  ratio ($2\sigma$ bounds)
---|---|---|---
$\lambda_{\text{max}}$ | $375$ | $4540$ | $[0.058,0.146]$
$\lambda_{\text{min}}{}^{\dagger}$ | $0.008\,55$ | $0.000\,005\,52$ | $[1090,2480]$
$\lambda_{\text{max}}/\lambda_{\text{min}}{}^{\dagger}$ | $25\,600$ | $112\,000\,000$ | $[0.0002,0.0005]$
Variance | $11.6$ | $422$ | $[0.031,0.050]$
Skewness | $24.1$ | $87.5$ | $[0.259,0.372]$
Kurtosis | $1\,350$ | $13\,600$ | $[0.105,0.209]$

As the Hessian is symmetric and positive semidefinite, its distance-scaling
properties are characterized by its eigenvalue spectrum. While the full
spectrum is intractable, we compute the extremal eigenvalues by power
iteration, as well as approximate their mean, variance, skewness and kurtosis
by sampling. Details can be found in the supplemental material. The adjacent
table shows means of these descriptors computed over 10 different anchor
images $x_{0}$ of size $400\times 400$. For scale-invariance between the
metrics, we normalize all values with the corresponding mean eigenvalues. The
smallest eigenvalues of e-lpips are challenging to compute due to the metric’s
stochastic nature. We thus evaluate the results marked with $\dagger$ in lower
resolution ($64\times 64$), and use a weakened version of e-lpips with no
dropout and with a fixed ensemble of 256 input transformations. We notice a
clear trend of improved results when the ensemble size is increased,
suggesting still better conditioning for full e-lpips.

The maximal eigenvalues of lpips are consistently an order of magnitude larger
than e-lpips’, the minimal eigenvalues are consistently three orders of
magnitude smaller, and lpips’ condition number is three orders of magnitude
larger on average. These numbers indicate that lpips is much more ill-
conditioned. Moreover, the distribution of lpips’ eigenvalues is more
positively skew — indicating that more eigenvalues are clustered towards zero
from the mean and consequently more directions where the metric is blind to
changes — as well as more kurtotic, implying more concentration at the
extremal ends. These findings are consistent with the large-scale behavior
observed earlier.

The supplemental material contains videos demonstrating random perturbations
in the linear space spanned by the 16 largest eigenvectors of both lpips and
e-lpips around the central image $x_{0}$. Though informal, the perturbations
sampled from e-lpips eigenvectors appear more natural and consistent with the
image structure.

#### Correlation with Human Judgment.

We now study the power of e-lpips in predicting human assessment of image
similarity, the original goal of the lpips metric that ours is based on. We
use the data of, and directly follow the evaluation process presented by Zhang
et al. [30]. We train the feature weights to produce distances $d(A,C)$ and
$d(B,C)$ which are fed to a small two-hidden-layer MLP that predicts the ratio
of human choices of A and B. We enable input transformations only for testing
after the weights have been trained, and keep the weights of the underlying
VGG network fixed.

The resulting accuracy in predicting human answers in the two-alternative
forced choice test (2AFC) is 69.16% for e-lpips, falling directly between the
VGG (68.65%) and SqueezeNet (70.07%) versions of the non-ensembled lpips. This
indicates that predictive power does not suffer as a result of the ensembling,
i.e., additional robustness and more intuitive geometry over images come at no
cost. To put the numbers in perspective, Zhang et al. report a mean human
score of 73.9%111The score is not 100%, as humans have difficulty predicting
how other humans perceive image similarity., while simple metrics such as
pixel-wise $L_{2}$ distance and SSIM achieve the score of 63%. The
supplemental material contains a table with an accuracy breakdown over
different classes of image corruptions.

#### Performance as a Loss Function.

Using an image similarity metric as a loss function in training an image
generation or restoration model is a potential adversarial situation: if the
metric cannot reliably distinguish between perceptual similarity and
dissimilarity, the optimizer may drive the model to produce nonsensical
results. As a case study, we train a 10M parameter standard convolutional
U-net with skip connections [23] for removing additive Gaussian noise from
photographs using different loss functions, as well as for 4-fold single-image
super-resolution. (We train super-resolution in a supervised fashion [13] as a
case study, aware that state-of-the-art techniques utilize more sophisticated
models [7, 18].)

Although the differences are not large on average, we find that e-lpips
consistently results in somewhat better results than plain lpips, both
numerically and by visual inspection. In particular, the non-ensembled lpips
metric sometimes falls in strange local minima on some network architectures
(e.g. transposed convolutions, see supplemental). This is never observed with
e-lpips. Despite the increased robustness and somewhat better results, there
is no difference in wall-clock training speed between lpips and e-lpips.
Visual and numerical results can be found in the supplemental material.

###  3.3 Geometry of Natural Images

$x_{1}$

$x_{2}$

$\cdots$ $x_{10}$ e-lpips (ours) $L_{2}$ lpips

$\cdots$$\Longrightarrow$

$\cdots$$\Longrightarrow$

Input images Barycenters

Figure 4: Barycenters of similar-looking images $x_{1},x_{2},\cdots,x_{10}$
under various distance metrics. Top row: 10 noise realizations of the same
image. Bottom row: small translations of the same image. The $L_{2}$
barycenter simply averages the inputs. Unlike $L_{2}$ and lpips, the e-lpips
barycenter retains much of the appearance of the input images in both cases.

#### Barycenters of similar-looking images.

From a perceptual point of view, it would seem reasonable that an average of
two or more images that look the same should still look the same. To study the
properties of averages of multiple images taken under e-lpips, we first
compute barycenters, $\operatorname*{\text{arg
min}}_{x}\sum_{i}d(x,n_{i})^{2}$, over several independent realizations
$\\{n_{i}\\}$ of images transformed by the same random process, by direct
numerical optimization over pixels.

As examples, we study i.i.d. additive Gaussian noise and small random offsets
in Figure 4. The $L_{2}$ barycenter is simply the pixelwise average of the
inputs, and appears quite different from the individual input images:
averaging zero-mean noise cancels it out, and averaging shifted images results
in a blur. Despite having converged into a local optimum, the non-ensembled
lpips barycenters deviate visually even further, providing another view into
the metric’s large null space. In contrast, the appearance of the e-lpips
barycenters quite closely matches the individual realizations. This suggests a
property we call _perceptual convexity_ : the barycenter — which can be seen
as a convex combination — of a set of visually similar images is itself
visually similar to the others. The supplemental material contains more
examples, and a study of pairwise averages.

#### Discrete geodesics.

Input frames $L_{2}$ lpips e-lpips (ours)

Time Evol. $\rightarrow$

Time Evol. $\rightarrow$

Figure 5: Discrete geodesics between Images A and B, computed in three
metrics. Each column shows a single frame from the discrete geodesic, as well
as the time evolution of the scanline indicated in red. The reader is
encouraged to view the supplemental animations.

A discrete geodesic between images $a$ and $b$ is a sequence of images that
minimizes the total pairwise squared distance between adjacent frames [10]:

$\operatorname*{\text{arg
min}}_{x_{1},\ldots,x_{N-1}}\sum_{i=0}^{N-1}d(x_{i},x_{i+1})^{2},\quad\text{with
}x_{0}=a,\;x_{N}=b.$ (5)

The metric $d(\cdot,\cdot)$ directly determines the visual properties of the
solution, as the free images $x_{1},\ldots,x_{N-1}$ affect the optimization
objective only through it. The $L_{2}$ geodesic is a pixelwise linear cross-
fade. We numerically compute discrete geodesics of 8 in-between frames for the
e-lpips, lpips, and $L_{2}$ metrics by initializing the free frames to random
noise, and simultaneously optimize all of them using Adam. See the
supplemental material for details on the optimization procedure.

As shown in Figure 5, geodesic interpolation between adjacent video frames
appears to create meaningful frame interpolations in the e-lpips metric:
instead of fading in and out like the $L_{2}$ crossfade, many image features
translate and rotate as if reprojected along optical flow. Note that no
correspondence or explicit reprojection is performed; all effects emerge from
the metric. In addition to frame interpolation, the geodesics result in
interesting fusions between different textures. Some evidence of similar
behavior is seen in lpips geodesics, but the image fusion is generally of
poorer quality, and translation and rotation are not modeled as naturally. The
reader is encouraged to view the videos in the supplemental material. Due to
the difficulty of the optimization problem, we consider these results an
existence proof that warrants detailed further study.

##  4 Related Work

#### Adversarial attacks.

The output of deep neural networks can often be altered drastically with small
adversarial perturbations of the input found by gradient-based optimization
[25, 5, 9, 21, 4, 22]. Random self-ensembling has been suggested before as a
defense against attacks on classifiers. Li et al. [19] inject noise layers
into an image classifier. Xie et al. [29] and Athalye et al. [2] apply random
transformations to the input. These defenses succumb to the Expectation over
Transformations (EoT) attack [1] — random transformations are not strong
enough to robustify a classifier.

#### Properties of image metrics.

Weaknesses in classical image similarity metrics have been previously
demonstrated in the same fashion as our attacks [26, 27]. We are, however, not
aware of prior successful attacks against state-of-the-art neural perceptual
similarity metrics. Hessian eigenanalysis of deep convolutional features has
been previously employed for studying perturbations of minimal and maximal
discriminability on different levels of VGG [3].

Prior study has hinted toward similar interesting properties of the discrete
geodesics of the “VGG feature” image metric: Hénaff and Simoncelli [10] find
evidence of linearization of effects such as small translation and rotation,
but only when aided with additional projections into pixel-space geodesics. In
contrast, our geodesics are computed by optimizing only the e-lpips and lpips
metrics. This, together with the large lpips null space evident in the
barycenter results (Figure 4), suggests that e-lpips imposes a more robust
geometric structure on the space of images.

##  5 Conclusion

By constructing adversarial examples that demonstrate high perceptual non-
uniformity in lpips [30], we showed that image similarity metrics based on
convolutional classifier feature distances exhibit susceptibility to white-box
gradient-based perturbation attacks, much like image classifiers. While this
is not surprising, the observation explains minor practical robustness issues
faced when using such metrics as optimization targets without extra
regularization.

We further extended the fragile lpips metric into an effectively infinite
self-ensemble through applying random, simple image transformations and
dropout. The resulting e-lpips image similarity metric (’E’ for ensemble) is
much more resilient to various direct attacks mounted with full knowledge of
the ensemble. This presents an interesting contrast between perceptual feature
differences and image classifiers: it is known that similar defenses do not
suffice against similar attacks [1]. Our preliminary studies indicate the
difference can be explained by the much larger dimensionality of the hidden
convolutional features — attacking the similarity metric requires modification
of _all_ convolutional features, as opposed to just the output layer. While
supporting evidence is found from local eigenvalue analysis, we stress that we
do not assert theoretical robustness guarantees without further study. Still,
the observed higher practical resilience to attack is visible as increased
robustness when the similarity metric is used as a loss function in
optimization. Moreover, the new metric remains a good predictor of human
judgment of image similarity, like lpips.

By computing barycenters of image sets and discrete geodesics between image
pairs, we found evidence of “perceptual convexity” in the e-lpips metric:
averages taken under it retain a much closer appearance to the images being
averaged than under lpips or $L_{2}$, and geodesic sequences interestingly
linearize effects such as small translation and rotation, without explicit
correspondence or optical flow computation. Both findings require systematic
further study.

The supplemental material and source code are available at
github.com/mkettune/elpips.

## Acknowledgments

We thank Pauli Kemppinen, Frédo Durand, Miika Aittala, Sylvain Paris, Alexei
Efros, Richard Zhang, Taesung Park, Tero Karras, Samuli Laine, Timo Aila, and
Antti Tarvainen for in-depth discussions; and Seyoung Park for helping with
the TensorFlow port of LPIPS. We acknowledge the computational resources
provided by the Aalto Science-IT project.

## References

  * [1] A. Athalye, N. Carlini, and D. A. Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In Proc. 35th International Conference on Machine Learning (ICML), volume 80, pages 274--283. PMLR, 2018.
  * [2] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok.  Synthesizing robust adversarial examples.  In Proc. 35th International Conference on Machine Learning (ICML), volume 80, pages 284--293. PMLR, 2018.
  * [3] A. Berardino, J. Ballé, V. Laparra, and E. P. Simoncelli.  Eigen-distortions of hierarchical representations.  In Proc. Neural Information Processing Systems (NeurIPS), 2017.
  * [4] A. N. Bhagoji, D. Cullina, and P. Mittal.  Dimensionality reduction as a defense against evasion attacks on machine learning classifiers.  CoRR, abs/1704.02654, 2016.
  * [5] N. Carlini and D. A. Wagner.  Towards evaluating the robustness of neural networks.  2017 IEEE Symposium on Security and Privacy (SP), pages 39--57, 2017\.
  * [6] Q. Chen and V. Koltun.  Photographic image synthesis with cascaded refinement networks.  2017 IEEE International Conference on Computer Vision (ICCV), pages 1520--1529, 2017.
  * [7] R. Dahl, M. Norouzi, and J. Shlens.  Pixel recursive super resolution.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5449--5458, Oct 2017.
  * [8] L. Gatys, A. S. Ecker, and M. Bethge.  Texture synthesis using convolutional neural networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Proc. Neural Information Processing Systems (NeurIPS), pages 262--270. Curran Associates, Inc., 2015.
  * [9] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  CoRR, abs/1412.6572, 2014.
  * [10] O. J. Henaff and E. P. Simoncelli.  Geodesics of learned representations.  In Proc. International Conference on Learning Representations (ICLR), 2016.
  * [11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proc. Neural Information Processing Systems (NeurIPS), pages 6626--6637. Curran Associates, Inc., 2017\.
  * [12] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and $<$0.5mb model size.  arXiv:1602.07360, 2016.
  * [13] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [14] J. Kim, J. K. Lee, and K. M. Lee.  Accurate image super-resolution using very deep convolutional networks.  CoRR, abs/1511.04587, 2015.
  * [15] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization, 2014.
  * [16] A. Krizhevsky.  One weird trick for parallelizing convolutional neural networks.  CoRR, abs/1404.5997, 2014.
  * [17] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  arXiv:1811.00982, 2018.
  * [18] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. P. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105--114, 2017.
  * [19] X. Liu, M. Cheng, H. Zhang, and C.-J. Hsieh.  Towards robust neural networks via random self-ensemble.  In Proc. European Conference on Computer Vision (ECCV), 2018.
  * [20] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich.  Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.  ACM Trans. Graph., 30(4):40:1--40:14, July 2011.
  * [21] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard.  Deepfool: A simple and accurate method to fool deep neural networks.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574--2582, 2016.
  * [22] N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and A. Swami.  Distillation as a defense to adversarial perturbations against deep neural networks.  2016 IEEE Symposium on Security and Privacy (SP), pages 582--597, 2016.
  * [23] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  MICCAI, 9351:234--241, 2015.
  * [24] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  CoRR, abs/1409.1556, 2014.
  * [25] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.  Intriguing properties of neural networks.  In International Conference on Learning Representations, 2014.
  * [26] Z. Wang and A. C. Bovik.  Mean squared error: Love it or leave it? a new look at signal fidelity measures.  IEEE Signal Processing Magazine, 26:98--117, 2009.
  * [27] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600--612, April 2004\.
  * [28] Z. Wang, E. P. Simoncelli, and A. C. Bovik.  Multiscale structural similarity for image quality assessment.  In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pages 1398--1402 Vol.2, Nov 2003.
  * [29] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. L. Yuille.  Mitigating adversarial effects through randomization.  CoRR, abs/1711.01991, 2017.
  * [30] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.


# Right to be Forgotten in the Era of Large Language Models: Implications,
Challenges, and Solutions

Dawen Zhang1,2, Pamela Finckenberg-Broman1, Thong Hoang1,
Shidong Pan1,2, Zhenchang Xing1,2, Mark Staples1, Xiwei Xu1

CSIRO’s Data611
Australian National University2
<EMAIL_ADDRESS>
###### Abstract

The Right to be Forgotten (RTBF) was first established as the result of the
ruling of Google Spain SL, Google Inc. v AEPD, Mario Costeja González, and was
later included as the Right to Erasure under the General Data Protection
Regulation (GDPR) of European Union to allow individuals the right to request
personal data be deleted by organizations. Specifically for search engines,
individuals can send requests to organizations to exclude their information
from the query results. It was a significant emergent right as the result of
the evolution of technology. With the recent development of Large Language
Models (LLMs) and their use in chatbots, LLM-enabled software systems have
become popular. But they are not excluded from the RTBF. Compared with the
indexing approach used by search engines, LLMs store, and process information
in a completely different way. This poses new challenges for compliance with
the RTBF. In this paper, we explore these challenges and provide our insights
on how to implement technical solutions for the RTBF, including the use of
differential privacy, machine unlearning, model editing, and prompt
engineering. With the rapid advancement of AI and the increasing need of
regulating this powerful technology, learning from the case of RTBF can
provide valuable lessons for technical practitioners, legal experts,
organizations, and authorities.

Keywords: GDPR, Right to be forgotten, Privacy, Large Language Models

##  1 The Legal Principles behind Right to be Forgotten

This section aims to introduce the Right to be Forgotten and its relationship
with relevant legal principles behind the law, i.e., the General Data
Protection Regulation.

###  1.1 The Origin of RTBF

Privacy is a fundamental human right. This is affirmed by a collection of
documents, including the International Covenant on Civil and Political Rights
(ICCPR) Article 17111https://www.ohchr.org/en/instruments-
mechanisms/instruments/international-covenant-civil-and-political-
rights#article-17, Universal Declaration of Human Rights (UDHR) Article
12222https://www.un.org/en/about-us/universal-declaration-of-human-rights, and
the European Convention on Human Rights (ECHR) Article
8333https://fra.europa.eu/en/law-reference/european-convention-human-rights-
article-8-0. It is explicitly declared in ECHR that ”Everyone has the right to
respect for his or her private and family life, home, and communications”. The
Right to be Forgotten (RTBF) is a critical aspect of the fundamental human
right to Privacy. RTBF evolved and established itself as a legal principle
from the case law of the Court of Justice of the European Union (the CJEU)
within the European Union (EU) in response to the rise of the data-driven
society, where vast quantities of data and information are gathered,
processed, stored, and exchanged for diverse motives. In the European Union (
the EU), the right to Privacy is set as primary law and binds all legal
subjects within the EU Member states. The Charter of Fundamental Rights of the
European Union (the Charter)444https://commission.europa.eu/aid-development-
cooperation-fundamental-rights/your-rights-eu/eu-charter-fundamental-rights_en
details on the right to Privacy, including communication, under Article 7, and
personal data protection under Article 8. Article 8 specifies what is included
in data protection.

“Everyone has the right to the protection of personal data concerning him or
her. Such data must be processed fairly for specified purposes and on the
basis of the consent of the person concerned or some other legitimate basis
laid down by law. Everyone has the right of access to data which has been
collected concerning him or her, and the right to have it rectified.
Compliance with these rules shall be subject to control by an independent
authority.”
— Article 8 - Protection of personal data, EU Charter of Fundamental Rights.

These Fundamental rights are echoed in secondary laws, such as the General
Data Protection Regulation (GDPR) and Directive 95/46, which, according to
Articles 1-3, protect the fundamental rights and freedoms of natural persons,
their right to privacy with respect to the processing of personal data, and of
removing obstacles to the free flow of such data.

Most nations protect privacy on different levels. In some, the Right to
Privacy is protected through constitutional provisions; in others, by national
laws, and entering international treaties. Notably, the extent of protection
afforded to individuals, the types of data that can be accessed and utilised,
and the parties covered by data protection laws differ significantly across
jurisdictions. This includes determining whose data can be accessed,
processed, and exploited within legal bounds and whether data protection
encompasses both private and public entities engaging in exploitation and
surveillance activities.

The Right to be Forgotten (RTBF) evolved and was established from the case of
Google Spain SL, Google Inc. v AEPD, Mario Costeja González555Case C-131/12
Google Spain SL, Google Inc. v Agencia Española de Protección de Datos (AEPD)
and Mario Costeja González (2014) ECLI:EU:C:2014:317: https://eur-
lex.europa.eu/legal-content/EN/TXT/?uri=ecli:ECLI:EU:C:2014:317. An
advertisement related to the data subject was published in a newspaper and
indexed by Google, which the data subject later discovered by searching his
name. The data subject submitted a request to remove his name from the
newspaper and Google search engine; however, both authorities rejected the
request. A legal case between the data subject and Google went to the Court of
Justice of the European Union. Their ruling required the search engine to
remove the personal data of individuals upon their requests. The ruling
referred to multiple principles, notably privacy, legitimate interests, and
balancing of interests.

  * •

Privacy: The ruling cites Articles 7 and 8 of the Charter of Fundamental
Rights of the European Union (the Charter), declaring that the processing of
personal data should respect the privacy of data subjects. Interestingly, the
ruling explicitly mentioned that the personal information of the data subject
would not be ubiquitously available and interconnected without the existence
of the internet and specifically search engines in modern society.

  * •

Legitimate interests: The ruling found that the operators of search engines
are the controllers of their data, as their processing of data has different
legitimate interests and consequences from the original publishers of
information.

  * •

Balancing of interests: The ruling acknowledges that the processing of
personal information is necessary if the processing is for legitimate
interests, but that such interests can be overridden by the interests of the
data subject’s fundamental rights and freedoms.

###  1.2 RTBF as a Part of GDPR

The RTBF was later included as the Right to
Erasure666https://gdpr.eu/article-17-right-to-be-forgotten/ under Article
17(1) of the General Data Protection Regulation (GDPR)777The European
Parliament and of the Council Regulation (EU) 2016/679 of 27 April 2016 on the
protection of natural persons with regard to the processing of personal data
and on the free movement of such data, and repealing Directive 95/46/EC
(General Data Protection Regulation): https://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=CELEX:32016R0679 of European Union, as a result of EU Data
Protection Reform888https://commission.europa.eu/law/law-topic/data-
protection/reform_en. It furthers the protection of rights established through
the original case by codifying a collection of rights (Art. 12–23) into the
law, notably the Right to be Informed (Art. 13, Art. 14), Right of Access
(Art. 15), Right to Rectification (Art. 16), and Right to Erasure (Art. 17).

  * •

Right to be Informed (Art. 13, Art. 14) Individuals have the right to be
informed when their data is collected and used. If the data is obtained from
the data subject directly, the data subject must be informed at the time when
personal data are obtained; and if the data is obtained from other sources,
the data subject must be informed within a reasonable period of time which
should not exceed one month.

  * •

Right of Access (Art. 15) Individuals have the right to request information
about the processing of their personal data. Specifically, the information may
include whether or not their personal data is processed, access to the
personal data, and information about the processing of personal data such as
purpose, duration, etc. The right of access also allows the data subjects to
learn about their personal data being processed before practicing other rights
where the access to their personal data being processed is necessary or
helpful.

  * •

Right to Rectification (Art. 16) The data subjects have the right to have
their inaccurate personal data rectified from the controller.

  * •

Right to Erasure (Art. 17) The data subjects have the right to request erasure
of their personal data from the controller without undue delay. The right may
also apply to circumstances where the personal data is inaccurate999TU and RE
v Google LLC: https://eur-lex.europa.eu/legal-
content/en/TXT/?uri=CELEX:62020CJ0460. The undue delay is commonly regarded as
one month, and may be extended by 2 months for complex or multiple requests
where the data subjects shall be informed101010Responding to requests - Data
protection under GDPR: https://europa.eu/youreurope/business/dealing-with-
customers/data-protection/data-protection-gdpr/index_en.htm.

Moreover, RTBF is not an absolute right. GDPR explicitly defined six grounds
for the erasure of personal data, including situations where data collection
or processing is no longer relevant or consent has been revoked. Additionally,
there are four circumstances in which this right shall not apply, such as when
it conflicts with freedom of expression or certain public interests. Other
jurisdictions also recognize such rights, including Argentina, India, and
South Korea.

Since the establishment of the case, Google published an online
form111111https://reportcontent.google.com/forms/rtbf for users to request
results related to them being delisted. According to their published paper
[1], they processed 3,231,694 requests about RTBF during the five years from
May 30th, 2014, to May 31st, 2019. They found that news, government, social
media, and directory sites were the most targeted for submitting RTBF
requests.

##  2 Large Language Models and Data Practices

This section aims to introduce Large Language Models and their challenges.

###  2.1 The Training of LLMs

Large Language Models (LLMs) are language models based on deep neural networks
(DNNs) with billions of parameters trained on vast amounts of data [2]. The
training data for these LLMs consists of a large proportion of public data on
the internet. For example, Common Crawl data is 60% of the training data used
in GPT-3 [3]; 50% of PaLM’s training dataset is social media conversations
[4]; and OpenAI and Google have extensively used Reddit user posts in their
large language models [5, 6, 7]. Moreover, these tech companies such as OpenAI
may also collect user interactions with large language models for future
training121212New ways to manage your data in ChatGPT:
https://openai.com/blog/new-ways-to-manage-your-data-in-chatgpt.

Unlike previous pre-trained language models, i.e., ELMo [8] and BERT [9],
recent LLMs have become more end-user-facing, which requires users to interact
with them through a prompting interface. Specifically, users need to
understand how LLMs work and guide LLMs to understand their problems. For
example, GPT-3 [10] popularized the paradigm of prompting, in which users
input a carefully crafted text prompt, and by completing the text, GPT-3 may
be able to output useful information for certain tasks. Some researchers
called such capabilities that only exist in large models but not in small
models the ”emergent abilities” of LLMs [11]. To exploit such capabilities of
LLMs to follow human instructions, more advanced models, such as InstructGPT
and ChatGPT, rely on human feedback during training, which involves a reward
function trained on manually generated prompt-response pairs through
Reinforcement Learning from Human Feedback (RLHF), to steer the LLM to follow
human instructions [12]. Based on these techniques, technology companies and
the community have launched chatbots based on LLMs, including OpenAI’s
ChatGPT131313https://openai.com/blog/chatgpt, Google’s
Flan-T5141414https://huggingface.co/docs/transformers/model_doc/flan-t5,
Meta’s LLaMA151515https://ai.facebook.com/blog/large-language-model-llama-
meta-ai/, Anthropic’s Claude161616https://www.anthropic.com/product, and
Stanford’s Alpaca171717https://crfm.stanford.edu/2023/03/13/alpaca.html. LLMs
have been embedded into other tools, including search engines like Microsoft’s
Bing181818https://www.bing.com/new and GitHub’s
Copilot191919https://github.com/features/copilot. Moreover, some LLMs have
been given access to other tools, such as Google’s
Bard202020https://blog.google/technology/ai/bard-google-ai-search-updates/ or
OpenAI’s ChatGPT Plugins212121https://openai.com/blog/chatgpt-plugins.

###  2.2 Issues of LLMs related to Personal Data

However, though LLMs are impressive and considered a disruptive technology by
many, they suffer major issues related to data. Training data memorization and
hallucination are two typical issues.

Training Data Memorization. It has been observed that LLMs may memorize
personal data, and this data can appear in their output [13]. Even though
companies try to remove personal data from training datasets, there could
still be personal data contained in the training
dataset222222https://openai.com/blog/our-approach-to-ai-safety.

Hallucination. Large language models may output factually incorrect content,
which is known as ”hallucination” [14], and producing such output does not
require factually incorrect information to be present in the training dataset.
Even if the context is given, the LLM-based generative search engines are
likely to give incorrect citations, or draw wrong conclusions from the context
[15].

Memorization and hallucination may happen without explicitly asking for
related information (e.g. “Please tell me about [person name]”). The authors
encountered a case where the LLM was asked to format a text related to person
A, and the LLM additionally inserted a paragraph about person B who was never
mentioned in the prompt context. The example is shown in Fig 1 and the
personal information is masked for privacy.

Figure 1: An example of ChatGPT memorization and hallucination happening at
the same time when not being explicitly asked for information. Actual names
and descriptions are masked for privacy.

##  3 Large Language Models and RTBF

This section focuses on the comparison between LLMs and search engines and the
challenges in applying LLMs to solve the Right to be Forgotten problem.

###  3.1 Comparing LLMs and Search Engines

The RTBF was first established via a case targeting search engines.
Interestingly, many on social media debate whether ChatGPT will replace search
engines232323https://www.nytimes.com/2022/12/21/technology/ai-chatgpt-google-
search.html. For this reason, we make a technical comparison between LLMs and
search engines and discuss whether the operations of LLMs would fall under the
grounds of search engines. We identify three major similarities and three main
differences between LLMs and search engines mentioned as follows.

#### Similarities

  * •

Organizing internet data. LLMs and search engines have sourced data from the
internet. Specifically, LLMs are deep neural networks trained on crawled data
also scraped from web pages, with the data embedded into these LLMs as
weights, while search engines use crawlers to scrape web pages and index this
data.

  * •

Used to access information. Users often employ LLMs and search engines to
access information. While LLMs are trained on a vast amount of online
information and provide their output in a generative way, search engines are
used to search through online information via indexes. This usage has led to a
debate about whether LLMs can replace search engines.

  * •

Intertwined with each other. LLMs have been embedded into search engines,
e.g., Microsoft’s Bing, while search engines are also now embedded into LLMs,
e.g., Google’s Bard.

#### Disimilarities

  * •

Predicting words vs. indexing information. LLMs are trained to predict the
next word in a sentence, and the relationship between words does not
necessarily reflect the actual information of the trained models. Search
engines are created to collect, index, and rank relevant webpages based on
user queries.

  * •

Conversational chatbots vs. search box. One trend of LLMs is they aim to solve
users’ problems by employing the interfaces of conversational chatbots, in
which users refine inputs about their problems in multi-round conversations
with LLMs. On the other hand, search engines provide services through a user
interface with a search box that receives users’ queries and outputs a list of
relevant web pages.

Overall, LLMs have similar source data to search engines, and the datasets
used to develop these models may contain personal data, causing privacy
concerns. ChatGPT, a chatbot built upon GPT-3.5, a typical large language
model, has become the fastest-growing application in
history242424https://www.reuters.com/technology/chatgpt-sets-record-fastest-
growing-user-base-analyst-note-2023-02-01/, which may also place LLMs in a
similar position as search engines mentioned in the court ruling. Similar to
search engines, LLMs do not share the same legitimate interests as the
original publishers from whom the data were crawled. Instead, these data are
used by organizations to train models and provide chatbot services to users.
The use of these data also has various consequences for the original
publishers. Therefore, organizations training LLMs should be considered data
controllers.

The balancing of interests allows the use of data to be justified by
legitimate interests such as research purposes or freedom of speech. While
OpenAI labels ChatGPT as a research preview and Google marks Bard as an
experiment, these labels do not remove their commercial nature. Even if they
can be justified by research purposes, the balancing of interests allows these
legitimate interests to be overridden by the requests of data subjects.

###  3.2 Challenges of Applying RTBF on LLMs

We have shown that LLMs are not immune to the ruling of the original RTBF
case. We further summarize the scenarios where RTBF-related GDPR articles may
apply in Fig. 2.

$\begin{dcases}\text{User Chat History}\\\ \text{In-model
Data}\begin{dcases}\text{Memorized Data}\\\ \text{Hallucinated Data}\\\
\end{dcases}\end{dcases}$ Figure 2: Scenarios where RTBF-related GDPR articles
may apply.

#### User chat history.

LLMs provide services in the form of chatbots, where users converse with the
system instead of just typing in a few simple keywords or questions. This
leads to a greater likelihood of users providing context and personal
information to the system through multiple rounds of exchange [16]. For
example, some users shared their experience of using ChatGPT for medical
consultations252525https://www.scientificamerican.com/article/ai-chatbots-can-
diagnose-medical-conditions-at-home-how-good-are-they/, which may contain
information regarded as personal data according to GDPR. Therefore, user chat
history data is likely within the scope of RTBF. To comply with the law,
consent must be given before user chat history is collected and used for
purposes like training models. Users should be able to withdraw their consent
and have their data erased if they no longer want their data used in this way,
i.e., by building large language models.

#### In-model data.

Personal data may also exist in the model and can be collected by prompting
the model. These personal data in models are the result of model training with
datasets containing personal information. However, the output of such
extraction can be accurate memorized data or inaccurate hallucinated data (see
Fig. 1). According to GDPR, users may request access, rectification, or
deletion of these data. However, this may pose major challenges for LLMs to
achieve. We again use the comparison between LLMs and search engines to
demonstrate these challenges.

#### Right of access.

In search engines, data subjects can practice their right to access by simply
querying keywords, which is how the original RTBF case started. Though the
technologies now used by search engines are far more sophisticated than
before, the information is still organized through indexing and can be easily
accessed by users via queries. In contrast, in LLMs, it is hard to know what
personal data are used in training and how to attribute these data to
particular individuals. Data subjects can only learn about their personal data
in these LLMs by either inspecting the original training dataset or perhaps by
prompting the model. However, training datasets are sometimes not disclosed,
especially those that are proprietary. Prompting the trained models also
cannot guarantee the outputs contain the full list of information stored in
the model weights. For example, we prompted ChatGPT about a person in Fig. 1;
however, ChatGPT responded that said it could not find any information about
her, as shown in Fig. 3. Moreover, there is no stable and robust way to access
hallucinated data.

Figure 3: ChatGPT cannot provide information about a person it output with the
prompt in Fig 1, which shows practicing right of access by prompting the model
is not feasible. Actual names and descriptions are masked for privacy.

#### Right to erasure.

In search engines, data can be erased by two methods. The first method is to
start at the root by removing the source web pages containing personal
information. The search engine entries will automatically become invalid after
the web page is removed and the search engine cache is expired, as the indexes
are linked to the original web pages. The second method is to delist the
particular entries of links related to personal data from the search engine
indexes. In this way, the original web pages will remain available, but will
no longer be included in the results of queries. However, we are unable to
apply these two methods to LLMs. Removing personal data from training datasets
does not directly affect the trained model, but can be effective only after
the next training of the model. Training a large language model may take
months. For example, LLaMA was trained between December 2022 and February
2023262626https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md.
This far exceeds the “undue delay” required by GDPR, which is considered to be
about one month272727https://gdpr.eu/right-to-be-forgotten/. Moreover, it is
difficult to remove data from a trained model, as model weights are a complex
integration of the whole collection of training data. Another problem is that
removing hallucinated data is challenging. Hallucinated data are not contained
in the training dataset of the model, and hallucinated data from the model is
hard to eliminate. Even if some hallucinated data could be removed from the
model, side effects and new hallucination might be introduced. Eliminating
hallucination from LLMs is still impossible now.

#### Right to rectification.

The approaches to rectifying data are similar to the right to erasure. Search
engines can do this by either rectifying the original websites or the links.
For LLMs, rectifying data from models is still a difficult task. We can see
that the root causes of these challenges are the issues of training data
memorization and hallucination mentioned in Section 2.2. The problem is even
more tricky due to a large amount of time required for training or retraining
the model and the inability to remove or modify data directly in the model.
Similar to the right to erasure, rectifying hallucination from LLMs is also
difficult to achieve.

##  4 Technical Solutions

While the above challenges and their root causes remain unsolved, ongoing
research efforts are focusing on addressing these issues. Based on the
targeted stages within the lifecycle of AI models, We divided the solutions
into two types: privacy-preserving machine learning (PPML) and post-training
methods. With reference to the two streams of solutions in search engines
mentioned above, i.e. removing the source web page and delisting, We further
map the works in post-training methods into two categories: fixing the
original model and band-aid approaches. These methods have the potential to
provide solutions for the challenges and can also be combined to solve the
problem. However, all of these approaches require further research. This
section does not aim to provide an exhaustive list of relevant research
advances in these areas, but rather point out potential solution space for
addressing RTBF in LLMs.

###  4.1 Privacy-preserving Machine Learning

Privacy-preserving Machine Learning (PPML) represents a line of research
focusing on preserving the privacy throughout the machine learning process by
addressing various threats to privacy, including Private Data in the Clear,
Model Inversion Attacks, Membership Inference Attacks, De-anonymization, and
Reconstruction Attacks [17]. One particular area in PPML is Differential
Privacy (DP), which provides formal guarantee for the privacy of training
data. Yue et al. [18] demonstrated that models can be trained with DP while
maintaining competitive performance in synthetic text generation tasks and the
also performance of the subsequent model trained on the generated synthetic
text. This shows DP can be a promising solution for preventing privacy issues
in LLMs. Further research efforts are needed in expanding the solution to
other tasks, as well as the ”emergent abilities” of LLMs.

###  4.2 Fixing the original model

The methods in this category aim to solve the root of the issue by fixing the
original model and fall within the idea of machine unlearning [19]. Though
several methods have been proposed, there is a lack of known use cases in
industry practices.

#### Exact Machine Unlearning

The exact machine unlearning methods remove the exact data points from the
model through an accelerated re-training process achieved with training
dataset partitioning. SISA [20] is a typical exact machine unlearning method
that adopts a voting-based aggregation mechanism on top of sharding and
slicing of the training dataset, and the re-training happens at the checkpoint
of a shard where the data point being removed has yet to be fed into the
model. The process can be even faster when the sharding and slicing leverage a
priori probabilities of data point removal. However, such acceleration may
come at the cost of fairness [21, 22].

#### Approximate Machine Unlearning

The approximate machine unlearning methods adjust the model weights by
approximating the effects of removing the data from the training set. This can
be achieved in various ways. One way is by calculating the influence of data
on weights through methods such as Newton steps [23, 24]; and another way is
by storing the information, such as weight updates, during the training for
later data deletion [25]. Despite these techniques, approximate unlearning is
susceptible to the issue of over-unlearning, which can significantly
compromise the model’s efficacy [26].

###  4.3 Band-aid Approaches

The methods in this category do not deal with the original model but instead
introduce side paths to change its behaviors. These methods were not
originally designed for RTBF; however, they may be used as potential solutions
for this RTBF problem.

#### Model Editing

Model editing methods [27, 28] leave the original model as it is and store the
modifications separately. The output of the model will use both the original
model and the stored modifications. However, these methods suffer from the
inconsistency of downstream knowledge, and the relevant knowledge might not be
updated accordingly.

#### Prompting

By providing RTBF requests in the prompts, LLMs may follow the instructions
for data removal requests, as shown in Fig. 4. With the growth of the number
of requests received, external storage like vector databases can be leveraged.
However, this method may face threats such as prompt injection or extraction.
Moreover, due to the nature of this method, the data is not actually removed
in accordance with the law, so the method should not be used as a
comprehensive solution to RTBF.

(a) Famous scandals were used in the prompt, and it is clear that ChatGPT
learned the event in the training data.

(b) Once being asked about the event, ChatGPT followed instructions of RTBF
requests provided in the prompt.

Figure 4: By providing RTBF requests in the prompts, ChatGPT can follow the
instruction of RTBF requests. For the privacy of data subjects, the names and
relevant information are masked.

##  5 Legal Perspectives

The technology has been evolving rapidly, leading to the emergence of new
challenges in the field of law, but the principle of privacy as a fundamental
human right should not be changed, and people’s rights should not be
compromised as a result of technological advancements. However, LLMs are
trending and have become significantly influential. Fresh interpretations of
laws related to the cases of LLMs may have to make trade-offs. For example,
the “undue delay” in GDPR, though not strictly specified in the article, is
commonly interpreted as one month. Considering the current technical
capabilities, this time frame might be impossible to achieve. Moreover, as
mentioned in 3.1, the companies are labelling LLM-based products as “research
preview” or “experiment”, justifying them under the legitimate interests of
RTBF. Though these products are the result or a part of research activities,
they are far beyond the definition of scientific research in the common sense.
How to balance the interests and interpret the law in a time of rapid
technological advancements and accelerated commercialization of research
outcomes, has become a question for all stakeholders involved in legal
matters.

##  6 On-going Discussion

There has been on-going discussion on the laws and AI. New regulations are
being drafted or planned, such as Proposal for AI Act by the European
Union282828https://eur-lex.europa.eu/legal-
content/EN/TXT/HTML/?uri=CELEX:52021PC0206, Interim Administrative Measures
for Generative Artificial Intelligence Services by
China292929http://www.cac.gov.cn/2023-07/13/c_1690898327029107.htm, and
Blueprint for an AI Bill of Rights by the White House of the United
States303030https://www.whitehouse.gov/ostp/ai-bill-of-rights/. In terms of
RTBF, how it could be applied to AI has been widely discussed in the research
community from both technical and legal perspectives [29, 30, 31]. In fact,
the Right to be Forgotten, as an ”emergent right” [32] distinct from
traditional rights, has been controversial since its emergence [33]. It was a
significant attempt to address the power gap between the legal protection of
privacy and the interference from the evolving technology. With the current
trend that the AI technology is becoming increasingly powerful, RTBF, as a
valuable precedent, may serve as a meaningful reference for the current and
future development of laws.

##  7 Conclusion

In this work, we demonstrate the implications of the Right to be Forgotten and
identified unique challenges brought by the large language models. We discuss
four potential technical solutions and provide insights on the issue from
legal perspectives. We believe that our work benefits practitioners and
stakeholders and helps them understand the issues around RTBF. We call for
more attention and further efforts into this matter.

## References

  * [1] T. Bertram, E. Bursztein, S. Caro, H. Chao, R. C. Feman, P. Fleischer, A. Gustafsson, J. Hemerly, C. Hibbert, L. Invernizzi, L. K. Donnelly, J. Ketover, J. Laefer, P. Nicholas, Y. Niu, H. Obhi, D. Price, A. Strait, K. Thomas, and A. Verney, “Five years of the right to be forgotten,” in _Proceedings of the Conference on Computer and Communications Security_ , 2019\.
  * [2] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020.
  * [4] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel, “Palm: Scaling language modeling with pathways,” 2022.
  * [5] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [6] Y. Yang, S. Yuan, D. Cer, S.-y. Kong, N. Constant, P. Pilar, H. Ge, Y.-H. Sung, B. Strope, and R. Kurzweil, “Learning semantic textual similarity from conversations,” _arXiv preprint arXiv:1804.07754_ , 2018.
  * [7] OpenAI, “Gpt-4 technical report,” 2023.
  * [8] S. Edunov, A. Baevski, and M. Auli, “Pre-trained language model representations for language generation,” _arXiv preprint arXiv:1903.09722_ , 2019.
  * [9] A. Wang and K. Cho, “Bert has a mouth, and it must speak: Bert as a markov random field language model,” _arXiv preprint arXiv:1902.04094_ , 2019.
  * [10] L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” _Minds and Machines_ , vol. 30, pp. 681–694, 2020.
  * [11] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler _et al._ , “Emergent abilities of large language models,” _arXiv preprint arXiv:2206.07682_ , 2022.
  * [12] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [13] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson _et al._ , “Extracting training data from large language models.” in _USENIX Security Symposium_ , vol. 6, 2021.
  * [14] J. Maynez, S. Narayan, B. Bohnet, and R. T. Mcdonald, “On faithfulness and factuality in abstractive summarization,” in _Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2020.
  * [15] N. F. Liu, T. Zhang, and P. Liang, “Evaluating verifiability in generative search engines,” 2023.
  * [16] K. Saffarizadeh, M. Boodraj, T. M. Alashoor _et al._ , “Conversational assistants: Investigating privacy concerns, trust, and self-disclosure.” in _ICIS_ , 2017.
  * [17] M. Al-Rubaie and J. M. Chang, “Privacy-preserving machine learning: Threats and solutions,” _IEEE Security & Privacy_, vol. 17, no. 2, pp. 49–58, 2019\.
  * [18] X. Yue, H. Inan, X. Li, G. Kumar, J. McAnallen, H. Shajari, H. Sun, D. Levitan, and R. Sim, “Synthetic text generation with differential privacy: A simple and practical recipe,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 1321–1342. [Online]. Available: https://aclanthology.org/2023.acl-long.74
  * [19] Y. Cao and J. Yang, “Towards making systems forget with machine unlearning,” in _2015 IEEE Symposium on Security and Privacy_. IEEE, 2015, pp. 463–480.
  * [20] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,” in _2021 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2021, pp. 141–159.
  * [21] K. Koch and M. Soll, “No matter how you slice it: Machine unlearning with SISA comes at the expense of minority classes,” in _First IEEE Conference on Secure and Trustworthy Machine Learning_ , 2023. [Online]. Available: https://openreview.net/forum?id=RBX1H-SGdT
  * [22] D. Zhang, S. Pan, T. Hoang, Z. Xing, M. Staples, X. Xu, L. Yao, Q. Lu, and L. Zhu, “To be forgotten or to be fair: Unveiling fairness implications of machine unlearning methods,” _arXiv preprint arXiv:2302.03350_ , 2023.
  * [23] C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, “Certified data removal from machine learning models,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 3832–3842. [Online]. Available: https://proceedings.mlr.press/v119/guo20c.html
  * [24] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9304–9312.
  * [25] L. Graves, V. Nagisetty, and V. Ganesh, “Amnesiac machine learning,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 13, pp. 11 516–11 524, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/17371
  * [26] H. Hu, S. Wang, J. Chang, H. Zhong, R. Sun, S. Hao, H. Zhu, and M. Xue, “A duty to forget, a right to be assured? exposing vulnerabilities in machine unlearning services,” 2023.
  * [27] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, “Fast model editing at scale,” _arXiv preprint arXiv:2110.11309_ , 2021.
  * [28] E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn, “Memory-based model editing at scale,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 15 817–15 831.
  * [29] E. F. Villaronga, P. Kieseberg, and T. Li, “Humans forget, machines remember: Artificial intelligence and the right to be forgotten,” _Computer Law & Security Review_, vol. 34, no. 2, pp. 304–313, 2018.
  * [30] Q.-V. Dang, “Right to be forgotten in the age of machine learning,” in _Advances in Digital Science: ICADS 2021_. Springer, 2021, pp. 403–411.
  * [31] E. Esposito, “Algorithmic memory and the right to be forgotten on the web,” _Big Data & Society_, vol. 4, no. 1, p. 2053951717703996, 2017.
  * [32] F. Werro, _The Right to be Forgotten: A Comparative Study of the Emergent Right’s Evolution and Application in Europe, the Americas, and Asia_. Springer Nature, 2020, vol. 40.
  * [33] D. Lindsay, “The ‘right to be forgotten’by search engines under data privacy law: A legal analysis of the costeja ruling,” _Journal of Media Law_ , vol. 6, no. 2, pp. 159–179, 2014.