# Cosine Model Watermarking Against Ensemble Distillation

Laurent Charette1, Lingyang Chu2, Yizhou Chen3, Jian Pei3, Lanjun Wang4, Yong
Zhang1

###### Abstract

Many model watermarking methods have been developed to prevent valuable
deployed commercial models from being stealthily stolen by model
distillations. However, watermarks produced by most existing model
watermarking methods can be easily evaded by ensemble distillation, because
averaging the outputs of multiple ensembled models can significantly reduce or
even erase the watermarks. In this paper, we focus on tackling the challenging
task of defending against ensemble distillation. We propose a novel
watermarking technique named CosWM to achieve outstanding model watermarking
performance against ensemble distillation. CosWM is not only elegant in
design, but also comes with desirable theoretical guarantees. Our extensive
experiments on public data sets demonstrate the excellent performance of CosWM
and its advantages over the state-of-the-art baselines.

##  1 Introduction

High-performance machine learning models are valuable assets of many large
companies. These models are typically deployed as web services where the
outputs of models can be queried using public application programming
interfaces (APIs) (Ribeiro, Grolinger, and Capretz 2015).

A major risk of deploying models through APIs is that the deployed models are
easy to steal (Tramèr et al. 2016). By querying the outputs of a deployed
model through its API, many model distillation methods (Orekondy, Schiele, and
Fritz 2019; Jagielski et al. 2019; Papernot et al. 2017) can be used to train
a replicate model with comparable performance as the deployed model. Following
the context of model distillation (Hinton, Vinyals, and Dean 2015), a
replicate model is called a _student model_ ; and the deployed model is called
a _teacher model_.

A model distillation process is often imperceptible because it queries APIs in
the same way as a normal user (Orekondy, Schiele, and Fritz 2019). To protect
teacher models from being stolen, one of the most effective ways is model
watermarking (Szyller et al. 2019). The key idea is to embed a unique
watermark in a teacher model, such that a student model distilled from the
teacher model will also carry the same watermark. By checking the watermark,
the owner of the teacher model can identify and reclaim ownership of a student
model.

Some model watermarking methods have been proposed to identify student models
produced by single model distillation (Szyller et al. 2019; Lukas, Zhang, and
Kerschbaum 2019; Jia et al. 2021). However, as we discuss in Section 2,
watermarks produced by these methods can be significantly weakened or even
erased by _ensemble distillation_ (Hinton, Vinyals, and Dean 2015), which uses
the average of outputs queried from multiple different teacher models to train
a replicate model.

Ensemble distillation has been well-demonstrated to be highly effective at
compressing multiple large models into a small size student model with high
performance (Buciluǎ, Caruana, and Niculescu-Mizil 2006; Ba and Caruana 2014).
On the other hand, the effectiveness of ensemble distillation also poses a
critical threat to the safety of deployed models.

As shown by extensive experimental results in Section 5, ensemble distillation
not only generates student models with better prediction performance, but also
significantly reduces the effectiveness of existing model watermarking methods
in identifying student models. As a result, accurately identifying student
models produced by ensemble distillation is an emergent task with top priority
to protect teacher models from being stolen.

In this paper, we focus on defending against model distillation, and we
successfully tackle this task by introducing a novel model watermarking method
named CosWM. To the best of our knowledge, our method is the first model
watermarking method with a theoretical guarantee to accurately identify
student models produced by ensemble distillation. We make the following
contributions.

First, we present a novel method named CosWM that embeds a watermark as a
cosine signal within the output of a teacher model. Since the cosine signal is
difficult to erase by averaging the outputs of multiple models, student models
produced by ensemble distillation will still carry a strong watermark signal.

Second, under reasonable assumptions, we prove that a student model with a
smaller training loss value will carry a stronger watermark signal. This means
a student model will have to carry a stronger watermark in order to achieve a
better performance. Therefore, a student model intending to weaken the
watermark will not be able to achieve a good performance.

Third, we also design CosWM to allow each teacher model to embed a unique
watermark by projecting the cosine signal in different directions in the high-
dimensional feature space of the teacher model. In this way, owners of teacher
models can independently identify their own watermarks from a student model.

Last, extensive experiment results demonstrate the outstanding performance of
CosWM and its advantages over state-of-the-art methods.

##  2 Related Works

In this section, we introduce two major categories of model watermarking
methods and discuss why these methods can be easily evaded by ensemble
distillation.

The first category of methods (Uchida et al. 2017; Rouhani, Chen, and
Koushanfar 2018; Adi et al. 2018; Zhang et al. 2018; Le Merrer, Pérez, and
Trédan 2019) aim to protect machine learning models from being exactly copied.
To produce a watermark, an effective idea is to embed a unique pattern by
manipulating the values of parameters of the model to protect (Uchida et al.
2017; Rouhani, Chen, and Koushanfar 2018). If a protected model is exactly
copied, the parameters of the copied model will carry the same pattern, which
can be used as a watermark to identify the ownership of the copied model.
Another idea to produce a watermark is to use backdoor images that trigger
prescribed model predictions (Adi et al. 2018; Zhang et al. 2018; Le Merrer,
Pérez, and Trédan 2019). The same backdoor image will trigger the same
prescribed model prediction on an exactly copied model. Thus, the backdoor
images are also effective in identifying exactly copied models.

The above methods focus on identifying exactly copied models, but they cannot
be straight-forwardly extended to identify a student model produced by
ensemble distillation (Hinton, Vinyals, and Dean 2015). Because the model
parameters of the student model can be substantially different from the
teacher model; and simple backdoor images of the teacher model are often not
transferable to the student model, that is, the backdoor images may not
trigger the prescribed model prediction on the student model (Lukas, Zhang,
and Kerschbaum 2019).

The second category of methods aim to identify student models that are
distilled from a single teacher model by single model distillation (Tramèr et
al. 2016).

PRADA (Juuti et al. 2019) is designed to identify model distillations using
synthetic queries that tend to be out-of-distribution. It analyzes the
distribution of API queries and detects potential distillation activities when
the distribution of queries deviates from the benign distribution. However, it
is not effective in identifying the queries launched by ensemble
distillations, because these queries are mostly natural queries that are not
out-of-distribution.

Another typical idea is to produce transferable backdoor images that are
likely to trigger the same prescribed model prediction on both the teacher
model and the student model. DAWN (Szyller et al. 2019) generates transferable
backdoor images by dynamically changing the outputs of the API of a protected
teacher model on a small subset of querying images. Fingerprinting (Lukas,
Zhang, and Kerschbaum 2019) makes backdoor images more transferable by finding
common adversarial images that trigger the same adversarial prediction on a
teacher model and any student model distilled from the teacher model.
Entangled Watermarks (Jia et al. 2021) forces a teacher model to learn
features for classifying data sampled from the legitimate data and watermarked
data.

The above methods are effective in identifying student models produced by
single model distillation, but they cannot accurately identify student models
produced by ensemble distillation.

The reason is that, when an ensemble distillation averages the outputs of a
watermarked teacher model and multiple other teacher models without a
watermark, the prescribed model predictions of the watermarked teacher model
will be weakened or even erased by the normal predictions of the other teacher
models. If multiple watermarked teacher models are used for ensemble
distillation, the prescribed model prediction of one teacher model can still
be weakened or erased when averaged with the predictions of the other teacher
models, because the prescribed model predictions of different teacher models
are not consistent with each other.

The proposed CosWM method is substantially different from the other
watermarking methods (Szyller et al. 2019; Lukas, Zhang, and Kerschbaum 2019;
Jia et al. 2021). The watermark of CosWM is produced by coupling a cosine
signal with the output function of a protected teacher model. As proved in
Theorem 1 and demonstrated by extensive experiments in Section 5, when an
ensemble distillation averages the outputs of multiple teacher models, the
embedded cosine signal will persist. As a result, the watermarks produced by
CosWM are highly effective in identifying student models produced by ensemble
distillation.

##  3 Problem Definition

Ensemble methods, such as bagging (Bühlmann and Yu 2002), aggregate the
probability predictions of all models in an ensemble to create a more accurate
model on average. Ensemble models and distillation have been applied jointly
since the first seminal studies on distillation (Buciluǎ, Caruana, and
Niculescu-Mizil 2006; Ba and Caruana 2014; Hinton, Vinyals, and Dean 2015).
These distillation methods use a combination of KL loss (Kullback and Leibler
1951) and cross-entropy loss (Bishop 2006) in the training process. Cross-
entropy loss requires ground truth labels. Some recent state-of-the-art
distillation methods (Vongkulbhisal, Vinayavekhin, and Visentini-Scarzanella
2019; Shen and Savvides 2020) only use KL loss, and thus can work without
access to the ground truth values. This allows adversaries to replicate high
performance models using ensemble model distillation and without ground truth
labels.

Technically, let $\mathcal{R}=\\{R_{1},\ldots,R_{N}\\}$ be a set of $N$ models
trained to perform the same $m$-class classification task. Each model $R_{i}$
outputs a probability prediction vector $R_{i}(\mathbf{x})$ on an input sample
$\mathbf{x}\in\mathbb{R}^{n}$. An adversary may effectively build an ensemble
model by querying an unlabeled data set
$X^{S}=\\{\mathbf{x}^{1},\ldots,\mathbf{x}^{L}\\}$ to each model
$R_{1},\ldots,R_{N}$ and averaging the outputs, i.e.,
$\mathbf{\bar{q}}^{l}=\frac{1}{N}\sum_{i=1}^{N}R_{i}(\mathbf{x}^{l})$ for
$l=1,\ldots,L$. The averaged output $\mathbf{\bar{q}}^{l}$ can then be used as
the soft pseudo labels to train a student model $S$.

We now formulate the task of watermarking against distillation from ensembles.
Assume a model $R$ to be protected and the watermarked version $w(R)$, where
$w(\cdot)$ is a watermarking function. Denote by $h(R)$ a function measuring
the accuracy of model $R$ (on a given test data set) and by $g(R)$ a function
measuring the strength of the watermark signal in model $R$.

Let $S$ be an arbitrary model that is replicated from an ensemble distillation
using $w(R)$ as a teacher. $S$ may use some additional other teacher models.
Let $S^{\prime}$ be another arbitrary model that is replicated from an
ensemble distillation where $w(R)$ is not a teacher. The task of model
watermarking is to design watermarking function $w(\cdot)$ such that it meets
two requirements. First, the accuracy loss in watermarking is within a
specified tolerance range $\alpha>0$, i.e., $h(R)-h(w(R))\leq\alpha$. Second,
the watermark signal model in $S$ is stronger than that in $S^{\prime}$, i.e.,
$g(S)>g(S^{\prime})$.

##  4 CosWM

In this section, we present our watermarking method CosWM. We first explain
the intuition of our method. Then, we develop our watermarking framework to
embed a periodic signal to a teacher model. Third, we describe how the
embedded signal can be extracted from a student model learned using a
watermarked teacher model. Next, we provide strong theoretical results to
justify our design. Last, we discuss possible extensions to ensembles
containing multiple watermarked models.

###  4.1 Intuitions

The main idea of CosWM is to introduce a perturbation to the output of a
teacher model. This perturbation is transferred onto a student model distilled
from the teacher model and remains detectable with access to the output of the
student model.

Figure 1: The idea of CosWM, where $\mathbf{q}_{i}(\mathbf{x})$ is an model
output component for image $\mathbf{x}$, $p(\mathbf{x})$ is a projection as
described in equation (3), $f$ and $P(f)$ are frequency and power spectrum
values of a $p(\mathbf{x})$-$\mathbf{q}_{i}(\mathbf{x})$ graph.

The idea is illustrated in Figure 1. Let $R$ be a model to be watermarked and
$\mathbf{q}=R(\mathbf{x})$ be the output of the model $R$ on input
$\mathbf{x}$. We also convert $\mathbf{x}$ into a number $p(\mathbf{x})$ in a
finite range. We can select a class $i^{*}$ and use the model prediction
output $\mathbf{q}_{i^{*}}$ on the class to load our watermark. Let
$\mathbf{q}_{i^{*}}(\mathbf{x})$ be the $i^{*}$-th element of vector
$R(\mathbf{x})$. Figure 1(a) plots
$(\mathbf{q}_{i^{*}}(\mathbf{x}),p(\mathbf{x}))$ without any added watermark
signal. After adding a periodic perturbation $\phi(p(\mathbf{x}))$ of
frequency $f_{w}$ to the output of $R$, the new output
$\mathbf{q}_{i^{*}}(\mathbf{x})$ demonstrates some oscillations, as shown in
Figure 1(b). We keep the perturbation small enough so that the model
predictions are mostly unaffected and the effect of the watermark on the
model’s performance is minimal.

A student model trying to replicate the behavior of the teacher model
passively features a similar oscillation at the same frequency $f_{w}$. In
addition, even with the averaging effect of an ensemble of teacher models on
the outputs, the periodic signal should still be present in some form. Since
the averaging is linear, the amplitude is diminished by a factor of the number
of the ensemble models as shown in Figure 1(c). By applying a Fourier
transform, the perturbation can be re-identified by the presence of a peak in
the power spectrum at the frequency $f_{w}$ as shown in Figure 1(d).

###  4.2 Embedding Watermarks to a Teacher Model

Normally, an output $\mathbf{q}$ of a model $R$ on a given data point
$\mathbf{x}$ is calculated from the softmax of the logits
$\mathbf{z}\in\mathbb{R}^{m}$, i.e.,

$\mathbf{q}_{i}=\frac{e^{\mathbf{z}_{i}}}{\sum_{j=1}^{m}e^{\mathbf{z}_{j}}},\
\mbox{for}\ i=1,...,m,$ (1)

where $\mathbf{z}$ is a function of $\mathbf{x}$, and $\mathbf{q}_{i}$ is the
$i$-th element of vector $\mathbf{q}$. As a result, the output $\mathbf{q}$
has the following property.

######  Property 1.

Let $\mathbf{q}$ be a softmax of the logit output $\mathbf{z}$ of a model $R$.
Then,

  1. 1.

$0\leq\mathbf{q}_{i}\leq 1$ for $i=1,\ldots,m$,

  2. 2.

$\sum_{i=1}^{m}\mathbf{q}_{i}=1$.

We want to substitute $\mathbf{q}$ in the model inference by a modified output
$\mathbf{\hat{q}}\in\mathbb{R}^{m}$ which features the periodic signal and
satisfies Property 1. However, only modifying $\mathbf{q}$ in the model
inference by itself may degrade the performance of the model, and the loss in
accuracy cannot be bounded. In order to mitigate this effect, we also use the
modified output $\mathbf{\hat{q}}$ in training $R$. That is, we use
$\mathbf{\hat{q}}$ to compute cross entropy loss in the training process.

To embed watermarks, we first define a watermark key $K$ that consists of a
target class $i^{*}\in\\{1,\ldots,m\\}$, an angular frequency
$f_{w}\in\mathbb{R}$, and a random unit projection vector
$\mathbf{v}\in\mathbb{R}^{n}$, i.e., $K=(i^{*},f_{w},\mathbf{v})$. Using $K$,
we define a periodic signal function

$\mathbf{a}_{i}(\mathbf{x})=\left\\{\begin{aligned}
&\cos\left(f_{w}p(\mathbf{x})\right),&\mbox{if}\ i=i^{*},\\\
&\cos\left(f_{w}p(\mathbf{x})+\pi\right),&\mbox{otherwise},\end{aligned}\right.$
(2)

for $i\in\\{1,\ldots,m\\}$, where

$p(\mathbf{x})=\mathbf{v}^{\intercal}\mathbf{x}.$ (3)

We consider single-frequency signals in this work and we plan to study
watermark signals with mixed frequencies in our future work. We adopt linear
projections since they are simple one-dimensional functions of input data and
can easily form a high-dimensional function space. This leads to a large-
dimensional space to select $\mathbf{v}$ from, and generally little
interference between two arbitrary choices of $\mathbf{v}$. As a consequence,
we get a large choice of possible watermarks, and each watermark is concealed
to adversaries trying to source back the signal with arbitrary projections.

We inject the periodic signal into output $\mathbf{q}$ to obtain
$\mathbf{\hat{q}}$ as follows. For $i\in\\{1,\ldots,m\\}$,

$\mathbf{\hat{q}}_{i}=\left\\{\begin{aligned}
&\frac{\mathbf{q}_{i}+\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))}{1+2\varepsilon},&\mbox{if}\
i=i^{*},\\\
&\frac{\mathbf{q}_{i}+\frac{\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))}{m-1}}{1+2\varepsilon},&\mbox{otherwise},\end{aligned}\right.$
(4)

where $\varepsilon$ is an amplitude component for the watermark periodic
signal. As proved in Lemma 1 in Appendix A.2, the modified output
$\mathbf{\hat{q}}$ still satisfies both requirements in Property 1. Therefore,
it is natural to replace $\mathbf{q}$ by $\mathbf{\hat{q}}$ in inference.

Nevertheless, if we only modify $\mathbf{q}$ into $\mathbf{\hat{q}}$ in
inference, the inference performance can be degraded by this perturbation.
Since the modified output satisfies Property 1, we can use it in training as
well to compensate for the potential performance drop. To do that, we directly
replace $\mathbf{q}$ by $\mathbf{\hat{q}}$ in the cross-entropy loss function.
Specifically, for a data point $\mathbf{x}$ with one-hot encoding true label
$\mathbf{y}^{t}\in\mathbb{R}^{m}$, the cross-entropy loss during training can
be replaced by

$L_{CE,wm}=-\sum_{j=1}^{m}\mathbf{y}^{t}_{j}\log\left(\mathbf{\hat{q}}_{j}\right).$
(5)

The model $R_{w}$ trained as such carries the watermark. By directly modifying
the output, we ensure that the signal is present in every output, even for
input data not used during training. This generally results in a clear signal
function in the output of the teacher model $R_{w}$ that is harder to conceal
by noise caused by distillation training or by dampening due to ensemble
averaging.

###  4.3 Extracting Signals in Student Models

Let $S$ be a student model that is suspected of being distilled from a
watermarked model $R_{w}$ or multiple ensembled teacher models including
$R_{w}$. To extract the possible watermark from $S$, we need to query $S$ with
a sample of student training data
$\widetilde{X}^{S}=\\{\mathbf{x}^{1},\ldots,\mathbf{x}^{\widetilde{L}}\\}$.
According to (Szyller et al. 2019), the owner of a teacher model can easily
obtain $\widetilde{X}^{S}$ because the owner may store any query input sent by
an adversary to the API.

Let the output of model $S$ on the input data $\widetilde{X}^{S}$ be
$\widetilde{Q}^{S}=\\{\mathbf{q}^{1},\ldots,\mathbf{q}^{\widetilde{L}}\\}$,
where $\mathbf{q}^{l}\in\mathbb{R}^{m}$ for $l=1,\ldots,{\widetilde{L}}$. For
every pair $(\mathbf{x}^{l},\mathbf{q}^{l})$, we extract a pair of results
$(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})$, where
$\mathbf{p}_{l}=\mathbf{v}^{\intercal}\mathbf{x}^{l}$ as per Equation (3),
$\mathbf{v}$ is in the watermark key of $R_{w}$ and $i^{*}$ is the target
class when embedding watermarks to $R_{w}$. We filter out the pairs
$(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})$ with $\mathbf{q}_{i^{*}}^{l}\leq
q_{min}$ in order to remove outputs with low confidence, where the threshold
value $q_{min}$ is a constant parameter of the extraction process. The
surviving pairs are re-indexed into a set
$\widetilde{D}^{S}=\\{(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})\\}_{l=1,\ldots,\widetilde{M}}$,
where $\widetilde{M}$ is the number of remaining pairs. These surviving pairs
$(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})\in\widetilde{D}^{S}$ are then used to
compute the Fourier power spectrum, for evenly spaced frequency values
spanning a large interval containing the frequency $f$.

To approximate the power spectrum, we use the Lomb-Scargle periodogram method
(Scargle 1982), which allows one to approximate the power spectrum $P(f)$ at
frequency $f$ using unevenly sampled data. We give the formal definition of
$P(f)$ in Section 4.4 when we analyze the theoretical bounds of $P(f)$. Due to
noise in the model outputs, it is preferable to have more sample pairs in
$\widetilde{D}^{S}$ than the few required to detect a pure cosine signal. In
our experience, we reliably detect a watermark signal using 100 pairs for a
single watermarked model and 1,000 pairs for an 8-model ensemble.

To measure the signal strength of the watermark, we define a maximum frequency
$F$ and a window $\left[f_{w}-\frac{\delta}{2},f_{w}+\frac{\delta}{2}\right]$,
where $\delta$ is a parameter for the width of the window and $f_{w}$ is the
frequency in watermark key of $R_{w}$. Then, we calculate $P_{signal}$ and
$P_{noise}$ by averaging spectrum values $P(f)$ on frequencies inside and
outside the window, i.e.,
$P_{signal}=\frac{1}{\delta}\int_{f_{w}-\frac{\delta}{2}}^{f_{w}+\frac{\delta}{2}}P(f)df$
and
$P_{noise}=\frac{1}{F-\delta}\left[\int_{0}^{f_{w}-\frac{\delta}{2}}P(f)df+\int_{f_{w}+\frac{\delta}{2}}^{F}P(f)df\right]$,
respectively. We use the signal-to-noise ratio to measure the signal strength
of the watermark, i.e.,

$P_{snr}=P_{signal}/P_{noise}.$ (6)

The algorithm is summarized in Algorithm 1.

Inputs : A suspected model $S$,

Samples $\widetilde{X}_{S}$ of the training data of $S$,

A watermark key $K=(i^{*},f_{w},\mathbf{v})$ of the

watermarked model $R_{w}$,

Filtering threshold value $q_{min}$.

Output : Signal strength.

1 Query $\widetilde{X}_{S}$ to $S$ and obtain outputs
$\widetilde{Q}^{S}=\\{\mathbf{q}^{1},\ldots,\mathbf{q}^{\widetilde{L}}\\}$.

2Compute projections
$\mathbf{p}_{l}=\mathbf{v}^{\intercal}\cdot\mathbf{x}^{l}$, for
$l=1,\ldots,\widetilde{L}$.

3Filter out outputs where $\mathbf{q}_{i^{*}}^{l}\leq q_{min}$, remaining
pairs form the set
$\widetilde{D}^{S}=\\{(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})\\}_{l=1,\ldots,\widetilde{M}}$.

4Compute the Lomb-Scargle periodogram from the pairs
$(\mathbf{p}_{l},\mathbf{q}_{i^{*}}^{l})$ in $\widetilde{D}^{S}$.

5Compute $P_{signal}$ and $P_{noise}$ by averaging spectrum values on
frequencies inside and outside the window
$\left[f_{w}-\frac{\delta}{2},f_{w}+\frac{\delta}{2}\right]$, respectively.

6Compute $P_{snr}=P_{signal}/P_{noise}$.

7return Signal strength $P_{snr}$.

Algorithm 1 Extracting signal in a model

###  4.4 Theoretical Analysis

Here, we analyze the signal strength of $P_{signal}$ and $P_{noise}$ and
provide theoretical bounds for the power spectrum $P(f)$. Let us first recall
two results from (Scargle 1982).

Given a paired data set
$D=\\{(\mathbf{a}^{l},\mathbf{b}_{l})\in\mathbb{R}^{n}\times\mathbb{R},l=1,\ldots,L\\}$,
an angular frequency $f$, a projection vector $\mathbf{v}$, and a sinusoidal
function
$s(\mathbf{x})=\alpha+\beta\cos(f\mathbf{v}^{\intercal}\mathbf{x}+\gamma),$
where $\alpha$, $\beta$ and $\gamma$ are the parameters of $s(\mathbf{x})$,
the _best fitting points_ $\mathbf{s}^{*}(D)$ for this paired data are

$[\mathbf{s}^{*}(D)]_{l}=\alpha^{*}+\beta^{*}\cos(f\mathbf{v}^{\intercal}\mathbf{a}^{l}+\gamma^{*})\
\mbox{for}\ l=1,\ldots,L,$ (7)

where the parameters $\alpha^{*}$, $\beta^{*}$, $\gamma^{*}$ minimize the
square error
$\chi^{2}_{f}(D)=\sum_{l=1}^{L}[\mathbf{b}_{l}-s(\mathbf{a}^{l})]^{2}.$

Moreover, given a paired data set
$D=\\{(\mathbf{a}^{l},\mathbf{b}_{l})\in\mathbb{R}^{n}\times\mathbb{R},l=1,\ldots,L\\}$
and a frequency $f$, the _unnormalized Lomb-Scargle periodogram_ can be
written as

$P_{D}(f)=\frac{1}{2}\left[\chi_{0}^{2}(D)-\chi_{f}^{2}(D)\right],$ (8)

where $\chi_{0}^{2}(D)$ is the square error of the best constant fit to
$\mathbf{b}_{1},\ldots,\mathbf{b}_{L}$.

Now we are ready to give a theoretical bound on $P_{D}(f)$ for the output of
the student model.

######  Theorem 1.

Suppose there are $N$ teacher models $R_{1},\ldots,R_{N}$. Without loss of
generality, let $R_{1}$ be a watermarked teacher model with watermark key
$K=(i^{*},f_{w},\mathbf{v})$, and $S$ a student model distilled from an
ensemble model of $R_{1},\ldots,R_{N}$ on the student training data $X^{S}$.
Let $\widetilde{X}^{S}=\\{\mathbf{x}^{1},\ldots,\mathbf{x}^{L}\\}$ be a sample
subset of $X^{S}$. Let $\mathbf{\hat{q}}^{l}=R_{1}(\mathbf{x}^{l})$ be the
output of model $R_{1}$,
$\mathbf{\widetilde{q}}^{l}=\frac{1}{N-1}\sum_{i=2}^{N}R_{i}(\mathbf{x}^{l})$
be the output of the ensemble model of $R_{2},\ldots,R_{N}$,
$\mathbf{\bar{q}}^{l}=\frac{1}{N}(\mathbf{\hat{q}}^{l}+(N-1)\mathbf{\widetilde{q}}^{l})$
be the output of the ensemble model of $R_{1},\ldots,R_{N}$, and
$\mathbf{q}^{l}=S(\mathbf{x}^{l})$ the output of $S$ for the training data
point $\mathbf{x}^{l}$. Let
$\hat{D}=\\{(\mathbf{x}^{l},\mathbf{\hat{q}}^{l}_{i^{*}}),l=1,\ldots,L\\}$,
$\widetilde{D}=\\{(\mathbf{x}^{l},\mathbf{\widetilde{q}}^{l}_{i^{*}}),l=1,\ldots,L\\}$,
$\bar{D}=\\{(\mathbf{x}^{l},\mathbf{\bar{q}}^{l}_{i^{*}}),l=1,\ldots,L\\}$ and
$D=\\{(\mathbf{x}^{l},\mathbf{q}^{l}_{i^{*}}),l=1,\ldots,L\\}$ be paired data
sets. Then, the unnormalized Lomb-Scargle periodogram value $P_{D}(f)$ for the
student output at angular frequency $f$ has the following bounds

$\frac{1}{2}{\left[\chi_{0}^{2}(D){-}\tau_{1}{+}L_{se}\right]}{\geq}P_{D}(f){\geq}\frac{1}{2}{\left[\chi_{0}^{2}(D){-}\tau_{2}{-}L_{se}\right]},$
(9)

where

$\displaystyle\tau_{1}={\chi}^{2}_{f}(\bar{D}),\ $
$\displaystyle\tau_{2}=\frac{1}{N^{2}}{\chi}^{2}_{f}(\hat{D})+\left(\frac{N-1}{N}\right)^{2}{\chi}^{2}_{f}(\widetilde{D}),$
$\displaystyle
L_{se}=\sum_{l=1}^{L}\left(\mathbf{\bar{q}}^{l}_{i^{*}}-\mathbf{q}^{l}_{i^{*}}\right)^{2}.$

###### Proof.

See Appendix A.1.

Theorem 1 provides several insights.

######  Remark 1.

When a student model is well trained by a teacher model, $L_{se}$ is generally
small.

######  Remark 2.

Consider the case where $f=f_{w}$. If we choose our sample $\widetilde{X}^{S}$
with high confidence output scores on the $i^{*}$-th class, for example by
filtering as described in Algorithm 1, ${\chi}^{2}_{f_{w}}(\hat{D})$ should be
small enough to be negligible by our watermark design in the teacher model. We
then discuss the following two cases.

Case I: When $N=1$, there is only one watermarked teacher to distill a student
model. Then, after neglecting ${\chi}^{2}_{f_{w}}(\hat{D})$, the left
inequality of Equation (9) becomes

$P_{D}(f_{w})\geq\frac{1}{2}\left[\chi_{0}^{2}(D)-L_{se}\right].$

This implies that we can observe a significant signal for the output of the
student model at frequency $f_{w}$ when the output of the student model is
close to that of the teacher model.

Case II: When $N\neq 1$, since there is no sinusoidal signal in
$\widetilde{\mathbf{q}}^{l}_{i^{*}}$, for $l=1,\ldots,L$, and the sinusoidal
signal in $\bar{\mathbf{q}}^{l}_{i^{*}}$, for $l=1,\ldots,L$ is, proportional
to $\frac{\varepsilon}{N}$, $\tau_{2}$ increases as $N$ increases. However, to
keep the watermark signal significant in the output of the student model, one
can increase the watermark signal amplitude $\varepsilon$ in the teacher model
$R_{1}$, which indirectly increases $\chi_{0}^{2}(D)$. This is due to the fact
that if $\varepsilon$ increases, $\chi_{0}^{2}(\hat{D})$ also increases. Since
$L_{se}$ is small when a student model is well trained by the teacher model,
$\chi_{0}^{2}(D)$ increases as well. This implies that we can detect the
watermark in the output of the student model at frequency $f_{w}$ by
increasing the watermark signal in the teacher model $R_{1}$ when $N$ is
large. We validate this observation in our experiments in Section 5.3.

######  Remark 3.

When $f\neq f_{w}$, since there is no sinusoidal signal at frequency $f$ in
$\hat{\mathbf{q}}^{l}_{i^{*}}$, $\widetilde{\mathbf{q}}^{l}_{i^{*}}$, and
$\bar{\mathbf{q}}^{l}_{i^{*}}$ for $l=1,\ldots,L$, $\chi^{2}_{f}(\hat{D})$,
$\chi^{2}_{f}(\widetilde{D})$ and $\chi^{2}_{f}(\bar{D})$ are generally large.
Thus, the values of both sides of the inequality in Equation (9) are small,
which implies that there is no sinusoidal signal for the output of the student
model at frequency $f\neq f_{w}$.

###  4.5 Multiple Watermarked Teacher Models

Consider a student model trained on the output of an ensemble model that
consists of two or more teacher models with watermarks. Can those watermarks
be detected in the student model?

We argue that it should be possible to extract each signal if the watermark
keys are different. The reason for this is that a signal embedded using
watermark key $K_{1}=(i_{1},f_{1},\mathbf{v}^{1})$ appears as noise for an
independent watermark $K_{2}=(i_{2},f_{2},\mathbf{v}^{2})$. Since noise has
low overall spectrum values, the resulting ensemble output spectrum will be
similar to an ensemble with only one watermarked model. Therefore, each signal
should be detectable using its respective key. This highlights the importance
that $\mathbf{v}$ should preferably be a high dimensional vector that can
provide more independent random choices for the watermark key $K$.

(a) Unwatermarked (b) Watermarked – matching projection (c) Watermarked – non-
matching projection

Figure 2:  A case study of the watermarking mechanism in CosWM. The black
vertical line indicates $f=30.0$. In each subgraph, the left plots the target
class output $\mathbf{q}_{i}(\mathbf{x})$ of the teacher model and the student
model as a function of projection value $p(\mathbf{x})$, and the right plots
the power spectrum value $P(f)$ for the output of the student model as a
function of frequency $f$.

##  5 Experiments

In this section, we evaluate the performance of CosWM on the model
watermarking task. We first describe the settings and data sets in Section
5.1. Then we present a case study to demonstrate the working process of CosWM
in Section 5.2. We compare the performance of all the methods in two scenarios
in Sections 5.3 and 5.4. We analyze the effect of the amplitude parameter
$\varepsilon$ and the signal frequency parameter $f_{w}$ on the performance of
CosWM in Appendices B.2 and B.3, respectively. We also analyze the effects of
using ground truth labels during distillation in Appendix B.5.

###  5.1 Experiment Settings and Data Sets

We compare CosWM with two state-of-the-art methods, DAWN (Szyller et al. 2019)
and Fingerprinting (Lukas, Zhang, and Kerschbaum 2019). We implement CosWM and
replicate DAWN in PyTorch 1.3. The Fingerprinting code is provided by the
authors of the corresponding paper (Lukas, Zhang, and Kerschbaum 2019) and is
implemented in Keras using a TensorFlow v2.1 backend. All the experiments are
conducted on Dell Alienware with Intel(R) Core(TM) i9-9980XE CPU, 128G memory,
NVIDIA 1080Ti, and Ubuntu 16.04.

We conduct all the experiments using two public data sets, FMNIST (Xiao,
Rasul, and Vollgraf 2017), and CIFAR10 (Krizhevsky 2009). We report the
experimental results on CIFAR10 in this section and the results on FMNIST in
Appendix B.1.

The CIFAR10 data set contains natural images in 10 classes. It consists of a
training set of 50,000 examples and a test set of 10,000 examples. We
partition all the training examples randomly into two halves, with use one
half for training the teacher models and the other half for distilling the
student models. For each data set the feature vectors are normalized to the
range $[0,1]$.

In all experiments, we use ResNet18 (He et al. 2016). All models are trained
or distilled for 100 epochs to guarantee convergence. The models with the best
testing accuracy during training/distillation are retained.

###  5.2 A Case Study

We conduct a case study to demonstrate the watermarking mechanism in CosWM. We
first train one watermarked teacher model and one non-watermarked teacher
model using the first half of the training data, and then distill one student
model from each teacher model using the second half of the training data. To
train the watermarked teacher model, we set the signal amplitude
$\varepsilon=0.05$ and the watermark key $K=(f_{w},i^{*},\mathbf{v}^{0})$ with
$f_{w}=30.0$, $i^{*}=0$ and $\mathbf{v}^{0}$ a unit random vector. For
extraction, we set $q_{min}$ to be the first quartile of all
$\mathbf{q}_{i^{*}}(\mathbf{x})$ values for 1,000 randomly selected training
examples whose ground truth is class $i^{*}$. Code for this case study is
available online
111https://developer.huaweicloud.com/develop/aigallery/notebook/detail?id=2d937a91-1692-4f88-94ca-82e1ae8d4d79.

We analyze the output of the teacher models and the student models for both
the time and frequency domains in Figure 2 for three cases. In Figures 2(a),
(b), and (c) for the three cases, we plot $\mathbf{q}_{i^{*}}(\mathbf{x})$ vs.
$p(\mathbf{x})$ in time domain for both the teacher model and the student
model in the left graph, and $P(f)$ vs. $f$ in the frequency domain for the
student model in the right graph.

In the first case, Figure 2(a) shows the results for the non-watermarked
teacher model and the student model. There is no sinusoidal signal in the
output for either the teacher model or the student model at frequency $f_{w}$
with projection vector $\mathbf{v}^{0}$.

In the second case, Figure 2(b) shows the results for the watermarked teacher
model and the student model. The accuracy loss of the watermarked teacher
model is within $1\%$ of the accuracy of the unwatermarked teacher model in
Figure 2(a). We extract the output of the watermarked teacher model and the
student model using the watermark key $K$. The output of the teacher follows
an almost perfect sinusoidal function and the output of the student model is
close to the output of the teacher model in the time domain. In the frequency
domain, the student model has a very prominent peak at frequency $f_{w}$. This
observation validates Remark 2 in Section 4.4 when $N=1$.

In the last case, we replace $\mathbf{v}^{0}$ by a different unit random
vector $\mathbf{v}^{1}$ in the watermark key $K$ to extract the output of the
watermarked teacher model and the student model. The results are shown in
Figure 2(c). The output of both the teacher model and the student model is
almost indiscernible from noise. Thus, there is no significant peak for the
output of the student model in the power spectrum at frequency $f_{w}$. This
observation validates Remark 3 in Section 4.4.

(a) Single Teacher (b) 2-model Ensemble (c) 4-model Ensemble (d) 8-model
Ensemble

Figure 3: mAP of CosWM, DAWN, and Fingerprinting under different parameter
values as a function of accuracy of the watermarked model. Each watermarked
model is part of an ensemble of teacher models and is the only watermarked
model within that ensemble.

(a) 2-model Ensemble (b) 4-model Ensemble (c) 6-model Ensemble (d) 8-model
Ensemble

Figure 4: mAP of CosWM, DAWN and Fingerprinting under different parameter
values as a function of accuracy of the watermarked model. Each watermarked
model is part of an ensemble of teacher models where every model is
watermarked.

###  5.3 Protection with a Single Watermarked Teacher

To compare CosWM with DAWN and Fingerprinting in protecting watermarked
teacher models, we set up a series of ranking tasks with different ensemble
size $N$. In each ranking task, we have $10$ student models distilled from the
watermarked teacher model (positive student models) and $100$ student models
not distilled from the watermarked teacher model (negative students). For
different methods, we use their own watermark signal strength values to rank
those $110$ students. Specifically, we use $P_{snr}$ defined in Equation (6)
for CosWM, the fraction of matching watermark predictions for DAWN, and the
fraction of matching fingerprint predictions for Fingerprinting. To evaluate
the performance of all three methods, we compute the average precision (AP)
for each ranking task and repeat each task for all $10$ watermarked models to
calculate the mean average precision (mAP) and its standard deviation.

For all three methods, we use the first half of the training data to train
$10$ unwatermarked teacher models with different initialization and $10$
teacher models with different watermark or fingerprint keys. We tune the
parameters to make sure that the accuracy losses of all watermarked teacher
models are within $1\%$ of the averaged accuracy of the unwatermarked teacher
models. To create a ranking task with $110$ student models, for every
watermarked teacher model we assemble it with $N-1$ randomly selected
unwatermarked teacher models to distill $10$ student models with different
initialization. In addition, we train $10$ independent student models with
ground truth labels and different initialization. The above process gives us
$10$ positive and $100$ negative student models for each watermarked teacher
model.

For CosWM, all watermarked teacher models have the same frequency $f_{w}=30.0$
and target class $i^{*}=0$, but have $10$ different unit random projection
vectors $\mathbf{v^{0}},\ldots,\mathbf{v^{9}}$. We set $q_{min}$ to the median
of all $\mathbf{q}_{i^{*}}$ values and vary the watermark amplitude
$\varepsilon$ in $0.025$, $0.05$, $0.1$, and $0.2$. For DAWN, we vary the
fraction of watermarked input $\tau$ in $0.0005$, $0.001$, $0.002$, and
$0.005$. For Fingerprinting, we generate one single set of fingerprint input
per teacher model using parameter $\varepsilon_{fp}=0.095$, which results in a
large enough set of fingerprints with the best conferrability score. During
extraction, all fingerprint input and labels are tested on a model to compute
the fingerprint strength value for ranking.

Figure 3 shows the results on the CIFAR10 data set for different ensemble size
values $N=1,2,4,8$. In this figure, we plot the mAP scores as a function of
the average teacher model accuracy. As a baseline, we add a Random method that
ranks all student models randomly, whose mAP and standard deviation is
represented by the horizontal red dashed line. The vertical purple dashed line
shows the average and standard deviation of the accuracy of the unwatermarked
teacher models.

As shown for both CosWM and DAWN in Figure 3, a stronger watermark will
negatively affect model performance. A model owner must consider this effect
when tuning the watermark.

When the ensemble size is small, i.e., $1$ and $2$, the best mAP of CosWM and
DAWN are generally comparable, and are both significantly larger than that of
Fingerprinting, as shown in Figures 3(a) and (b). When the ensemble size is
larger, i.e., $4$ and $8$, the best mAP of CosWM is significantly larger than
that of DAWN and Fingerprinting, whose watermarked model is consistently
outnumbered, as shown in Figures 3(c) and 3(d). This superior performance of
CosWM is due to our watermark signal design that is robust to ensemble
distillation. When the ensemble size increases, CosWM needs a larger
$\varepsilon$ to keep mAP high. This confirms the discussion in Remark 2 in
Section 4.4.

In addition, we observe a trade-off between ensemble size and mAP when
choosing different signal amplitude $\varepsilon$ for CosWM and different
fraction $\tau$ of watermarked input for DAWN. We analyze the effect of the
amplitude parameter $\varepsilon$ in more details in Appendix B.2.

###  5.4 Protection with Multiple Watermarked Teachers

We compare CosWM with DAWN and Fingerprinting in assembling only watermarked
teacher models to train a student model by undertaking another series of
ranking tasks for different ensemble sizes $N$. We train 10 watermarked
teacher models as described in Section 5.3, and assemble $10$ sets of teacher
models for each ensemble size in a round-robin manner. The training of all
other models and watermark settings in this experiment remain exactly the same
as described in Subsection 5.3. As a result, in an $N$-ensemble teacher model
experiment, each ranking task associated to a teacher model has $10N$ positive
and $110-10N$ negative student models.

Figure 4 shows the results on the CIFAR10 data set for different ensemble size
values, i.e., $N=2,4,6,8$. It is plotted in the same way as in Figure 3,
described in Section 5.3. Similar to the previous experiments, we also add the
Random baseline to provide a lower bound performance.

The accuracy losses of all watermarked models are within $1\%$ of the average
accuracy of all unwatermarked teacher models. When the ensemble size is small,
i.e., $N=2$, the best mAP of CosWM and DAWN are generally comparable to each
other, and are both significantly larger than that of Fingerprinting, as shown
in Figure 4(a). However, CosWM has a significantly higher best mAP for larger
ensemble sizes, i.e., $N=4,6,8$, as shown in Figures 4(b), (c) and (d). This
shows that CosWM watermarks are generally unaffected by other watermarks in a
teacher ensemble and confirms the possibility of detecting watermarks if the
ensemble features multiple watermarked teacher models as discussed in Section
4.5.

We also observe a similar trade-off between ensemble size and mAP when
choosing different signal amplitude $\varepsilon$ for CosWM and different
fraction $\tau$ of watermarked input for DAWN. This is further analyzed in
Appendix B.2.

##  6 Conclusion

In this paper, we tackle a novel problem of protecting neural network models
against ensemble distillation. We propose CosWM, an effective method relying
on a signal embedded into all output of a watermarked model, and therefore
transferring the signal to training data for student models. We prove that the
embedded signal in CosWM is strong in a well-trained student model by
providing lower and upper bounds on the watermark strength metric. In
addition, CosWM can be extended to identify student models distilled from an
ensemble featuring multiple watermarked models. Our extensive experiments
demonstrate the superior performance of CosWM in providing models defense from
ensemble distillation.

## References

  * Adi et al. (2018) Adi, Y.; Baum, C.; Cisse, M.; Pinkas, B.; and Keshet, J. 2018.  Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring.  _arXiv preprint arXiv:1802.04633_.
  * Ba and Caruana (2014) Ba, J.; and Caruana, R. 2014.  Do Deep Nets Really Need to be Deep?  In _Advances in Neural Information Processing Systems_ , volume 27.
  * Bishop (2006) Bishop, C. M. 2006.  _Pattern Recognition and Machine Learning (Information Science and Statistics)_.  Springer New York.
  * Buciluǎ, Caruana, and Niculescu-Mizil (2006) Buciluǎ, C.; Caruana, R.; and Niculescu-Mizil, A. 2006.  Model Compression.  In _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 535–541.
  * Bühlmann and Yu (2002) Bühlmann, P.; and Yu, B. 2002.  Analyzing bagging.  _The Annals of Statistics_ , 30(4): 927 – 961.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the Knowledge in a Neural Network.  _arXiv preprint arXiv:1503.02531_.
  * Jagielski et al. (2019) Jagielski, M.; Carlini, N.; Berthelot, D.; Kurakin, A.; and Papernot, N. 2019.  High Accuracy and High Fidelity Extraction of Neural Networks.  _arXiv preprint arXiv:1909.01838_.
  * Jia et al. (2021) Jia, H.; Choquette-Choo, C. A.; Chandrasekaran, V.; and Papernot, N. 2021.  Entangled Watermarks as a Defense against Model Extraction.  _arXiv preprint arXiv:2002.12200_.
  * Juuti et al. (2019) Juuti, M.; Szyller, S.; Marchal, S.; and Asokan, N. 2019.  PRADA: Protecting Against DNN Model Stealing Attacks.  _IEEE European Symposium on Security and Privacy_.
  * Krizhevsky (2009) Krizhevsky, A. 2009.  Learning multiple layers of features from tiny images.  Technical report.
  * Kullback and Leibler (1951) Kullback, S.; and Leibler, R. A. 1951.  On Information and Sufficiency.  _The Annals of Mathematical Statistics_ , 22(1): 79 – 86.
  * Le Merrer, Pérez, and Trédan (2019) Le Merrer, E.; Pérez, P.; and Trédan, G. 2019.  Adversarial frontier stitching for remote neural network watermarking.  _Neural Computing and Applications_ , 32(13): 9233–9244.
  * Lukas, Zhang, and Kerschbaum (2019) Lukas, N.; Zhang, Y.; and Kerschbaum, F. 2019.  Deep Neural Network Fingerprinting by Conferrable Adversarial Examples.  _arXiv preprint arXiv:1912.00888_.
  * Orekondy, Schiele, and Fritz (2019) Orekondy, T.; Schiele, B.; and Fritz, M. 2019.  Knockoff Nets: Stealing Functionality of Black-Box Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Papernot et al. (2017) Papernot, N.; McDaniel, P.; Goodfellow, I.; Jha, S.; Celik, Z. B.; and Swami, A. 2017.  Practical Black-Box Attacks against Machine Learning.  In _Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security_ , 506–519.
  * Ribeiro, Grolinger, and Capretz (2015) Ribeiro, M.; Grolinger, K.; and Capretz, M. A. M. 2015.  MLaaS: Machine Learning as a Service.  In _IEEE 14th International Conference on Machine Learning and Applications (ICMLA)_ , 896–902.
  * Rouhani, Chen, and Koushanfar (2018) Rouhani, B. D.; Chen, H.; and Koushanfar, F. 2018.  DeepSigns: A Generic Watermarking Framework for IP Protection of Deep Learning Models.  _arXiv preprint arXiv:1804.00750_.
  * Scargle (1982) Scargle, J. D. 1982.  Studies in astronomical time series analysis. II-Statistical aspects of spectral analysis of unevenly spaced data.  _The Astrophysical Journal_ , 263: 835–853.
  * Shen and Savvides (2020) Shen, Z.; and Savvides, M. 2020.  MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks.  _arXiv preprint arXiv:2009.08453_.
  * Szyller et al. (2019) Szyller, S.; Atli, B. G.; Marchal, S.; and Asokan, N. 2019.  DAWN: Dynamic Adversarial Watermarking of Neural Networks.  _arXiv preprint arXiv:1906.00830_.
  * Tramèr et al. (2016) Tramèr, F.; Zhang, F.; Juels, A.; Reiter, M. K.; and Ristenpart, T. 2016.  Stealing Machine Learning Models via Prediction APIs.  _arXiv preprint arXiv:1609.02943_.
  * Uchida et al. (2017) Uchida, Y.; Nagai, Y.; Sakazawa, S.; and Satoh, S. 2017.  Embedding Watermarks into Deep Neural Networks.  In _Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval_ , 269–277.
  * Vongkulbhisal, Vinayavekhin, and Visentini-Scarzanella (2019) Vongkulbhisal, J.; Vinayavekhin, P.; and Visentini-Scarzanella, M. 2019.  Unifying heterogeneous classifiers with distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3175–3184.
  * Xiao, Rasul, and Vollgraf (2017) Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms.  _arXiv preprint arXiv:1708.07747_.
  * Zhang et al. (2018) Zhang, J.; Gu, Z.; Jang, J.; Wu, H.; Stoecklin, M. P.; Huang, H.; and Molloy, I. 2018.  Protecting Intellectual Property of Deep Neural Networks with Watermarking.  In _Proceedings of the 2018 on Asia Conference on Computer and Communications Security_ , 159–172.

Appendix

In this appendix, we provide the proofs for Theorem 1 in Subsection A.1 and a
lemma on properties of the modified softmax outputs in Subsection A.2. In
addition, we show more extensive experimental results in Subsection B.

##  Appendix A Proofs

###  A.1 Proof of Theorem 1

###### Proof.

We first prove the left inequality of (9). By using the triangular inequality
and the fact that $\mathbf{s}^{*}(\bar{D})$ is the best sinusoidal fit for
$\bar{D}$, we have

$\displaystyle\chi^{2}_{f}(D)$
$\displaystyle\geq\sum_{l=1}^{L}\left[\mathbf{\bar{q}}^{l}_{i^{*}}-[\mathbf{s}^{*}(D)]_{l}\right]^{2}-\sum_{l=1}^{L}\left[\mathbf{\bar{q}}^{l}_{i^{*}}-\mathbf{q}^{l}_{i^{*}}\right]^{2}$
(10)
$\displaystyle\geq\sum_{l=1}^{L}\left[\mathbf{\bar{q}}^{l}_{i^{*}}-[\mathbf{s}^{*}(\bar{D})]_{l}\right]^{2}-L_{SE}$
(11) $\displaystyle={\chi}^{2}_{f}(\bar{D})-L_{SE}.$ (12)

Combining the above inequality with equation (8) we obtain the left inequality
of (9).

Next we prove the right inequality of (9). Since $\mathbf{s}^{*}(\hat{D})$ and
$\mathbf{s}^{*}(\widetilde{D})$ are points from the graph of sinusoidal
functions with same frequency $f$, then
$\frac{1}{N}\mathbf{s}^{*}(\hat{D})+\frac{N-1}{N}\mathbf{s}^{*}(\widetilde{D})$
is also a set of points from the graph of a sinusoidal function. As
$\mathbf{s}^{*}(D)$ is the best sinusoidal fit for $D$, we have

$\displaystyle\chi^{2}_{f}$
$\displaystyle(D)=\sum_{l=1}^{L}\left[\mathbf{q}^{l}_{i^{*}}-[{\mathbf{s}}^{*}(D)]_{l}\right]^{2}$
(13)
$\displaystyle\leq\sum_{l=1}^{L}\left[\mathbf{q}^{l}_{i^{*}}-\left[\frac{1}{N}\mathbf{s}^{*}(\hat{D})+\frac{N-1}{N}\mathbf{s}^{*}(\widetilde{D})\right]_{l}\right]^{2}.$
(14)

By using the above inequality and triangular inequality, we get the following:

$\displaystyle\chi^{2}_{f}(D)\leq\sum_{l=1}^{L}\left[\mathbf{q}^{l}_{i^{*}}{-}\frac{1}{N}[\mathbf{s}^{*}(\hat{D})]_{l}{-}\frac{N-1}{N}[\mathbf{s}^{*}(\widetilde{D})]_{l}\right]^{2}$
(15)
$\displaystyle\leq\sum_{l=1}^{L}\left[\mathbf{q}^{l}_{i^{*}}-\frac{1}{N}\hat{\mathbf{q}}^{l}_{i^{*}}-\frac{N-1}{N}\mathbf{\widetilde{q}}^{l}_{i^{*}}\right]^{2}+$
(16)
$\displaystyle\quad{\sum_{l=1}^{L}}\left[\frac{1}{N}\hat{\mathbf{q}}^{l}_{i^{*}}{+}\frac{N-1}{N}\mathbf{\widetilde{q}}^{l}_{i^{*}}{-}\frac{1}{N}[\mathbf{s}^{*}(\hat{D})]_{l}{-}\frac{N-1}{N}[\mathbf{s}^{*}(\widetilde{D})]_{l}\right]^{2}$
$\displaystyle\leq L_{SE}+$ (17)
$\displaystyle\sum_{l=1}^{L}\left[\frac{1}{N}\left[\hat{\mathbf{q}}^{l}_{i^{*}}-[\mathbf{s}^{*}(\hat{D})]_{l}\right]+\frac{N-1}{N}\left[\mathbf{\widetilde{q}}^{l}_{i^{*}}-[\mathbf{s}^{*}(\widetilde{D})]_{l}\right]\right]^{2}$
$\displaystyle\leq
L_{SE}+\frac{1}{N^{2}}{\chi}^{2}_{f}(\hat{D})+\left(\frac{N-1}{N}\right)^{2}{\chi}^{2}_{f}((\widetilde{D}).$
(18)

Combining the above inequality with equation (8) we obtain the right
inequality of (9).

∎

###  A.2 Lemma 1: Modified Softmax Properties

######  Lemma 1.

Let $\mathbf{q}$ be the softmax output of a model $R$, then the modified
softmax $\hat{\mathbf{q}}$, as defined in (4) satisfies Property 1.

###### Proof.

By the definition of softmax (1), for all $i$ we have

$\displaystyle 0\leq\mathbf{q}_{i}\leq 1,$ (19)
$\displaystyle-1\leq\mathbf{a}_{i}(\mathbf{x})\leq 1.$ (20)

Therefore, when $i=i^{*}$, we have

$0\leq\mathbf{q}_{i}+\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))\leq
1+2\varepsilon,$ (21)

and then

$0\leq\frac{\mathbf{q}_{i}+\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))}{1+2\varepsilon}\leq
1.$ (22)

When $i\neq i^{*}$, since $m\geq 2$, we have

$0\leq\mathbf{q}_{i}+\frac{\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))}{m-1}\leq
1+\frac{2\varepsilon}{m-1}\leq 1+2\varepsilon,$ (23)

and then

$0\leq\frac{\mathbf{q}_{i}+\frac{\varepsilon(1+\mathbf{a}_{i}(\mathbf{x}))}{m-1}}{1+2\varepsilon}\leq
1.$ (24)

Thus, $\hat{\mathbf{q}}$ satisfies clause 1 of Property 1.

To prove clause 2 of Property 1, we use the fact that
$\mathbf{a}_{i^{*}}+\mathbf{a}_{j\neq i^{*}}=0$ and obtain

$\displaystyle\sum_{i}^{m}\hat{\mathbf{q}}_{i}=\frac{\mathbf{q}_{i^{*}}+\varepsilon(1+\mathbf{a}_{i^{*}})}{1+2\varepsilon}+\sum_{j\neq
i^{*}}\frac{\mathbf{q}_{j}+\frac{\varepsilon(1+\mathbf{a}_{j})}{m-1}}{1+2\varepsilon}$
(25)
$\displaystyle=\left(\sum_{j=1}^{m}\frac{\mathbf{q}_{i}}{1+2\varepsilon}\right)+\left(\sum_{j\neq
i^{*}}\frac{\varepsilon(1+\mathbf{a}_{i^{*}}+1+\mathbf{a}_{j})}{(m-1)(1+2\varepsilon)}\right)$
(26)
$\displaystyle=\frac{1}{1+2\varepsilon}+\frac{2\varepsilon}{1+2\varepsilon}$
(27) $\displaystyle=1.$ (28)

Thus, clause 2 of Property 1 is also satisfied. ∎

##  Appendix B More Experiment Results

###  B.1 FMNIST Results

We include here the results of the experiments described in Sections 5.3 and
5.4 for models trained on the FMNIST data set. The FMNIST data set contains
fashion item images in 10 classes. It consists of a training set of 60,000
examples and a test set of 10,000 examples. We partition all the training
examples randomly into a teacher half and a student half just as we did with
CIFAR10.

Figure 5 shows the result of the single watermark experiment described in
section 5.3 using the FMNIST data set. The figure plots the mAP of the
different watermarks against the test accuracy of the watermarked model. All
of the experiment parameters and baselines are the same as shown in the
equivalent CIFAR10 results shown in Figure 3.

The accuracy losses of all the watermarked models are on average negligible
when compared to equivalent unwatermarked models. Similar to the CIFAR10
results, the mAP of CosWM and DAWN watermarks are comparable for smaller
ensemble sizes, as shown in Figures 3(a) and (b), and the CosWM watermarks are
significantly stronger for larger ensembles, as shown in Figures 3(c) and (d).
This solidifies our claim that the design of CosWM makes it the most robust
method against ensemble distillation.

(a) Single Teacher (b) 2-model Ensemble (c) 4-model Ensemble (d) 8-model
Ensemble

Figure 5: mAP of CosWM, DAWN, and Fingerprinting under different parameter
values as a function of accuracy of the watermarked model trained on the
FMNIST data set. Each watermarked model is part of an ensemble of teacher
models and is the only watermarked model within that ensemble.

(a) 2-model Ensemble (b) 4-model Ensemble (c) 6-model Ensemble (d) 8-model
Ensemble

Figure 6: mAP of CosWM, DAWN and Fingerprinting under different parameter
values as a function of accuracy of the watermarked model trained on the
FMNIST data set. Each watermarked model is part of an ensemble of teacher
models where every model is watermarked.

Figure 6 shows the result of the multiple watermark experiment described in
section 5.4 on the FMNIST data set. Once more, the figure plots mAP against
watermarked model test accuracy for many watermark methods. All of the
experiment parameters and baselines are the same as shown in the equivalent
CIFAR10 results shown in Figure 4.

The accuracy losses of all the watermarked models are still negligible
compared to unwatermarked models. Similar to the CIFAR10 results and the
single model experiments The watermark produced by CosWM is significantly more
robust than DAWN or Fingerprinting when used with larger ensemble sizes of
$N=4,6,8$. This solidifies our claims that CosWM has the ability to discern
each distinct watermark featured in an ensemble from a student model.

###  B.2 Amplitude Parameter Analysis

To demonstrate the flexibility and tuning capabilities of CosWM, we analyze
the effect of the watermark parameters to the identification performance. We
start with the amplitude parameter $\varepsilon$. In general, because a larger
amplitude signal results in a larger $P_{snr}$ value, a watermark signal with
a larger amplitude should make it more likely to be successfully extracted.
Therefore we expect watermarks with higher $\varepsilon$ to have a higher mAP.

We can observe from Figures 3-6 that this is generally the case for both CosWM
and DAWN. A stronger signal usually infers higher mAP scores, at the cost of
lower watermarked model performance. Therefore, while setting the amplitude of
the watermark signal, one must consider striking a balance between model
performance and watermark performance. A similar balancing act exists when
setting the ratio of trigger inputs $\tau$ in a DAWN watermark.

Figure 7: Watermark strength $P_{snr}$ as a function of signal frequency
$f_{w}$ for different ensemble sizes. Each ensemble contains only one
watermarked model.

(a) Single Teacher (b) 2-model Ensemble (c) 4-model Ensemble (d) 8-model
Ensemble

Figure 8: mAP of CosWM under different parameter values as a function of
accuracy of the watermarked model. All models are trained on the CIFAR10 data
set. Each watermarked model is part of an ensemble of teacher models and is
the only watermarked model within that ensemble. Students are distilled with a
combination of KL loss and cross entropy loss.

###  B.3 Signal Frequency Parameter Analysis

As a demonstration of CosWM’s versatility, we show that the method can be
effective at extracting the signal function for a wide range of frequencies
$f_{w}$.

For the different values of $f_{w}$ of $0.01$, $1.0$, $30.0$, $100.0$,
$10^{4}$, $10^{6}$, $10^{8}$, we train 5 watermarked teacher models and
distill 10 student models per an ensemble which includes only one watermarked
teacher. We use parameter values $\varepsilon=0.2$, and five randomly
generated unit projection vectors $\mathbf{v}^{0},...,\mathbf{v}^{4}$. For
each student model, we compute the $P_{snr}$ value of the matching teacher
watermark at the frequency $f_{w}$ and compare its value as we change $f_{w}$.
We also compute the standard deviation of $P_{snr}$ to use as a confidence
interval. We repeat the experiment for ensemble sizes $N=2,4,8$, where each
model ensemble assembles only one watermarked teacher model with randomly
generated unwatermarked teacher models as described in Subsection 5.3.

Figure 7 shows the results of this analysis. We clearly see that CosWM
performs very well for a very wide range of $f_{w}$ values, but becomes less
effective if $f_{w}$ is too small or too large. Frequency values of
$f_{w}=30,100,10^{4},10^{6}$ all result in $P_{snr}$ values generally above
$5.0$ for $N=1,2,4$. From our previous experiments under similar settings in
Subsection 5.3, such high values will always result in an mAP of exactly or
extremely close to $1.0$. Even with $N=8$, we observe a noticeable increase in
$P_{snr}$ between $f_{w}=30$ and $10^{4}$, which will increase mAP to
relatively high scores, compared to another method like DAWN or
Fingerprinting, as shown in Figures 3(d) and 5(d). This shows that on top of
having a variety of choice for the projection vector $\mathbf{v}$, CosWM also
offers a large range in the choice of frequency $f_{w}$.

###  B.4 Selecting Extraction Parameters

Here we explain how to properly select the extraction parameters $q_{min}$ and
$\delta$ in the extraction algorithm described in Subsection 4.3.

The value of $q_{min}$ should be selected so that we only keep points inside
the range of the signal function. However, this can be difficult to determine
systematically, as the amplitude of the signal in the student will decrease
when the ensemble size increases. To remedy this, we choose $q_{min}$ to be
proportional to the median or the first quartile of the output values, which
allows us to determine its value automatically. From our experience, the first
quartile value works well in the single teacher case, while the median value
better generalizes for larger ensembles.

In applications where few data points have high confidence outputs, such as
data sets with many classes, one may instead filter outputs below a set
$q_{max}$ threshold. The idea is to substitute high confidence outputs that
would be close to $1$, as in Figure 1(a) for an unwatermarked model with low
confidence outputs that would be close to $0$. Applying the perturbation in
either case should result in a cosine signal with a clearer peak in its power
spectrum.

The value of $\delta$ should be chosen as to only contain a significant part
of the power spectrum peak. In numerical computations, frequency values are
determined by an uniformly spaced sample, and choosing $\delta$ is equivalent
to choosing a fixed number of closest frequencies to $f_{w}$. Peaks can vary
in width depending on the experiment, but choosing a narrower window will
generally have little impact on the computed value of $P_{snr}$. We have found
that a sufficient window can contain only a few (less than 10) frequency
samples to generalize well across all experiments.

###  B.5 Combination of KL loss and cross entropy loss

In this subsection, we conduct experiments on CosWM to extract watermarks from
student models distilled with a combination of KL loss and cross entropy loss,
in a similar way to several common distillation processes such as (Ba and
Caruana 2014; Buciluǎ, Caruana, and Niculescu-Mizil 2006; Hinton, Vinyals, and
Dean 2015). We use the same training settings as the CIFAR10 experiments
described in Subsection 5.3. In each task, we distill 100 student models with
an equally weighted combination of KL loss from the ensemble’s outputs and
cross entropy loss from the ground truth labels. We set $q_{min}$ to the
median of the $\mathbf{q}_{i^{*}}$ values of the sampled inputs with ground
truth label $i^{*}$, and vary the watermark amplitude $\varepsilon$ in
$0.025$, $0.05$, $0.1$, $0.2$, and $0.4$.

Figure 8 shows the results on the CIFAR10 data set for different ensemble size
values, i.e., $N=1,2,4,8$. Results are plotted in the same way as in Figure 3,
shown in Subsection 5.3. Similar to the previous experiments, we also add a
Random baseline to provide a lower bound performance for all the methods.

The accuracy losses of all watermarked models are within $1\%$ of the average
accuracy of all unwatermarked teacher models except when $\varepsilon=0.4$.
CosWM has very high mAP for ensemble sizes $N=1,2,4$, as shown in Figure 8(a),
(b), and (c). However, for the ensemble size $N=8$, CosWM needs to sacrifice
more accuracy to achieve high mAP as shown in Figure 8(d). The major reason
for this observation is that the watermark signal has been diluted by the
cross-entropy loss since the student models are distilled with an equally
weighted combination of KL loss and cross entropy loss.