# HARD: Hard Augmentations for Robust Distillation

Arne F. Nix1-2,*, Max F. Burg2-3, Fabian H. Sinz1-2, **
1 Institute for Bioinformatics and Medical Informatics, University of Tübingen
2 Institute for Computer Science and Campus Institute Data Science, University
of Göttingen
3 Institute for Theoretical Physics, University of Tübingen

<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) is a simple and successful method to transfer
knowledge from a teacher to a student model solely based on functional
activity. However, current KD has a few shortcomings: it has recently been
shown that this method is unsuitable to transfer simple inductive biases like
shift equivariance, struggles to transfer out of domain generalization, and
optimization time is magnitudes longer compared to default non-KD model
training. To improve these aspects of KD, we propose Hard Augmentations for
Robust Distillation (HARD), a generally applicable data augmentation
framework, that generates synthetic data points for which the teacher and the
student disagree. We show in a simple toy example that our augmentation
framework solves the problem of transferring simple equivariances with KD. We
then apply our framework in real-world tasks for a variety of augmentation
models, ranging from simple spatial transformations to unconstrained image
manipulations with a pretrained variational autoencoder. We find that our
learned augmentations significantly improve KD performance on in-domain and
out-of-domain evaluation. Moreover, our method outperforms even state-of-the-
art data augmentations and since the augmented training inputs can be
visualized, they offer a qualitative insight into the properties that are
transferred from the teacher to the student. Thus HARD represents a generally
applicable, dynamically optimized data augmentation technique tailored to
improve the generalization and convergence speed of models trained with
KD.111Code available at https://github.com/sinzlab/HARD

##  1 Introduction

Knowledge distillation (KD) methods [27, 37, 60] are powerful and flexible
tools to transfer the knowledge of a given _teacher_ model to the transfer
target, the _student_ model, without copying the weights. Instead, these
methods match the student’s functional activity (e.g. the softmax output) to
that of the teacher for the presented inputs. Hence, those methods are
independent of architectural details and allow knowledge distillation to be
applied in scenarios like model compression [7, 27], continual learning [42,
52, 4], or even neuroscience [35], where traditional transfer learning would
be impossible to use. KD methods also appear to be key to training new models
that trade off inductive biases for more flexibility and more parameters [55,
17, 53] on smaller data [54, 9, 40]. However, Nix et al. [40] recently showed
that current KD methods fail to transfer even simple equivariances between
teacher and student. Additionally, previous work showed that KD leads to a
larger gap between student and teacher on out-of-domain evaluation performance
compared to within domain performance [41, 6], even in cases where the student
almost perfectly matches the teacher [6] (see Table 5). This phenomenon is
especially pronounced for particularly robust teachers [41]. Thus we expect
that transferring robustness properties is a difficult problem for KD in
general.

We hypothesize that KD methods are in principle capable of transferring most
knowledge from a teacher to a student if the training data is chosen
adequately. We confirm this hypothesis on a small toy example (Section 3),
showing the importance of input data for KD. Motivated by this demonstration,
we propose our _Hard Augmentations for Robust Distillation (HARD)_ method, a
general framework (Section 4) to generate augmented training inputs which
improve knowledge transfer by maximizing the distance between teacher and
student while leaving the teacher’s output unchanged. Consequently, our
framework moves the input in directions that the teacher is invariant to but
which are most challenging for the student. Our experiments (Section 5) show
that our task-agnostic framework improves transfer effectiveness and thereby
solves the problem of of KD not being able to transfer shift equivariance
[40]. Additionally, as part of our framework, we propose several parameterized
augmentations (Section 4.1) that can be integrated with most existing KD
methods and are applicable to a variety of different computer vision tasks.
Finally, we demonstrate across multiple different models on the tasks of
CIFAR10 and ImageNet that our framework learns interpretable augmentations
that improve KD to the same level and in many cases even beyond established
data augmentation methods, even when evaluated in an out-of-domain setting.

##  2 Related Work

Figure 1: Our task-agnostic HARD framework switches between training the
student to match the teacher and training the augmentor to generate new
samples on which the student underperforms while maintaining high teacher
performance. We optimize the augmentor and student in interchanging phases
through a student-teacher loss
$\mathcal{L}_{\tilde{\mathrm{s}}\leftrightarrow\tilde{\mathrm{t}}}$ and
teacher-teacher loss
$\mathcal{L}_{\tilde{\mathrm{t}}\leftrightarrow\mathrm{t}}$. We switch between
the two phases by comparing the default loss
$\mathcal{L}_{\tilde{\mathrm{s}}}$ on augmented data to pre-defined
thresholds.

There is a long tradition in using data augmentations to artificially extend
training data for deep learning models and particularly in computer vision, be
it through adding Gaussian noise, random crops, shifts, flips, or rotations
[33, 18]. In recent years, data augmentations became more complex [24, 39, 12,
61, 59, 28], employing a multitude of different heuristics with the aim to
improve generalization and in some cases also out-of-domain performance [24].
A particularly popular augmentation method is _Mixup_ [61], which randomly
interpolates two input samples and their labels respectively. Similarly,
_Cutmix_ [59] combines two input images by pasting a random crop of one image
on top of the other. Also, many studies use parameterized augmentations
optimized to improve a given objective [11, 48, 25, 67], and some even
optimize the augmentations to improve on an adversarial objective [56, 3, 63,
64, 65, 20, 2], however, without applying them for knowledge transfer.

In KD, applying data augmentations is a very effective tool to improve
matching between student and teacher [58, 6] and optimizing on a meta level
can be useful to aide the teaching [43]. Similar to our work, Rashid et al.
[45], Haidar et al. [21], Zhang et al. [62] utilized adversarial objectives to
optimize data augmentations for KD, however, they were solely focused on
natural language processing tasks and do not optimize the augmentations
towards invariance.

Inspired by this large body of work we formulate a task-agnostic framework
containing only one building block that is specific to the data-domain – the
instantiation of the augmentor model generating the augmented data samples –
for which we offer a variety of reasonable model choices based on spatial
transformer modules [29], Mixup [61], and variational autoencoders [31, 10,
34].

##  3 Input Data Matters for Functional Transfer

x$\displaystyle{-2}$$\displaystyle{0}$$\displaystyle{2}$yMSE:
5.08ABaselineMSE: 3.24BMix-UpMSE: 1.96CGaussian NoiseMSE:
0.10DHARDTeacherStudentTrain
Data$\displaystyle{-10}$$\displaystyle{0}$$\displaystyle{10}$$\displaystyle{0.0}$$\displaystyle{0.2}$$\displaystyle{0.4}$DensityE$\displaystyle{-10}$$\displaystyle{0}$$\displaystyle{10}$F$\displaystyle{-10}$$\displaystyle{0}$$\displaystyle{10}$G$\displaystyle{-10}$$\displaystyle{0}$$\displaystyle{10}$H
Figure 2: Fitting the student, a three-layer ReLU MLP, to the teacher
function, $\cos(x)$, for $10,000$ iterations. We show results for 10 random
seeds (A-D) and the distribution of (augmented) training inputs as a
normalized histogram (E-H). We compare baseline (no augmentations) with Mixup,
Gaussian noise and an HARD-optimized noise distribution. We report mean-
squared-error (MSE) on 100 test inputs sampled from $\mathcal{U}_{[-10,10]}$.

We hypothesize that the choice of input data is crucial to successfully
knowledge distillation and we illustrate the impact of training data by a
simple toy example. To demonstrate this, consider a simple KD task in which we
instantiate the teacher-model by the true function $f_{\mathrm{t}}(x)=\cos(x)$
and the student $f_{\mathrm{s}}(x)$ by a three layer Multilayer Perceptron
(MLP) with ReLU activation [1]. We use input data $x$ chosen such that it does
not capture the teacher’s $\cos(x)$ periodicity (orange points in Figure 2A).
Simple KD does neither interpolate between the given training points nor
extrapolates beyond them (Figure 2E). Hence the student neural network does
not learn the teacher’s periodicity and fails to interpolate and extrapolate
beyond the training data (Figure 2A).

Augmenting the training data with more helpful inputs $\tilde{x}$ and teacher
labels $f_{\mathrm{t}}(\tilde{x})=\cos(\tilde{x})$ could mitigate this
problem. One method successfully applied to KD [6] is to extend the input data
through Mixup [61]. When applying this to our illustrative example, we create
new training inputs $\tilde{x}$ through linear interpolation between pairs of
input points $\tilde{x}=(1-\alpha)x_{1}+\alpha x_{2}$ (Figure 2F), and
recording the corresponding teacher responses
$f_{\mathrm{t}}(\tilde{x})=\cos(\tilde{x})$. Thus, the student learns to
interpolate between training points, but mixup does not enhance extrapolation
(Figure 2B).

To generate datapoints that would interpolate and extrapolate beyond already
available training points, we could simply augment by adding Gaussian noise
$\epsilon$ to the available data points, $\tilde{x}=x+\epsilon$, hence
interpolating and extrapolating beyond the training data (Figure 2G). This
strategy helps our student to match the teacher also outside the original
training regime (Figure 2C). However, the student only improves within a fixed
margin that is determined by the noise distribution’s mean and variance.

We could obviously improve interpolation and extrapolation by increasing the
noise distribution’s variance or shifting it’s mean, however, as we move to a
high dimensional image input space
($x\in\mathbb{R}\rightarrow\vec{x}\in\mathbb{R}^{N}$) it becomes unclear how
to heuristically select helpful new samples and at the same time random
exploration strategies become computationally infeasible. Instead, we propose
to optimize a parameterized augmentation to efficiently generate new, hard
training samples on which the student lacks performance, as here the student
could improve the most. In our toy example, we illustrate this by optimizing
the Gaussian’s parameters (mean and variance) according to our augmentation
framework _HARD_ , which we will present in the next section. This provides us
with a noise distribution which we use to draw new helpful training examples
$\tilde{x}$ that transfer inter- and extrapolation to the student network
(Figure 2D,H). Overall, this toy example shows that learning hard
augmentations to select new helpful data points is crucial to efficiently
improve extrapolation beyond the training distribution.

##  4 Learning Hard Augmentations for Robust Distillation (HARD)

Our task-agnostic HARD framework learns augmenting training images to most
efficiently help knowledge-transfer from a teacher to a student model. Our
method requires three main components: a teacher model with frozen parameters,
a student model that should learn knowledge from the teacher, and a
parameterized augmentation model that learns to augment images such that most
of the teacher’s knowledge is transferred to the student.

In classical KD methods[27], the objective is to minimize a distance
$\mathcal{D}\left[f_{\mathrm{s}}(x),f_{\mathrm{t}}(x)\right]$ between the
student’s activation $f_{\mathrm{s}}(x)$ and the teacher’s activation
$f_{\mathrm{t}}(x)$ on given inputs $x\in\mathbb{R}^{n}$. Usually, this would
be the Kullback-Leibler divergence between the softmax distributions of
teacher and student. Unfortunately, only considering training data could miss
properties of the teacher (eg. shift invariance) that might be crucial for
generalization (see Section 3 for an illustrative example). To resolve this
issue, we learn a parametrized augmentation model $g_{\mathrm{a}}$ to generate
new input data points $\tilde{x}=g_{\mathrm{a}}(x)$ transferring such
invariance properties from the teacher to the student. Hence, we define a
teacher-student loss considering the more general case of matching student and
teacher on augmented inputs $\tilde{x}\in\mathbb{R}^{n}$:

$\displaystyle\mathcal{L}_{\tilde{\mathrm{s}}\leftrightarrow\tilde{\mathrm{t}}}=\mathcal{D}\left[f_{\mathrm{s}}(\tilde{x}),f_{\mathrm{t}}(\tilde{x})\right]\
.$ (1)

To specifically transfer the teacher’s invariance properties to the student,
we propose a teacher-teacher loss pushing the augmentor towards generating
data points on which the teacher is invariant,

$\displaystyle\mathcal{L}_{\tilde{\mathrm{t}}\leftrightarrow\mathrm{t}}=\mathcal{D}\left[f_{\mathrm{t}}(\tilde{x}),f_{\mathrm{t}}(x)\right]\
,$ (2)

as these are often useful augmentations for generalization. Using both of
these losses, we optimize the augmentor’s parameters $\theta_{\mathrm{a}}$ to
generate augmented samples on which the teacher results in similar activations
but the student differs from them (Figure 1 top) and simultaneously we
optimize the student’s parameters $\theta_{\mathrm{s}}$ to perform well on
those augmentations (Figure 1 bottom):

$\displaystyle\max_{\theta_{\mathrm{a}}}\
\lambda_{\mathrm{s}}\mathcal{L}_{\tilde{\mathrm{s}}\leftrightarrow\tilde{\mathrm{t}}}-\lambda_{\mathrm{t}}\mathcal{L}_{\tilde{\mathrm{t}}\leftrightarrow\mathrm{t}}\text{\quad\quad
and \quad\quad}\min_{\theta_{\mathrm{s}}}\
\mathcal{L}_{\tilde{\mathrm{s}}\leftrightarrow\tilde{\mathrm{t}}}\ .$ (3)

Here, $\lambda_{\mathrm{s}}$ and $\lambda_{\mathrm{t}}$ trade off the loss
terms and are treated as hyper-parameters. We train both components separately
switching from training the augmentor to training the student when the
student’s performance on augmented data gets worse than a pre-defined
threshold ($\mathcal{L}_{\tilde{\mathrm{s}}}>\ell_{\max}$) and we switch back
from student to augmentor training when the student’s performance on augmented
data surpasses a pre-defined threshold
($\mathcal{L}_{\tilde{\mathrm{s}}}<\ell_{\min}$; Figure 1). To prevent
catastrophic forgetting, we save augmentors at every switch and employ an
augmentor randomly chosen out of the set of previously saved augmentors in
each iteration when training the student.

###  4.1 The augmentor models

To generate new input data points it is important to choose an augmentor that
suits the desired application and is powerful enough to generate useful
augmentations. Usually, we do not know a priori what useful augmentations are
and thus should try to allow as much flexibility as possible. Additionally,
some variance over augmentations could benefit the transfer. Thus, all
augmentors in our study introduce randomness in the model by adding Gaussian
noise into the computation of the augmentation through the reparametrization
trick [31]. While our framework is universally applicable across domains,
choosing an effective augmentation model likely needs to be addressed for each
task individually. In our experiments, we use the following augmentor models:

Figure 3: Illustration of the augmentor models used in our experiments. (A)
HARD-Mix: Image-dependent patch-wise interpolation of multiple images. (B)
HARD-Affine: Learned distribution of affine transformations in the pixel
coordinates. (C) HARD-VAE: Finetuning (parts of) a pretrained VAE.

##### HARD-Affine

In the simplest model, we limit the augmentations to affine transformations of
the coordinate grid of pixel locations, i.e. shifts, rotations, scalings, and
shears of images. Models implementing such transformations are known as
_spatial transformers_ [29]. We leverage this model for our augmentor by
learning a distribution over the entries of an affine transformation matrix
$\vartheta\in\mathbb{R}^{2\times 3}$ that defines the transformation of the
sampling grid, i.e. a transformation that maps the pixel positions from the
original image to the augmented image (Figure 3A).

##### HARD-Mix

Additionally we consider a slightly more complex augmentor model, which is an
adaptive variant of the commonly used Mixup [61] and Cutmix [59]
augmentations. However, instead of randomly sampling the ratio and cutout
position that are used to combine images, we learn how to combine the images
dependent on the input images. We achieve this by performing a patch-wise
projection of the input image, followed by comparing each patch with the same
query vector sampled from a learned distribution (Figure 3B). We normalize
similarities for each patch over each group of images and use the resulting
weights to combine the original image patches, giving a combined image. This
mechanism allows our augmentor to decide which features of which image are
shown to the student, enabling it to explore the interpolated space between
images systematically, instead of randomly. As it would not make sense for the
teacher to be invariant to an interpolation as it is generated by HARD-Mix, we
do not consider the teacher-teacher-loss
$\mathcal{L}_{\tilde{\mathrm{t}}\leftrightarrow\mathrm{t}}$ in this case and
optimize student and augmentor jointly instead.

##### HARD-VAE

To lift constraints further, we wanted to use a more powerful augmentor that
could generate a large variety of images across the entire image-space. As the
augmentor has to generate new samples on-the-fly during the student training,
the generation process needs to be very fast, limiting the choice of useful
generative models. For this reason, we focus on variants of the variational
autoencoder architecture [31], allowing for good image reconstructions which
can be achieved reasonably fast in a single forward pass (Figure 3D). For
CIFAR, we choose the _very deep VAE_ [10] model, which we finetune by solely
optimizing parameters of the posterior network from layer 10 onward in the
decoder. For the experiments on ImageNet, we use a Residual-Quantized VAE (RQ-
VAE) [34] pretrained on ImageNet, which we finetune in its entirety and add a
noise vector on the latent state. Hence, as training progresses, the model
changes from generating plain reconstructions of a given image to input
conditioned generations that serve as our augmentations.

##  5 Experiments

Table 1: MNIST (columns “Centered”) and MNIST-C (columns “Shifted”) test accuracies (mean and standard error of the mean across 4 random seeds) comparing KD without augmentation and our HARD-Affine method to Orbit transfer [40], which also learns and transfers equivariances. The left two columns show the transfer results from a small CNN teacher to a MLP student. The right columns show analogous experiments between a ResNet18 teacher and a small ViT student. The best performing transfer is shown in bold for each column. Examples of our HARD-Affine learned data augmentations shown on the right. We include the controls Random Affine and MNIST-C Shifts (marked by italics).  |  | CNN $\rightarrow$ MLP | ResNet18 $\rightarrow$ ViT
---|---|---
Method | Centered | Shifted | Centered | Shifted
Teacher only | 99.0 $\pm$ 0.0 | 91.3 $\pm$ 0.5 | 99.5 $\pm$ 0.0 | 92.8 $\pm$ 0.5
Student only | 98.4 $\pm$ 0.0 | 35.2 $\pm$ 0.7 | 98.3 $\pm$ 0.0 | 40.4 $\pm$ 0.8
\+ Random Affine | 92.1 $\pm$ 0.6 | 81.0 $\pm$ 2.0 | 95.4 $\pm$ 0.3 | 90.4 $\pm$ 1.0
\+ MNIST-C Shifts | 98.1 $\pm$ 0.1 | 86.5 $\pm$ 0.3 | 98.5 $\pm$ 0.0 | 93.7 $\pm$ 0.2
Orbit [40] | 98.8 | 95.2 | 98.4 | 84.0
KD | 98.6 $\pm$ 0.0 | 40.3 $\pm$ 0.6 | 98.6 $\pm$ 0.1 | 44.7 $\pm$ 1.9
\+ HARD-Affine | 98.6 $\pm$ 0.1 | 68.9 $\pm$ 2.5 |  99.2 $\pm$ 0.0 |  84.1 $\pm$ 2.3

###  5.1 Transferring equivariance

For our initial experiment, we reproduce the setup from Nix et al. [40] to
test whether we can transfer the inductive bias from a shift equivariant
teacher, CNN and ResNet18 [22], to a student that does not have this inductive
bias built into its architecture: a Multi-Layer Perceptron (MLP) and a Vision
Transformer (ViT) [17]. When training the students and teachers by themselves
on standard MNIST [15] training data, we observe a small drop of
generalization performance (-0.6% and -1.2%) between teacher and student on
the MNIST test set and a large gap (-56.1% and -52.4%) when we evaluate on a
version of the test set in which digits were randomly shifted [38]. As another
baseline, we applied plain KD to transfer shift equivariance from teacher to
student. Consistent with the findings of Nix et al. [40], we only observe a
small improvement on the centered (+0.2% and +0.3%) and the shifted (+5.1% and
+4.3%) test sets, which likely result from the centered training data we use
for transfer.

We then test if combining KD with our augmentations produced by HARD-Affine
would outperform these baselines. The resulting student model improves
significantly on shifted inputs (+28.6% and +39.4%) compared to plain KD and
the generated images clearly show that the augmentor learns to shift the
digits within the image. Compared to Nix et al. [40] our approach outperforms
their results on the ViT task but, while improving the out-of-domain
generalization by 28.6% over baseline, stays behind the Orbit performance on
the MLP task. This demonstrates that our method while acting on fewer parts of
the network compared to Orbit and while being a more general method, can
improve or reach better performance when it comes to transferring invariances,
and can be generalized to bigger datasets, as we show below.

We verify that the student’s performance improvement is specifically due to
our data generation framework in two control experiments. The first experiment
(Random Affine) augments the training inputs of a stand-alone student model
with a random affine transformation akin to our augmentor model, but using
transformation parameters sampled uniformly from a pre-defined, reasonably
constrained range (i.e. ensuring the digit is always fully visible). This
student performs well on the shifted test set, however, performance
significantly degrades on the centered test set. In comparison, our HARD-
Affine model is unconstrained and learns more useful augmentations, leading to
better performance on the centered test sets.

In our second control (Shifts) we asked how much data augmentation could
improve the performance in the best case (without KD). For this, we augment
the inputs by the same random shifts that were applied to obtain the shifted
test data, leading to great improvements on the shifted test set. However, our
learned augmentations achieve scores in a similar range on the shifted
evaluation and outperform its results on the centered test set.

###  5.2 Transfer on natural images

After demonstrating that our method successfully captures the difference
between teacher and student and bridges a gap in inductive bias, we now want
to test whether this effect holds up in more realistic scenarios.

##### CIFAR experiments

Table 2: Test accuracies on the CIFAR10 test set. Standard error of the mean
is reported where available across three different seeds. Best transfer is
highlighted in bold. The ResNet101∗ models were pretrained on ImageNet.
Examples of augmented test images from ResNet18$\rightarrow$ViT experiments
with samples across different iterations are shown to the right.

ResNet18 $\downarrow$ ViT ResNet101∗ $\downarrow$ ViT ResNet101∗ $\downarrow$
ResNet18 Teacher only 92.5 $\pm$ 0.0 95.5 95.5 Student only 68.5 $\pm$ 0.5
68.5 $\pm$ 0.5 78.5 \+ Standard Aug. 78.3 $\pm$ 0.4 78.3 $\pm$ 0.4 92.6 \+
Random Affine Aug. 58.9 $\pm$ 0.4 58.9 $\pm$ 0.4 79.3 KD 67.9 $\pm$ 0.1 68.5
84.4 \+ Standard Aug. 80.9 $\pm$ 0.1 79.3 93.3 \+ HARD-Affine 87.8 $\pm$ 0.8
84.4 93.5 \+ HARD-VAE 81.9 $\pm$ 0.4 81.2 91.0 \+ HARD-VAE-Affine 87.6 $\pm$
0.6 87.1 94.0

We begin by applying our framework to CIFAR10 [32] on three different KD
scenarios (see Table 2). Specifically, we test scenarios where the student
lacks an inductive bias (ResNet18$\rightarrow$ViT), where the teacher has more
capacity and access to data than the student (ResNet101∗
$\rightarrow$ResNet18), and to scenarios combining both properties (ResNet101∗
$\rightarrow$ViT). For all experiments, we keep the experimental setup as
close to our previous MNIST experiments as possible (see Appendix A for
details).

We start by establishing baselines by training only the teacher and only the
student models on the data and evaluating default KD. We observe that on this
small data set a small ResNet18 performs better (78.5% accuracy) than a larger
ViT (68.5%), likely because of the ResNet’s superior inductive bias on this
task and small data set. Next, we find that adding default data augmentations
(random rotations, cropping, horizontal flips) to the student baselines
significantly boosts performance to 92.6% and 78.3% for the ResNet18 and ViT,
respectively. Adding these default augmentations to typical KD leads to a
great performance boost, too (see Table 2).

Given that adding default data augmentation to KD already leads to a
substantial performance boost, it is particularly noteworthy that the data
augmentations learned by HARD-Affine outperform this baseline for the ViT.
Qualitatively, the augmented images exhibit a large variety of spatial
transformations, suggesting that a difference in these examples lead to the
observed performance boost (Table 2, right).

We then investigated performance of our HARD-VAE augmentation strategy and
found performance improvement over the KD + standard augmentations baseline
for transfer to the ViT (+1.0% and +1.9%) student. However, inspecting the
augmented images indicates that our augmentor lacks the expected shifts of
object positions, but rather learns stylistic changes in the image (Table 2,
right). This motivated us to combine HARD-Affine and HARD-VAE augmentation
resulting in best performance (up to +7.8%) for all teacher-student pairings
(HARD-VAE-Affine in Table 2) and the resulting images demonstrate variability
in both style and spatial alignment (Table 2, right).

##### ImageNet experiments

Having established our methods’ performance for CIFAR10, we extend our results
to classification on ImageNet [14]. Here we aim to distill a ResNet50 [22]
teacher, trained with Deep-augment and AugMix data augmentations [25], into a
smaller ResNet18 and ViT-S (small vision transformer variant) [17] that we
want to be particularly robust to natural image corruptions,. The distillation
into ResNet18 allows us to investigate the capability for model compression,
because ResNet18 is a smaller network compared to ResNet50, but with a similar
architecture. Distillation into a ViT-S architecture with a patch-size of 14
tests additionally if KD transfers the ResNet50’s inductive bias of shift
equivariance on a larger dataset.

ResNet50 $\rightarrow$ ResNet18 ResNet50 $\rightarrow$ ViT-S Val ReaL V2 Val
ReaL V2 Teacher 75.8 83.1 63.7 75.8 83.1 63.7 Student 70.7 78.1 57.4 73.2 79.4
60.3 KD 70.7 78.7 58.1 75.3 82.8 62.9 \+ HARD-Affine 71.6 79.5 58.6 74.9 82.3
62.4 \+ HARD-Mix 71.4 79.4 58.6 75.7 83.0 63.3 \+ HARD-VAE 71.0 78.9 58.7 75.8
83.1 63.5

Table 3: In-domain evaluation for ImageNet: reporting Top-1 accuracy in % on
ImageNet-Validaton [14], ImageNet-ReaL [5] and ImageNet-V2 [46] with KD from a
robust ResNet50 [25] teacher to ResNet18 (columns 2-4) and ViT-S (columns 5-7)
students.

Table 4: In-domain evaluation for ImageNet: Reporting Top-1 accuracy in % on
ImageNet-A [26], ImageNet-R [25], ImageNet-Sketch [57] and ImageNet-Style [19]
and mean-corruption-error on ImageNet-C (lower is better) [23].

ResNet50 $\rightarrow$ ResNet18 ResNet50 $\rightarrow$ ViT-S Im-A Im-R Im-C
$\downarrow$ Sketch Style Im-A Im-R Im-C $\downarrow$ Sketch Style Teacher 3.8
46.8 53.0 32.6 21.2 3.8 46.8 53.0 32.6 21.2 Student 1.6 30.0 88.1 18.4 4.4 8.0
26.3 78.1 13.8 6.6 KD 1.6 40.2 69.2 26.0 13.4 3.3 45.0 56.8 29.6 18.7 \+ HARD-
Affine 1.5 38.2 73.1 24.9 10.4 3.4 40.8 62.2 26.2 14.5 \+ HARD-Mix 1.8 39.9
68.8 26.1 13.7 3.5 45.4 56.2 29.9 19.2 \+ HARD-VAE 1.7 39.5 72.5 25.8 12.1 3.4
45.4 57.4 30.7 18.1

We evaluate on common test sets for both in-domain (ID) [5, 46] and out-of-
domain (OOD) [23, 26, 25, 19, 57] generalization performance (Tables 3 and 4,
respectively). To properly investigate the extrapolation abilities of KD
training, we trained a strong KD baseline by applying several data
augmentations: we randomly switch between Cutmix [59] and Mixup [61], each
drawing their interpolation weight from a $\beta$-distribution with
$\alpha=1$, as well as AugMix [24] augmentations. For the standalone student
training, we additionally apply various lighter data augmentations (Cutmix
with $\alpha=1$, Mixup with $\alpha=0.1$, and Trivialaugment [39]). Since we
ask how KD can be improved in a setting of limited resources, we run our
experiments an order of magnitude shorter than proposed for the state-of-the-
art in KD [6] (200 epochs for all ResNet18 and 150 epochs for all ViT-S
experiments). For student and KD models, we perform a small grid search over
learning-rate and weight-decay hyperparameters. We then train the models with
our HARD framework based on the hyperparameters of our best performing KD
setting. The augmentor-specific settings are selected through a small grid-
search in the ResNet18 setting (for details see Appendix A).

We first evaluate the ID performance of our methods (Table 3) beginning with
the standalone teacher and student baselines, which reveal a larger
performance gap between the ResNet18 student and the ResNet50 teacher compared
to the ViT-S student (5.1% and 2.6% on the ImageNet validation set,
respectively). Plain KD significantly reduces this gap for the ViT-S (+2.1%
performance improvement compared to standalone). For the ResNet18 student KD
achieves only small (0.7% V2) improvements or no improvements (0.0% Val), even
though the initial gap between teacher and student is larger. Applying HARD-
Affine, HARD-Mix and HARD-VAE augmentation on this task improves over plain KD
across most augmentation models and test sets with student performance gains
of up to 0.9% for ResNet18 (HARD-Affine) and 0.6% for ViT-S (HARD-VAE). For
ViT-S, our best-performing HARD-VAE method even matches the teacher’s
performance on 2 out of 3 test sets.

For the OOD setting (Table 4), we observe that the initial gap between student
and teacher is larger than on ID data across all data sets (up to 35.1%
difference), except for Im-A in the ViT-S setting. The aggressive data
augmentations we apply for the plain KD baseline favor OOD performance, hence
it is expected that plain KD results in good performance improvement over the
standalone baseline (up to 21.3% imporvement on Im-C). All three HARD
approaches transfer some of the teacher’s generalization abilities leading to
improvements on a number of students and data sets, however, HARD-Affine fails
to reach the KD performance in both settings and HARD-VAE underperforms for
the ResNet18 student in these OOD scenarios. However, HARD-Mix and HARD-VAE
(for ViT-S) outperform plain KD on several test sets and are roughly on par on
all others, across the board. Given that we chose a very strong baseline by
applying aggressive state-of-the-art data augmentations we find these results
especially encouraging.

##  6 Discussion

Figure 4: Example augmentations applied to images of the ImageNet validation
set obtained from augmentor models in the ViT-S setting at the end of
training.

##### Interpretability

HARD enables us to gain insight into the distillation mechanism as the
augmented images illustrate the knowledge that is transferred (Figure 4). As
expected, HARD-Affine learns to downscale images to shift and rotate the
images such that the object in the image is shown in different places (row 2-4
in Figure 4) and scales such that the images is cropped (row 1). As HARD-Mix
is a dynamically learnable extension of mixup, it either merges two objects
into the same picture (row 1 and 4), especially if they are not in the same
position, or uses one image to change the style (row 2) or background (row 3)
of another. Finally, HARD-VAE mostly impacts the style of an image and
additionally adds small distortions to specific image regions, which is
noticable by the altered image brightness and the blurring of some high-
frequency features.

##### Limitations and broader impact

State-of-the-art knowledge distillation typically deals with huge models
(billions of parameters) and incredibly long training times (>9,000 epochs)
[6, 13]. In comparison, our study is computationally lightweight in requiring
approximately 400 A100GPU days across all our experiments. We believe
exploring even more flexible augmentor models with a semantically meaningful
latent space as for example diffusion models [50, 44, 49, 47] could improve
our proposed methods even further. However, generating a single image with
out-of-the-box diffusion models requires multiple seconds. This is
prohibitively long, so leave exploring their usability in our proposed dynamic
data augmentation technique for future work. In general, KD allows us to
distill smaller models that perform similar to large foundation models.
Improving the distillation process to be more efficient lowers the barrier of
applying KD across labs with various compute budget and decreases
environmental impact. At the same time, transferring generalization abilities
effectively and consistently results in smaller distilled models that are
appealing to use, thus we would expect such smaller models to be used
abundantly hence lowering the general carbon footprint for model usage. In
conclusion, our study proposes avenues to efficiently improve KD in terms of
performance, efficiency, and hence environmental impact.

##  7 Conclusion

In this work we introduced a general, task-agnostic, and modular framework to
extend knowledge distillation by learnable data augmentations. The
augmentation models are optimized to generate inputs on which teacher and
student disagree, keeping the teacher’s predictions unchanged at the same
time. We show that these augmentations can solve the issue of KD and transfer
equivariance properties, even in cases where the teacher’s inductive biases
are distinct from the student’s. We further demonstrate that our learned
augmentations achieve performance competitive to established classical data
augmentation techniques even when student and teacher share similar inductive
biases. Overall our framework offers a powerful tool that enhances transfer
performance and offers a unique insights into the transferred knowledge
through its interpretable augmentations.

#### Acknowledgements

Furthermore, we thank Felix Schlüter for his helpful insights into evaluation
problems as well as Mohammad Bashiri, Pawel Pierzchlewicz and Suhas
Shrinivasan for helpful comments and discussions. The authors thank the
International Max Planck Research School for Intelligent Systems (IMPRS-IS)
for supporting Arne Nix and Max F. Burg.

This work was supported by the Cyber Valley Research Fund (CyVy-RF-2019-01),
by the German Federal Ministry of Education and Research (BMBF) through the
Tübingen AI Center (FKZ: 01IS18039A), by the Deutsche Forschungsgemeinschaft
(DFG) in the SFB 1233, Robust Vision: Inference Principles and Neural
Mechanisms (TP12), project number: 276693517, and funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID
432680300 – SFB 1456. FHS is supported by the Carl-Zeiss-Stiftung and
acknowledges the support of the DFG Cluster of Excellence “Machine Learning –
New Perspectives for Science”, EXC 2064/1, project number 390727645.

## References

  * Agarap [2018] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  _arXiv preprint arXiv:1803.08375_ , 2018.
  * Antoniou et al. [2022] Anthreas Antoniou, Amos Storkey, and Harrison Edwards.  Data Augmentation Generative Adversarial Networks.  feb 2022.
  * Behpour et al. [2019] Sima Behpour, Kris M. Kitani, and Brian D. Ziebart.  ADA: Adversarial data augmentation for object detection.  _Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019_ , pages 1243–1252, mar 2019.  doi: 10.1109/WACV.2019.00137.
  * Benjamin et al. [2019] Ari S Benjamin, David Rolnick, and Konrad P Kording.  Measuring and regularizing networks in function space.  In _7th International Conference on Learning Representations, ICLR 2019_ , 2019.
  * Beyer et al. [2020] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord.  Are we done with imagenet?  _arXiv preprint arXiv:2006.07159_ , 2020.
  * Beyer et al. [2021] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  jun 2021.  URL https://arxiv.org/abs/2106.05237v1http://arxiv.org/abs/2106.05237.
  * Bucilǎ et al. [2006] Cristian Bucilǎ, Rich Caruana, and Alexandra Niculescu-Mizil.  Model compression.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , volume 2006, pages 535–541, 2006.  ISBN 1595933395.  doi: 10.1145/1150402.1150464.
  * Chen et al. [2023] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le.  Symbolic discovery of optimization algorithms, 2023.
  * Chen et al. [2022] Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao.  DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers.  apr 2022.  doi: 10.48550/arxiv.2204.12997.  URL https://arxiv.org/abs/2204.12997v2.
  * Child [2020] Rewon Child.  Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images.  nov 2020.  doi: 10.48550/arxiv.2011.10650.  URL https://arxiv.org/abs/2011.10650v2http://arxiv.org/abs/2011.10650.
  * Cubuk et al. [2018] Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le.  AutoAugment: Learning Augmentation Policies from Data.  _Cvpr 2019_ , (Section 3):113–123, may 2018.  doi: 10.48550/arxiv.1805.09501.  URL https://arxiv.org/abs/1805.09501v3.
  * Cubuk et al. [2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 702–703, 2020.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby.  Scaling vision transformers to 22 billion parameters, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Deng [2012] Li Deng.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.  8-bit optimizers via block-wise quantization.  _9th International Conference on Learning Representations, ICLR_ , 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  2020\.  URL https://github.com/http://arxiv.org/abs/2010.11929.
  * Engstrom et al. [2017] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry.  A rotation and a translation suffice: Fooling cnns with simple transformations.  2017\.
  * Geirhos et al. [2018] Robert Geirhos, Claudio Michaelis, Felix A. Wichmann, Patricia Rubisch, Matthias Bethge, and Wieland Brendel.  IMAGENET-TRAINED CNNS ARE BIASED TOWARDS TEXTURE; INCREASING SHAPE BIAS IMPROVES ACCURACY AND ROBUSTNESS, nov 2018.  ISSN 23318422.  URL http://arxiv.org/abs/1811.12231.
  * Gong et al. [2021] Kehong Gong, Jianfeng Zhang, and Jiashi Feng.  PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 8571–8580, may 2021.  ISSN 10636919.  doi: 10.1109/CVPR46437.2021.00847.  URL https://arxiv.org/abs/2105.02465v1.
  * Haidar et al. [2022] Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Philippe Langlais, and Pascal Poupart.  CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation.  apr 2022.  doi: 10.48550/arxiv.2204.07674.  URL https://arxiv.org/abs/2204.07674v1http://arxiv.org/abs/2204.07674.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2016-Decem:770–778, dec 2015.  ISSN 10636919.  doi: 10.1109/CVPR.2016.90.  URL https://arxiv.org/abs/1512.03385v1.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations, mar 2019.  ISSN 23318422.  URL http://arxiv.org/abs/1903.12261.
  * Hendrycks et al. [2019] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  _arXiv preprint arXiv:1912.02781_ , 2019.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 8320–8329, 2021a.  ISBN 9781665428125.  doi: 10.1109/ICCV48922.2021.00823.  URL https://github.com/hendrycks/imagenet-r.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _CVPR_ , 2021b.
  * Hinton and Dean [2015] Geoffrey Hinton and Jeff Dean.  Distilling the Knowledge in a Neural Network.  Technical report, 2015.
  * Jackson et al. [2019] Philip TG Jackson, Amir Atapour Abarghouei, Stephen Bonner, Toby P Breckon, and Boguslaw Obara.  Style augmentation: data augmentation via style randomization.  In _CVPR workshops_ , volume 6, pages 10–11, 2019.
  * Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In _Advances in Neural Information Processing Systems_ , volume 2015-Janua, pages 2017–2025. Neural information processing systems foundation, jun 2015.  URL https://arxiv.org/abs/1506.02025v3.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ , dec 2014.  URL https://arxiv.org/abs/1412.6980v9.
  * Kingma and Welling [2014] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In _2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings_. International Conference on Learning Representations, ICLR, dec 2014.  doi: 10.48550/arxiv.1312.6114.  URL https://arxiv.org/abs/1312.6114v10.
  * [32] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  CIFAR-10 (Canadian Institute for Advanced Research).  URL http://www.cs.toronto.edu/$∼$kriz/cifar.html.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.  URL https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  * Lee et al. [2022] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.  Autoregressive image generation using residual quantization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11523–11532, 2022.
  * Li et al. [2019] Zhu Li, Adrian Perez-Suay, Gustau Camps-Valls, and Dino Sejdinovic.  Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness.  Technical report, 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts, 2017.
  * McClure and Kriegeskorte [2016] Patrick McClure and Nikolaus Kriegeskorte.  Representational distance learning for deep neural networks.  _Frontiers in Computational Neuroscience_ , 10(DEC):131, dec 2016.  ISSN 16625188.  doi: 10.3389/fncom.2016.00131.  URL http://journal.frontiersin.org/article/10.3389/fncom.2016.00131/full.
  * Mu and Gilmer [2019] Norman Mu and Justin Gilmer.  MNIST-C: A Robustness Benchmark for Computer Vision.  jun 2019.  doi: 10.5281/zenodo.3237938.  URL https://arxiv.org/abs/1906.02337v1http://arxiv.org/abs/1906.02337.
  * Müller and Hutter [2021] Samuel G Müller and Frank Hutter.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 774–782, 2021.
  * Nix et al. [2022] Arne Nix, Suhas Shrinivasan, Edgar Y Walker, and Fabian Sinz.  Can Functional Transfer Methods Capture Simple Inductive Biases?  In Gustau Camps-Valls, Francisco J R Ruiz, and Isabel Valera, editors, _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , volume 151 of _Proceedings of Machine Learning Research_ , pages 10703–10717. PMLR, 2022.  URL https://proceedings.mlr.press/v151/nix22a.html.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Pan et al. [2020] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E Turner, and Mohammad Emtiyaz Khan.  Continual deep learning by functional regularisation of memorable past.  In _Advances in Neural Information Processing Systems_ , volume 2020-Decem, 2020.
  * Pham et al. [2021] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V. Le.  Meta Pseudo Labels.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 11553–11563, mar 2021.  ISSN 10636919.  doi: 10.1109/CVPR46437.2021.01139.  URL https://arxiv.org/abs/2003.10580v4.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022.  URL http://arxiv.org/abs/2204.06125.  arXiv:2204.06125 [cs].
  * Rashid et al. [2021] Ahmad Rashid, Vasileios Lioutas, and Mehdi Rezagholizadeh.  MATE-KD: Masked adversarial text, a companion to knowledge distillation.  In _ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference_ , pages 1062–1071, 2021.  ISBN 9781954085527.  doi: 10.18653/v1/2021.acl-long.86.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Rusak et al. [2020] Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel.  A simple way to make neural networks robust against diverse image corruptions.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 12348 LNCS:53–69, jan 2020.  ISSN 16113349.  doi: 10.48550/arxiv.2001.06057.  URL https://arxiv.org/abs/2001.06057v5.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, May 2022.  URL http://arxiv.org/abs/2205.11487.  arXiv:2205.11487 [cs].
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2826, 2016.  doi: 10.1109/CVPR.2016.308.
  * Titsias et al. [2019] Michalis K Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh.  Functional Regularisation for Continual Learning with Gaussian Processes.  2019\.  URL http://arxiv.org/abs/1901.11356.
  * Tolstikhin et al. [2021] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  MLP-Mixer: An all-MLP Architecture for Vision.  _Advances in Neural Information Processing Systems_ , 29:24261–24272, may 2021.  ISSN 10495258.  doi: 10.48550/arxiv.2105.01601.  URL https://arxiv.org/abs/2105.01601v4.
  * Touvron et al. [2020] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  pages 10347–10357, jul 2020.  ISSN 2640-3498.  URL https://proceedings.mlr.press/v139/touvron21a.htmlhttp://arxiv.org/abs/2012.12877.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 2017-Decem, pages 5999–6009, 2017.
  * Volpi et al. [2018] Riccardo Volpi, John Duchi, Hongseok Namkoong, Vittorio Murino, Ozan Sener, and Silvio Savarese.  Generalizing to Unseen Domains via Adversarial Data Augmentation.  _Advances in Neural Information Processing Systems_ , 2018-Decem:5334–5344, may 2018.  ISSN 10495258.  doi: 10.48550/arxiv.1805.12018.  URL https://arxiv.org/abs/1805.12018v2.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , pages 10506–10518, 2019.
  * Wang et al. [2020] Huan Wang, Suhas Lohit, Michael Jones, and Yun Fu.  Knowledge Distillation Thrives on Data Augmentation.  dec 2020.  doi: 10.48550/arxiv.2012.02909.  URL https://arxiv.org/abs/2012.02909v1http://arxiv.org/abs/2012.02909.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6023–6032, 2019.
  * Zagoruyko and Komodakis [2017] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In _5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings_ , 2017.  URL https://github.com/szagoruyko/attention-transfer.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond Empirical Risk Minimization.  _6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings_ , oct 2017.  doi: 10.48550/arxiv.1710.09412.  URL https://arxiv.org/abs/1710.09412v2.
  * Zhang et al. [2022] Minjia Zhang, Niranjan Uma Naresh, and Yuxiong He.  Adversarial Data Augmentation for Task-Specific Knowledge Distillation of Pre-trained Transformers.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):11685–11693, 2022.  ISSN 2159-5399.  doi: 10.1609/aaai.v36i10.21423.  URL www.aaai.org.
  * Zhang et al. [2019a] Xiaofeng Zhang, Zhangyang Wang, Dong Liu, and Qing Ling.  DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classification.  _ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings_ , 2019-May:2807–2811, may 2019a.  ISSN 15206149.  doi: 10.1109/ICASSP.2019.8683197.  URL https://arxiv.org/abs/1809.00981v1.
  * Zhang et al. [2019b] Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong.  Adversarial AutoAugment.  dec 2019b.  doi: 10.48550/arxiv.1912.11188.  URL https://arxiv.org/abs/1912.11188v1http://arxiv.org/abs/1912.11188.
  * Zhao et al. [2020] Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas.  Maximum-entropy adversarial data augmentation for improved generalization and robustness.  In _Advances in Neural Information Processing Systems_ , volume 2020-Decem, 2020.  URL https://github.com/garyzhao/ME-ADA.
  * Zhong et al. [2017] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation, 2017.
  * Zietlow et al. [2022] Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Chris Russell.  Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers.  2022\.  URL http://arxiv.org/abs/2203.04913.

##  Appendix A Setup Details

Our experiments on MNIST were meant to reproduce Nix et al. [40] and thus
follow their setup exactly, using the same training setup and model
architectures.

###  A.1 CIFAR10 Experiments

##### Training

We train on the entire CIFAR10 dataset (excluding 10% held-out as validation
set) for 300 epochs with a batch-size of 256. As an optimizer, we use Adam
[30] with a learning rate of 0.0003 and an L2-regularization of 2$\cdot$10-9.
Our training begins with a linear warmup of the learning rate for 20 epochs.
The validation accuracy is monitored after every epoch and if it has not
improved for 20 consecutive epochs, we decay the learning rate by a factor of
0.8 and restore the previously best performing model. The training is stopped
prematurely if we decay five times.

##### Models

The different models we use generally follow the standard architecture and
settings know from the literature. For the ViT, we use a smaller variant of it
on the CIFAR task. It consists of six layers and eight attention heads
throughout the network. The dropout rate is set to 0.1 and the hidden
dimension is chosen as 512 in all places.

##### KD and HARD

After initial experiments on MNIST, we decided to use a softmax temperature of
5.0 for all experiments involving KD. We furthermore rely solely on the KL-
Divergence loss to optimize our model. For the experiments with our
augmentation framework, we have the same settings as before for the student
(KD) training and separate settings for the augmentor training. There we have
different settings depeding on whether we use the VAE augmentor (or the Affine
augmentor). There we reduce the batch-size to 160 (128) and a learning-rate of
0.0001 (0.05). We initialize both augmentors to perform an identity
transformation, i.e. the VAE is taken pretrained from Child [10]. The
thresholds for switching are set as $\ell_{\min}=10\%(5\%)$ and
$\ell_{\max}=60\%(40\%)$. The train modi are switched if the threshold is
surpassed for 5 consecutive iterations. Both $\lambda_{\mathrm{s}}$ and
$\lambda_{\mathrm{t}}$ are set to 1 for the experiments. For the experiment
ResNet101∗ $\rightarrow$ ResNet18, we found a slightly different setting to be
more effective with $\ell_{\min}=$5% and $\ell_{\max}=40\%$ and a switch only
happening if the threshold is surpassed for 10 consecutive iterations.

###  A.2 ImageNet Experiments

##### Baseline Training

In general, all our ImageNet experiments follow a similar setup. We train with
a batch-size of 512 samples using the Lion optimizer [8] with a linear
learning-rate warmup to a defined initial learning-rate. Afterwards, we anneal
the learning-rate following a cosine schedule [36] with a final value of 0.
The training runs for 200 epochs for all ResNet18 experiments and 150 epochs
for the ViT-S experiments. Throughout the training, the validation accuracy is
monitored on a heldout set consisting of samples randomly chosen from training
set, making up 1% of the total number of samples. The validation performance
is used to pick the best performing epoch throughout training for final
evaluation and the best hyperparameters during grid-search. We train at a
resolution of 224 pixels with random resizing and cropping, as well as random
horizontal flips applied in all trainings. All training runs are performed
with automatic mixed precision and 8bit optimization [16].

##### Student Training

After a grid-search, we found that for the standalone student training, an
optimization with learning-rate 0.0001 with weight decay 0.1 for the ResNet18
student and learning-rate 0.00005 with weight decay 0.001 for the ViT-S
student worked best. For both students, we apply light augmentations during
training with Mixup ($\alpha=0.2$) [61] and CutMix ($\alpha=1.0$) [59]. For
the ViT-S baseline, we additionally apply Trivial-Augment [39] and randomly
erase pixels from the input image [66] with a probability of 0.1. We optimize
the standard cross-entropy loss with additional label-smoothing [51] mixed in
with a factor of 0.1.

##### KD and HARD

As described in the main paper, the configuration for the KD experiments
(including HARD) mainly differ in the choice of augmentation, as well as
learning-rate and weight-decay. The plain KD experiments use Mixup
($\alpha=1.0$) and CutMix ($\alpha=1.0$) as well as AugMix [24] augmentation.
The softmax temperature was chosen as 1.0 in prior experiments and kept for
all experiments. The learning-rate for all KD and HARD experiments was chosen
through a grid-search to be 0.0001 in all cases and weight-decay is 0.001 in
most cases, except for HARD experiments with a ResNet18 student where a
weight-decay of 0.05 is used.

##  Appendix B Knowledge Distillation Results from the Literature

We (re-)evaluated student and teacher models from two high-performing KD
experiments [6, 41] in the literature on both in-domain and out-of-domain test
sets.

Table 5: In-domain and out-of-domain performance for two KD experiments from
the literature. Showing that the gap in out-of-domain evaluations is larger
than in-domain.

In Domain Out of Domain Model Val ReaL V2 Im-A Im-R Im-C Sketch Style BiT
ResNet152 (Teacher) [6] 82.9 87.8 72.0 31.9 49.2 51.0 37.4 16.9 BiT ResNet50
(Distilled) 82.8 87.5 72.5 25.1 45.3 51.8 31.6 15.1 Dino Teacher (Teacher)
[41] 86.5 89.7 78.4 76.1 79.3 27.3 62.8 34.6 ViT-S/14 (Distilled) 81.2 86.7
71.2 34.4 55.1 53.4 42.2 13.5


Xing2023X23 Jeena2024J24

# LAMOST J1010+2358 is Not a Pair-Instability Supernova Relic

Pierre N. Thibodeaux 1,2 0000-0002-3867-3927 Alexander P. Ji 1,2,3
0000-0002-4863-8842 William Cerny 4 0000-0003-1697-7062 Evan N. Kirby5
0000-0001-6196-5162 Joshua D. Simon 6 1Department of Astronomy & Astrophysics,
University of Chicago, 5640 S Ellis Avenue, Chicago, IL 60637, USA  2Kavli
Institute for Cosmological Physics, University of Chicago, Chicago, IL 60637,
USA  3Joint Institute for Nuclear Astrophysics  4Department of Astronomy, Yale
University, New Haven, CT 06520, USA  5Department of Physics, University of
Notre Dame, Notre Dame, IN 46556, USA  6Observatories of the Carnegie
Institution for Science, 813 Santa Barbara St., Pasadena, CA 91101, USA
Corresponding author<EMAIL_ADDRESS>
###### Abstract

The discovery of a star formed out of pair-instability supernova ejecta would
have massive implications for the Population III star initial mass function
and the existence of stars over 100 $M_{\odot}$, but none have yet been found.
Recently, the star LAMOST J1010+2358 was claimed to be a star that formed out
of gas enriched by a pair-instability supernova. We present a non-LTE
abundance analysis of a new high-resolution Keck/HIRES spectrum of J1010+2358.
We determined the carbon and aluminum abundances needed to definitively
distinguish between enrichment by a pair-instability and core-collapse
supernova. Our new analysis demonstrates that J$1010+2358$ does not have the
unique abundance pattern of a a pair-instability supernova, but was instead
enriched by the ejecta of a low mass core-collapse supernova. Thus, there are
still no known stars displaying unambiguous signatures of pair-instability
supernovae.

###### keywords:

stars: abundances – first stars, galaxies, reionization

##  1 Introduction

The first Population III stars in the universe are thought to have been
extremely massive (Bromm et al., 2001; Hirano et al., 2014). The top-heavy
initial mass function should easily populate the mass range of
$140-260\mathrm{\,M_{\odot}}$, where stars are expected to explode as pair-
instability supernovae (PISNe, e.g., Heger et al. 2003; Yoon et al. 2012;
Nomoto et al. 2013). PISNe are expected to produce a unique abundance pattern
with extremely strong odd-even ratios (Heger & Woosley, 2002; Takahashi et
al., 2018). Detecting even one star with a PISN signature would have important
implications for the Population III initial mass function (e.g., Koutsouridou
et al., 2024).

However, stars exhibiting a clear PISN signature have remained elusive. The
first claimed PISN detection (Aoki et al., 2014) turned out to have an
abundance pattern more consistent with a core-collapse supernova (e.g.,
Takahashi et al., 2018). One difficulty with finding PISNe is that they likely
enrich stars to relatively high metallicities $\mathrm{\,[Fe/H]}\sim-2.5$ or
$-2.0$, so they may be missed in most surveys that primarily target stars with
$\mathrm{\,[Fe/H]}\lesssim-3$ (Karlsson et al., 2008). The PISN signatures are
rapidly erased by any contamination (e.g., Ji et al., 2015), so the signature
should be searched for in the presence of contamination by other supernovae
(Salvadori et al., 2019; Aguado et al., 2023).

Recently, Xing et al. (2023) discovered that the metal-poor star LAMOST
J1010$+$2358 (hereafter abbreviated J$1010+2358$) is a likely candidate that
has preserved a PISN signature. They found the star has high metallicity
($\mathrm{\,[Fe/H]}=-2.4$), low [Mg/Fe], and an extreme odd-even effect
including non-detections of Na, Sc, Zn, Sr, and Ba. The derived abundance
pattern is a clear match to a massive (260$\mathrm{\,M_{\odot}}$) PISN.
However, Jeena et al. (2024, hereafter ) and Koutsouridou et al. (2024)
pointed out that the abundance pattern could also be consistent with a core-
collapse supernova (CCSN). Key elements needed to strengthen the PISN claim
include low abundances of carbon, aluminum, and potassium, which were not
measured in the original analysis.

Here we present a new abundance analysis of J$1010+2358$ using a high signal-
to-noise ratio Keck/HIRES spectrum covering an expanded wavelength range. In
Section 2, we discuss the data and then explain our method of deriving
elemental abundances for J$1010+2358$. In Section 3, we present the best
fitting SN models to our abundances, and we compare our findings to Xing et
al. (2023, hereafter ) in Section 4. Contrary to their result, and consistent
with the alternate interpretation proposed by , we find that J$1010+2358$ is
best explained by a CCSN and not a PISN progenitor.

Figure 1: Keck/HIRES spectra (black) of J$1010+2358$ around aluminum, carbon,
magnesium, and sodium features. Our re-reduction of ’s Subaru/HDS spectrum is
plotted in blue for comparison, which shows clear Na D absorption. Our spectra
are less noisy, allowing for the measurement of carbon. The HDS spectrum also
lacks coverage of the aluminum lines.

##  2 Observations and Abundance Analysis

###  2.1 Observations

We obtained a spectrum of J$1010+2358$ on 2024 January 18 using Keck/HIRES
(Vogt et al., 1994) with its red cross-disperser. The exposure time was 8100s
over 5 exposures with a 11 slit and 2x1 binning, resulting in a resolution of
$R\sim 36000$. Data were reduced using MAKEE
v6.4111https://sites.astro.caltech.edu/~tb/ipac_staff/tab/makee/. We also
observed a nearby telluric star to verify that the lines used in this analysis
are unaffected by telluric absorption. The signal-to-noise (S/N) is 30 per
0.02 Å pixel at 3950 Å where the blue Al lines are located and 75 per 0.03 Å
pixel at 7700 Å where the K 7699 Å line is located. We used smhr
222https://github.com/andycasey/smhr/tree/py38-mpl313 (Casey, 2014) to correct
the radial velocity, normalize the echelle orders, and stitch the spectrum.
Figure 1 shows portions of the spectrum around Al 3961 Å, the Na D lines, the
Mg b lines, and CH G band. In contrast to , we clearly detect two Na D lines
(see discussion in Section 4).

###  2.2 Stellar Parameters

We derived stellar parameters using Gaia DR3 photometry and astrometry (Gaia
Collaboration et al., 2016; Brown et al., 2021). We dereddened the Gaia
photometry using the Gaia DR3
relations333https://www.cosmos.esa.int/web/gaia/edr3-extinction-law, and
adopted color-$T_{\mathrm{eff}}$ relations from Mucciarelli et al. (2021). The
surface gravity ($\log\,g$) was determined using this $T_{\mathrm{eff}}$,
applying bolometric corrections from Casagrande & VandenBerg (2018), and the
distance from Bailer-Jones et al. (2018). To obtain the microturbulence
$v_{t}$, we fixed the $\log\,g$ and $T_{\mathrm{eff}}$ and used ATLAS
(Castelli & Kurucz, 2004) with MOOG in smhr to balance the [Fe I/H] abundances
with respect to the reduced equivalent widths. Our adopted stellar parameters
and uncertainties are $T_{\mathrm{eff}}=5700\pm 200$ K, $\log\,g=4.68\pm 0.3$,
and $v_{t}=1.45\pm 0.2$ $\mathrm{\,km\,s^{-1}}$, with
$[\alpha\mathrm{/Fe}]=0.0$ and $\mathrm{\,[M/H]}$ $=-2.5\pm 0.2$.

To estimate an uncertainty on the parameters, we first looked at , who report
$T_{\mathrm{eff}}=5860$ K, $\log\,g=3.6$, $\mathrm{\,[Fe/H]}=-2.42$, and
$v_{t}=1.5$ $\mathrm{\,km\,s^{-1}}$. However, the  value for $\log\,g$
(suggesting that J$1010+2358$ is a subgiant) is inconsistent with the
$\log\,g$ required by the Gaia DR3 parallax (which implies J$1010+2358$ is a
dwarf). Their reported temperature is consistent with the range of $5700-5900$
K we found based on using different colors (all combinations of $BP$, $G$,
$RP$, and $K_{s}$) of the Gaia photometry.

We also determined spectroscopic stellar parameters in two ways: first, we
followed the method of Frebel et al. (2013), using excitation, ionization, and
line strength balance and then applying a correction to the photometric
temperature scale to determine $T_{\mathrm{eff}}=5540$ K, $\log\,g=4.20$,
$\nu_{t}=1.62~{}\mathrm{\,km\,s^{-1}}$. Second, we performed a parameter
balance using LOTUS (Li & Ezzeddine, 2023) in non-LTE mode. We obtained
$T_{\mathrm{eff}}=5520\pm 165$ K, $\log\,g=4.8\pm 0.4$, $\nu_{t}=0.98\pm
0.25~{}\mathrm{\,km\,s^{-1}}$. The LOTUS value is consistent with our
$\log\,g$ from parallax but the Frebel et al. (2013) value is not; this is
expected from $\log\,g$ discrepancies due to using LTE instead of non-LTE
(Ezzeddine et al., 2020).

The effect of stellar parameter uncertainties on the final abundances is
dominated by $T_{\mathrm{eff}}$, and we adopt a conservative uncertainty of
$\pm 200$ K to cover the range of values that we found. We adopt a $\log\,g$
uncertainty of 0.3 dex to include our range of $\log\,g$ values, between the
non-LTE-adjusted Frebel et al. (2013) value and the LOTUS value. Our $v_{t}$
uncertainty is also 0.2 $\mathrm{\,km\,s^{-1}}$, which is consistent with the
Frebel et al. (2013) value and includes the upper end of the best-fit $v_{t}$
range from LOTUS . We adopted a model metallicity $\mathrm{\,[M/H]}=-2.5$ with
an uncertainty of 0.2 dex because the ATLAS model atmosphere grid for
$[\alpha/\mathrm{Fe}]=0.0$ does not extend below $-2.5$. While this value
departs from our final $\mathrm{\,[Fe/H]}$ fit, the dependence of the model
atmospheres on the $\mathrm{\,[M/H]}$ is weak and does not contribute strongly
to our final abundance uncertainty.

###  2.3 LTE Abundances

We conducted an abundance analysis using a similar setup as : Castelli &
Kurucz (2004) model atmospheres and MOOG radiative transfer that assumes local
thermodynamic equilibrium (LTE) but includes scattering (Sneden, 1973; Sobeck
et al., 2011). Linelists for the analysis were adopted from Ji et al. (2020),
with the atomic data originating from linemake (Placco et al., 2021). We fit
equivalent widths and best-fit syntheses in smhr including formal 5$\sigma$
upper limits for undetected lines (see Ji et al. 2020 for details).

We also used TSFitPy (Gerber et al., 2023), which uses Turbospectrum (Plez,
2012) and the standard MARCS model atmospheres (Gustafsson et al., 2008) to
fit selected lines with syntheses. The abundances derived via this method were
consistent with the MOOG abundances for most lines, and we adopt line-by-line
differences between MOOG/ATLAS and Turbospectrum/MARCS as systematic
uncertainties for each line.

We calculated an abundance and an associated error for each detected line in
our analysis. The total error $E_{i,\mathrm{tot}}$ for each line $i$ was
calculated as the quadrature sum of the statistical error of the MOOG fits,
the abundance difference of the MOOG and TSFitPy fits, the abundance
difference that comes from increasing one of $T_{\mathrm{eff}}$, $\log\,g$,
$\mathrm{\,[M/H]}$, and $v_{t}$ by the adopted uncertainties, and a minimum
systematic error of 0.1 dex for any unmodeled effects. The final LTE
abundances are weighted means of the individual line abundances for each
species, using inverse variance weights $w_{i}=1/E_{i,\mathrm{tot}}^{2}$. The
error on the abundance is the uncertainty of the weighted mean,
$\sigma_{\epsilon}=(\sum_{i}w_{i})^{-1/2}$. The uncertainty of the [X/Fe]
ratios, $\sigma_{\mathrm{XFe}}$, differ from $\sigma_{\epsilon}$ because they
account for correlations in X and Fe due to the stellar parameter
uncertainties.

###  2.4 NLTE Corrections

We used TSFitPy to fit our spectrum with both LTE and non-LTE (NLTE) to
determine NLTE corrections for each line. The elements calculated in NLTE and
the pre-computed departure coefficient grids used are: Na (Larsen et al.,
2022), Mg (Bergemann et al., 2017), Ca (Mashonkina et al., 2017; Semenova et
al., 2020), Ti (Bergemann, 2011), Mn (Bergemann et al., 2019), Fe (Bergemann
et al., 2012a; Semenova et al., 2020), Co (Bergemann et al., 2010; Yakovleva
et al., 2020), Ni (Bergemann et al., 2021; Voronov et al., 2022), Sr
(Bergemann et al., 2012b). The NLTE correction for a single line is computed
by taking the difference of the NLTE abundance and the LTE abundance from
TSFitPy. The NLTE corrections for the Aluminum lines were calculated
seperately from Nordlander & Lind (2017). The total NLTE correction uses the
MOOG line weights.

Our final abundances for J$1010+2358$ are the weighted average MOOG abundances
with the weighted average TSFitPy NLTE correction. These final abundances are
listed in Table 1 as $\log\epsilon$, $[\mathrm{X/H}]$, and $[\mathrm{X/Fe}]$
alongside their error and the NLTE corrections (which are already incorporated
into the abundances). We adopt solar abundances from Magg et al. (2022), which
are revised from Asplund et al. (2009).

Table 1: Chemical Abundances

ID | N | log $\epsilon$ | [X/H] | [X/Fe] | $\sigma_{\epsilon}$ | $\sigma_{\mathrm{XFe}}$ | $\Delta_{\mathrm{NLTE}}$
---|---|---|---|---|---|---|---
CH | 2 | $6.10$ | $-2.46$ | $+0.16$ | $0.26$ | 0.21 | $\dots$
O I | 1 | ${<}{8.59}$ | ${<}{-0.18}$ | ${<}{+2.44}$ | limit | limit | $\dots$
Na I | 2 | $2.25$ | $-4.04$ | $-1.42$ | $0.13$ | 0.08 | $-0.03$
Mg I | 4 | $4.30$ | $-3.25$ | $-0.63$ | $0.11$ | 0.08 | $+0.02$
Al I | 2 | $2.70$ | $-3.73$ | $-1.11$ | $0.20$ | 0.15 | $+0.61$
K I | 1 | ${<}{2.93}$ | ${<}{-2.21}$ | ${<}{+0.41}$ | limit | limit | $\dots$
Ca I | 4 | $3.33$ | $-3.04$ | $-0.42$ | $0.08$ | 0.08 | $+0.04$
Sc II | 2 | $-0.18$ | $-3.25$ | $-0.63$ | $0.19$ | 0.21 | $\dots$
Ti I | 3 | $2.28$ | $-2.66$ | $-0.04$ | $0.15$ | 0.10 | $+0.20$
Ti II | 5 | $2.05$ | $-2.89$ | $-0.27$ | $0.07$ | 0.12 | $+0.15$
V I | 1 | ${<}{1.52}$ | ${<}{-2.37}$ | ${<}{+0.25}$ | limit | limit | $\dots$
V II | 1 | ${<}{2.08}$ | ${<}{-1.81}$ | ${<}{+0.81}$ | limit | limit | $\dots$
Cr I | 2 | $2.73$ | $-3.01$ | $-0.39$ | $0.15$ | 0.08 | $\dots$
Mn I | 2 | $2.52$ | $-3.00$ | $-0.37$ | $0.13$ | 0.11 | $+0.23$
Fe I | 73 | $4.88$ | $-2.62$ | $0.00$ | $0.02$ | 0.02 | $+0.09$
Fe II | 6 | $4.78$ | $-2.72$ | $-0.10$ | $0.07$ | 0.16 | $+0.01$
Co I | 2 | $2.18$ | $-2.77$ | $-0.14$ | $0.15$ | 0.09 | $+0.16$
Ni I | 1 | $3.57$ | $-2.67$ | $-0.05$ | $0.20$ | 0.12 | $+0.36$
Zn I | 1 | ${<}{2.64}$ | ${<}{-1.92}$ | ${<}{+0.70}$ | limit | limit | $\dots$
Sr II | 2 | $-1.19$ | $-4.06$ | $-1.44$ | $0.13$ | 0.16 | $+0.07$
Ba II | 1 | ${<}{-1.52}$ | ${<}{-3.70}$ | ${<}{-1.08}$ | limit | limit | $\dots$
Eu II | 1 | ${<}{-0.85}$ | ${<}{-1.37}$ | ${<}{+1.25}$ | limit | limit | $\dots$

NOTE: The Magg et al. (2022) solar abundances were used for normalization.
NLTE corrections are already applied to the $\log\epsilon$ and other abundance
values. [X/Fe] has an additional NLTE correction for [Fe/H] already applied.
We use the species Fe I, Ti II, and V II for SN model fitting.

The final abundances are plotted in Figure 2 as $[\mathrm{X/Fe}]$ in
comparison to the original  abundances and the distribution of abundances for
stars with $-3<\mbox{[Fe/H]}<-1.5$ from the SAGA database (Suda et al., 2008).
We adjust the SAGA aluminum abundances by our adopted NLTE correction since
they were mainly computed in LTE. Compared to the SAGA stars, J$1010+2358$ is
low in magnesium, calcium, and barium. Additionally, we find that the star is
low in sodium, scandium, titanium, and strontium, though not as low as
reported.

Figure 2: [X/Fe] values for J$1010+2358$ against those from  and against SAGA
abundance ranges. The uncertainties on our abundances are
$\sigma_{\mathrm{X/Fe}}$ from Table 1. The SAGA solar normalizations have been
adjusted from Asplund et al. (2009) to Magg et al. (2022). We have applied our
NLTE correction to the SAGA aluminum value. The colors of the element labels
correspond to the elements only measured by us (red), only measured in
(blue), or in both (black).

##  3 SN Yield Fits

We compare our abundances to a grid of 16800 CCSN nucleosynthesis models
(Heger & Woosley, 2010) and 14 PISN models (Heger & Woosley, 2002). We
determine the best-fit model by minimizing the mean absolute deviation between
the abundances of J$1010+2358$ and those of the models for elements $Z=6-30$,
allowing our [X/H] measurements to scale with a dilution parameter. Compared
to a chi-squared ($\chi^{2}$) minimization, this penalizes individual bad
element fits less severely, though we still use the $\chi^{2}$ to
qualitatively visualize the range of well-fitting models. We reject any
solutions that violate an upper limit, though this does not affect our final
best fit.

We first verified that our SN fitting program worked by using the original
abundances to find the well-fitting (within $5\sigma$) models from our grid,
as shown in Figure 3(a). In this plot, we display models with a $\chi^{2}$
value within $5\sigma$ of the optimal value. The best-fitting model is
displayed in gold, whereas the well-fitting models are plotted in black, both
in the Z vs. [X/H] plot and in the progenitor mass histogram. The total search
space of progenitor masses is also plotted in gray in the right plot. The
reduced $\chi^{2}$ is calculated as $\chi^{2}/(\mathrm{d.o.f})$, where the
degrees of freedom is the number of elements fit (excluding upper limits)
minus the number of model parameters (4 for CCSN, 2 for PISN).

With the  abundances, we recover their solution of a 260 $M_{\odot}$ PISN
model as the best-fitting solution but also found some CCSN models among the
well-fitting models. These models are not within $4\sigma$ of the optimal
$\chi^{2}$ fit, so they are not preferred; however, it shows that a CCSN model
could be found to explain their data. This possibility was first pointed out
in , as was the fact that J$1010+2358$’s low $[\mathrm{Mg/Fe}]$ does not
preclude it from being enriched by a CCSN.

With our updated abundances, we find the best-fitting model (reduced
$\chi^{2}=2.8$) to be an 11 $M_{\odot}$ CCSN, as shown in Figure 3(b). As
suggested by , the measurements of aluminum (Z=13) and carbon (Z=6) clearly
distinguish between a CCSN and PISN. However, the rest of our abundances are
consistent with a CCSN model as well, and we recover the same result when we
remove carbon and aluminum from our fits. Even when excluding C and Al, PISN
models are completely excluded from the well-fitting models.

When we restrict our search to just PISN models, as displayed in Figure 3(c),
the best-fitting model is a 220 $M_{\odot}$ PISN. However, this is a lower-
quality fit (reduced $\chi^{2}=28.8$). With the exception of sodium (Z=11) and
magnesium (Z=12), we do not see a strong odd-even pattern in our measured
abundances. The iron peak elements also lack the strong odd-even pattern that
we would expect for a PISN. The difference in conclusions between our analysis
and  stems from disagreements in the abundance measurements, most notably
sodium, calcium (Z=20), scandium (Z=21), titanium (Z=25), and cobalt (Z=27).

(a) Our SN fitting of the abundances from . Their result of a $260M_{\odot}$
PISN is recovered as the best solution, though CCSN models are possible within
$5\sigma$ of their measurements, as first noted by .

(b) The best-fitting model over all surveyed models is an 11 $M_{\odot}$ CCSN
model (Heger & Woosley, 2010). The reduced chi-squared of this fit is 2.8, and
the reduced chi-squareds of the other ”well-fitting” models range between 2.2
and 6.4. PISNe are completely excluded.

(c) The best-fitting PISN model (Heger & Woosley, 2002) compared to our
abundances. The reduced $\chi^{2}$ of this fit is 28.8, which is clearly a
worse fit than the best CCSN model, especially for carbon (Z=6), sodium
(Z=11), aluminum (Z=13), calcium (Z=20), scandium (Z=21), chromium (Z=24), and
cobalt (Z=27).

Figure 3: Our model search. For the left plot in each subfigure, the best
model (gold) is the one that minimizes the mean absolute error from the [X/H]
abundances. This includes minimizing over a dilution parameter, which would
shift the [X/H] of the model up or down. The uncertainties in these plots
correspond to $\sigma_{\epsilon}$ in Table 1. The black “well-fitting” models
are those whose $\chi^{2}$ fall within a $5\sigma$ range of the minimum
$\chi^{2}$ value, and their opacities correspond to their strength-of-fit. Red
squares correspond to the abundances found in this work, and the blue circles
correspond to the abundances of . On the right of each subfigure, the
progenitor masses of all the models are plotted as the grey histogram. The
progenitor masses of the well-fitting models are plotted in black, and the
best-fitting model is plotted in yellow.

##  4 Comparison to Xing et al. 2023

Because it substantially affects the interpretation, we investigated possible
reasons for our abundance differences compared to . First, they adopted
different stellar parameters; most importantly, their $\log\,g$ of $3.6$,
differs from our value of $4.68$. Though our spectroscopic and photometric
parameters disagree with this result, we analyzed our spectrum with their
stellar parameters and in LTE. The discrepancies between our values of
scandium and magnesium are attributable to the different stellar parameters,
but other elements remain inconsistent.

Our uncorrected, LTE abundances have a slightly stronger odd-even effect in
the Fe peak elements; however, repeating our SN fitting procedure yielded the
same result, with the same exclusion of PISNe from the well-fitting models. We
could find no changes in the analysis of our spectrum that would reproduce the
low Na or Si detection reported by .

In particular, our Na D detection is in direct conflict with the clear non-
detection shown in their extended data figure 1. We thus downloaded the data
used in  from the SMOKA archive (Baba et al., 2002) and reduced it in IRAF
(Tody, 1986, 1993; Fitzpatrick et al., 2024) with the HDS routines. The S/N of
the HDS spectrum is lower than the HIRES spectrum, but we clearly see the two
stellar Na D lines, as well as the stronger interstellar absorption components
(Fig 1). The stellar Na lines are also visible on the raw 2D science image,
where we identify some cosmic rays near the Na D lines. We suspect cosmic ray
removal could have impacted the data reduction in . Furthermore, the 2D
inspection shows that the star is not centered on the slit, making sky
subtraction more difficult. As a final check, we measured equivalent widths of
the lines we analyzed in our reduction of the HDS spectrum, finding no
significant differences after accounting for spectrum noise. In any case, our
higher S/N Keck/HIRES spectrum should provide a more reliable abundance
analysis.

##  5 Conclusion

We present a new abundance analysis of the PISN candidate, J$1010+2358$. We
collected a Keck/HIRES spectrum and calculated abundances using both
equivalent width fitting and spectral synthesis fitting. Our new abundances
confirm that the abundance pattern of J$1010+2358$ originated in a core-
collapse supernova, and not a pair-instability supernova, as previously
reported. Consistent with the suggestion of Jeena et al. (2024), our new
carbon and aluminum measurements strongly favor a CCSN explanation over a PISN
explanation. We find other discrepancies in our abundances compared with those
determined by Xing et al. (2023). Our result means that a true PISN candidate
is yet to be found.

## acknowledgments

P.T. and A.P.J. acknowledge support by the National Science Foundation under
grants AST-2206264 and AST-2307599. W.C acknowledges support from a Gruber
Science Fellowship at Yale University. E.N.K. acknowledges support from NSF
CAREER grant AST-2233781. This research has made use of the Keck Observatory
Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA
Exoplanet Science Institute (NExScI), under contract with the National
Aeronautics and Space Administration. Based in part on data collected at
Subaru Telescope and obtained from the SMOKA, which is operated by the
Astronomy Data Center, National Astronomical Observatory of Japan.

The authors wish to recognize and acknowledge the very significant cultural
role and reverence that the summit of Maunakea has always had within the
Native Hawaiian community. We are most fortunate to have the opportunity to
conduct observations from this mountain.

## References

  * Aguado et al. (2023) Aguado, D. S., Salvadori, S., Skúladóttir, Á., et al. 2023, MNRAS, 520, 866
  * Aoki et al. (2014) Aoki, W., Tominaga, N., Beers, T. C., Honda, S., & Lee, Y. S. 2014, Science, 345, 912
  * Asplund et al. (2009) Asplund, M., Grevesse, N., Sauval, A. J., & Scott, P. 2009, ARA&A, 47, 481
  * Baba et al. (2002) Baba, H., Yasuda, N., Ichikawa, S.-I., et al. 2002, Report of the National Astronomical Observatory of Japan, 6, 23
  * Bailer-Jones et al. (2018) Bailer-Jones, C. A. L., Rybizki, J., Fouesneau, M., Mantelet, G., & Andrae, R. 2018, The Astronomical Journal, 156, 58
  * Bergemann (2011) Bergemann, M. 2011, MNRAS, 413, 2184
  * Bergemann et al. (2017) Bergemann, M., Collet, R., Amarsi, A. M., et al. 2017, ApJ, 847, 15
  * Bergemann et al. (2012a) Bergemann, M., Hansen, C. J., Bautista, M., & Ruchti, G. 2012a, A&A, 546, A90
  * Bergemann et al. (2012b) —. 2012b, A&A, 546, A90
  * Bergemann et al. (2010) Bergemann, M., Pickering, J. C., & Gehren, T. 2010, MNRAS, 401, 1334
  * Bergemann et al. (2019) Bergemann, M., Gallagher, A. J., Eitner, P., et al. 2019, A&A, 631, A80
  * Bergemann et al. (2021) Bergemann, M., Hoppe, R., Semenova, E., et al. 2021, MNRAS, 508, 2236
  * Bromm et al. (2001) Bromm, V., Ferrara, A., Coppi, P. S., & Larson, R. B. 2001, MNRAS, 328, 969
  * Brown et al. (2021) Brown, A. G., Vallenari, A., Prusti, T., et al. 2021, Astronomy and Astrophysics, 649, 1
  * Casagrande & VandenBerg (2018) Casagrande, L., & VandenBerg, D. A. 2018, MNRAS, 479, L102
  * Casey (2014) Casey, A. R. 2014, ArXiv e-prints, arXiv:1405.5968
  * Castelli & Kurucz (2004) Castelli, F., & Kurucz, R. L. 2004, New Grids of ATLAS9 Model Atmospheres, , , arXiv:astro-ph/0405087
  * Ezzeddine et al. (2020) Ezzeddine, R., Rasmussen, K., Frebel, A., et al. 2020, The Astrophysical Journal, 898, 150
  * Fitzpatrick et al. (2024) Fitzpatrick, M., Placco, V., Bolton, A., et al. 2024, arXiv e-prints, arXiv:2401.01982
  * Frebel et al. (2013) Frebel, A., Casey, A. R., Jacobson, H. R., & Yu, Q. 2013, ApJ, 769, 57
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Prusti, T., de Bruijne, J. H. J., et al. 2016, A&A, 595, A1
  * Gerber et al. (2023) Gerber, J. M., Magg, E., Plez, B., et al. 2023, A&A, 669, A43
  * Gustafsson et al. (2008) Gustafsson, B., Edvardsson, B., Eriksson, K., et al. 2008, A&A, 486, 951
  * Heger et al. (2003) Heger, A., Fryer, C. L., Woosley, S. E., Langer, N., & Hartmann, D. H. 2003, ApJ, 591, 288
  * Heger & Woosley (2002) Heger, A., & Woosley, S. E. 2002, ApJ, 567, 532
  * Heger & Woosley (2010) —. 2010, ApJ, 724, 341
  * Hirano et al. (2014) Hirano, S., Hosokawa, T., Yoshida, N., et al. 2014, ApJ, 781, 60
  * Jeena et al. (2024) Jeena, S. K., Banerjee, P., & Heger, A. 2024, MNRAS, 527, 4790
  * Ji et al. (2015) Ji, A. P., Frebel, A., & Bromm, V. 2015, MNRAS, 454, 659
  * Ji et al. (2020) Ji, A. P., Li, T. S., Hansen, T. T., et al. 2020, AJ, 160, 181
  * Karlsson et al. (2008) Karlsson, T., Johnson, J. L., & Bromm, V. 2008, ApJ, 679, 6
  * Koutsouridou et al. (2024) Koutsouridou, I., Salvadori, S., & Ása Skúladóttir. 2024, The Astrophysical Journal Letters, 962, L26
  * Larsen et al. (2022) Larsen, S. S., Eitner, P., Magg, E., et al. 2022, A&A, 660, A88
  * Li & Ezzeddine (2023) Li, Y., & Ezzeddine, R. 2023, The Astronomical Journal, 165, 145
  * Magg et al. (2022) Magg, E., Bergemann, M., Serenelli, A., et al. 2022, A&A, 661, A140
  * Mashonkina et al. (2017) Mashonkina, L., Sitnova, T., & Belyaev, A. K. 2017, A&A, 605, A53
  * Mucciarelli et al. (2021) Mucciarelli, A., Bellazzini, M., & Massari, D. 2021, A&A, 653, A90
  * Nomoto et al. (2013) Nomoto, K., Kobayashi, C., & Tominaga, N. 2013, ARA&A, 51, 457
  * Nordlander & Lind (2017) Nordlander, T., & Lind, K. 2017, A&A, 607, A75
  * Placco et al. (2021) Placco, V. M., Sneden, C., Roederer, I. U., et al. 2021, linemake: Line list generator, Astrophysics Source Code Library, record ascl:2104.027, , , ascl:2104.027
  * Plez (2012) Plez, B. 2012, Turbospectrum: Code for spectral synthesis, Astrophysics Source Code Library, record ascl:1205.004, , , ascl:1205.004
  * Salvadori et al. (2019) Salvadori, S., Bonifacio, P., Caffau, E., et al. 2019, MNRAS, 487, 4261
  * Semenova et al. (2020) Semenova, E., Bergemann, M., Deal, M., et al. 2020, A&A, 643, A164
  * Sneden (1973) Sneden, C. A. 1973, PhD thesis, The University of Texas at Austin.
  * Sobeck et al. (2011) Sobeck, J. S., Kraft, R. P., Sneden, C., et al. 2011, AJ, 141, 175
  * Suda et al. (2008) Suda, T., Katsuta, Y., Yamada, S., et al. 2008, PASJ, 60, 1159
  * Takahashi et al. (2018) Takahashi, K., Yoshida, T., & Umeda, H. 2018, ApJ, 857, 111
  * Tody (1986) Tody, D. 1986, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 627, Instrumentation in astronomy VI, ed. D. L. Crawford, 733
  * Tody (1993) Tody, D. 1993, in Astronomical Society of the Pacific Conference Series, Vol. 52, Astronomical Data Analysis Software and Systems II, ed. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes, 173
  * Vogt et al. (1994) Vogt, S. S., Allen, S. L., Bigelow, B. C., et al. 1994, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 2198, Instrumentation in Astronomy VIII, ed. D. L. Crawford & E. R. Craine, 362
  * Voronov et al. (2022) Voronov, Y. V., Yakovleva, S. A., & Belyaev, A. K. 2022, ApJ, 926, 173
  * Xing et al. (2023) Xing, Q.-F., Zhao, G., Liu, Z.-W., et al. 2023, Nature, 618, 712
  * Yakovleva et al. (2020) Yakovleva, S. A., Belyaev, A. K., & Bergemann, M. 2020, Atoms, 8, doi:10.3390/atoms8030034
  * Yoon et al. (2012) Yoon, S. C., Dierks, A., & Langer, N. 2012, A&A, 542, A113