classifier. Notably, the counts go to zero for $H_{\scriptscriptstyle\rm
E}>22.5$, hence there are no target galaxies fainter than this magnitude. This
explains the rapid gain in precision one obtains by simply cutting the full
sample (here shown by the orange histogram) at brighter and brighter values of
$H_{\scriptscriptstyle\rm E}$ (see Table 2). Looking at the other histograms,
we see that the application of the neural network effectively cuts the
distribution down to the correct $H_{\scriptscriptstyle\rm E}$. When using
only Euclid bands (blue histogram), this leaves an excess of sources, which
are either outside the redshift range or below the chosen
H$\alpha+\left[{N\textsc{ii}}\right]$ flux limit, which are significantly
reduced by adding the ground-based information (magenta dashed histogram).
Note also how a selection over the target redshift range $[0.9,1.8]$ using
photometric redshifts clearly does not effectively cut on the
$H_{\scriptscriptstyle\rm E}$ magnitude, leaving a large population of faint
objects. Nevertheless, we remind the reader that this discussion is specific
to Flagship2. In the case of EL-COSMOS, also galaxies fainter than
$H_{\scriptscriptstyle\rm E}\simeq 22.5$ are part of the target sample (see
Fig. 4 and Table 2).

Figure 10: False positive rate as function of redshift and $f_{\rm
H\alpha+\left[{N\textsc{ii}}\right]}$. The plot allows us to identify the
origin of non-targets that enter the selected samples. The solid blue, dashed
orange, and dotted purple curves, respectively, correspond to the neural
networks trained with Euclid photometry, Euclid plus morphological data, and
Euclid plus ground-based photometry. The dash-dotted red line is the false
positive rate of the photo-$z$ selection. _Top_ : False positive rate as a
function of $z$. The green shaded area marks the target redshift range.
_Bottom_ : False positive rate as a function of $f_{\rm
H\alpha+\left[{N\textsc{ii}}\right]}$. The green shaded area corresponds to
the H$\alpha$ limiting flux. There is a peak in the false positive rate just
below the flux limit used to define the target sample, although we note that
these galaxies can still give correct redshift measurements.

We can use the false positive rate (see Sect. 2.1) to interpret the origin of
misclassified galaxies as a function of redshift and emission-line flux. In
the top panel of Fig. 10 this quantity is plotted as a function of redshift.
The sample produced using Euclid photometry alone shows an excess of false
positives at $z<1$. This explicitly shows the inability with only the Euclid
bands to properly exclude low-redshift galaxies, as well as some with flux
below the flux limit. The addition of ground-based photometry effectively
cures this, removing all galaxies at $z<0.9$, leaving only a fraction of
misidentified objects fainter than $$2\text{\times}{10}^{-16}$\,{\rm
erg}\,{\rm s}^{-1}\,{\rm cm}^{-2}$ inside the target redshift range. It is
interesting to note that the Euclid-only and the Euclid plus ground curves
become indistinguishable at $z\gtrsim 1.4$. This is consistent with the
redshift at which the $4000\,\AA$ break enters the $Y_{\scriptscriptstyle\rm
E}$ band (at $9600\,\AA$) and indicates that in this range the combination of
$I_{\scriptscriptstyle\rm E}$ and $Y_{\scriptscriptstyle\rm E}$,
$J_{\scriptscriptstyle\rm E}$, and $H_{\scriptscriptstyle\rm E}$ provides, in
general, sufficient spectral leverage to break degeneracies to both capture
the correct redshift and identify emission-line targets. As also shown, a
photometric redshift selection is effective at removing low-redshift galaxies,
but keeps in the sample all the low-flux galaxies (as is expected, since we
are selecting on redshift alone).

In Fig. 10 bottom panel, instead, we plot the false positive rate as a
function of $f_{\rm H\alpha+\left[{N\textsc{ii}}\right]}$. In this case, the
peak and discontinuity evident at the flux limit,
$$2\text{\times}{10}^{-16}$\,{\rm erg}\,{\rm s}^{-1}\,{\rm cm}^{-2}$, is due
to sources just below the flux limit, which enter the sample as false
positives. Above the flux limit, instead, false positives arise from galaxies
that are outside the redshift range. For this reason, the photo-$z$ selection
gives the lowest false positive rate, followed by the Euclid and ground-based
classification. This does not tell the full story, however. The photo-$z$
selection includes a number of false positives entering the sample at low
fluxes, which are the cause of the very low precision shown by this selection.
The classifier trained with ground-based photometry provides the best solution
by balancing the two conditions of removing objects below the line flux limit
and outside the redshift range.

Complementarily, it is also interesting to look at the true negative rate (Eq.
6) of the whole selected sample, which gives an insight into the fraction of
non-targets removed from the sample. When we select galaxies using Euclid
photometry only, the true negative rate is $87\%$; the combination with
ground-based data increases this metric up to $97\%$. Conversely, the true
negative rate of the photo-$z$ selection is $59\%$. Again, the better
performance of the classifiers in comparison to the photo-$z$ selection
reflects the fact that the latter does not make a selection in the emission-
line limiting flux.

Finally, the machine learning algorithms identify regions in the full colour-
magnitude space with a higher density of targets. In the case of the
classifier trained on Euclid photometry, this is a four-dimensional space. In
Appendix C we present slices through the four-dimensional probability maps
constructed from each classifier, showing how the selection depends on colour.
It is interesting to visualise the boundaries constructed by each classifier.
There is no visible separation between target and non-target galaxies in the
colour planes and the classification algorithms define complex boundaries in
the four-dimensional space. The support vector classifier and the neural
network produce particularly smooth boundaries, while the self-organising map
and tree-based classifiers do not. The irregular boundary is an indication
that the classifier is over-fitting the training set and will not generalise
well. In addition, we verified that the $5\%$ of the targets that we lose by
imposing the $95\%$ recall value are uniformly distributed in colour and are
not part of any particular object class. We note that the lost targets are
mainly faint objects.

##  5 Purity and completeness

The final purity of the spectroscopic sample will depend on the combination of
the photometric information with the selection criteria applied to the
spectroscopic measurements, as described by the flow diagram of Fig. 1. To
provide a concrete, yet preliminary, example, we would like to quantify here
the improvement in the final redshift purity and sample completeness produced
by our photometric selection process. This work is based on a set of simulated
spectra that were processed by the Euclid spectroscopic measurement pipeline
(the SPE processing function). Although the simulated data were not yet fully
realistic, they are nevertheless very useful for understanding how a machine
learning-based photometric classification can aid in the sample selection.
Also, the simulated spectra were built from the EL-COSMOS sample described in
Sect. 3.1.1, which helps in making this test self-consistent. Two-dimensional
spectral images were generated using the FastSpec code based on the spectral
energy distribution and morphological parameters of the galaxies. These images
were convolved with the NISP instrumental point spread function and realistic
noise was added according to the detector model. Multiple exposures were
simulated for each source and stacked with one to four exposures. One-
dimensional spectra were extracted from the images and input to the Euclid
spectroscopic measurement processing function to measure the redshift and
spectral features.

The spectroscopic measurement pipeline carries out a likelihood analysis using
spectral templates to estimate the redshift. It produces a probability
distribution function of the redshift that is typically sharply peaked with a
few primary redshift solutions. The integral of the peak provides a useful
measure of the reliability of the solution. We vary the threshold in this
reliability value to select spectroscopic samples and build the relationship
between redshift purity and completeness, as shown by the SPE solid blue line
in Fig. 11.

In the following discussion we focus on the results from the neural network
classifier applied to the simulated spectroscopic sample. The purity and
completeness values should be taken as indicative of the general trends and
not as accurate forecasts of the pipeline performance. The values depend on
the specific distribution of simulated sources and instrumental configuration.
The target sample is defined as described in Sect. 3.3, using the total flux
of the H$\alpha$ and N II complex.

Figure 11: Redshift purity and sample completeness as a function of
spectroscopic reliability threshold. The solid blue, dashed red, dotted green,
and dash-dotted orange lines respectively correspond to a selection using only
SPE reliability, SPE reliability combined with a photometric classification
based on Euclid data, with the classification that uses Euclid and ground-
based photometry, with the $H_{\scriptscriptstyle\rm E}$ magnitude limit
selection, and with the colour selection in the $(I_{\scriptscriptstyle\rm
E}-H_{\scriptscriptstyle\rm E})$-$H_{\scriptscriptstyle\rm E}$ plane

. In all cases the recall of the photometric classification is set to $95\%$.

Figure 12: Spectroscopic redshift purity and completeness with the addition of
the photometric classification. _Top_ : the curves are colour-coded as a
function of the recall of the photometric classification. The spectroscopic
reliability threshold varies along each curve, while varying the threshold on
the photometric classification probability shifts the curve. The purity
improves as recall increases, reaching a maximum for recall ${\sim}\,95\%$ and
declining after. For better visualisation, the first lines are labelled with
the corresponding recall value. At recall values above $95\%$ the curves are
tightly packed. The solid black line corresponds to $100\%$ recall, while the
dashed line to $95\%$ recall, the value we chose to benchmark our results.
_Bottom_ : redshift purity as a function of the recall of the photometric
classification, fixing the value of sample completeness to $45\%$.

Figure 11 shows how redshift purity versus sample completeness plot improves
when we complement the pure spectroscopic reliability cut selection (blue
solid line) with increasing information provided by the photometric neural
network classifier for the two configurations using Euclid-only or Euclid and
ground-based photometry. The curve corresponding to the
$H_{\scriptscriptstyle\rm E}$ magnitude-limit selection that gives $95\%$
recall (see Table 2) is also plotted together with the curve corresponding to
the colour selection presented in Sect. 4.1. These two curves visually
overlap, but the colour cut curve (dash-dot-dotted purple line) is marginally
higher than the simple magnitude cut curve (dash-dotted orange line). The
figure shows that in the range between $40\%$ and $60\%$ completeness, the
photometric classification improves the redshift purity. For example, at a
fixed value of $45\%$ sample completeness, the classification based on Euclid-
only bands improves the purity by ${\sim}\,20\%$, when we add ground-based
photometry the improvement rises to ${\sim}\,45\%$. The simple
$H_{\scriptscriptstyle\rm E}$ magnitude limit selection, at that same
completeness value, gives an improvement of a few per cent only
(${\lesssim}\,10\%$), evidencing the importance of exploiting all available
photometric information.

To examine the effect of the photometric classification in more detail, in the
top panel of Fig. 12 we show the redshift purity and sample completeness as a
function of the reliability threshold imposed on the spectroscopic redshift
measurement. The photometric classification has its own threshold parameter on
the classification probability, which when combined with the spectroscopic
selection, produces a family of curves. We label these curves based on their
recall values. The bottom panel shows the dependence of redshift purity on the
photometric selection recall, when the completeness is fixed to $45\%$. As we
see, a recall value of $95\%$ approximately maximises the purity-completeness
curve, which justifies the choice made in Sect. 2.1.

The main conclusion from this exercise is that the impact of properly
elaborated photometric information on the final purity and completeness of the
Euclid spectroscopic sample is very significant, with a major improvement
especially when ground-based visible bands are included. The precise gain,
however, will depend on the galaxy distribution, the survey configuration and
the instrument model.

##  6 Conclusions

We have investigated the benefits of combining photometric information with
the spectroscopic measurement criteria for selecting Euclid spectroscopic
samples. Euclid spectroscopy will give estimates of the galaxy redshifts,
fluxes of the emission lines, and confidence intervals. However, since
emission-line galaxies make up only a small fraction of the photometric
sample, measurement noise can reduce the redshift purity and completeness of
the sample and degrade the figure of merit for the galaxy clustering probe.
The addition of photometric criteria in the selection can allow us to improve
the purity of the sample by identifying sources that are likely to be bright
emission-line galaxies at the target redshift.

To this end, we compared a set of machine learning classification algorithms
with the aim of photometrically selecting emission-line target galaxies that
are likely to give good redshift measurements in the Euclid Wide Survey. We
used two catalogues to benchmark the classification performance, EL-COSMOS and
Flagship2. Both catalogues have Euclid and ground-based simulated photometry.
We produced noisy realisations of the catalogues assuming background-limited
observations. The two catalogues yield similar results when using as input
Euclid-only photometry, but when this is combined with ground-based data, the
results using Flagship2 outperform those with EL-COSMOS. This is related to
the differences in the H$\alpha$ luminosity function and colour distribution
of the two catalogues. In addition to these two configurations (Euclid-only
and Euclid plus ground) we also considered adding morphological information
(half-light radius and the axial ratio). We find that in general, while the
addition of ground-based data strongly improves the precision (doubling it in
the case of Flagship2), including morphological information (at least in the
form provided here) gives negligible improvement.

The purity of the final spectroscopic sample will depend on the combination of
the photometric classification with further selection criteria based on the
properties of the spectroscopic data (see diagram in Fig. 1). To investigate
this requires full end-to-end simulations of the spectroscopic reduction
pipeline. We presented a preliminary exercise to assess the relative gain when
the spectroscopic data are complemented by the photometric selection discussed
here. This will be expanded in future work. We showed that in the range
between $40\%$ and $60\%$ completeness the purity is boosted by ${\sim}\,20\%$
when using Euclid-only bands, and between $40\%$ and $100\%$ when including
ground-based photometry. We consider this a remarkable indication.

The introduction of ground-based data significantly improves the purity of the
sample, but in the practical application can also bring additional nuisance in
the form of systematic errors. The ground-based photometry will come from
multiple surveys and so will not be fully homogeneous. It will also suffer
from additional selection effects correlated with the observing conditions
that can propagate as systematic errors to the galaxy clustering measurements
and cosmological constraints. Thus, the gains in purity from incorporating
ground-based data must be carefully weighed against the potential of adding
systematic errors, also considering the specific requirements of the science
analysis to be carried out. We foresee that ground-based data may be used in
analyses where a higher level of purity is desired, such as for studying the
galaxy halo occupation distribution or galaxy evolution as a function of
environment.

Photometric redshifts can also play a key role in sample selection. We used
the Euclid photometric redshift estimates to select galaxies in the target
redshift range and compared the performance of such a selection to that of the
colour-based machine learning classifiers. Figure 10 shows that the photo-$z$
selection is very efficient for redshift classification, especially to remove
low redshift interlopers, but is not effective in identifying emission-line
galaxies. Indeed, the photo-$z$ selection has the highest fraction of false
positives from faint galaxies with $f_{\rm
H\alpha+\left[{N\textsc{ii}}\right]}<$2\text{\times}{10}^{-16}$\,{\rm
erg}\,{\rm s}^{-1}\,{\rm cm}^{-2}$, but the lowest for bright ones with
$f_{\rm H\alpha+\left[{N\textsc{ii}}\right]}>$2\text{\times}{10}^{-16}$\,{\rm
erg}\,{\rm s}^{-1}\,{\rm cm}^{-2}$, which means that it makes a better
redshift selection than the algorithms presented in this work. Photometric
redshifts could be used with additional constraints from spectral energy
distribution fits to identify bright emission-line galaxy targets. In
particular, the Euclid photometric redshift pipeline will output estimates of
galaxy physical properties including the star-formation rate and dust
attenuation, which will allow us to select emission-line galaxy samples. We
expect that a classifier developed based on photometric redshifts and
estimates of physical properties from spectral energy distribution fitting
would perform similarly to the pure colour and magnitude-based classifiers
that we tested, since the underlying photometric information is the same.
Analogously, we expect a classifier trained to make a selection in redshift
alone to perform similarly to the photo-$z$ selection. Alternative classifiers
that use the estimates of galaxy physical properties from the Euclid
photometric redshift pipeline for sample selection will be investigated in a
future work.

It is important to note that in this study some of the complications that will
be present in real Euclid data were not considered. First, we assume an ideal
training set, which is fully representative of the Wide Survey data. In the
actual Euclid Wide Survey, the training set will come from the Deep Fields,
which will total $\sim 50\,\text{deg}^{2}$. Shallow and full-depth photometric
measurements will be available for the Euclid photometry in the Deep Fields;
however, we will only have the full-depth measurements for the ground-based
photometry. As they are currently trained, the machine learning algorithms
learn to classify the targets at a given noise level and it is not necessarily
true that they will be able to generalise their results when trained and
tested on samples with different noise levels. Therefore, if ground-based
photometry is used, it will be necessary to degrade the measurements to match
the noise level in the Wide Survey. Since the ground-based photometry will
come from multiple surveys, this operation will not be simple, and residual
variations in homogeneity in the noise can lead to systematic variations in
the classifier performance.

Moreover, the effective emission-line flux limit will vary across the Wide
Survey due to foreground emission including zodiacal light and scattered
stellar light (Euclid Collaboration: Scaramella et al., 2022). In this study,
we used a fixed flux limit to build the training set of emission-line
galaxies. In practice, this does not impose a sharp flux cut in the measured
sample. However, when developing a classifier on real data, we will be able to
use the Deep Survey to define the training set as the set of galaxies that are
correctly measured by the Euclid pipeline, without imposing any specific
constraints on their physical properties. It will also be possible to
construct a classifier that accounts for variations in the noise level across
the Wide Survey to optimise the sample.

Finally, a further complication that must be considered is contamination from
stars in the galaxy catalogue that can impact the purity. The photometric
classifier can be trained to maximise the precision in the presence of stars.
This work will require us to incorporate a star-galaxy classifier, which is
based both on size and photometric colours. Here, the morphological
measurements will be important.

In the next stage of this work, we will consider the full set of spectroscopic
and photometric selection criteria in order to compute the redshift purity,
sample completeness and ultimately cosmologically relevant figures of merit.
This requires running the spectroscopic reduction pipeline on mock data in
order to produce end-to-end simulations. Such simulations will allow us to
optimise the sample selection criteria, possibly with the use of machine
learning classifiers. With Euclid observations beginning in fall 2023, we will
be able to further tune the selection based on the actual telescope
performance and ultimately construct the spectroscopic galaxy sample that will
be used to test the cosmological model.

###### Acknowledgements.

The Euclid Consortium acknowledges the European Space Agency and a number of
agencies and institutes that have supported the development of Euclid, in
particular the Agenzia Spaziale Italiana, the Belgian Science Policy, the
Canadian Euclid Consortium, the French Centre National d’Etudes Spatiales, the
Deutsches Zentrum für Luft- und Raumfahrt, the Danish Space Research
Institute, the Fundação para a Ciência e a Tecnologia, the Hungarian Academy
of Sciences, the Ministerio de Ciencia, Innovación y Universidades, the
National Aeronautics and Space Administration, the National Astronomical
Observatory of Japan, the Netherlandse Onderzoekschool Voor Astronomie, the
Norwegian Space Agency, the Research Council of Finland, the Romanian Space
Agency, the State Secretariat for Education, Research and Innovation (SERI) at
the Swiss Space Office (SSO), and the United Kingdom Space Agency. A complete
and detailed list is available on the Euclid web site (http://www.euclid-
ec.org). This work has made use of CosmoHub (Carretero et al., 2017; Tallada
et al., 2020). CosmoHub has been developed by the Port d’Informació Científica
(PIC), maintained through a collaboration of the Institut de Física d’Altes
Energies (IFAE) and the Centro de Investigaciones Energéticas,
Medioambientales y Tecnológicas (CIEMAT) and the Institute of Space Sciences
(CSIC & IEEC), and was partially funded by the ”Plan Estatal de Investigación
Científica y Técnica y de Innovación” program of the Spanish government.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., et al. 2015, TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, software available from tensorflow.org
  * Addison et al. (2019) Addison, G. E., Bennett, C. L., Jeong, D., Komatsu, E., & Weiland, J. L. 2019, ApJ, 879, 15
  * Bauer & Kohavi (1999) Bauer, E. & Kohavi, R. 1999, Mach. Learn., 36, 105
  * Boser et al. (1992) Boser, B. E., Guyon, I. M., & Vapnik, V. N. 1992, in Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92 (New York, NY, USA: Association for Computing Machinery), 144–152
  * Breiman (2001) Breiman, L. 2001, Mach. Learn., 45, 5
  * Carretero et al. (2017) Carretero, J., Tallada, P., Casals, J., et al. 2017, in Proceedings of the European Physical Society Conference on High Energy Physics. 5-12 July, 488
  * Comparat et al. (2016) Comparat, J., Delubac, T., Jouvel, S., et al. 2016, A&A, 592, A121
  * Cropper et al. (2016) Cropper, M., Pottinger, S., Niemi, S., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9904, Space Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave, ed. H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, & E. C. Tong, 99040Q
  * Eisenstein et al. (2001) Eisenstein, D. J., Annis, J., Gunn, J. E., et al. 2001, AJ, 122, 2267
  * Euclid Collaboration: Blanchard et al. (2020) Euclid Collaboration: Blanchard, A., Camera, S., Carbone, C., et al. 2020, A&A, 642, A191
  * Euclid Collaboration: Desprez et al. (2020) Euclid Collaboration: Desprez, G., Paltani, S., Coupon, J., et al. 2020, A&A, 644, A31
  * Euclid Collaboration: Gabarra et al. (2023) Euclid Collaboration: Gabarra, L., Mancini, C., Rodriguez Munoz, L., et al. 2023, arXiv:2302.09372
  * Euclid Collaboration: Pocino et al. (2021) Euclid Collaboration: Pocino, A., Tutusaus, I., Castander, F. J., et al. 2021, A&A, 655, A44
  * Euclid Collaboration: Scaramella et al. (2022) Euclid Collaboration: Scaramella, R., Amiaux, J., Mellier, Y., et al. 2022, A&A, 662, A112
  * Euclid Collaboration: Schirmer et al. (2022) Euclid Collaboration: Schirmer, M., Jahnke, K., Seidel, G., et al. 2022, A&A, 662, A92
  * Flaugher (2005) Flaugher, B. 2005, International Journal of Modern Physics A, 20, 3121
  * Freund & Schapire (1995) Freund, Y. & Schapire, R. E. 1995, in Computational Learning Theory, ed. P. Vitányi (Berlin, Heidelberg: Springer Berlin Heidelberg), 23–37
  * Geurts et al. (2006) Geurts, P., Ernst, D., & Wehenkel, L. 2006, Mach. Learn., 63, 3
  * Guzzo et al. (2014) Guzzo, L., Scodeggio, M., Garilli, B., et al. 2014, A&A, 566, A108
  * Hastie et al. (2001) Hastie, T., Tibshirani, R., & Friedman, J. 2001, The Elements of Statistical Learning, Springer Series in Statistics (New York, NY, USA: Springer New York Inc.)
  * Ioffe & Szegedy (2015) Ioffe, S. & Szegedy, C. 2015, arXiv:1502.03167
  * Klambauer et al. (2017) Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. 2017, arXiv:1706.02515
  * Kohonen (1982) Kohonen, T. 1982, Biological Cybernetics, 43, 59
  * Kohonen (1990) Kohonen, T. 1990, Proceedings of the IEEE, 78, 1464
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, arXiv e-prints, arXiv:1110.3193
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., & Hinton, G. 2015, Nature, 521, 436
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. 2017, arXiv:1708.02002
  * LSST Science Collaboration et al. (2009) LSST Science Collaboration, Abell, P. A., Allison, J., et al. 2009, arXiv:0912.0201
  * Maciaszek et al. (2022) Maciaszek, T., Ealet, A., Gillard, W., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave, ed. L. E. Coyle, S. Matsuura, & M. D. Perrin, 121801K
  * Moosavi et al. (2014) Moosavi, V., Packmann, S., & Vallés, I. 2014, SOMPY: A Python Library for Self Organizing Map (SOM), gitHub.[Online]. Available: https://github. com/sevamoo/SOMPY
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., et al. 2011, Journal of Machine Learning Research, 12, 12
  * Pozzetti et al. (2016) Pozzetti, L., Hirata, C. M., Geach, J. E., et al. 2016, A&A, 590, A3
  * Raichoor et al. (2023) Raichoor, A., Moustakas, J., Newman, J. A., et al. 2023, AJ, 165, 126
  * Ross et al. (2011) Ross, A. J., Ho, S., Cuesta, A. J., et al. 2011, MNRAS, 417, 1350
  * Rowe et al. (2015) Rowe, B. T. P., Jarvis, M., Mandelbaum, R., et al. 2015, Astronomy and Computing, 10, 121
  * Saito et al. (2020) Saito, S., de la Torre, S., Ilbert, O., et al. 2020, MNRAS, 494, 199
  * Scoville et al. (2007) Scoville, N., Aussel, H., Brusa, M., et al. 2007, ApJS, 172, 1
  * Stanford et al. (2021) Stanford, S. A., Masters, D., Darvish, B., et al. 2021, ApJS, 256, 9
  * Tallada et al. (2020) Tallada, P., Carretero, J., Casals, J., et al. 2020, Astronomy and Computing, 32, 100391
  * Tanaka et al. (2018) Tanaka, M., Coupon, J., Hsieh, B.-C., et al. 2018, Publications of the Astronomical Society of Japan, 70, S9
  * Weaver et al. (2021) Weaver, J. R., Kauffmann, O., Shuntov, M., et al. 2021, in American Astronomical Society Meeting Abstracts, Vol. 53, American Astronomical Society Meeting Abstracts, 215.06

##  Appendix A Colour-magnitude projection planes

Figure 13: Target and non-target distributions in colour-colour and colour-
magnitudes planes for the Flagship2 catalogue. The contours contain $99\%$,
$50\%$, and $25\%$ of the samples.

We present in Fig. 13 the colour-colour and colour-magnitude distributions for
targets and non-targets in the Flagship2 catalogue for three different
combinations. The contours contain $99\%$, $50\%$, and $25\%$ of the samples.
There is a nearly complete overlap of the targets and non-targets in the
colours.

##  Appendix B Photo-$z$ as input variables

Figure 14: Comparison of the precision versus recall curves of two neural
networks trained with and without photo-$z$s as an input feature. The two
neural networks were trained with Euclid and ground-based photometry, but in
the case of the solid blue line the algorithm takes the photo-$z$ of the
galaxy as an additional feature. The solid vertical line corresponds to $95\%$
recall.

As an additional test, we trained a neural network with Flagship2 data in the
Euclid plus ground-based configuration with the additional information of the
measured photo-$z$. In principle, the neural network can extrapolate the
redshift from the photometric information. However, by directly providing the
photo-$z$ we may facilitate the selection process as the network will be able
to put more attention on the emission line flux.

The precision at $95\%$ recall is $47.9\%$ in the case without photo-$z$, as
reported in Table 3. When we add the photo-$z$ of the galaxy as an input
feature the precision rises to $50.1\%$. Nevertheless, the addition of the
photo-$z$ to the input information makes the classifier dependent on the
complex process used to produce the photo-$z$s, including the training sets,
spectral energy distribution models, and algorithms used. We postpone to a
future work the detailed study of these dependencies and the performance on
realistic data.

##  Appendix C Selection probability maps

Figure 15: Probability maps in colour-colour and colour-magnitude planes, for
the six classifiers tested in this paper, trained using Flagship2 Euclid
photometry only. The thick white contour marks the probability threshold that
gives 95% recall.

Figure 15 gives a visualisation of the selection probability for each
classifier in planes through the parameter space. We show the results from the
Flagship2 catalogue for the case when classifiers are trained with Euclid
photometry alone. Each row shows the colour-colour and colour-magnitude plots
for a given algorithm. The parameter space is four-dimensional, and the two-
dimensional planes are made by fixing two of the parameters to their median
values.

One notices that the different classifiers identify a similar region of
maximum probability for a given pair of features. The shape and the gradients
of these regions, however, vary for each algorithm. This is due to the
differences in the selection algorithms and possible projection effect when
the boundaries are represented on the planes. In the case of the single
classifiers (top three rows: self-organising map, neural network, and support
vector classifier) they are compact and well defined, unlike for the cases of
voting classifiers based on decision trees (bottom three rows). Also, the
contours and gradients are less smooth for the self-organising map than for
the neural network and the support vector classifier. The probability gradient
of the support vector classifier is very steep, especially in comparison to
the neural network.

The probability maps for the voting classifiers (bottom three rows) show
orthogonal contours. This is due to the common base classifier of these
algorithms, the decision tree, which tends to produce decision rules
orthogonal to one another. At the same time, the three algorithms have very
different probability contours. These differences are related to the batch
selection rule used to train the decision trees (see Sect. 2). We expect that
the algorithms that give a classification model with irregular and steep
contours (such as the self-organising map) or stepped contours (such as the
decision trees) will be prone to over-fitting and will show poorer performance
than algorithms that give smooth probability contours.


# Contrastive Neighborhood Alignment

Pengkai Zhu
Amazon/AWS AI equal contribution Zhaowei Cai11footnotemark: 1
Amazon/AWS AI Yuanjun Xiong11footnotemark: 1
Amazon/AWS AI Zhuowen Tu11footnotemark: 1
Amazon/AWS AI Luis Goncalves
Amazon/AWS AI Vijay Mahadevan
Amazon/AWS AI Stefano Soatto
Amazon/AWS AI

###### Abstract

We present Contrastive Neighborhood Alignment (CNA), a manifold learning
approach to maintain the topology of learned features whereby data points that
are mapped to nearby representations by the source (teacher) model are also
mapped to neighbors by the target (student) model. The target model aims to
mimic the local structure of the source’s representation space using a
contrastive loss. CNA is an unsupervised learning algorithm that does not
require ground-truth labels for the individual samples. CNA is illustrated in
three scenarios: manifold learning, where the model maintains the local
topology of the original data in a dimension-reduced space; model
distillation, where a small student model is trained to mimic a larger
teacher; and legacy model update, where an older model is replaced by a more
powerful one. Experiments show that CNA is able to capture the manifold in a
high-dimensional space and improves performance compared to the competing
methods in their domains.

##  1 Introduction

In this paper, we present a new algorithm, Contrastive Neighborhood Alignment
(CNA), that preserves the local topology of learned features between source
and target models by mapping neighbors in one space (source) to neighbors also
in the target (student) space. CNA overcomes the optimization challenge in the
formulation of traditional manifold learning methods for large-scale computing
by designing a contrastive loss that can be trained effectively to preserve
the local topology of the feature space. The model learned through CNA is
inductive and hence can generalize to novel data. CNA is an unsupervised
approach and requires no ground-truth labels. Fig. 1 illustrates the local
topology preservation by CNA, that the neighbors ($x_{5},x_{7},x_{8}$) of a
sample $x_{1}$ (Fig. 1 left) in the source space is well-preserved in the
target model space trained by CNA(Fig. 1 right).

Figure 1: The local topology of feature space in source (left) and target
(right) models. CNA preserves the local structure between models, such that
the neighbors ($x_{5},x_{7},x_{8}$ for $x_{1}$) from the source model stay the
same in the target model.

CNA does not impose constraints on source models and it can be adopted to
performing student-teacher learning for tasks such as knowledge distillation
(KD) [16] and reducing model regression in model updating [51], beyond the
standard manifold learning tasks [34, 9, 42]. Conventional knowledge
distillation and student-teacher (S-T) learning [16, 47, 1, 7] approaches
primarily follow the design principle of matching features or classification
distributions between the teacher and student models [47]. Despite the
effectiveness of the feature/distribution matching criterion, S-T learning
methods are not without limitations. Typically, student-teacher training tries
to force the student to mimic the input-output behavior of the teacher, but
the regularization term (model bias) often results in a direct conflict with
the cross-entropy (CE) loss, i.e., KL-divergence/logit-matching vs. softmax.
This has two problems. First, the criterion trades off error rate
minimization. Second, the criterion is too rigid in that it mimics the
teacher’s predictions without the awareness of the data manifold. In contrast,
CNA tackles the student-teacher learning problem by modeling the knowledge in
the models using the local structures in their feature space, which is
distinct from the existing methods in model distillation [16].

CNA is a new inductive principle: instead of mimicking the input-output
behavior, it transfers the local neighborhood structure. Neighbors remain
neighbors, when transferring from the source to the target, regardless of the
label. This has two distinct advantages: first, it does not require ground
truth labels. Second, it allows more flexibility to deviate from the inference
of the source model when it is mistaken. In CNA, we impose that samples are
neighbors in the source (teacher) representation space remain neighbors in the
target (student) representation space. This alignment of local neighborhoods
is achieved by adding to the cross-entropy loss a contrastive term that is in
the same form (softmax function) as the cross-entropy term, making the
optimization well conditioned; as both the cross-entropy and the contrastive
loss are in similar unit spaces (normalized softmax function), balancing the
two terms becomes relatively easy. The contributions of our work are listed
below:

  * •

We propose a new method for manifold learning that maintains the local
neighborhood in inductive models.

  * •

We introduce an instance-level contrastive loss for preserving the
neighborhood structures that is different from how contrastive loss is
motivated and implemented previously [49, 30, 15].

  * •

We propose to use a new loss function in Student-Teacher training, which is
distinct from the existing methods in this field.

The effectiveness of CNA is demonstrated on three tasks, 1) dimensionality
reduction for manifold learning [45], 2) model distillation [16], and 3) model
update regression minimization [51]. CNA is capable of capturing the manifold
from the source space and can be applied to various downstream tasks in
knowledge distillation.

##  2 Related work

Manifold/metric learning: Preserving local neighborhood structure is a popular
direction in machine learning that has been explored in both metric learning
[52, 8, 26, 17] and manifold learning [34]. Standard manifold learning
approaches perform dimensionality reduction by mapping the feature
representation from the original space to the new one [45]. They attempt to
maintain the local structure through the reconstruction by the convex
combination of same neighbors [34, 9], by retaining the neighborhood distances
while unfolding the manifold [48], or by preserving pairwise geodesic
distances between all data points [42]. These manifold learning methods learn
the manifold transductively, which limits their deployments in modern deep
networks. In contrast, Our CNA adopts an instance-level contrastive loss that
attempts to preserve the manifold structure of feature space in an inductive
model, which is significantly different from these works. Additionally, CNA
can be combined with classification loss (e.g. a cross-entropy loss) to create
classifiers when training deep models.

Contrastive learning and self-supervised learning: Contrastive learning [12]
is used for either supervised [41, 37, 20] or unsupervised [49, 6, 14]
representation learning. It has recently become popular in self-supervised
learning through the instance discrimination pretext task [49]. Contrastive
representation distillation (CRD) [43] extends the idea of contrastive
learning to the teacher-student training case. Though not directly minimizing
a certain distance function, CRD still needs to calculate the distance between
student’s feature vectors and that of the teacher. CNA uses the contrastive
loss form in teacher student training but does not measure the “cross model
distance”. It only concerns the local structures within the models’ feature
spaces. The supervised contrastive learning method [20] is also different from
CNA since we focus on preserving the local neighborhoods between models as
opposed to using class labels to learn visual representation.

Model distillation: There has been a significant amount of work to study how
knowledge can be shared between multiple models. Model distillation [16, 47]
transfers knowledge from a larger, more powerful teacher model to a smaller
student model by matching their output/representation. CNA does not rely on
matching outputs [16, 51] or reusing the old models’ parameters [38]. Instead,
knowledge is shared between models through the information encoded in the
neighborhood structure of a model’s feature space.

Regression minimization for model upgrading: The presence of new errors that
were not manifest when using the old model (negative flips) can cause a
perceived reduction of performance by users, referred to in the industry as
“regression.” Although progress in the design of the architecture of backbone
models [40, 39, 15, 50, 18, 19, 46], new optimization and regularization
schemes [24, 10, 4], and the availability of new datasets [35, 25, 28, 3, 33,
22] have driven a fast-paced reduction of the average error rate (AER) of
image classifiers, regression is still a major obstacle to the deployment of
improved models. Even a modest number of negative flips can nullify the
benefit of a large decrease in AER. Overall, the problem of reducing the
negative flip rate (NFR), along with the average error rate (AER), has been
referred to as Positive-Congruent Training, or PC-Training [51]. Minimizing
model regression during model upgrading is an emerging problem that is of
great practical importance [51]. In [51], NFR is proposed as the metric to
measure regression in classification model updates and an approach based on
focal distillation, a variant of model distillation [16], is proposed to
reduce regression. Our work addresses the same problem through contrastive
neighbor alignment instead of relying on model distillation.

##  3 Contrastive Neighborhood Alignment

Figure 2: Qualitative results of manifold learning on synthetic datasets.
Rows(top to bottom): S-curve, Swiss Roll, Sphere. Columns(left to right):
original 3D data points, transductive methods (Isomap [42], LLE [34], Hessian
LLE [9]), inductive methods (MVU [48], CNA).

Given a possibly pretrained source model $S$ and a dataset
$\mathcal{D}=\\{x_{i}\\}$, it is a common task to transfer information from
the source model $S$ to a newly trained target model $T$. Let $f_{S}(\cdot)$
and $f_{T}(\cdot)$ denote the feature representation of the source and target
model, respectively. The two models could have different architectures or
different feature dimensions. For example, in manifold learning [45], the
high-dimensional source feature space is cast to a target space in which the
low-dim manifold is unveiled. In model distillation, knowledge is shared
between models by minimizing the sample-wise pseudo-distance between the two
models’ classification posterior probabilities. In this work, we propose CNA,
a contrastive loss that transfers information between models by preserving of
the local structures of the source model feature space in that of the target.
An illustration of this idea is shown in Fig. 1. Below, we detail the
derivation of the CNA loss function and its applications in model
distillation/update tasks.

###  3.1 Preserving Local Structure

The idea of replicating the local structure in a new feature space has been
explored in the manifold learning literature [45]. For example, Isomap [42]
preserves pairwise geodesic (or curvilinear) distances between samples. The
geodesic distances are computed by constructing a neighborhood graph in which
every sample $x_{i}$ is connected with its top-$K$ nearest neighbors. Locally
linear embedding (LLE) [34] studied the local structure in the non-linear
feature space. It has shown that by embedding high-dimensional feature vectors
in their local neighborhoods, we can obtain effective low-dimensional
embedding of the data points. LLE assumes each data point and its neighbors
lie on a locally linear patch in the source space, and characterizes the local
structure by a linear reconstruction using the neighbors:

$\epsilon(W)=\sum_{i}\|f_{S}(x_{i})-\sum_{j}W_{i,j}f_{S}(x_{j})\|_{2}^{2}$ (1)

where $f_{S}(x_{i})$ is the feature representation of $x_{i}$ in the source
space, $\epsilon$ is the reconstruction error and $W$ is the coefficient
matrix. $W_{i,j}$ is enforced to 0 if $x_{j}$ is not one of the top-$K$
neighbors of $x_{i}$ and $\sum_{j}W_{i,j}=1$. The solution
$W^{*}=\operatorname*{arg\,min}\epsilon(W)$ is then used to obtain the target
representation by minimizing the reconstruction error in the target space.
Maximum Variance Unfolding (MVU) [48] is another work that focuses on
retaining pairwise distances in a neighborhood graph. It maximizes the sum of
distances between all data points in the target space:

$\max\sum_{i,j}d(f_{T}(x_{i}),f_{T}(x_{j})),\\\ $ (2)

under the constraint that the distances between the neighbors are preserved:
$d(f_{T}^{(i)},f_{T}^{(j)})=d(f_{S}^{(i)},f_{S}^{(j)})$ if $x_{j}$ is one of
the top-$K$ neighbors of $x_{i}$. Here $d(\cdot,\cdot)$ is the $\ell_{2}$
distance.

However, these manifold learning methods are difficult to be employed in
modern deep learning frameworks. They often need to store a local structure
graph for the entire dataset, which occupies a lot of memory. Inferring the
representation for each data point is also not viable when the dataset is
large. The features are assumed to live in the same metric space defined by
the $\ell_{2}$ distance function, which is often not the case for highly non-
linear deep neural networks. The generalization to novel data is also
difficult due to the transductive nature of these methods (see discussions in
Sec. 3.2).

Inspired by these approaches, and to remedy these issues, CNA is proposed to
preserve the local structure in the target model’s feature space by enforcing
the same neighbors from the source space.

###  3.2 Contrastive Neighborhood Alignment Loss

We hypothesize that the knowledge can be shared between two models when the
nearest neighbors of $x_{i}$ in the source model’s feature space are still its
nearest neighbors in the target model’s feature space. For simplicity, we can
set $K=1$ in Eq. 1, resulting in each sample’s feature $f_{T}(x_{i})$ being
represented only by its nearest neighbor $f_{T}(x_{i^{\star}})$ in the feature
space, with $W_{ii^{\star}}=1$. Then the learning objective can make the
target model maintain the local structure encoded by $W$ in its own feature
space. By resorting to instance-based contrastive loss, we propose the
contrastive neighborhood alignment (CNA) loss that achieves the goal of
matching the same neighbors in an easy to optimize manner.

Specifically, for each sample $x_{i}$ in a batch of $M$ samples, we locate its
nearest neighbor $\aleph(x_{i})$ in the feature space of the source model:

$\aleph(x_{i})=x_{i^{\star}},$ (3)

where

$i^{\star}=\operatorname*{arg\,max}_{j=1,\ldots,M,j\neq
i}\frac{f_{S}(x_{i})\cdot
f_{S}(x_{j})}{|f_{S}(x_{i})|_{2}\cdot|f_{S}(x_{j})|_{2}},$ (4)

where $f_{S}(\cdot)$ is the feature representation of the source model and
$|\cdot|_{2}$ is the $\ell_{2}$ norm of the feature. The nearest neighborhood
is the indicator we use for the model local topology. And the contrastive
neighborhood alignment loss is designed to encourage the target model to
maintain the same local neighborhood structure,

$\ell_{\mathrm{CNA}}(x_{i})=-\log\frac{\exp[f_{T}(x_{i})\cdot
f_{T}(\aleph(x_{i}))/\tau]}{\sum_{j\neq i}^{M}\exp(f_{T}(x_{i})\cdot
f_{T}(x_{j})/\tau)},$ (5)

where $\tau$ is the temperature, and $f_{T}(\cdot)$ the target feature
extractor and its output is assumed normalized. Instead of directly minimizing
the distance between neighbors, in Eq. 5, the neighbor pairs defined by the
source model are pulled together but the non-neighbor pairs will be pushed
away during the target model training. In this way, the local structure is
preserved, and the knowledge encoded in the structure is transferred to the
new model.

The contrastive loss has the same softmax form as the cross-entropy loss, and
this helps when both losses need to be jointly optimized. More importantly,
there is no longer a direct requirement to match the source and target
outputs. This relaxation enables the CNA to be applicable even when the target
model is trained with different loss functions or has different feature
dimensions.

While the CNA loss of Eq. 5 only considers the nearest neighbor, it can be
easily generalized to top-$K$ nearest neighbors,

$\ell_{\mathrm{CNA}}^{K}(x_{i})=\frac{1}{K}\sum_{\tilde{x}\in\aleph^{K}(x_{i})}-\log\frac{\exp[f_{T}(x_{i})\cdot
f_{T}(\tilde{x})/\tau]}{\sum_{j\neq i}^{M}\exp(f_{T}(x_{i})\cdot
f_{T}(x_{j})/\tau)},$ (6)

where $\aleph^{K}(x_{i})$ is the top-$K$ nearest neighbors of sample $x_{i}$.
This formulation simply averages the CNA losses of multiple neighbors.

_Transductive vs. Inductive:_ Traditional manifold learning methods are
transductive as they infer the target representation for each data point
directly by optimizing the objective. It is advantageous in capturing the
manifold of training data since the projection is not limited by any function.
But they are hard to generalize to novel data as it requires modification of
the original graph. Our CNA is fundamentally different as we learn an
inductive target model by making the new model mimic the local structure of
the source representation space. In experiments, we show that CNA can also be
used to learn an inductive model, and it is non-trivial to adapt the
transductive method into an inductive model.

###  3.3 Model Distillation/Update

By distilling the knowledge of the source (teacher) model to the target
(student) model, the target model can achieve higher accuracy [16] or maintain
consistency when the model is updated [51]. There are various definitions of
knowledge which leads to different approaches to transfer it. The objective
can be to minimize the sample-wise feature distance between two models with
respect to a feature distance function, usually $\ell_{2}$-distance [36].
Recent works also proposed to use the contrastive loss [20] instead of
minimizing absolute distance. Our CNA can also be used in these tasks, by
carrying the knowledge of the teacher model to the student model by preserving
the local structure of the feature space.

Let us consider a classification problem. Let $y_{i}\in\\{1,2,\ldots C\\}$
denote the class label for each data sample $x_{i}$. $h(\cdot,\cdot)$ is the
classifier following the feature extractor $f(\cdot)$ to predict the class
label, $\hat{y}=\arg\max_{y}h(f(x),y)$. In order to learn the full target
classification model $\\{f_{T},h_{T}\\}$ capable of preserving the local
topology of the teacher model, the final objective is defined as the weighted
sum of the standard cross-entropy loss and the CNA loss:

$\mathcal{L}_{\mathrm{CNA}}=\frac{1}{M}\sum_{i}^{M}\left[\ell_{CE}(x_{i},y_{i})+\lambda\ell_{\mathrm{CNA}}(x_{i})\right],$
(7)

where $\lambda$ is the trade-off coefficient, and $\ell_{CE}$ is the cross-
entropy loss,

$\ell_{CE}(x_{i},y_{i})=-\log\frac{\exp(h_{T}(f(x_{i}),y_{i}))}{\sum_{c}\exp(h_{T}(f(x_{i}),y_{c}))},$
(8)

where $y_{c}=1,2,\ldots,C$ refers to each class in the datasets. Note that the
prediction of the source model $h_{S}(\cdot,\cdot)$ is not present in the loss
function Eq. 7, unlike the distillation term in knowledge distillation methods
[16]. Thus, the two terms in Eq. 7 stay congruous even if the source model
makes mistakes. As stated before, another major advantage of our proposed CNA
loss is the formulation consistency between the two terms, and balancing the
two terms in the loss function is relatively easy.

##  4 Experiments

We first evaluate the performance of the proposed CNA method on the
unsupervised dimensionality reduction task and compare to other manifold
learning techniques. CNA is then evaluated on two real-world application
tasks: model distillation (from a large to small model) and model update (from
a small to a large model).

###  4.1 Manifold Learning/Dimensionality Reduction

In a dimensionality reduction task, the goal is to learn a low-dimensional
embedding that captures the manifold in the high-dim data space and favors
downstream classifications. The source model $f_{S}(\cdot)$ reduces to an
identity function and the target model $f_{T}(\cdot)$ is a projector that maps
data to a lower dimensional space. We perform the experiments on synthetic
datasets and natural datasets. On synthetic datasets, we show qualitative
results to demonstrate that CNA is capable of discovering the manifold in a
low-dimensional space. On natural datasets, we report quantitative results to
measure the quality of the learned embeddings. In this task, CNA is employed
in an unsupervised fashion where the cross-entropy loss in Eq. 7 is absent.

Baselines: We compare to three transductive manifold learning methods: Isomap
[42], LLE [34] and Hessian LLE [9]. We use CNA to learn an inductive function
through neural networks on both synthetic and real-world data. We also propose
an inductive baseline trained by a modified MVU [48] objective. Specifically,
this baseline uses the same inductive network as CNA and is trained by:

$\ell_{MVU}=\frac{1}{M}\sum_{i}^{M}\Big{\\{}\frac{1}{K}\sum_{j:x_{j}\in\aleph^{K}(x_{i})}^{M}[d(f_{S}(x_{i}),f_{S}(x_{j}))\\\
-d(f_{T}(x_{i}),f_{T}(x_{j}))]^{2}-\gamma\sum_{\begin{subarray}{c}j:j\neq i\\\
x_{j}\notin\aleph^{K}(x_{i})\end{subarray}}d(f_{T}(x_{i}),f_{T}(x_{j}))\Big{\\}}$
(9)

where $\gamma$ is a balance weight. Basically, this loss function will push
non-neighbor data points far away while keeping the distance between neighbors
in the target space. During the experiments, we found a small $\gamma$ is
necessary (1e-6 e.g.) to avoid trivial solutions where every sample is
isolated.

Implementations: For Isomap, LLE and Hessian LLE, the only free parameter is
the number of neighbors $K$. We set it to 10 on synthetic data and sweep over
[5, 10, 50, 100, 200] on real-world datasets. For MVU and CNA, we use a
3-layer multi-layer perceptron (MLP) with Tanh activation function as the
inductive projection model. All networks are trained by Adam optimizer[21]
with batch size 256. We fine-tuned the learning rate, number of neighbors $K$,
temperature $\tau$ and $\gamma$ in Eq. 9 on a held-out validation set.

####  4.1.1 Synthetic Data

Table 1: Quantitative results in % on real-world data. Local-err: local error
on training set; 5-NN Acc: generalization accuracy on test set using a 5-NN
classifier trained on the low-dimensional space. The inputs are projected to
40-dim space. *:Hessian LLE uses a 10-dim space due to the constraint on
available neighbors [48]. $\dagger$: CNA can reduce training error but with
lower test accuracy.

Methods | Tr./Ind. | MNIST | CIFAR10
---|---|---|---
Local-err | 5-NN Acc | Local-err | 5-NN Acc
Isomap | Tr. | 9.3 | 91.2 | 4.2 | 88.7
LLE | Tr. | 10.0 | 91.3 | 3.4 | 88.9
Hessian* | Tr. | 15.4 | 88.2 | 4.1 | 88.9
MVU | Ind. | 34.1 | 72.3 | 3.9 | 88.2
CNA | Ind. (ours) | 8.7 | 92.3 | 3.9 | 89.3
CNA$\dagger$ | Ind. (ours) | - | - | 3.2 | 88.7

We performed experiments on three synthetic datasets: S-Curve, Swiss Roll, and
Sphere. Data points lie on a two-dimensional manifold in the original 3D
space. The plots of the datasets are shown in the left column of Fig. 2. All
datasets contain 2000 samples. The learning is considered good if the manifold
is unfolded and the local structure is retained in the projected 2D space. In
the experiments, the MVU and CNA networks have an intermediate layer with 5
nodes and hence 60 total parameters. All networks are trained for 2000 epochs.
We visualize the learned manifolds for all the methods in Fig. 2.

The results show that the proposed CNA is capable of unfolding the manifold on
all three synthetic datasets. The data points with similar colors stay
neighbors in the projected space. Hence the local topology is well maintained.
Transductive manifold learning methods can also unfold the manifold. However,
on Swiss Roll dataset, the inductive MVU method fails to retrain the manifold
even though we extensively fine-tuned the parameters. This indicates the non-
triviality of achieving manifold learning using inductive networks. Notice
that although the reconstructed manifolds of CNA for Swill Roll have a non-
linear warping, they are not considered poor as the local structure of the two
manifolds is identical to that of Isomap and Hessian LLE. Morever, compared to
trasductive methods, CNA can learn the manifold using an inductive function
with a handful parameters.

####  4.1.2 Real-world Data

We conducted experiments on two real-world datasets: MNIST [27] and CIFAR10
[23]. On MNIST we use the original data (784-dim) as source space, and on
CIFAR10 we use the perceptual features (512-dim) of a pretrained ResNet18
model as the source data. For computational efficiency, we randomly select
4000 samples as training data and 1000 samples as test data. The intermediate
layer of the MVU and CNA networks has 512 nodes. The networks are trained by
1000 epochs. The data is projected to a 40-dim space except for Hessian LLE
because the number of neighbors required is proportional to the square of
dimension [48]. The baseline transductive methods are more competitive, not
less, as we found dim=40 works the best for most transuctive methods.

Metrics: We evaluated two metrics: local error and 5-NN generalization
accuracy [45]. Local error is the rate of samples in the training set of which
the nearest neighbor is not from the same class. It measures whether the
neighbors are maintained in the low-dimensional space. 5-NN generalization
accuracy is defined as the accuracy on the test set using a 5-neareast
neighbor classifier trained on the projected space. It reflects the quality of
the learned data representation regarding the downstream classification task.

The quantitative results on real-word datasets are tabulated in Tab. 1. The
proposed CNA method shows competitive results across the board. It outperforms
all other methods on MNIST and achieves the best 5-NN accuracy on CIFAR10. CNA
has higher training local error on CIFAR10. We argue that CNA focuses on
broader local topology rather than the nearest neighbor, in order to achieve
better generalization when the data is more complex. It can reach lower
training error with a sacrifice on the test accuracy (see CNA$\dagger$ in Tab.
1). Based on these observations, we conclude that CNA can effectively conduct
manifold learning with an inductive function.

###  4.2 Real-World Applications

We evaluated the proposed CNA method on two real-world tasks: model
distillation and model update. In both tasks, information from one model is
expected to transfer to another model, and our CNA method can be applied by
considering the teacher (old) model as source and the student (new) model as
target in model distillation (update). ImageNet [35] and CIFAR100 [23] are the
two major datasets used for evaluation. The used network architectures are the
standard ResNet variants [15] on ImageNet, and ResNet [15] and ShuffleNet [53,
29] on CIFAR100 following [43].

Training Details: For CIFAR100 (ImageNet), the base initial learning rate was
0.2 (0.1) for batch size of 256, linear scaling [11] was used for other batch
sizes, and cosine learning rate decay schedule was adopted. Weight decay was
set as 0.0001 for all ImageNet experiments, but it is sensitive and we
searched the best one from $\\{0.0001,0.0002,0.0005\\}$ for CIFAR100. SGD was
used for optimization in all experiments. For CNA, we set $\lambda=1.0$ in Eq.
7 and temperature $\tau=0.01$ in Eq. 5 for all experiments. Each experiment of
CIFAR100 (ImageNet) is conducted on 4 (8) V100 GPUs for about 1 (15) hour(s).
The results were averaged on 5 runs for all CIFAR100 experiments. More details
will be introduced in the following section.

Table 2: The model distillation results (top-1 accuracy in %) on CIFAR100. *
denoted results run by us. The other results of the competitors are from [43].

Teacher (Source) | ResNet56 | ResNet110 | ResNet110 | ResNet32x4 | ResNet32x4 | ResNet32x4
---|---|---|---|---|---|---
Student (Target) | ResNet20 | ResNet20 | ResNet32 | ResNet8x4 | ShuffleNetV1 | ShuffleNetV2
Teacher (Source) | 72.90 | 74.25 | 74.25 | 79.67 | 79.67 | 79.67
Student (Target) | 69.96 | 69.96 | 71.26 | 72.68 | 70.98 | 71.76
KD* [16] | 71.82 | 71.33 | 73.81 | 73.55 | 75.05 | 75.54
SP [44] | 69.67 | 70.04 | 72.69 | 72.94 | 73.48 | 74.56
CC [32] | 69.63 | 69.48 | 71.48 | 72.97 | 71.14 | 71.29
VID [2] | 70.38 | 70.16 | 72.61 | 73.09 | 73.38 | 73.40
RKD [31] | 69.61 | 69.25 | 71.82 | 71.90 | 72.28 | 73.21
LFA* | 71.85 | 71.75 | 73.79 | 72.41 | 75.07 | 75.43
CNA (ours) | 71.96 | 71.30 | 73.63 | 74.66 | 75.20 | 75.44

####  4.2.1 Baselines

Table 3: The model distillation results (top-1/top-5 accuracy in %) on
ImageNet.

| ResNet34 | ResNet18 | KD [16] | CRD [43] | LFA | CNA (ours)
---|---|---|---|---|---|---
top-1 | 73.31 | 69.76 | 71.33 | 71.17 | 71.41 | 71.38
top-5 | 91.42 | 89.08 | 90.35 | 90.13 | 90.36 | 90.19
| ResNet50 | ResNet18 | KD [16] | CRD [43] | LFA | CNA
top-1 | 76.13 | 69.76 | 71.25 | - | 70.80 | 71.43
top-5 | 93.55 | 89.08 | 90.53 | - | 90.13 | 90.27

Knowledge Distillation was proposed in [16] to distill the knowledge from a
pre-trained source (teacher) model to a newly trained target (student) model.
It forces the target output probabilities to mimic those of the source, by
minimizing a KL divergence loss between them,

$\ell_{KD}(h_{T}(x_{i}),h_{S}(x_{i}))=\tau^{2}KL(h_{T}(x_{i})/\tau,h_{S}(x_{i})/\tau),$
(10)

where $h(\cdot)$ is the classifier prediction, $KL$ is KL divergence and
$\tau$ is the temperature.

Local Feature Alignment: To address the issue that LLE [34] lacks the ability
of generalizing to points unknown to training data, [13] proposed to learn a
non-linear mapping to preserve the local neighborhood by contrastive learning.
This idea can also be applied here to align the local feature representations,
called local feature alignment (LFA). Given the feature representations of the
source model $f_{S}(x_{i})$ and the target model $f_{T}(x_{j})$, the idea is
to pull them together if $x_{i}$ and $x_{j}$ are the same sample, otherwise
repel them. The loss is defined as:

$\ell_{FA}(x_{i},x_{j})=\mathds{1}[i=j]\frac{1}{2}||f_{T}(x_{j})-f_{S}(x_{i})||_{2}^{2}\\\
+\mathds{1}[i\neq
j]\frac{1}{2}[\max(0,\xi-||f_{T}(x_{j})-f_{S}(x_{i})||_{2})]^{2}$ (11)

where $\mathds{1}[\cdot]$ is an indicator function and $\xi$ is a margin
hyper-parameter in the hinge loss term. Similar to CNA loss of Eq. 5, local
structure can be aligned by LFA, but the difference is LFA directly optimizes
on the $\ell_{2}$ feature distance.

Similar to Eq. 7, the KD loss of Eq. 10 and FA loss of Eq. 11 is jointly
optimized with a cross-entropy loss with trade-off coefficient $\lambda$ in
the model distillation task.

Figure 3: Results of model upgrade on CIFAR100 and ImageNet. X-axis: negative
flips rate (NFR) in %, lower is better. Y-axis: new model accuracy in % higher
is better. The idea model is expected to be located in the top-left corner.

####  4.2.2 Model distillation

We compare CNA to the KD and LFA baselines as introduced in Sec. 4.2.1, and
the recent contrastive representation distillation (CRD) [43]. We followed
[43] for the experimental settings for model distillation. In detail, the
models on CIFAR100 (ImageNet) were trained for 240 (100) epochs. Differently,
we used cosine learning rate scheduling instead of a hand designed schedule.
The default batch size is 1024 (512) for CNA and 512 for KD and LFA, on
ImageNet (CIFAR100). We re-implemented the KD baseline following [16], but set
loss trade-off $\lambda=1$ and temperature $\tau=100$, which yielded better
results. For LFA, the margin $\xi=3.0$ for all experiments, but we searched
for the best trade-off $\lambda$ since it is sensitive.

The results are shown in Tab. 2 for CIFAR100 and Tab. 3 for ImageNet. In the
first part of Tab. 2, the target and source models are of the same
architectural style, but of different styles in the second part. Besides KD
[16], LFA and CRD [43], some recent works of SP [44], CC [32], VID [2] and RKD
[31] are also compared. First, we found that our KD and LFA baselines already
achieved very good results, better than SP, CC, VID and RKD in most of cases,
and comparable to CRD. But the results of LFA are sometimes unreliable. For
example, when distilling “ResNet32x4” to “ResNet8x4” on CIFAR100 or distilling
“ResNet50” to “ResNet18” on ImageNet, LFA underperforms KD, CRD and CNA by a
large margin. This is probably due to the direct optimization of LFA on the
$\ell_{2}$ feature distance. Next, our CNA achieved reliable results
comparable to KD and CRD, although CNA only tries to preserve the local
topology, irrespective of the representation knowledge as distilled by KD.
This shows the effectiveness of the proposed CNA even when it is evaluated for
a classification task. The observations are similar in the experiments of
ImageNet in Tab. 3, showing that the CNA can generalize well on different
datasets.

####  4.2.3 Reducing Regression in Model Upgrade

In [51] the problem of regression in model upgrade is introduced. A variant of
KL-divergence based distillation loss was used to reduce the amount of new
errors introduced by a new model in comparison to an old reference model. We
expect the capacity of CNA in preserving the structure knowledge would also
help reduce the regression. We compare CNA to (i) a naive baseline approach to
train the new model using the standard cross-entropy loss - no attempt to
address model regression is considered in this case (ii) the latest approach
using focal distillation (FD) [51].

Table 4: Ablation studies on batch size.

batch size | 32 | 64 | 128 | 256 | 512 | 1024
---|---|---|---|---|---|---
CIFAR100 | 71.51 | 71.36 | 71.87 | 71.79 | 71.96 | -
ImageNet | - | - | 70.75 | 70.87 | 71.08 | 71.38

Table 5: Ablation studies on Number of MLP.

# MLP | 0 | 1 | 2
---|---|---|---
CIFAR100 | 71.96 | 71.56 | 71.49
ImageNet | 71.38 | 71.01 | 71.00

Metrics: We evaluated the negative flip rate (NFR) [51] and the accuracy for
each model. The NFR is defined as the fraction of negative flips:

$\mathrm{NFR}=\frac{1}{N}\sum_{i=1}^{N}\mathds{1}[\hat{y}_{n}^{(i)}\neq
y_{i}\;and\;\;\hat{y}_{o}^{(i)}=y_{i}]$ (12)

where $\hat{y}_{n}^{(i)}$ and $\hat{y}_{o}^{(i)}$ are the predicted labels of
the new model and the old model for input image $x_{i}$, and $N$ is the number
of samples in the test set $\mathcal{S}_{test}=\\{(x_{i},y_{i}),i=1..N\\}$.

All models in these experiments were trained with batch size 512. On CIFAR100
(ImageNet), the model was trained for 240 (90) epochs, with learning rate
decreased by 0.1 every 80 (30) epochs. For CNA, the trade-off $\lambda$ in Eq.
7 is chosen by validation as $1$ for CIFAR100 and $0.2$ for ImageNet. To
ensure a fair comparison, the hyper-parameters for focal distillation were
also chosen through the same validation split of CNA.

We consider the common model update scenario: the new model architecture is
larger than that of the old model. We evaluated a variety of architecture
changes, including ShuffleNet and ResNet variants. The models were both
trained on the same dataset. The results are visualized in Fig. 3. We observe
that CNA can consistently reduce the NFR. In particular, the relative NFR
reduction compared to the baseline new model is 29.9% on CIFAR100 and 14.1% on
ImageNet. On ImageNet, CNA slightly improves the model accuracy from 76.13% to
76.28%. CNA outperforms focal distillation by obtaining higher accuracy with
comparable or lower NFR. This performance improvement can be attributed to the
benefits of imposing direct representation regularization instead of mimicking
the input-output behavior via knowledge distillation. The latter in the
scenario of model upgrade would cause inevitable accuracy degradation of the
new model, which the CNA does not suffer from.

###  4.3 Ablation Studies

Table 6: Ablation studies on temperature.

temperature | 0.001 | 0.01 | 0.05 | 0.1
---|---|---|---|---
CIFAR100 | 71.42 | 71.96 | 71.44 | 70.40
ImageNet | 70.87 | 71.38 | 70.84 | 70.52

Table 7: Ablation studies on number of neighbors $K$.

$K$ | 1 | 2 | 4 | 8
---|---|---|---|---
CIFAR100 | 71.96 | 71.94 | 71.50 | 71.41
ImageNet | 71.38 | 71.26 | 71.14 | 71.02

To further understand the performance of CNA, we conducted a series of model
distillation ablation studies. The source/target model is ResNet56/ResNet20 on
CIFAR100 and ResNet34/ResNet18 on ImageNet. The default settings are:
temperature $\tau=0.01$, number of neighbors $K=1$, number of MLP is 0 and
batch size is 512 (1024) for CIFAR100 (ImageNet).

Batch Size. The effect of batch size is investigated in Tab. 4. It can be
found that the CNA algorithm is quite robust to different batch sizes on
CIFAR100, as long as the batch size is not smaller than 128. But the
performance decreases when the batch size becomes smaller on ImageNet. This is
probably because there are 1,000 classes for ImageNet and it is possible there
is no very close neighbor in a single batch if the batch size is too small.
Relatively large batch size is preferred for CNA to better preserve the local
structure.

Temperature $\tau$ was ablated in Tab. 6. It shows the best choice is around
$\tau=0.01$. When it is decreased or increased too much, the performances
drop. This temperature is also robust to different datasets.

Number of neighbors $K$ in Eq. 6 was ablated in Tab. 7. On both datasets,
$K=1$ performs considerably well, but the performances decrease when more
neighbors were counted in the loss function of Eq. 6. This is probably because
the contrastive loss formulation (Eq. 6) is curated for single pair contrast.
Directly including the other neighbors without considering their further
structure information, e.g. neighbor ranking, from the old model may impose
inaccurate neighborhood regulation on the new model, thus degrading the
performance.

Number of MLPs. The hidden MLP layers have been shown to be very helpful for
representation learning [6, 14]. Results from investigations of its effect on
CNA are shown in Tab. 5. It shows that the hidden MLP layer does not help in
CNA. The possible reason could be that the goal of CNA, maintaining the local
data structure, is different from the representation learning [6, 14].

##  5 Conclusion

In this paper, we proposed a new method, contrastive neighborhood alignment
(CNA), that preserves the local structure of feature spaces between models.
The effectiveness of CNA is illustrated on three problems: manifold learning,
where the topology of the original data is maintained in a low-dimensional
space using an inductive model, model distillation, where a compact student is
trained to mimic a larger one, and model upgrade, where the new model is more
powerful than the old one. Existing methods in model distillation are
primarily focused on matching the output predictions/logits for the pair of
models in question whereas CNA is motivated differently by preserving the
local neighborhood structures of the samples in the representation spaces. The
instance-level contrastive loss in CNA is empirically shown to work
harmoniously with the standard cross-entropy loss.

Limitations: The concept of aligning the local neighborhoods is intuitive but
its performance gain in model distillation still has room to improve.
Nonetheless, CNA provides a new means for preserving the integrity of the
model by matching the feature manifold using a contrastive term that has not
been previously explored.

Potential Negative Societal Impacts: CNA may accidentally transfer the bias in
the source feature embedding [5] to the target model. As a result it may
unintentionally amplify these biases when deploying the model. Selecting
unbiased source model or applying debiasing algorithm to the learned new model
can mitigate this negative effect.

## References

  * [1] https://github.com/dkozlov/awesome-knowledge-distillation.
  * [2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai.  Variational information distillation for knowledge transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019.
  * [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In ICCV, 2015.
  * [4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv:1607.06450, 2016.
  * [5] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  arXiv preprint arXiv:1607.06520, 2016.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [7] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang.  A survey of model compression and acceleration for deep neural networks.  arXiv preprint arXiv:1710.09282, 2017.
  * [8] Jason V Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S Dhillon.  Information-theoretic metric learning.  In Proceedings of the 24th international conference on Machine learning, pages 209–216, 2007.
  * [9] David L Donoho and Carrie Grimes.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data.  Proceedings of the National Academy of Sciences, 100(10):5591–5596, 2003.
  * [10] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of machine learning research, 12(7), 2011.
  * [11] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [12] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [13] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  NIPS Deep Learning and Representation Learning Workshop, preprint arXiv:1503.02531, 2015.
  * [17] Elad Hoffer and Nir Ailon.  Deep metric learning using triplet network.  In International workshop on similarity-based pattern recognition, pages 84–92, 2015.
  * [18] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [19] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * [20] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Advances in Neural Information Processing Systems, 2020.
  * [21] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [23] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, 2012.
  * [25] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In ICCV, 2011.
  * [26] Brian Kulis et al.  Metric learning: A survey.  Foundations and trends in machine learning, 5(4):287–364, 2012\.
  * [27] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [29] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv:1807.03748, 2018.
  * [31] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [32] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019.
  * [33] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100, 000+ questions for machine comprehension of text.  In EMNLP, 2016.
  * [34] Sam T Roweis and Lawrence K Saul.  Nonlinear dimensionality reduction by locally linear embedding.  science, 290(5500):2323–2326, 2000.
  * [35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  Int. J. Comput. Vis., 115(3):211–252, 2015.
  * [36] Bharat Bhusan Sau and Vineeth N Balasubramanian.  Deep model compression: Distilling knowledge from noisy teachers.  arXiv preprint arXiv:1610.09650, 2016.
  * [37] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [38] Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto.  Towards backward-compatible representation learning.  In CVPR, 2020.
  * [39] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [41] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701–1708, 2014.
  * [42] Joshua B Tenenbaum, Vin De Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  science, 290(5500):2319–2323, 2000.
  * [43] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2020.
  * [44] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365–1374, 2019.
  * [45] Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, et al.  Dimensionality reduction: a comparative.  J Mach Learn Res, 10(66-71):13, 2009.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, 2017.
  * [47] Lin Wang and Kuk-Jin Yoon.  Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [48] Kilian Q Weinberger, Fei Sha, and Lawrence K Saul.  Learning a kernel matrix for nonlinear dimensionality reduction.  In Proceedings of the twenty-first international conference on Machine learning, page 106, 2004.
  * [49] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, 2018.
  * [50] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, 2017.
  * [51] Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto.  Positive-congruent training: Towards regression-free model updates.  arXiv:2011.09161, 2020.
  * [52] Liu Yang and Rong Jin.  Distance metric learning: A comprehensive survey.  Michigan State Universiy, 2(2):4, 2006.
  * [53] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.