# Quantum surface effects in strong coupling dynamics

Vasilios Karanikolas<EMAIL_ADDRESS>International Center
for Young Scientists (ICYS), National Institute for Materials Science (NIMS)
1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan  Ioannis Thanopulos  Materials
Science Department, School of Natural Sciences, University of Patras, Patras
265 04, Greece  Joel D. Cox  Center for Nano Optics, University of Southern
Denmark, Campusvej 55, DK-5230 Odense M, Denmark  Danish Institute for
Advanced Study, University of Southern Denmark, Campusvej 55, DK-5230 Odense
M, Denmark  Takashi Kuroda  National Institute for Materials Science (NIMS)
1-1 Namiki, Tsukuba, Ibaraki 305-0044, Japan  Jun-ichi Inoue  National
Institute for Materials Science (NIMS) 1-1 Namiki, Tsukuba, Ibaraki 305-0044,
Japan  N. Asger Mortensen  Center for Nano Optics, University of Southern
Denmark, Campusvej 55, DK-5230 Odense M, Denmark  Danish Institute for
Advanced Study, University of Southern Denmark, Campusvej 55, DK-5230 Odense
M, Denmark  Emmanuel Paspalakis  Materials Science Department, School of
Natural Sciences, University of Patras, Patras 265 04, Greece  Christos
Tserkezis<EMAIL_ADDRESS>Center for Nano Optics, University of Southern
Denmark, Campusvej 55, DK-5230 Odense M, Denmark

(August 27, 2024)

###### Abstract

Plasmons in nanostructured metals are widely utilized to trigger strong
light–matter interactions with quantum light sources. While the nonclassical
behavior of such quantum emitters (QEs) is well-understood in this context,
the role of quantum and surface effects in the plasmonic resonator is usually
neglected. Here, we combine the Green’s tensor approach with the Feibelman
$d$-parameter formalism to theoretically explore the influence of quantum
surface effects in metal-dielectric layered nanostructures on the relaxation
dynamics of a proximal two-level QE. Having identified electron spill-out as
the dominant source of quantum effects in jellium-like metals, we focus our
study on sodium. Our results reveal a clear splitting in the emission
spectrum, indicative of having reached the strong-coupling regime, and, more
importantly, non-Markovian relaxation dynamics of the emitter. Our findings
establish that strong light–matter coupling is not suppressed by the emergence
of nonclassical surface effects in the optical response of the metal.

Introduction — Confining light in sub-diffraction volumes via surface plasmon
polaritons (SPPs) has become common practice to enhance nanoscale light–matter
interactions in the past decade ozbay_sci311 . In combination with multi-level
quantum emitters (QEs), plasmonic nanostructures play the role of the cavity
torma_rpp78 ; Pelton_natphot ; bozhevolnyi_natphot2017 ; baumberg_natmat2019
in cavity quantum electrodynamics (QED) Mandel_Cambridge1995 ;
Scully_Cambridge1997 , while the QEs, depending on the desired functionality,
can be natural (atoms, molecules) or artificial (quantum wells and dots,
defects in nanodiamonds, collective states in transition metal dichalcogenides
or hexagonal boron nitride) akimov_nat450 ; chang_natphys3 ; goban_natcom5 ;
hood_pnas113 ; fernandez_acsphot5 ; koperski_pnas117 . Advanced fabrication
techniques now enable the engineering of metal cavities with characteristic
dimensions below $10$ nm, as well as precise positioning of QEs within them,
to produce promising templates for bright single-photon emitters
manolatou_ieeejqe44 ; akselrod_natphot8 ; rose_nl14 ; chikkaraddy_nat535 ;
May2020 . The plasmonic cavity is typically formed by a metallic substrate
interacting with another metallic film or with dropcasted nanoparticles;
separating the two plasmonic components by a thin dielectric spacer containing
QEs in the form of defects, molecules, or quantum dots bogdanov_nl18 ;
katzen_acsami12 ; ojambati_acsphot7 ; Ding_nanophotonics , the resulting
intense electromagnetic (EM) fields, confined within minimized volumes,
trigger extreme light–matter interactions.

Traditionally, in the local-response approximation (LRA)
Hohenester_Springer2020 , the optical response of metals is described through
the Drude model, or through experimentally measured bulk permittivities. But,
as plasmonic cavities become narrower, the interaction strengths with the
therein confined QEs are overestimated by LRA, as compared to experimental
results ciraci_sci337 ; savage_nat491 ; scholl_nat483 , calling for amendment
of the theoretical models romero_oex14 ; esteban_natcom3 ; luo_prl111 ;
raza_nanophot2 ; yang_nat576 ; the main missing element is information about
quantum effects in the metal. As a first extension of LRA, the hydrodynamic
Drude model (HDM), which describes the motion of the compressible free-
electron gas as a convective fluid, has met with considerable success
moreau_prb87 ; raza_prb88 ; mortensen_natcom5 ; tserkezis_acsphot5b , yet
effects associated with electron spill-out still require a self-consistent
treatment toscano_natcom6 ; yan_prb2015 ; ciraci_prb93 ; ciraci2019 . However,
the ultimate goal of an _ab initio_ optical response calculation for
structures with sizes of order $100$ nm at room temperature remains out of
computational reach varas_nanophot5 .

A more efficient approach is based on the work of Feibelman feibelman_pss12 ,
who bridged EM and _ab initio_ calculations through appropriate surface-
response functions, the $d$-parameters ($d_{\perp}$ and $d_{\parallel}$),
which need be calculated only once for a given surface to be implemented in a
quantum-informed surface-response formalism (SRF) yan_PRL2015 ;
christensen_prl118 ; yang_nat576 ; Goncalves_natcom11 ; Mortensen2021 . The
full QED problem is then described by quantum-corrected mesoscopic boundary
conditions at the metal surface yan_PRL2015 ; yang_nat576 . Macroscopic QED
Rivera2020 ; Head-Marsden2020  allows to directly apply the rigorous Green’s
function approach developed within LRA, now corrected by SRF, without invoking
quantum effects in a quasi-normal mode formalism Lalanne_lpr2018 ;
Dezfouli_optica2017 ; Franke_PRL2019 .

The relaxation process of a QE placed in a nanostructured environment has been
extensively investigated in the context of the macroscopic QED theory. The
photonic environment provides an enhanced density of optical states —often,
but not restrictively, in the form of an optical resonance— at a specific
energy that, when matched with the transition energy of the QE, leads to
considerable QE–environment light–matter coupling-strength enhancement.
Usually, the interaction is described within a Lorentzian model of the
spectral density which, under strong-coupling conditions, results in a Rabi
splitting in the emission spectrum and oscillations in the relaxation dynamics
of the QE. However, this approach fails to describe possible partially excited
state population trapping effects Yang2017 ; thanopulos_prb99 ; Yang2019 .
Here, we go beyond this Lorentzian spectral density model approach, taking
into account the influence of the exact QE–nanostructure spectral density on
the light–matter interaction strength.

To date, focus has been mostly cast on the interaction of QEs with plasmonic
nanostructures described by nonlocal dielectric functions within HDM,
identifying significant differences in both weak tserkezis_nscale8 ;
christensen_nn8  and strong-coupling regimes tserkezis_acsphot5a ;
tserkezis_rpp83 , as compared to LRA. Here, we go one step further to explore,
for the first time, the role of quantum-informed plasmonics in the population
dynamics of the QE excited state under strong-coupling conditions, where QE
and environment coherently exchange energy. We focus on insulator/metal (IM),
insulator/metal/insulator (IMI), and metal/insulator/metal (MIM) geometries
smith_nanoscale ; raza_prb88 , for which the strong-coupling regime proves
reachable even when quantum corrections in the metal are taken into account.

Theory — The QE is approximated as a two-level system with ground state
$|g\rangle$ and excited state $|e\rangle$, transition frequency $\omega_{0}$,
and dipole moment $\boldsymbol{\mu}$. Initially, the QE is in the excited
state, and the EM field is in its vacuum state
$|i\rangle=|e\rangle\otimes|0\rangle$, while the final state is
$|f\rangle=|g\rangle\otimes\hat{f}_{i}^{\dagger}(\mathbf{r},\omega)|0\rangle$,
where the QE relaxes to the ground state by emitting a photon or exciting SPPs
dressed states supported by the metallic nanostructures dung_pra65 , and
$\hat{f}^{\dagger}$ denotes the bosonic creation operator of the dressed state
$i$. The QE relaxation rate $\Gamma$ is given by
$\Gamma(\mathbf{r},\omega)=\Gamma_{0}\tilde{\Gamma}(\mathbf{r},\omega)$, where
$\Gamma_{0}=\omega^{3}\mu^{2}/3\pi c^{3}\hbar\varepsilon_{0}$ is the vacuum
rate and $\tilde{\Gamma}(\mathbf{r},\omega)$ is the Purcell factor of a QE
placed at $\mathbf{r}=(0,0,z)$ above a metal/dielectric interface; for a
transition dipole moment perpendicular to a single insulator/metal interface,
the Purcell factor has the form

$\tilde{\Gamma}_{z}(\mathbf{r},\omega)=\sqrt{\varepsilon_{d}}-\frac{3c}{2\omega}\mathrm{Re}\Bigg{\\{}\int\limits_{0}^{\infty}\text{d}k_{s}\frac{k_{s}^{3}R_{\rm
TM}}{k_{d,z}k_{d}^{2}}e^{2ik_{d,z}z}\Bigg{\\}},$ (1)

where $R_{\rm TM}$ is the generalized transverse magnetic (TM) Fresnel
coefficient Tai_Oxford1994 ; Scheel2009 . We note that the metal-dielectric
structures considered here are as close to QED cavities as possible, dominated
by a single mode, thus allowing the use of a Purcell factor without ambiguity
koenderink_ol35 .

Our material of choice is sodium (Na), whose work function is lower than that
of the common plasmonic metals gold and silver, so as to focus on electron
spill-out effects abajo_jpcc112 ; echarri_arxiv2020 . For the single
dielectric/metal interface, the TM reflection coefficients take the form
(assuming $d_{\parallel}=0$, valid for Na) Goncalves_natcom11

$R_{\rm TM}=\frac{\varepsilon_{\rm m}k_{{\rm d},z}-\varepsilon_{d}k_{{\rm
m},z}+\mathrm{i}\left(\varepsilon_{\rm m}-\varepsilon_{\rm
d}\right)k_{s}^{2}d_{\perp}(\omega)}{\varepsilon_{\rm m}k_{{\rm
d},z}+\varepsilon_{\rm d}k_{{\rm m},z}-\mathrm{i}\left(\varepsilon_{\rm
m}-\varepsilon_{\rm d}\right)k_{s}^{2}d_{\perp}(\omega)},$ (2)

where $\varepsilon_{\rm d}$ and $\varepsilon_{\rm m}$ denote the permittivity
of the dielectric ($\mathrm{d}$) and metal ($\mathrm{m}$), respectively,
$k_{0}=\omega/c$ is the free-space wave vector,
$k_{j}=\sqrt{\varepsilon_{j}}k_{0}$ (with $j=\mathrm{m,d}$) is the wavevector
of each medium, analyzed in in-plane ($k_{\mathrm{s}}$) and normal
($k_{j,z}=\sqrt{k_{j}{2}-k_{\mathrm{s}}^{2}}$) components. For Na, we use a
Drude model $\varepsilon(\omega)=1-\omega_{\rm
p}^{2}/\left(\omega^{2}+\mathrm{i}\omega\gamma\right)$, with $\omega_{\rm p}$
being the plasma frequency and $\gamma$ the damping rate, taken as
$\hbar\omega_{\rm p}=5.9$ eV and $\hbar\gamma=0.1$ eV, while $d_{\perp}$ is
given by Lorentzian-fitted data extracted from _ab initio_ calculations for
jellium christensen_prl118 ; Goncalves_natcom11  (see Supplemental Material
SupportingInformation ).

The relaxation of the quantum emitter is described by its emission spectrum
vanvlack_prb85 ; Karanikolas2020

$S(\omega,\mathbf{r})=\frac{1}{2\pi}\left|\frac{(\mu^{2}\omega^{2})(\varepsilon_{0}c^{2})\mathbf{\hat{n}}\cdot\mathbf{G}(\omega,\mathbf{r},\mathbf{r}_{d})}{\omega_{0}-\omega-\int_{0}^{\infty}\mathrm{d}\omega^{\prime}J(\omega_{0},\omega^{\prime},\mathbf{r})/(\omega^{\prime}-\omega)}\right|^{2}$
(3)

in the frequency domain, and by the time-dependent QE excited state
probability amplitude, $c_{1}(t)$, given by the solution of the integro-
differential equation Tudela2014a ; Li2016PRLa ; thanopulos_prb95 ;
thanopulos_prb99

$\frac{\mathrm{d}c_{1}(t)}{\mathrm{d}t}=\mathrm{i}\int_{0}^{t}\mathrm{d}t^{\prime}K(t-t^{\prime})c_{1}(t^{\prime}).$
(4)

The kernel of Eq. (4) is given by
$K(\tau)=\mathrm{i}e^{\mathrm{i}\omega_{0}\tau}\int_{0}^{\infty}\mathrm{d}\omega
J(\omega_{0},\omega,\mathbf{r})e^{-\mathrm{i}\omega\tau}$, where
$J(\omega_{0},\omega,\mathbf{r})=\tilde{\Gamma}_{z}(\omega,\mathbf{r})/[2\pi\tau_{0}]$
is the spectral density, $\omega_{0}$ is the energy difference between the
ground and excited QE states, and $\tau_{0}=1/\Gamma_{0}(\omega_{0})$ is its
free-space lifetime Tudela2014a ; Li2016PRLa ; thanopulos_prb95 ;
thanopulos_prb99 . In Eq. (3), $\mathbf{r}_{d}$ is the position where the
signal is detected, and $\mathbf{r}$ is the position of the QE, while $\omega$
is the emission frequency of the combined QE–Na system vanvlack_prb85 ;
Karanikolas2020 . Eq. (3) implies that, if the QE–nanostructure coupling
strength is high enough, the emission spectrum will feature a doublet of
emission peaks, whose energy difference defines the Rabi splitting
$\hbar\Omega$. More details about the macroscopic QED model used can be found
in the Supplemental Material SupportingInformation . Solving Eq. (4), the
population dynamics of the QE is calculated under the rotating wave
approximation (RWA), which is well-justified under the conditions considered
here; as we show in SupportingInformation , results obtained by relaxing the
RWA exhibit the same quantitative behavior thanopulos_prb95 ; thanopulos_prb99
.

Results and discussion — To investigate the role of spill-out on the emission
properties of QE, we first revisit the simple IM case involving a single SPP
resonance. We note here that, in general, there is a competition between
relevant contributions from SPPs and a pseudomode stemming from EM modes with
higher in-plane momentum Tudela2014a ; delga_prl112 ; effectively, which
contribution dominates depends on the QE–metal separation. In Fig. 1(a) we
present the Purcell factor as a function of the QE transition energy, for a QE
placed $1\,$nm away from an air–Na IM interface, as evaluated within LRA and
SRF. We directly observe that the highest Purcell factor value within SRF has
a redshifted peak value that is one order of magnitude smaller than the
corresponding LRA result, and the SPP-originated resonance around
$\hbar\omega=\hbar\omega_{\rm p}/\sqrt{2}=4.125$ eV becomes significantly
broader. These alterations are related to additional material losses due to
surface-enabled Landau damping tserkezis_prb96 . Nevertheless, at lower
frequencies, away from the SPP resonance, the SRF Purcell factor is in fact
higher than the one predicted by LRA. Similar results have been discussed in
Ref. Goncalves_natcom11 , thus validating the Green’s tensor formalism
employed here. Interestingly, a second spectral feature appears at around $5$
eV as a shoulder in the Purcell spectrum; this is directly related to spill-
out, and is known as the Bennett multipole surface plasmon bennett_prb1 ;
svendsen_jpcm32 .

Figure 1: A QE with $z$-oriented transition dipole moment, placed $1\,$nm away
from an air/Na IM interface. (a) Purcell factor and (b) emission spectrum
(arbitrary units) within LRA (black dashed lines) and SRF (red solid lines).

The emission spectrum of a QE with transition energy
$\hbar\omega_{0}=4.125\,$eV and free-space lifetime $\tau_{0}=0.1$ ns is
presented in Fig. 1(b), where the Purcell factors of Fig. 1(a) are used. In
the LRA description, the emission spectrum features two sharp peaks, splitting
by $\hbar\Omega=1.06$ eV around the QE transition energy. When the quantum
aspects of the response of the metal are introduced through SRF, the two
emission peaks become broader, due to the higher material losses, but the Rabi
splitting increases to $\hbar\Omega=1.3$ eV; the latter phenomenon stems from
the dependence of the QE emission on the _entire spectrum_ of the Purcell
factor, which is higher within SRF away from the SPP resonance [Fig. 1(a)]
christensen_prl118 .

Figure 2: Purcell factor within (a) LRA and (b) SRF of a QE placed $5$ nm away
from an IMI nanostructure, for different metal thicknesses $D=10$ (black solid
line), 20 (red dotted line) and $50$ nm (blue dashed line). The thin purple
shows the corresponding IM result.

Turning to the IMI case, where air is considered as the insulating medium, we
examine the corresponding Purcell factor in Figs. 2(a) and (b) within LRA and
SRF, respectively, for metal layer thicknesses $D=10,\,20$, and $50\,$nm.
Specifically, we set the distance between the QE and the insulator/metal
interface to $z_{\text{QE}}=5$ nm, noting that the Purcell factors calculated
for the IM and the IMI geometry exhibit negligible differences for smaller
values $z_{\text{QE}}\sim 2\,$ nm. In the case of LRA, the Purcell factor has
its highest value at the SPP energy for all layer thicknesses. On the other
hand, when SRF is taken into account, the maximum Purcell factor is roughly
halved, and its peak positions redshift significantly as the metal thickness
decreases. For the thinnest layer ($D=10\,$nm), the resonance within SRF has
shifted by more than $\hbar\omega=0.2\,$eV compared to LRA. For increasing
metal thickness (above $D=50\,$nm), the Purcell factor for the IMI case
converges to the IM result.

Figure 3: Contour plot of the Purcell factor of a QE, as a function of
emission energy and dielectric layer thickness $D$, calculated within (a) LRA
and (b) SRF, and the corresponding contours of the emission spectra (arbitrary
units) as a function of the QE emission and transition energies obtained with
(c) LRA and (d) SRF. The QE, with $\tau_{0}=0.1\,$ns, is placed in the middle
of the insulator layer of an MIM structure with dielectric-layer thickness
$D=4\,$nm.

While the IMI architecture can be instructive, in the remainder of the paper
we will focus on the MIM structure, which is expected to display many of the
advantages of plasmonic cavities discussed in the introduction —apart,
perhaps, from the antenna effect that characterizes nanoparticle-on-mirror
cavities baumberg_natmat2019 , which contain the QE within them and, when
relatively large nanoparticles are involved, one can approximately assume that
the QE experiences (locally) the cavity as a MIM. The investigation of QE
relaxation inside MIM structures is thus an important first (exactly solvable)
step towards understanding timely experimental works.

In Figs. 3(a,b) we compare LRA and SRF contour plots of the Purcell factor as
a function of the QE emission energy and the thickness $D$ of the insulator
layer (assumed air here) for a QE placed in the middle of the insulating gap
($z_{\text{QE}}=0$). In both plots, we observe that, for small $D$, the
Purcell factor is significantly enhanced compared to the reference free-space
value. In LRA, the peak of the Purcell factor coincides with the SPP
resonance, while for higher energies there is a sharp drop of its value,
related to the absence of plasmon polariton modes in the bandgap region of the
MIM structure marocico_pra84 . On the other hand, SRF reproduces the expected
redshift of the Purcell factor spectrum, while considerable enhancement is
observed inside the bandgap, as a result of the quantum effects captured by
the Feibelman parameters. As the thickness of the insulator increases, surface
effects become less important and the spectra obtained within the two models
converge. What is evident from the large Purcell-factor enhancements of Figs.
3(a,b) is that the QE–MIM interaction can enter the strong coupling regime, as
verified by the emission-spectrum contour plots of Figs. 3(c,d), where clear
anticrossings can be observed not only within LRA, but also in the SRF case.
Naturally, the emission double-peak features are sharper in the LRA
description, and are broadened when additional loss channels due to surface
effects are considered, but the observed Rabi splitting clearly survives.

Having established the strong coupling regime, we turn now to the main focus
of this paper, which is the QE dynamics. The QE excited-state population
dynamics presented in Fig. 4 exhibits strong oscillations, with very different
features from simple exponential relaxation within the Markovian (weak
coupling) approximation, where memory effects in Eq. (4) are ignored. For a QE
with transition energy $\hbar\omega_{0}=3\,$eV and vacuum relaxation rate
$\tau_{0}=0.5\,$ns, Fig. 4(a) shows the excited state dynamics when
$z_{\text{QE}}=1\,$nm in the IM geometry. In the LRA description, we observe
that the relaxation follows an exponential pattern modified by rapid
oscillations, where after a few of them the QE relaxes to the ground state
within a time-span of $60\,$fs. Faster relaxation to the ground state is
predicted in SRF, although a closer look in the inset of Fig. 4(a) reveals
that in this case the excited state remains partially populated,
$\left|c_{\text{nL}}\right|^{2}=0.003$, even at longer times, although is
expected to fully relax due to interaction with the environment.

Figure 4: Population of the excited state of a two-level QE with
$\hbar\omega_{0}=3\,$eV and $\tau_{0}=0.5\,$ns, as a function of time $t$. (a)
The QE is placed $1\,$nm away from an IM geometry. (b) The QE is placed in the
middle of a MIM geometry with dielectric-layer thickness $D=2\,$nm. In both
panels, black and red lines correspond to results within LRA and SRF,
respectively.

To directly compare the IM and MIM geometries, we consider a QE centered in an
insulating (air) layer of $D=2\,$nm thickness sandwiched between metal
regions, i.e., maintaining a $1\,$nm distance from the metal/dielectric
interfaces. Here, the QE population dynamics presented in Fig. 4(b) is
characterized by strong oscillatory behavior, with larger population values
persisting over longer time spans. While in LRA calculations the QE eventually
relaxes to the ground state within the considered time scale, the SRF
calculations accounting for the quantum response predict that $\sim 3$% of the
initial population remains in the excited state on the same time scale, a
$10$-fold increase compared to the IM structure. The time scale of the plot is
short compared to the QE lifetime, and complete relaxation will ultimately be
achieved.

In Fig. 5 we present the excited-state population dynamics of a QE placed in
the middle of a MIM cavity having an insulator (air) layer thickness of
$D=2\,$nm when different free-space relaxation rates $\tau_{0}$ are
considered. Choosing the associated QE transition energy matching the highest
value shown in the Purcell factor spectra of Fig. 3, LRA and SRF models are
contrasted in Figs. 5(a,b) for $\hbar\omega_{0}=4.13\,$eV and
$\hbar\omega_{0}=3.43\,$eV, respectively. In both panels we observe strong and
rapid Rabi oscillations; as the value of the free-space lifetime decreases
from $\tau_{0}=10$ to $0.1\,$ns, the QE/MIM coupling increases, the
oscillation period decreases, and non-Markovian effects become stronger, as
anticipated. We also observe that the population oscillations are denser in
the SRF case; the reason is that what is considered is the full Purcell-factor
spectrum which, compared to the LRA case is broader, and the contribution to
the bandgap region is also substantial, thus leading to a higher QE/MIM cavity
interaction. The different values of $\tau$ are connected with the different
possible natural or artificial QEs available for experimental
manipulationBricks2018 ; Koperski2020 ; Epstein2020 .

Figure 5: Population of the excited state of a two-level QE as a function of
time, within LRA (a,c) and SRF (b,d). The QE is placed in the middle of a MIM
geometry of thickness $D=2\,$nm. (a,b) The values of $\tau_{0}$ considered are
displayed in the insets. The transition energy of the QE is
$\hbar\omega_{0}=4.13\,$eV (LRA) and $\hbar\omega_{0}=3.43\,$eV (SRF). (c,d)
The values of $\hbar\omega_{0}$ considered are shown in the insets, for a QE
with $\tau_{0}=0.5\,$ns.

Finally, in Figs. 5(c,d) we explore the influence of the QE transition energy,
$\hbar\omega_{0}$, on the population dynamics, taking a QE free-space lifetime
of $\tau_{0}=0.5\,$ns. Different values of the QE transition energy are
considered, including those matching the SPP energies indicated in the Purcell
factor spectrum of Fig. 3. Fig. 5(c) shows the LRA calculation, where we
observe that, after a few oscillations, the population dynamics remains
partially trapped—although, eventually, it fully relaxes at later times. As
the QE transition energy decreases, moving away from the SPP resonance energy
associated with the highest Purcell factor enhancement, the partially
transient trapped population steadily decreases. In the SRF case, we also
observe a highly oscillatory behavior of the excited state population density,
a sign that the non-Markovian signature is retained. Although the partially
excited population trapping is smaller, strong oscillations are still present,
with reduced period, indicating that the overall QE–MIM interaction is higher
when the quantum aspects of the metal response are taken into account.

Summary — We theoretically explored the emission properties of a QE placed in
proximity to IM, IMI, and MIM geometries in the weak- and strong-coupling
regimes. Focusing on Na as a plasmonic metal whose nanoscopic optical response
is dominated by electron spill-out, we adopted the quantum-informed SRF
approach, in which the Feibelman $d$ parameters for the centroid of induced
charge incorporate first-principles calculations. Although quantum effects
introduce additional losses in the metal, the QE/MIM cavity system is found to
operate in the strong-coupling regime, manifested as a Rabi splitting in the
emission spectrum and through Rabi oscillations in the relaxation dynamics of
the QE excited state, with a highly non-Markovian character. SRF bridges,
therefore, _ab initio_ approaches with classical EM calculations in a simple
and elegant way, suitable not only for classical but also for quantum-optical
dynamic studies at the nanoscale.

###### Acknowledgements.

Acknowledgments — V.K. research was supported by JSPS KAKENHI Grant Number
JP21K13868. J. D. C. is a Sapere Aude research leader supported by Independent
Research Fund Denmark (grant No. 0165-00051B). N. A. M. is a VILLUM
Investigator supported by VILLUM FONDEN (grant No. 16498). The Center for Nano
Optics is financially supported by the University of Southern Denmark (SDU
2020 funding). E.P.’s work is co-financed by Greece and the European Union
(project code name POLISIMULATOR).

## References

  * (1) E. Ozbay, Science 311, 189 (2006).
  * (2) P. Törmä and W. L. Barnes, Reports on Progress in Physics 78, 013901 (2015).
  * (3) M. Pelton, Nature Photonics 9, 427 (2015).
  * (4) S. I. Bozhevolnyi and J. B. Khurgin, Nature Photonics 11, 398 (2017).
  * (5) J. J. Baumberg, J. Aizpurua, M. H. Mikkelsen, and D. R. Smith, Nature Materials 18, 668-678 (2019).
  * (6) L. Mandel and E. Wolf, Optical Coherence and Quantum Optics (Cambridge University Press, 1995).
  * (7) M. O. Scully and M. S. Zubairy, Quantum Optics (Cambridge University Press, 1997).
  * (8) A. V. Akimov, A. Mukherjee, C. L. Yu, D. E. Chang, A. S. Zibrov, P. R. Hemmer, H. Park, and M. D. Lukin, Nature 450, 402 (2007).
  * (9) D. E. Chang, A. S. Sørensen, E. A. Demler, and M. D. Lukin, Nature Physics 3, 807 (2007).
  * (10) A. Goban, C.-L. Hung, S.-P. Yu, J. Hood, J. Muniz, J. Lee, M. Martin, A. McClung, K. Choi, D. Chang, O. Painter, and H. Kimble, Nature Communications 5, 3808 (2014).
  * (11) J. D. Hood, A. Goban, A. Asenjo-Garcia, M. Lu, S.-P. Yu, D. E. Chang, and H. J. Kimble, Proceedings of the National Academy of Sciences 113, 10507 (2016).
  * (12) A. I. Fernández-Domínguez, S. I. Bozhevolnyi, and N. A. Mortensen, ACS Photonics 5, 3447 (2018).
  * (13) M. Koperski, D. Vaclavkova, K. Watanabe, T. Taniguchi, K. S. Novoselov, and M. Potemski, Proceedings of the National Academy of Sciences 117, 13214 (2020).
  * (14) C. Manolatou and F. Rana, IEEE Journal of Quantum Electronics 44, 435 (2008).
  * (15) G. M. Akselrod, C. Argyropoulos, T. B. Hoang, C. Ciracì, C. Fang, J. Huang, D. R. Smith, and M. H. Mikkelsen, Nature Photonics 8, 835 (2014).
  * (16) A. Rose, T. B. Hoang, F. McGuire, J. J. Mock, C. Ciracì, D. R. Smith, and M. H. Mikkelsen, Nano Letters 14, 4797 (2014).
  * (17) R. Chikkaraddy, B. de Nijs, F. Benz, S. J. Barrow, O. A. Scherman, E. Rosta, A. Demetriadou, P. Fox, O. Hess, and J. J. Baumberg, Nature 535,127 (2016).
  * (18) M. A. May, D. Fialkow, T. Wu, K.-D. Park, H. Leng, J. A. Kropp, T. Gougousi, P. Lalanne, M. Pelton, and M. B. Raschke, Advanced Quad. Tech. 3, 1900087 (2020).
  * (19) S. I. Bogdanov, M. Y. Shalaginov, A. S. Lagutchev, C.-C. Chiang, D. Shah, A. S. Baburin, I. A. Ryzhikov, I. A. Rodionov, A. V. Kildishev, A. Boltasseva, and V. M. Shalaev, Nano Letters 18, 4837 (2018).
  * (20) J. M. Katzen, C. Tserkezis, Q. Cai, L. H. Li, J. M. Kim, G. Lee, G.-R. Yi, W. R. Hendren, E. J. G. Santos, R. M. Bowman, and F. Huang, ACS Applied Materials & Interfaces 12, 19866 (2020).
  * (21) O. S. Ojambati, W. M. Deacon, R. Chikkaraddy, C. Readman, Q. Lin, Z. Koczor-Benda, E. Rosta, O. A. Scherman, and J. J. Baumberg, ACS Photonics 7, 2337 (2020).
  * (22) F. Ding, Y. Yang, R. A. Deshpande, and S. I. Bozhevolnyi, Nanophotonics 7, 1129 (2018).
  * (23) U. Hohenester, Nano and Quantum Optics: An Introduction to Basic Principles and Theory (Springer, 2020).
  * (24) C. Ciracì, R. T. Hill, J. J. Mock, Y. Urzhumov, A. I. Fernández-Domínguez, S. A. Maier, J. B. Pendry, A. Chilkoti, and D. R. Smith, Science 337, 1072 (2012).
  * (25) K. J. Savage, M. M. Hawkeye, R. Esteban, A. G. Borisov, J. Aizpurua, and J. J. Baumberg, Nature 491, 574 (2012).
  * (26) J. A. Scholl, A. L. Koh, and J. A. Dionne, Nature 483, 421 (2012).
  * (27) I. Romero, J. Aizpurua, G. W. Bryant, F. J. García de Abajo, Optics Express 14, 9988 (2006).
  * (28) R. Esteban, A. G. Borisov, P. Nordlander and J. Aizpurua, Nature Communications 3, 825 (2012).
  * (29) Y. Luo, A. I. Fernández-Domínguez, A. Wiener, S. A. Maier and J. B. Pendry, Physical Review Letters 111, 093901 (2013).
  * (30) S. Raza, N. Stenger, S. Kadkhodazadeh, S. V. Fischer, N .Kostesha, A.-P. Jauho, A. Burrows, M. Wubs, and N. A. Mortensen, Nanophotonics 2, 131 (2013).
  * (31) Y. Yang, D. Zhu, W. Yan, A. Agarwal, M. Zheng, J. D. Joannopoulos, P. Lalanne, T. Christensen, K. K. Berggren, and M. Soljačić, Nature 576, 248 (2019).
  * (32) A. Moreau, C. Ciracì, and D. R. Smith, Physical Review B 87, 045401 (2013).
  * (33) S. Raza, T. Christensen, M. Wubs, S. I. Bozhevolnyi, and N. A. Mortensen, Physical Review B 88, 115401 (2013).
  * (34) N. A. Mortensen, S. Raza, M. Wubs, T. Søndergaard, and S. I. Bozhevolnyi, Nature Communications 5, 3809 (2014).
  * (35) C. Tserkezis, A. T. M. Yeşilyurt, J.-S. Huang, and N. A. Mortensen, ACS Photonics 5, 5017 (2018).
  * (36) G. Toscano, J. Straubel, A. Kwiatkowski, C. Rockstuhl, F. Evers, H. Xu, N. A. Mortensen and M. Wubs, Nature Communications 6, 7132 (2015).
  * (37) W. Yan, Physical Review B 91, 115416 (2015).
  * (38) C. Ciracì and F. Della Sala, Physical Review B 93, 205405 (2016).
  * (39) C. Ciracì, R. Jurga, M. Khalid, and F. Della Sala, Nanophotonics 8, 1821 (2019).
  * (40) A. Varas, P. García-González, J. Feist, F. García-Vidal, and A. Rubio, Nanophotonics 5, 409 (2016).
  * (41) P. Feibelman, Progress in Surface Science 12, 287 (1982).
  * (42) W. Yan, M. Wubs, and N. Asger Mortensen, Physical Review Letters 115, 137403 (2015).
  * (43) T. Christensen, W. Yan, A.-P. Jauho, M. Soljačić, and N. A. Mortensen, Physical Review Letters 118, 157402 (2017).
  * (44) P. A. D. Gonçalves, T. Christensen, N. Rivera, A.-P. Jauho, N. A. Mortensen, and M. Soljačić, Nature Communications 11, 366 (2020).
  * (45) N. A. Mortensen, Nanophotonics 10, 2563 (2021).
  * (46) N. Rivera and I. Kaminer, Nature Reviews Physics 2, 538 (2020).
  * (47) K. Head-Marsden, J. Flick, C. J. Ciccarino, and P. Narang, Chemical Reviews 121 (2021), 10.1021/acs.chemrev.0c00620.
  * (48) P. Lalanne, W. Yan, K. Vynck, C. Sauvan, and J.-P. Hugonin, Laser & Photonics Reviews 12, 1700113 (2018).
  * (49) M. K. Dezfouli, C. Tserkezis, N. A. Mortensen, and S. Hughes, Optica 4, 1503 (2017).
  * (50) S. Franke, S. Hughes, M. K. Dezfouli, P. T. Kristensen, K. Busch, A. Knorr, and M. Richter, Physical Review Letters 122, 213901 (2019).
  * (51) C.-J. Yang and J.-H. An, Phys. Rev. B 95, 161408(R) (2017).
  * (52) I. Thanopulos, V. Karanikolas, N. Iliopoulos, and E. Paspalakis, Physical Review B 99, 195412 (2019).
  * (53) C.-J. Yang, J.-H. An, and H.-Q. Lin, Phys. Rev. Research 1, 023027, (2019).
  * (54) C. Tserkezis, N. Stefanou, M. Wubs, and N. A. Mortensen, Nanoscale 8, 17532 (2016).
  * (55) T. Christensen, W. Yan, S. Raza, A.-P. Jauho, N. A. Mortensen, and M. Wubs, ACS Nano 8, 1745 (2014).
  * (56) C. Tserkezis, M. Wubs, and N. A. Mortensen, ACS Photonics 5, 133 (2018).
  * (57) C. Tserkezis, A. I. Fernández-Domínguez, P. A. D. Gonçalves, F. Todisco, J. D. Cox, K. Busch, N. Stenger, S. I. Bozhevolnyi, N. A. Mortensen, and C. Wolff, Reports on Progress in Physics 83, 082401 (2020).
  * (58) C. L. C. Smith, N. Stenger, A. Kristensen, N. A. Mortensen, and S. I. Bozhevolnyi, Nanoscale 7, 9355 (2015).
  * (59) H. T. Dung, L. Knöll, and D.-G. Welsch, Physical Review A 65, 043813 (2002).
  * (60) C. T. Tai, Dyadic Green Functions in Electromagnetic Theory (Oxford University Press, Oxford, 1994).
  * (61) S. Scheel and S. Y. Buhmann, Acta Physical Slovaca 58 675 (2009).
  * (62) A. F. Koenderink, Opt. Lett. 35, 4208 (2010).
  * (63) F. J. García de Abajo, Journal of Physical Chemistry C 112, 17983 (2008).
  * (64) A. Rodríguez Echarri, P. A. D. Gonçalves, C. Tserkezis, F. J. García de Abajo, N. A. Mortensen, and J. D. Cox, Optica 8, 710 (2021).
  * (65) See Supplemental Material for details on the calculation of the Green’s tensor for the IM, IMI and MIM geometries including the SRF boundury conditions. The macroscopic QED theory is used to extract Eqs. (3) and (4).
  * (66) C. Van Vlack, P. T. Kristensen, and S. Hughes, Physical Review B 85, 075303 (2012).
  * (67) V. Karanikolas, I. Thanopulos, and E. Paspalakis, Physical Review Research 2, 033141 (2020).
  * (68) A. Gonzalez-Tudela, P. A. Huidobro, L. Martín-Moreno, C. Tejedor, and F. J. García-Vidal, Physical Review B 89, 041402(R) (2014).
  * (69) R.-Q. Li, D. Hernángomez-Peréz, F. J. García-Vidal, and A. I. Fernández-Domínguez, Physical Review Letters 117, 107401 (2016).
  * (70) I. Thanopulos, V. Yannopapas, and E. Paspalakis, Physical Review B 95, 075412 (2017).
  * (71) A. Delga, J. Feist, J. Bravo-Abad, and F. J. Garcia-Vidal, Physical Review Letters 112, 253601 (2014).
  * (72) C. Tserkezis, N. A. Mortensen, and M. Wubs, Physical Review B 96, 085413 (2017).
  * (73) A. J. Bennett, Phys. Rev. B 1, 203 (1970).
  * (74) M. K. Svendsen, C. Wolff, A.-P. Jauho, N. A. Mortensen, and C. Tserkezis, J. Phys.: Condens. Matter 32, 395702 (2020).
  * (75) C. A. Marocico and J. Knoester, Physical Review A 84, 053824 (2011).
  * (76) J.L. Bricks, Y.L. Slominskii, I.D. Panas, and A.P. Demchenko, Methods Appl. Fluoresc. 6, 012001 (2018).
  * (77) M. Koperski, D. Vaclavkova, K. Watanabe, T. Taniguchi, K.S. Novoselov, and M. Potemski, PNAS 117, 13214 (2020).
  * (78) I. Epstein, A.J. Chaves, D.A. Rhodes, B. Frank, K. Watanabe, T. Taniguchi, H. Giessen, J.C. Hone, N.M.R. Peres, and F.H.L. Koppens, 2D Materials 7, 035031 (2020).


Subhabrata<EMAIL_ADDRESS>Iro<EMAIL_ADDRESS>Christian<EMAIL_ADDRESS>Andrea
<EMAIL_ADDRESS>Visual Geometry Group
University of Oxford
Oxford, UK The Curious Layperson

# The Curious Layperson: Fine-Grained Image Recognition without Expert Labels

###### Abstract

Most of us are not experts in specific fields, such as ornithology.
Nonetheless, we do have general image and language understanding capabilities
that we use to match what we see to expert resources. This allows us to expand
our knowledge and perform novel tasks without ad-hoc external supervision. On
the contrary, machines have a much harder time consulting expert-curated
knowledge bases unless trained specifically with that knowledge in mind. Thus,
in this paper we consider a new problem: fine-grained image recognition
without expert annotations, which we address by leveraging the vast knowledge
available in web encyclopedias. First, we learn a model to describe the visual
appearance of objects using non-expert image descriptions. We then train a
fine-grained textual similarity model that matches image descriptions with
documents on a sentence-level basis. We evaluate the method on two datasets
and compare with several strong baselines and the state of the art in cross-
modal retrieval. Code is available at: https://github.com/subhc/clever.

##  1 Introduction

Figure 1: Fine-Grained Image Recognition without Expert Labels. We propose a
novel task that enables fine-grained classification without using expert class
information (_e.g_ bird species) during training. We frame the problem as
document retrieval from general image descriptions by leveraging existing
textual knowledge bases, such as Wikipedia.

Deep learning and the availability of large-scale labelled datasets have led
to remarkable advances in image recognition tasks, including fine-grained
recognition [Wah et al. (2011), Nilsback and Zisserman (2006), Horn et al.
(2017)]. The problem of fine-grained image recognition amounts to identifying
subordinate-level categories, such as different species of birds, dogs or
plants. Thus, the supervised learning regime in this case requires annotations
provided by domain _experts_ or citizen scientists [Van Horn et al. (2015)].

While most people, unless professionally trained or enthusiasts, do not have
knowledge in such specific domains, they are generally capable of consulting
existing expert resources such as books or online encyclopedias, _e.g_
Wikipedia. As an example, let us consider bird identification. Amateur bird
watchers typically rely on field guides to identify observed species. As a
general instruction, one has to answer the question “what is most noticeable
about this bird?” before skimming through the guide to find the best match to
their observation. The answer to this question is typically a detailed
description of the bird’s shape, size, plumage colors and patterns. Indeed, in
Fig. 1, the non-expert observer might not be able to directly identify a bird
as a “Vermillion Flycatcher”, but they _can_ simply describe the appearance of
the bird: “this is a bright red bird with black wings and tail and a pointed
beak”. This description can be matched to an expert corpus to obtain the
species and other expert-level information.

On the other hand, machines have a much harder time consulting off-the-shelf
expert-curated knowledge bases. In particular, most algorithmic solutions are
designed to address a _specific_ task with datasets constructed _ad-hoc_ to
serve precisely this purpose. Our goal, instead, is to investigate whether it
is possible to re-purpose general image and text understanding capabilities to
allow machines to consult already existing _textual_ knowledge bases to
address a new task, such as recognizing a bird.

We introduce a novel task inspired by the way a layperson would tackle fine-
grained recognition from visual input; we name this CLEVER, _i.e_ Curious
Layperson-to-Expert Visual Entity Recognition. Given an image of a
subordinate-level object category, the task is to retrieve the relevant
document from a large, expertly-curated text corpus; to this end, we only
allow non-expert supervision for learning to describe the image. We assume
that: (1) the corpus dedicates a separate entry to each category, as is, for
example, the case in encyclopedia entries for bird or plant species, etc., (2)
there exist no paired data of images and documents or expert labels during
training, and (3) to model a layperson’s capabilities, we have access to
general image and text understanding tools that do not use expert knowledge,
such as image descriptions or language models.

Given this definition, the task classifies as weakly-supervised in the
taxonomy of learning problems. We note that there are fundamental differences
to related topics, such as image-to-text retrieval and unsupervised image
classification. Despite a significant amount of prior work in image-to-text or
text-to-image retrieval [Peng et al. (2017a), Wang et al. (2017), Zhen et al.
(2019), Hu et al. (2019), He et al. (2019)], the general assumption is that
images and corresponding documents are paired for training a model. In
contrast to unsupervised image classification, the difference is that here we
are interested in _semantically_ labelling images using a secondary modality,
instead of grouping similar images [Asano et al. (2020), Caron et al. (2020),
Van Gansbeke et al. (2020)].

To the best of our knowledge, we are the first to tackle the task of fine-
grained image recognition without expert supervision. Since the target corpus
is not required during training, the search domain is easily extendable to any
number of categories/species—an ideal use case when retrieving documents from
dynamic knowledge bases, such as Wikipedia. We provide extensive evaluation of
our method and also compare to approaches in cross-modal retrieval, despite
using significantly reduced supervision.

##  2 Related Work

In this paper, we address a novel problem (CLEVER). Next we describe in detail
how it differs from related problems in the computer vision and natural
language processing literature and summarise the differences with respect to
how class information is used in Table 1.

#### Fine-Grained Recognition.

| Class Information
---|---
Task | Train | Test
FGVR | K | K
ZSL | K | U
GZSL | K | K + U
CLEVER | U | U
Table 1: Overview of related topics (K: known, U: unknown).

The goal of fine-grained visual recognition (FGVR) is categorising objects at
sub-ordinate level, such as species of animals or plants [Wah et al. (2011),
Van Horn et al. (2015), Van Horn et al. (2018), Nilsback and Zisserman (2008),
Kumar et al. (2012)]. Large-scale annotated datasets require domain experts
and are thus difficult to collect. FGVR is more challenging than coarse-level
image classification as it involves categories with fewer discriminative cues
and fewer labeled samples. To address this problem, supervised methods exploit
side information such as part annotations [Zhang et al. (2014)], attributes
[Vedaldi et al. (2014)], natural language descriptions [He and Peng (2017)],
noisy web data [Krause et al. (2016), Xu et al. (2016), Gebru et al. (2017)]
or humans in the loop [Branson et al. (2010), Deng et al. (2015), Cui et al.
(2016)]. Attempts to reduce supervision in FGVR are mostly targeted towards
eliminating auxiliary labels, _e.g_ part annotations [Zheng et al. (2017),
Simon and Rodner (2015), Ge et al. (2019), Huang and Li (2020)]. In contrast,
our goal is fine-grained recognition without access to _categorical_ labels
during training. Our approach only relies on side information (captions)
provided by laymen and is thus unsupervised from the perspective of “expert
knowledge”.

#### Zero/Few Shot Learning.

Zero-shot learning (ZSL) is the task of learning a classifier for unseen
classes [Xian et al. (2018a)]. A classifier is generated from a description of
an object in a secondary modality, mapping semantic representations to class
space in order to recognize said object in images [Socher et al. (2013)].
Various modalities have been used as auxiliary information: word embeddings
[Frome et al. (2013), Xian et al. (2016)], hierarchical embeddings
[Kampffmeyer et al. (2019)], attributes [Farhadi et al. (2009), Akata et al.
(2015)] or Wikipedia articles [Elhoseiny et al. (2017), Zhu et al. (2018),
Elhoseiny et al. (2016), Qiao et al. (2016)]. Most recent work uses generative
models conditioned on class descriptions to synthesize training examples for
unseen categories [Long et al. (2017), Kodirov et al. (2017), Felix et al.
(2018), Xian et al. (2019), Vyas et al. (2020), Xian et al. (2018b)]. The
multi-modal and often fine-grained nature of the standard and generalised
(G)ZSL task renders it related to our problem. However, different from the
(G)ZSL settings our method uses neither class supervision during training nor
image-document pairs as in [Elhoseiny et al. (2017), Zhu et al. (2018),
Elhoseiny et al. (2016), Qiao et al. (2016)].

#### Cross-Modal and Information Retrieval.

While information retrieval deals with extracting information from document
collections [Manning et al. (2008)], cross-modal retrieval aims at retrieving
relevant information across various modalities, _e.g_ image-to-text or vice
versa. One of the core problems in information retrieval is ranking documents
given some query, with a classical example being Okapi BM25 [Robertson et al.
(1995)]. With the advent of transformers [Vaswani et al. (2017)] and BERT
[Devlin et al. (2019)], state-of-the-art document retrieval is achieved in
two-steps; an initial ranking based on keywords followed by computationally
intensive BERT-based re-ranking [Nogueira and Cho (2019), Nogueira et al.
(2020), Yilmaz et al. (2019), MacAvaney et al. (2019)]. In cross-modal
retrieval, the common approach is to learn a shared representation space for
multiple modalities [Peng et al. (2017a), Andrew et al. (2013), Wang and
Livescu (2016), Peng et al. (2016), Peng et al. (2017b), Wang et al. (2017),
Zhen et al. (2019), Hu et al. (2019), He et al. (2019)]. In addition to paired
data in various domains, some methods also exploit auxiliary semantic labels;
for example, the Wikipedia benchmark [Pereira et al. (2013)] provides broad
category labels such as _history_ , _music_ , _sport_ , etc.

We depart substantially from the typical assumptions made in this area.
Notably, with the exception of [He et al. (2019), Wang et al. (2009)], this
setting has not been explored in fine-grained domains, but generally targets
higher-level content association between images and documents. Furthermore,
one major difference between our approach and cross-modal retrieval, including
[He et al. (2019), Wang et al. (2009)], is that we do not assume paired data
between the input domain (images) and the target domain (documents). We
address the lack of such pairs using an intermediary modality (captions) that
allows us to perform retrieval directly in the text domain.

#### Natural Language Inference (NLI) and Semantic Textual Similarity (STS).

Also related to our work, in natural language processing, the goal of the NLI
task is to recognize textual entailment, _i.e_ given a pair of sentences
(premise and hypothesis), the goal is to label the hypothesis as entailment
(true), contradiction (false) or neutral (undetermined) with respect to the
premise [Bowman et al. (2015), Williams et al. (2018)]. STS measures the
degree of semantic similarity between two sentences [Agirre et al. (2012),
Agirre et al. (2013)]. Both tasks play an important role in semantic search
and information retrieval and are currently dominated by the transformer
architecture [Vaswani et al. (2017), Devlin et al. (2019), Liu et al. (2019),
Reimers and Gurevych (2019)]. Inspired by these tasks, we propose a sentence
similarity regime that is domain-specific, paying attention to fine-grained
semantics.

##  3 Method

We introduce the problem of layperson-to-expert visual entity recognition
(CLEVER), which we address via image-based document retrieval. Formally, we
are given a set of images $x_{i}\in\mathcal{I}$ to be labelled given a corpus
of expert documents $D_{j}\in\mathcal{D}$, where each document corresponds to
a fine-grained image category and there exist $K=|D|$ categories in total. As
a concrete example, $\mathcal{I}$ can be a set of images of various bird
species and $\mathcal{D}$ a bird identification corpus constructed from
specialized websites (with one article per species). Crucially, the pairing of
$x_{i}$ and $D_{j}$ is not known, _i.e_ no expert task supervision is
available during training. Therefore, the mapping from images to documents
cannot be learned directly but can be discovered through the use of non-expert
image descriptions $\mathcal{C}_{i}$ for image $x_{i}$.

Our method consists of three distinct parts. First, we learn, using
“layperson’s supervision”, an image captioning model that uses simple color,
shape and part descriptions. Second, we train a model for Fine-Grained
Sentence Matching (FGSM). The FGSM model takes as input a pair of sentences
and predicts whether they are descriptions of the same object. Finally, we use
the FGSM to score the documents in the expert corpus via voting. As there is
one document per class, the species corresponding to the highest-scoring
document is returned as the final class prediction for the image. The overall
inference process is illustrated in Fig. 2.

###  3.1 Fine-grained Sentence Matching

The overall goal of our method is to match images to expert documents —
however, in absence of paired training data, learning a cross-domain mapping
is not possible. On the other hand, describing an image is an easy task for
most humans, as it usually does not require domain knowledge. It is therefore
possible to leverage image descriptions as an intermediary for learning to map
images to an expert corpus.

To that end, the core component of our approach is the FGSM model
$f(c_{1},c_{2})\in\mathbb{R}$ that scores the visual similarity of two
descriptions $c_{1}$ and $c_{2}$. We propose to train $f$ in a manner similar
to the textual entailment (NLI) task in natural language processing. The
difference to NLI is that the information that needs to be extracted here is
fine-grained and domain-specific _e.g_ “a bird with blue wings” vs. “this is a
uniformly yellow bird”. Since we do not have annotated sentence pairs for this
task, we have to create them synthetically. Instead of the terms entailment
and contradiction, here we use positive and negative to emphasize that the
goal is to find matches (or mismatches) between image descriptions.

Figure 2: Overview. We train a model for fine-grained sentence matching (FGSM)
using layerperson’s annotations, _i.e_ class-agnostic image descriptions. At
test time, we score documents from a relevant corpus and use the top-ranked
document to label the image.

We propose to model $f$ as a sentence encoder, performing the semantic
comparison of $c_{1},c_{2}$ in embedding space. Despite their widespread
success in downstream tasks, most transformer-based language models are
notoriously bad at producing semantically meaningful sentence embeddings
[Reimers and Gurevych (2019), Li et al. (2020)]. We thus follow [Reimers and
Gurevych (2019)] in learning an appropriate textual similarity model with a
Siamese architecture built on a pre-trained language transformer. This also
allows us to leverage the power of large language models while maintaining
efficiency by computing an embedding for each input independently and only
compare embeddings as a last step. To this end, we compute a similarity score
for $c_{1}$ and $c_{2}$ as
$f(c_{1},\,c_{2})=h\left(\left[\phi_{1};\,\phi_{2};\,|\phi_{1}-\phi_{2}|\right]\right)$,
where $[\cdot]$ denotes concatenation, and $h$ and $\phi$ are lightweight MLPs
operating on the average-pooled output of a large language model
$\operatorname{T}(\cdot)$ with the shorthand notation
$\phi_{1}=\phi(\operatorname{T}(c_{1}))$.

#### Training.

One requirement is that the FGSM model should be able to identify _fine-
grained_ similarities between pairs of sentences. This is in contrast to the
standard STS and NLI tasks in natural language understanding which determine
the relationship (or degree of similarity) of a sentence pair on a _coarser_
semantic level. Since our end-goal is visual recognition, we instead train the
model to emphasize visual cues and nuanced appearance differences.

Let $\mathcal{C}_{i}$ be the set of human-annotated descriptions for a given
image $x_{i}$. Positive training pairs are generated by exploiting the fact
that, commonly, each image has been described by multiple annotators; for
example in CUB-200 [Wah et al. (2011)] there are $|\mathcal{C}_{i}|=10$
captions per image. Thus, each pair (from
$\mathcal{C}_{i}\times\mathcal{C}_{i}$) of descriptions of the same image can
be used as a positive pair. The negative counterparts are then sampled from
the complement $\bar{\mathcal{C}}_{i}=\bigcup_{l\neq i}\mathcal{C}_{l}$, _i.e_
among the available descriptions for all other images in the dataset. We
construct this dataset with an equal amount of samples for both classes and
train $f$ with a binary cross entropy loss.

#### Inference.

During inference the sentence embeddings $\phi$ for each sentence in each
document can be precomputed and only $h$ needs to be evaluated dynamically
given an image and its corresponding captions, as described in the next
section. This greatly reduces the memory and time requirements.

###  3.2 Document Scoring

Although trained from image descriptions alone, the FGSM model can take any
sentence as input and, at test time, we use the trained model to score
sentences from an expert corpus against image descriptions. Specifically, we
assign a score $z_{ij}\in\mathbb{R}$ to each expert document $D_{j}$ given a
set of descriptions for the $i$-th image:

$z_{ij}=\frac{1}{|\mathcal{C}_{i}\times
D_{j}|}\sum_{(c,s)\in\mathcal{C}_{i}\times D_{j}}{f(c,s)},$ (1)

Since there are several descriptions in $\mathcal{C}_{i}$ and sentences in
$D_{j}$, we compute the final document score as an average of individual
predictions (scores) of all pairs of descriptions and sentences. Aggregating
scores across the whole corpus $\mathcal{D}$, we can then compute the
probability $p(D_{j}\>|\>x_{i})=\frac{e^{-z_{ij}}}{\sum_{k}e^{-z_{ik}}}$ of a
document $D_{j}\in\mathcal{D}$ given image $x_{i}$ and assign the document
(and consequently class) with the highest probability to the image.

###  3.3 Bridging the Domain Gap

While training the FGSM model, we have so far only used laypersons’
descriptions, disregarding the expert corpus. However, we can expect the
documents to contain significantly more information than visual descriptions.
In the case of bird species, encyclopedia entries usually also describe
behavior, migration, conservation status, etc. In this section, we thus employ
two mechanisms to bridge the gap between the image descriptions and the
documents.

#### Neutral Sentences.

We introduce a third, neutral class to the classification problem, designed to
capture sentences that do not provide relevant (visual) information. We
generate neutral training examples by pairing an image description with
sentences from the documents (or other descriptions) that do not have any
_nouns_ in common. Instead of binary cross entropy, we train the three-class
model (positive/neutral/negative) with softmax cross entropy.

#### Score Distribution Prior.

Despite the absence of paired training data, we can still impose priors on the
document scoring. To this end, we consider the probability distribution
$p(\mathcal{D}\>|\>x)$ over the entire corpus $\mathcal{D}$ given an image $x$
in a training batch $\mathcal{B}$. We can then derive a regularizer
$R(\mathcal{B})$ that operates at batch-level:

$R(\mathcal{B})=\sum_{x\in\mathcal{B}}\Big{(}-\langle p(\mathcal{D}\>|\>x),\
p(\mathcal{D}\>|\>x)\rangle\;\\\ +\sum_{x^{\prime}\in\mathcal{B}\setminus
x}\langle p(\mathcal{D}\>|\>x),\ p(\mathcal{D}\>|\>x^{\prime})\rangle\Big{)}$
(2)

where $\langle\cdot,\cdot\rangle$ denotes the inner product of two vectors.
The intuition of the two terms of the regularizer is as follows. $\langle
p(D\>|\>x),\>p(D\>|\>x)\rangle$ is maximal when the distribution assigns all
mass to a single document. Since the score $z_{ij}$ is averaged over all
captions of one image, this additionally has the side effect of encouraging
all captions of one image to vote for the same document. The second term of
$R(\mathcal{B})$ then encourages the distributions of two different images to
be orthogonal, favoring the assignment of images uniformly across all
documents.

Since $R(\mathcal{B})$ requires evaluation over the whole document corpus for
every image, we first pre-train $f$, including the large transformer model
$T$, (c.f. Section 3.1). After convergence, we extract sentence features for
all documents and image descriptions and train only the MLPs $\phi$ and $h$
with $\mathcal{L}+\lambda R$, where $\lambda$ balances the 3-class cross
entropy loss $\mathcal{L}$ and the regularizer.

##  4 Experiments

We validate our method empirically for bird and plant identification. To the
best of our knowledge, we are the first to consider this task, thus in absence
of state-of-the-art methods, we ablate the different components of our model
and compare to several strong baselines.

###  4.1 Datasets and Experimental Setup

#### Datasets.

We evaluate our method on Caltech-UCSD Birds-200-2011 (CUB-200) [Wah et al.
(2011)] and the Oxford-102 Flowers (FLO) dataset [Nilsback and Zisserman
(2006)]. For both datasets, Reed et al. (2016) have collected several visual
descriptions per image by crowd-sourcing to non-experts on Amazon Mechanical
Turk (AMT). We further collect for each class a corresponding expert document
from specialised websites, such as AllAboutBirds111https://allaboutbirds.com
(AAB) and Wikipedia.

| CUB-200 | FLO
---|---|---
Method | top-1$\uparrow$ | top-5$\uparrow$ | MR$\downarrow$ | top-1$\uparrow$ | top-5$\uparrow$ | MR$\downarrow$
random guess | 0.5 | 2.5 | 100.0 | 0.9 | 4.9 | 51.0
SRoBERTa-STSb Reimers and Gurevych (2019) (no-ft) | 1.3 | 6.4 | 73.4 | 1.1 | 7.7 | 45.2
SRoBERTa-NLI Liu et al. (2019) (no-ft) | 1.9 | 5.3 | 81.3 | 0.9 | 5.7 | 48.2
Okapi BM25 Robertson et al. (1995) | 1.0 | 7.5 | 78.2 | 1.6 | 8.0 | 43.9
TF-IDF Jones (1972) | 2.2 | 9.7 | 72.1 | 1.4 | 5.0 | 45.2
RoBERTa Liu et al. (2019) | 4.3 | 16.6 | 44.6 | 1.1 | 9.6 | 42.6
ours | 7.9 | 28.6 | 31.9 | 6.2 | 14.2 | 39.7
Table 2: Comparison to baselines. We report the retrieval performance of our
method on CUB-200 and Oxford-102 Flowers (FLO) and compare to various strong
baselines.

#### Setup.

We use the image-caption pairs to train two image captioning models: “Show,
Attend and Tell” (SAT) Xu et al. (2015) and AoANet Huang et al. (2019). Unless
otherwise specified, we report the performance of our model based on their
ensemble, _i.e_ combining predictions from both models. As the backbone $T$ of
our sentence transformer model, we use RoBERTa-large Liu et al. (2019) fine-
tuned on NLI and STS datasets using the setup of Reimers and Gurevych (2019).
Please see the appendix for further implementation, architecture, dataset and
training details.

We use three metrics to evaluate the performance on the benchmark datasets. We
compute top-1 and top-5 per-class retrieval accuracy and report the overall
average. Additionally, we compute the mean rank (MR) of the target document
for each class. Here, retrieval accuracy is identical to classification
accuracy, since there is only a single relevant article per category.

###  4.2 Baseline Comparisons

Since this work is the first to explore the mapping of images to expert
documents without expert supervision, we compare our method to several strong
baselines (Table 2).

Our FGSM performs text-based retrieval, we evaluate current text retrieval
systems. TF-IDF: Term frequency-inverse document frequency (TF-IDF) is widely
used for unsupervised document retrieval Jones (1972). For each image, we use
the predicted captions as queries and use the TF-IDF textual representation
for document ranking instead of our model. We empirically found the cosine
distance and $n$-grams with $n={2,3}$ to perform best for TF-IDF. BM25:
Similar to TF-IDF, BM25 Robertson et al. (1995) is another common measure for
document ranking based on $n$-gram frequencies. We use the BM25 Okapi
implementation from the python package rank-bm25 with default settings.
RoBERTa: One advantage of processing caption-sentence pairs with a Siamese
architecture, such as SBERT/SRoBERTa Reimers and Gurevych (2019), is the
reduced complexity. Nonetheless, we have trained a transformer baseline for
text classification, using the same backbone Liu et al. (2019), concatenating
each sentence pair with a SEP token and training as a binary classification
problem. We apply this model to score documents, instead of FGSM, aggregating
scores at sentence-level. SRoBERTa-NLI/STSb: Finally, to evaluate the
importance of learning _fine-grained_ sentence similarities, we also measure
the performance of the same model trained only on the NLI and STSb benchmarks
Reimers and Gurevych (2019), without further fine-tuning. Following Reimers
and Gurevych (2019) we rank documents based on the cosine similarity between
the caption and sentence embeddings.

Our method outperforms all bag-of-words and learned baselines. Approaches such
as TF-IDF and BM25 are very efficient, albeit less performant than learned
models. Notably, the closest in performance to our model is the transformer
baseline (RoBERTa), which comes at a large computational cost ($347$ sec vs.
$0.55$ sec for our model per image on CUB-200).

Method | top-1$\uparrow$ | top-5$\uparrow$ | MR$\downarrow$
---|---|---|---
user interaction | 11.9 | 37.5 | 24.8
FGSM + cosine | 4.5 | 17.8 | 35.5
FGSM w/ SAT | 4.3 | 15.0 | 42.9
FGSM w/ AoANet | 5.7 | 20.8 | 38.3
FGSM w/ ensemble | 5.9 | 20.0 | 36.1
FGSM $+\;R(\mathcal{B})$ [2-cls] | 7.4 | 24.6 | 29.9
FGSM $+\;R(\mathcal{B})$ [3-cls] | 7.9 | 28.6 | 31.9
Table 3: Ablation and user study. On CUB-200 we evaluate scoring functions,
captioning models and the regularizer $R(\mathcal{B})$.

Method | sup. | top-1$\uparrow$ | top-5$\uparrow$ | MR$\downarrow$
---|---|---|---|---
random guess | ✗ | 2.0 | 10.0 | 25.0
ViLBERT Lu et al. (2019) | ✗ | 3.5 | 14.8 | 20.2
TF-IDF Jones (1972) | ✗ | 7.2 | 28.6 | 18.9
CLIP Radford et al. (2021) | ✓ | 10.0 | 32.9 | 14.0
DSCMR Zhen et al. (2019) | ✓ | 13.5 | 34.7 | 15.2
ours | ✗ | 20.9 | 50.7 | 9.6
Table 4: Comparison to cross-media retrieval. We evaluate the performance of
methods on the ZSL split of CUB-200. Our method performs favorably against
existing approaches trained with more supervision.

###  4.3 Ablation & User Interaction

We ablate the different components of our approach in Table 4. We first
investigate the use of a different scoring mechanism, _i.e_ the cosine
similarity between the embeddings of $c$ and $s$ as in Reimers and Gurevych
(2019); we found this to perform worse (FGSM + cosine). We also study the
influence of the captioning model on the final performance. We evaluate
captions obtained by two methods, SAT Xu et al. (2015) and AoANet Huang et al.
(2019), as well as their ensemble. The ensemble improves performance thanks to
higher variability in the image descriptions. Next, we evaluate the
performance of our model after the final training phase, with the proposed
regularizer and the inclusion of neutral pairs (Section 3.3). $R(\mathcal{B})$
imposes prior knowledge about the expected class distribution over the dataset
and thus stabilizes the training, resulting in improved performance ([2-cls]).
Further, through the regularizer and neutral sentences ([3-cls]), FGSM is
exposed to the target corpus during training, which helps reduce the domain
shift during inference compared to training on image descriptions alone (FGSM
w/ ensemble).

Finally, our method enables user interaction, _i.e_ allowing a user to
directly enter own descriptions, replacing the automatic description model. In
Table 4 we have simulated this by evaluating with ground-truth instead of
predicted descriptions. Naturally, we find that human descriptions indeed
perform better, though the performance gap is small. We attribute this gap to
a much higher diversity in the human annotations. Current image captioning
models still have diversity issues, which also explains why our ensemble
variant improves the results.

Figure 3: Qualitative Results (CUB-200). We show examples of input images and
their predicted captions, followed by the top-5 retrieved documents (classes).
For illustration purposes, we show a random image for each document; the image
is not used for matching.

###  4.4 Comparison with Cross-Modal Retrieval

Since the nature of the problem presented here is in fact cross-modal, we
adapt a representative method, DSCMR Zhen et al. (2019), to our data to
compare to the state of the art in cross-media retrieval. We note that such an
approach requires image-document pairs as training samples, thus using more
supervision than our method. Instead of using image descriptions as an
intermediary for retrieval, DSCMR thus performs retrieval monolithically,
mapping the modalities in a shared representation space. We argue that,
although this is the go-to approach in broader category domains, it may be
sub-optimal in the context of fine-grained categorization.

Since in our setting each category (species) is represented by a single
article, in the scenario that a supervised model sees _all_ available
categories during training, the cross-modal retrieval problem degenerates to a
classification task. Hence, for a meaningful comparison, we train both our
model and DSCMR on the CUB-200 splits for ZSL Xian et al. (2018a) to evaluate
on 50 _unseen_ categories. We report the results in Table 4, including a TF-
IDF baseline on the same split. Despite using no image-documents pairs for
training, our method still performs significantly better.

Additionally, we compare to representative methods from the vision-and-
language representation learning space. ViLBERT Lu et al. (2019) is a multi-
modal transformer model capable of learning joint representations of visual
content and natural language. It is pre-trained on 3.3M image-caption pairs
with two proxy tasks. We use their multi-modal alignment prediction mechanism
to compute the alignment of the sentences in a document to a target image,
similar to ViLBERT’s zero-shot experiments. The sentence scores are averaged
to get the document alignment score and the document with the maximum score is
chosen as the class. Finally, we compare to CLIP Radford et al. (2021), that
learns a multimodal embedding space from 400M image-text pairs. CLIP predicts
image and sentence embeddings with separate encoders. For a target image we
score each sentence using cosine similarity and average across the document
for the final score. CLIP’s training data is not public, but we find that
there is a high possibility it does indeed contain expert labels as removing
class names from documents hurts its performance.

###  4.5 Qualitative Results

In Fig. 3, we show qualitative retrieval results. The input image is shown on
the left followed by the predicted descriptions. We then show the top-5
retrieved documents/classes together with an example image for the reader.
Note that the example images are not used for matching, as the FGSM module
operates on text only. We find that in most cases, even when the retrieved
document does not match the ground truth class, the visual appearance is still
similar. This is especially noticeable in families of birds for which
discriminating among individual species is considered to be particularly
difficult even for humans, _e.g_ warblers (last row).

##  5 Discussion

Like with any method that aims to reduce supervision, our method is not
perfect. There are multiple avenues where our approach can be further
optimized.

First, we observe that models trained for image captioning tend to produce
short sentences that lack distinctiveness, focusing on the major features of
the object rather than providing detailed fine-grained descriptions of the
object’s unique aspects. We believe there is a scope for improvement if the
captioning models could extensively describe each different part and attribute
of the object. We have tried to mitigate this issue by using an ensemble of
two popular captioning networks. However, using multiple models and sampling
multiple descriptions may lead to redundancy. Devising image captioning models
that produce diverse and distinct fine-grained image descriptions may provide
improved performance on CLEVER task; there is an active area of research Wang
et al. (2020a, b) that is looking into this problem.

Second, the proposed approach to scoring a document given an image uses _all_
the sentences in the document classifying them as positive, negative or
neutral with respect to each input caption. Given that the information
provided by an expert document might be noisy, _i.e_ not necessarily related
to the _visual_ domain, it is likely worthwhile to develop a filtering
mechanism for relevancy, effectively using only a subset of the sentences for
scoring.

Finally, in-domain regularization results in a significant performance boost
(Table 4), which implies that the CLEVER task is susceptible to the domain gap
between laypeople’s descriptions and the expert corpus. Language models such
as BERT/RoBERTa partially address this problem already by learning general
vocabulary, semantics and grammar during pre-training on large text corpora,
enabling generalization to a new corpus without explicit training. However,
further research in reducing this domain gap seems worthwhile.

##  6 Conclusion

We have shown that it is possible to address fine-grained image recognition
without the use of expert training labels by leveraging existing knowledge
bases, such as Wikipedia. This is the first work to tackle this challenging
problem, with performance gains over the state of the art on cross-media
retrieval, despite their training with image-document pairs. While humans can
easily access and retrieve information from such knowledge bases, CLEVER
remains a challenging learning problem that merits future research.

## Acknowledgments

S. C. is supported by a scholarship sponsored by Facebook. I. L. is supported
by the European Research Council (ERC) grant IDIU-638009 and EPSRC VisualAI
EP/T028572/1. C. R. is supported by Innovate UK (project 71653) on behalf of
UK Research and Innovation (UKRI) and ERC grant IDIU-638009. A. V. is
supported by ERC grant IDIU-638009. We thank Andrew Brown for valuable
discussions.

## References

  * Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre.  SemEval-2012 task 6: A pilot on semantic textual similarity.  In _SEM 2012_ , pages 385–393, 7-8 June 2012.
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo.  *SEM 2013 shared task: Semantic textual similarity.  In _SEM 2013_ , pages 32–43, June 2013.
  * Akata et al. (2015) Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid.  Label-embedding for image classification.  _TPAMI_ , 38(7):1425–1438, 2015.
  * Andrew et al. (2013) Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu.  Deep canonical correlation analysis.  In _ICML_ , pages 1247–1255. PMLR, 2013.
  * Asano et al. (2020) Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  In _ICLR_ , 2020.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In _EMNLP_. Association for Computational Linguistics, 2015.
  * Branson et al. (2010) Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie.  Visual recognition with humans in the loop.  In _ECCV_ , pages 438–451. Springer, 2010.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _NeurIPS_ , 33, 2020.
  * Cui et al. (2016) Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie.  Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop.  In _CVPR_ , pages 1153–1162, 2016.
  * Deng et al. (2015) Jia Deng, Jonathan Krause, Michael Stark, and Li Fei-Fei.  Leveraging the wisdom of the crowd for fine-grained recognition.  _TPAMI_ , 38(4):666–676, 2015.
  * Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * Elhoseiny et al. (2016) Mohamed Elhoseiny, Ahmed Elgammal, and Babak Saleh.  Write a classifier: Predicting visual classifiers from unstructured text.  _TPAMI_ , 39(12):2539–2553, 2016.
  * Elhoseiny et al. (2017) Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal.  Link the head to the" beak": Zero shot learning from noisy text description at part precision.  In _CVPR_ , pages 6288–6297. IEEE, 2017.
  * Farhadi et al. (2009) A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth.  Describing objects by their attributes.  In _CVPR_ , pages 1778–1785, 2009.
  * Felix et al. (2018) Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro.  Multi-modal cycle-consistent generalized zero-shot learning.  In _ECCV_ , pages 21–37, 2018.
  * Frome et al. (2013) Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  DeViSE: A deep visual-semantic embedding model.  In _Proc. NeurIPS_ , 2013.
  * Ge et al. (2019) Weifeng Ge, Xiangru Lin, and Yizhou Yu.  Weakly supervised complementary parts models for fine-grained image classification from the bottom up.  In _CVPR_ , pages 3034–3043, 2019.
  * Gebru et al. (2017) Timnit Gebru, Judy Hoffman, and Li Fei-Fei.  Fine-grained recognition in the wild: A multi-task domain adaptation approach.  In _ICCV_ , pages 1349–1358, 2017.
  * He and Peng (2017) Xiangteng He and Yuxin Peng.  Fine-grained image classification via combining vision and language.  In _CVPR_ , pages 5994–6002, 2017.
  * He et al. (2019) Xiangteng He, Yuxin Peng, and Liu Xie.  A new benchmark and approach for fine-grained cross-media retrieval.  In _ACM Multimedia_ , pages 1740–1748, 2019.
  * Horn et al. (2017) Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie.  The iNaturalist challenge 2017 dataset.  _arXiv.cs_ , abs/1707.06642, 2017.
  * Hu et al. (2019) Peng Hu, Xu Wang, Liangli Zhen, and Dezhong Peng.  Separated variational hashing networks for cross-modal retrieval.  In _ACM Multimedia_ , pages 1721–1729, 2019.
  * Huang et al. (2019) Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In _ICCV_ , pages 4634–4643, 2019.
  * Huang and Li (2020) Zixuan Huang and Yin Li.  Interpretable and accurate fine-grained recognition via region grouping.  In _CVPR_ , pages 8662–8672, 2020.
  * Jones (1972) Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_ , 1972.
  * Kampffmeyer et al. (2019) Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing.  Rethinking knowledge graph propagation for zero-shot learning.  In _CVPR_ , pages 11487–11496, 2019.
  * Kodirov et al. (2017) Elyor Kodirov, Tao Xiang, and Shaogang Gong.  Semantic autoencoder for zero-shot learning.  In _CVPR_ , pages 3174–3183, 2017.
  * Krause et al. (2016) Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei.  The unreasonable effectiveness of noisy data for fine-grained recognition.  In _ECCV_ , pages 301–320. Springer, 2016.
  * Kumar et al. (2012) Neeraj Kumar, Peter N Belhumeur, Arijit Biswas, David W Jacobs, W John Kress, Ida C Lopez, and João VB Soares.  Leafsnap: A computer vision system for automatic plant species identification.  In _ECCV_ , pages 502–516. Springer, 2012.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li.  On the sentence embeddings from bert for semantic textual similarity.  In _EMNLP_ , pages 9119–9130, 2020.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Long et al. (2017) Yang Long, Li Liu, Ling Shao, Fumin Shen, Guiguang Ding, and Jungong Han.  From zero-shot learning to conventional supervised classification: Unseen visual data synthesis.  In _CVPR_ , pages 1627–1636, 2017.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_ , pages 13–23, 2019.
  * MacAvaney et al. (2019) Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian.  Cedr: Contextualized embeddings for document ranking.  In _Proceedings of the 42nd Intl. ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1101–1104, 2019.
  * Manning et al. (2008) Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan.  _Introduction to information retrieval_.  Cambridge university press, 2008.
  * Nilsback and Zisserman (2006) Maria-Elena Nilsback and Andrew Zisserman.  A visual vocabulary for flower classification.  In _CVPR_ , volume 2, pages 1447–1454, 2006.
  * Nilsback and Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, pages 722–729. IEEE, 2008.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_ , 2019.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin.  Document ranking with a pretrained sequence-to-sequence model.  _EMNLP_ , 2020.
  * Peng et al. (2016) Yuxin Peng, Xin Huang, and Jinwei Qi.  Cross-media shared representation by hierarchical learning with multiple deep networks.  In _IJCAI_ , pages 3846–3853, 2016.
  * Peng et al. (2017a) Yuxin Peng, Xin Huang, and Yunzhen Zhao.  An overview of cross-media retrieval: Concepts, methodologies, benchmarks, and challenges.  _IEEE Transactions on circuits and systems for video technology_ , 28(9):2372–2385, 2017a.
  * Peng et al. (2017b) Yuxin Peng, Jinwei Qi, Xin Huang, and Yuxin Yuan.  Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network.  _IEEE Transactions on Multimedia_ , 20(2):405–420, 2017b.
  * Pereira et al. (2013) Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Nikhil Rasiwasia, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos.  On the role of correlation and abstraction in cross-modal multimedia retrieval.  _TPAMI_ , 36(3):521–535, 2013.
  * Qiao et al. (2016) Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel.  Less is more: zero-shot learning from online textual documents with noise suppression.  In _CVPR_ , pages 2249–2257, 2016.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  _ICML_ , 2021.
  * Reed et al. (2016) Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele.  Learning deep representations of fine-grained visual descriptions.  In _CVPR_ , pages 49–58, 2016.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _EMNLP-IJCNLP_ , pages 3973–3983, 2019.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al.  Okapi at trec-3.  _Nist Special Publication Sp_ , 109:109, 1995.
  * Simon and Rodner (2015) Marcel Simon and Erik Rodner.  Neural activation constellations: Unsupervised part model discovery with convolutional networks.  In _ICCV_ , pages 1143–1151, 2015.
  * Socher et al. (2013) Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng.  Zero-shot learning through cross-modal transfer.  In _NeurIPS_ , pages 935–943, 2013.
  * Van Gansbeke et al. (2020) Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Scan: Learning to classify images without labels.  In _ECCV_ , 2020.
  * Van Horn et al. (2015) Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In _CVPR_ , pages 595–604, 2015.
  * Van Horn et al. (2018) Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _CVPR_ , pages 8769–8778, 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , pages 5998–6008, 2017.
  * Vedaldi et al. (2014) Andrea Vedaldi, Siddharth Mahendran, Stavros Tsogkas, Subhransu Maji, Ross Girshick, Juho Kannala, Esa Rahtu, Iasonas Kokkinos, Matthew B Blaschko, David Weiss, et al.  Understanding objects in detail with fine-grained attributes.  In _CVPR_ , pages 3622–3629, 2014.
  * Vyas et al. (2020) Maunil R. Vyas, Hemanth Venkateswara, and Sethuraman Panchanathan.  Leveraging seen and unseen semantic relationships for generative zero-shot learning.  abs/2007.09549, 2020.
  * Wah et al. (2011) Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. (2017) Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen.  Adversarial cross-modal retrieval.  In _ACM Multimedia_ , pages 154–162, 2017.
  * Wang et al. (2020a) Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan.  Compare and reweight: Distinctive image captioning using similar images sets.  In _ECCV_ , 2020a.
  * Wang et al. (2009) Josiah Wang, Katja Markert, and Mark Everingham.  Learning models for object recognition from natural language descriptions.  In _BMVC_ , 2009.
  * Wang et al. (2020b) Qingzhong Wang, Jia Wan, and Antoni B. Chan.  On diversity in image captioning: Metrics and methods.  _TPAMI_ , PP, 2020b.
  * Wang and Livescu (2016) Weiran Wang and Karen Livescu.  Large-scale approximate kernel canonical correlation analysis.  _ICLR_ , 2016.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, 2018.
  * Xian et al. (2016) Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele.  Latent embeddings for zero-shot classification.  In _CVPR_ , pages 69–77, 2016.
  * Xian et al. (2018a) Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  _TPAMI_ , 41(9):2251–2265, 2018a.
  * Xian et al. (2018b) Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata.  Feature generating networks for zero-shot learning.  In _CVPR_ , pages 5542–5551, 2018b.
  * Xian et al. (2019) Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata.  f-vaegan-d2: A feature generating framework for any-shot learning.  In _CVPR_ , pages 10275–10284, 2019.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _ICML_ , pages 2048–2057, 2015.
  * Xu et al. (2016) Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao.  Webly-supervised fine-grained visual categorization via deep domain adaptation.  _TPAMI_ , 40(5):1100–1113, 2016.
  * Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin.  Cross-domain modeling of sentence-level evidence for document retrieval.  In _EMNLP-IJCNLP_ , pages 3481–3487, 2019.
  * Zhang et al. (2014) Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell.  Part-based r-cnns for fine-grained category detection.  In _ECCV_ , pages 834–849. Springer, 2014.
  * Zhen et al. (2019) Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng.  Deep supervised cross-modal retrieval.  In _Proc. CVPR_ , pages 10394–10403, 2019.
  * Zheng et al. (2017) Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo.  Learning multi-attention convolutional neural network for fine-grained image recognition.  In _ICCV_ , pages 5209–5217, 2017.
  * Zhu et al. (2018) Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal.  A generative adversarial approach for zero-shot learning from noisy texts.  In _CVPR_ , pages 1004–1013, 2018.