# Coherent Control of Collective Spontaneous Emission through Self-
interference

Lei Qiao<EMAIL_ADDRESS>Department of Physics, National University of
Singapore, Singapore 117551, Singapore  Jiangbin Gong<EMAIL_ADDRESS>Department of Physics, National University of Singapore, Singapore 117551,
Singapore

###### Abstract

As one of the central topics in quantum optics, collective spontaneous
emission such as superradiance has been realized in a variety of systems. This
work proposes an innovative scheme to coherently control collective emission
rates via a self-interference mechanism in a nonlinear waveguide setting. The
self-interference is made possible by photon backward scattering incurred by
quantum scatterers in a waveguide working as quantum switches. Whether the
interference is constructive or destructive is found to depend strongly on the
distance between the scatterers and the emitters. The interference between two
propagation pathways of the same photon leads to controllable superradiance
and subradiance, with their collective decay rates much enhanced or suppressed
(also leading to hyperradiance or population trapping). Furthermore, the self-
interference mechanism is manifested by an abrupt change in the emission rates
in real time. An experimental setup based on superconducting transmission line
resonators and transmon qubits is further proposed to realize controllable
collective emission rates.

_Introduction._ —Waveguide quantum electrodynamics has recently been a growing
area in quantum optics with important applications in quantum information
processing [1, 2, 3, 4, 5]. Different integrations of quantum emitters (QEs)
with nanophotonic structures have been achieved, such as guided surface
plasmons confined on a conducting nanowire with individual optical emitters
[6, 7], photonic nanowire coupled by embedded quantum dots [8, 9], or
superconducting transmission line with superconducting qubits [10, 11, 12].
These physical platforms make it possible to let QEs interact with one-
dimensional bosonic modes with nontrivial dispersions, leading to intriguing
dynamics such as persistent quantum beats [13, 14], supercorrelated radiance
[15], and single photons by quenching the vacuum [16]. Indeed, enhanced light-
matter interaction because of dimensionality reduction in waveguide quantum
electrodynamics continues to attract a great deal of attention [17, 18, 19,
20, 21, 22, 23, 24, 25, 26, 27]. Advances in designing and probing light-
matter interactions have allowed the investigation of collective phenomena
such as superradiance, subradiance [28, 29, 30, 31, 32, 33, 34, 35, 36] ,
cavity antiresonance spectroscopy [37], and nonequilibrium collective phase
transition [38, 39].

As a typical collective emission, Dicke superradiance has been demonstrated in
systems of hot atoms [40, 41], cold atoms [42, 43], trapped ions [44, 45, 46],
and superconducting qubits [47, 48], etc. Its counterpart with reduced
emission rate is termed subradiance. Transitions between superradiant and
subradiant states have been realized in superconducting circuits by initially
applying phase gate on each qubit [49]. Superradiance and subradiance are
highly relevant to quantum memory as their roles can be important in the
writing and reading of quantum information [50, 51]. However, to date
continuously controllable collective emission rate without using external
driving fields [52, 53, 54] remains a challenge for almost all QE systems.

In this letter we reveal an unknown aspect of spontaneous emission in a
nonlinear waveguide setting. We consider quantum scatterers in addition to
general quantum emitters in the same waveguide. The emitted photon propagating
in the waveguide can be bounced back by the scatterers as quantum switches.
The backward scattered photon then interferes with the other branch of the
photon propagating in the opposite direction. Such self-interference is
exploited to achieve continuous and extensive control of the spontaneous
emission rate of QEs. As shown below, even the transition from superradiance
to subradiance can be readily achieved if we control certain features of the
quantum switches, such as its resonance frequency and the QE-scatterer
distance. The underlying self-interference mechanism is analyzed both
qualitatively and quantitatively, with excellent agreement between physical
insights and theoretical studies. In particular, the QE-scatterer distance is
found to be a crucial parameter to induce constructive or destructive
interference. An experimental setup based on superconducting transmission line
resonators and transmon qubits is further proposed to realize continuously
controllable collective emission rates.

_Model._ —The system we consider consists of a one-dimensional array of
tunneling-coupled cavities which accomodate one assembly of QEs at position
$x=x_{1}$, as well as a second collection of two-level atoms at $x_{2}$
respectively. A schematic plot of this configuration is shown in Fig. 1. Atoms
at $x_{2}$ play the role of quantum scatterers, through which the spontaneous
emission dynamics of QEs at $x_{1}$ is to be manipulated. Though playing two
different roles, these two collections of atoms will be treated with similar
notation, indexed by $A$ and $B$ respectively, and assumed to have excited
states $|e^{A}\rangle$, $|e^{B}\rangle$ and ground states $|g^{A}\rangle$,
$|g^{B}\rangle$, separated in energy by frequencies $\Omega_{A}$ and
$\Omega_{B}$ (we set $\hbar=1$ throughout). The tunneling-coupled photonic
waveguide forms a lattice, modeled by the following tight-binding Hamiltonian

$H_{\text{ph}}=\sum_{x}\omega_{c}a_{x}^{{\dagger}}a_{x}+J\sum_{x}\left(a_{x+1}^{{\dagger}}a_{x}+a_{x}^{{\dagger}}a_{x+1}\right)\text{,}$
(1)

where $a_{x}^{{\dagger}}$ is the creation operator of the waveguide mode at
position $x$, and $\omega_{c}$ is the resonance frequency of a single cavity.
For convenience, we assume the lattice constant to be $a=1$ throughout. The
total Hamiltonian describing the system is then

$\displaystyle H$
$\displaystyle=H_{\text{ph}}+\sum_{j}\Omega_{j,A}\left|e_{j}^{A}\right\rangle\left\langle
e_{j}^{A}\right|+\sum_{j}\Omega_{j,B}\left|e_{j}^{B}\right\rangle\left\langle
e_{j}^{B}\right|$
$\displaystyle+\sum_{j}\left(V_{j,A}\sigma_{j,A}^{+}a_{x_{1}}+V_{j,B}\sigma_{j,B}^{+}a_{x_{2}}+h.c\right)\text{,}$
(2)

where $\sigma_{j,A}^{+}$ and $\sigma_{j,B}^{+}$ are the creation operators for
the $j$th atom in each assembly and $V_{j,A}$ and $V_{j,B}$ are the respective
coupling strengths. For the case of only one QE with weak coupling ($V\ll J$),
the QE decays with the radiation rate $\Gamma=V^{2}/J$ if the emitter is near
resonance with the frequency of a single cavity [55]. In the single excitation
subspace, the spectrum of Eq. (2) comprises discrete localized bound states
and a continuum of delocalized dressed states with energy
$\omega_{k}=\omega_{c}+2J\cos\left(k\right)$ vs the mode wavevector $k$, thus
forming a scattering band with $\omega_{c}$ being the band central frequency
with bandwidth $4J$ ($J>0$). The said bound states result in the known
fractional trapping of an emitted photon and nonexponential dynamics of the
spontaneous emission [56, 57]. If the excitation frequency of QEs is equal to
the single-cavity frequency $\omega_{c}$ (a condition assumed below), then the
momentum of a radiated photon is around $k_{m}=\pm\pi/2$. Note also that the
peak photon group velocity is $|v_{g}^{m}|=2J$ reached by the wavevector
$k_{m}$.

Figure 1: Schematic of a waveguide setup. A one-dimensional array of
resonators with nearest-neighbor tunneling $J$ hosts an assembly of quantum
emitters at $x_{1}$ and a second collection of two-level atoms as quantum
scatterers at $x_{2}$.

_General theoretical considerations._ —Let us now assume that the QEs and the
quantum scatterers are separated by a distance $\Delta x\equiv|x_{2}-x_{1}|$,
which is less than the half of the coherence length $\sim v_{g}^{m}/\Gamma$ of
a spontaneously emitted photon. The initial state of the whole system is that
one of the QEs is excited, or a superpositions of such configurations, with
the photon field in vacuum. This hence places the whole wavefunction in the
single-excitation invariant subspace. The time-evolving state at time $t$ can
be written as

$\left|\psi\left(t\right)\right\rangle=\left\\{\sum_{i}\sum_{j=1}^{M_{i}}C_{j}^{i}\left(t\right)\sigma_{j,i}^{+}+\sum_{k}C_{k}\left(t\right)a_{k}^{{\dagger}}\right\\}\left|g\text{,vac}\right\rangle\text{,}$
(3)

where $i=A,B$ and
$a_{k}^{{\dagger}}=(1/\sqrt{N})\sum_{x}e^{ikx}a_{x}^{{\dagger}}$. $M_{i}$ is
the number of the QEs or of the scatterers. $C_{j}^{i}$ is the excitation
amplitude for the $j$th atom in each collection of atoms, $C_{k}$ is the
amplitude of the waveguide mode with momentum $k$. Without loss of generality,
we assume that only QEs with indices $j_{n}$ may be excited at time zero,
i.e., $C_{j_{n}}^{A}(0)\neq 0$, $C_{j}^{A}(0)=0$ ($j\neq j_{n}$), and
$C_{j}^{B}(0)=C_{k}(0)=0$. If initially at most two excited QEs indexed by
$j_{1}$ and $j_{2}$ are involved in the initial excitation, then exact results
about their ensuing time dependence can be obtained: [58]

$\displaystyle C_{j_{1}}^{A}\left(t\right)$
$\displaystyle=\frac{L_{1}(s)C_{j_{1}}^{A}\left(0\right)+L_{2}(s)C_{j_{2}}^{A}\left(0\right)}{G(s)}e^{st}|_{s=-i\Omega_{A}}$
$\displaystyle+\sum_{m}\frac{iL_{2}(s)\left[C_{j_{1}}^{A}\left(0\right)+C_{j_{2}}^{A}\left(0\right)\right]}{\left(is-\Omega_{A}\right)[G(s)]^{\prime}}e^{st}|_{s=x_{m}}$
$\displaystyle-\sum_{\alpha=\pm}\int_{-1}^{1}\frac{\left[Q_{1}^{\alpha}C_{j_{1}}^{A}\left(0\right)+Q_{2}^{\alpha}C_{j_{2}}^{A}\left(0\right)\right]e^{i2Jyt}}{2\pi
i\alpha\left(y+\Omega_{A}\right)[Q_{1}^{\alpha}-Q_{2}^{\alpha}]}dy\text{,}$
(4)

where $L_{1,2}(s)$ and $G(s)$ strongly depend on the separation parameter
$\Delta x$, $Q_{1,2}^{\pm}$ are explicit functions of the integration variable
$y$, and $x_{m}$ is the roots of the equation $G(s)=0$ [58]. The imaginary
part of each $x_{m}$ corresponds to the inverse of system’s eigenenergies of
the localized photon-QE dressed states. In fact, the second term on the right-
hand side of Eq. (4) originates from the system’s photon-QE bound states with
nonzero field amplitudes. $C_{j_{2}}^{A}(t)$ can be obtained by exchanging the
positions of $C_{j_{1}}^{A}(0)$ and $C_{j_{2}}^{A}(0)$ in Eq. (4). In
obtaining the analytical expressions above, it has been assumed that each
collection of atoms (namely, among the QEs, or among the scatterers) are
identical and thus $\Omega_{j,i}$ and $V_{j,i}$ are independent of the atom
index $j$, denoted by $\Omega_{A}$, $\Omega_{B}$, $V_{A}$, and $V_{B}$.

_Cases with one single emitter._ —To gain some important insights first, we
consider a single QE indexed by $j_{1}$ coupled with mutiple scatterers
through the waveguide, with the initial state $|\psi(0)\rangle$ $=$
$\sigma_{A}^{+}|g$,vac$\rangle$. The time evolution of the excited state
population $P_{e}(t)=|C_{j_{1}}^{A}\left(t\right)|^{2}$ is shown in Fig. 2(a)
vs the detuning parameter $\Delta_{B}=\Omega_{B}-\omega_{c}$ depicting the
scatterers, with the QE-scatterer separation $\Delta x=7$ (i.e., 7 lattice
constants) as an example. At early times, the emission dynamics matches well
with that of a normal decay process $P_{e}(t)\approx e^{-\Gamma_{1}t}$ with
$\Gamma_{1}=V_{A}^{2}/J$ (assuming $V_{A}\ll J)$, without feeling the presence
of the scatterers. Later the scatterers makes a dramatic difference during the
spontaneous emission process. In particular, as the main reason to introduce
the scatterers in the first place, the scatterers as two-level systems can
extensively control the coherent transport of a single photon in the
waveguide, including a complete reflection of the emitted photon [5]. Hence,
once part or even the whole of the propagating photon towards the scatterers
comes back to the decaying QE, it will interfere with the other branch of the
photon propagating along the other direction. Supporting this physical
picture, Fig. 2(a) depicts an abrupt change of the radiation rate once $t$
reaches $t_{0}\approx 2\Delta x/(2J)$, yielding a cascade of stimulated
emission. In terms of the real-time dynamics, we are witnessing an intriguing
scenario that an emitted photon, when being bounced back, can further boost
the emission process that has not been completed yet. The degree of
enhancement is continuously controlled by the frequency of the scatterers, as
evidenced by the shown strong dependence of the emission rates vs the detuning
parameter $\Delta_{B}$.

Figure 2: (a) Excited state population $P_{e}(t)$ vs detuning $\Delta_{B}$ for
$\Delta x=7$. (b) Excited state population $P_{e}(t)$ vs detuning $\Delta_{B}$
for $\Delta x=8$. Time is in units of $1/(2J)$. The parameters are
$V_{A}/(2J)=0.08$, $V_{B}/(2J)=1.8$, $\Delta_{A}/(2J)=0$, the number of
emitters is $M_{A}=1$ and that of scatterers is $M_{B}=2$. In both (a) and (b)
the vertical dashed lines indicate the time of arrival of a reflected photon.

Continuing our investigations of the cases with $\Delta x=2n+1$ ($n$ being an
integer), let us further assume that the quantum scatterers are near resonance
with each single cavity with small $\Delta x$, plus the conditions $V_{A}\ll
2J$ and $\sqrt{M_{B}}V_{B}\sim 2J$. The enhanced emission rate under these
conditions are found to be
$P_{e}(t)\approx[M_{B}V_{B}^{2}/(4J^{2})-\alpha_{1}/2]^{2}e^{-2J\alpha_{1}t}/\beta_{1}$
with $\alpha_{1}=V_{A}^{2}/(J^{2}-\Delta xV_{A}^{2}/2)$ and $\beta_{1}$ is a
time-independent quantity [58]. As can be seen in Fig. 2(a), our approximate
theoretical results agree well with the exact results obtained from Eq. (4).
For $\Delta x$ being small enough, the radiance rate is two times the normal
decay rate. Although there is only one QE here, the emission rate is much
enhanced and even beyond two-QE Dicke superradiance. The physical
understanding is the following: The QE interferes with itself via a delayed
photon, thus it can effectively realize collective interference and hence
achieve superradiance.

Next we consider cases with $\Delta x=2n$. A few computational examples with
$\Delta x=8$ are shown in Fig. 2(b), where suppressed emission rates are
clearly observed. As the frequency of the scatterers is tuned from
$\omega_{c}$ to the values far away from the photon band, the suppression
becomes weak and ultimately the emission comes back to the normal decay. It is
curious to qualitatively understand why superradiance and subradiance are
observed for $\Delta=2n+1$ and $\Delta=2n$ in Fig. 2(a) and Fig. 2(b),
respectively. If $\Delta x=2n+1$, the phase difference incurred by the round
travel of the bounced photon can be estimated as $|k_{m}|(2\Delta
x)=(2n+1)\pi$, if considering the main wave component around $k_{m}=\pm\pi/2$
with the largest group velocity. Also accounting for the $\pi$ shift
associated with a complete photon reflection, the overall phase difference
between the bounced photon and the original photon is thus $2n\pi$, yielding
constructive interference and hence enhanced emission. By contrast, if $\Delta
x=2n$ is chosen, then the overall phase difference between the two interfering
pathways is $(2n+1)\pi$, thus producing destructive interference and leading
to suppression of the emission and thus subradiance. Confirming this
understanding, in Fig. 3(a), we further show how the emission rates for weak
coupling $V_{B}$ is changed over a wide range if the QE-scatterer distance
$\Delta x$ is adjusted.

Our results above have clearly indicated the important role of the backward
scattering in the self-interference mechanism. It is hence useful to examine
some details of the scattering process. When the radiation field reaches the
scatterers, part of the field is reflected with the reflection amplitude
$r_{k}$ given by
$r_{k}=M_{B}V_{B}^{2}[i2J|\sin(k)|(\omega_{k}-\Omega_{B})-M_{B}V_{B}^{2}]^{-1}$
[58]. Around the resonance, the reflection spectrum yields the so-called
Breit-Wigner line shape with the spectrum width given by
$M_{B}V_{B}^{2}/|J\sin(k_{0})|$, where $k_{0}$ is determined by the relation
$4J^{2}\sin^{2}(k)(\omega_{k}-\Omega_{B})^{2}=M_{B}^{2}V_{B}^{4}$ [5]. In
particular, if the photon energy $\omega_{k}$ is under resonance with the two-
level scatterers, namely, $\omega_{k}=\Omega_{B}$, one then obtains complete
reflection with $r_{k}=-1$ (hence the above-mentioned $\pi$ phase shift).
Under the parameter setting $\omega_{c}=\Omega_{B}$, the resonance scattering
condition $\omega_{k}=w_{c}+2J\cos(k)=\Omega_{B}$ occurs for $k=\pi/2$. From
the expression of $r_{k}$, it is also seen that if $k=0$ or $k=\pi$, complete
reflection happens also. However, this is irrelevant to our self-interference
mechanism because such components have a vanishing group velocity in the
waveguide. In Fig. 3(b), we show the reflection and transmission spectra vs
momentum $k$.

Figure 3: (a) Excited state population $P_{e}(t)$ vs $\Delta x$, with
$V_{A}/(2J)=0.09$, $V_{B}/(2J)=0.07$, $\Delta_{A}/(2J)=0$,
$\Delta_{B}/(2J)=0$, $M_{A}=1$ and $M_{B}=2$. (b) Reflection coefficient
$R=\left|r_{k}\right|^{2}$ and transmission coefficient $T=|1+r_{k}|^{2}$ as a
function of momentum $k$. Parameters are $\Delta_{B}/(\sqrt{M_{B}}V_{B})=0$,
$J/(\sqrt{M_{B}}V_{B})=1.13$. (c) Excited state population
$2|C_{\text{2}}\left(t\right)|^{2}$, with $\Delta_{A}/(2J)=0$,
$\Delta_{B}/(2J)=0$, $V_{A}/(2J)=0.08$ , $V_{B}/(2J)=1.27$, $M_{A}=2$ and
$M_{B}=2$. (d) Excited state population $5|C_{\text{5}}\left(t\right)|^{2}$,
with $\Delta_{A}/(2J)=0$, $\Delta_{B}/(2J)=0$, $V_{A}/(2J)=0.04$ ,
$V_{B}/(2J)=1.0$, $M_{A}=5$ and $M_{B}=2$. Time is in units of $1/(2J)$.
Different vertical dashed lines indicate the different times of arrival of a
reflected photon.

_Cases with two and more emitters._ —We now examine how collective emission
rates with two QEs can be manipulated by exploiting self-interference. The two
emitters are prepared in typical single-photon entangled states
$|\psi^{\pm}\rangle$ $=$
$(1/\sqrt{2})(\sigma_{j_{1},A}^{+}\pm\sigma_{j_{2},A}^{+})|g$,vac$\rangle$.
For $|\psi^{-}\rangle$, the decay of QEs is completely suppressed since it is
a dark state that cannot emit a photon. Thus we focus on the emission dynamics
emanating from $|\psi^{+}\rangle$. When $V_{B}=0$ and the frequencies of QEs
lie outside the scattering band with $|\Delta_{A}\pm 2J|$ $\gg V_{A}$, the
evolution of $|\psi^{+}\rangle$ is dominated by a trapping regime due to the
presence of bound states [56]. For the case $V_{A}\ll 2J$, the decay of the
amplitudes $C_{j_{1}}^{A}(t)=C_{j_{2}}^{A}(t)\equiv
C_{\text{2}}\left(t\right)$ is basically exponential, with a very slowly
changing radiation rate as $\Omega_{A}$ is tuned from from $\omega_{c}\pm J$
to $\omega_{c}$. This is what one expects from the Wigner-Weisskopf and
Markovian perturbative theories, which predict
$|C_{\text{2}}\left(t\right)|^{2}\approx(1/2)e^{-\Gamma_{s}(\Delta_{A})t}$,
with a decay rate $\Gamma_{s}(\Delta_{A})=4\pi V_{A}^{2}D(\Delta_{A})$, where
$D(\Delta_{A})$ is the density of states for the photon Hamiltonian
$H_{\text{ph}}$. $D(\Delta_{A})$ reaches its extremum under the resonance
condition $\Delta_{A}=0$ and $\Gamma_{s}(0)\equiv\Gamma_{s}=2V_{A}^{2}/J$,
which is twice the normal decay rate shown in Fig. 2. Indeed, this is what the
standard superradiance theory predicts.

Consider now what happens if the excitation frequency $\Omega_{A}$ of QEs is
around the middle of the band with $|\Delta_{A}|$ $\ll 2J$ in the presence of
scatterers. The frequency $\Omega_{B}$ of the scatterers controls the position
of transmission valley of the radiated photon while $M_{B}$ and $V_{B}$
determines the valley width. As in the case of one QE, before $t$ reaches
$t_{0}$, $|C_{\text{2}}\left(t\right)|^{2}$ behaves as a superradiant state
without sensing the quantum scatterers. Beyond $t_{0}$, it is found that the
superradiance rate can be much enhanced if $\Delta x=2n+1$ and much suppressed
if $\Delta x=2n$. Representative results are shown in Fig. 3(c). It is seen
that the self-interference mechanism works effectively in the case of
superradiance. In particular, the enhanced superradiance by constructive self-
interference may be termed hyperradiance, a terminology also used previously
but due to different physics [59]. Under the conditions $V_{A}\ll 2J$ and
$\sqrt{M_{B}}V_{B}\sim 2J$, we can make reasonable approximations using our
general theoretical expressions and find the following hyperradiance dynamics
with small $\Delta x$ [58]:

$C_{\text{2}}\left(t\right)\approx\frac{M_{B}V_{B}^{2}-J\Gamma_{h}}{\sqrt{2\left(\chi^{2}-16M_{B}V_{A}^{2}V_{B}^{2}\right)}}e^{-\frac{\Gamma_{h}}{2}t}\text{.}$
(5)

Here the radiance rate $\Gamma_{h}=4JV_{A}^{2}/(J^{2}-\Delta xV_{A}^{2})$ with
$\chi=M_{B}V_{B}^{2}-\Delta xM_{B}V_{A}^{2}V_{B}^{2}/J^{2}$. The dotted line
shown in Fig. 3(c) is obtained from Eq. (5), in excellent agreement with exact
results obtained directly from Eq. (4). Under the limit $\Delta x$ being small
enough, $\Gamma_{h}$ is found to be just two times the superradiance rate
$\Gamma_{h}=2\Gamma_{s}$. On the other hand, Eq. (5) indicates that
$\Gamma_{h}$ becomes large as $\Delta x$ increases. As such, tuning $\Delta x$
allows us to further boost hyperradiance rates.

Finally, we investigate how the self-interference mechanism works when there
are multiple QEs. In this case, we rely fully on computational studies since
it becomes tedious to find analytical results with more than two QEs being
initially excited. To investigate if the above hyperradiance dynamics can be
extended to cases with multiple QEs, we consider the following initial
amplitudes $C_{j_{1}}^{A}(0)=...=C_{j_{M_{A}}}^{A}(0)=1/\sqrt{M_{A}}$. Fig.
3(d) depicts the results with $M_{A}=5$ QEs for different values of $\Delta
x$. In the absence of scatterers, under the condition $\sqrt{M_{A}}V_{A}\ll
2J$, the amplitudes $C_{j_{1}}^{A}(t)$ = … = $C_{j_{M_{A}}}^{A}(t)\equiv
C_{M_{A}}(t)$ can be approximately described by
$|C_{M_{A}}\left(t\right)|^{2}\approx(1/M_{A})e^{-\Gamma_{s}t}$ with
$\Gamma_{s}=M_{A}V_{A}^{2}/J$, which is nothing but the Dicke superradiance.

However, for cases with $\Delta x=2n+1$, Fig. 3(d) shows that the self-
interference mechanism further boosts the collective emission rates by a
factor of two for relatively small $\Delta x$. Furthermore, as $\Delta x$
increases, the emission rate continues to be enhanced and hence surpasses
$2\Gamma_{s}$. This echoes with our observation in the case of two emitters.
To understand this intriguing trend due to increasing $\Delta x$, we first
note that the wavevevectors of the photon that will be backward scattered are
spread around $k=\pi/2$ (in the direction towards the scatterers), but only
the component with precisely $k=\pi/2$ of the largest group velocity can
optimally induce the constructive self-interference. If $\Delta x$ increases,
the potential phase dispersion among these components along the the
propagation pathway increases. This imposes a more strict selection on the
wave components that can contribute to the self-interference. These selected
wavecomponents also tend to induce the self-interference more synchronously.

For cases with $\Delta x=2n$, it is clearly observed in Fig. 3(d) that the
emission rates are much suppressed due to the destructive self-interference.
For suppressed subradiance, the asymptotic values of the excited state
population is finite at sufficiently long time. Indeed, under $\Delta_{B}=0$,
the emitted photon is first bounced back by the scatterers. Once the scattered
photon meets the QEs, destructive self-interference suppresses the collective
emission and as such the QEs tend to reflect the emitted photon as well, thus
also dynamically trapping the photon between the QEs and the scatterers. These
results and insights indicate the role of bound states in fully explaining the
population trapping on the excited state.

_Discussion and conclusions._ —In a one-dimensional nonlinear waveguide
setting, we have shown that the self-interference incurred by a backward
scattered photon originally emitted from quantum emitters can dramatically
change the emission rate. The backward scattering is caused by a collection of
quantum scatterers with a preselected distance. For an initial state as a
superposition of excitation of not more than two emitters, we obtain an exact
analytical solution to predict how quantum scatterers can be used to control
the emission rate. In both our theoretical treatments and our qualitative
analysis, it is found that the distance between the emitters and the
scatterers plays a critical role in deciding whether constructive or
destructive interference occurs. This self-interference mechanism then leads
to extensive control of the collective emission dynamics, ranging from
hyperrardiance to strongly suppressed subradiance. The theory and the physics
communicated in this work are rather general. Indeed, the considered quantum
emitters can be of different types, including both natural and artificial
ones.

It might not be straightforward to tune the separation between the emitters
and the scatterers if they are already grown in a nanophotonic structure. We
propose to install several different groups of quantum scatterers at different
positions in the waveguide. When the transition frequencies of the scatterers
are tuned to be outside the band and far away from the resonance frequency of
a single cavity, this group of scatterers can be considered to be turned off
and hence irrelevant to our self-interference mechanism. For this reason, one
can effectively realize the position tuning of the scatterers by adjusting
their resonance frequencies.

Finally, we discuss how the main idea of this work can be realized on an
experimental platform consisting of superconducting transmission line
resonators and transmon qubits. The coplanar transmission line resonators,
which offer the continuum for the coherent transport of nonlinear photons, can
be constructed by many equal transmission-line segments coupling with each
other by dielectric materials [5, 60]. The transmon qubit is viewed as a
Cooper pair box shunted by a capacitor that is large relative to the stray
capacitance of the Josephson junction [61]. The frequencies of transmon qubits
can be manipulated via the external magnetic flux intersecting the loop formed
by the SQUID [61, 62]. The practicable values for the coupling strength
between resonator and transmon qubits range from a few to hundreds of MHz
while the qubit frequency can be controlled from a few to tens of GHz that is
similar to the range of the resonance frequency of each resonator [2, 63].

_Acknowledgments._ — J.G. acknowledges fund support by the Singapore Ministry
of Education Academic Research Fund Tier-3 grant No. MOE2017-T3-1-001.

## References

  * [1] X. Gu, A. F. Kockum, A. Miranowicz, Y. X. Liu, and F. Nori, Microwave photonics with superconducting quantum circuits, Phys. Rep. 718–719, 1 (2017).
  * [2] G. Wendin, Quantum information processing with superconducting circuits: a review, Rep. Prog. Phys. 80, 106001 (2017).
  * [3] D. Roy, C. M. Wilson, and O. Firstenberg, Strongly interacting photons in one-dimensional continuum, Rev. Mod. Phys. 89, 021001 (2017).
  * [4] J.-T. Shen and S. Fan, Coherent single photon transport in a one-dimensional waveguide coupled with superconducting quantum bits, Phys. Rev. Lett. 95, 213001 (2005).
  * [5] L. Zhou, Z. R. Gong, Y. X. Liu, C. P. Sun, and F. Nori, Controllable scattering of a single photon inside a one-dimensional resonator waveguide, Phys. Rev. Lett. 101, 100501 (2008).
  * [6] D. E. Chang, A. S. Sorensen, E. A. Demler, and M. D. Lukin, A single-photon transistor using nanoscale surface plasmons, Nat. Phys. 3, 807 (2007).
  * [7] M. Rycenga, C. M. Cobley, J. Zeng, W. Li, C. H. Moran, Q. Zhang, D. Qin, and Y. Xia, Controlling the synthesis and assembly of silver nanostructures for plasmonic applications, Chem. Rev. 111, 3669 (2011).
  * [8] J. Claudon, J. Bleuse, N. S. Malik, M. Bazin, P. Jaffrennou, N. Gregersen, C. Sauvan, P. Lalanne, and J.-M. Gerard, A highly efficient single-photon source based on a quantum dot in a photonic nanowire, Nat. Photonics 4, 174 (2010).
  * [9] P. Lodahl, S. Mahmoodian, and S. Stobbe, Interfacing single photons and single quantum dots with photonic nanostructures, Rev. Mod. Phys. 87, 347 (2015).
  * [10] O. Astafiev, A. M. Zagoskin, A. A. Abdumalikov Jr., Y. A. Pashkin, T. Yamamoto, K. Inomata, Y. Nakamura, and J. S. Tsai, Resonance fluorescence of a single artificial atom, Science 327, 840 (2010).
  * [11] J. You and F. Nori, Atomic physics and quantum optics using superconducting circuits, Nature 474, 589 (2011).
  * [12] A. A. Houck, H. E. Tureci, and J. Koch, On-chip quantum simulation with superconducting circuits, Nat Phys 8, 292 (2012).
  * [13] H. Zheng and H. U. Baranger, Persistent quantum beats and long-distance entanglement from waveguide-mediated interactions, Phys. Rev. Lett. 110, 113601 (2013).
  * [14] G. Z. Song, J. L. Guo, W. Nie, L. C. Kwek, and G. L. Long, Optical properties of a waveguide-mediated chain of randomly positioned atoms, Opt. Express 29, 1903 (2021).
  * [15] Z. H. Wang, T. Jaako, P. Kirton, and P. Rabl, Supercorrelated radiance in nonlinear photonic waveguides, Phys. Rev. Lett. 124, 213601 (2020).
  * [16] E. Sánchez-Burillo, L. Martín-Moreno, J. J. García-Ripoll, and D. Zueco, Single photons by quenching the vacuum, Phys. Rev. Lett. 123, 013601 (2019).
  * [17] L. Guo, A. Frisk Kockum, F. Marquardt, and G. Johansson, Oscillating bound states for a giant atom, Phys. Rev. Research 2, 043014 (2020).
  * [18] G. Andersson, B. Suri, L. Guo, T. Aref, and P. Delsing, Non-exponential decay of a giant artificial atom, Nat. Phys. 15, 1123 (2019).
  * [19] H. Pichler and P. Zoller, Photonic circuits with time delays and quantum feedback, Phys. Rev. Lett. 116, 093601 (2016).
  * [20] P. Facchi, S. Pascazio, F. V. Pepe, and D. Pomarico, Correlated photon emission by two excited atoms in a waveguide, Phys. Rev. A 98, 063823 (2018).
  * [21] A. González-Tudela and J. I. Cirac, Quantum emitters in two-dimensional structured reservoirs in the nonperturbative regime, Phys. Rev. Lett. 119, 143602 (2017).
  * [22] S. Garmon, K. Noba, G. Ordonez, and D. Segal, Non-Markovian dynamics revealed at a bound state in the continuum, Phys. Rev. A 99, 010102(R) (2019).
  * [23] G. Calajó, Y.-L. L. Fang, H. U. Baranger, and F. Ciccarello, Exciting a bound state in the continuum through multiphoton scattering plus delayed quantum feedback, Phys. Rev. Lett. 122, 073601 (2019).
  * [24] F. Dinc and A. M. Brańczyk, Non-Markovian super-superradiance in a linear chain of up to 100 qubits, Phys. Rev. Research 1, 032042(R) (2019).
  * [25] E. Sánchez-Burillo, D. Zueco, L. Martín-Moreno, and J. J. García-Ripoll, Dynamical signatures of bound states in waveguide QED, Phys. Rev. A 96, 023831 (2017).
  * [26] L. Qiao and C. P. Sun, Atom-photon bound states and non-markovian cooperative dynamics in coupled-resonator waveguides, Phys. Rev. A 100, 063806 (2019).
  * [27] L. Qiao and C. P. Sun, Universal trapping law induced by an atomic cloud in single-photon cooperative dynamics, Phys. Rev. A 101, 063831 (2020).
  * [28] R. H. Dicke, Coherence in spontaneous radiation processes, Phys. Rev. 93, 99 (1954).
  * [29] M. Gross and S. Haroche, Superradiance: An essay on the theory of collective spontaneous emission, Phys. Rep. 93, 301 (1982).
  * [30] K. Sinha, P. Meystre, E. A. Goldschmidt, F. K. Fatemi, S. L. Rolston, and P. Solano, Non-Markovian collective emission from macroscopically separated emitters, Phys. Rev. Lett. 124, 043603 (2020).
  * [31] A. A. Svidzinsky, J. T. Chang, and M. O. Scully, Dynamical Evolution of Correlated Spontaneous Emission of a Single Photon from a Uniformly Excited Cloud of $N$ Atoms, Phys. Rev. Lett. 100, 160504 (2008).
  * [32] G. Y. Slepyan and A. Boag, Quantum nonreciprocity of nanoscale antenna arrays in timed Dicke states, Phys. Rev. Lett. 111, 023602 (2013).
  * [33] S. D. Jenkins, J. Ruostekoski, N. Papasimakis, S. Savo, and N. I. Zheludev, Many-body subradiant excitations in metamaterial arrays: Experiment and theory, Phys. Rev. Lett. 119, 053901 (2017).
  * [34] Y. Ke, A. V. Poshakinskiy, C. Lee, Y. S. Kivshar, and A. N. Poddubny, Inelastic scattering of photon pairs in qubit arrays with subradiant states, Phys. Rev. Lett. 123, 253601 (2019).
  * [35] Y.-X. Zhang and K. Molmer, Theory of subradiant states of a one-dimensional two-level atom chain, Phys. Rev. Lett. 122, 203605 (2019).
  * [36] T. Bienaimé, N. Piovella, and R. Kaiser, Controlled Dicke subradiance from a large cloud of two-level systems, Phys. Rev. Lett. 108, 123602 (2012).
  * [37] D. Plankensteiner, C. Sommer, H. Ritsch, and C. Genes, Cavity antiresonance spectroscopy of dipole coupled subradiant arrays, Phys. Rev. Lett. 119, 093601 (2017).
  * [38] M. R. Bakhtiari, A. Hemmerich, H. Ritsch, and M. Thorwart, Nonequilibrium phase transition of interacting bosons in an intra-cavity optical lattice, Phys. Rev. Lett. 114, 123601 (2015).
  * [39] K. Yamamoto, M. Nakagawa, N. Tsuji, M. Ueda, and N. Kawakami, Collective Excitations and Nonequilibrium Phase Transition in Dissipative Fermionic Superfluids, Phys. Rev. Lett. 127, 055301 (2021).
  * [40] N. Skribanowitz, I. P. Herman, J. C. MacGillivray, and M. S. Feld, Observation of Dicke superradiance in optically pumped HF gas, Phys. Rev. Lett. 30, 309 (1973).
  * [41] M. Gross, C. Fabre, P. Pillet, and S. Haroche, Observation of near-infrared Dicke superradiance on cascading transitions in atomic sodium, Phys. Rev. Lett. 36, 1035 (1976).
  * [42] S. Inouye, A. Chikkatur, D. Stamper-Kurn, J. Stenger, D. Pritchard, and W. Ketterle, Superradiant Rayleigh scattering from a Bose-Einstein condensate, Science 285, 571 (1999).
  * [43] K. Baumann, C. Guerlin, F. Brennecke, and T. Esslinger, Dicke quantum phase transition with a superfluid gas in an optical cavity, Nature (London) 464, 1301 (2010).
  * [44] R. G. DeVoe and R. G. Brewer, Observation of superradiant and subradiant spontaneous emission of two trapped ions, Phys. Rev. Lett. 76, 2049 (1996).
  * [45] S. Begley, M. Vogt, G. K. Gulati, H. Takahashi, and M. Keller, Optimized multi-ion cavity coupling, Phys. Rev. Lett. 116, 223001 (2016).
  * [46] B. Casabone, K. Friebe, B. Brandstatter, K. Schuppert, R. Blatt, and T. E. Northup, Enhanced quantum interface with collective ion-cavity coupling, Phys. Rev. Lett. 114, 023602 (2015).
  * [47] J. A. Mlynek, A. A. Abdumalikov, C. Eichler, and A. Wallraff, Observation of Dicke superradiance for two artificial atoms in a cavity with high decay rate, Nat. Commun. 5, 5186 (2014).
  * [48] S. Filipp, A. F. van Loo, M. Baur, L. Steffen, and A. Wallraff, Preparation of subradiant states using local qubit control in circuit QED, Phys. Rev. A 84, 061805 (2011).
  * [49] Z. Wang, H. Li, W. Feng, X. Song, C. Song, W. Liu, Q. Guo, X. Zhang, H. Dong, D. Zheng, H. Wang, and D.-W. Wang, Controllable switching between superradiant and subradiant states in a 10-qubit superconducting circuit, Phys. Rev. Lett. 124, 013601 (2020).
  * [50] M. S. Mendes, P. L. Saldanha, J. W. R. Tabosa, and D. Felinto, Dynamics of the reading process of a quantum memory, New J. Phys. 15, 075030 (2013).
  * [51] M. O. Scully, Single photon subradiance: quantum control of spontaneous emission and ultrafast readout, Phys. Rev. Lett. 115, 243602 (2015).
  * [52] M. Woldeyohannes and S. John, Coherent control of spontaneous emission near a photonic band edge: A qubit for quantum computation, Phys. Rev. A 60, 5046 (1999).
  * [53] M. Woldeyohannes and S. John, Coherent control of spontaneous emission near a photonic band edge, J. Opt. B 5, R43 (2003).
  * [54] X. Q. Jiang, B. Zhang, Z. W. Lu, and X. D. Sun, Control of spontaneous emission from a microwave-field-coupled three-level $\Lambda$-type atom in photonic crystals, Phys. Rev. A 83, 053823 (2011).
  * [55] F. Lombardo, F. Ciccarello, and G. M. Palma, Photon localization versus population trapping in a coupled-cavity array, Phys. Rev. A 89, 053826 (2014).
  * [56] S. John and T. Quang, Spontaneous emission near the edge of a photonic band gap, Phys. Rev. A 50, 1764 (1994).
  * [57] L. Qiao, Y. J. Song, and C. P. Sun, Quantum phase transition and interference trapping of populations in a coupled-resonator waveguide, Phys. Rev. A 100, 013825 (2019).
  * [58] In this Supplemental Material, in which the Refs. [5, 56, 57] are cited, we provide further details on various theoretical treatments in solving the Schrödinger equation and on a theory of photon scattering due to the presence of the scatterers as quantum switchers.
  * [59] M.-O. Pleinert, J. von Zanthier, and G. S. Agarwal, Hyperradiance from collective behavior of coherently driven atoms, Optica 4, 779 (2017).
  * [60] L. Zhou, Y. B. Gao, Z. Song, and C. P. Sun, Coherent output of photons from coupled superconducting transmission line resonators controlled by charge qubits, Phys. Rev. A 77, 013831 (2008).
  * [61] J. Koch, T. M. Yu, J. Gambetta, A. A. Houck, D. I. Schuster, J. Majer, A. Blais, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Charge-insensitive qubit design derived from the Cooper pair box, Phys. Rev. A 76, 042319 (2007).
  * [62] M. A. Sillanpää, J. I. Park, and R. W. Simmonds, Coherent quantum state storage and transfer between two phase qubits via a resonant cavity, Nature (London) 449, 438 (2007).
  * [63] A. Blais, A. L. Grimsmo, S. M. Girvin, and A. Wallraff, Circuit quantum electrodynamics, Rev. Mod. Phys. 93, 025005 (2021).

Supplementary material for:
Coherent Control of Collective Spontaneous Emission through self-interference

##  SM1 Dynamics of collective Spontaneous emission

The time evolution of the amplitudes defined in Eq. (3) of the main text is
given by

$i\frac{\partial}{\partial
t}C_{j}^{i}\left(t\right)=\Omega_{j,i}C_{j}^{i}\left(t\right)+\sum_{k}\frac{V_{j,i}e^{ikx_{\left(i\right)}}}{\sqrt{N}}C_{k}\left(t\right)\text{,}$
(SM1) $i\frac{\partial}{\partial
t}C_{k}\left(t\right)=\omega_{k}C_{k}\left(t\right)+\sum_{j,i}\frac{V_{j,i}e^{-ikx_{\left(i\right)}}}{\sqrt{N}}C_{j}^{i}\left(t\right)\text{,}$
(SM2)

where $x_{\left(A\right)}=x_{1}$ and $x_{\left(B\right)}=x_{2}$. To proceed
further one may invoke the Wigner-Weisskopf and Markovian theories by
neglecting the contributions of bound states, but this treatment would not be
able to capture the fractional trapping [1, 2]. To go beyond these
approximations, we take the Laplace transform of Eqs. (SM1) and (SM2),
yielding

$i\left[-C_{j}^{i}\left(0\right)+s\tilde{C}_{j}^{i}\left(s\right)\right]=\Omega_{j,i}\tilde{C}_{j}^{i}\left(s\right)+\sum_{k}\frac{V_{j,i}e^{ikx_{\left(i\right)}}}{\sqrt{N}}\tilde{C}_{k}\left(s\right)\text{,}$
(SM3)
$i\left[-C_{k}\left(0\right)+s\tilde{C}_{k}\left(s\right)\right]=\omega_{k}\tilde{C}_{k}\left(s\right)+\sum_{j,i}\frac{V_{j,i}e^{-ikx_{\left(i\right)}}}{\sqrt{N}}\tilde{C}_{j}^{i}\left(s\right)\text{.}$
(SM4)

The expression of $\tilde{C}_{j}^{i}\left(s\right)$ is written as

$i\left[-C_{j}^{A}\left(0\right)+s\tilde{C}_{j}^{A}\left(s\right)\right]=\Omega_{A}\tilde{C}_{j}^{A}\left(s\right)+\frac{V_{A}}{N}\sum_{k}\frac{1}{is-\omega_{k}}\left[\sum_{j^{\prime}}V_{A}\tilde{C}_{j^{\prime}}^{A}\left(s\right)+\sum_{j^{\prime}}V_{B}e^{-ik\Delta
x}\tilde{C}_{j^{\prime}}^{B}\left(s\right)\right]\text{,}$ (SM5)
$is\tilde{C}_{j}^{B}\left(s\right)=\Omega_{B}\tilde{C}_{j}^{B}\left(s\right)+\frac{V_{B}}{N}\sum_{k}\frac{1}{is-\omega_{k}}\left[\sum_{j^{\prime}}V_{A}e^{ik\Delta
x}\tilde{C}_{j^{\prime}}^{A}\left(s\right)+\sum_{j^{\prime}}V_{B}\tilde{C}_{j^{\prime}}^{B}\left(s\right)\right]\text{.}$
(SM6)

Substituting $\tilde{C}_{j}^{B}\left(s\right)$ in Eq. (SM5) into Eq. (SM6) and
denoting the initial excited QEs by $j_{n}$, so
$\tilde{C}_{j_{n}}^{A}\left(s\right)$ can be found as the following:

$\displaystyle\tilde{C}_{j_{1}}^{A}\left(s\right)$
$\displaystyle=i\frac{K_{A}\left(s,M_{A}-1\right)K_{B}\left(s,M_{B}\right)-\left(M_{A}-1\right)M_{B}\left[V_{A}V_{B}F\left(s,\Delta
x\right)\right]^{2}}{\left(is-\Omega_{A}\right)\left\\{K_{A}\left(s,M_{A}\right)K_{B}\left(s,M_{B}\right)-M_{A}M_{B}\left[V_{A}V_{B}F\left(s,\Delta
x\right)\right]^{2}\right\\}}C_{j_{1}}^{A}\left(0\right)$
$\displaystyle+i\frac{K_{B}\left(s,M_{B}\right)V_{A}^{2}F\left(s,0\right)+M_{B}\left[V_{A}V_{B}F\left(s,\Delta
x\right)\right]^{2}}{\left(is-\Omega_{A}\right)\left\\{K_{A}\left(s,M_{A}\right)K_{B}\left(s,M_{B}\right)-M_{A}M_{B}\left[V_{A}V_{B}F\left(s,\Delta
x\right)\right]^{2}\right\\}}C_{j_{2}}^{A}\left(0\right)\text{,}$ (SM7)

where $K_{i}(s,M_{i})=is-\Omega_{i}-M_{i}V_{i}^{2}F(s,0)$. $F(s,x)$ are given
by

$F\left(s,x\right)=\frac{\left(is-
is\sqrt{\left(s^{2}+4J^{2}\right)/s^{2}}\right)^{\left|x\right|}}{is\sqrt{\left(s^{2}+4J^{2}\right)/s^{2}}\left(2J\right)^{\left|x\right|}}\text{.}$
(SM8)

In the calculations above, we have used the formula [3]

$\frac{1}{2\pi}\int
dk\frac{e^{ikx}}{z+2J\cos\left(k\right)}=\frac{\left(-\frac{z}{2J}+\frac{z}{2J}\sqrt{1-\left(\frac{2J}{z}\right)^{2}}\right)^{\left|x\right|}}{z\sqrt{1-\left(\frac{2J}{z}\right)^{2}}}\text{.}$

Without loss of generality, we set $\omega_{c}$ to be zero as reference
energy. By exchanging the positions of $C_{j_{1}}^{A}(0)$ and
$C_{j_{2}}^{A}(0)$ in Eq. (SM7), one can analogously obtain the time evolution
expression of $\tilde{C}_{j_{2}}^{A}(s)$. The amplitude $C_{j_{n}}^{A}(t)$ is
given by the inverse Laplace transform $C_{j}^{i}(t)=(1/2\pi
i)\int_{\sigma-i\infty}^{\sigma+i\infty}\tilde{C}_{j}^{i}(s)e^{st}ds$. To
evaluate this integral, we consider the analytic behavior of
$\tilde{C}_{j_{1}}^{A}(s)$ in the whole complex plane except a branch cut
along the imaginary axis from $-i2J$ to $i2J$. With the residue theorem, the
time dependence of $C_{j_{1}}^{A}\left(t\right)$ can then be obtained:

$\displaystyle C_{j_{1}}^{A}\left(t\right)$
$\displaystyle=\frac{L_{1}(s)C_{j_{1}}^{A}\left(0\right)+L_{2}(s)C_{j_{2}}^{A}\left(0\right)}{G(s)}e^{st}|_{s=-i\Omega_{A}}+\sum_{m}\frac{iL_{2}(s)\left[C_{j_{1}}^{A}\left(0\right)+C_{j_{2}}^{A}\left(0\right)\right]}{\left(is-\Omega_{A}\right)[G(s)]^{\prime}}e^{st}|_{s=x_{m}^{\left(2\right)}}$
$\displaystyle-\sum_{\alpha=\pm}\int_{-1}^{1}\frac{\left[Q_{1}^{\alpha}C_{j_{1}}^{A}\left(0\right)+Q_{2}^{\alpha}C_{j_{2}}^{A}\left(0\right)\right]e^{i2Jyt}}{2\pi
i\alpha\left(y+\Omega_{A}\right)[Q_{1}^{\alpha}-Q_{2}^{\alpha}]}dy\text{,}$
(SM9)

where $L_{1}(s)$ and $L_{2}(s)$ are defined as

$L_{1}(s)=K_{A}(s,M_{A}-1)K_{B}(s,M_{B})-(M_{A}-1)M_{B}[V_{A}V_{B}F(s,\Delta
x)]^{2}$

and

$L_{2}(s)=K_{B}(s,M_{B})V_{A}^{2}F(s,0)+M_{B}[V_{A}V_{B}F(s,\Delta
x)]^{2}\text{.}$

The function $G(s)=L_{1}(s)-L_{2}(s)$ and $[G(s)]^{\prime}$ means the
derivative of $G(s)$ with respect to $s$. Here $x_{m}$ is pure imaginary and
the equation $G(-iE)=0$ is nothing but the system’s eigenenergy equation of
the localized photon-QE dressed states. The functions $Q_{1}^{\pm}$ and
$Q_{2}^{\pm}$ are given by

$Q_{1}^{\pm}=U_{A}^{\pm}(y,M_{A}-1)U_{B}^{\pm}(y,M_{B})-(M_{A}-1)M_{B}[V_{A}V_{B}f_{\pm}(y,\Delta
x)]^{2}$

and

$Q_{2}^{\pm}=U_{B}^{\pm}(y,M_{B})V_{A}^{2}f_{\pm}(y,0)+M_{B}[V_{A}V_{B}f_{\pm}(y,\Delta
x)]^{2}\text{,}$

where $U_{i}^{\pm}(y,M_{i})=-2Jy-\Omega_{i}-M_{i}V_{i}^{2}f_{\pm}(y,0)$ and
$f_{\pm}(y,x)$ are given by

$f_{\pm}(y,x)=\pm i\frac{\left(-y\pm
i\sqrt{1-y^{2}}\right)^{\left|x\right|}}{2J\sqrt{1-y^{2}}}\text{.}$ (SM10)

By exchanging the positions of $C_{j_{1}}^{A}(0)$ and $C_{j_{2}}^{A}(0)$ in
Eq. (SM9), one can get the time evolution of $C_{j_{2}}^{A}(t)$.

For $M_{A}=1$ and $V_{A}\ll 2J$ with $\Delta x=2n+1$ ($n$ being in integer),
the first two terms in Eq. (SM9) are found to be numerically much smaller than
the third term when both type-A and type-B QEs are near resonance with a
single cavity. Dropping then first two terms, $C_{j_{1}}^{A}\left(t\right)$
can thus be approximately obtained as the following:

$C_{j_{1}}^{A}\left(t\right)\approx\frac{J}{\pi
i}\int_{-1}^{1}\left\\{\frac{2Jy-i\frac{M_{B}V_{B}^{2}}{2J\sqrt{1-y^{2}}}}{\left(2Jy-i\frac{V_{A}^{2}}{2J\sqrt{1-y^{2}}}\right)\left(2Jy-i\frac{M_{B}V_{B}^{2}}{2J\sqrt{1-y^{2}}}\right)+M_{B}V_{A}^{2}V_{B}^{2}\frac{\left(-y-i\sqrt{1-y^{2}}\right)^{2\Delta
x}}{\left(2J\sqrt{1-y^{2}}\right)^{2}}}-h.c\right\\}e^{i2Jyt}dy\text{.}$
(SM11)

After the radiation field comes back to the decaying QE and under the
condition $\sqrt{M_{B}}V_{B}<$ or $\sim 2J$, the oscillating term $e^{i2Jyt}$
in the above expression indicates that the above integrand is insensitive to
the lower or upper limit of this integration with small $\Delta x$ (see the
following comparison with different values of $\Delta x$ between the
approximate result and exact result in Fig. SM1). As such, the integration
range can be safely extended to infinity. Then $C_{j_{1}}^{A}\left(t\right)$
can be written as

$C_{j_{1}}^{A}\left(t\right)\approx\frac{J}{\pi
i}\int_{-\infty}^{\infty}\left\\{\frac{2Jy-i\frac{M_{B}V_{B}^{2}}{2J}}{\left(2Jy-i\frac{V_{A}^{2}}{2J}\right)\left(2Jy-i\frac{M_{B}V_{B}^{2}}{2J}\right)+M_{B}V_{A}^{2}V_{B}^{2}i^{2\Delta
x}\left(1-2\Delta xyi\right)}-h.c\right\\}e^{i2Jyt}dy\text{.}$ (SM12)

The poles of the first term in the integrand are

$\displaystyle y_{0,\pm}$
$\displaystyle=i\frac{1}{2}\left(\frac{M_{B}V_{B}^{2}}{4J^{2}}+\frac{V_{A}^{2}}{4J^{2}}-2\Delta
xM_{B}\frac{V_{A}^{2}}{4J^{2}}\frac{V_{B}^{2}}{4J^{2}}\right)$
$\displaystyle\pm\frac{1}{2}\sqrt{-\left(\frac{M_{B}V_{B}^{2}}{4J^{2}}+\frac{V_{A}^{2}}{4J^{2}}-2\Delta
xM_{B}\frac{V_{A}^{2}}{4J^{2}}\frac{V_{B}^{2}}{4J^{2}}\right)^{2}+8M_{B}\frac{V_{A}^{2}}{4J^{2}}\frac{V_{B}^{2}}{4J^{2}}}\text{.}$
(SM13)

For $\sqrt{M_{B}}V_{B}$ $\sim 2J$, $y_{0,\pm}$ are pure imaginary. Using the
residue theorem, one reduces $C_{j_{1}}^{A}\left(t\right)$ to the following:

$C_{j_{1}}^{A}\left(t\right)\approx\frac{M_{B}V_{B}^{2}/(4J^{2})-\alpha_{1}/2}{\sqrt{\beta_{1}}}e^{-J\alpha_{1}t}\text{,}$
(SM14)

where $\alpha_{1}=V_{A}^{2}/[J^{2}-\Delta xV_{A}^{2}/2]$ and
$\beta_{1}=\\{[2J^{2}M_{B}V_{B}^{2}-\Delta
xM_{B}V_{A}^{2}V_{B}^{2}]/(8J^{4})\\}^{2}-M_{B}V_{A}^{2}V_{B}^{2}/(2J^{4})$.

In Fig. SM1, we compare the approximate results (dotted lines) in Eq. (SM14)
with the exact results (solid lines) in Eq. (SM9) for different values of
$\Delta x$. As $\Delta x$ decreases, the approximate results agree better with
the exact results.

Figure SM1: Excited state population
$P_{e}(t)=|C_{j_{1}}^{A}\left(t\right)|^{2}$ with $\Delta x=1$ in (a), $\Delta
x=5$ in (b), $\Delta x=9$ in (c), $\Delta x=13$ in (d), $\Delta x=17$ in (e),
$\Delta x=21$ in (f). The parameters are $V_{A}/(2J)=0.07$, $V_{B}/(2J)=1.8$,
$\Delta_{A}/(2J)=0$, $\Delta_{B}/(2J)=0$, $M_{A}=1$ and $M_{B}=2$. Time is in
units of $1/(2J)$.

Similar to our treatments in getting the result in Eq. (SM14), in the case of
two QEs with the initial state $|\psi^{+}\rangle=$
$(1/\sqrt{2})(\sigma_{j_{1},A}^{+}+\sigma_{j_{2},A}^{+})|g$,vac$\rangle$, the
dynamics of hyperradiance for $V_{A}\ll 2J$ and $\sqrt{M_{B}}V_{B}\sim$ $2J$
can also be found with small $\Delta x$, namely

$C_{\text{sup}}\left(t\right)\approx\frac{M_{B}V_{B}^{2}\left(J^{2}-\Delta
xV_{A}^{2}\right)-4J^{2}V_{A}^{2}}{\left(J^{2}-\Delta
xV_{A}^{2}\right)\sqrt{2\left(\chi^{2}-16M_{B}V_{A}^{2}V_{B}^{2}\right)}}e^{-\frac{\Gamma_{h}}{2}t}\text{,}$
(SM15)

where the emission rate $\Gamma_{h}=4JV_{A}^{2}/(J^{2}-\Delta xV_{A}^{2})$ and
$\chi=M_{B}V_{B}^{2}-\Delta xM_{B}V_{A}^{2}V_{B}^{2}/J^{2}$.

##  SM2 Theory of photon scattering in a waveguide

To describe the radiation field propagating towards and interacting with the
quantum scatterers, we start with the Green’s function defined in terms of the
system Hamiltonian $H_{\text{sca}}$ (assuming there are only scatterers but no
emitters)

$G\left(z\right)=\frac{1}{z-H_{\text{sca}}}$ (SM16)

where $z$ is a complex energy variable and the expression of $H_{\text{sca}}$
is

$H_{\text{sca}}=H_{\text{ph}}^{0}+V_{\text{ph}}$ (SM17)

and

$H_{\text{ph}}^{0}=\sum_{k}\omega_{k}a_{k}^{{\dagger}}a_{k}+\sum_{j}\Omega_{B}\left|e_{j}^{B}\right\rangle\left\langle
e_{j}^{B}\right|\text{,}$ (SM18)
$V_{\text{ph}}=\sum_{j}\sum_{k}\frac{V_{B}}{\sqrt{N}}\left(\sigma_{j}^{B+}a_{k}e^{ikx_{2}}+\sigma_{j}^{B-}a_{k}^{{\dagger}}e^{-ikx_{2}}\right)\text{.}$
(SM19)

The Green’s function $G(z)$ is an analytic function on the complex $z$ plane
except at those points and branch cut of the real axis (which correspond to
the eigenvalues of bound states and scattering states). $G(z)$ satisfies the
resolvent equations [3]

$G\left(z\right)=G_{0}\left(z\right)+G_{0}\left(z\right)V_{\text{ph}}G\left(z\right)\text{.}$
(SM20)

Here $G_{0}(z)$ is the free Green’s function and is written as

$G_{0}\left(z\right)=\frac{1}{z-H_{\text{ph}}^{0}}\text{.}$ (SM21)

The S-matrix element in this scattering process is given by

$S_{p,k}=\delta_{p,k}-i2\pi\delta\left(\omega_{p}-\omega_{k}\right)T_{p,k}\left(\omega_{k}+i\epsilon\right)\text{,}$
(SM22)

where $T(z)=V_{\text{ph}}+V_{\text{ph}}G(z)V_{\text{ph}}$ is the T-matrix and
$T_{p,k}(\omega_{k}+i\epsilon)=\left\langle
g\text{,vac}\right|a_{p}T(\omega_{k}+i\epsilon)a_{k}^{{\dagger}}\left|g\text{,vac}\right\rangle$.
With the method of self-consistent equations [2],
$T_{p,k}(\omega_{k}+i\epsilon)$ is found to be

$T_{p,k}\left(\omega_{k}+i\epsilon\right)=\frac{\frac{M_{B}V_{B}^{2}}{N}}{\omega_{k}-\Omega_{B}-\frac{M_{B}V_{B}^{2}}{N}\sum_{k^{\prime}}\frac{1}{\omega_{k}-\omega_{k^{\prime}}+i\epsilon}}\text{.}$
(SM23)

So the S-matrix element is

$S_{p,k}=\left(1+r_{k}\right)\delta_{p,k}+r_{k}\delta_{-p,k}$ (SM24)

with the reflection amplitude

$r_{k}=\frac{-iM_{B}V_{B}^{2}}{\left|\partial\omega_{k}/\partial
k\right|(\omega_{k}-\Omega_{B})+iM_{B}V_{B}^{2}}\text{.}$ (SM25)

For the case of $M_{B}=1$, Eq. (SM25) comes back to the result obtained in
[4]. The transmission amplitude can be obtained from relations $t_{k}=1+r_{k}$
and $\left|r_{k}\right|^{2}+\left|t_{k}\right|^{2}=1$.

## References

  * [1] S. John and T. Quang, Spontaneous emission near the edge of a photonic band gap, Phys. Rev. A 50, 1764 (1994).
  * [2] L. Qiao, Y. J. Song, and C. P. Sun, Phys. Rev. A 100, 013825 (2019).
  * [3] E. N. Economou, Green Functions in Quantum Physics (Springer-Verlag, Berlin, 1979).
  * [4] L. Zhou, Z. R. Gong, Y. X. Liu, C. P. Sun, and F. Nori, Phys. Rev. Lett. 101, 100501 (2008).


# Why Exposure Bias Matters: An Imitation Learning Perspective of Error
Accumulation in Language Generation

Kushal Arora1 Layla El Asri2 Hareesh Bahuleyan3 Jackie Chi Kit Cheung1

1Mila / McGill University 2Borealis AI 3 Zalando SE
{kushal.arora@mail, <EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>A part of this work
was done when the author was an intern at Borealis AI. During a part of this
work, the author was an employee at Borealis AI. During a part of this work,
the author was an Academic Advisor at Borealis AI.

###### Abstract

Current language generation models suffer from issues such as repetition,
incoherence, and hallucinations. An often-repeated hypothesis is that this
brittleness of generation models is caused by the training and the generation
procedure mismatch, also referred to as exposure bias. In this paper, we
verify this hypothesis by analyzing exposure bias from an imitation learning
perspective. We show that exposure bias leads to an accumulation of errors,
analyze why perplexity fails to capture this accumulation, and empirically
show that this accumulation results in poor generation quality. 111Source code
to reproduce these experiments is available at
https://github.com/kushalarora/quantifying_exposure_bias.

##  1 Introduction

Large-scale neural language models have made great strides in language
generation tasks such as machine translation Bahdanau et al. (2014); Vaswani
et al. (2017); Raffel et al. , text summarization See et al. (2017); Lewis et
al. (2019); Zhang et al. (2019a), conversational dialog generation Serban et
al. (2015); Lowe et al. (2017); Roller et al. (2020); Zhang et al. (2020),
etc.

However, despite the successes achieved by these models on several conditional
generation tasks, they continue to suffer from degenerate behaviors such as
repetition, a lack of diversity, dullness, and incoherence, especially in
open-ended generation settings such as text completion and dialog modeling
Holtzman et al. (2019). This degenerate behavior is often attributed to a
mismatch between the maximum likelihood training and generation procedure
mismatch Welleck et al. (2019); Choi et al. (2020); Li et al. (2016).

Maximum likelihood training, also referred to as teacher forcing Williams and
Zipser (1989), factorizes the language model as a linear chain, and maximizes
the log-likelihood of this factorized language model on a training corpus.
During this maximum likelihood training, the model learns a distribution of
the next tokens conditioned on the contexts from the ground-truth training
data.

A concern with MLE-based training is that ground-truth contexts from the
training corpus are not available during generation. Rather, the conditioning
contexts during this phase comprise tokens previously generated by the model
itself. The distribution of these contexts seen during the generation phase
might be very different from the ones encountered during the training phase.
This mismatch is referred to as exposure bias Ranzato et al. (2016); Bengio et
al. (2015).

A side effect of exposure bias is that an error at any step during generation
might have a cascading effect as the next context will incorporate this
erroneous prediction, deviating away from the ground truth context
distribution, leading to more errors. Several authors Welleck et al. (2019);
Choi et al. (2020); Li et al. (2016) have speculated that these errors might
result in sequences that degenerate over the sequence length resulting in
incoherent text, lack of vocabulary diversity, and hallucinations, and word-
and phrase-level repetition.

There is an active debate in the language generation community on the impact
of exposure bias in language generation. Authors have both validated Xu et al.
(2019); Zhang et al. (2019b) and questioned He et al. (2019) the impact of
exposure bias on language generation. Previous works have also linked exposure
bias to out-of-distribution generalization Schmidt (2019), and out-of-domain
generalization and hallucinations Wang and Sennrich (2020) but these claims
remain weak in absence of a clear and principled formalization of the exposure
bias issue. Finally, several approaches have been proposed to mitigate
exposure bias (Ranzato et al., 2016; Shen et al., 2016; Bahdanau et al., 2017;
Chen et al., 2020; Leblond et al., 2018; Welleck et al., 2019), though these
approaches improve the performance on the downstream tasks, the authors
neither formalized exposure bias nor provided any empirical evidence that the
downstream improvements are directly linked to the mitigation of exposure bias
issue.

In this paper, we attempt to clarify this confusion by formalizing exposure
bias in terms of accumulation of errors and by analyzing its impact on
generation quality. We do this by providing a theoretically grounded
understanding of the exposure bias issue by analyzing it from an imitation
learning perspective. We use this perspective to show that behavior cloning—an
imitation learning algorithm—is equivalent to teacher forcing under the choice
of a particular loss function. We then exploit this equivalence by borrowing
the bound on error accumulation caused by behavior cloning and use it to
formalize exposure bias and analyze error accumulation in language generation.

Finally, we use this quantifiable definition of exposure bias to demonstrate
that models trained using teacher forcing do suffer from an accumulation of
errors. We also show, both analytically and empirically, why perplexity fails
to capture this error accumulation, and how a lower exposure bias correlates
with better generation quality.

##  2 Language Generation Formulation

Given a finite-sized vocabulary set $\mathcal{V}$, language generation is
posed as a problem of generating a variable-length sequence
$w_{0}^{n}\in\mathcal{V}^{n}$ from a language model $p_{\theta}$, either
unconditionally or conditioned on a source $\bf{x}$, using a decoding
algorithm $\mathcal{F}$:

$w_{0}^{n}=\mathcal{F}(p_{\theta};\bf{x})$ (1)

Language modeling is the problem of learning this parameterized model
$p_{\theta}$ that approximates an oracle model $o$.

Maximum likelihood-based training factorizes the probability distribution
model, $p_{\theta}(w_{0}^{n})$, into a linear chain, i.e.,

$p_{\theta}(w_{0}^{n};\textbf{x})=\prod_{i=1}^{n}p_{\theta}(w_{i}|w_{0}^{i-1};\textbf{x})p(w_{0}),$
(2)

where $w_{i}$ is the token to be generated at step $i$ and $w_{0}^{i-1}$ is
the context at time $i$; i.e., all the tokens seen from step 0 to step
$i-1$.222As $w_{0}$ is usually a fixed SOS token, $p(w_{0})=1$. We will drop
$p(w_{0})$ from the subsequent equations for brevity.

During maximum likelihood training, the language model is trained by
minimizing the negative log-likelihood on the corpus $\mathcal{D}$, i.e.,

$\theta^{*}=\underset{\theta}{\text{argmin}}\frac{-1}{|\mathcal{D}|}\sum_{w_{0}^{n}\in\mathcal{D}}\sum_{i=0}^{n}\log
p_{\theta}(w_{i}|w_{0}^{i-1}),$ (3)

where $|\mathcal{D}|$ is the number of tokens in the corpus.

Given a trained language model $p_{\theta}$, the simplest strategy for
generating a target sequence is to greedily sample the model;i.e., at each
step $i$, pick the most probable token $w_{i}=\arg\max
p_{\theta}(\cdot|w_{0}^{i-1};x)$—as its prediction. For the next step $i+1$,
we use $w_{i}$ to generate the context $w_{0}^{i}=w_{0}^{i-1}w_{i}$, and use
it to predict the next token. This continues either until the maximum sequence
length ($T$) is reached, or a special end-of-sequence token (EOS) is
generated.

##  3 An Imitation Learning Perspective of Language Generation

In this section, we will present an imitation learning perspective of language
generation. This framing will allow us to borrow theoretical machinery from
the imitation learning literature to formalize the exposure bias issue and
analyze it in terms of the accumulation of errors due to a procedural mismatch
between MLE-based training and generation.

We start by posing language generation as a sequential decision-making problem
and language modeling as an instance of imitation learning. We exploit these
parallels to show behavior cloning, an imitation learning algorithm, is
equivalent to teacher forcing under a choice of a particular loss function. We
then exploit this equivalence to quantify the error accumulation due to
exposure bias.

#### Language Generation is a Sequential Decision-Making Problem:

A sequential decision-making problem can be formalized as learning a policy
$\pi(a_{t}|s_{t})$ over a space of actions $a_{t}\in\mathcal{A}$ and states
$s_{t}\in S$ where the next state $s_{t+1}$ is conditioned on the current
state-action pair and is determined by the transition distribution
$P(s_{t+1}|s_{t},a_{t})$. We can use this framework to pose language
generation as an instance of a sequential decision-making problem with
language model $p_{\theta}$ as the policy, contexts
$w_{0}^{t-1}\in\mathcal{V}*$ as states, the next token prediction
$w_{t}\in\mathcal{V}$ as actions, and concatenation as the transition
function.

This perspective allows us to appreciate the fact that, during generation,
predictions at previous steps affect the next predictions, and errors over
time can cascade resulting in incoherent sequences.

#### Language Modeling is Imitation Learning:

Imitation learning is a class of methods to solve a sequential decision-making
problem while having access to the oracle policy $o$ or data generated by the
oracle; i.e., $\mathcal{D}=\\{(s_{t},a_{t})|s_{t}\sim d^{t}_{o},a_{t}\sim
o(\cdot|s_{t})\\}$. Here, $d_{o}^{t}$ is the oracle-induced state-visitation
distribution at time $t$.

In imitation learning, an agent learns a model policy $\pi$ that reproduces
the expert policy $o$ but on the state-visitation distribution $d_{\pi}^{t}$
that has been induced by the model policy $\pi$, i.e.:

$\pi^{*}=\underset{\pi}{\arg\min}\ \sum_{t=1}^{T}\mathbb{E}_{s\sim
d_{\pi}^{t}}\Big{[}l(\pi,s;o)\Big{]}.$

where $l(\pi,s;o)$ is the expected per-step cost of model $\pi$ mimicing
oracle $o$ in state $s$, $d_{\pi}^{t}$ is the state-visitation distribution at
step $t$ induced by following policy $\pi$ from step $1$ to $t-1$.

The sequential decision-making perspective of language generation allows us to
pose language modeling as an instance of imitation learning—learning a model
for a sequential decision-making problem with the help of an expert oracle (in
RL-based methods) or using the data generated by the oracle (for MLE-based
methods).

#### Teacher Forcing is Behavior Cloning:

The assumption of access to an oracle is unrealistic in many scenarios.
Behavior cloning is an approach to solving an imitation learning problem using
only the training data generated by an oracle. In this setup, the state-action
pairs in the training data are assumed to be identically and independently
distributed. This is equivalent to reducing a sequential decision-making
problem to a supervised multi-class classification learning problem.

Concretely, this learning problem can be seen as minimizing the expected per-
step loss under the state distribution induced by the oracle:

$\displaystyle L^{BC}(\pi)=$ $\displaystyle\sum_{t=1}^{T}\mathbb{E}_{s_{t}\sim
d_{o}^{t}}\left[l(s_{t},\pi;o)\right]$ (4) $\displaystyle\approx$
$\displaystyle\frac{-1}{|\mathcal{D}|}\sum_{(s_{t},a_{t})\in\mathcal{D}}l(s_{t},\pi;o),$
(5)

Here, $L^{BC}(\pi)$ is the behavior cloning loss, and $l(s,\pi;o)$ is the per-
step loss.

Similarly, in practical scenarios, language models are also trained on a
finite training corpus, $\mathcal{D}$, that is assumed to be generated by the
oracle; i.e., $\mathcal{D}=\\{(w_{0}^{t-1},w_{t})|w_{0}^{t-1}\sim
d_{o}^{t-1},w_{t}\sim o(\cdot|w_{0}^{t-1})\\}$.

The maximum likelihood training loss from Equation 3, can be reformulated as
learning the distribution over the next tokens, conditioned on the training
contexts generated by the oracle, $w_{0}^{t-1}\sim d_{o}^{t-1}$:

$\displaystyle
L^{\text{TF}}(p_{\theta})=\frac{-1}{|\mathcal{D}|}\sum_{(w_{0}^{i-1},w_{i})\in\mathcal{D}}\log
p_{\theta}(w_{i}|w_{0}^{i-1})$ , (6)
$\displaystyle\approx\sum_{t=1}^{T}\underset{\begin{subarray}{c}w_{0}^{t-1}\sim
d_{o}^{t}\\\ w_{t}\sim
o(\cdot|w_{0}^{t-1})\end{subarray}}{\mathbb{E}}\left[-\log
p_{\theta}(w_{t}|w_{0}^{t-1})\right]$ , (7)

The behavior cloning loss (Equation 4) is equivalent to the language modeling
loss (Equation 7) with $l(p_{\theta},w_{0}^{t-1};o)$ defined as,

$l(p_{\theta},w_{0}^{t-1};o)=\underset{w_{t}\sim
o(\cdot|w_{0}^{t-1})}{\mathbb{E}}\left[-\log
p_{\theta}(w_{t}|w_{0}^{t-1})\right]$

For our analysis though, we define per-step loss for language modeling,
$l(p_{\theta},w_{0}^{t-1};o)$ as:

$\displaystyle l(p_{\theta},w_{0}^{t-1};o)=\underset{w_{t}\sim
o(\cdot|w_{0}^{t-1})}{\mathbb{E}}\log\frac{o(w_{t}|w_{0}^{t-1})}{p_{\theta}(w_{t}|w_{0}^{t-1})},$
(8)

This definition ensures that the per-step loss for the oracle is zero; i.e.,
$l(o,w_{0}^{t-1};o)=0$.

The per-step loss function defined by equation 8 ensures that the behavior
cloning loss, $L^{BC}(p)$, under our definition is equivalent to teacher
forcing loss, $L^{TF}(p)$, up to a constant term. This equivalence of
$L^{BC}(p)$ and $L^{TF}(p)$ ensures that the model learned by minimizing
either of the two losses will be identical.

#### Language Generation is Policy Rollouts:

During policy rollouts, an agent in state $s_{t}$ executes the action $a_{t}$,
sampled from policy $\pi$, and ends up in state $s_{t+1}$. The agent’s next
state is dependent upon its own actions. This state evolution can be
formulated as sampling from state-visitation distribution induced by the
policy $\pi$, i.e., $s_{t+1}\sim d_{\pi}^{t+1}$.

The performance of policy $\pi$ during rollouts can be measured using the loss
(cost) of executing the policy $\pi$:

$\displaystyle L^{I}(\pi)=$ $\displaystyle\sum_{t=1}^{T}\mathbb{E}_{s_{t}\sim
d_{\pi}^{t}}\left[l(s_{t},\pi;o)\right]$ (9)

We can also formulate language generation in terms of policy rollouts from
imitation learning. Mathematically, we can express generation as sampling
contexts from the model’s context distribution, i.e., $w_{0}^{j-1}\sim
d_{p_{\theta},\mathcal{F}}^{j}$, and generating the next token $w_{j}$
conditioned on $w_{0}^{j-1}$, using the decoding algorithm $\mathcal{F}$:

$\\{w_{j}=\mathcal{F}(p_{\theta},w_{0}^{j-1})|w_{0}^{j-1}\sim
d_{p_{\theta},\mathcal{F}}^{j}\\}$ (10)

We can now define the inference-time loss, $L^{I}(p_{\theta})$, as the
accumulated $T$-step loss of model $p_{\theta}$ imitating oracle $o$ on the
context distribution induced by the model:

$\displaystyle L^{I}(p_{\theta})$
$\displaystyle=\sum_{t=1}^{T}\underset{\begin{subarray}{c}w_{0}^{t-1}\sim
d_{p_{\theta},\mathcal{F}}^{t}\\\ w_{t}\sim
o(\cdot|w_{0}^{t-1})\end{subarray}}{\mathbb{E}}\log\frac{o(w_{t}|w_{0}^{t-1})}{p_{\theta}(w_{t}|w_{0}^{t-1})},$
(11)

where $d_{p_{\theta},\mathcal{F}}^{t}(w_{0}^{t})\coloneqq
p_{\theta}(w_{0}^{t-1})$, is the context distribution at step $t$, induced due
to use of model $p_{\theta}$ and the decoding algorithm $\mathcal{F}$, from
step $1$ to $t-1$.

##  4 Exposure Bias and Error Accumulation

Ranzato et al. (2016) defined exposure bias as a behavioral mismatch between
maximum likelihood-based training and the generation procedure at inference
time. During maximum likelihood-based training, the next token distribution is
conditioned on ground truth data whereas, during generation, it has to rely on
the model’s own previously generated tokens. They also postulated that this
training and generation context distribution mismatch might result in an
accumulation of errors during generation.

Intuitively, when the model produces a token $w_{i}$ that makes the resulting
context $w_{0}^{i}$ unfamiliar, it might not be able to continue the
generation adequately and is likely to produce another token which will
further make the context flawed. This phenomenon reinforces itself as the
context drifts further from what the oracle would produce, leading to an
accumulation of errors.

In the imitation learning literature, the accumulation of errors while rolling
out a policy trained using behavior cloning is analyzed in the terms of
inference-time regret of the behavior cloning policy, $\pi_{BC}$, with respect
to the oracle policy, $o$, Ross and Bagnell (2010); Ross et al. (2011) i.e.,

$\mathcal{R}(\pi_{BC})=L^{I}(\pi_{BC})-L^{I}(o)$ (12)

Let $\epsilon_{t}$ be the expected error of executing policy $\pi$ at step $t$
on the state-visitation distribution induced by the oracle $o$, i.e.,

$\epsilon_{t}=\mathbb{E}_{s\sim d^{t}_{o}}[l(s,{\pi};o)]$ (13)

Let $\epsilon$ be the average expected error of executing policy $\pi$ over
$T$ step, i.e., $\epsilon=1/T\sum_{t=1}^{T}\epsilon_{t}$. Assuming
$l(s,\pi,o)$ is an upper bound on $[0,1]$ loss, we can bound the regret for a
policy ${\pi}_{BC}$ as,

$T\epsilon\leq\mathcal{R}(\pi_{BC})\leq T^{2}\epsilon.$ (14)

The lower bound in Equation 14 assumes no accumulation of error, hence an
expected error of $\epsilon$ at each step, whereas the upper bound assumes the
worst-case scenario, resulting in linear growth in error at each step and
overall quadratic accumulative growth w.r.t. maximum sequence length $T$.

Relying on the imitation learning perspective of language generation presented
in the previous section, we can now borrow this regret-based analysis from
imitation learning literature to similarly bound the regret of a language
generation model as

$T\epsilon\leq\mathcal{R}(p_{\theta},\mathcal{F})\leq T^{2}\epsilon.$ (15)

where $p_{\theta}$ is the model being used for generation, $\mathcal{F}$ is
the decoding method being used for generation,
$\epsilon=1/T\sum_{t=1}^{T}\epsilon_{t}$ and $\epsilon_{t}$ is defined as

$\epsilon_{t}=\underset{\begin{subarray}{c}w_{0}^{t-1}\sim d_{o}^{t}\\\
w_{t}\sim
o(\cdot|w_{0}^{t-1})\end{subarray}}{\mathbb{E}}\log\frac{o(w_{t}|w_{0}^{t-1})}{p_{\theta}(w_{t}|w_{0}^{t-1})}$
(16)

We will now use these bounds on the regret to analyze and quantify the error
accumulation due to exposure bias in language generation.

##  5 Quantifying Error Accumulation due to Exposure Bias

In our analysis, we use two metrics, $\text{AccErr}_{\leq}(l)$ and
$\text{\%ExAccErr}_{\leq}(l)$ to measure the impact of error accumulation due
to exposure bias.

We define accumulated errors up to length $l$, $\text{AccErr}_{\leq}(l)$, as

$\text{AccErr}_{\leq}(l)=\mathcal{R}_{\leq
l}(p_{\theta},\mathcal{F})/\epsilon_{\leq l}$ (17)

Here, $\mathcal{R}_{\leq l}(p_{\theta},\mathcal{F})$ be the regret due to the
use of language model $p_{\theta}$ and decoding method, $\mathcal{F}$, up to
sequence length $l$, and $\epsilon_{\leq l}=1/l\sum_{t=1}^{l}\epsilon_{t}$ is
the expected per-step error up to length $l$.

This metric captures the growth of error w.r.t. sequence length $l$. If
exposure bias does indeed leads to error accumulation,
$\text{AccErr}_{\leq}(l)$ should grow super-linearly w.r.t. $l$.

We define our second metric, $\text{\%ExAccErr}_{\leq}(l)$, as the percentage
of excess errors committed by the model that can be attributed to exposure
bias, i.e.,

$\text{\%ExAccErr}_{\leq}(l)=\frac{\mathcal{R}_{\leq
l}(p_{\theta},\mathcal{F})-l\epsilon_{\leq l}}{l\epsilon_{\leq l}}*100$

Here, $l\epsilon_{\leq l}$ is the lower bound on the regret and is the minimum
number of errors ($\epsilon$ per step) a model would make if there was no
accumulation of errors.

$\text{\%ExAccErr}_{\leq}(l)$ allows us to compare models, training
algorithms, and decoding strategies on the extra error that might be
caused/mitigated by their use. A model, training algorithm, or decoding
strategy that perfectly mitigates the exposure bias will result in zero excess
accumulated errors.

In the rest of the paper, we use these definitions to show: 1) error
accumulation in language generation is real, 2) perplexity fails to capture
this error accumulation, 3) lower exposure bias correlates with a higher
quality generation that is more coherent, uses more diverse vocabulary, and is
less repetitive.

(a) $AccErr_{\leq}(l)$ vs $l$

(b) $\text{\%ExError}_{\leq}(l)$ vs $l$.

Figure 1: Figure 1(a) plots accumulated error till length $l$
($AccErr_{\leq}(l))$) w.r.t. $l$. This graph shows the quadratic growth of
accumulated errors w.r.t to sequence length ($l$) as predicted by the theory.
Figure 1(b) plots % excess errors due to error accumulation
($\text{\%ExError}_{\leq}(l)$) caused by exposure bias. This indicates that
extra errors due to exposure bias grows near-linearly with the sequence
length, and decoding using greedy search results in over 70% more errors.

##  6 Study Setup: Open-ended Generation

#### Text Completion Setup:

Text completion is the standard experimental setup to measure the quality of
generation in open-ended language generation Holtzman et al. (2019); Welleck
et al. (2019). It is also a generalization of numerous practical language
generation applications such as story generation Fan et al. (2018), contextual
text completion Radford et al. (2019), dialog modeling Zhang et al. (2018).

Text completion models take a text passage or prefix $w_{0}^{j}\sim o$ as an
input and generate a coherent continuation of the prefix, $w_{j+1}^{n}$ using
the language model $p_{\theta}$ and the decoding algorithm $\mathcal{F}$,
i.e., $w_{j+1}^{n}=\mathcal{F}(p_{\theta},w_{0}^{j})$. In this paper, we use
this text-completion setup to analyze the error accumulation due to exposure
bias and its correlation with language generation quality.

#### Language Model and Dataset:

We conduct our analysis using the GPT2 language model Radford et al. (2019).
We use the GPT2-117M model as our evaluation language model and use the train
split of Wikitext-103 Merity et al. (2016) for prompts. We rely on GPT-2 model
fine-tuned on Wikitext-103 as our approximate oracle. We tokenize the
Wikitext-103 dataset using GPT-2’s tokenization scheme. We chunk
Wikitext-103’s train split into sequences of length $512$. Of these, we use
the first $50$ tokens as prompts for our generation experiments and generate
the completions to a maximum length of $512$ or up to the end of the sequence
token. We use a total of $20k$ prompts for our evaluation.

##  7 Results

###  7.1 Error Accumulation in Language Generation is Real!

Figure 1(a) plots $\text{AccErr}_{\leq}(l)$ w.r.t. sequence length, $l$. The
support (dotted, orange line) $y=x$, captures the linear growth. It shows
$\text{AccErr}_{\leq}(l)$ grows near-quadratically w.r.t. sequence length,
empirically validating the theory that exposure bias would lead to the
accumulation of errors. Figure 1(b), further strengthens this claim by
demonstrating near-linear growth in excess errors w.r.t. to the sequence
length.

We hypothesize that these excess errors would manifest in the form of language
degeneration, especially in the latter part of the sequence, and would cause
issues such as hallucinations, limited vocabulary, and word- and phrase-level
repetitions.

Figure 2: Analyzing (log) perplexity ($H_{\leq l}$) w.r.t to average per-step
error ($\epsilon_{\leq l}$), and length-normalized exposure bias regret
($\mathcal{R}_{\leq l}(p_{\theta},\mathcal{F})/l$). We observe that perplexity
strongly correlates with average per-step error ($\rho=0.9997$), but it has a
weaker correlation with length-normalized regret ($\rho=0.4003$).

Search |  | Generation Quality
---|---|---
%ExErrAcc ($\downarrow$) | seq-rep-4 ($\downarrow$) | rep ($\downarrow$) | wrep ($\downarrow$) | uniq ($\uparrow$)
Greedy | 60.96% | 0.8990 | 0.4423 | 0.4136 | 7833
Beam (k=5) | 69.72% | 0.8094 | 0.4064 | 0.3787 | 10966
Sampling
w/ Temp (temp=1) | 39.37% | 0.1883 | 0.2547 | 0.2301 | 23729
w/ Temp (temp=1.2) | 24.75% | 0.1556 | 0.2271 | 0.2033 | 25225
w/ top-k (k=100) | 35.37% | 0.1690 | 0.2409 | 0.2166 | 26251
w/ top-p (p=0.94) | 48.71% | 0.2218 | 0.2743 | 0.2490 | 22582
Human | - | 0.0274 | 0.4338 | - | 28739

Table 1: Impact of error accumulation on generation quality. We observe that
stochastic decoding methods not only lead to diverse language generation but
also have lower exposure bias than the deterministic methods.

###  7.2 Perplexity is Not Enough

Perplexity is a standard measure used to evaluate the quality of a language
model. It is often used as a proxy measure for the text generation quality of
the language model. In this section, we argue perplexity paints an incomplete
picture regarding a model’s ability to generate high-quality, coherent text.
It only captures the average per-step error generalization gap (or lack of it)
but fails to account for the error accumulation due to exposure bias. These
accumulated errors, as seen in the previous section, can grow near-
quadratically and can prove to be a major concern for any generation model
that generates sequences longer than a few words.

Perplexity can be seen as scaled exponentiated average per-step error,
$\epsilon$, computed over a held-out test set, $\mathcal{D}_{h}$:

$\displaystyle\epsilon$
$\displaystyle=1/T\sum_{t=1}^{T}\underset{\begin{subarray}{c}w_{0}^{t-1}\sim
d_{o}^{t}\\\ w_{t}\sim
o(\cdot|w_{0}^{t-1})\end{subarray}}{\mathbb{E}}\log\frac{o(w_{t}|w_{0}^{t-1})}{p(w_{t}|w_{0}^{t-1})}.$
(18)
$\displaystyle\approx\frac{-1}{|\mathcal{D}_{h}|}\sum_{(w_{0}^{i-1},w_{i})\in\mathcal{D}_{h}}\log
p_{\theta}(w_{i}|w_{0}^{i-1})+c,$ (19)
$\displaystyle=H(p_{\theta};\mathcal{D}_{h})+c.`^{\prime}$ (20)

where $H(p_{\theta};\mathcal{D}_{h})$ is the entropy rate (log perplexity) of
the model $p_{\theta}$ on the held-out test set $\mathcal{D}_{h}$.

As entropy rate is a linear function of average per-step error, we hypothesize
that it will only be able to measure the per-step generalization gap of the
model and will fail to capture the error accumulation caused by reducing a
sequential decision-making problem to a supervised learning problem.

In Figure 2, we plot the entropy rate, $H(p_{\theta};\mathcal{D}_{h})_{\leq
l}$, w.r.t. average per-step error, $\epsilon_{\leq l}$, and length-normalized
regret up to length $l$, $\mathcal{R}_{\leq l}(p_{\theta},\mathcal{F})/l$. We
observe a strong correlation between the entropy rate and average per-step
error ($\rho=0.9997$) validating our theoretical observation that perplexity
can capture the per-step generalization gap of language model $p_{\theta}$. On
the other hand, the length-normalized regret exhibits a poor correlation with
the entropy rate ($\rho=0.4003$) indicating perplexity’s failure to capture
the error accumulation due to exposure bias.

A case in point of perplexity’s inability to capture error accumulation is the
degenerate behavior of GPT-2 Radford et al. (2019) while generating moderately
long sequences under greedy or beam search. This happens despite GPT2 having a
low zero-shot perplexity on the held-out set of the Wikitext-103 dataset
(perplexity: $37.50$). We hypothesize that one of the reasons for the
degenerate behaviors of large pre-trained language models such as repetition,
low vocabulary usage, and a lack of coherent generation is the result of this
accumulation of errors. An example of this behavior is presented in Table 2
where we observe GPT2 generating repetitive and incoherent text completion for
a WikiText-103 prompt under deterministic decoding schemes such as greedy and
beam decoding.

###  7.3 Error Accumulation Impacts Generation Quality

Context: … He made one attempt to break through the blockade on 23 December,
but it proved impossible to do as the Union controlled every channel from
Savannah, aided …
---
Greedy: … Savannah, aided by the British, to the North. The blockade was
broken on 23 December by the British and the Americans. The blockade was
broken on 23 December by the British and the Americans. The blockade was
broken on 23 December by the British and the Americans. The blockade was
broken on 23 December by the British and the …
BeamSearch (k=5): … Savannah, aided by the North Carolina Railroad. In the
spring of 1788, in the midst of the blockade, he made a move to join the
Confederacy, but it was too late. On 23 December 1788, he marched to the
battle of Custer’s Creek in the South Carolina River. In the following weeks,
he returned to Savannah on 23 December, but it was too late. On 24 December
1788, he made another attempt to break through the blockade on 23 December,
but it was too late. …
Sampling w/ Temp. (temp=1.2): … Savannah, aided by US Navy ships, and took him
by surprise. One of the most famous incidents involved General Grant’s arrest
at the Battle of Camp Amish on 16 August. He was being held by the Union Army
in Atlanta and accused of taking down General Joseph E. Douglas, a Marine
commander who was in command of Fort Hood, an Army outpost near Fort Laramie,
Mississippi. …
Top-k Sampling (k=100): … Savannah, aided by a little naval warplane
bombardment which had crippled many parts of the ship. As an early American
merchant vessel he was carried forward as a heavy merchant, while at some time
it was necessary to take up residence in Savannah to be used as a regular
transport vessel. In the meantime, on the 20th of January 1816, Jefferson
died. He was buried in an unmarked grave in the chapel inside Savannah in the
family home at Glenbrook, where the rest of the family resides. …
Top-p Sampling (p=0.94): … Savannah, aided in the capture of his land and his
supplies. He made two attempts to stop a blockade on 28 December, but failed
to make it. There were, however, six people in Savannah who were convinced
that Captain Walker was the cause of the blockade. He then made a trip to
North Carolina where he gave up hope. …
Gold: … He made one attempt to break through the blockade on 23 December, but
it proved impossible to do as the Union controlled every channel from
Savannah, aided by their occupation of Tybee Island at the mouth of the
Savannah River. Bulloch reported to Mallory in late January 1862 that breaking
out was hopeless so Mallory ordered him to turn the ship over to another
officer and to return to Europe some other way. …
Table 2: Examples of completions using various decoding methods. We observe
that the deterministic decoding schemes produce less diverse, incoherent, and
more repetitive (highlighted in red) text.

Finally, we examine the hypothesis that poor text generation capabilities of
pre-trained large language models under greedy decoding might be due to the
error accumulation caused by a procedural mismatch between generation and
maximum likelihood training (Vijayakumar et al., 2016; Welleck et al., 2019;
Holtzman et al., 2019).

The regret-based definition of error accumulation allows us to analyze
exposure bias along two axes of variation: the trained language model,
$p_{\theta}$, and the decoding algorithm, $\mathcal{F}$. In this set of
experiments, we explore the impact of various decoding schemes on error
accumulation due to exposure bias and the quality of the completed text.

For a quantitative analysis of the impact of various decoding algorithms on
the quality of language generation, we measure the completion quality by using
the same metrics as Welleck et al. (2019). These metrics are: 1.) rep/128
measures if the prediction token at step $t$ occurs in previous $128$ steps,
2.) wrep/128 counts the prediction’s repetition at step $t$ only if the
predicted token is not the ground-truth token at that position, 3.) seq-rep-4
measure the repetition at the 4-gram level, and 4.) uniq measure the
vocabulary diversity by accounting for the number of unique tokens generated
by the model.

Table 7.1 shows that various sampling-based decoding algorithms result in
diverse and more coherent language generation and a lower percentage of excess
errors. Sampling with temperature (with temp=$1.2$) leads to the least amount
of repetition (both at the token and the n-gram level), second highest
vocabulary diversity, and the least amount of excess errors due to exposure
bias. This also bears out from our qualitative analysis in Table 2 as sampling
with temperature produces the most coherent text. Greedy and beam search
decoding schemes, in contrast, fare poorly in both reducing exposure bias and
language generation quality metrics, producing repetitive and incoherent text.
These quantitative and qualitative experiments offer us evidence that reducing
exposure bias does lead to more coherent text generation.

We hypothesize that the reasonable amount of randomness introduced by
stochastic sampling helps the model avoid sampling the most likely token at
each time step, thus avoiding possible divergent contexts that might have
resulted in a degenerate completion in the future. We conjecture that this
timely intervention prevents the generation context distribution from
diverging too far away from the training context distribution, helping it
avoid the compounding of errors. This is also borne out by qualitative
analysis as a reasonable amount of stochasticity does result in texts which
look more coherent and oracle-like. A broader analysis of this behavior though
is beyond the scope of this work and is left for future work.

Considering that choice of decoding algorithm does not impact average per-step
error, $\epsilon$, this rules out the role of modeling and model training in
language degeneration. Hence, it is reasonable to assume that both qualitative
and quantitative improvement in language quality observed in this experiment
is strongly linked to the reduction in error accumulation due to exposure
bias.

##  8 Related Work

Non-MLE Training Methods: Several approaches have been proposed to mitigate
the exposure bias issue including RL-based optimization objectives (Ranzato et
al., 2016; Shen et al., 2016; Bahdanau et al., 2017; Chen et al., 2020),
learning to search (Leblond et al., 2018), energy-based models (Deng et al.,
2020), imitation learning (Du and Ji, 2019), generative adversarial networks
(Yu et al., 2017) and knowledge distillation (Liu et al., 2019). Although
these methods motivate their approaches as intending to reduce exposure bias,
they neither formally analyze exposure bias nor provide any empirical evidence
that these methods mitigate the effect of exposure bias. In this paper, we
analyze the exposure bias from a principled imitation learning perspective in
terms of the accumulation of errors. This definition can be adapted to
evaluate various novel training and modeling approaches on their ability to
reduce exposure bias.

Smarter Decoding Methods: Large language models have unusually low test
perplexities but they falter at coherent and diverse language generation
specifically in open-ended language generation tasks especially while using
deterministic decoding schemes. Several authors (Vijayakumar et al., 2016;
Welleck et al., 2019; Holtzman et al., 2019) have hypothesized that training
and inference mismatch due to MLE-based training is responsible for the
degenerate behavior. They have proposed smarter decoding schemes to mitigate
the side effects of exposure bias resulting in better generation quality.
Despite this being an active area of research, this often-repeated hypothesis
for degenerate generation behavior has not received serious treatment until
now. In this paper, we take a step towards explaining this discrepancy and
show that error accumulation due to exposure bias might be the reason for this
degenerate behavior and explain why perplexity has a handicap in capturing
this compounding of errors.

Analyzing Exposure Bias: Schmidt (2019) and Wang and Sennrich (2020) link
exposure bias to a generalization gap due to distribution and domain shift
respectively. Performance degradation under domain and distribution shift is a
major issue with language generation, and direct evidence supporting this
hypothesis will provide insights into building more robust language generation
models. Unfortunately, neither of the papers formally analyzes the exposure
bias issue or empirically links the generalization gap to exposure bias
directly.

Three recent papers, Xu et al. (2019); Zhang et al. (2019b); He et al. (2019),
have tried to empirically evaluate the impact of exposure bias on language
generation. The first two papers validate the existence of exposure bias
whereas He et al. (2019) show language models have self-recovering ability
negating the impact of exposure bias. All three analyses are based on the
empirical definition of exposure bias which, in turn, is based on the informal
formulation by Ranzato et al. (2016).

In this paper, we provide a principled and theoretically grounded approach to
analyze exposure bias in language generation and show that it is indeed a
problem and that it might explain the degeneration issue with large language
models on open-ended tasks under deterministic decoding.

##  9 Discussion

In this paper, we analyze language generation from an imitation learning
perspective. We use this analysis to arrive at a theoretical bound on error
accumulation due to exposure bias. This bound predicts a super-linear growth
in error accumulation during generation due to exposure bias. In our
experiments, we validate this bound and show that accumulation due to exposure
bias indeed results in super-linear growth in errors.

We then show, both analytically and empirically, why perplexity is not enough
to capture this accumulation of errors and hypothesize that this accumulation
of errors is responsible for the degenerate language generation. Finally, we
provide some evidence for this hypothesis by evaluating the impact of various
decoding schemes on error accumulation and generation quality. We show that
techniques that improve the generation quality do result in a lower error
accumulation and this indicates that excess error accumulation due to exposure
bias might be a factor affecting language generation quality.

Our analysis provides a principled and theoretically grounded way to
understand exposure bias. We believe this analysis can pave way for developing
smarter training and decoding algorithms to address this error accumulation
resulting in more robust language generation models.

## Acknowledgments

We would like to thank the reviewers for their valuable feedback. This work is
supported by funding from Samsung Electronics. The last author is supported by
the Canada CIFAR AI Chair program. This research was enabled in part by
support provided by Calcul Québec 333https://www.calculquebec.ca, and Compute
Canada 444https://www.computecanada.ca. We would also like to thank Khimya
Khetarpal, Sachin Grover, Ankit Anand, and Jayakumar Subramanian for feedback
on current and previous drafts of this paper, and colleagues at Borealis AI
for their valuable inputs and discussions during the first author’s internship
at Borealis AI.

## References

  * Bahdanau et al. (2017) Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017.  An actor-critic algorithm for sequence prediction.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural Machine Translation by Jointly Learning to Align and Translate.  _arXiv:1409.0473 [cs, stat]_.  ArXiv: 1409.0473.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 1171–1179. Curran Associates, Inc.
  * Chen et al. (2020) Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020.  Reinforcement learning based graph-to-sequence model for natural question generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Choi et al. (2020) Byung-Ju Choi, Jimin Hong, David Park, and Sang Wan Lee. 2020.  F2̂-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9167–9182, Online. Association for Computational Linguistics.
  * Deng et al. (2020) Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. 2020\.  Residual energy-based models for text generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Du and Ji (2019) Wanyu Du and Yangfeng Ji. 2019.  An empirical comparison on imitation learning and reinforcement learning for paraphrase generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6011–6017. Association for Computational Linguistics.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * He et al. (2019) Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James R. Glass. 2019.  Quantifying exposure bias for neural language generation.  _CoRR_ , abs/1905.10617.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019.  The Curious Case of Neural Text Degeneration.  _arXiv:1904.09751 [cs]_.  ArXiv: 1904.09751.
  * Leblond et al. (2018) Rémi Leblond, Jean-Baptiste Alayrac, Anton Osokin, and Simon Lacoste-Julien. 2018.  SEARNN: training rnns with global-local losses.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  _arXiv:1910.13461 [cs, stat]_.  00007 arXiv: 1910.13461.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016.  A Diversity-Promoting Objective Function for Neural Conversation Models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California. Association for Computational Linguistics.
  * Liu et al. (2019) Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, and Haizhou Li. 2019\.  Teacher-student training for robust tacotron-based TTS.  _CoRR_ , abs/1911.02839.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses.  _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1:1116–1126.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer Sentinel Mixture Models.  _arXiv:1609.07843 [cs]_.  00218 arXiv: 1609.07843.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * (18) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  page 67.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y.-Lan Boureau, and Jason Weston. 2020.  Recipes for building an open-domain chatbot.  _arXiv:2004.13637 [cs]_.
  * Ross and Bagnell (2010) Stéphane Ross and Drew Bagnell. 2010.  Efficient reductions for imitation learning.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010_ , volume 9 of _JMLR Proceedings_ , pages 661–668.
  * Ross et al. (2011) Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.  In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , pages 627–635.
  * Schmidt (2019) Florian Schmidt. 2019.  Generalization in Generation: A closer look at Exposure Bias.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 157–167, Hong Kong. Association for Computational Linguistics.  00002\.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get To The Point: Summarization with Pointer-Generator Networks.  _arXiv:1704.04368 [cs]_.  ArXiv: 1704.04368.
  * Serban et al. (2015) Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models.  _arXiv:1507.04808 [cs]_.  ArXiv: 1507.04808.
  * Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Minimum Risk Training for Neural Machine Translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1683–1692, Berlin, Germany. Association for Computational Linguistics.  00202\.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _arXiv:1706.03762 [cs]_.  ArXiv: 1706.03762.
  * Vijayakumar et al. (2016) Ashwin K. Vijayakumar, Michael Cogswell, Ramprasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016.  Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models.
  * Wang and Sennrich (2020) Chaojun Wang and Rico Sennrich. 2020.  On exposure bias, hallucination and domain shift in neural machine translation.  _CoRR_ , abs/2005.03642.
  * Welleck et al. (2019) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019.  Neural Text Generation with Unlikelihood Training.  _arXiv:1908.04319 [cs, stat]_.  00000 arXiv: 1908.04319.
  * Williams and Zipser (1989) Ronald J. Williams and David Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280.
  * Xu et al. (2019) Yifan Xu, Kening Zhang, Haoyu Dong, Yuezhou Sun, Wenlong Zhao, and Zhuowen Tu. 2019\.  Rethinking exposure bias in language modeling.  _CoRR_ , abs/1910.11235.
  * Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017.  Seqgan: Sequence generative adversarial nets with policy gradient.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 2852–2858. AAAI Press.
  * Zhang et al. (2018) Amy Zhang, Adam Lerer, Sainbayar Sukhbaatar, Rob Fergus, and Arthur Szlam. 2018\.  Composable Planning with Attributes.  _arXiv:1803.00512 [cs]_.  ArXiv: 1803.00512.
  * Zhang et al. (2019a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
  * Zhang et al. (2019b) Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019b.  Bridging the gap between training and inference for neural machine translation.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4334–4343. Association for Computational Linguistics.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278, Online. Association for Computational Linguistics.