# Intravalley Andreev reflection in the multi-terminal device with Y-shaped
Kekulé graphene superlattices

Chao Wang1,2, Peipei Zhang2, Yu-Xian Li2∗, Juntao Song2∗ 1College of Physics,
Shijiazhuang University, Shijiazhuang 050035, People’s Republic of China
2College of Physics $\&$ Information Engineering and Hebei Advanced Thin Films
Laboratory, Hebei Normal University, Shijiazhuang 050024, People’s Republic of
China

###### Abstract

Using the tight-binding model, a multi-terminal superconductor(S) device is
proposed, where the structures of the center region are primitive grpahene(G)
and Y-shaped Kekulé graphene superlattice(GS), respectively. The intravalley
Andreev reflection is studied in this model through the utilization of the
non-equilibrium Green’s function method. In the G/S device, due to the time-
reverse symmetry the dominant process of Andreev reflection is intervalley
reflection. In a three-terminal GS/S device, it has been observed that the
coefficient of intravalley Andreev reflection can surpass that of intervalley
reflection in crossed Andreev reflection. This is attributed to the coupling
between valleys K and -K within the first Brillouin zone, resulting in
enhanced intervalley scattering. The valley-dependent transport of Andreev
reflection can be influenced by the phase difference between superconductor
terminals. The valley polarization of the local Andreev reflection and the
crossed Andreev reflection could be controlled by adjusting the structure of
the central region.

##  I Introduction

Since graphene(G)Novoselov (1, 2, 6, 3, 7, 8, 9, 4, 5) was successfully
exploited in the laboratory, which is essentially a single layer of pure
carbon atoms bonded together in a honeycomb lattice, it has been the most
interesting two-dimensional material in the research community due to its
special properties. As is known, the unit cell of graphene consists of two
structure-different carbon A and B atoms. The six corners of the hexagonal
Brillouin zone are divided into two inequivalent groups, labeled as the $K$
and $-K$ valleys. The two valleys are related by the time reversal symmetry
and the spatial inversion symmetry, so electrons in graphene have extra
pseudo-spin and valley degrees of freedom besides the usual charge and spin
ones.

Figure 1: (Color online) (a) The diagram of the Y-shaped Kekulé graphene
superlattice and the Brillouin zone(green hexagon). The two basis vectors for
the graphene (superlattices) are labeled as $a_{1}$($b_{1}$) and
$a_{2}$($b_{2}$). (b) The four-terminal graphene-superconductor device with
the Y-shaped Kekule superlattice. The incident electrons travel into the
superconductors, which are represented by the solid arrow lines (black lines),
and the reflected holes travel back to the terminal $1$(blue lines) or
terminal $2$ (brown lines), respectively. Terminal $1$ and $2$ are the
primitive graphene ribbons. $L=2$ ($W=3$) is the length(width) of the center
region of the graphene-superconductor junction.

The large momentum difference between the two valleys in clean graphene
samples effectively suppresses intervalley scattering, thereby conserving the
valley degree of freedom as a quantum number in electron transports. This
conservation can be harnessed to utilize the valley degree of freedom as an
information carrierPesin2012 (10, 11), which is similar to how the spin of the
electron is utilized. Recently, the valley-dependent transport of electron has
drawn much attention in graphene-like materials, where the valley of the
electron can be detected and manipulated. One kind of graphene
superlattice(GS) structures can be constructed by the Kekulé
distortionGiovannetti2015 (12, 13, 14, 15). In the Kekulé GS, the unit cell is
enlarged by $\sqrt{3}\times\sqrt{3}$ so that the points $K$ and $-K$ of the
pristine graphene are folded into the point $\Gamma$ of the GS structure in
the first Brillouin zone, which is shown in Fig. 1 (a). In G/GS and GS/G/GS
structures, valley inverse effectbeenakker2018 (16) and valley
supercurrentwang2020 (17) have been predicted and studied in theory,
respectively.

In a conductor-superconductor(C/S) heterojunction, the conductance is
determined by the Andreev reflectionBeenakker (18, 21, 23, 24, 25, 26, 28, 29,
27, 19, 22, 20) when the bias is less than the superconductor gap. The process
of Andreev reflectionAndreev (30, 31, 32) is that incident electrons from the
conductor are reflected as holes back to the conductor at the interface
between the conductor and superconductor. Due to the time-reversal symmetry,
the incident electron and the reflected hole belong to the different valleys
in the graphene-superconductor structure. In some conditions, the valley
degrees of freedom of the electron and the hole can be manipulatedNiu2007 (33,
37, 36, 34, 35). For example, when the time reversal is broken, the electron
and the hole can come from the same valleywangchao2020 (38, 39).

In our model shown in Figs. 1 (b), it is a four-terminal device. The central
region is the Y-shaped Kekulé superlattices, which are connected with two
graphene terminals and two superconductor terminals. For a three-terminal
device there is only one superconductor terminal. Assuming that incident
electrons from terminal 1 travel into the center region and flow to the
superconductor terminal, there are holes reflected back to the graphene
terminal 1 and 2. The valley degrees of freedom of the electron and the hole
can exhibit either identical or opposite characteristics due to the
intervalley scattering effect. When incident electrons and reflected holes
have the opposite valley degree of freedom, it is intervalley Andreev
reflection. Conversely, for the intravalley Andreev reflection electrons and
holes have the same valley degree of freedom.

The present paper discusses the Andreev reflection in the multi-terminal
device for different valley degrees of freedom, with a particular focus on
intravalley Andreev reflection. For the Y-shaped Kekulé graphene superlattice
structure, whose energy band are shown in Fig. 2 (b), the two valleys couple
together, and the chirality of them are unchanged. After adjusting the on-site
energy of one of six atoms in the unit cell of the superlattice, the chirality
of one valley is broken and the one of the other valley is unbroken, which is
showed in Fig. 2 (c). The coefficients of Andreev reflection, which include
the local Andreev reflection and the crossed Andreev reflection represented as
$T_{A11}$ and $T_{A12}$, are calculated in our works. Comparing the
coefficients of Andreev reflection in the G/S structure and the GS/S
structure, it is found that the intravalley Andreev reflection can be enhanced
in the GS/S structure due to the intervalley scattering effect. The valley
polarization of the Andreev reflection can be controlled by adjusting the
incident energy of electrons in a three-terminal or a four-terminal model. As
a consequence, the results we received provide a new way to the design and
development of electronic components.

The rest of this paper is arranged as follows. In Sec.II, the model
Hamiltonian for the system is presented and the formalisms for calculating the
Andreev reflection coefficients and the valley polarization are derived. Our
main results are shown and discussed in Sec.III. Finally, a brief conclusion
is presented in Sec.IV.

##  II Model and Formula

The superconductor heterojunction investigated here is a Y-shaped Kekulé
graphene superlattice in the central region, which connects with the graphene
terminals and the superconductor terminals. The Hamiltonian of the junction is

$\ H=H_{GS}+H_{S}+H_{T},$ (1)

where $H_{GS}$, $H_{S}$ and $H_{T}$ are the Hamiltonians of the graphene
superlattices ribbon, superconductor terminals, and coupling of the center
region and the superconductor terminals, respectively.

Figure 2: (Color online) The energy band for $E_{0}=-0.05t$. The structure are
(a) Graphene ribbon, (b) Kekulé superlattice with
$U_{A_{\alpha}/B_{\alpha}}=0$ and (c) Kekulé superlattice with only
$U_{A_{3}}=0.5t$. The red dashed line represents the Fermi level $E_{f}=0$.
The width is $W=80$.

In the tight-binding representation, $H_{GS}$ is given byNeto (2)

$\displaystyle H_{GS}=$
$\displaystyle\sum_{m,n}\sum_{\alpha=1,2,3}[E_{A_{\alpha}}a^{{\dagger}}_{\emph{m,n}}a_{\emph{m,n}}+E_{B_{\alpha}}b^{{\dagger}}_{\emph{m,n}}b_{\emph{m,n}}]$
(2) $\displaystyle-
t_{\alpha}[a_{\emph{m,n}}^{\dagger}b_{\emph{m,n}}+a_{\emph{m,n}}^{\dagger}b_{\emph{m-1,n}}+a_{\emph{m,n}}^{\dagger}b_{\emph{m,n-1}}+H.c.],$

where $a_{\emph{m,n}}^{\dagger}$ ($a_{\emph{m,n}}$) and
$b_{\emph{m,n}}^{\dagger}$ ($b_{\emph{m,n}}$) are the creation (annihilation)
operator of the sublattices $A_{\alpha}$ and $B_{\alpha}$. $E_{A_{\alpha}}$
and $E_{B_{\alpha}}$ stand for the energy of the superlattice site, where
$E_{A_{\alpha}/B_{\alpha}}=E_{0}+U_{A_{\alpha}/B_{\alpha}}$ and $E_{0}$ is the
primitive on-site energy and $U_{A_{\alpha}/B_{\alpha}}$ is the modification
of the on-site energy. The second term in the Hamiltonian stands for the
nearest-neighbor hopping, where $t_{\alpha}$ is the hopping energy. For the
Y-shaped Kekulé superlattice the hopping energy is restructured as
$t_{1/2}=t(1-\delta)$ and $t_{3}=t(1+2\delta)$, where $t$ is the hopping
energy in the primitive graphene. It is considered that the region of the
graphene superlattice is directly coupled to the superconductor terminals.
Described by a continuum model the superconductor terminal is represented by
BCS Hamiltonian,

$H_{\emph{S}}=\sum_{\textbf{\emph{k}},\sigma}\varepsilon_{\textbf{\emph{k}}}C^{\dagger}_{\emph{{k}}\sigma}C_{\textbf{\emph{k}}\sigma}+\sum_{\textbf{\emph{k}}}(\Delta_{0}C_{\textbf{\emph{k}}\downarrow}C_{-\textbf{\emph{k}}\uparrow}+\Delta_{0}^{*}C^{\dagger}_{\textbf{\emph{-k}}\uparrow}C^{\dagger}_{\textbf{\emph{k}}\downarrow}),$
(3)

where $\Delta_{0}=\Delta e^{\emph{i}\theta}$. Here $\Delta$ is the
superconductor gap and $\theta$ is the superconductor phase. The coupling
between the superconductor terminal and the center region is described by

$H_{\emph{T}}=\sum_{\emph{m,n},\sigma}\emph{t}a^{\dagger}_{\emph{m,n}\sigma}C_{\sigma}(x_{\emph{i}})+h.c..$
(4)

Here $x_{\emph{i}}$ is the position of the carbon atom i coupling to the
superconductor terminal and
$C_{\sigma}(x)=\sum_{\emph{k}_{x}}e^{\emph{i}k_{x}x}C_{\textbf{\emph{k}}\sigma}$.

The process of the Andreev reflection is that an incident electron with the
energy $\varepsilon$ flows from the graphene terminal into the superlattice
region, then it is reflected as a hole at the interface between the center
region and the superconductor. The reflected hole can flow out of the center
region to either of the two graphene terminals. Using the nonequilibrium Green
function method, we can calculate the retarded and advanced Green function
$G^{r}(\varepsilon)=[G^{a}]^{\dagger}=1/(\varepsilon I-H_{C}-\Sigma^{r})$,
where $H_{C}$ is the Hamiltonian of the center region in the Nambu
representation and $I$ is the unit matrix with the same dimension as $H_{C}$.
The center region is the rectangular region surrounded by the dashed line in
Fig. 1 (b). $\Sigma^{r}$ are the retarded self-energy due to the coupling to
the terminal $\alpha$ and the superconductor terminalsXie2009 (23, 26, 38).

Figure 3: (Color online) In three-terminal GS/S junction $T_{A,11}$ and
$T_{A,12}$ vs the incident energy $\varepsilon$ with $E_{0}=-5\Delta$ when
$W=80$ and $L=50$. The structure of the center region are primitive graphene
in (a) and (b), Y-shaped Kekulé superlattices with $U_{A_{3}}=0$ in (c) and
(d), and superlattices with $U_{A_{3}}=0.5t$ in (e) and (f).

In the Nambu representation, the retard Green function can be described by

${G^{r}}=\left({\begin{array}[]{*{20}{c}}{{G^{r}_{\uparrow\uparrow}}}&{{G^{r}_{\uparrow\downarrow}}}\\\
{{G^{r}_{\downarrow\uparrow}}}&{{G^{r}_{\downarrow\downarrow}}}\end{array}}\right),$
(5)

where the subscripts $\upuparrows$, $\upuparrows$,$\uparrow\downarrow$ and
$\downarrow\uparrow$ represent the 11, 22, 12 and 21 matrix elements,
respectively. Marking the valley degree of freedom of the incident electron
and the reflected hole as $\alpha$ and $\beta$ respectively, the valley-
dependent transport of Andreev reflection is that the incident electron with
velocity $\nu^{\alpha}_{j}$ from terminal 1 is reflected back as the hole with
velocity $\mu^{\beta}_{i}$ to terminal 1 or 2, namely the local Andreev
reflection or the crossed Andreev reflection. We obtain the Andreev reflection
coefficientsAndo1991 (40, 41):

$\displaystyle
T^{\alpha,\beta}_{A11}(\varepsilon)=\sum_{ij}\frac{\mu^{\beta}_{i}}{\nu^{\alpha}_{j}}|\Gamma^{11}_{ij}|^{2},$
(6) $\displaystyle
T^{\alpha,\beta}_{A12}(\varepsilon)=\sum_{ij}\frac{\mu^{\beta}_{i}}{\nu^{\alpha}_{j}}|\Gamma^{12}_{ij}|^{2},$

where

$\displaystyle\Gamma^{11}=\phi^{-1}_{h}(-)G^{r}_{\downarrow\uparrow
11}V_{e}\phi_{e}(+),$ (7)
$\displaystyle\Gamma^{12}=\phi^{-1}_{h}(+)G^{r}_{\downarrow\uparrow
L1}V_{e}\phi_{e}(+).$

In the equations, $\alpha(\beta)$ represents the valley degree of freedom,
which is $K$ or $-K$. $\phi_{e}(\pm)$ and $\phi_{h}(\pm)$ are the wave
function of electron and hole in the graphene terminals, respectively.
$+$($-$) represents that the direction of the wave function is
forward(backward). $\chi_{e}(\pm)$ and $\chi_{h}(\pm)$ are the matrixes of the
eigenvalue of the wave functions, so here is the expression
$V_{e}=tI(\phi_{e}(+)\chi^{-1}_{e}(+)\phi^{-1}_{e}(+)-\phi_{e}(-)\chi^{-1}_{e}(-)\phi^{-1}_{e}(-))$.
From the equations above, the polarization of the Andreev reflection is
defined as

$\displaystyle
P_{A11}=\frac{T^{K,K}_{A11}+T^{-K,K}_{A11}-T^{K,-K}_{A11}-T^{-K,-K}_{A11}}{\sum_{\alpha,\beta}T^{\alpha,\beta}_{A11}},$
(8) $\displaystyle
P_{A12}=\frac{T^{K,K}_{A12}+T^{-K,K}_{A12}-T^{K,-K}_{A12}-T^{-K,-K}_{A12}}{\sum_{\alpha,\beta}T^{\alpha,\beta}_{A12}}.$

##  III Results and discussion

In the numerical calculation, we set the hopping energy $t=2.75$ $eV$. The
length of the nearest-neighbor C-C bond is set to be $a_{0}=0.142$
$\mathrm{nm}$ as in the real graphene sample. The superconductor gap is set to
be $\Delta=0.01t$. For the structure of the central region, there are three
different cases. When $\delta=0$, the center region is primitive graphene.
When $\delta=0.1$, the center region is the Y-shaped Kekulé superlattice. For
the superlattice structure, there are two scenarios: one where all
modifications of the on-site energy are set to zero, and another where only
one modification of the on-site energy is non-zero, such as $U_{A_{3}}=0.5t$.
To facilitate the discussion on valley polarization, we set the Fermi energy
at $E_{f}=0$.

In Fig. 2 (a), we observe that incident electrons can originate from two
distinct valleys, exhibiting different degrees of freedom in terms of valley
states. Due to $|E_{0}|=0.05t>\Delta$, the process of the Andreev reflection
in our study is an intraband reflectionBeenakker (18, 26, 38). Due to the
time-reversal symmetry, in the primitive graphene, electrons and holes taking
parting into the Andreev reflection have the opposite valley degree of
freedom, which is intervalley Andreev refleciton. In Figs. 2 (b) and (c), the
energy band structures of the superlattices are presented, where the two
valleys are coupled together. In the superlattices, due to the coupling
valleys and the stronger intervalley scattering effect, it is possible that
electrons and holes have the same valley degree of freedom, which is
intravalley Andreev reflection.

Figure 3 depicts the coefficients of Andreev reflection in the three-terminal
device at $E_{0}=-5\Delta$. When the structure of the central region is
primitive graphene, it is clear that electrons taking part in local Andreev
reflection mainly come from the valley $K$, and most of them are reflected as
holes belonging to the valley $-K$, which showed in Fig. 3 (a). For $T_{A11}$,
the values of the coefficients of the intravalley Andreev reflection are
always tiny. In other words, in the local Andreev reflection the intravalley
Andreev refleciton is suppressed when the structure of the central region is
primitive graphene, and the dominant contribution comes from the coefficients
of intervalley Andreev reflection. The maximum value of $T^{K,-K}_{A11}$
reaches its peak at $1.35$ in Fig. 3 (a). It reduces to $0.9$ in Fig. 3 (c),
where the structure is superlattices. When the modification $U_{A_{3}}=0.5t$
is introduced into the superlattices, the peak value of $T^{K,-K}_{A11}$ in
Fig. 3 (e) undergoes a reduction to $0.45$. The decrease in $T^{K,-K}_{A11}$
can be attributed to the heightened impact of intervalley scattering, which is
significantly more pronounced in the superlattice configuration than in
primitive graphene. As depicted in Fig. 3 (e), a comparison with Figs. 3 (a)
and (c) reveals an augmentation of intravalley Andreev reflection.

Figure 4: (Color online) In the three-terminal device the valley polarization
of the Andreev reflection vs $\varepsilon$ when $E_{0}=-5\Delta$ for three
different center regions.

For $T_{A12}$, in the primitive graphene, the intervalley reflection is the
dominant process, and the intravalley reflection is nearly suppressed, which
are shown in Fig. 3 (b). When the central region adopts a Y-shaped Kekulé
superlattice structure, with $\varepsilon$ varying within the range of
$(-\Delta,\Delta)$, the curve value $T^{K,K}_{A12}$ consistently surpasses
that of the other three curves depicted in Fig. 3 (d). This implies that a
significant portion of incident electrons originating from valley $K$ are
reflected as holes back into valley $K$, ultimately flowing towards terminal
2. It explains why the peaks of $T^{K,-K}_{A11}$ reduce clearly in the Figs. 3
(c) and (e). In Fig. 2 (b), we can see that the chirality of the two coupling
valley is unbroken although the two valleys couple together. Due to the
coupling valleys, there are more intense intervalley scattering in the central
region. In Fig. 3 (d), the intravalley reflection in the crossed Andreev
reflection is significantly enhanced due to intervalley scattering, while the
intervalley reflection is effectively suppressed, exhibiting a distinct
deviation from the curves observed in Fig. 3 (b). After introducing
$U_{A_{3}}$ the chirality of one valley is broken, which is showed in Fig. 2
(c). The breaking of chiral symmetry results in an intensified intervalley
scattering effect. Therefore, in Fig. 3 (f), although the primary contribution
to $T_{A12}$ originates from $T^{K,K}_{A12}$, it is crucial not to overlook
the contribution of intervalley reflections.

Figure 5: (Color online) The valley polarization of the Andreev reflection in
the three-terminal device for Y-shaped Kek superlattices with $U_{A_{3}}=0$.
The dashed lines corresponds to the value $E_{0}=-5\Delta$.

According to the coefficients of valley-dependent Andreev reflection mentioned
above, the valley polarizations of three distinct structures in the central
region are calculated and depicted in Fig. 4. For $P_{A11}$, small plateaus
can be observed around $\varepsilon=0$, while as $|\varepsilon|$ increases,
the curves exhibit oscillations with consistently positive polarization
values. From Fig. 4(a), it is evident that when transitioning from primitive
graphene to a Y-shaped Kekulé superlattice structure in the central region,
$P_{A11}$ increases from $0.7$ to $0.85$ at $\varepsilon=0$. However, upon
introducing $U_{A_{3}}=0.5t$, $P_{A11}$ decreases back to $0.7$ again. This
indicates that modulation of the center region’s structure can effectively
control the valley polarization of Andreev reflection.

The curves in Fig. 4 (b) are the valley polarization of the crossed Andreev
reflection with $\varepsilon$ changing. When the central region consists of
primitive graphene, it exhibits a distinct behavior compared to $P_{A11}$, as
$P_{A12}$ can transition from a negative to positive value within the range of
$\varepsilon$ changing from $-0.5\Delta$ to $0.5\Delta$. In the case of
crossed Andreev reflection in primitive graphene, the sign of valley
polarization is dependent on the incident energy. By modifying the structure
into superlattices, $P_{A12}$ maintains its negative value regardless of
$\varepsilon$, with values reaching as low as $-0.9$ for
$|\varepsilon|<0.5\Delta$. Introducing $U_{A_{3}}$ into these superlattices
results in an increase in $P_{A12}$ to approximately $-0.25$ at
$\varepsilon=0$, while still retaining its negative nature throughout. This
suggests that manipulating the structure of the central region may offer
potential control over valley polarization in Andreev reflection.

Figure 6: (Color online) $T_{A11}$ vs the incident energy $\varepsilon$ in the
four-terminal device for three different center region. The phase difference
between the superconductor terminals are $\Delta\theta=0$ in (a) (c) and (e),
and $\Delta\theta=\pi$ in (b) (d) and (f). The structure of the center region
are primitive graphene in (a) and (b), Y-shaped Kekulé superlattices with
$U_{A_{3}}=0$ in (c) and (d), and superlattices with $U_{A_{3}}=0.5t$ in (e)
and (f). The length of the centre region is $L=50$ and the width is $W=80$.

In order to investigate the variation of valley polarization in Andreev
reflection within a three-terminal device featuring a Y-shaped Kekulé
superlattice, both the incident energy $\varepsilon$ and on-site energy
$E_{0}$ are simultaneously adjusted. As shown in Fig. 5, distinct boundaries
can be observed for both $P_{A11}$ and $P_{A12}$, dividing the image into two
regions with positive and negative values of valley polarization,
respectively. In proximity to these boundaries, $P_{A11}$ and $P_{A12}$ can
attain extreme values of $\pm 1$. By manipulating the on-site energy $E_{0}$,
it becomes possible to control the distance between the Dirac point and Fermi
energy within the energy band. In Fig. 2 (a), when the Fermi level intersects
only with the first band, both the incident electron and the reflected hole
possess a fixed valley degree of freedom, resulting in maximum valley
polarization. By increasing $|E_{0}|$ to sufficiently large values, such as
$E_{0}=-5\Delta$, due to intervalley scattering in the Y-shaped Kekulé
superlattice, the reflected hole can have either $K$ or $-K$ as its valley
degree of freedom when it flows into the graphene terminals. It is reasonable
for a slight decrease in valley polarization for large $|E_{0}|$. In Fig. 5,
it is evident that the sign of valley polarization primarily depends on the
on-site energy $E_{0}$ in devices with Y-shaped Kekulé superlattices.

Figure 7: (Color online) $T_{A12}$ vs the incident energy $\varepsilon$ in the
four-terminal device for three different center region. The phase difference
between the superconductor terminals are $\Delta\theta=0$ in (a) (c) and (e),
and $\Delta\theta=\pi$ in (b) (d) and (f). The structure of the center region
are primitive graphene in (a) and (b), Y-shaped Kekulé superlattices with
$U_{A_{3}}=0$ in (c) and (d), and superlattices with $U_{A_{3}}=0.5t$ in (e)
and (f). The length of the center region is $L=50$ and the width is $W=80$.

In previous worksCheng (26, 38, 32), Andreev reflection can be affected by the
phase difference between the two superconductor terminals in the four-terminal
device. In the four-terminal device, the coefficients of Andreev reflection,
namely $T_{A11}$ and $T_{A12}$, exhibit variations with incident energy under
different superconductor phase differences $\Delta\theta$, as depicted in Fig.
6 and Fig. 7 respectively. By comparing the coefficients at $\Delta\theta=0$
and $\Delta\theta=\pi$, it becomes evident that the superconductor phase
difference significantly impacts Andreev reflection. Considering a primitive
graphene structure for the central region, we calculate the coefficients of
$T_{A11}$ presented in Fig. 6 (a) and (b). Regardless of the magnitude of the
superconductor phase difference, $T^{K,-K}_{A11}$ consistently surpasses other
coefficients. Within the range of $0.5\Delta<|\varepsilon|<1.0\Delta$, there
is a remarkable decrease in $T^{K,-K}_{A11}$ when adjusting from
$\Delta\theta=0$ to $\Delta\theta=\pi$. By comparison, the adjustment of the
superconductor phase difference has little effect on the intravalley
reflection $T^{K,K}_{A11}$ and $T^{-K,-K}_{A11}$, whose values are always
tiny. Then, the central region undergoes a structural transformation into a
Y-shaped Kekulé superlattice, as depicted in Figs. 6 (c) and (d). It is
observed that within the range of $0.5\Delta<|\varepsilon|<1.0\Delta$,
adjusting the phase difference of the superconductors from $\Delta\theta=0$ to
$\Delta\theta=\pi$ enhances $T^{K,-K}_{A11}$. However, upon introducing
$U_{A_{3}}=0.5t$ into the superlattice, there is a decrease in the peak value
of $T^{K,-K}_{A11}$ and its dependence on the superconductor phase difference
becomes limited. Comparing Figs. 6 (e) and (f), it can be observed that both
coefficients for intravalley reflections, namely $T^{K,K}_{A11}$ and
$T^{-K,-K}_{A11}$, are consistently lower than $T^{K,-K}_{A11}$. Nevertheless,
when the superconductor phase difference is set at $\Delta\theta=\pi$, these
coefficients no longer exhibit symmetry about $\varepsilon=0$.

Under the different superconductor phase difference $\Delta\theta$ the
variation of $T_{A11}$ has been discussed in the four-terminal device. In Fig.
7 it shows how the coefficients of $T_{A12}$ change with the incident energy
$\varepsilon$ under different superconductor phase difference. When the
central region consists of primitive graphene, $T_{A12}$ exhibits distinct
characteristics compared to the three-terminal device. Specifically,
irrespective of the phase difference between the superconductors, the
intravalley reflection $T^{K,K}_{A12}$ exhibits a greater intensity compared
to that observed in the three-terminal device. Moreover, for
$\Delta\theta=\pi$, there is an enhancement in the intervalley reflections
$T^{K,-K}_{A12}$ and $T^{-K,K}_{A12}$. However, when we consider a Y-shaped
Kekulé superlattice as the structure of the central region in a four-terminal
device, we observe that intravalley reflection remains as the dominant process
for crossed Andreev reflection, similar to its behavior in a three-terminal
device. The curves depicting $T^{K,K}_{A12}$ can be seen in Fig. 7 (c) and
(d). Around $\varepsilon=0$, there exists a minimum value of $T^{K,K}_{A12}$
when $\Delta\theta=0$. Conversely, $T^{K,K}_{A12}$ exhibits a peak value
around $\varepsilon=0$ when $\Delta\theta=\pi$. It is evident that the
strength of the crossed Andreev reflection coefficients can be controlled by
adjusting the phase difference in the superconductors. Upon introducing
$U_{A_{3}}=0.5t$ into the superlattice, the impact of the superconductor phase
difference on $T_{A12}$ diminishes. The curves depicted in Fig. 7 (e) resemble
those in Fig. 7 (f). From Figs. 6 and 7, it can be observed that the phase
difference between two superconductor terminals influences Andreev reflection;
however, this influence weakens due to broken chiral symmetry.

Figure 8: (Color online) In the four-terminal device $P_{A12}$ vs the incident
energy $\varepsilon$ for three different center regions. The difference of the
superconductor phase are (a) $\Delta\theta=0$ , and (b) $\Delta\theta=\pi$,
respectively. The length of the centre region is $L=50$ and the width is
$W=80$.

The curves in Fig. 8 illustrate the valley polarization of crossed Andreev
reflection in a four-terminal device for two different phase differences
between the superconductors, namely $\Delta\theta=0$ and $\Delta\theta=\pi$.
In Fig. 8 (a), except for the black curve, the values of $P_{A12}$ are
consistently negative within the range of $|\varepsilon|<\Delta$. For the
black curve, as $\varepsilon$ varies from $0.2\Delta$ to $0.6\Delta$, the
valley polarization exhibits a plateau-like behavior approaching -1,
indicating that this corresponds to a primitive graphene structure in the
central region. However, when transitioning to a Y-shaped Kekulé superlattice
structure in the central region, there is an increase in plateau value around
-0.9 for the red curve. After incorporating $U_{A_{3}}=0.5t$ into the
superlattice, for $-0.5\Delta<\varepsilon<0.5\Delta$, the value of $P_{A12}$
increases up to -0.5 as depicted by the blue curve. By setting the phase
difference between the superconductors as $\Delta\theta=\pi$, Fig. 8 (b)
illustrates that the black curve exhibits a positive value within the range of
$-\Delta<\varepsilon<0$ and transitions to a negative value for
$0<\varepsilon<\Delta$. Notably, there is a plateau with a positive value
around 0.5 observed for $-0.9\Delta<\varepsilon<-0.5\Delta$. Interestingly,
within this same range, the red curve displays a plateau with a negative value
around $-0.5$, indicating an inversion of valley polarization due to changes
in the structure of the central region.

##  IV Conclusions

In this study, we calculate the coefficients of intervalley and intravalley
Andreev reflection in a multi-terminal device that we propose. The central
region of the device consists of primitive graphene and a Y-shaped Kekulé
graphene superlattice. From the energy band structure of the Y-shaped Kekulé
graphene superlattice, it is evident that the two valleys are coupled
together, resulting in enhanced intervalley scattering. Consequently, it is
possible for both electrons and holes involved in Andreev reflection to
originate from the same valley. In the three-terminal device, the coefficient
of intravalley reflection $T^{K,K}_{A12}$ is greater than that of intervalley
reflection when employing a Y-shaped Kekulé graphene superlattice as the
structure in the center region. The valley polarization of Andreev reflection
can be controlled by adjusting both the structure and on-site energy. In a
four-terminal device, the valley-dependent transport of Andreev reflection can
be influenced by manipulating the phase difference between superconductor
terminals. To summarize, it is possible to manipulate the degree of freedom
associated with valleys during Andreev reflection, providing novel ideas for
designing and developing electronic components.

## ACKNOWLEDGMENTS

This work was supported by the National Natural Science Foundation of China
(Grant No. 12274305 and No. 11874139), the Natural Science Foundation of Hebei
(Grant No. A2022106001 and No. 2019205190).

## References

  * (1) Corresponding authors:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
  * (2) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Gregorieva, and A. A. Firsov, Science 306, 666 (2004).
  * (3) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov and A. K. Geim, Rev. Mod. Phys. 81, 109-162 (2009).
  * (4) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, M. I. Katsnelson, I. V. Grigorieva, S. V. Dubonos, and A. A. Firsov, Nature(London) 438, 197 (2005).
  * (5) Andre K. Geim, Rev. Mod. Phys. 83, 851-862 (2011).
  * (6) S. Das Sarma, Shaffique Adam, E. H. Hwang and Enrico Rossi, Rev. Mod. Phys. 83, 407-470 (2011).
  * (7) F. D. M. Haldane, Phys. Rev. Lett. 61, 2015 (1988).
  * (8) V. V. Cheianov and V. I. Fal’ko, Phys. Rev. B 74, 041403 (2006).
  * (9) A. F. Young and P. Kim, Nat. Phys. 5, 222 (2009).
  * (10) K. Nakada, M. Fujita, G. Dresselhaus and M. S. Dresselhaus, Phys. Rev. B 54, 17954 (1996).
  * (11) D. Pesin and A. H. MacDonald, Nat. Mater. 11, 409 (2012).
  * (12) D. Xiao, W. Yao, and Q. Niu, Phys. Rev. Lett. 99, 236809 (2007).
  * (13) G. Giovannetti, M. Capone, J. van den Brink, and C. Ortix, Phy. Rev. B. 91, 121417 (2015).
  * (14) J. J. Wang, S. Liu, J. Wang, and J. F. Liu, Phys. Rev. B 98, 195436 (2018).
  * (15) E. Andrade, R. C. Bastos, P. A. Pantaleón, and F. Mireles, Journal of Applied Physics 127, 054304 (2020).
  * (16) C. Gutiérrez, C. J. Kim, L. Brown, T. Schiros, D. Nordlund, E. B.Lochocki, K. M. Shen, J. Park, and A. N. Pasupathy, Nature Physics 12, 950-958 (2016).
  * (17) C. W. J. Beenakker, N. V. Gnezdilov, E. Dresselhaus, V. P. Ostroukh, Y. Herasymenko, I. Adagideli, and J. Tworzydlo, Phys. Rev. B 97, 241403(R) (2018).
  * (18) J. J. Wang, S. Liu, J. Wang, and J. F. Liu, Phys. Rev. B 101, 245428 (2020).
  * (19) C. W. Beenakker, Phys. Rev. Lett. 97, 067007 (2006); C. W. Beenakker, Rev. Mod. Phys. 80, 1337-1354 (2008).
  * (20) F. Amet, C. T. Ke, I. V. Borzenets, J. Wang, K. Watanabe, T. Taniguchi, R. S. Deacon, M. Yamamoto, Y. Bomze, S.Tarucha and G. Finkelstein, Science 352, 6288 (2016).
  * (21) P. Rickhaus, M. Weiss, L. Marot, and C. Schönenberger, Nano Lett. 12, 1942-1945 (2012).
  * (22) M. Titov, and C. W. Beenakker, Phys. Rev. B 74, 041401(R) (2006).
  * (23) J. Schelter, B. Trauzettel and P. Recher, Phys. Rev. Lett. 108, 106603 (2012).
  * (24) Q. F. Sun and X. C. Xie, J. Phys. Condens Matter 21, 344204 (2009).
  * (25) D. K. Efetov, C. Handschin, K. B. Efetov, J. Shuang, R. Cava, T. Taniguchi, K. Watanabe, J. Hone, C. R. Dean and P. Kim, Nature Physics 12, 328-332 (2016).
  * (26) J. Linder, and A. Sudb$\phi$, Phys. Rev. B 76, 214508 (2007).
  * (27) S. G. Cheng, Y. Xing, J. Wang and Q. F. Sun, Phys. Rev. Lett. 103, 167003 (2009).
  * (28) Yanxia Xing, Jian Wang and Qing-feng Sun, Phys. Rev. B 83, 205418 (2011).
  * (29) Q. Zhang, D. Fu, B. Wang, R. Zhang, and D. Y. Xing, Phys. Rev. Lett. 101, 047005 (2008).
  * (30) D. Rainis, F. Taadei, F. Dolcini, M. Polini and R. Fazio, Phys. Rev. B 79, 115131 (2009).
  * (31) A. F. Andreev, Sov. Phys. JETP 19, 1228 (1964).
  * (32) M. J. M. de Jong and C. W. J. Beenakker, Phys. Rev. Lett. 74, 1657-1660 (1995).
  * (33) Lin Zhang, Chao Wang, Peipei Zhang, and Yu-Xian Li Phys. E 141, 115237 (2022).
  * (34) D. Xiao, W. Yao and Q. Niu, Phys. Rev. Lett. 99, 236809 (2007).
  * (35) A. Rycerz, J. Tworzydlo, and C.W. J. Beenakker, Nature Phys. 3, 172 (2007).
  * (36) D. Xiao, G. B. Liu, W.X.Feng, X. D. Xu and W. Yao, Phys. Rev. Lett. 108, 196802 (2012).
  * (37) R. V. Gorbachev, J. C. W. Song, G. L. Yu, A. V. Kretinin, F. Withers, Y. Cao, A. Mishchenko, I. V. Grigorieva, K. S. Novoselov, L. S. Levitov and A. K. Geim, Science 346, 6208 (2014)
  * (38) W. Yao, D. Xiao, and Q. Niu, Phys. Rev. B 77, 235406 (2009).
  * (39) C. Wang, L. Zhang, P.P. Zhang, J. T. Song, and Y.X. Li, Phys. Rev. B 101, 045407 (2020).
  * (40) W. Zeng, and R. Shen, Phys. Rev. B 104, 075436 (2021).
  * (41) T. Ando, Phys. Rev. B 44, 8017 (1991).
  * (42) X.T. An and W. Yao, Physical Review Applied 14, 014039 (2020).


# TempoSum: Evaluating the Temporal Generalization of
Abstractive Summarization

Chi Seng Cheang1 Hou Pong Chan1 Derek F. Wong1 Xuebo Liu2
Zhaocong Li1 Yanming Sun1 Shudong Liu1 Lidia S. Chao1
1NLP2CT Lab, Department of Computer and Information Science, University of
Macau
2Institute of Computing and Intelligence, Harbin Institute of Technology,
Shenzhen, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent pre-trained language models (PLMs) achieve promising results in
existing abstractive summarization datasets. However, existing summarization
benchmarks overlap in time with the standard pre-training corpora and
finetuning datasets. Hence, the strong performance of PLMs may rely on the
parametric knowledge that is memorized during pre-training and fine-tuning.
Moreover, the knowledge memorized by PLMs may quickly become outdated, which
affects the generalization performance of PLMs on future data. In this work,
we propose TempoSum, a novel benchmark that contains data samples from 2010 to
2022, to understand the temporal generalization ability of abstractive
summarization models. Through extensive human evaluation, we show that
parametric knowledge stored in summarization models significantly affects the
faithfulness of the generated summaries on future data. Moreover, existing
faithfulness enhancement methods cannot reliably improve the faithfulness of
summarization models on future data. Finally, we discuss several
recommendations to the research community on how to evaluate and improve the
temporal generalization capability of text summarization models111Our TempoSum
benchmark will be available at https://github.com/AndyCheang/TempoSum..

TempoSum: Evaluating the Temporal Generalization of
Abstractive Summarization

Chi Seng Cheang1 Hou Pong Chan1 Derek F. Wong1 Xuebo Liu2 Zhaocong Li1 Yanming
Sun1 Shudong Liu1 Lidia S. Chao1 1NLP2CT Lab, Department of Computer and
Information Science, University of Macau 2Institute of Computing and
Intelligence, Harbin Institute of Technology, Shenzhen, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Abstractive summarization aims to generate a concise summary that contains the
critical information of the source text while ensuring the generated text is
fluent and faithful. In recent years, pre-trained language models (PLMs) have
accomplished state-of-the-art outcomes across numerous summarization
benchmarks Zhang et al. (2020); Lewis et al. (2020). Nonetheless, the test
sets within these benchmarks are predominantly comprised of data samples prior
to 2019, resulting in a substantial temporal overlap with both the training
sets and standard pre-training corpora. As summarization systems are commonly
deployed in practical applications for processing text articles from future
time periods, it is crucial to consider their performance on such data. PLMs
that excel on data originating from the same temporal context as the pre-
trained corpus may not necessarily exhibit robust generalization capabilities
when applied to future datasets.

News article from CNN in 2015:
---
Add Vice President Joe Biden to the list of potential 2016 candidates
traveling to the first-in-the-nation caucus state. Biden will travel on
official White House business next week to Des Moines, Iowa where he will
deliver a speech on the administration’ s economic policies, his office
announced Friday …
Generated Summary:
Vice President Joe Biden will travel to Iowa next week …
News article from CNN in 2021:
President Joe Biden’s mission to speed up the lackluster rollout of
coronavirus vaccines. Biden signed a slew of executive orders this week aimed
at accelerating the distribution of vaccines and tests …
Generated Summary:
Vice President Joe Biden has signed executive orders to speed up vaccine
distribution …
Figure 1:  Two sample CNN news articles in 2015 and 2021, respectively, as
well as their summaries generated by the PEGASUS Zhang et al. (2020) model
that is fine-tuned on the CNN/DM dataset. We highlight the entity mentions of
Joe Biden in the articles and summary in red color. We underline the factual
error in the summaries.

In Figure 1, we present two CNN news articles in 2015 and 2021, respectively,
as well as their summaries generated by the PEGASUS Zhang et al. (2020) model
that is fine-tuned on the CNN/DM dataset. From the summary of the article in
2015, we observe that PEGASUS generates “Vice President Joe Biden”, which is
faithful to the source document. However, since both the training set of
CNN/DM and the pre-training corpora of PEGASUS contain text data in 2015, it
is not clear whether the model utilizes the document content or the parametric
world knowledge that is memorized during pre-training and fine-tuning to
generate the correct information about Joe Biden.

On the other hand, the news article in 2021 from Figure 1 indicates that Joe
Biden is a president. Since both the CNN/DM training set and standard pre-
training corpora do not contain data after 2019, the PEGASUS model must
leverage the document content to generate the expression “President Joe
Biden”. Unfortunately, the PEGASUS model hallucinates the content of “Vice
President Joe Biden” in the summary which is unfaithful.

Based on the above qualitative results, we suspect that the promising
performance of PLMs in existing summarization benchmarks relies on their
parametric world knowledge. In order to reliably evaluate the temporal
generalization ability of pre-trained summarization models, the source
articles in an evaluation benchmark should contain knowledge conflicts, i.e.,
the articles should present knowledge that contradicts the parametric world
knowledge in PLMs.

To this end, we propose the TempoSum benchmark, which contains text articles
with knowledge conflicts to assess the temporal generalization capability of
text summarization models. Specifically, our benchmark consists of two
summarization datasets of different abstractiveness collected from CNN and BBC
News ranging from 2010 to 2022. For each dataset, we construct a future test
set, in which all the articles are published after 2019 and contain knowledge
conflicts to evaluate the robustness of pre-trained summarization on articles
from a future temporal period.

Based on our benchmark, we present the first systematic human evaluation of
the temporal generalization ability of text summarization models. This paper
aims to answer two important research questions. First, how do the pre-
training process and fine-tuning datasets affect the temporal generalization
abilities of summarization models? We compare the performance of the SOTA pre-
trained text summarization model, PEGASUS, and a non-pretrained Transformer
model on our proposed benchmark. Our results demonstrate that although the
pre-training process of PEGASUS helps the model generates more meaningful and
grammatical summaries, it also encourages the model to hallucinate outdated
information according to its parametric world knowledge. Moreover, we observe
that the divergence between the source documents and their reference summaries
in the finetuning dataset causes summarization models to rely on their
parametric world knowledge instead of the source text.

Second, are recent faithfulness enhancement methods and faithfulness
evaluation methods effective on future data? We first analyze the performance
of two recent faithfulness enhancement methods, CLIFF Cao and Wang (2021) and
ENT Xiao and Carenini (2022), which are built on contrastive learning and copy
mechanism, respectively. We find that the contrastive learning-based method
outperforms the copy-based method in terms of faithfulness improvement on
future data. However, the performance of these two faithfulness enhancement
methods heavily depends on the domain of the data. Namely, both of these
methods fail to improve the faithfulness of the text summarization in future
CNN news articles. Then, we investigate the human correlations of two popular
faithfulness evaluation metrics, FactCC Kryscinski et al. (2020) and
QAFactEval Fabbri et al. (2022), on future data. The results show that both of
these metrics have a weak correlation with human judgments.

Taken together, it is essential to use temporally split test sets that contain
knowledge conflicts to accurately estimate the temporal generalization
performance of text summarization models. Our findings suggest that PLMs are
prone to rely on their parametric knowledge and exhibit a tendency to generate
hallucinations containing outdated knowledge. Therefore, a new evaluation
protocol is required to estimate the extent to which PLMs rely on their
parametric world knowledge.

##  2 Related Work

###  2.1 Text Summarization Datasets

Most of the existing text summarization datasets, such as CNN/Dailymail
Hermann et al. (2015); Nallapati et al. (2016), XSum Narayan et al. (2018),
Newsroom Grusky et al. (2018), focus on single document summarization in the
news domain. The Multi-News dataset Fabbri et al. (2019) is proposed for
multi-document news summarization, in which the input is a set of news
documents. The SumREN dataset Reddy et al. (2022) is constructed for reported
speech summarization that aims to summarize the reported statements made by a
specific person in news documents. Other text summarization datasets focused
on summarizing scientific articles Cohan et al. (2018); Lu et al. (2020),
legal documents Kornilova and Eidelman (2019), or dialogues Gliwa et al.
(2019); Zhu et al. (2021); Chen et al. (2021). However, the test sets of the
above datasets do not evaluate the temporal generalization capability of
summarization models. A realistic evaluation benchmark should include data
from different temporal periods to ensure an accurate evaluation. As most of
the existing PLMs are pre-trained on corpora before 2019, TempoSum includes
the data collected from 2010 to 2022. The extended collection periods allow
the research community to explore the temporal generalization ability of these
models. Recently, Goyal et al. (2022) collect 100 news articles in 2022 from
CNN and BBC, but they do not select articles that contain knowledge conflicts.
In contrast, our benchmark comprises recent news articles that contain
knowledge conflicts with existing PLMs, thereby enabling us to examine the
temporal generalization ability of PLMs.

###  2.2 Evaluation of Faithfulness and Factuality

There are several datasets that annotate the faithfulness of generated
summaries. Some of these datasets Kryscinski et al. (2020); Wang et al.
(2020); Fabbri et al. (2021) annotate an overall faithfulness label for each
summary. Pagnoni et al. (2021); Goyal and Durrett (2021); Huang et al. (2020)
define a more fine-grained typology of faithfulness errors. Other datasets
Maynez et al. (2020); Cao and Wang (2021); Cao et al. (2022) annotate
factuality labels to indicate whether the generated summaries are factually
correct. All the above datasets annotate summaries in the existing CNN/DM and
XSum benchmarks. By contrast, we collect recent news articles that contain
knowledge conflicts to study the temporal generalization ability of PLMs.

###  2.3 Out-of-distribution Generalization of Language Models

Some previous works Raffel et al. (2020); Guo and Yu (2022); Grangier and Iter
(2022) study the generalization capability of language models (LMs) in out-of-
domain data. Other studies Lazaridou et al. (2021); Luu et al. (2022); Röttger
and Pierrehumbert (2021); Rosin et al. (2022); Loureiro et al. (2022); Agarwal
and Nenkova (2022) analyze the temporal generalization ability of LMs on
future test sets that do not have temporal overlap with the training set and
standard pre-training corpora, but these studies do not analyze the effects of
parametric world knowledge on the temporal generalization ability of PLMs.
Recently, Longpre et al. (2021) propose a method to artificially generate data
samples that contain knowledge conflicts and study how parametric world
knowledge affects the performance of question-answering models. In contrast,
our method selects recent news articles that present knowledge conflicts.
Thus, the distribution of the samples created by our method is more similar to
the distribution of real data. Moreover, we conduct extensive human
evaluations to analyze the types of errors made by pre-trained summarization
models.

##  3 TempoSum Benchmark

We first propose the TempoSum benchmark to investigate the temporal
generalization capability of PLMs on future documents that present knowledge
conflicts, i.e., the scenarios where the knowledge presented in the source
document contradicts the parametric world knowledge in PLMs.

###  3.1 Data Source

Previous pre-trained summarization models are mostly fine-tuned on the XSum
Narayan et al. (2018) or CNN/DM Hermann et al. (2015) datasets. To evaluate
the temporal generalization ability of PLMs that are finetuned on XSum and
CNN/DM, we create the following two datasets that contain articles from an
extended time period. We filter out all the samples that overlap with the
training sets of XSum and CNN/DM.

##### BBC dataset:

The existing XSum dataset consists of news articles from BBC news between 2010
to 2017. The reference summaries in XSum are highly abstractive. In our
benchmark, we crawl news articles from BBC news between 2010 to 2022 to
construct the BBC dataset.

##### CNN dataset:

The existing CNN/DM dataset contains news articles from CNN news and Dailymail
news between 2007 and 2015. The reference summaries in CNN/DM are highly
extractive. We collect news articles from CNN news222We exclude DailyMail from
our benchmark because this news agency is reported to have low credibility
Goyal et al. (2022) between 2012 to 2022 to construct the CNN dataset in our
benchmark.

###  3.2 Test Set Construction

To understand the temporal generalization ability of pre-trained summarization
models, we construct a future test set and an in-distribution test set for
each dataset in our benchmark. A future test set consists of samples that
present knowledge conflicts, while an in-distribution test set comprises
samples that are consistent with the parametric knowledge of pre-trained
summarization models. To construct the test sets in our benchmark, we first
collect evolving facts that change over time. Next, we assume that the
collected evolving facts that occurred after 2019 are knowledge conflicts and
use them to select news articles.

####  3.2.1 Identifying evolving facts

We use Wikidata Vrandecic and Krötzsch (2014) as the knowledge base to collect
evolving facts. Wikidata store facts in the form of (subject, relation,
object) triples. Since standard pre-training corpora only obtain data up to
2019, we assume that any fact that is changed after 2019 is unaware by PLMs.
Concretely, if the object of a (subject, relation) pair is changed after 2019,
then all the facts that contain the (subject, relation) pair are considered as
evolving facts. For example, Wikidata contains a fact (Joe Biden, position-
held, Vice President of the United States) since 2009. However, the object of
(Joe Biden, position-held) is changed to “President of the United States” in
2021. Thus, both the facts (Joe Biden, position-held, Vice President of the
United States) and (Joe Biden, position-held, President of the United States)
are considered as evolving facts.

As the space of evolving facts is extremely large, our work focuses on the
facts that are related to politicians as a proxy to study the effects of
conflicted knowledge on PLMs. Specifically, we only extract evolving facts
that have the “position-held” or “position-held-by” relation. We select these
two relations because the standard text summarization fine-tuning datasets
(e.g., CNN/DM, XSum) focus on the news domain, and most of their samples
contain politicians. Moreover, as the positions of government officials often
change, the facts that contain these two relations have strong temporal
dynamics Lazaridou et al. (2021); Dhingra et al. (2022).

####  3.2.2 Selecting articles with evolving facts

For both BBC and CNN datasets in our benchmark, we only crawl the news
articles in which the salient content contains the subject or object entity of
an evolving fact333We crawl archived BBC and CNN articles from Internet
Archive (https://web.archive.org/). It is because PLMs prone to rely on their
parametric knowledge when summarizing the entities that are related to
evolving facts, as shown in Figure 1. Specifically, all the crawled articles
satisfy the following two criteria: (1) the article includes both the subject
and object of at least one evolving fact; (2) the reference summary includes
either the subject or the object of an evolving fact. In this way, all our
crawled articles that are published after 2019 should contain facts that
contradict the knowledge stored in PLMs (knowledge conflicts). For each of the
BBC and CNN datasets, we partition the crawled articles into future test set
and in-distribution test set according to their publication time. The crawled
articles that were published after 2019 belong to the future test set; the
remaining articles belong to the in-distribution test set. We present the
statistics of our constructed test sets in Table 1.

###  3.3 Human Evaluation Protocol

One major harmful effect of parametric world knowledge is that it may cause
summarization models to hallucinate contents that are not faithful to the
source document, as shown in Figure 1. To understand how parametric world
knowledge affects the faithfulness of the generated summaries, we define three
fine-grained categories of hallucinations, which include outdated
hallucinations, non-verifiable hallucinations and factual hallucination. Table
2 presents the complete definition of our proposed typology of hallucinations.
In contrast, previous works Maynez et al. (2020) only determine whether the
hallucinations are factual or non-factual.

For each generated summary, we first decide whether the sentence is faithful,
i.e., the information can be entailed by the source article. If it is not
faithful, we verify it through Wikipedia to decide whether they are factual to
world knowledge at the time the article was published or outdated if it could
be connected to the knowledge from the past. If there is no information found
to support the unfaithful entities, then labeled as non-verifiable errors. The
complete human evaluation instruction is in Appendix A.

Dataset | Test set | Year | # Samples
---|---|---|---
BBC | In-dist. | 2010-2017 | 6,254
Future | 2020-2022 | 2,260
CNN | In-dist. | 2012-2015 | 970
Future | 2020-2022 | 3,250
Table 1:  The dataset statistics of TempoSum. Our evaluation benchmark consists of two datasets: BBC and CNN. We construct an in-distribution (In-dist.) and a future test set for each dataset.  Category |  Definition |  Example
---|---|---
Outdated hallucinations |  Hallucinations that are consistent with world knowledge before the time in which the article is published. |  Vice President Joe Biden has signed executive orders to speed up vaccine distribution.
Non-verifiable hallucinations |  Hallucinations that cannot be verified by world knowledge at any time stamp. If a summary contains totally ungrammatical sentences, we will also consider the grammatical errors as non-verifiable hallucinations. |  President Thomas Biden has signed executive orders to speed up vaccine distribution.
Factual hallucinations |  Hallucinations that are consistent with world knowledge at the time the article is being published. |  US President Joe Biden has signed executive orders to speed up vaccine distribution.
Table 2: Our proposed typology for hallucinations in system-generated
summaries. The word spans that are inconsistent with the source text
(hallucinations) are highlighted in red color. The truncated source article
(published in 2021) for the examples: President Joe Biden’s mission to speed
up the lackluster rollout of coronavirus vaccines. Biden signed a slew of
executive orders this week aimed at accelerating the distribution of vaccine
and tests.

##  4 Experiment Settings

###  4.1 Summarization Models

We perform extensive experiments using the following models. More
implementation details are described in Appendix D.

##### PEGASUS:

The state-of-the-art pre-trained text summarization model. This model is a
Transformer model that is pre-trained using the gap-sentence prediction
objective Zhang et al. (2020).

##### Transformer:

To study how the pre-training process affects the temporal generalization
performance of summarization models, we also evaluate the performance of a
non-pretrained vanilla Transformer model Vaswani et al. (2017) in our
experiments.

##### CLIFF:

A faithfulness enhancement method that applies contrastive learning to teach a
summarization model to distinguish faithful summaries from unfaithful
summaries Cao et al. (2022). This method implicitly encourages text
summarization models to generate faithful summaries. We apply the CLIFF method
to the PEGASUS model in our experiments.

##### ENT:

A faithfulness enhancement method that encourages summarization models to
explicitly copy salient entities from the source text via an entity span copy
mechanism Xiao and Carenini (2022). We apply the ENT method to the PEGASUS
model in our experiments.

###  4.2 Human Evaluation

We conduct extensive human evaluations to annotate the types of hallucinations
made by system-generated summaries444We use the annotation tool proposed by
Nakayama et al. (2018) to conduct our human evaluation.. Four authors from
this paper annotate the above model-generated summaries. We annotate 150
random articles from each test split (a total of 600 articles for both the BBC
and CNN datasets). Each sample is annotated by two of our authors. The
matching rate of annotations on the in-distribution test set was 68% and 76%
for the BBC and CNN datasets. As for the future test set, the annotations
matched 72% and 85% for the BBC and CNN datasets, respectively.

###  4.3 Automatic Evaluation Metrics

##### FactCC:

An entailment-based faithfulness evaluation metric proposed by Kryscinski et
al. (2020). Previous studies Pagnoni et al. (2021) show that FactCC has a
strong correlation with human judgments on existing benchmarks.

##### QAFactEval:

The state-of-the-art faithfulness evaluation for text summarization Fabbri et
al. (2022). This metric utilizes question-generation and question-answering
models to estimate the faithfulness of a summary.

##  5 Result Analysis

###  5.1 How do the pre-training process and fine-tuning datasets affect the
temporal generalization abilities of summarization models?

(a) BBC Dataset

(b) CNN Dataset

Figure 2: Human evaluation results on the types of hallucinations made by
different models on the TempoSum benchmark.

We illustrate the distribution of hallucinations made by different models on
the BBC and CNN datasets in Figure 2(a) and Figure 2(b), respectively. From
these two figures, we draw the following observations.

##### Pre-training improves the faithfulness of a summarization model on
future data.

We observe that PEGASUS produces significantly fewer hallucinations than its
non-pretrained counterpart in the future test sets of both BBC and CNN
datasets. The results suggest that pre-training helps a summarization model
generates more faithful summaries even if the source articles contain
knowledge conflicts.

##### Pre-training encourages a summarization model to hallucinate outdated
information based on its parametric knowledge.

We observe that on the future test set of BBC, PEGASUS generates substantially
more outdated hallucinations than Transformer (39.81% vs. 33.33%). While on
the future test set of CNN, PEGASUS and Transformer obtain a similar number of
outdated hallucinations (3.94% vs 4.72%). Our human evaluation results reveal
that the parametric world knowledge learned during pre-training harms the
faithfulness performance of PEGASUS on future test sets. Previous studies
mainly consider the benefits of parametric knowledge Cao et al. (2022); Maynez
et al. (2020), but few of them critically study its negative impacts. Our
results indicate that parametric knowledge can be harmful and may guide the
model to generate unfaithful information.

##### The source-target divergence in the finetuning dataset encourages the
model to generate summaries with less reliance on the source.

Source article |  The prime minister was speaking in the Commons two days after announcing the lockdown at a televised press conference… UK recorded 18,950 new confirmed cases of coronavirus and 136 deaths … Boris Johnson is stuck in between two groups…
---|---
Summary by PEGASUS |  Prime Minister David Cameron has defended the government ’s response to the outbreak of the coronavirus, which has killed 136 people in the UK.
Summary by CLIFF |  Prime Minister Boris Johnson has defended the government’s response to the outbreak of coronavirus in the UK.
Figure 3: An example of a BBC news article from 2020, and the summaries
generated by PEGASUS and CLIFF. We highlight the entity mentions of Prime
Minister and Boris Johnson in the article and the generated summaries in red
color. We underline the factual error in the summaries.

Our evaluation results show that models that are fine-tuned on XSum generate
more hallucinations on future data. The reason is that a significant portion
of reference summaries in the XSum dataset have the entity missing problem
Tejaswin et al. (2021) (i.e., reference summaries contain entities that do not
appear in the source text). Thus, models trained on this dataset are
encouraged to hallucinate contents that are not inferable from the source
article.

We observe that models that are fine-tuned on the XSum dataset learn spurious
correlations from the training data. Namely, these models learn to generate
entities that frequently co-occur on the training data, e.g., Prime Minister
and David Cameron. Figure 3 shows an example of a BBC news article published
in 2020. The content of the article implies that Boris Johnson is the prime
minister of the UK. However, PEGASUS generates “Prime Minister David Cameron”
in the summary, which is an outdated hallucination. This example suggests that
PEGASUS memorizes a strong correlation between Prime Minister and David
Cameron from the finetuning dataset. Learning such a spurious correlation is
undesirable since a text summarization model should learn to ground the
generation decision on the facts presented in the source text. Moreover, this
kind of behavior harms the temporal generalization ability of text
summarization since the knowledge of our world evolves constantly.

##### The pre-training process helps generate meaningful and grammatical
summaries.

On the BBC dataset, PEGASUS generates much fewer non-verifiable errors than
Transformer for both the in-distribution test set (21.36% vs. 82.52%) and
future test set (24.07% vs. 67.59%). Similarly, on the CNN dataset, PEGASUS
produces less non-variable error than Transformer for both the in-distribution
test set (7.89% vs. 28.07%) and future test set (4.72% vs. 26.77%). The reason
may lie in that the pre-training process of PEGASUS allows the model to learn
more general linguistic features and time-invariant world knowledge (e.g.,
London is the capital city of the United Kingdom), which can be generalized to
the data from different temporal periods.

##### Summary.

Our experiment results show that fine-tuning datasets plays a crucial role in
the temporal generalization performance of text summarization models. The
divergence between the source document and reference summary encourages models
to heavily rely on their parametric knowledge, limiting their temporal
generalization ability. On the other hand, outdated hallucination is a
critical concern for the temporal generalization ability. It becomes a non-
neglectable factor in the faithfulness performance on the future test set.

###  5.2 Are recent faithfulness enhancement methods and faithfulness
evaluation methods effective on future data?

We first investigate the temporal generalization of two recent faithfulness
enhancement methods, CLIFF and ENT. Then, we evaluate the human correlations
of two popular faithfulness metrics, FactCC and QAFactEval.

####  5.2.1 Performance of the CLIFF method

Figure 2 shows the annotation results of CLIFF on both datasets. We have the
following observations.

##### CLIFF effectively prevents summarization models from relying on their
parametric knowledge on the BBC dataset.

We observe that CLIFF generates significantly fewer hallucinations than
PEGASUS on the future test set of BBC (41.67% vs. 61.11%). The remarkable
faithfulness improvement of CLIFF is attributed to its effectiveness in
encouraging models not to rely on their parametric knowledge. On the future
test set, CLIFF generated fewer summaries containing outdated hallucinations
than PEGASUS (25.00% vs 39.81%). Moreover, CLIFF shows its robustness in
information fusion, Figure 3 shows an example of a BBC article from 2020 and
the summary generated by CLIFF. It successfully fuses the information from the
source article and generates a faithful summary with new knowledge “Prime
Minister Boris Johnson”, while PEGASUS utilizes its parametric knowledge to
generate a summary with outdated hallucination.

##### The effectiveness of CLIFF on future data highly depends on the
finetuning dataset.

Different from the competitive performance observed on the BBC dataset, CLIFF
has negative impacts on the CNN dataset. CLIFF slightly improves the
faithfulness performance on the in-distribution test set, producing fewer
hallucinations than PEGASUS (6.14% vs. 7.89%). CLIFF is ineffective on the
future test set and generates more hallucinations than PEGASUS (18.11% vs.
10.24%). Moreover, the training objective of CLIFF fails to mitigate the
reliance on parametric knowledge and yields more outdated hallucinations than
PEGASUS (6.30% vs. 3.94%).

While CLIFF performs well on the in-distribution test set, its performance on
the future test set is inconsistent across the two datasets in the TempoSum
benchmark. It remains challenging to develop faithfulness enhancement methods
that are generalizable to the future.

####  5.2.2 Performance of the ENT method

Figure 2 shows the annotation results for the ENT method on both datasets. We
draw the following observations.

##### The ENT method fails to improve the faithfulness of summarization models
on future data.

From Figure 2, we observe that ENT and PEGASUS produce a similar amount of
hallucinated content (61.11% vs. 61.11%) on the future test set of the BBC
dataset. Whereas in the future test set of CNN, the ENT model produces more
hallucinations than the PEGASUS model (40.94% vs. 10.24%). Moreover, ENT does
not reduce the tendency of a model to rely on its parametric world knowledge.
In comparison to PEGASUS, ENT produces a higher percentage of outdated
hallucinations on the future test set of BBC (44.44% vs. 39.81%). Whereas in
the future test set of the CNN dataset, ENT produces the same level of
outdated hallucinations as PEGASUS (3.93% vs. 3.93%).

We suspect that the ineffectiveness of ENT is due to the restrictions in its
copy mechanism. The copy mechanism of ENT only copies named entities from the
source text that are identified by a NER tool. However, unnamed entities like
government officials (e.g., the prime minister) often occur in the source text
and reference summaries in our benchmark. This type of unnamed entity cannot
be recognized by NER and is excluded from the copy mechanism of ENT.

Source | Metric | In-distribution | Future
---|---|---|---
BBC | FactCC | -0.03 | 0.138
QAFactEval | 0.406 | 0.317
CNN | FactCC | 0.219 | 0.235
QAFactEval | 0.303 | 0.377
Table 3:  Spearman’s rank correlation between human judgments and faithfulness
metrics in different test sets.

####  5.2.3 Human correlations results of faithfulness evaluation metrics

##### Recent faithfulness evaluation metrics demonstrate weak correlations
with human judgments on future data.

We present the human correlation results of FactCC and QAFactEval on the
TempoSum benchmark in Table 3. Although QAFactEval consistently obtains higher
human correlations than FactCC on the future test sets of CNN and BBC, the
Spearman’s correlation coefficient scores Myers and Sirois (2004) obtain by
these two metrics are lower than 0.4, suggesting that these metrics only have
weak correlations with human judgments. Thus, we still need to develop more
accurate faithfulness evaluation metrics for future data.

####  5.2.4 Summary

Enhancing and evaluating the faithfulness of summarization models to future
data poses a significant challenge. Current faithfulness enhancement and
evaluation methods are typically developed based on observations made from
existing evaluation benchmarks, which may not be generalizable to future data.
Instead, we argue that the use of non-temporal overlap samples for evaluation
purposes can more accurately reflect the generalization capability of
summarization systems and metrics, while also providing a fair comparison of
performance between different faithfulness enhancement and evaluation methods.

##  6 Recommendations

Based on the observations from our experiments, we recommend the following
four takeaways to the summarization research community.

  1. 1.

We should use temporally split test sets that contain knowledge conflicts to
evaluate the performance of summarization systems. This kind of test set
allows us to isolate the effects of parametric world knowledge to accurately
evaluate the temporal generalization performance of text summarization.

  2. 2.

Human evaluations for pre-trained summarization models should annotate the
number of generated summaries that contain outdated hallucinations. This
practice allows us to assess the extent to which pre-trained summarization
models rely on their parametric world knowledge.

  3. 3.

In order to improve the temporal generalization capability of summarization
models, we need to develop techniques to prevent summarization models from
learning spurious correlations among entities during pre-training and fine-
tuning. Our experiments show that spurious correlation is a key factor that
affects the temporal generalization ability of summarization models on future
data.

  4. 4.

Users need to be cautious when using existing automatic metrics to evaluate
the faithfulness of summaries on future data. The community should develop
accurate automatic faithfulness metrics that can generalize well on future
data.

##  7 Conclusion

In this paper, we propose TempoSum, a novel evaluation benchmark for assessing
the temporal generalization capability of text summarization models. We
collect samples from a broad temporal range which allow us to study the
performance of PLMs on data from different temporal periods. Through extensive
experiments on TempoSum, we find that PLMs are vulnerable to generating
summaries with outdated hallucinations and ignore the supporting evidence
provided by source articles. It is crucial to develop techniques to prevent
summarization systems from generating outdated information and design
automatic evaluation metrics to detect outdated hallucinations. TempoSum
provides a valuable data corpus which covers a wide range of temporal periods,
along with human judgements from various summarization systems, to support
future studies in this direction.

## Limitations

Our study of temporal generalization is limited to news domain datasets. In
the future, we still need empirical studies for the temporal generalization
abilities of text summarization models on other domains (e.g., dialog and
scientific domains). Moreover, our analysis of knowledge conflicts mainly
focuses on entity relationships with politicians. We can consider other
categories of entities in future work. Furthermore, as most of the existing
PLMs (e.g., PEGASUS, BART, T5, etc.) are pre-trained on corpora before 2019,
TempoSum is designed to evaluate the temporal generalization ability of PLMs
that were pre-trained on corpora before 2019. To access the temporal
generalization capability of PLMs trained on more recent data (e.g., data from
2019 to 2021), we need to construct an evaluation benchmark that includes data
from future periods (e.g, data after 2021).

## Ethics Statement

This work proposes a new benchmark for accessing the temporal generalization
ability of text summarization models. We only collect news articles from
creditable news agencies to avoid including offensive content in our
benchmark. The news articles used in our benchmark are publicly available on
the internet, and therefore do not reveal any private information about
individuals.

## Acknowledgements

We would like to thank the anonymous reviewers and meta-reviewer for their
valuable feedback on this work. This work was supported in part by the Science
and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ,
FDCT/0070/2022/AMJ) and the Multi-year Research Grant from the University of
Macau (Grant No. MYRG2020-00054-FST). This work was performed in part at SICC
which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University
of Macau.

## References

  * Agarwal and Nenkova (2022) Oshin Agarwal and Ani Nenkova. 2022.  Temporal effects on pre-trained models for language processing tasks.  _Trans. Assoc. Comput. Linguistics_ , 10:904–921.
  * Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2022.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 3340–3354. Association for Computational Linguistics.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 6633–6649. Association for Computational Linguistics.
  * Chen et al. (2021) Wang Chen, Piji Li, Hou Pong Chan, and Irwin King. 2021.  Dialogue summarization with supporting utterance flow modelling and fact regularization.  _Knowl. Based Syst._ , 229:107328.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers)_ , pages 615–621. Association for Computational Linguistics.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022.  Time-aware language models as temporal knowledge bases.  _Trans. Assoc. Comput. Linguistics_ , 10:257–273.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021.  Summeval: Re-evaluating summarization evaluation.  _Trans. Assoc. Comput. Linguistics_ , 9:391–409.
  * Fabbri et al. (2019) Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019\.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 1074–1084. Association for Computational Linguistics.
  * Fabbri et al. (2022) Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.  Qafacteval: Improved qa-based factual consistency evaluation for summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 2587–2601. Association for Computational Linguistics.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 1449–1462. Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of GPT-3.  _CoRR_ , abs/2209.12356.
  * Grangier and Iter (2022) David Grangier and Dan Iter. 2022.  The trade-offs of domain adaptation for neural language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 3802–3813. Association for Computational Linguistics.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 708–719. Association for Computational Linguistics.
  * Guo and Yu (2022) Xu Guo and Han Yu. 2022.  On the domain adaptation and generalization of pretrained language models: A survey.  _CoRR_ , abs/2211.03154.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Huang et al. (2020) Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020.  What have we achieved on text summarization? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 446–469. Association for Computational Linguistics.
  * Kornilova and Eidelman (2019) Anastassia Kornilova and Vladimir Eidelman. 2019.  BillSum: A corpus for automatic summarization of US legislation.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 9332–9346. Association for Computational Linguistics.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  _Advances in Neural Information Processing Systems_ , 34:29348–29363.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7052–7063. Association for Computational Linguistics.
  * Loureiro et al. (2022) Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and José Camacho-Collados. 2022.  Timelms: Diachronic language models from twitter.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, May 22-27, 2022_ , pages 251–260. Association for Computational Linguistics.
  * Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020.  Multi-xscience: A large-scale dataset for extreme multi-document summarization of scientific articles.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 8068–8074. Association for Computational Linguistics.
  * Luu et al. (2022) Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022.  Time waits for no one! analysis and challenges of temporal misalignment.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 5944–5958. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 1906–1919. Association for Computational Linguistics.
  * Myers and Sirois (2004) Leann Myers and Maria J Sirois. 2004.  Spearman correlation coefficients, differences between.  _Encyclopedia of statistical sciences_ , 12.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018\.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016_ , pages 280–290. ACL.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 1797–1807. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Reddy et al. (2022) Revanth Gangi Reddy, Heba Elfardy, Hou Pong Chan, Kevin Small, and Heng Ji. 2022\.  Sumren: Summarizing reported speech about events in news.  _CoRR_ , abs/2212.01146.
  * Rosin et al. (2022) Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022.  Time masking for temporal language models.  In _WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 - 25, 2022_ , pages 833–841. ACM.
  * Röttger and Pierrehumbert (2021) Paul Röttger and Janet B. Pierrehumbert. 2021.  Temporal adaptation of BERT and performance on downstream document classification: Insights from social media.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 2400–2412. Association for Computational Linguistics.
  * Tejaswin et al. (2021) Priyam Tejaswin, Dhruv Naik, and Pengfei Liu. 2021.  How well do you know your summarization datasets? In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3436–3449, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 5008–5020. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.
  * Xiao and Carenini (2022) Wen Xiao and Giuseppe Carenini. 2022.  Entity-based spancopy for abstractive summarization to improve the factual consistency.  _CoRR_ , abs/2209.03479.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  PEGASUS: pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhu et al. (2021) Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021.  Mediasum: A large-scale media interview dataset for dialogue summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 5927–5934. Association for Computational Linguistics.

##  Appendix A Human Evaluation Instruction

We present the instruction of our human evaluation experiments in Figure 4.

In this study, we investigate the faithfulness performance of summarization
models for data from different temporal periods.
---
You are required to identify the faithfulness errors in the generated
summaries by using the error typologies described below.
Factual A summary contains a text span that is not covered by the source input
but is consistent with world knowledge at the time of the article being
published.
Outdated A summary contains a text span that is not covered by the source
input but is consistent with world knowledge before the time the article is
published.
Non-verifiable A summary contains a text span that is not covered by the
source input and cannot be verified by any time of world knowledge.
Please follow the instruction shown below to identify the faithfulness of the
generated summaries.
Step 1 Read the source document and the generated summary carefully.
Step 2 Check if all the information within the summaries can be directly
entailed by the source article. If yes, mark the sample as faithful. If no, go
to the Step 3.
Step 3 If any summary contains entities that are not entailed by the source.
Please verify it by using Wikipedia with the knowledge of when the article was
published. If it is consistent with the knowledge of the time the article was
published, mark this entity as factual. If not, but is consistent with the
knowledge before the article was published, mark this entity as outdated. If
this entity cannot be verified by any time of knowledge, mark this entity as
non-verifiable.
Step 4 If the summary is not sensical or ungrammatical, please mark the sample
as non-verifiable.
Figure 4:  Our human annotation instruction.

##  Appendix B Automatic Evaluation Results

Table 4 shows the results of FactCC and QAFactEval scores obtained by
different models. We observe that CLIFF outperforms all other methods on all
the test sets in terms of QAFactEval scores. On the other hand, ENT attains
the best FactCC scores on the BBC datasets, while CLIFF achieves the highest
FactCC scores on the CNN dataset.

##  Appendix C Scientific Artifacts

Our proposed TempoSum benchmark consists of two English news summarization
datasets. All the data is collected from the Internet
Archive555https://archive.org/. The licenses of the original news sources are
applied.

##  Appendix D Implementation Details

In this section, we describe the checkpoints that we use for our experiments.

##### PEGASUS:

We use “google/pegasus-large-xsum” and “google/pegasus-large-cnn_dailymail”
checkpoints provided by Wolf et al. (2019) in our experiments.

##### CLIFF:

We use the Pegasus+CLIFF checkpoints from the official code repository of
CLIFF Cao and Wang (2021).

##### ENT:

We use the source codes provided by the authors to fine-tune PEGASUS. The
authors find that the proposed mechanism is performs better on a filtered
dataset (i.e., all the entities in the reference summaries are present in the
source article). We follow the authors by using the filtered version of XSum
and CNN/DM to fine-tune PEGASUS.

##### Transformer:

We use the transformer-base model from Huggingface Wolf et al. (2019).

Data | Test set | Model | FactCC | QAFactEval
---|---|---|---|---
BBC | In-dist. | PEGASUS | 14.44 | 27.99
Trans. | 17.97 | 3.99
CLIFF | 15.40 | 30.38
ENT | 17.99 | 20.16
Future | PEGASUS | 11.24 | 24.97
Trans. | 15.45 | 2.28
CLIFF | 12.04 | 29.12
ENT | 16.86 | 18.87
CNN | In-dist. | PEGASUS | 50.57 | 73.86
Trans. | 51.70 | 71.69
CLIFF | 57.11 | 78.41
ENT | 33.02 | 61.39
Future | PEGASUS | 61.64 | 77.64
Trans. | 54.54 | 73.44
CLIFF | 63.42 | 80.44
ENT | 35.83 | 65.23
Table 4:  FactCC and QAFactEval scores obtained by different summarization
systems on the TempoSum benchmark.