# Scattering mechanisms in state-of-the-art GaAs/AlGaAs quantum wells

Yi Huang (黄奕<EMAIL_ADDRESS>B. I. Shklovskii  M. A. Zudov  School of
Physics and Astronomy, University of Minnesota, Minneapolis, Minnesota 55455,
USA

(August 28, 2024)

###### Abstract

Motivated by recent breakthrough in molecular beam epitaxy of GaAs/AlGaAs
quantum wells [Y. J. Chung et al., Nature Materials 20, 632 (2021)], we
examine contributions to mobility and quantum mobility from various scattering
mechanisms and their dependencies on the electron density. We find that at
lower electron densities, $n_{e}\lesssim 1\times 10^{11}$ cm-2, both transport
and quantum mobility are limited by unintentional background impurities and
follow a power law dependence, $\propto n_{e}^{\alpha}$, with $\alpha\approx
0.85$. Our predictions for quantum mobility are in reasonable agreement with
an estimate obtained from the resistivity at filling factor $\nu=1/2$ in a
sample of Y. J. Chung et al. with $n_{e}=1\times 10^{11}$ cm-2. Consideration
of other scattering mechanisms indicates that interface roughness (remote
donors) is a likely limiting factor of transport (quantum) mobility at higher
electron densities. Future measurements of quantum mobility should yield
information on the distribution of background impurities in GaAs and AlGaAs.

Advances in molecular beam epitaxy, particularly the purification of source
materials and improved vacuum conditions, have recently lead to a new
generation of GaAs/AlGaAs quantum wells Chung  _et al._ (2021) in which the
mobility reached a record value of $\mu=4.4\times 10^{7}$ cm2V-1s-1 at
electron density $n_{e}=2.0\times 10^{11}$ cm-2. The increase in mobility was
especially pronounced at lower densities ($n_{e}\lesssim 1\times 10^{11}$
cm-2), where $\mu$ was twice the previously reported record values. At higher
densities ($n_{e}\gtrsim 2\times 10^{11}$ cm-2), however, the mobility has
decreased to $\mu\approx 3.5\times 10^{7}$ cm2V-1s-1 at $n_{e}\approx
2.7\times 10^{11}$ cm-2, approaching previously reported values. While the
increase of $\mu$ at low $n_{e}$ could be readily attributed to a reduced
concentration of unintentional background impurities, subsequent reduction of
$\mu$ at higher $n_{e}$ calls for revisiting other scattering sources.

In this Letter, we theoretically examine both transport and quantum mobility
($\mu_{\rm q}$) considering scattering by background impurities (BI), remote
ionized donors in the doping layers (RI), interface roughness (IR), and alloy
disorder (AD). We find that both $\mu$ and $\mu_{\rm q}$ are limited by BI
scattering at low $n_{e}$, as expected. With increasing $n_{e}$, however,
scattering on IR (and eventually on AD) becomes important, as far as $\mu$ is
concerned, whereas $\mu_{\rm q}$ becomes limited by RI scattering (hwa, ).

Modern GaAs-based heterostructures, such as those reported in Ref. Chung  _et
al._ , 2021, consist of a GaAs quantum well of width $w$ surrounded by
AlxGa1-xAs barriers of thickness $d$. A two-dimensional electron gas (2DEG)
with a concentration $n_{e}$ is supplied by two remote doping layers, each
located at a distance $d_{w}=d+w/2$ from the center of the GaAs quantum well.
These layers have a sophisticated doping well design, which helps to
substantially reduce the scattering by ionized donors owing to excess electron
screening Sammon  _et al._ (2018a); Chung  _et al._ (2020); Akiho and Muraki
(2021, 2022). In this design, a $\delta-$layer of silicon atoms with
concentration $n\gg n_{e}$ is implanted into a narrow ($\approx 3$ nm) GaAs
quantum well, which is sandwiched between thin ($\approx 2$ nm) AlAs layers dw
(2). In our calculations we use $n\approx 1.5\times 10^{12}$ cm-2, Al mole
fraction $x=0.12$, and take into account the electron density dependencies of
the effective spacer thickness $d_{w}$ and of the quantum well width $w$. More
specifically, we use $k_{F}w=3.9$, where $k_{F}=\sqrt{2\pi n_{e}}$ is the
Fermi wave number, and $d_{w}^{-1}=an_{e}$, where $a=3.55$ nm (wdw, ). These
constraints were obtained by fitting samples parameters of Ref. Chung  _et
al._ , 2021, as detailed in the Supplemental Material (not, ).

We start with transport mobility, $\mu=e\tau/m^{\star}$, where
$m^{\star}=0.067\,m_{0}$ is the effective mass of an electron in GaAs (mas, ),
$m_{0}$ is the free electron mass,

$\displaystyle\frac{1}{\tau}=\frac{4m^{\star}}{\pi\hbar^{3}}\int\limits_{0}^{2k_{F}}\frac{dq}{\sqrt{4k_{F}^{2}-q^{2}}}\quantity(\frac{q}{2k_{F}})^{2}\expectationvalue{\absolutevalue{U(q)}^{2}}\,,$
(1)

is the transport scattering rate, and $U(q)$ is the screened potential of a
given scattering source.

_Coulomb background impurities._ The screened potential squared averaged over
impurity positions is given by

$\displaystyle\expectationvalue{\absolutevalue{U_{\rm
BI}(q)}^{2}}=\int\limits_{-\infty}^{+\infty}dz\;N(z)U_{1}^{2}(q,z)\,,$ (2)

where $U_{1}^{2}(q,z)$ is the screened potential squared from a single Coulomb
impurity, and $N(z)$ is the 3D concentration of impurities at a distance $z$
from the center of the 2DEG. Since the main contribution to momentum
relaxation comes from impurities located close to the quantum well, for which
excess electron screening (Sammon  _et al._ , 2018a) is not important and
$U_{1}(q,z)$ can be obtained taking into account screening by electrons in the
quantum well only. Using Thomas-Fermi (TF) approximation (Sammon  _et al._ ,
2018a), and following Refs. Jonson, 1976; Gold, 1986, 1987, 1989, 2013 , we
can write (more detailed discussion can be found in the Supplemental Material
(not, ))

$\displaystyle U_{1}(q,z)=\frac{2\pi e^{2}}{\kappa
q\epsilon(q)}\int\limits_{-\infty}^{+\infty}dz^{\prime}\;\absolutevalue{\psi(z^{\prime})}^{2}e^{-q\absolutevalue{z-z^{\prime}}}\,,$
(3)

where $\psi(z)$ is the wave function along $z$ direction and the dielectric
function is given by

$\displaystyle\epsilon(q)=1+(q_{\rm TF}/q)F_{c}(qw)[1-G(q)]\,.$ (4)

Here, $q_{\rm TF}=2/a_{B}$, $a_{B}=\kappa\hbar^{2}/m^{\star}e^{2}$ is the
effective Bohr radius, $\kappa=12.9$ is the dielectric constant of GaAs,
$G(q)=q/(2\sqrt{q^{2}+k_{F}^{2}})$ is the local field correction using Hubbard
approximation Jonson (1976), and the form factor $F_{c}(qw)$ is given by

$F_{c}(qw)=\iint\limits_{-\infty}^{+\infty}dzdz^{\prime}\absolutevalue{\psi(z)}^{2}\absolutevalue{\psi(z^{\prime})}^{2}\exp(-q\absolutevalue{z-z^{\prime}})\,.$
(5)

For small concentration $n_{e}<1\times 10^{11}$ cm-2, it suffices to use an
infinite-potential-well approximation and
$\psi(z)=\sqrt{2/w}\cos{(z\pi/w)}\Theta(w/2-\absolutevalue{z})$ in both Eq.
(3) and Eq. (5).

Following Ref. Sammon  _et al._ , 2018a, we define BI density as

$\displaystyle N(z)=\begin{cases}N_{1},\;&w/2<\absolutevalue{z}<d_{w}\,,\\\
N_{2},\;&\absolutevalue{z}<w/2\,,\end{cases}$ (6)

where $N_{1}$ ($N_{2}$) represents the 3D concentration of impurities in
AlGaAs (GaAs). The results of our calculations for uniform impurity
distribution ($N_{1}=N_{2}$) and for no impurities in GaAs ($N_{2}=0$) are
shown in Fig. 1 along with the experimental data of Ref. Chung  _et al._ ,
2021 (circles). We find that the data at $n_{e}\lesssim 1\times 10^{11}$ cm-2
can be better described by $N_{1}=1.4\times 10^{14}$ cm-3 and $N_{2}=0$ (solid
line). Assuming uniform distribution of impurities (dashed curve) yields
$N_{1}=N_{2}=5\times 10^{12}$ cm-3, which is close to an estimate $1\times
10^{13}$ cm-3 of Ref. Chung  _et al._ , 2021 111This uniform impurities
concentration is also close to a recent estimate found in Ref. Ahn and Das
Sarma, 2022. However, they used fixed quantum well width $w=30$ nm in the
calculations, while we take into account the density dependence such that
$k_{F}w=3.9$.. We also note that $\mu_{\rm BI}(N_{2}=0)$ is well described by
$\mu_{\rm BI}=38.3\,n_{e}^{\alpha}$, where $\alpha\approx 0.85$, whereas
$\mu_{\rm BI}(N_{1}=N_{2})$ follows $\mu_{\rm BI}=43.1\,n_{e}^{\alpha}$, where
$\alpha\approx 1.12$. Here and in what follows we assume that the electron
density is in units of $10^{11}$ cm-2 and the mobility is in units of $10^{6}$
cm2V-1s-1.

While BI scattering can describe the experimental data reasonably well at
$n_{e}\leq 1$, it is clear that BI scattering alone cannot explain
experimental $\mu$ at higher $n_{e}$. Indeed, obtained power laws represent
crossovers from $\mu_{\rm BI}\propto n_{e}^{1/2}$ at low $n_{e}$ Sammon  _et
al._ (2018a) to $\mu_{\rm BI}\propto n_{e}^{3/2}$ at higher $n_{e}$ Dmitriev
_et al._ (2012). To explain the experimental $\mu$ at higher $n_{e}$, one
needs to examine other scattering sources.

_Remote impurities._ One obvious candidate for reduced $\mu$ at higher $n_{e}$
is remote ionized impurity scattering. For electron densities $n_{e}<10$ and
for a given doping concentration $n=1.5\times 10^{12}$ cm-2, the fraction of
ionized donors in each doping layer is small, $1-f=n_{e}/2n<0.5$, and we can
use Eq. (22) from Ref. Sammon  _et al._ , 2018a rem ,

$\displaystyle\mu_{\rm RI}$
$\displaystyle=7.7\,\frac{e}{\hbar}\frac{nk_{F}^{3}d_{w}^{5}}{n_{e}}=1.6\times
10^{6}\,n_{e}^{-9/2}\,,$ (7)

where in the final expression of Eq. (7) we used $n=15$, $k_{F}=\sqrt{2\pi
n_{e}}$, and $d_{w}^{-1}=an_{e}$ wdw . Equation (7) gives $\mu_{\rm RI}\sim
10^{4}$ at $n_{e}=3$, which is more than 300 times larger than experimental
$\mu$, so the RI scattering is still irrelevant in this regime. We note,
however, that additional mechanisms of disorder in the doping layers may lead
to an increase of RI scattering, as discussed in Sec. V of Ref. Sammon  _et
al._ , 2018a, although quantitative understanding of these mechanisms is still
lacking.

Figure 1: Mobility $\mu$ versus electron density $n_{e}$. Circles are
experimental data from Ref. Chung  _et al._ , 2021. Solid (dashed) line
represents $\mu_{\rm BI}$ calculated for $N_{1}=1.4\times 10^{14}$ cm-3,
$N_{2}=0$ ($N_{1}=N_{2}=5\times 10^{12}$ cm-3).

_Interface roughness._ Scattering on interface roughness originates from
fluctuations of the ground state energy due to $1-2$ monolayer local
variations of the quantum well width $w$. For the infinite barrier height
($V\rightarrow\infty$), the corresponding transport scattering rate
$\tau^{-1}_{\rm IR}\propto w^{-6}$ Gold (1986, 1987) and such dependence was
confirmed in narrow $w<10$ nm GaAs quantum wells with AlAs barriers Sakaki
_et al._ (1987); Kamburov  _et al._ (2016). However, in AlxGa1-xAs/GaAs
quantum wells the barrier height is significantly reduced ($V\approx 0.75x$ eV
for $x<0.45$), and fluctuations of the ground state energy are diminished due
to finite penetration of the electron wave function into the barrier Gold
(1989); Li  _et al._ (2005). As a result, $\tau^{-1}_{\rm IR}$ is considerably
suppressed compared to the case $V\rightarrow\infty$, and its dependence on
$w$ weakens Gottinger  _et al._ (1988); Li  _et al._ (2005); Luhman  _et al._
(2007); Kamburov  _et al._ (2016).

We calculate the IR-limited scattering rates following the approach of Refs.
Gold, 1989; Li  _et al._ , 2005. We use the barrier height for $x=0.12$ and
take into account the difference in the electron effective mass in the GaAs
well ($m^{\star}=0.067\,m_{0}$) and in the Al0.12Ga0.88As barriers
($m_{B}=0.067+0.083x=0.077\,m_{0}$). At the end we also impose a constraint
$k_{F}w=3.9$ (wdw, ).

Using the correlator of local well width variations
$\expectationvalue{\Delta(\mathbf{r})\Delta(\mathbf{r}^{\prime})}=\Delta^{2}\exp(-\absolutevalue{\mathbf{r}-\mathbf{r}^{\prime}}^{2}/\Lambda^{2})$,
where $\Delta$ is the roughness height and $\Lambda$ is the roughness lateral
size, the scattering potential due to interface roughness is given by

$\displaystyle\expectationvalue{\absolutevalue{U_{\rm
IR}(q)}^{2}}=\frac{\pi}{\epsilon^{2}(q)}\Delta^{2}\Lambda^{2}\quantity(\partialderivative{E}{w})^{2}e^{-q^{2}\Lambda^{2}/4}\,,$
(8)

where $E$ is the ground state energy for the finite potential well. Here, the
dielectric function $\epsilon(q)$, see Eq. (4), is computed with the form
factor $F_{c}(qw)$, see Eq. (5), using finite-potential-well wave function.
(See Supplemental Material for details (not, )).

By substituting Eq. (8) into Eq. (1) we obtain the mobility due to interface
roughness $\mu_{\rm IR}$ as shown in Fig. 2. In order to reproduce the
experimental data at $n_{e}>2$, we arrived at $\Delta=2.83$ Å and $\Lambda=80$
Å. With these roughness parameters, $\mu_{\rm IR}$ becomes equal to $\mu_{\rm
BI}(N_{2}=0)$ at $n_{e}\approx 2.4$ and, in the vicinity of this density, can
be described by $\mu_{\rm IR}\simeq 4.7\times 10^{2}\,n_{e}^{-2}$. We have
also examined the effect of $x$ on the mobility limited by IR. By raising $x$
from 0.12 to 0.24, a value which is common for previous generation of samples,
$\mu_{\rm IR}$ at $n_{e}=3$ becomes smaller by 24 %, although this effect
weakens at lower $n_{e}$.

Figure 2:  Mobility $\mu$ versus electron density $n_{e}$. Circles are
experimental data from Ref. Chung  _et al._ , 2021. Thin lines marked by BI,
IR, AD, and RI represent $\mu_{\rm BI}$ ($N_{1}=1.4\times 10^{14}$ cm-3,
$N_{2}=0$), $\mu_{\rm IR}$ ($\Delta=0.283$ nm, $\Lambda=8$ nm, $x=0.12$),
$\mu_{\rm AD}$ ($x=0.12)$, and $\mu_{\rm RI}$ [Eq. (7)], respectively. Thick
line represents $\mu$ limited by all contributions.  Figure 3: Quantum
mobility $\mu_{\rm q}$ versus electron density $n_{e}$. Thin solid lines
marked by BI, IR, AD, and RI represent $\mu_{\rm q,BI}$ ($N_{1}=1.4\times
10^{14}$ cm-3, $N_{2}=0$), $\mu_{\rm q,IR}$ ($\Delta=0.283$ nm, $\Lambda=8$
nm, $x=0.12$), $\mu_{\rm q,AD}$ ($x=0.12)$, and $\mu_{\rm q,RI}$ (Eq. (11)),
respectively. Thick solid line represents $\mu_{\rm q}$ limited by all
contributions. Thick dashed line is $\mu_{\rm q}$ limited by all contributions
but with $\mu_{\rm q,BI}$ computed for $N_{1}=N_{2}=5\times 10^{12}$ cm-3.
Solid circle shows quantum mobility obtained from the resistivity at filling
factor $\nu=1/2$ measured in Ref. Chung  _et al._ , 2021.

_Alloy disorder._ Alloy disorder (AD) scatters electrons due to the wave
function tails extending into the AlxGa1-xAs barriers. Usually, in typical
high mobility samples, this scattering mechanism is deemed irrelevant, see
e.g., Ref. Li  _et al._ , 2005; Umansky and Heiblum, 2013. However, in light
of lower $x$ used in the new generation of samples (Chung  _et al._ , 2021),
it is important to revisit this scattering source. Following Refs. Ando, 1982;
Li  _et al._ , 2005; Gold, 2013, we write the scattering potential as

$\expectationvalue{\absolutevalue{U_{\rm AD}(q)}^{2}}=\frac{(\Delta
E_{c})^{2}}{\epsilon^{2}(q)}\Omega
x(1-x)\int\limits_{\absolutevalue{z}>w/2}\absolutevalue{\psi(z)}^{4}dz,$ (9)

where $\Omega=a^{3}/4$, $a=5.67$ $\mathrm{\SIUnitSymbolAngstrom}$ is the
lattice constant, and $\Delta E_{c}\simeq 1$ eV is the conduction band
discontinuity at the $\Gamma$-point of GaAs/AlAs interface. As for the case of
IR, we use finite-potential-well wave function to calculate the form factor in
$\epsilon(q)$ (see Supplementary Material (not, )). Substituting Eq. (9) into
Eq. (1) and applying the constraint $k_{F}w=3.9$ (wdw, ), we obtain the
mobility limited by AD scattering $\mu_{\rm AD}$ as shown in Fig. 2. We note
that $\mu_{\rm AD}$ can be well described by $\mu_{\rm AD}\simeq 7.1\times
10^{3}\,n_{e}^{-3}$ and that it becomes equal to $\mu_{\rm BI}$ at
$n_{e}\approx 3.7$ and to $\mu_{\rm IR}$ at $n_{e}\approx 7$. We thus conclude
that alloy disorder scattering cannot be ignored at higher densities. We have
also examined the effect of $x$ on mobility limited by alloy disorder. By
increasing $x$ from 0.12 to 0.24, $\mu_{\rm AD}$ increases by a factor of
$2.4$ for $n_{e}<10$.

We now turn to the effects of different scattering mechanisms on quantum
mobility $\mu_{\rm q}=e\tau_{\rm q}/m^{\star}$, where the quantum scattering
rate is given by

$\displaystyle\frac{1}{\tau_{\rm
q}}=\frac{2m^{\star}}{\pi\hbar^{3}}\int\limits_{0}^{2k_{F}}\frac{dq}{\sqrt{4k_{F}^{2}-q^{2}}}\,\expectationvalue{\absolutevalue{U(q)}^{2}}.$
(10)

_Background impurities._ Since background impurities far away from the quantum
well contributes to $\mu_{\rm q}$ significantly, excess electron screening
cannot be ignored and the scattering potential is no longer given by Eq. (3).
As a very good approximation we can think of a perfect screening, such that
the excess electron screening length is zero. However, the expression of
scattering potential $U(q)$ in this approximation is still cumbersome so we
leave it to Supplemental Material (not, ) as a reference for an interested
reader. As discussed above, two different impurity distributions, $N_{2}=0$
and $N_{1}=N_{2}$, can describe the experimental $\mu$ reasonably well at
$n_{e}<1$. However, these distributions give very different values for
$\mu_{\rm q}$, as shown in Fig. 3. Indeed, we find that $\mu_{\rm
q,BI}(N_{2}=0)$ (solid curve marked “BI”) is an order of magnitude smaller
than $\mu_{\rm q,BI}(N_{1}=N_{2})$ (dashed curve marked “BI”). More
specifically, we find that $\mu_{\rm q,BI}(N_{2}=0)\simeq 1.4\,n_{e}^{0.85}$,
whereas $\mu_{\rm q,BI}(N_{1}=N_{2})\simeq 13.3\,n_{e}^{0.87}$. As a result,
future experiments measuring quantum mobility should be able to distinguish
between these two distributions.

_Remote donors._ For quantum mobility limited by RI scattering, we use Eq.
(23) from Ref. Sammon  _et al._ , 2018a valid at $1-f=n_{e}/2n<0.5$ for
$n_{e}<10$ rem :

$\displaystyle\mu_{\rm q,RI}$
$\displaystyle=6.5\,\frac{e}{\hbar}\frac{nk_{F}d_{w}^{3}}{n_{e}}=2.6\times
10^{3}\,n_{e}^{-7/2}\,,$ (11)

where in the final expression we used $n=15$, $k_{F}=\sqrt{2\pi n_{e}}$, and
$d_{w}^{-1}=an_{e}$ wdw . Even though Eq. (11) yields very large $\mu_{\rm
q,RI}\simeq 56$ at $n_{e}=3$, RI scattering is still expected to limit
$\mu_{\rm q}$ at higher $n_{e}$, see Fig. 3.

_Interface roughness and alloy disorder._ Contributions from IR and AD can be
calculated using Eqs. (8), (9), and (10). In particular we find that $\mu_{\rm
q,IR}\simeq 1.2\mu_{\rm IR}$ and $\mu_{\rm q,AD}\simeq 1.3\mu_{\rm AD}$. As
shown in Fig. 3, these contributions are considerably smaller than the RI
contribution at all relevant $n_{e}$. As a result, one expects a crossover
from BI-limited to RI-limited quantum mobility for either model of BI
distribution. The value and the position of the maximum at the quantum
mobility crossover should yield information on the BI distribution.

Next, we comment on the relation between the quantum mobility and the mobility
of composite fermions $\mu_{\rm CF}$ at filling factor $\nu=1/2$. By comparing
the expression of the longitudinal resistivity at $\nu=1/2$, Eq. (5.11) of
Ref. Halperin  _et al._ , 1993,

$\displaystyle\rho_{1/2}=\frac{n_{i}}{n_{e}}\frac{1}{k_{F}z}\frac{2\sqrt{2}\pi\hbar}{e^{2}},$
(12)

and the expression for quantum mobility, Eq. (6) in Ref. Sammon  _et al._ ,
2018a,

$\displaystyle\mu_{\rm q}=\frac{2e}{\pi\hbar}\frac{k_{F}z}{n_{i}},$ (13)

one can conclude that

$\displaystyle\mu_{\rm q}=\frac{4\sqrt{2}}{en_{e}\rho_{1/2}},$ (14)

which implies that $\mu_{\rm q}=4\sqrt{2}\mu_{\rm CF}$. Here, $n_{i}$ is the
2D concentration of random impurities in a thin layer at a distance $z$ away
from the center of the quantum well. In order to obtain $\rho_{1/2}$ and
$\mu_{\rm q}^{-1}$ for impurities with 3D concentration $N(z)$, one should
replace $n_{i}$ by $N(z)dz$ and integrate over $z$ 222To avoid the divergence
at $z=0$, one can average $z$ in the denominator using
$\expectationvalue{z}=\int
dz^{\prime}\absolutevalue{\psi(z^{\prime})}^{2}\absolutevalue{z-z^{\prime}}$,
where $\psi(z)$ is the wave function along $z$ direction.. This integration
does not change the relation Eq. (14) between $\mu_{q}$ and $\rho_{1/2}$, so
that it holds for both BI and RI scattering 333On the other hand, AD and IR do
not change the 2D electron concentration inside the quantum well, so they do
not contribute to the gauge field fluctuations and $\rho_{1/2}$..

From the experimental data in Ref. Chung  _et al._ , 2021, the longitudinal
resistance is $R_{1/2}=35$ $\mathrm{\SIUnitSymbolOhm}$ at $\nu=1/2$ and
$n_{e}=1$. Using the geometry of experiments in Ref. Chung  _et al._ , 2021,
we estimate $\rho_{1/2}\approx 2.5R_{1/2}$. As a result, the quantum mobility
at $n_{e}=1$ is estimated as $\mu_{\rm q}=4.0$. This data point, filled circle
in Fig. 3, falls in between our predictions for $N_{2}=0$ and $N_{1}=N_{2}$.
In order to fit this point, while keeping $\mu$ the same as shown in Figs. 1
and 2, we need $N_{1}\approx 4\times 10^{13}$ cm-3 and $N_{2}\approx 4\times
10^{12}$ cm-3.

Finally, we mention that hydrodynamics (Alekseev, 2016) or scattering on oval
defects (Bockhorn  _et al._ , 2014) might affect the zero-field resistivity
and, as a result, the inferred mobility. Both mechanisms manifest as negative
magnetoresistance in weak magnetic field (Dai  _et al._ , 2010; Hatke  _et
al._ , 2011; Bockhorn  _et al._ , 2011; Hatke  _et al._ , 2012; Shi  _et al._
, 2014a, b; Bockhorn  _et al._ , 2014) which can also be seen in Fig. 3(b) of
Ref. Chung  _et al._ , 2021. However, since no experimental studies of this
magnetoresistance are yet available, we cannot comment on its origin.

In summary, we have examined roles of different scattering sources on the
transport and quantum mobilities in the new generation of ultrahigh mobility
GaAs/Al0.12As0.88 quantum wells (Chung  _et al._ , 2021). While at lower
electron densities both mobilities are limited by background impurity
scattering, interface roughness (remote impurity) scattering is the likely
source limiting transport (quantum) mobility at large electron densities. Our
predictions for quantum mobility are in agreement with the value estimated
from the mobility of the composite fermions at filling factor $\nu=1/2$ in a
sample with $n_{e}=1\times 10^{11}$ cm-2 Chung  _et al._ (2021). Future
measurements of quantum mobility should provide insight on the distribution of
background impurities in the GaAs quantum well and in the AlGaAs barriers.

###### Acknowledgements.

We thank L. N. Pfeiffer and M. Shayegan for discussions and clarifying the
experimental details. Y.H. was partially supported by the William I. Fine
Theoretical Physics Institute. M.A.Z. acknowledges support by the U.S.
Department of Energy, Office of Science, Basic Energy Sciences, under Award
No. ER 46640-SC0002567.

## References

  * Chung  _et al._ (2021) Y. J. Chung, K. A. Villegas Rosales, K. W. Baldwin, P. T. Madathil, K. W. West, M. Shayegan, and L. N. Pfeiffer, Nature Materials 20, 632 (2021).
  * (2) We note that both AD and IR scattering were not considered to be limiting factors in theories considering structures with mobilities even higher than reported in Ref. Chung  _et al._ (2021) (see, e.g., Ref. Hwang and Das Sarma, 2008).
  * Sammon  _et al._ (2018a) M. Sammon, M. A. Zudov, and B. I. Shklovskii, Phys. Rev. Materials 2, 064604 (2018a).
  * Chung  _et al._ (2020) Y. J. Chung, K. A. Villegas Rosales, K. W. Baldwin, K. W. West, M. Shayegan, and L. N. Pfeiffer, Phys. Rev. Materials 4, 044003 (2020).
  * Akiho and Muraki (2021) T. Akiho and K. Muraki, Phys. Rev. Applied 15, 024003 (2021).
  * Akiho and Muraki (2022) T. Akiho and K. Muraki, arXiv:2203.14575 (2022).
  * dw (2) To our knowledge, such design, which is a special case of a short-period GaAs/AlAs superlattice Baba  _et al._ (1983); Friedland  _et al._ (1996), was first realized in single heterointerfaces doped on one side in $1990$’s Du  _et al._ (1993); lor .
  * (8) The condition $k_{F}w=3.9$ defines maximum electron density at which the second subband of the GaAs quantum well remains unoccupied, wheres the dependence $d_{w}^{-1}=an_{e}$, where $a=3.55$ nm, is governed by electrostatics. These conditions were obtained from the fits to experimental data Chung  _et al._ (2021), see Supplemental Material (not, ).
  * (9) See Supplemental Material at [URL].
  * (10) In our calculations we ignore the density dependence of the effective mass, which is rather weak in the considered density range (Coleridge  _et al._ , 1996; Tan  _et al._ , 2005; Hatke  _et al._ , 2013; Shchepetilnikov  _et al._ , 2017; Fu  _et al._ , 2017).
  * Jonson (1976) M. Jonson, J. of Phys. C: Solid State Phys. 9, 3055 (1976).
  * Gold (1986) A. Gold, Solid State Commun. 60, 531 (1986).
  * Gold (1987) A. Gold, Phys. Rev. B 35, 723 (1987).
  * Gold (1989) A. Gold, Zeitschrift für Physik B Condensed Matter 74, 53 (1989).
  * Gold (2013) A. Gold, JETP Lett. 98, 416 (2013).
  * Note (1) This uniform impurities concentration is also close to a recent estimate found in Ref. ahn:2022. However, they used fixed quantum well width $w=30$ nm in the calculations, while we take into account the density dependence such that $k_{F}w=3.9$.
  * Dmitriev  _et al._ (2012) I. A. Dmitriev, A. D. Mirlin, D. G. Polyakov, and M. A. Zudov, Rev. Mod. Phys. 84, 1709 (2012).
  * (18) Eqs. (22) and (23) in Ref. Sammon  _et al._ , 2018a apply to a single doping layer. Here, we consider two doping layers playing the same role, so the mobility is reduced by 2.
  * Sakaki  _et al._ (1987) H. Sakaki, T. Noda, K. Hirakawa, M. Tanaka, and T. Matsusue, Appl. Phys. Lett. 51, 1934 (1987).
  * Kamburov  _et al._ (2016) D. Kamburov, K. W. Baldwin, K. W. West, M. Shayegan, and L. N. Pfeiffer, Appl. Phys. Lett. 109, 232105 (2016).
  * Li  _et al._ (2005) J. M. Li, J. J. Wu, X. X. Han, Y. W. Lu, X. L. Liu, Q. S. Zhu, and Z. G. Wang, Semicond. Sci. Tech. 20, 1207 (2005).
  * Gottinger  _et al._ (1988) R. Gottinger, A. Gold, G. Abstreiter, G. Weimann, and W. Schlapp, Europhys. Lett. 6, 183 (1988).
  * Luhman  _et al._ (2007) D. R. Luhman, D. C. Tsui, L. N. Pfeiffer, and K. W. West, Appl. Phys. Lett. 91, 072104 (2007).
  * Umansky and Heiblum (2013) V. Umansky and M. Heiblum,  _Molecular Beam Epitaxy: From research to mass production_ , edited by M. Henini (Elsevier Inc., 2013) Chap. MBE growth of high-mobility 2DEG, pp. 121–137.
  * Ando (1982) T. Ando, J. Phys. Soc. Jpn. 51, 3900 (1982).
  * Halperin  _et al._ (1993) B. I. Halperin, P. A. Lee, and N. Read, Phys. Rev. B 47, 7312 (1993).
  * Note (2) To avoid the divergence at $z=0$, one can average $z$ in the denominator using $\expectationvalue{z}=\intop\ilimits@dz^{\prime}\absolutevalue{\psi(z^{\prime})}^{2}\absolutevalue{z-z^{\prime}}$, where $\psi(z)$ is the wave function along $z$ direction.
  * Note (3) On the other hand, AD and IR do not change the 2D electron concentration inside the quantum well, so they do not contribute to the gauge field fluctuations and $\rho_{1/2}$.
  * Alekseev (2016) P. S. Alekseev, Phys. Rev. Lett. 117, 166601 (2016).
  * Bockhorn  _et al._ (2014) L. Bockhorn, I. V. Gornyi, D. Schuh, C. Reichl, W. Wegscheider, and R. J. Haug, Phys. Rev. B 90, 165434 (2014).
  * Dai  _et al._ (2010) Y. Dai, R. R. Du, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 105, 246802 (2010).
  * Hatke  _et al._ (2011) A. T. Hatke, M. A. Zudov, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 83, 121301(R) (2011).
  * Bockhorn  _et al._ (2011) L. Bockhorn, P. Barthold, D. Schuh, W. Wegscheider, and R. J. Haug, Phys. Rev. B 83, 113301 (2011).
  * Hatke  _et al._ (2012) A. T. Hatke, M. A. Zudov, J. L. Reno, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 85, 081304(R) (2012).
  * Shi  _et al._ (2014a) Q. Shi, P. D. Martin, Q. A. Ebner, M. A. Zudov, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 89, 201301(R) (2014a).
  * Shi  _et al._ (2014b) Q. Shi, M. A. Zudov, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 90, 201301(R) (2014b).
  * Hwang and Das Sarma (2008) E. H. Hwang and S. Das Sarma, Phys. Rev. B 77, 235437 (2008).
  * Baba  _et al._ (1983) T. Baba, T. Mizutani, and M. Ogawa, Jpn. J. Appl. Phys. 22, L627 (1983).
  * Friedland  _et al._ (1996) K.-J. Friedland, R. Hey, H. Kostial, R. Klann, and K. Ploog, Phys. Rev. Lett. 77, 4616 (1996).
  * Du  _et al._ (1993) R. R. Du, H. L. Stormer, D. C. Tsui, L. N. Pfeiffer, and K. W. West, Phys. Rev. Lett. 70, 2944 (1993).
  * (41) L. N Pfeiffer, private communication.
  * Coleridge  _et al._ (1996) P. Coleridge, M. Hayne, P. Zawadzki, and A. Sachrajda, Surf. Sci. 361, 560 (1996).
  * Tan  _et al._ (2005) Y.-W. Tan, J. Zhu, H. L. Stormer, L. N. Pfeiffer, K. W. Baldwin, and K. W. West, Phys. Rev. Lett. 94, 016405 (2005).
  * Hatke  _et al._ (2013) A. T. Hatke, M. A. Zudov, J. D. Watson, M. J. Manfra, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 87, 161307(R) (2013).
  * Shchepetilnikov  _et al._ (2017) A. V. Shchepetilnikov, D. D. Frolov, Y. A. Nefyodov, I. V. Kukushkin, and S. Schmult, Phys. Rev. B 95, 161305(R) (2017).
  * Fu  _et al._ (2017) X. Fu, Q. A. Ebner, Q. Shi, M. A. Zudov, Q. Qian, J. D. Watson, and M. J. Manfra, Phys. Rev. B 95, 235415 (2017).
  * Ahn and Das Sarma (2022) S. Ahn and S. Das Sarma, Phys. Rev. Materials 6, 014603 (2022).
  * Sammon  _et al._ (2018b) M. Sammon, T. Chen, and B. I. Shklovskii, Phys. Rev. Materials 2, 104001 (2018b).
  * Davies (1997) J. H. Davies,  _The Physics of Low-dimensional Semiconductors: An Introduction_ (Cambridge University Press, 1997).
  * Note (4) Since background impurities scattering dominates at $n_{e}<3\ll n$, $1-f<0.1\ll 1$ and we can use $r_{s}\simeq 0.5n^{-1/2}$ from Ref. sammon:2018a.
  * Note (5) See for example, Eq. (5c) in Ref. gold:1989b or Section 4.9 in Ref. davies:1997.

##  I Supplementary materials

###  I.1 Well width $w$ and setback $d_{w}$ versus electron density

To obtain the constraints on the quantum well width $w$ and on the effective
spacer width $d_{w}=d+w/2$ we have fitted the experimental data of Ref. Chung
_et al._ , 2021 as illustrated in Fig. S1. Open circles in panel (a) and (b)
of Fig. S1 represent $w$ and $d_{w}$, respectively, as a function of electron
density $n_{e}$. Solid lines are power law fits to the data, yielding
$w=49\,n_{e}^{-1/2}$ and $d_{w}=0.28\,n_{e}^{-1}$, in units as marked. These
relationships are equivalent to $k_{F}w=3.9$, where $k_{F}=\sqrt{2\pi n_{e}}$
is the Fermi wave number, and $d_{w}^{-1}=an_{e}$, where $a=3.55$ nm, used in
the main text. Here, the first condition corresponds to a maximum electron
density at which the second subband of the GaAs quantum well still remains
unoccupied, whereas the second condition is governed by electrostatics of the
device.

Figure S1:  (a) Quantum well width $w$, and (b) effective setback distance
$d_{w}=d+w/2$ as a function of the carrier density $n_{e}$ in a new
generation, doping-well samples from Ref. Chung  _et al._ , 2021. Solid lines
are power-law fits yielding $w=49\,n_{e}^{-1/2}$ and $d_{w}=0.28\,n_{e}^{-1}$,
in units as marked. These fits correspond to constraints $k_{F}w=3.9$ and
$d_{w}^{-1}=an_{e}$, where $a=3.55$ nm.

###  I.2 Scattering potential of a single background impurity

We derive the screened potential of a single charged impurity at a distance
$z$ away from the center of a quantum well of thickness $w$ using random phase
approximation (RPA) as in Ref. Sammon  _et al._ , 2018a. There are two sources
of screening for background impurities: they can be screened by 2DEG in the
quantum well or by excess electrons (EEs) remaining in the remote doping wells
separated from the center of the 2DEG by distance $d_{w}=d+w/2$. We introduce
the notations $U_{{\rm i},k}$ for the screened impurity interaction and
${\tilde{U}}_{{\rm i},k}$ for the unscreened impurity interaction, where the
subscript i denotes the impurity and subscript $k$ can be equal to 1 (2DEG) or
2 (EEs). [Used below subscript $l$ can be equal to 1 (2DEG) or 2 (EEs) as
well]. Following Ref. Sammon  _et al._ , 2018a, we obtain the screened Coulomb
interaction between an impurity and an electron in the 2DEG,

$\displaystyle U_{{\rm i},1}=\frac{{\tilde{U}}_{{\rm
i},1}(1-\Pi_{2}{\tilde{U}}_{2,2})+{\tilde{U}}_{{\rm
i},2}\Pi_{2}{\tilde{U}}_{2,1}}{(1-\Pi_{1}{\tilde{U}}_{1,1})(1-\Pi_{2}{\tilde{U}}_{2,2})-\Pi_{1}\Pi_{2}{\tilde{U}}_{1,2}^{2}}\,,$
(S1)

where bare interactions ${\tilde{U}}_{{\rm i},k}$ and ${\tilde{U}}_{k,l}$ are
given by

$\displaystyle{\tilde{U}}_{{\rm i},k}(q,z)=\frac{2\pi e^{2}}{\kappa
q}\int\limits_{-\infty}^{+\infty}dz^{\prime}\;\lambda_{k}(z^{\prime})e^{-q\absolutevalue{z-z^{\prime}}}\,,$
(S2) $\displaystyle{\tilde{U}}_{k,l}(q)=\frac{2\pi e^{2}}{\kappa
q}\iint\limits_{-\infty}^{+\infty}dz\;dz^{\prime}\;\lambda_{k}(z)\lambda_{l}(z^{\prime})e^{-q\absolutevalue{z-z^{\prime}}}\,.$
(S3)

Here, $\lambda_{1}(z)=\absolutevalue{\psi(z)}^{2}$ and
$\lambda_{2}(z)=\delta(\absolutevalue{z}-d_{w})$ are the linear densities of
the 2DEG and of the EEs respectively, and the polarization functions
$\Pi_{1}=-(\kappa q_{\rm TF}/2\pi e^{2})[1-G(h)]$ and $\Pi_{2}=-(\kappa
r_{s}^{-1}/2\pi e^{2})[1-G(h)]$ are calculated using Thomas-Fermi
approximation with local field corrections.

Within infinite-potential-well approximation,
$\lambda_{1}(z)=(2/w)\cos^{2}(\pi z/w)\Theta(w/2-\absolutevalue{z})$, we can
write down the analytical expressions for bare interactions. For
${\tilde{U}}_{{\rm i},1}$, we find

$\displaystyle{\tilde{U}}_{{\rm i},1}(q,z)=\frac{2\pi e^{2}}{\kappa
q}\begin{cases}F_{0}(qw)e^{-q\absolutevalue{z}},&\absolutevalue{z}>w/2\,,\\\
G_{0}(q,z),&\absolutevalue{z}<w/2\,,\end{cases}$ (S4)

where

$\displaystyle F_{0}(x)=\frac{8\pi^{2}}{x(x^{2}+4\pi^{2})}\sinh(x/2)\,,$ (S5)

and

$\displaystyle G_{0}(q,z)$ $\displaystyle=F_{0}(qw){\rm
csch}(qw/2)(1-e^{-qw/2}\cosh(qz))$ $\displaystyle+\frac{4qw\cos^{2}(\pi
z/w)}{4\pi^{2}+q^{2}w^{2}}\,.$ (S6)

For ${\tilde{U}}_{{\rm i},2}$, we have

$\displaystyle{\tilde{U}}_{{\rm i},2}=\frac{2\pi e^{2}}{\kappa
q}(e^{-q\absolutevalue{z-d_{w}}}+e^{-q\absolutevalue{z+d_{w}}})$ (S7)

The electron-electron interactions are given by

$\displaystyle{\tilde{U}}_{1,1}=\frac{2\pi e^{2}}{\kappa q}F_{c}(qw),$ (S8)
$\displaystyle{\tilde{U}}_{1,2}={\tilde{U}}_{2,1}=\frac{2\pi e^{2}}{\kappa
q}2e^{-qd_{w}}F_{0}(qw),$ (S9) $\displaystyle{\tilde{U}}_{2,2}=\frac{2\pi
e^{2}}{\kappa q}(2+2e^{-2qd_{w}}),$ (S10)

where

$\displaystyle
F_{c}(x)=\frac{20\pi^{2}x^{3}+3x^{5}-32\pi^{4}(1-e^{-x}-x)}{x^{2}(4\pi^{2}+x^{2})^{2}}.$
(S11)

Screening by EEs affects only the potential of impurities located in the
vicinity of the doping wells. These impurities practically do not contribute
to momentum relaxation, so we can calculate the mobility assuming $\Pi_{2}=0$.
The corresponding $U_{i,1}$ in Eq. (S1) with $\Pi_{2}=0$ can be simplified as

$\displaystyle U_{i,1}=\frac{{\tilde{U}}_{{\rm
i},1}}{1-\Pi_{1}{\tilde{U}}_{1,1}},$ (S12)

which is equal to $U_{1}$ defined in Eq. (3) of the main text. However, for
quantum mobility EE screening plays an important role. The screening radius of
EEs is given by $r_{s}\simeq 0.5n^{-1/2}$ Sammon  _et al._ (2018b), valid for
large filling fraction of remote donors $f=1-n_{e}/2n>0.5$ 444 Since
background impurities scattering dominates at $n_{e}<3\ll n$, $1-f<0.1\ll 1$
and we can use $r_{s}\simeq 0.5n^{-1/2}$ from Ref. Sammon  _et al._ , 2018b. ,
where $n_{e}$ is the 2DEG density and $n=1.5\times 10^{12}$ cm-2 is the
concentration of donors. We have confirmed that our results for $\mu_{\rm q}$
with $r_{s}\simeq 0.5n^{-1/2}\approx 4$ nm are practically the same as for
$r_{s}=0$ or $\Pi_{2}=\infty$.

###  I.3 Finite-potential-well solution

The Schrödinger equation is given by

$-\frac{\hbar^{2}}{2m(z)}\derivative[2]{\psi(z)}{z}+V(z)\psi(z)=E\psi(z)\,,$
(S13)

where $m(z)=m^{\star}=0.067\,m_{0}$ at $\absolutevalue{z}<w/2$ and
$m(z)=m_{B}=(0.067+0.083x)m_{0}=0.077\,m_{0}$ at $\absolutevalue{z}>w/2$, and
the effective finite potential well is described by

$V(z)=\begin{cases}0,\quad\,&\absolutevalue{z}<w/2\,,\\\
V_{0}-E_{F},\quad\,&\absolutevalue{z}>w/2\,,\end{cases}$ (S14)

where $V_{0}=90$ meV ($x=0.12$) and $E_{F}=\hbar^{2}k_{F}^{2}/2m^{\star}$ is
the Fermi energy. Below we use $V$ to represent $V_{0}-E_{F}$ for simplicity.
The bound state solutions ($E<V$) for the lowest subband are

$\psi(z)=\begin{cases}Ce^{-\eta\absolutevalue{z}},\,&\absolutevalue{z}>w/2\,,\\\
D\cos(kz),\,&\absolutevalue{z}<w/2\,,\\\ \end{cases}$ (S15)

where $k=\sqrt{2m^{\star}E/\hbar^{2}}$, $\eta=\sqrt{2m_{B}(V-E)/\hbar^{2}}$.
At $z=\pm w/2$, $\psi(z)$ is continuous while $\psi^{\prime}(z)$ has a finite
discontinuity because of the difference in effective masses:

$\displaystyle Ce^{-\eta w/2}=D\cos(kw/2)\,,$ (S16)
$\displaystyle\frac{C}{m_{B}}\eta e^{-\eta
w/2}=\frac{D}{m^{\star}}k\sin(kw/2)\,.$ (S17)

The normalization of $\psi(z)$ gives

$C^{2}\int_{w/2}^{\infty}e^{-2\eta
z}dz+D^{2}\int_{0}^{w/2}\cos^{2}(kz)dz=1/2,$ (S18)

Combining with Eq. (S16), one has

$C=\quantity[\frac{1}{\eta}e^{-\eta
w}+\frac{kw+\sin(kw)}{2k\cos^{2}(kw/2)}e^{-\eta w}]^{-1/2}.$ (S19)

By dividing Eq. (S17) by Eq. (S16), one obtains

$\tan(kw/2)=\frac{m^{\star}}{m_{B}}\frac{\eta}{k}\,.$ (S20)

Introducing dimensionless quantities $\tilde{E}=E/E_{0}$, $\tilde{V}=V/E_{0}$,
where $E_{0}=\hbar^{2}\pi^{2}/2m^{\star}w^{2}$ is the ground state energy for
infinite well, in Eq. (S20) we arrive at 555See for example, Eq. (5c) in Ref.
Gold, 1989 or Section 4.9 in Ref. Davies, 1997.

$\tilde{E}+\frac{m_{B}}{m^{\star}}\tilde{E}\tan^{2}{\quantity(\sqrt{\tilde{E}}\pi/2)}=\tilde{V}\,.$
(S21)

If $\tilde{V}\to\infty$ and $m_{B}=m^{\star}$, then $\tilde{E}=1$. For any
finite $\tilde{V}$, we have $0<\tilde{E}<1$. For example, at small electron
concentration $n_{e}<1\times 10^{11}$ cm-2 with $k_{F}w=3.9$, we have
$\tilde{V}>37\gg 1$, which justifies our infinite-potential-well approximation
at lower $n_{e}$. On the other hand, at large electron concentration
$n_{e}\in(5,15)\times 10^{11}$ cm-2, $\tilde{V}$ decreased significantly from
6.2 to 1.1, and therefore finite-potential-well effect such that
$0<\tilde{E}<1$ has to be taken into account.

The local energy fluctuation due to interface roughness is given by

$\delta E(\mathbf{r})=\partialderivative{E}{w}\Delta(\mathbf{r})\,,$ (S22)

where $\Delta(\mathbf{r})$ is the local variation of the well width at a
position $\mathbf{r}$. Taking derivative with respect to $w$ for both sides in
Eq. (S21), we obtain the expression for $\partialderivative*{E}{w}$

$\partialderivative{E}{w}=-\frac{2E}{w}\quantity{1+g\quantity(\frac{m_{B}}{m^{\star}},\frac{E}{V}){\tilde{V}}^{-1/2}}^{-1}\,,$
(S23)

and

$g(x,y)=\frac{2}{\pi}\quantity[x^{-1/2}+y(x^{1/2}-x^{-1/2})]^{-1}(1-y)^{-1/2}\,.$
(S24)

Substituting Eq. (S23) in Eq. (8) in the main text, we obtain the scattering
potential for the interface roughness.


11institutetext: The University of Hong Kong, Hong Kong SAR
11email: {fuyingw@connect.,shenghui<EMAIL_ADDRESS>
# Author Guidelines for ECCV Submission

Fuying Wang 11 0000-0002-3313-6479 Shenghui Du 11 Lequan Yu 11
0000-0002-9315-6527

# HERGen: Elevating Radiology Report Generation with Longitudinal Data

Fuying Wang 11 0000-0002-3313-6479 Shenghui Du 11 Lequan Yu 11
0000-0002-9315-6527

###### Abstract

Radiology reports provide detailed descriptions of medical imaging integrated
with patients’ medical histories, while report writing is traditionally labor-
intensive, increasing radiologists’ workload and the risk of diagnostic
errors. Recent efforts in automating this process seek to mitigate these
issues by enhancing accuracy and clinical efficiency. However, existing
automated approaches are based on a single timestamp and often neglect the
critical temporal aspect of patients’ imaging histories, which is essential
for accurate longitudinal analysis. To address this gap, we propose a novel
History Enhanced Radiology Report Generation (HERGen) framework that employs a
group causal transformer to efficiently integrate longitudinal data across
patient visits. Our approach not only allows for comprehensive analysis of
varied historical data but also improves the quality of generated reports
through an auxiliary contrastive objective that aligns image sequences with
their corresponding reports. More importantly, we introduce a curriculum
learning-based strategy to adeptly handle the inherent complexity of
longitudinal radiology data and thus stabilize the optimization of our
framework. The extensive evaluations across three datasets demonstrate that
our framework surpasses existing methods in generating accurate radiology
reports and effectively predicting disease progression from medical images.

###### Keywords:

Radiology Report Generation Longitudinal Study Vision-Language Learning

##  1 Introduction

Figure 1:  Overview of our HERGen for radiology report generation: Our model
processes longitudinal data for each patient and utilizes the comprehensive
historical information within these longitudinal data to generate robust and
precise radiology reports.

Chest X-rays are a cornerstone in diagnosing thoracic conditions, including
pneumonia and lung cancer [19, 35]. Given a chest X-ray, radiologists will
meticulously examine each anatomical section in the X-ray and document their
observations with detailed text descriptions. The generated report is crucial
to diagnose diseases (_e.g_., lung cancer, scoliosis) and assess the position
of the treatment devices (_e.g_., tracheostomy tubes, pacemakers).
Particularly, when prior images are available, radiologists commonly compare
the clinical findings of the current scan with prior scans to assess the
evolution of disease over time, which is essential in regular clinical
evaluations. However, the high volume of chest X-rays overwhelms radiologists,
exacerbating the impact of the global shortfall in this workforce [8, 37].

Automated chest X-ray report generation has emerged as a key research area,
aiming to ease radiologists’ workload and improve patient care [43].
Mainstream approaches focus on improving clinical accuracy and completeness of
individual reports [10, 9, 27, 42], often overlooking the chronological
consistency in longitudinal imaging. Modeling such inherent temporal
information in chest X-rays has shown to be crucial for generating precise
radiology reports [20, 5, 59, 39]. Some recent studies integrate prior images
for temporal representation and enhance report generation [5, 39]. However,
they are limited to the use of only one prior image for the current report,
failing to capture high-level disease progression evident across a patient’s
history. This highlights the need for a framework that learns accurate
representations from both study-level and patient-level images, thereby
producing reports closely aligned with radiologists’ analyses.

In this paper, we propose a novel History Enhanced radiology Report Gener-
ation framework (HERGen) to effectively capture the temporal information of
longitudinal data for generating comprehensive and temporally coherent
radiology reports, as shown in Fig. 1. The key part is a causal transformer
model, which treats all visual tokens from the same image as a group and uses
a group causal attention mechanism to handle it. Viewing all visual tokens of
each patient as a sequence, this mechanism groups visual tokens from the same
image, facilitating intra-image interactions of visual tokens and inter-image
interactions of tokens only across previous studies. Notably, it treats each
patient’s X-ray series as a distinct sequence, adeptly handling the
variability in the number of longitudinal images per patient. Moreover, we
further refine the model’s capability to chart disease progression through a
cross-modal contrastive learning objective, ensuring the alignment of
longitudinal visual representations with their narrative reports. Due to the
inherent complexity of longitudinal data, it is non-trivial to optimize the
whole framework. We thereby introduce a new curriculum learning-based
optimization strategy in three progressive steps to enhance and stabilize the
learning process of our framework. The model is first trained to generate
radiology reports for individual images and then, we employ the auxiliary
contrastive alignment module to optimize the latent space. After that, the
entire framework is trained with the integration of a temporal aggregation
module, enabling it to learn from the patient’s historical information.
Extensive experimental results on radiology report generation and temporal
medical image classification tasks demonstrate the superiority of our
framework in generating accurate radiology reports and effectively predicting
the disease progression from medical images. The source code is available at
https://github.com/fuying-wang/HERGen.

##  2 Related Work

Automated Report Generation. Radiology report generation, inspired by image
captioning techniques [11, 56, 54, 46], face unique challenges due to the
complexity and variability in radiology reports [1, 9, 10, 18, 25, 23, 49,
55]. Initial approaches, primarily based on CNN-RNN [17, 18, 49, 57], have
evolved with the adoption of transformer [44]. Recent advancements include
memory-driven transformers for enhanced cross-modal interactions [10, 9],
alignment of visual features with disease tags [55], and contrastive methods
for anomaly detection [25]. Integration of knowledge graphs [23, 57], warm
starting strategies [27], and interactive frameworks for region-specific
reports [42] have also been explored. However, these methods often treat
X-rays and reports as independent entities, overlooking the temporal aspects
inherent in various radiology modalities.

Longitudinal Chest X-ray Representations. Radiology studies, inherently
chronological, are crucial for accurate reporting, yet the temporal dimension
is often under-addressed in research. [34] indirectly acknowledged the
importance of sequential context by proposing a method to reduce language
model hallucinations. [5] introduced a self-supervised framework capturing the
longitudinal evolution of chest X-ray findings. Similarly, [59] developed a
cross-attention-based multi-modal fusion framework utilizing patient record
chronology to enhance report pre-filling. [20] employed graph attention
networks [45] for an anatomy-aware approach to tracking disease progression in
longitudinal CXR data. [39] used Faster R-CNN [36] to project longitudinal
studies into a composite representation highlighting anatomical changes over
time. However, most of these methods primarily focus on learning
representations rather than generating reports. Furthermore, these methods
often treat two consecutive image-text pairs, lacking flexibility for varying
patient history lengths and are limited in capturing the complex progression
of diseases.

Biomedical Vision-language Pretraining. Radiology reports, paired with chest
X-rays, offer rich labels for learning visual representations. Building on the
CLIP framework [32], [58, 15, 7, 47] demonstrate the efficacy of self-
supervised vision-language pretraining in biomedical imaging tasks.
Particularly, [58] use a contrastive objective [29] for modality alignment,
[15] focus on local alignment for detailed feature learning, and [7] develop
CXR-BERT, employing masked language modeling for enhanced image feature
learning from radiology language. Furthermore, [5] adapt BioViL [7] for
longitudinal analysis in radiology, improving temporal aspects in report
generation and classification tasks. Our work further explores the application
of vision-language pretraining on longitudinal data, aiming to effectively
capture disease progression in patient records.

##  3 Method

###  3.1 Problem Formulation

The overall framework of the proposed method is shown in Fig. 2. We analyze a
dataset comprising chest X-rays from $M$ patients, denoted as
$\\{\mathcal{I}_{i}\\}_{i=1,2,...,M}$, where
$\mathcal{I}_{i}=\\{\mathbf{I}_{j}^{(i)}\\}_{j=1,2,...,N_{i}}$ represents the
set of X-rays for the $i$-th patient and $N_{i}$ is the number of studies
(visits). For each patient $i$, their X-rays are chronologically ordered based
on their associated study dates
$\mathcal{T}_{i}=\\{\mathbf{T}_{j}^{(i)}\\}_{j=1,2,...,N_{i}}$. The objective
of our method is to generate a set of radiology reports
$\\{\mathcal{\hat{R}}_{i}\\}_{i=1,2,...,M}$ for each patient, aiming to
closely approximate the ground truth reports
$\\{\mathcal{R}_{i}\\}_{i=1,2,...,M}$. In the following, we use $i$ and $j$ to
index patient and study respectively.

###  3.2 History-enhanced Report Generation

Extract Representations of Single Images. In our approach, each X-ray image
$\mathbf{I}_{j}^{(i)}\in\mathbb{R}^{C\times W\times H}$ is first encoded into
a feature representation $\mathbf{P}_{j}^{(i)}\in\mathbb{R}^{S\times F}$ with
an image encoder. Here, $C$, $W$, and $H$ denote the number of channels,
width, and height of the image, respectively, while $S$ and $F$ represent the
number of visual tokens and the feature dimension per token. Following
CvT-212DistilGPT2 [27], we utilize the CvT architecture [52], pretrained on
ImageNet-21K, as our image encoder, while our framework can take other encoder
backbones. To tailor the dimensions of $S$ and $F$ to our requirements, we
introduce an encoder projection layer $E_{proj}$. This layer comprises a
$1\times 1$ convolution layer followed by a linear projection layer,
transforming each $\mathbf{P}_{j}^{(i)}$ into a more compact visual
representation $\mathbf{V}_{j}^{(i)}\in\mathbb{R}^{S^{\prime}\times
F^{\prime}}$, where $S^{\prime}$ and $F^{\prime}$ denote the adjusted number
of visual tokens and their new dimensionality, respectively.

Sequential Date-aware Temporal Embedding. Temporal embeddings are especially
critical for our group causal transformer to learn longitudinal information.
Standard positional embeddings typically assume equidistant intervals between
tokens, an assumption that is not applicable in our context due to the varying
time gaps between consecutive chest X-rays. For example, the clinical
progression captured in X-rays taken a month apart is significantly different
from that in X-rays taken a year apart. To tackle this challenge, we introduce
study date-aware positional embeddings,
$\mathbf{p}_{j}^{(i)}\in\mathbb{R}^{S^{*}\times F^{\prime}}$ for each study.
These embeddings are conditioned on the study dates $\mathbf{T}_{j}^{(i)}$,
offering a more precise representation of the temporal intervals between
X-rays. In detail, we first calculate the relative study date for each X-ray
image as
$\mathbf{T^{\prime}}_{j}^{(i)}=\mathbf{T}_{j}^{(i)}-\mathbf{T}_{0}^{(i)}$.
Then, we identify the maximum relative study date in the training set and
create a learnable embedding vocabulary of the corresponding length. Each
temporal embedding is defined as:
$\mathbf{p}_{j}^{(i)}=\mathrm{Embedding}(\mathbf{T^{\prime}}_{j}^{(i)})\in\mathbb{R}^{1\times
F^{\prime}}$. The visual token embeddings $\mathbf{V}_{j}^{(i)}$ are then
added with the temporal embedding $\mathbf{p}_{j}^{(i)}$ to form
$\mathbf{Z}_{j}^{(i)}=\mathbf{V}_{j}^{(i)}+\mathbf{p}_{j}^{(i)}$, where
$\mathbf{Z}_{j}^{(i)}\in\mathbb{R}^{S^{\prime}\times F^{\prime}}$. Finally, we
concatenate all visual token embeddings for each patient to create a patient-
level sequence
$\mathbf{\tilde{Z}}_{i}=\mathrm{Concat}([\\{\mathbf{Z}_{j}^{(i)}\\}_{j=1,2,...,N_{i}}])$
where $\mathbf{\tilde{Z}}_{i}\in\mathbb{R}^{S^{(i)}\times F^{\prime}}$ and
$S^{(i)}=N_{i}\times S^{\prime}$, which is then fed into the group causal
transformer for temporal aggregation.

Figure 2:  History Enhanced Radiology Report Generation (HERGen): the
framework processes patient-level chest X-rays using the CvT* (CvT combined
with the encoder projection layer), which then aggregates temporal information
through a group causal transformer. Subsequently, GPT2 serves as the decoder
for predicting the radiology report, which was optimized by a cross-entropy
(CE) loss. Additionally, an auxiliary contrastive alignment module is employed
to enhance the alignment of the latent spaces between image and text
modalities, thereby producing more consistent reports. Note that in the group
causal transformer block, thick lines represent image-level interactions,
while thin lines indicate token-level interactions.

Group Causal Transformer. Our group causal transformer comprises $L$ group
causal blocks, designed to aggregate longitudinal information from patient
data. In block $l$, for every visual token (indexed by $p$), we first compute
the query, key, and value vectors from its preceding block’s representation
$\mathbf{z}^{(l-1)}_{(p)}\in\mathbb{R}^{F^{\prime}}$ as:

$\displaystyle\mathbf{q}_{(p)}^{(l,a)}=\mathbf{W}_{Q}^{(l,a)}\mathrm{LN}(\mathbf{z}_{(p)}^{(l-1)})\in\mathbb{R}^{D_{h}},$
$\displaystyle\mathbf{k}_{(p)}^{(l,a)}=\mathbf{W}_{K}^{(l,a)}\mathrm{LN}(\mathbf{z}_{(p)}^{(l-1)})\in\mathbb{R}^{D_{h}},$
$\displaystyle\mathbf{v}_{(p)}^{(l,a)}=\mathbf{W}_{V}^{(l,a)}\mathrm{LN}(\mathbf{z}_{(p)}^{(l-1)})\in\mathbb{R}^{D_{h}}.$
(1)

Figure 3: Comparison between (a) bidirectional attention, (b) causal attention
and (c) our group causal attention.

Figure 4: Illustration of the proposed curriculum training strategy.

Here, $\mathrm{LN}$ denotes LayerNorm and $a=1,...,A$ indexes the $A$
attention heads with the latent dimensionality for each head being
$D_{h}=F^{\prime}/A$. The initial representation $\mathbf{z}^{(0)}$
corresponds to the input sequence $\mathbf{\tilde{Z}}_{i}$. $\mathbf{W}_{Q}$,
$\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ are learnable matrices. For simplicity,
we have omitted the patient index $i$ in our notation. Then, the process for
computing dot-product self-attention weights, along with subsequent steps, is
defined as follows:

$\displaystyle\mathbf{\alpha}_{(p)}^{(l,a)}=\mathrm{SM}(\frac{\mathbf{q}_{(p)}^{(l,a)}}{D_{h}}\cdot[\mathbf{k}_{(p^{\prime})}^{(l,a)}]_{p^{\prime}\in
0,...,S^{(i)}-1}),$
$\displaystyle\mathbf{s}{(p)}^{(l,a)}=\sum_{p^{\prime}}\mathbf{M}{(p^{\prime})}\mathbf{\alpha}{(p)(p^{\prime})}^{(l,a)}\mathbf{v}{(p^{\prime})}^{(l,a)},$
(2)

where $\mathbf{\alpha}\in\mathbb{R}^{S^{(i)}}$. The group causal attention
matrix $\mathbf{M}(p^{\prime})$, as illustrated in Fig. 4, differs
fundamentally from the bidirectional attention used in BERT [12] and the
causal attention in GPT [33]. It ensures that each visual token within an
image not only interacts with others in the same image but also with tokens
from preceding images.

This design reflects our intention to make the transformer cognizant of the
temporal sequence in radiological data, a crucial aspect for accurately
capturing disease progression over time. Subsequently, we perform
concatenation followed by a Multi-Layer Perceptron (MLP) with residual
connections to get the output. This can be mathematically represented as:

$\displaystyle\mathbf{z^{\prime}}_{(p)}^{(l)}$
$\displaystyle=\mathbf{W}_{O}[\mathbf{s}_{(p)}^{(l,1)}...\mathbf{s}_{(p)}^{(l,A)}]+\mathbf{z}_{(p)}^{(l-1)},$
$\displaystyle\mathbf{z}_{(p)}^{(l)}$
$\displaystyle=\mathrm{MLP}(\mathrm{LN}(\mathbf{z^{\prime}}_{(p)}^{(l)}))+\mathbf{z^{\prime}}_{(p)}^{(l)},$
(3)

where $\mathbf{W}_{O}$ is a learnable matrix. Then, the output sequence of the
group causal transformer, denoted as
$\mathbf{z}^{(L)}\in\mathbb{R}^{S^{(i)}\times F^{\prime}}$, is split into a
series of representations of studies
$\\{\mathbf{D}_{j}^{(i)}\\}_{j=1,...,N_{i}}$ and
$\mathbf{D}_{j}^{(i)}\in\mathbb{R}^{S^{\prime}\times F^{\prime}}$.

Report Generation and Auxiliary Contrastive Alignment. Each temporally
aggregated visual representation $\mathbf{D}_{j}^{(i)}$ is input into a text
decoding module for generating radiology reports. Note that We chose GPT-2 as
the text decoder following [27], which shows DistilGPT2 [38] outperforms other
alternatives like ClinicalBERT [2], PubMedBERT [13], and SciBERT [6]. We
minimize a cross-entropy loss $\mathcal{L}_{\mathrm{CE}}$ to ensure predicted
reports are close to ground truth reports. To improve the coherence of
generated reports, we introduce an auxiliary contrastive alignment module.
This module is designed to align the distributions of the visual and textual
modalities, thereby enhancing the model’s overall performance. Initially, for
each visual token embedding
$\mathbf{D}_{j}^{(i)}\in\mathbb{R}^{S^{\prime}\times F^{\prime}}$, we perform
a mean pooling operation along the first dimension and it results in a global
representation of the entire image, denoted as
$\mathbf{\tilde{D}}_{j}^{(i)}\in\mathbb{R}^{F^{\prime}}$. Then, we use a text
encoder to encode each report $\mathbf{R}_{j}^{(i)}$ into a representation
$\mathbf{E}_{j}^{(i)}$. Subsequently, we concatenate all visual and text
embeddings within the same minibatch to form a combined set:
$\\{\mathbf{\tilde{D}}_{s}\\}_{s=1,2,...,N_{B}}$ and
$\\{\mathbf{E}_{s}\\}_{s=1,2,...,N_{B}}$, respectively. Note that $N_{B}$
represents the total number of studies within the mini-batch. The contrastive
loss is defined as follows:

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{Cont}}=\sum_{r=1}^{N_{B}}\frac{1}{2N_{B}}&(-\mathrm{log}\frac{\mathrm{exp}(\mathrm{sim}(\mathbf{\tilde{D}}_{r},\mathbf{E}_{r})/\tau)}{\sum_{s^{\prime}=1}^{N_{B}}\mathrm{exp}(\mathrm{sim}(\mathbf{\tilde{D}}_{r},\mathbf{E}_{s^{\prime}})/\tau)}\\\
&-\mathrm{log}\frac{\mathrm{exp}(\mathrm{sim}(\mathbf{E}_{r},\mathbf{\tilde{D}}_{r})/\tau)}{\sum_{s^{\prime}=1}^{N_{B}}\mathrm{exp}(\mathrm{sim}(\mathbf{E}_{r},\mathbf{\tilde{D}}_{s^{\prime}})/\tau)})\end{split}$
(4)

where $\tau$ is the temperature hyperparameter.

Learning Objectives. Finally, our model is optimized by jointly minimizing
these two objectives:

$\mathcal{L}=\mathcal{L}_{\mathrm{CE}}+\lambda\cdot\mathcal{L}_{\mathrm{Cont}}.$
(5)

Here, $\lambda$ is a hyperparameter used to balance these two losses. Based on
empirical studies, we set $\lambda$ as $1.0$. The ablation results of the
hyperparameter $r$ can be found in the Supplementary Material.

###  3.3 Curriculum Training

As shown in Fig. 4, we introduce a curriculum learning strategy, unfolding in
three stages to progressively enhance our model’s performance:

  * •

Stage 1: Encoder-Decoder Report Generation: Initially, reports are generated
using an encoder-decoder architecture trained on individual chest X-ray image-
text pairs. This foundational step focuses solely on static data without
temporal context.

  * •

Stage 2: Alignment Refinement with Text Encoder: Subsequently, a text encoder
is incorporated, utilizing contrastive learning to refine the alignment
between the visual and textual data.

  * •

Stage 3: Temporal Information Learning:The final stage expands the model’s
capability to a longitudinal perspective. Here, we integrate the group causal
transformer to process sequences of chest X-rays, thereby incorporating
temporal information into the report generation.

These stages collectively develop a robust and comprehensive model, which is
then systematically evaluated to assess its effectiveness in generating
accurate and contextually relevant radiology reports.

##  4 Experiments

###  4.1 Experimental Setup

Dataset and Preprocessing. We evaluate the performance of our model on two
clinical tasks: radiology report generation and temporal medical image
classification. The used datasets are as follows:

  * •

MIMIC-CXR: We utilize the MIMIC-CXR dataset [19], which originally comprises
$377,110$ chest X-ray images and $227,835$ reports, to evaluate our model.
Aligning with previous work [10, 9, 27], we adopt the official split of the
MIMIC-CXR dataset in our experiment. However, the original dataset includes
multiple lateral images, which could introduce inconsistency in longitudinal
analyses. Additionally, we observed duplicate images within the same study,
bringing noise to patient-level progression analysis. Therefore, we
meticulously curated the dataset by removing lateral images and duplicates
within studies for each train/validation/test set, resulting in a preprocessed
dataset consisting of $145,471$ pairs for training, $1,151$ for validation,
and $2,210$ for testing. We then follow [27] to preprocess images and reports.
Specifically, we resize all images to 384$\times$384 while preserving aspect
ratios. Report preprocessing involved truncating to 60 words, converting to
lowercase, removing special characters, and replacing infrequent terms with
placeholders. Crucially, we organized the image-report pairs chronologically
based on the “StudyDate" metadata, preserving temporal integrity for analyzing
each patient’s radiological history. Further details on dataset curation and
preprocessing are available in the Supplementary Material. Note that we re-run
the publicly released code of compared methods on our curated MIMIC-CXR
dataset to ensure a fair comparison.

  * •

Longitudinal MIMIC-CXR: We further devise the Longitudinal MIMIC-CXR dataset,
derived from the preprocessed MIMIC-CXR-JPG dataset, to assess our model’s
capability in generating temporally coherent reports, following [59]. This
subset includes only patients with at least two consecutive visits. It is
worth noting that the training, validation, and test splits of the
Longitudinal-MIMIC dataset correspond to the official divisions of the MIMIC-
CXR dataset.

  * •

MS-CXR-T: We also assess our model’s capacity for capturing temporal
information using the MS-CXR-T dataset [4]. This dataset consists of $1326$
multi-image frontal chest X-rays, each annotated with one of five findings.
For each finding, there are three possible states reflecting disease
progression: “Improving," “Stable," and “Worsening".

Radiology Report Generation. We evaluate the performance of our method in
radiology report generation on both the MIMIC-CXR dataset and the Longitudinal
MIMIC-CXR dataset. We compare HERGen with $7$ state-of-the-art (SOTA)
radiology report generation models, including $\mathcal{M}^{2}$Transformer
[11], R2Gen [10], R2GenCMN [9], $\mathcal{M}^{2}$TR.PROGRESSIVE [28], XProNet
[48], CvT-212DistilGPT2 [27] and DCL [21]. To ensure a fair comparison, we
rerun the publicly released code of these methods on our curated MIMIC-CXR
dataset. Note that $5$ SOTA models (PPKER [23], ContrastiveAttention [25],
AlignTransformer [55], KIUT [16], and METransformer [50]) lack publicly
available source code, thus results are cited from their original papers for
reference. However, we note that these can’t directly compared to our results
due to our additional dataset preprocessing. Additionally, the results in the
RGRG [42] paper, employing the Chest ImaGenome [53] split instead of the
official MIMIC-CXR split, are also for reference only but not directly
comparable. On the Longitudinal MIMIC-CXR dataset, we compare our model with
both single-image based baselines, i.e., R2Gen [10], R2CMN [9],
CvT-212DistilGPT2 [27], etc. and longitudinal image-based baseline, i.e.,
Prefilling [59].

Temporal Image Classification. The temporal image classification task is
evaluated on the MS-CXR-T dataset [4]. This evaluation serves as an additional
task to assess how well our model can understand and process disease
progression in medical images. We compare our approach with both temporal
image-based vision language pretraining methods (_e.g_., BioViL-T) and single
image pretraining methods (_e.g_., BioViL). More information about this
experiment is available in the Supplementary Material.

Evaluation Metrics. In line with previous studies [31, 10, 27, 42], we
employed a combination of Natural Language Generation (NLG) and Clinical
Efficiency (CE) metrics to evaluate our report generation performance. For
NLG, we used established metrics including BLEU-n [30], which measures n-gram
overlap, METEOR [3], that accounts for recall through an $F_{\beta}$ score,
ROUGE-L [22], based on the longest common subsequence. Recognizing that NLG
metrics may not fully reflect clinical accuracy, we further integrated CE
metrics following previous work [10, 9, 16, 50]. Specifically, we apply
CheXbert [40] to label the generated reports into 14 categories (related to
thoracic diseases and support devices), and then compute precision, recall,
and F1 scores against ground truths. The macro-averaged results over $14$
classes are reported, given the susceptibility of micro-averaged metrics to
minor class imbalances [41]. As for the temporal image classification, we
predict one of “improving", “stable", and “worsening" for each one of the $5$
findings: Consolidation, Pleural Effusion, Pneumonia, Pneumothorax and Edema.
Following BioViL-T [5], we use macro-accuracy across the 5 classes to evaluate
the performance.

###  4.2 Implementation Details

We set the minibatch size to $16$ for single image-text pair training and to
$4$ for temporal training. Our model training was limited to a maximum of 5
studies per patient to accommodate resource limitations. We employed the AdamW
[24] optimizer for model optimization. The learning rate was adjusted
according to the training stage, with detailed strategies provided in the
Supplementary Materials. The training was early stopped if the validation
BLEU-4 score did not improve over 10 consecutive epochs. All experiments were
conducted using two Nvidia GeForce RTX 3090 GPUs.

Table 1:  Natural Language Generation (NLG) and Clinical Efficacy (CE) metrics
on MIMIC-CXR. The Best and second-best results of each metric are shown in
bold and underline, respectively. $\dagger$ indicates the results are cited
from their original papers. Since our study involves necessary data cleaning
for longitudinal analysis, these results are not strictly comparable to ours.
Results without $\dagger$ were obtained by re-running publicly available code
on the same preprocessed dataset used in our study.

|  | NLG | CE
---|---|---|---
Method | Year | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | Precision | Recall | F1
$\mathcal{M}^{2}$Transformer [11] | 2019 | $0.352$ | $0.211$ | $0.138$ | $0.096$ | $0.128$ | $0.263$ | $0.239$ | $0.173$ | $0.173$
R2Gen [10] | 2020 | $0.339$ | $0.211$ | $0.143$ | $0.103$ | $0.138$ | $0.279$ | $0.297$ | $0.189$ | $0.193$
R2GenCMN [9] | 2021 | $0.345$ | $0.213$ | $0.143$ | $0.101$ | $0.140$ | $0.274$ | $0.354$ | $0.271$ | $0.275$
$\mathcal{M}^{2}$TR.PROGRESSIVE [28] | 2021 | $0.349$ | $0.204$ | $0.132$ | $0.091$ | $0.124$ | $0.255$ | $0.220$ | $0.209$ | $0.236$
XProNet [48] | 2022 | $0.303$ | $0.188$ | $0.127$ | $0.091$ | $0.128$ | $0.268$ | 0.419 | $0.230$ | $0.242$
CvT-212DistilGPT2 [27] | 2022 | 0.372 | 0.231 | 0.155 | 0.111 | 0.149 | 0.280 | $0.417$ | $0.295$ | $0.306$
DCL [21] | 2023 | $0.263$ | $0.153$ | $0.099$ | $0.071$ | $0.117$ | $0.211$ | $0.303$ | $0.232$ | $0.229$
HERGen(Ours) | 2024 | 0.395 | 0.248 | 0.169 | 0.122 | 0.156 | 0.285 | $0.415$ | 0.301 | 0.317
Results below are not strictly comparable due to our dataset preprocessing.
For reference only.
$\mathrm{PPKED}^{\dagger}$ [23] | 2021 | 0.360 | 0.224 | 0.149 | 0.106 | 0.149 | 0.284 | $-$ | $-$ | $-$
$\mathrm{ContrastiveAttention}^{\dagger}$ [25] | 2021 | 0.350 | 0.219 | 0.152 | 0.109 | 0.151 | 0.283 | $0.352$ | $0.298$ | $0.303$
$\mathrm{AlignTransformer}^{\dagger}$ [55] | 2021 | 0.378 | 0.235 | 0.156 | 0.112 | 0.158 | 0.283 | $-$ | $-$ | $-$
$\mathrm{RGRG}^{\dagger}$ [42] | 2023 | 0.373 | 0.249 | 0.175 | 0.126 | 0.168 | 0.264 | $-$ | $-$ | $-$
$\mathrm{KIUT}^{\dagger}$ [16] | 2023 | $0.393$ | $0.243$ | $0.159$ | $0.113$ | $0.160$ | $0.285$ | $0.371$ | $0.318$ | $0.321$
$\mathrm{METransformer}^{\dagger}$ [50] | 2023 | $0.386$ | $0.250$ | $0.169$ | $0.124$ | $0.152$ | $0.291$ | $0.364$ | $0.309$ | $0.311$

###  4.3 Results of Radiology Report Generation

Results on MIMIC-CXR. Our model exhibits excellent radiology report generation
capabilities, outperforming state-of-the-art models in both Natural Language
Generation (NLG) and Clinical Efficiency (CE) metrics, as shown in Table 1.
For NLG metrics, it notably surpasses all baseline models, notably improving
over the second-best model, CvT-212DistilGPT2, by significant margins.
Specifically, compared with CvT-212DistilGPT2, our model achieves a
$\Delta+5.9\%$ overall improvement on the averaged NLG metrics compared with
CvT-212DistilGPT2. In CE metrics, our model enhances recall and F1 by
$\Delta+2.0\%$ and $\Delta+3.6\%$, respectively, compared to the second-best
results. Our precision score of $0.415$ closely approaches the best score of
$0.419$. Additionally, we incorporate micro-based metrics for five common
observations, following the methodologies of other studies [26, 42], to
provide further evaluation of our method. These results are available in the
Supplementary Material. Furthermore, our statistical analysis verifies that
our model significantly outperforms the second-best approach, as detailed in
Table 4.

Results on Longitudinal MIMIC-CXR. Table 2 presents a comparison of our model
against various baseline methods in terms of Natural Language Generation (NLG)
and Clinical Efficiency (CE) metrics. Our model outperforms both single-image
and longitudinal-image-based methods in all evaluated NLG and CE metrics.
Notably, our model achieves an increase of $\Delta+6.5\%$ on the averaged NLG
metrics compared with the second-best approach CvT-212DistilGPT2. In terms of
CE metrics, our model also outperforms CvT-212DistilGPT2 in all cases,
achieving the improvements of $\Delta+14.7\%$ in precision, $\Delta+12.0\%$ in
recall, and $\Delta+13.0\%$ in F1 score, respectively. Notably, our model also
significantly surpasses longitudinal-image-based baseline [59], which also
utilizes prior images and reports for current report generation, underscoring
the effectiveness of our proposed temporal data integration strategy.

Table 2:  Results of NLG metrics (BLEU (BL), METEOR (M), ROUGE-L ($R_{L}$))
and CE metrics (Precision, Recall and F1) on the Longitudinal MIMIC-CXR
dataset. Results marked with a dagger ($\dagger$) are cited from published
literature Prefilling [59]. Since our curation of the Longitudinal-MIMIC
dataset aligns with the approach in Prefilling [59], these results are
directly comparable to ours.

| NLG | CE
---|---|---
Method | BL-1 | BL-2 | BL-3 | BL-4 | M | $R_{L}$ | Precision | Recall | F1
Baselines based on single images |  |  |
$\mathrm{AoANet}^{\dagger}$ [14] | $0.272$ | $0.168$ | $0.112$ | $0.080$ | $0.115$ | $0.249$ | - | - | -
$\mathrm{CNN+Trans}^{\dagger}$ | $0.299$ | $0.186$ | $0.124$ | $0.088$ | $0.120$ | $0.263$ | - | - | -
$\mathrm{Transformer}^{\dagger}$ | $0.294$ | $0.178$ | $0.119$ | $0.085$ | $0.123$ | $0.256$ | - | - | -
$\mathrm{R2Gen}^{\dagger}$ [10] | $0.302$ | $0.183$ | $0.122$ | $0.087$ | $0.124$ | $0.259$ | - | - | -
$\mathrm{R2CMN}^{\dagger}$ [9] | $0.305$ | $0.184$ | $0.122$ | $0.085$ | $0.126$ | $0.265$ | - | - | -
CvT-212DistilGPT2 [27] | $0.365$ | $0.226$ | $0.151$ | $0.107$ | $0.143$ | $0.275$ | $0.367$ | $0.258$ | $0.261$
Baselines based on longitudinal images |  |  |
$\mathrm{Prefilling}^{\dagger}$ [59] | $0.343$ | $0.210$ | $0.140$ | $0.099$ | $0.137$ | $0.271$ | - | - | -
HERGen(Ours) | 0.389 | 0.242 | 0.163 | 0.117 | 0.155 | 0.282 | 0.421 | 0.289 | 0.295

###  4.4 Results of Temporal Image Classification

The temporal image classification performance on MS-CXR-T is shown in Table.
3. We divided the dataset into training, validation, and test sets with a
$70\%$ / $10\%$ / $20\%$ ratio. In the finetuning phrase, we employ our
pretrained image encoder and group causal transformer (these two modules
remain frozen) to extract representations from pairs of images, and then only
train a linear layer to make predictions. It is observed that HERGen achieves
the best performance across $4$ diseases and achieves the second-best
performance on edema. Specifically, our model improve the macro-accuracy than
the second best results by $\Delta+11.1\%$, $\Delta+4.9\%$, $\Delta+14.7\%$
and $\Delta+3.7\%$ on consolidation, pleural effusion, pneumonia,
pneumothorax, respectively. These advancements further underscore the
effectiveness of our proposed group causal transformer in capturing the
progression of diseases and extracting semantics from longitudinal studies.

Table 3:  Temporal medical image classification performance on MS-CXR-T.
Macro-accuracy ([$\%$]) are used as the metric. The Best and second-best
results are shown in bold and underline, respectively. Note that Pl.effusion
denotes pleural effusion.

Method | Pre-train | Consolidation | Pl. effusion | Pneumonia | Pneumothorax | Edema
---|---|---|---|---|---|---
Random | - | 32.3 | 31.6 | 30.3 | 39.0 | 34.9
ResNet | ImageNet | 37.5 | 39.0 | 48.4 | 45.3 | 42.5
BioViL [7] | Static | 42.9 | 41.4 | 47.9 | 42.8 | 40.7
BioViL-T [5] | Temporal | 45.0 | 46.3 | 52.0 | 50.1 | 52.0
HERGen(Ours) | Temporal | 56.1 | 51.2 | 66.7 | 54.8 | 48.1

Table 4:  Comparison of CvT-212DistilGPT2 and HERGen with $95\%$ confidence
intervals, which are computed using non-parametric bootstrap.

Method | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L
---|---|---|---|---|---|---
CvT-212DistilGPT2 | 0.372 | 0.231 | 0.155 | 0.111 | 0.280 | 0.149
(0.367, 0.377) | (0.227, 0.235) | (0.151, 0.158) | (0.107, 0.114) | (0.276, 0.283) | (0.147, 0.151)
HERGen(Ours) | 0.396 | 0.248 | 0.168 | 0.122 | 0.285 | 0.156
(0.392, 0.400) | (0.244, 0.252) | (0.164, 0.172) | (0.118, 0.125) | (0.281, 0.288) | (0.154, 0.157)
Gains | +0.023 | +0.017 | +0.014 | +0.011 | +0.005 | +0.006
(+0.017, +0.030) | (+0.011, +0.023) | (+0.008, +0.019) | (+0.005, +0.016) | (0.0, +0.01) | (+0.003, +0.009)

Table 5:  Ablation study of different components (“CL" represents the
auxiliary contrastive alignment module, and “Temporal" denotes the group
causal transformer for capturing longitudinal information). On each dataset,
the row 1, 2 and 4 corresponds to the Stage 1, Stage 2, and Stage 3 of our
curriculum learning strategy, respectively. Row 3 showcases a variant trained
using our temporal approach without the contrastive learning component for
comparison. The relative improvements in the average of all NLG metrics
compared with baseline is presented in the “AVG.$\Delta$" column.

Dataset | CL | Temporal | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | AVG.$\Delta$
---|---|---|---|---|---|---|---|---|---
MIMIC-CXR |  |  | 0.372 | 0.231 | 0.155 | 0.111 | 0.149 | 0.280 | $-$
✓ |  | 0.390 | 0.239 | 0.161 | 0.117 | 0.153 | 0.280 | $+3.3\%$
| ✓ | $0.388$ | $0.240$ | $0.162$ | $0.116$ | $0.153$ | $0.283$ | $+3.4\%$
✓ | ✓ | 0.395 | 0.248 | 0.169 | 0.122 | 0.156 | 0.285 | $+5.9\%$
Lon-MIMIC |  |  | $0.365$ | $0.225$ | $0.151$ | $0.107$ | $0.143$ | $0.275$ | $-$
✓ |  | $0.375$ | $0.232$ | $0.155$ | $0.110$ | $0.146$ | $0.277$ | $+2.3\%$
| ✓ | $0.380$ | $0.237$ | $0.160$ | $0.115$ | $0.153$ | 0.283 | $+4.9\%$
✓ | ✓ | 0.389 | 0.242 | 0.163 | 0.117 | 0.155 | $\underline{0.282}$ | $+6.5\%$

###  4.5 Ablated Analysis of Our Framework

Effect of Auxiliary Contrastive Alignment. We delve into the impacts of
incorporating our auxiliary contrastive alignment module, as delineated in
Table 5. It is observed that incorporating the contrastive learning objective
yields improvements in all NLG metrics compared to the baseline for both
datasets, suggesting enhanced consistency in report generation. Notably, we
observed a augmentation of $+3.3\%$ in average NLG metrics for the MIMIC-CXR
dataset and a $+2.3\%$ improvement for the Longitudinal MIMIC-CXR dataset,
underscoring the value of our contrastive alignment module in report
generation.

Effect of Temporal Aggregation Module. We evaluate the impact of integrating
our temporal aggregation module in Table 5. It is observed that this module
significantly enhances the NLG metrics across the MIMIC-CXR and Longitudinal
MIMIC-CXR datasets. Specifically, on the Longitudinal MIMIC-CXR dataset, it
achieves $+4.9\%$ improvement in the averaged NLG metrics, compared to the
baseline upon integrating this module. When combined with a model trained
using contrastive learning, the improvement on the averaged NLG metrics
compared with baseline further increases to $+6.5\%$, which marks a
significant enhancement than $+2.3\%$ (Row 2). This pattern is consistent
across datasets, underscoring the temporal aggregation module’s effectiveness
in leveraging patient histories for generating more accurate reports.

Effect of Curriculum Learning Strategy. We evaluate the impact of our
curriculum learning strategy on model performance in Table 5. Our analysis
reveals that, across both the MIMIC-CXR and Longitudinal MIMIC-CXR datasets,
the models incorporating contrastive learning alignment consistently
outperform the baseline. Furthermore, our final model, which integrates both
contrastive learning and temporal aggregation module, shows the best
performance across the majority of metrics, highlighting the combined benefits
of these approaches. For a detailed comparison between joint and curriculum-
based training, please refer to the Supplementary Material.

(a) CvT-212DistilGPT2

(b) Our approach

Figure 5:  Embedding visualization of MIMIC-CXR images in CvT-212DistilGPT2
and our model with t-SNE.  Figure 6:  This case study compares radiology
report predictions for a patient by our model and CvT-212DistilGPT2. Text
highlighted in gray indicates words or their synonyms found in both the
predicted and ground truth reports. Purple highlights denote similar matches
in the baseline-generated (CvT-212DistilGPT2) reports and ground truth, while
red highlights show similar matches in our model’s reports and ground truth.
From top to bottom, the chest X-rays are chronologically ordered. Here $T$
denotes the study date of the first study.

###  4.6 Qualitative Results

Case Study of Generated Reports. Fig. 6 presents a case study comparing
reports generated by our model with those from CvT-212DistilGPT2 for a given
patient. The comparison shows that reports from our model align more clinical
findings with the ground truth. Moreover, our model correctly generates more
comparative statements, such as “appear stable" or “appear unchanged",
suggesting its superiority to capture temporal information. These findings
underscore our model’s proficiency in report generation by (1) identifying
disease-specific features through consistent anatomical structures in patient-
level CXRs and (2) generating time-comparative sentences.

Visualization of Learned Embeddings. Fig. 5 shows T-SNE visualization of image
embeddings from a dataset constructed following $5\times 200$ MIMIC-CXR
dataset proposed in MedCLIP [51]. We calculated the $50$-th percentile of the
cosine similarity matrix within each class, revealing that our model achieves
a higher average similarity (0.0919) compared to CvT-212DistilGPT2 (0.0245),
indicating better disease-specific feature extraction.

##  5 Conclusion

In this paper, we present a novel framework to enhance radiology report
generation by utilizing the varying-size patient histories. By integrating a
novel group casual transformer, our model effectively aggregates temporal
information of longitudinal data. Besides, our framework optimize an auxiliary
contrastive alignment module to further align image and textual data.
Moreover, a curriculum learning strategy is employed to sequentially optimize
these modules, thereby progressively improving model performance. Our
extensive experiments demonstrate the model’s capability to generate
clinically precise reports and extract meaningful insights from historical
data.

Limitations and Future Work. One potential limitation of our method is that
the model’s alignment operates within the embedding space without accounting
for anatomical consistencies in longitudinal studies. Additionally, we plan to
expand HERGen into a more comprehensive representation learning model, thereby
broadening its utility across varied downstream tasks.

## Acknowledgement

This work was partially supported by the Research Grants Council of Hong Kong
(27206123 and T45-401/22-N), the Hong Kong Innovation and Technology Fund
(ITS/273/22), and the National Natural Science Foundation of China (No.
62201483).

## References

  * [1] Alfarghaly, O., Khaled, R., Elkorany, A., Helal, M., Fahmy, A.: Automated radiology report generation using conditioned transformers. Informatics in Medicine Unlocked 24, 100557 (2021)
  * [2] Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.H., Jin, D., Naumann, T., McDermott, M.: Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019)
  * [3] Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
  * [4] Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., de Castro, D.C., Boecking, B., Sharma, H., Bouzid, K., Schwaighofer, A., et al.: Ms-cxr-t: Learning to exploit temporal structure for biomedical vision-language processing (2023)
  * [5] Bannur, S., Hyland, S., Liu, Q., Perez-Garcia, F., Ilse, M., Castro, D.C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al.: Learning to exploit temporal structure for biomedical vision-language processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15016–15027 (2023)
  * [6] Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)
  * [7] Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: European conference on computer vision. pp. 1–21. Springer (2022)
  * [8] Cao, D.J., Hurrell, C., Patlas, M.N.: Current status of burnout in canadian radiology. Canadian Association of Radiologists Journal 74(1), 37–43 (2023)
  * [9] Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. arXiv preprint arXiv:2204.13258 (2022)
  * [10] Chen, Z., Song, Y., Chang, T.H., Wan, X.: Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056 (2020)
  * [11] Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10578–10587 (2020)
  * [12] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [13] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3(1), 1–23 (2021)
  * [14] Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4634–4643 (2019)
  * [15] Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3942–3951 (2021)
  * [16] Huang, Z., Zhang, X., Zhang, S.: Kiut: Knowledge-injected u-transformer for radiology report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19809–19818 (2023)
  * [17] Jing, B., Wang, Z., Xing, E.: Show, describe and conclude: On exploiting the structure information of chest x-ray reports. arXiv preprint arXiv:2004.12274 (2020)
  * [18] Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint arXiv:1711.08195 (2017)
  * [19] Johnson, A., Lungren, M., Peng, Y., Lu, Z., Mark, R., Berkowitz, S., Horng, S.: Mimic-cxr-jpg-chest radiographs with structured labels. PhysioNet (2019)
  * [20] Karwande, G., Mbakwe, A.B., Wu, J.T., Celi, L.A., Moradi, M., Lourentzou, I.: Chexrelnet: An anatomy-aware model for tracking longitudinal relationships between chest x-rays. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 581–591. Springer (2022)
  * [21] Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., Chang, X.: Dynamic graph enhanced contrastive learning for chest x-ray report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3334–3343 (2023)
  * [22] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)
  * [23] Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13753–13762 (2021)
  * [24] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [25] Ma, X., Liu, F., Yin, C., Wu, X., Ge, S., Zou, Y., Zhang, P., Sun, X.: Contrastive attention for automatic chest x-ray report generation. arXiv preprint arXiv:2106.06965 (2021)
  * [26] Miura, Y., Zhang, Y., Tsai, E.B., Langlotz, C.P., Jurafsky, D.: Improving factual completeness and consistency of image-to-text radiology report generation. arXiv preprint arXiv:2010.10042 (2020)
  * [27] Nicolson, A., Dowling, J., Koopman, B.: Improving chest x-ray report generation by leveraging warm starting. Artificial Intelligence in Medicine 144, 102633 (2023)
  * [28] Nooralahzadeh, F., Gonzalez, N.P., Frauenfelder, T., Fujimoto, K., Krauthammer, M.: Progressive transformer-based generation of radiology reports. arXiv preprint arXiv:2102.09777 (2021)
  * [29] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [30] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)
  * [31] Pavlopoulos, J., Kougia, V., Androutsopoulos, I., Papamichail, D.: Diagnostic captioning: a survey. Knowledge and Information Systems 64(7), 1691–1722 (2022)
  * [32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [33] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [34] Ramesh, V., Chi, N.A., Rajpurkar, P.: Improving radiology report generation systems by removing hallucinated references to non-existent priors. In: Machine Learning for Health. pp. 456–473. PMLR (2022)
  * [35] Raoof, S., Feigin, D., Sung, A., Raoof, S., Irugulpati, L., Rosenow III, E.C.: Interpretation of plain chest roentgenogram. Chest 141(2), 545–558 (2012)
  * [36] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
  * [37] Rimmer, A.: Radiologist shortage leaves patient care at risk, warns royal college. BMJ: British Medical Journal (Online) 359 (2017)
  * [38] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
  * [39] Serra, F.D., Wang, C., Deligianni, F., Dalton, J., O’Neil, A.Q.: Controllable chest x-ray report generation from longitudinal representations. arXiv preprint arXiv:2310.05881 (2023)
  * [40] Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A.Y., Lungren, M.P.: Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint arXiv:2004.09167 (2020)
  * [41] Sorower, M.S.: A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis 18(1), 25 (2010)
  * [42] Tanida, T., Müller, P., Kaissis, G., Rueckert, D.: Interactive and explainable region-guided radiology report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7433–7442 (2023)
  * [43] Thrall, J.H., Li, X., Li, Q., Cruz, C., Do, S., Dreyer, K., Brink, J.: Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success. Journal of the American College of Radiology 15(3), 504–508 (2018)
  * [44] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [45] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
  * [46] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2015)
  * [47] Wang, F., Zhou, Y., Wang, S., Vardhanabhuti, V., Yu, L.: Multi-granularity cross-modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems 35, 33536–33549 (2022)
  * [48] Wang, J., Bhalerao, A., He, Y.: Cross-modal prototype driven network for radiology report generation. In: European Conference on Computer Vision. pp. 563–579. Springer (2022)
  * [49] Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M.: Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9049–9058 (2018)
  * [50] Wang, Z., Liu, L., Wang, L., Zhou, L.: Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11558–11567 (2023)
  * [51] Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163 (2022)
  * [52] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: Cvt: Introducing convolutions to vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 22–31 (2021)
  * [53] Wu, J.T., Agu, N.N., Lourentzou, I., Sharma, A., Paguio, J.A., Yao, J.S., Dee, E.C., Mitchell, W., Kashyap, S., Giovannini, A., et al.: Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv:2108.00316 (2021)
  * [54] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057. PMLR (2015)
  * [55] You, D., Liu, F., Ge, S., Xie, X., Zhang, J., Wu, X.: Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24. pp. 72–82. Springer (2021)
  * [56] You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4651–4659 (2016)
  * [57] Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D.: When radiology report generation meets knowledge graph. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12910–12917 (2020)
  * [58] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference. pp. 2–25. PMLR (2022)
  * [59] Zhu, Q., Mathai, T.S., Mukherjee, P., Peng, Y., Summers, R.M., Lu, Z.: Utilizing longitudinal chest x-rays and reports to pre-fill radiology reports. arXiv preprint arXiv:2306.08749 (2023)