# Attosecond imaging of photo-induced dynamics in molecules using time-
resolved photoelectron momentum microscopy

Marvin Reuner  I. Institute for Theoretical Physics and Centre for Free-
Electron Laser Science, Universität Hamburg, Notkestraße 9, 22607 Hamburg,
Germany  Daria Popova-Gorelova  I. Institute for Theoretical Physics and
Centre for Free-Electron Laser Science, Universität Hamburg, Notkestraße 9,
22607 Hamburg, Germany  The Hamburg Centre for Ultrafast Imaging (CUI),
Luruper Chaussee 149, 22607 Hamburg, Germany

###### Abstract

We explore the novel capabilities offered by attosecond extreme ultraviolet
and x-ray pulses that can be now generated by free-electron lasers and high-
harmonics generation sources for probing photon-induced electron dynamics in
molecules. We theoretically analyze how spatial and temporal dependence of
charge migration in a pentacene molecule can be followed by means of time-
resolved photoelectron microscopy on the attosecond time scale. Performing the
analysis, we accurately take into account that an attosecond probe pulse leads
to considerable spectral broadening. We demonstrate that the excited-state
dynamics of a neutral pentacene molecule in the real space map onto unique
features of photoelectron momentum maps.

††preprint: APS/123-QED

##  I Introduction

Photoelectron momentum microscopy is the technique, in which a light pulse
ionizes a probe leading to the detachment of a photoelectron. The momentum
distribution of the detached photoelectron encodes information about the shape
of an orbital or a bond in the real space from which the electron was detached
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. This connection is facilitated by the
Fourier transform, which links the real and momentum space. The spatial
resolution of this technique is given by the de-Broglie wavelength of the
emitted photoelectron, which is determined by the kinetic energy of the
photoelectron. If an extreme ultraviolet (XUV) probe pulse detaches an
electron from a valence orbital, the photoelectron would also have a kinetic
energy in the XUV range corresponding to the resolution of a few Angstroms.

In recent years, there has been a remarkable progress in atomic-scale imaging
of photo-excited states and dynamics using photoelectron momentum microscopy.
Momentum-space distribution of transiently excited electrons was measured
[13], the dynamics of crystalline bonds have been followed during the photo-
induced phase transition [9] and the technique allowed to detect singlet
fission with orbital resolution [14]. Time-resolved photoelectron momentum
microscopy probing photo-induced dynamics of pentacene films on sub-picosecond
time scale has been achieved at the free-electron laser FLASH [15]. So far,
these experiments were performed at a sub-picosecond time scale and were not
capable of capturing pure electron dynamics in real time on its natural time
scale. This barrier is now feasible to overcome due to the new capabilities to
produce attosecond x-ray pulses at free-electron lasers [16, 17, 18] or
attosecond XUV pulses using high-harmonics generation sources [19, 20, 21, 22,
23, 24, 25]. These sources have already been successfully employed to study
electronic processes with few-femtosecond to attosecond time and atomic-scale
space resolution [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36], and new
experimental schemes are being explored theoretically [37, 38, 39, 40, 41, 42,
43, 44, 45, 46, 47, 48, 49]. The interpretation of a signal from attosecond
probe pulses is challenging, since they have a broad bandwidth in the energy
domain, which smears out spectral lines in a signal. In this article, we
analyze how attosecond XUV pulses can be employed to measure charge migration
in a pentacene molecule and analyze how to extract the real-space and real-
time information about charge migration using the momentum-resolved
photoelectron microscopy.

Pentacene is a prototypical organic semiconductor composed of pentacene
molecules, which is among candidate materials for efficient organic
photovoltaics [50, 51]. The understanding of the process of solar-energy
conversion in organic semiconductors strongly relies on the insight into both
intra- and intermolecular photo-excited processes [52]. Exciton migration is
one of the fundamental processes governing solar-energy conversion and the
role of electronic coherence processes becomes especially relevant near
conical intersections [53, 54, 55]. Electronic charge-migration dynamics
launched as a coherent superposition of electronic states have already been
studied in detail [56, 57, 58, 59, 60, 61, 62, 63, 64, 65]. In this paper, we
focus on the description of how such coherent electron dynamics can be probed
on nanoscale by broadband probe pulses. Particularly, we study how attosecond
photoelectron momentum microscopy can be employed to probe intramolecular
coherent exciton dynamics in pentacene.

It has been shown that time-resolved photoelectron spectroscopy can be applied
to probe ultrafast electronic and nuclear molecular dynamics [66, 67, 68, 43,
69, 70, 71]. However, angle-unresolved photoelectron spectra do not carry any
spatial information about the dynamics, which is lost due to angle averaging.
In photoelectron momentum microscopy, momentum distribution of photoelectrons
is detected. The great advantage of this technique over angle-averaged
spectroscopy is that the distribution provides details of electron dynamics
within a molecule, if photoelectrons are detached with high kinetic energies.
We will also show that photoelectron spectra can loose important time-resolved
information about the dynamics after angle averaging.

In this article, we apply the general theoretical formalism to describe the
time and momentum-depended photoelectron probability by a broad-bandwidth
probe pulse derived in Ref. [7]. That study proposed to probe hole dynamics in
a positively ionized molecule by emitting electrons from orbitals that were
initially occupied. Here, we propose to apply attosecond XUV pulses to probe
photo-excited states of a neutral molecule, which is especially relevant to
reveal information about exciton dynamics in the energy conversion processes.

The article is organized as follows. We develop the theoretical description of
time-resolved photoelectron momentum microscopy of excited-state dynamics in
molecules taking electron correlations and broad-bandwidth of a probe pulse
into account in Sec. II. In Sec. III, we calculate photoelectron spectra and
photoelectron momentum maps from coherent electron-hole dynamics in a
pentacene molecule obtained by an attosecond XUV probe pulse. We reveal how
time-resolved atomic-scale changes in electron density are correlated with
time-resolved changes in photoelectron momentum maps.

##  II Method

Throughout this article, we consider a molecule with coherently evolving
electron dynamics. In this case, the $N_{\text{el}}$-electron wave function of
a molecule is a coherent superposition of several excited electronic
eigenstates $\left|{\Phi_{I}}\right>$

$\left|{\Psi(t)}\right>=\sum_{I}C_{I}e^{-iE_{I}(t-t_{0})}\left|{\Phi_{I}}\right>,$
(1)

where $E_{I}$ is the corresponding eigenenergy and the coefficient $C_{I}$
determines the population distribution for each eigenstate. The coherent
superposition has emerged at time $t_{0}$ due to crossing a conical
intersection or the excitation by a pump pulse. We assume that an ultrashort
XUV probe pulse interacts with the molecule at time $t_{p}$, which leads to an
emission of a photoelectron. The momentum distribution of the photoelectron
encodes information about the electronic state at the time $t_{p}$. In our
study, we propose to probe electron dynamics by studying the momentum
distribution of photoelectrons emitted from outermost orbitals that became
populated due to the photoexcitation of the molecule. We use atomic units for
this and following expressions.

Assuming a probe pulse with the central photon energy $\omega_{\text{in}}$,
polarization $\bm{\epsilon}_{\text{in}}$, pulse duration $\tau_{p}$ and
intensity profile $I(t)=I_{0}e^{-4\ln{2}[(t-t_{p})/\tau_{p}]^{2}}$, the
general expression for the photoelectron probability is [7]

$\displaystyle\begin{split}P(\mathbf{q},t_{p})=&\frac{\tau_{p}^{2}I_{0}\left|\bm{\epsilon}_{\text{in}}\cdot\mathbf{q}\right|^{2}}{8\pi\ln
2\omega_{\text{in}}^{2}c}\sum_{F,\sigma}e^{-(\Omega_{F}-\epsilon_{e})^{2}\tau_{p}^{2}/(4\ln
2)}\\\ &\times\left|\chi_{\sigma}^{\dagger}\int\mathrm{d}^{3}r\
e^{-i\mathbf{q}\cdot\mathbf{r}}\phi_{F}^{D}(\mathbf{r},t_{p})\right|^{2},\end{split}$
(2)

where $\mathbf{q}$ is the momentum of the photoelectron with the corresponding
energy $\epsilon_{e}=\frac{|\mathbf{q}|^{2}}{2}$ and $\chi_{\sigma}$ is the
photoelectron spin state. The summation is over all possible final states of
the molecule $\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|$ that can be produced
after the photoionization by the probe pulse, $E_{F}$ is their corresponding
eigenenergy.
$\phi_{F}^{D}(\mathbf{r},t_{p})=\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|\hat{\psi}(\mathbf{r})\left|{\Psi(t_{p})}\right>$
is the Dyson orbital defined as the overlap between the $N_{el}$ wavefunction
of the initial state of the system and the $(N_{el}-1)$ wavefunction of the
final state, where $\hat{\psi}(\mathbf{r})$ is the electron annihilation field
operator. $\Omega_{F}=\omega_{\text{in}}+\left<E\right>-E_{F}$, where
$\left<E\right>$ is the mean energy of the eigenstates involved in the wave-
packet in Eq. (1). Here, we assumed the sudden approximation, i. e. the
molecular state after photoionization and the state of the photoelectron are
decoupled, which is justified for high kinetic photoelectron energies [72]. It
is also assumed that the probe-pulse duration is much shorter than the
characteristic time scale of the electron dynamics. This means that the energy
splittings of the eigenstates involved in the dynamics are much smaller than
the bandwidth of the probe pulse and can be substituted by their mean energy.
If this assumption fails, the eigenenergies must enter the expression
explicitly and Eq. (2) gets slightly modified

$\displaystyle P(\mathbf{q},t_{p})\propto\sum_{F,\sigma}\Biggl{|}$
$\displaystyle\sum_{I}e^{-(\omega_{\text{in}}+E_{I}-E_{F}-\epsilon_{e})^{2}\tau_{p}^{2}/(8\ln
2)}$ (3) $\displaystyle\times\chi_{\sigma}^{\dagger}\int\mathrm{d}^{3}r\
e^{-i\mathbf{q}\cdot\mathbf{r}}\phi_{FI}^{D}(\mathbf{r},t_{p})\Biggr{|}^{2},$

where
$\phi_{FI}^{D}(\mathbf{r},t_{p})=\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|\hat{\psi}(\mathbf{r})\left|{C_{I}e^{-iE_{I}(t_{p}-t_{0})}}\right>$.
Time- and angle-resolved photoelectron probability for molecules with electron
dynamics has also been previously theoretically studied [73, 66, 74, 75]. Eqs.
(2) and (3) take a broad bandwidth of a probe pulse accurately into account,
which was not the case in these studies.

It follows from the expression Eq. (2) that the momentum-resolved
photoelectron probability encodes the Fourier transform of the Dyson orbital.
If the molecule is in the ground state, the overlap integral between a singly
ionized molecule and the molecule before the ionization is well approximated
by the Hartree-Fock (HF) orbital, from which an electron was detached. Due to
this connection, photoelectron momentum microscopy is used as an orbital
imaging method [76, 77, 78].

When a molecule in an excited state is ionized by a probe pulse, the Dyson
orbital cannot be approximated by a molecular orbital and the interpretation
of the Dyson orbital becomes more challenging. In order to carefully describe
the excited state dynamics as well as accurately treat the Dyson orbital, we
describe the excited states involved in the dynamics $\Phi_{I}$ and the states
of the ionized molecule $\Phi_{F}^{N_{\text{el}}-1}$ using the configuration
interaction method. We express each excited state based on the configuration
interaction approach limited to single excitations as

$\left|{\Phi_{I}}\right>=\sum_{i_{1},a_{1}}\tilde{c}_{i_{1}}^{a_{1}}(I)\left|{\phi_{i_{1}}^{a_{1},S(I)}}\right>,$
(4)

where $\left|{\phi_{i_{1}}^{a_{1},S(I)}}\right>$ denotes a configuration state
function (CSF), where an electron of the HF ground state has been excited from
the orbital $i_{1}$ to the orbital $a_{1}$. Such a state has a hole in orbital
$i_{1}$ and an electron in orbital $a_{1}$, and $S(I)$ describes the total
spin and the spin projection value. For the description of the final state, we
use a similar approach, but add one additional hole in the configuration
space. Thus, the final state is

$\begin{split}\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|=\sum_{i_{2},j_{2},a_{2}}&\tilde{c}_{i_{2},j_{2}}^{a_{2}}(F)\left<{\phi_{i_{2},j_{2}}^{a_{2},S(F)}}\right|\\\
&+\sum_{i_{2}}\tilde{c}_{i_{2}}(F)\left<{\phi_{i_{2}}^{S(F)}}\right|,\end{split}$
(5)

where $\left<{\phi_{i_{2},j_{2}}^{a_{2},S(F)}}\right|$ describes a CSF which
contains two holes in orbitals $i_{2}$ and $j_{2}$ and an electron in orbital
$a_{2}$. We denote an $N_{\text{el}}-1$-electron CSF with one hole among
originally occupied orbitals of the molecule (HOMO, HOMO-1, …) and no
electrons in the originally unoccupied molecular orbitals (LUMO, LUMO+1, …) as
$\left<{\phi_{i_{2}}^{S(F)}}\right|$. In both cases, $S(F)$ describes the
total spin and the spin projection value of the final state $F$. The
coefficients $\tilde{c}_{i_{2},j_{2}}^{a_{2}}(F)$ and $\tilde{c}_{i_{2}}(F)$
determine the contribution of a CSF to the final state. We perform
calculations employing RASSCF method [79] and use the same active orbitals for
the treatment of the states $\left|{\Phi_{I}}\right>$ and
$\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|$. With these approximations, the
overlap integrals between neutral and singly-ionized eigenstates of a molecule
are given by a linear combination of the overlap integrals between
$N_{\text{el}}$\- and $(N_{\text{el}}-1)$-electron CSFs. Such an overlap
integral provide either a zero or a molecule orbital, which is occupied in a
$N_{\text{el}}$-electron CSF and unoccupied in a $(N_{\text{el}}-1)$-electron
CSF. Thus, a Dyson orbital can be represented as a linear combination of
molecular orbitals.

##  III Imaging photo-induced dynamics in pentacene

Based on the previous considerations, we present an example for imaging
excited-state dynamics in aligned isolated pentacene molecules. The molecular
alignment can be achieved either for gas-phase molecules or molecular films
adsorbed on substrates. We recently observed that photo-excited dynamics of
pentacene molecules in a top pentacene layer of a bilayer pentacene films
adsorbed on an Ag(110) surface behave similarly to the dynamics of an isolated
pentacene [15]. These dynamics were probed using time-resolved photoelectron
momentum microscopy on a time scale of several hundreds of femtoseconds. This
experiment established conditions for time-resolved photoelectron microscopy
at free-electron lasers, which makes it feasible to perform such experiments
at much shorter timescales with brilliant and ultrashort light pulses from
free-electron laser sources.

In our study, we assume that coherent electron dynamics in pentacene were
excited by a pump pulse via an optical excitation from the ground state. Since
direct optical transitions from the ground state to triplet excited states are
dipole forbidden, we truncate our considerations to spin-singlet excited
states.

We calculate the vertical excited spin-singlet states of pentacene with MOLCAS
[80] using the RASSCF method [79], with a CC-PVDZ basis set for the atoms [81,
82]. The calculation of the first excited states of pentacene is converged
with an active space of 22 orbitals, containing 11$\pi$ and 11$\pi^{*}$
orbitals. Including more orbitals in the active space does not lead to
significant changes in the energy spectrum and the structure of the
eigenstates. The $\pi$ orbitals contain a maximum of one hole, while the
$\pi^{*}$ orbitals have a maximum of one electron. The energies of the first
four excited states and the changes in the occupation of the orbitals
(compared to the HF ground state) are shown in Fig. 1. We find that the third
and the fourth excited singlet states are dark states and the next bright
singlet states are energetically highly separated from the first and the
second excited state. Thus, one would be able to create a coherent
superposition of just two excited states of a pentacene in a possible
experiment. When the coherent superposition of the two excited states of
pentacene is created, its electron density starts to oscillate in time. The
oscillation period is determined by the energy difference between the excited
states. The energy difference and the electronic wavefunctions of the bright
excited states obtained in our calculation agree well with the experimental
and theoretical results in previous studies [83, 84, 85]. Also, the agreement
with a more accurate CASPT2 calculation [85] justifies our chosen theoretical
level of single excitation in the configuration space. We obtain that the
first excited state is predominantly characterized by the CSF obtained by the
excitation of an electron from the HOMO to the LUMO. The second excited state
is predominantly characterized by a linear combination of the CSF obtained by
the excitation of an electron from the HOMO to the LUMO+2 and the CSF obtained
by the excitation of an electron from the HOMO-2 to the LUMO.

Figure 1: Energy level diagram of the first four vertical spin-singlet excited
states of pentacene (left), with bright states in blue and dark states in red.
A wave-packet in a coherent superposition of the first two excited states has
an oscillation period of $5.73\ \text{fs}$. The corresponding orbital
excitations contribute to the wave function of the states (right), while the
arrow means an excitation from the left (HOMO-i) to the right (LUMO+j)
orbital. For the visualization of the orbitals, we used the software LUSCUS
[86]

The energy difference between the two excited states results in a oscillation
period of the electron density of $T=5.73\ \text{fs}$. To simplify our
considerations, we can neglect nuclear motions that should affect electron
dynamics on longer time scales. With Eq. (1) the time evolution of the wave-
packet after the excitation by the pump pulse at $t_{0}=0$ is

$\displaystyle\left|{\Psi(t)}\right>$
$\displaystyle=C_{1}e^{-iE_{1}t}\left|{\Phi_{1}}\right>+C_{2}e^{-iE_{2}t}\left|{\Phi_{2}}\right>,$
(6) $\displaystyle\left|{\Phi_{1}}\right>$
$\displaystyle=\left|{\phi_{H}^{L,0(0)}}\right>,$ (7)
$\displaystyle\left|{\Phi_{2}}\right>$
$\displaystyle=\frac{1}{\sqrt{2}}\left(\left|{\phi_{H}^{L+2,0(0)}}\right>-\left|{\phi_{H-2}^{L,0(0)}}\right>\right),$
(8)

with $|C_{1}|^{2}+|C_{2}|^{2}=1$. The coefficients $C_{1}$ and $C_{2}$ depend
on a particular process, which brought a molecule into a coherent
superposition of electronic states. Our results do not depend on the
particular values of the coefficients $C_{1}$ and $C_{2}$ and we assume an
equal population of both excited states, $C_{1}=C_{2}=\frac{1}{\sqrt{2}}$. The
orbital index $H-i$ denotes the $i$-th orbital below the HOMO and $L+j$ is the
$j$-th orbital above the LUMO. The photo-excitation leads to a time-dependent
change in the electron density relative to the ground state density

$\rho_{ex}(\mathbf{r},t)=\rho_{1}(\mathbf{r},t)-\rho_{0}(\mathbf{r}),$ (9)

where
$\rho_{0}(\mathbf{r})=\left<{\Phi_{0}}\right|\hat{\psi}^{\dagger}(\mathbf{r})\hat{\psi}(\mathbf{r})\left|{\Phi_{0}}\right>$
is the electron density of the ground state of pentacene described by the wave
function $\left|{\Phi_{0}}\right>$. The electron density for the pentacene
after the photo-excitation by a pump pulse is
$\rho_{1}(\mathbf{r},t)=\left<{\Psi(t)}\right|\hat{\psi}^{\dagger}(\mathbf{r})\hat{\psi}(\mathbf{r})\left|{\Psi(t)}\right>$.
We evaluate the electron density change and obtain

$\displaystyle\begin{split}\rho_{ex}(\mathbf{r},t)=&\left|C_{1}^{*}e^{iE_{1}t}\phi_{L}(\mathbf{r})+\frac{C_{2}^{*}}{\sqrt{2}}e^{iE_{2}t}\phi_{L+2}(\mathbf{r})\right|^{2}\\\
&+\frac{|C_{2}|^{2}}{2}|\phi_{L}(\mathbf{r})|^{2}-\frac{|C_{2}|^{2}}{2}|\phi_{H}(\mathbf{r})|^{2}\\\
&-\left|C_{1}^{*}e^{iE_{1}t}\phi_{H}(\mathbf{r})-\frac{C_{2}^{*}}{\sqrt{2}}e^{iE_{2}t}\phi_{H-2}(\mathbf{r})\right|^{2},\end{split}$
(10)

where $\phi_{H-i}(\mathbf{r})$ and $\phi_{L+j}(\mathbf{r})$ are the
wavefunctions of the i-th orbital below the HOMO and the j-th orbital above
the LUMO. The electron density change is shown in Fig. 2 at different times
during the oscillation period. The negative (blue) part is the reduction of
the electron density compared to the ground state density and the positive
(yellow) part is the increase of the electron density compared to the ground
state density. The time-dependent part of the electron density oscillates in
time as $\cos[(E_{1}-E_{2})t]$. From $t=0$ to $t=\frac{T}{2}$, the photo-
induced negative charge flows from the top left and bottom right of the
molecule to the top right and bottom left. The photo-induced positive charge
flows in the opposite direction. The spatial distribution of the electron
density change at $t=\frac{T}{2}$ can be mapped onto the electron density
change at $t=0$ by a reflection in the yz-plane. At two times during the
oscillation period, $t=\frac{T}{4}$ and $t=\frac{3T}{4}$, the photo-excited
change in the electron density coincides and has reflection symmetry.

Figure 2: The time evolution of the electron-hole density after the excitation of the pentacene molecule to the first two excited states with the pump pulse at $t_{0}=0\text{ fs}$. The blue isosurface shows the hole density (negative value) and the yellow the electron density (positive value). For the visualization we used the software VESTA [87]. F | $E_{F}$ | $\Omega_{F}$ | $\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|$
---|---|---|---
1 | $5.0\ \text{eV}$ | $98.9\ \text{eV}$ | $-0.95\left<{\Phi_{H}^{1/2(1/2)}}\right|$
2 | $6.7\ \text{eV}$ | $97.2\ \text{eV}$ | $-0.94\left<{\Phi_{H-2}^{1/2(1/2)}}\right|$
3 | $7.5\ \text{eV}$ | $96.4\ \text{eV}$ | $-0.83\left<{\Phi_{H,H}^{L,1/2(1/2)}}\right|+0.31\left<{\Phi_{H-4}^{1/2(1/2)}}\right|$
4 | $8.7\ \text{eV}$ | $95.2\ \text{eV}$ |  $-0.62\left<{\Phi_{H,H}^{L+1,1/2(1/2)}}\right|+0.5\left<{\Phi_{H-1,H}^{L,1/2(1/2)}}\right|_{udu}$ $+0.28\left<{\Phi_{H-1,H}^{L,1/2(1/2)}}\right|_{uud}$
5 | $9.6\ \text{eV}$ | $94.3\ \text{eV}$ | $0.70\left<{\Phi_{H-1,H}^{L,1/2(1/2)}}\right|_{uud}-0.45\left<{\Phi_{H-1,H}^{L,1/2(1/2)}}\right|_{udu}$
6 | $9.7\ \text{eV}$ | $94.2\ \text{eV}$ |  $-0.59\left<{\Phi_{H,H}^{L+2,1/2(1/2)}}\right|+0.62\left<{\Phi_{H-2,H}^{L,1/2(1/2)}}\right|_{udu}$ $-0.28\left<{\Phi_{H-2,H}^{L,1/2(1/2)}}\right|_{uud}$
Table 1:  First six final states $\left<{\Phi_{F}}\right|$ of ionized and
excited pentacene, which result in a nonzero Dyson orbital
$\phi_{F}^{D}(\mathbf{r})=\left<{\Phi_{F}^{N_{\text{el}}-1}}\right|\hat{\psi}(\mathbf{r})\left|{\Psi(t)}\right>$.
The corresponding energy is $E_{F}$ and
$\Omega_{F}=E_{F}-\left<E\right>-\omega_{\text{in}}+\epsilon_{e}$ is the
central energy in the photoelectron probability in Eq. (2) for the given final
state. The right column shows the contributing configuration state functions
of the final state. The indices $udu$ and $uud$ distinguish two different CSFs
that result in the total spin of 1/2 and spin projection of +1/2. Figure 3:
Angle-integrated photoelectron spectra for photoelectrons detached either from
the ground state $S_{0}$ or in the coherent superposition of the excited state
$S_{1}$ and $S_{2}$. The black dotted line shows the contribution at the
photoelectron energy $\epsilon_{e}=99\ \text{eV}$. The probe-pulse duration is
$0.5\ \text{fs}$ and has a central photon energy of $100\ \text{eV}$.

We assume that an XUV probe pulse with the central energy of $100\ \text{eV}$
and $500\ \text{as}$ duration photoionizes the molecule at the pump-probe time
delay $t_{p}$. The probe-pulse is linearly polarized along the $y$-direction,
which is perpendicular to the nodal plane of the molecule. For the calculation
of the final states after the photoionization by the probe pulse, we apply the
same method as in the calculation of the excited states of pentacene, but with
an additional hole in the 11 active $\pi$ orbitals. Tab. 1 shows the first six
final states of pentacene with the lowest energy that can be reached by
ionizing the initial state in Eq. (6).

If the photoionization was triggered by a pulse with a narrow bandwidth, the
photoelectron probability as a function of energy would consist of clear
spectral lines centred at energies
$\Omega_{F,I}=E_{F}-E_{I}-\omega_{\text{in}}+\epsilon_{e}$. Thus, the
photoelectron momentum distributions would be clearly assigned to a certain
Dyson orbital. In the case of the ultrashort probe pulse, each spectral line
has the spectral band width of $5.16\ \text{eV}$ and an individual Dyson
orbital contributing to the momentum distribution at a given energy is no
longer distinguished.

Let us first consider the angle-integrated photoelectron spectra. In a real
experiment, a probe pulse would interact with a number of pentacene molecules,
some of which would be excited by a pump pulse, and some of which would remain
unexcited. Thus, we first estimate, how the photoionization from unexcited
pentacene molecules would affect the photoelectron spectra from pentacene
molecules in the excited state. In Fig. 3, we show photoelectron spectra from
the pentacene molecules being in the ground state $S_{0}$ and from pentacene
molecules being in the coherent superposition of the excited state $S_{1}$ and
$S_{2}$ ($\left|{\Psi(t)}\right>$ in Eq. (6)). We obtain that the contribution
of photoelectrons detached from unexcited molecules is negligible at energies
higher than 97 eV. Thus, photoelectron spectra at energies higher than 97 eV
are beneficial to study the dynamics of photoexcited pentacene molecules,
since they would not contain a background signal from the unexcited molecules.

We find a remarkable result that the angle-averaged photoelectron spectra from
photo-excited pentacene do not vary with the probe-pulse arrival time although
the photo-induced charge distribution experiences considerable oscillations in
time (see Fig. 2). This is because the time-dependent part of the electronic
density change is highly symmetric in space. As a consequence, the
photoelectron momentum distributions are also highly symmetric and the
temporal dependence of the signal averages out after the angle integration.
The angle-integrated spectra do not reveal information about the time-
dependent photo-induced charge oscillations.

We now demonstrate that photoelectron momentum microscopy is indeed quite
sensitive to photo-induced electron dynamics. We consider photoelectron
momentum maps (PMMs) of a photoelectron with a fixed kinetic energy of
$\epsilon_{e}=99\ \text{eV}$. PMMs are constructed by a two-dimensional
projection of the three-dimensional photoelectron momentum distribution for a
constant energy $\epsilon_{e}$ (hemispherical cuts). Based on Eq. (2), the
photoelectron probability resulting from a particular final state is only
time-dependent, if the final state has a nonzero overlap with both eigenstates
in the initial state,
$\left<{\Phi_{F}}\right|\hat{\psi}(\mathbf{r})\left|{\Phi_{1}}\right>\neq 0$
and $\left<{\Phi_{F}}\right|\hat{\psi}(\mathbf{r})\left|{\Phi_{2}}\right>\neq
0$. At the chosen kinetic energy, three final states with the lowest energies,
with the indices 1, 2 and 3 in Table 1 contribute to the PMM. Thereby, among
these three final states, only the final state 1 contributes to a time-
dependent signal and the other two final states 2 and 3 contribute to a time-
independent background. The first final state results in a broad spectral line
centered approximately at the energy $\epsilon_{e}=99\ \text{eV}$.

We derive the explicit expressions for the Dyson orbital that determine the
angle-resolved photoelectron spectra at $\epsilon_{e}=99\ \text{eV}$. For the
first three final states, we obtain

$\displaystyle\begin{split}\phi_{F_{1}}^{D}(\mathbf{r})&=\left<{\Phi_{F_{1}}}\right|\hat{\psi}(\mathbf{r})\left|{\Psi(t)}\right>\\\
&=-C_{1}e^{-iE_{1}t_{p}}\frac{0.95}{\sqrt{2}}\phi_{L}(\mathbf{r})\chi_{\downarrow}\\\
&\hphantom{=}-C_{2}e^{-iE_{2}t_{p}}\frac{0.95}{2}\phi_{L+2}(\mathbf{r})\chi_{\downarrow},\end{split}$
(11)
$\displaystyle\begin{split}\phi_{F_{2}}^{D}(\mathbf{r})&=\left<{\Phi_{F_{2}}}\right|\hat{\psi}(\mathbf{r})\left|{\Psi(t)}\right>\\\
&=C_{2}e^{-iE_{2}t_{p}}\frac{0.94}{2}\phi_{L}(\mathbf{r})\chi_{\downarrow},\end{split}$
(12)
$\displaystyle\begin{split}\phi_{F_{3}}^{D}(\mathbf{r})&=\left<{\Phi_{F_{3}}}\right|\hat{\psi}(\mathbf{r})\left|{\Psi(t)}\right>\\\
&=-C_{1}e^{-iE_{1}t_{p}}\frac{0.83}{\sqrt{2}}\phi_{H}(\mathbf{r})\chi_{\downarrow}.\end{split}$
(13)

Figure 4: Time-dependent photoelectron momentum maps for photoelectrons
emitted from the excited pentacene at different probe-pulse arrival times
$t_{p}$. The probe-pulse duration is $0.5\ \text{fs}$ and the central photon
energy is $100\ \text{eV}$. The momentum maps are shown at the photoelectron
energy of $\epsilon_{e}=99\ \text{eV}$.

These three Dyson orbitals in Eqs. (11) - (13) enter the photoelectron
probability in Eq. (2) at the photoelectron energy $\epsilon=99\ \text{eV}$

$\displaystyle\begin{split}P(\mathbf{q},t_{p})\underset{\epsilon_{e}=99\
\text{eV}}{\propto}&\left|\bm{\epsilon}_{\text{in}}\cdot\mathbf{q}\right|^{2}\left[e^{-(\Omega_{F_{1}}-\epsilon_{e})^{2}\frac{\tau_{p}^{2}}{4\ln
2}}\left|C_{1}e^{-iE_{1}t_{p}}\frac{0.95}{\sqrt{2}}\mathcal{F}(\phi_{L}(\mathbf{r}))(\mathbf{q})\right.\right.\left.\left.+C_{2}e^{-iE_{2}t_{p}}\frac{0.95}{2}\mathcal{F}(\phi_{L+2}(\mathbf{r}))(\mathbf{q})\right|^{2}\right]\\\
+&\left|\bm{\epsilon}_{\text{in}}\cdot\mathbf{q}\right|^{2}\left[e^{-(\Omega_{F_{2}}-\epsilon_{e})^{2}\frac{\tau_{p}^{2}}{4\ln
2}}\left|C_{2}\frac{0.94}{2}\mathcal{F}(\phi_{L}(\mathbf{r}))(\mathbf{q})\right|^{2}\right.+\left.e^{-(\Omega_{F_{3}}-\epsilon_{e})^{2}\frac{\tau_{p}^{2}}{4\ln
2}}\left|C_{1}\frac{0.83}{\sqrt{2}}\mathcal{F}(\phi_{H}(\mathbf{r}))(\mathbf{q})\right|^{2}\right],\end{split}$
(14)

where
$\mathcal{F}(\phi(\mathbf{r}))(\mathbf{q})=(2\pi)^{-3/2}\int\text{d}r^{3}e^{i\mathbf{q}\cdot\mathbf{r}}\phi(\mathbf{r})$
denotes the Fourier transform of an orbital $\phi(\mathbf{r})$. The resulting
time-resolved PMMs shown in Fig. 4 vary strongly with time. The PMMs at
$t_{p}=0$ and $t_{p}=\frac{T}{2}$ are the point reflection image of each other
and have the same symmetry as the changes of the electron density at these
times (see Fig. 2). The PMMs at $t_{p}=\frac{T}{4}$ and $t_{p}=\frac{3T}{4}$,
when the electron density coincides, are equal and have reflection symmetry.
The most pronounced oscillating peaks are located in the PMMs at the momenta
$q_{x}=\pm 1.26\ \mbox{\AA}^{-1}$ and $q_{y}=\pm 1.97\ \mbox{\AA}^{-1}$.
According to Eq. (14) the time-dependent part of the PMMs arises from the
Fourier transform of LUMO and LUMO$+2$ orbitals. It is directly related to the
negatively-charged part of the induced electron-density change (yellow in Fig.
2), which is also specified by the LUMO and LUMO$+2$ orbitals (see Eq. (10)).
Thus, the time-dependent features in the PMMs encode the information about the
symmetry of the photo-induced electron density. Due to a high contrast of a
signal and a low background, these features can be detected in an experiment
even if the signal is weak. Due to the high spatial resolution, PMMs resolve
oscillations of the electron density within the molecule.

Figure 5: Time-dependent photoelectron momentum maps for photoelectrons
emitted from the excited pentacene for different photoelectron energies
$\epsilon_{e}$. Each set of momentum maps has a different probe-pulse arrival
time $t_{p}$. The probe-pulse duration is $0.5\ \text{fs}$ and the central
photon energy is $100\ \text{eV}$.

In Ref. [7, 88], we demonstrated that that a general time- and momentum-
resolved signal from a coherently evolving electronic wave packet can be
sensitive to electron currents within a sample. This means that a momentum-
resolved signal can be even different, when the electron density coincides,
but the charge flow direction differs. This applies to the electronic state in
Fig. 2 at times $t_{p}=\frac{T}{4}$ and $t_{p}=\frac{3T}{4}$. However, we do
not observe this effect in the considered case, since it also averages out due
to the high symmetry of the photo-induced charge distribution.

Finally, we investigate PMMs and their temporal dependence at different
photoelectron energies in Fig. 5. As follows from the angle-integrated spectra
in Fig. 3, the overall intensity of the PMMs increases at lower photoelectron
energies. The PMMs at energy $\epsilon_{e}=90\ \text{eV}$ and at energy
$\epsilon_{e}=96\ \text{eV}$ remain almost constant in time, whereas the PMMs
at $\epsilon_{e}=93\ \text{eV}$ and at $\epsilon_{e}=99\ \text{eV}$ show
pronounced temporal dependence. Although the absolute change of the signal in
time is approximately the same in the two former cases, the overall contrast
is much better for the PMMs at $\epsilon_{e}=99\ \text{eV}$ due to a lower
time-constant background. This temporal behavior of the signal can be
explained by the character of photo-induced changes in the electron density.
These changes are due to the electrons photo-excited into orbitals that were
unoccupied in the ground state and holes in the orbitals that were doubly
occupied in the ground state. If an electron, which evolves in the photo-
excited orbitals, is emitted by a probe pulse, then it directly carries the
information about the negatively-charged oscillations of the electron density.
If an electron is indeed emitted from lower lying orbitals, where an electron
hole evolves, then it is an ”indirect probe” of the evolving hole. Thus, the
signal at the photoelectron energy of 99 eV results from the emission of
electrons evolving in the photo-excited orbitals and is the most advantageous
to obtain a time-resolved signal with a high contrast.

Figure 6: Time-dependent photoelectron momentum maps (PMMs) for photoelectrons
emitted from the excited pentacene. In each column the PMMs are shown for
different probe-pulse arrival times (a),(e),(i) $t_{p}=0T$, (b),(f),(j)
$t_{p}=T/4$, (c),(g),(k) $t_{p}=T/2$, (d),(h),(l) $t_{p}=3T/4$. In the
different rows the PMMs are shown for different probe-pulse durations (a)-(d)
$\tau_{p}=0.5\ \text{fs}$, (e)-(h) $\tau_{p}=T/4$, (i)-(l) $\tau_{p}=T/2$. The
probe pulse has a central photon energy of $100\ \text{eV}$. The momentum maps
are obtained by averaging the signal over the range of $\epsilon_{e}=98.5\
\text{eV}$ to $\epsilon_{e}=99.5\ \text{eV}$ to simulate the energy resolution
of 1 eV.

In a real experiment, PMMs would be obtained with limited energy and time
resolution. We simulate the PMMs in the case of limited energy resolution by
averaging the photoelectron probability over the corresponding energy
interval. We show the PMMs at the photoelectron energy of 99 eV assuming the
energy resolution of 1 eV in Figs. 6(a)-(d), which are obtained by averaging
the signal over the range of 98.5 eV to 99.5 eV. We obtain that the momentum
distributions are almost unaffected by the limited energy resolution.

The temporal resolution of the experiment is determined by the duration of the
probe pulse and the timing jitter between the probe and the pump pulses. If
the probe-pulse duration cannot be assumed to be much shorter than the
oscillation period, Eq. (3) must be used for the calculation instead of Eq.
(2). We calculate the signal with Eq. (3) at different probe-pulse durations
$\tau_{p}$. We also assume the limited energy resolution of 1 eV. As shown in
Figs. 6(e)-(h) and Figs. 6(i)-(l), the temporal contrast of the resulting
signal is quite high even, if the probe-pulse duration is as large as a
quarter or half of the oscillation period.

In our calculation, we neglected electron-nuclear coupling in the description
of charge migration since the primary goal of our study is to investigate how
pure electron dynamics on their natural time scale can be measured. Let us
discuss, which consequences the coupling of electron dynamics to nuclear
dynamics would have for the signal. The coupling to nuclear motions would
indeed lead to decoherence of electron dynamics and should cause the decrease
of the oscillation amplitude [89, 65, 41, 90]. Former studies of coupled
electron-nuclear dynamics of similar molecules such as norbornadiene cations
[91] or benzene [92] have demonstrated that coherent electron motion can last
up to 20 fs in these molecules. It is expected that coherent electronic motion
in polycyclic aromatic hydrocarbons, which also include pentacene, would
experience similar time scales [92]. Since 20 fs corresponds to about five
cycles of oscillation in our case, our assumption that charge migration is
coherent during at least one full oscillation period is robust. At longer time
scales, the decay of the amplitude of the periodic charge oscillations due to
decoherence would result in the decrease of the time-dependent changes in the
PMMs. This would lead to a loss of temporal contrast of the signal, which
means that the signal at different times would become less distinguishable
from the time-independent part of the signal (see Fig. 5). This could serve
even as an advantage, since such an experiment would reveal the transition of
coherent dynamics to decoherent dynamics.

##  IV Conclusions

We showed theoretically how time-resolved photoelectron momentum microscopy
can be employed to image photo-induced coherently evolving electron dynamics
in a pentacene molecule. The assumed XUV probe pulse of 500 as duration
provides the temporal resolution, which is sensitive to pure intramolecular
electronic dynamics. The momentum-resolved signal revealed details of electron
dynamics with Ångstrom spatial resolution. Although the angle-averaged
photoelectron spectra were not sensitive to electron dynamics, momentum-
resolved spectra provided a highly contrast signal evolving in time. At high
photoelectron energies, the signal is provided by the electrons emitted from
photo-excited orbitals and is not affected from a possible contribution of
photoelectrons emitted by molecules that were not photo-excited. Such
photoelectron momentum maps are directly related to the negatively-charged
part of the photo-induced changes in the electron density and is sensitive to
its symmetry in real space. These findings demonstrate that attosecond
photoelectron momentum microscopy has a big potential to become a novel
technique to measure charge migration and electron coherence processes in
exciton transport in organic semiconductors with unprecedented time and
spatial details.

##  V Acknowledgements

We gratefully acknowledge the funding of the Volkswagen Foundation.

## References

  * [1] Christer Z. Bisgaard, Owen J. Clarkin, Guorong Wu, Anthony M. D. Lee, Oliver Geßner, Carl C. Hayden, and Albert Stolow.  Time-resolved molecular frame dynamics of fixed-in-space CS2 molecules.  Science, 323(5920):1464–1468, 2009.
  * [2] Peter Puschnig, Stephen Berkebile, Alexander J. Fleming, Georg Koller, Konstantin Emtsev, Thomas Seyller, John D. Riley, Claudia Ambrosch-Draxl, Falko P. Netzer, and Michael G. Ramsey.  Reconstruction of molecular orbital densities from photoemission data.  Science, 326(5953):702–706, 2009.
  * [3] Daniel Lüftner, Thomas Ules, Eva Maria Reinisch, Georg Koller, Serguei Soubatch, F. Stefan Tautz, Michael G. Ramsey, and Peter Puschnig.  Imaging the wave functions of adsorbed molecules.  Proceedings of the National Academy of Sciences, 111(2):605–610, 2014.
  * [4] B. Mignolet, R. D. Levine, and F. Remacle.  Control of electronic dynamics visualized by angularly resolved photoelectron spectra: A dynamical simulation with an ir pump and xuv attosecond-pulse-train probe.  Phys. Rev. A, 89:021403, Feb 2014.
  * [5] Peter Puschnig and Daniel Lüftner.  Simulation of angle-resolved photoemission spectra by approximating the final state by a plane wave: From graphene to polycyclic aromatic hydrocarbon molecules.  Journal of Electron Spectroscopy and Related Phenomena, 200(xx):193–208, 2015.
  * [6] S. Weiß, D. Lüftner, T. Ules, E. M. Reinisch, H. Kaser, A. Gottwald, M. Richter, S. Soubatch, G. Koller, M. G. Ramsey, F. S. Tautz, and P. Puschnig.  Exploring three-dimensional orbital imaging with energy-dependent photoemission tomography.  Nature Communications, 6:8287, October 2015.
  * [7] Daria Popova-Gorelova, Jochen Küpper, and Robin Santra.  Imaging electron dynamics with time- and angle-resolved photoelectron spectroscopy.  Phys. Rev. A, 94:013412, Jul 2016.
  * [8] P Kliuiev, T Latychevskaia, J Osterwalder, M Hengsberger, and L Castiglioni.  Application of iterative phase-retrieval algorithms to ARPES orbital tomography.  New Journal of Physics, 18(9):093041, sep 2016.
  * [9] C. W. Nicholson, A. Lücke, W. G. Schmidt, M. Puppin, L. Rettig, R. Ernstorfer, and M. Wolf.  Beyond the molecular movie: Dynamics of bands and bonds during a photoinduced phase transition.  Science, 362(6416):821–825, 2018.
  * [10] Pavel Kliuiev, Giovanni Zamborlini, Matteo Jugovac, Yeliz Gurdal, Karin von Arx, Kay Waltar, Stephan Schnidrig, Roger Alberto, Marcella Iannuzzi, Vitaliy Feyer, Matthias Hengsberger, Jürg Osterwalder, and Luca Castiglioni.  Combined orbital tomography study of multi-configurational molecular adsorbate systems.  Nature Communications, 10(1):5255, Nov 2019.
  * [11] C. Metzger, M. Graus, M. Grimm, G. Zamborlini, V. Feyer, M. Schwendt, D. Lüftner, P. Puschnig, A. Schöll, and F. Reinert.  Plane-wave final state for photoemission from nonplanar molecules at a metal-organic interface.  Physical Review B, 101(16):165421, April 2020.  Publisher: American Physical Society.
  * [12] G S M Jansen, M Keunecke, M Düvel, C Möller, D Schmitt, W Bennecke, F J S Kappert, D Steil, D R Luke, S Steil, and S Mathias.  Efficient orbital imaging based on ultrafast momentum microscopy and sparsity-driven phase retrieval.  New Journal of Physics, 22(6):063012, jun 2020.
  * [13] R. Wallauer, M. Raths, K. Stallberg, L. Münster, D. Brandstetter, X. Yang, J. Güdde, P. Puschnig, S. Soubatch, C. Kumpf, F. C. Bocquet, F. S. Tautz, and U. Höfer.  Tracing orbital images on ultrafast time scales.  Science, 371(6533):1056–1059, 2021.
  * [14] Alexander Neef, Samuel Beaulieu, Sebastian Hammer, Shuo Dong, Julian Maklar, Tommaso Pincelli, R. Patrick Xian, Martin Wolf, Laurenz Rettig, Jens Pflaum, and Ralph Ernstorfer.  Orbital-resolved observation of singlet fission, 2022.
  * [15] Kiana Baumgärtner, Marvin Reuner, Christian Metzger, Dmytro Kutnyakhov, Michael Heber, Federico Pressacco, Chul-Hee Min, Thiago RF Peixoto, Mario Reiser, Chan Kim, et al.  Ultrafast orbital tomography of a pentacene film using time-resolved momentum microscopy at a fel.  Nature Communications, 13(1):1–7, 2022.
  * [16] N. Hartmann, G. Hartmann, R. Heider, M. S. Wagner, M. Ilchen, J. Buck, A. O. Lindahl, C. Benko, J. Grünert, J. Krzywinski, J. Liu, A. A. Lutman, A. Marinelli, T. Maxwell, A. A. Miahnahri, S. P. Moeller, M. Planas, J. Robinson, A. K. Kazansky, N. M. Kabachnik, J. Viefhaus, T. Feurer, R. Kienberger, R. N. Coffee, and W. Helml.  Attosecond time–energy structure of x-ray free-electron laser pulses.  Nature Photonics, 12(4):215–220, Apr 2018.
  * [17] Praveen Kumar Maroju, Cesare Grazioli, Michele Di Fraia, Matteo Moioli, Dominik Ertel, Hamed Ahmadi, Oksana Plekan, Paola Finetti, Enrico Allaria, Luca Giannessi, Giovanni De Ninno, Carlo Spezzani, Giuseppe Penco, Simone Spampinati, Alexander Demidovich, Miltcho B. Danailov, Roberto Borghes, George Kourousias, Carlos Eduardo Sanches Dos Reis, Fulvio Billé, Alberto A. Lutman, Richard J. Squibb, Raimund Feifel, Paolo Carpeggiani, Maurizio Reduzzi, Tommaso Mazza, Michael Meyer, Samuel Bengtsson, Neven Ibrakovic, Emma Rose Simpson, Johan Mauritsson, Tamás Csizmadia, Mathieu Dumergue, Sergei Kühn, Harshitha Nandiga Gopalakrishna, Daehyun You, Kiyoshi Ueda, Marie Labeye, Jens Egebjerg Bækhøj, Kenneth J. Schafer, Elena V. Gryzlova, Alexei N. Grum-Grzhimailo, Kevin C. Prince, Carlo Callegari, and Giuseppe Sansone.  Attosecond pulse shaping using a seeded free-electron laser.  Nature, 578(7795):386–391, Feb 2020.
  * [18] Joseph Duris, Siqi Li, Taran Driver, Elio G Champenois, James P MacArthur, Alberto A Lutman, Zhen Zhang, Philipp Rosenberger, Jeff W Aldrich, Ryan Coffee, et al.  Tunable isolated attosecond x-ray pulses with gigawatt peak power from a free-electron laser.  Nature Photonics, 14(1):30–36, 2020.
  * [19] S. M. Teichmann, F. Silva, S. L. Cousin, M. Hemmer, and J. Biegert.  0.5-kev soft x-ray attosecond continua.  Nature Communications, 7(1):11493, May 2016.
  * [20] Jie Li, Xiaoming Ren, Yanchun Yin, Kun Zhao, Andrew Chew, Yan Cheng, Eric Cunningham, Yang Wang, Shuyuan Hu, Yi Wu, Michael Chini, and Zenghu Chang.  53-attosecond x-ray pulses reach the carbon k-edge.  Nature Communications, 8(1):186, Aug 2017.
  * [21] Xiaoming Ren, Jie Li, Yanchun Yin, Kun Zhao, Andrew Chew, Yang Wang, Shuyuan Hu, Yan Cheng, Eric Cunningham, Yi Wu, Michael Chini, and Zenghu Chang.  Attosecond light sources in the water window.  Journal of Optics, 20(2):023001, jan 2018.
  * [22] Jie Li, Jian Lu, Andrew Chew, Seunghwoi Han, Jialin Li, Yi Wu, He Wang, Shambhu Ghimire, and Zenghu Chang.  Attosecond science based on high harmonic generation from gases and solids.  Nature Communications, 11(1):2748, Jun 2020.
  * [23] Giulio Maria Rossi, Roland E. Mainz, Yudong Yang, Fabian Scheiba, Miguel A. Silva-Toledo, Shih-Hsuan Chia, Phillip D. Keathley, Shaobo Fang, Oliver D. Mücke, Cristian Manzoni, Giulio Cerullo, Giovanni Cirmi, and Franz X. Kärtner.  Sub-cycle millijoule-level parametric waveform synthesizer for attosecond science.  Nature Photonics, 14(10):629–635, Oct 2020.
  * [24] Bárbara Buades, Antonio Picón, Emma Berger, Iker León, Nicola Di Palo, Seth L. Cousin, Caterina Cocchi, Eric Pellegrin, Javier Herrero Martin, Samuel Mañas-Valero, Eugenio Coronado, Thomas Danz, Claudia Draxl, Mitsuharu Uemoto, Kazuhiro Yabana, Martin Schultze, Simon Wall, Michael Zürch, and Jens Biegert.  Attosecond state-resolved carrier motion in quantum materials probed by soft x-ray xanes.  Applied Physics Reviews, 8(1):011408, 2021.
  * [25] Katsumi Midorikawa.  Progress on table-top isolated attosecond light sources.  Nature Photonics, 16(4):267–278, Apr 2022.
  * [26] F. Calegari, D. Ayuso, A. Trabattoni, L. Belshaw, S. De Camillis, S. Anumula, F. Frassetto, L. Poletto, A. Palacios, P. Decleva, J. B. Greenwood, F. Martín, and M. Nisoli.  Ultrafast electron dynamics in phenylalanine initiated by attosecond pulses.  Science, 346(6207):336–339, 2014.
  * [27] P. M. Kraus, B. Mignolet, D. Baykusheva, A. Rupenyan, L. Horný, E. F. Penka, G. Grassi, O. I. Tolstikhin, J. Schneider, F. Jensen, L. B. Madsen, A. D. Bandrauk, F. Remacle, and H. J. Wörner.  Measurement and laser control of attosecond charge migration in ionized iodoacetylene.  Science, 350(6262):790–795, 2015.
  * [28] Aaron von Conta, Andres Tehlar, Albert Schletter, Yasuki Arasaki, Kazuo Takatsuka, and Hans Jakob Wörner.  Conical-intersection dynamics and ground-state chemistry probed by extreme-ultraviolet time-resolved photoelectron spectroscopy.  Nature communications, 9(1):1–10, 2018.
  * [29] F. Langer, C. P. Schmid, S. Schlauderer, M. Gmitra, J. Fabian, P. Nagler, C. Schüller, T. Korn, P. G. Hawkins, J. T. Steiner, U. Huttner, S. W. Koch, M. Kira, and R. Huber.  Lightwave valleytronics in a monolayer of tungsten diselenide.  Nature, 557(7703):76–80, May 2018.
  * [30] Florian Siegrist, Julia A. Gessner, Marcus Ossiander, Christian Denker, Yi-Ping Chang, Malte C. Schröder, Alexander Guggenmos, Yang Cui, Jakob Walowski, Ulrike Martens, J. K. Dewhurst, Ulf Kleineberg, Markus Münzenberg, Sangeeta Sharma, and Martin Schultze.  Light-wave dynamic control of magnetism.  Nature, 571(7764):240–244, Jul 2019.
  * [31] Peng Peng, Claude Marceau, and David M. Villeneuve.  Attosecond imaging of molecules using high harmonic spectroscopy.  Nature Reviews Physics, 1(2):144–155, Feb 2019.
  * [32] Jonathan P. Marangos.  Accessing the quantum spatial and temporal scales with xfels.  Nature Reviews Physics, 2(7):332–334, Jul 2020.
  * [33] Kristina S. Zinchenko, Fernando Ardana-Lamas, Issaka Seidu, Simon P. Neville, Joscelyn van der Veen, Valentina Utrio Lanfaloni, Michael S. Schuurman, and Hans Jakob Wörner.  Sub-7-femtosecond conical-intersection dynamics probed at the carbon k-edge.  Science, 371(6528):489–494, 2021.
  * [34] Marc Rebholz, Thomas Ding, Victor Despré, Lennart Aufleger, Maximilian Hartmann, Kristina Meyer, Veit Stooß, Alexander Magunia, David Wachs, Paul Birk, Yonghao Mi, Gergana Dimitrova Borisova, Carina da Costa Castanheira, Patrick Rupprecht, Georg Schmid, Kirsten Schnorr, Claus Dieter Schröter, Robert Moshammer, Zhi-Heng Loh, Andrew R. Attar, Stephen R. Leone, Thomas Gaumnitz, Hans Jakob Wörner, Sebastian Roling, Marco Butz, Helmut Zacharias, Stefan Düsterer, Rolf Treusch, Günter Brenner, Jonas Vester, Alexander I. Kuleff, Christian Ott, and Thomas Pfeifer.  All-xuv pump-probe transient absorption spectroscopy of the structural molecular dynamics of di-iodomethane.  Phys. Rev. X, 11:031001, Jul 2021.
  * [35] D. Garratt, L. Misiekis, D. Wood, E. W. Larsen, M. Matthews, O. Alexander, P. Ye, S. Jarosch, C. Ferchaud, C. Strüber, A. S. Johnson, A. A. Bakulin, T. J. Penfold, and J. P. Marangos.  Direct observation of ultrafast exciton localization in an organic semiconductor with soft x-ray transient absorption spectroscopy.  Nature Communications, 13(1):3414, Jun 2022.
  * [36] Erik P Månsson, Simone Latini, Fabio Covito, Vincent Wanie, Mara Galli, Enrico Perfetto, Gianluca Stefanucci, Umberto De Giovannini, Mattea C Castrovilli, Andrea Trabattoni, Fabio Frassetto, Luca Poletto, Jason B Greenwood, François Légaré, Mauro Nisoli, Angel Rubio, and Francesca Calegari.  Ultrafast dynamics of adenine following XUV ionization.  Journal of Physics: Photonics, 4(3):034003, may 2022.
  * [37] Gopal Dixit, Oriol Vendrell, and Robin Santra.  Imaging electronic quantum motion with light.  Proceedings of the National Academy of Sciences, 109(29):11636–11640, 2012.
  * [38] Daria Popova-Gorelova and Robin Santra.  Imaging instantaneous electron flow with ultrafast resonant x-ray scattering.  Phys. Rev. B, 91:184303, May 2015.
  * [39] Daria Popova-Gorelova and Robin Santra.  Imaging interatomic electron current in crystals with ultrafast resonant x-ray scattering.  Phys. Rev. B, 92:184304, Nov 2015.
  * [40] Maximilian Hollstein, Robin Santra, and Daniela Pfannkuche.  Correlation-driven charge migration following double ionization and attosecond transient absorption spectroscopy.  Phys. Rev. A, 95:053411, May 2017.
  * [41] Victor Despré, Nikolay V. Golubev, and Alexander I. Kuleff.  Charge migration in propiolic acid: A full quantum dynamical study.  Phys. Rev. Lett., 121:203002, Nov 2018.
  * [42] Mingrui He, Yang Li, Yueming Zhou, Min Li, Wei Cao, and Peixiang Lu.  Direct visualization of valence electron motion using strong-field photoelectron holography.  Phys. Rev. Lett., 120:133204, Mar 2018.
  * [43] Kai-Jun Yuan and André D. Bandrauk.  Ultrafast x-ray photoelectron imaging of attosecond electron dynamics in molecular coherent excitation.  The Journal of Physical Chemistry A, 123(7):1328–1336, 2019.  PMID: 30669842.
  * [44] Ludger Inhester, Loren Greenman, Artem Rudenko, Daniel Rolles, and Robin Santra.  Detecting coherent core-hole wave-packet dynamics in n2 by time- and angle-resolved inner-shell photoelectron spectroscopy.  The Journal of Chemical Physics, 151(5):054107, 2019.
  * [45] Khadijeh Khalili, Ludger Inhester, Caroline Arnold, Anders S. Gertsen, Jens Wenzel Andreasen, and Robin Santra.  Simulation of time-resolved x-ray absorption spectroscopy of ultrafast dynamics in particle-hole-excited 4‐(2-thienyl)-2,1,3-benzothiadiazole.  Structural Dynamics, 7(4):044101, 2020.
  * [46] Stefano M. Cavaletto, Daniel Keefer, Jérémy R. Rouxel, Flavia Aleotti, Francesco Segatta, Marco Garavelli, and Shaul Mukamel.  Unveiling the spatial distribution of molecular coherences at conical intersections by covariance x-ray diffraction signals.  Proceedings of the National Academy of Sciences, 118(22):e2105046118, 2021.
  * [47] François Mauger, Aderonke S. Folorunso, Kyle A. Hamer, Cristel Chandre, Mette B. Gaarde, Kenneth Lopata, and Kenneth J. Schafer.  Charge migration and attosecond solitons in conjugated organic molecules.  Phys. Rev. Research, 4:013073, Jan 2022.
  * [48] Yeonsig Nam, Francesco Montorsi, Daniel Keefer, Stefano M. Cavaletto, Jin Yong Lee, Artur Nenov, Marco Garavelli, and Shaul Mukamel.  Time-resolved optical pump-resonant x-ray probe spectroscopy of 4-thiouracil: A simulation study.  Journal of Chemical Theory and Computation, 18(5):3075–3088, 2022\.  PMID: 35476905.
  * [49] K. Chordiya, V. Despré, B. Nagyillés, F. Zeller, Z. Diveki, A. I. Kuleff, and M. U. Kahaly.  Photo-ionization initiated differential ultrafast charge migration: Impact of molecular symmetries and tautomeric forms, 2022.
  * [50] S. Yoo, B. Domercq, and B. Kippelen.  Efficient thin-film organic solar cells based on pentacene/c60 heterojunctions.  Applied Physics Letters, 85(22):5427–5429, 2004.
  * [51] Karl Leo.  Organic photovoltaics.  Nature Reviews Materials, 1(8):1–2, 2016.
  * [52] Anna Köhler and Heinz Bässler.  Electronic processes in organic semiconductors: An introduction.  John Wiley & Sons, 2015.
  * [53] Francesca Fassioli, Rayomond Dinshaw, Paul C. Arpin, and Gregory D. Scholes.  Photosynthetic light harvesting: excitons and coherence.  Journal of The Royal Society Interface, 11(92):20130901, 2014.
  * [54] Tammie R. Nelson, Dianelys Ondarse-Alvarez, Nicolas Oldani, Beatriz Rodriguez-Hernandez, Laura Alfonso-Hernandez, Johan F. Galindo, Valeria D. Kleiman, Sebastian Fernandez-Alberti, Adrian E. Roitberg, and Sergei Tretiak.  Coherent exciton-vibrational dynamics and energy transfer in conjugated organics.  Nature Communications, 9(1):2316, Jun 2018.
  * [55] Shahnawaz Rafiq and Gregory D. Scholes.  From fundamental theories to quantum coherences in electron transfer.  Journal of the American Chemical Society, 141(2):708–722, Jan 2019\.
  * [56] L.S. Cederbaum and J. Zobeley.  Ultrafast charge migration by electron correlation.  Chemical Physics Letters, 307(3):205–210, 1999.
  * [57] J. Breidbach and L. S. Cederbaum.  Migration of holes: Formalism, mechanisms, and illustrative applications.  The Journal of Chemical Physics, 118(9):3983–3996, 2003.
  * [58] Alexander I. Kuleff, Siegfried Lünnemann, and Lorenz S. Cederbaum.  Ultrafast charge migration following ionization in oligopeptides.  In Paul Corkum, Sandro Silvestri, Keith A. Nelson, Eberhard Riedle, and Robert W. Schoenlein, editors, Ultrafast Phenomena XVI, pages 586–588, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.
  * [59] Siegfried Lünnemann, Alexander I. Kuleff, and Lorenz S. Cederbaum.  Charge migration following ionization in systems with chromophore-donor and amine-acceptor sites.  The Journal of Chemical Physics, 129(10):104305, 2008.
  * [60] Anthony D. Dutoi, Lorenz S. Cederbaum, Michael Wormit, Jan Hendrik Starcke, and Andreas Dreuw.  Tracing molecular electronic excitation dynamics in real time and space.  The Journal of Chemical Physics, 132(14):144302, 2010.
  * [61] Anthony D. Dutoi, Michael Wormit, and Lorenz S. Cederbaum.  Ultrafast charge separation driven by differential particle and hole mobilities.  The Journal of Chemical Physics, 134(2):024303, 2011.
  * [62] Anthony D. Dutoi and Lorenz S. Cederbaum.  An excited electron avoiding a positive charge.  The Journal of Physical Chemistry Letters, 2(18):2300–2303, 2011\.
  * [63] F. Remacle and R. D. Levine.  Attosecond pumping of nonstationary electronic states of lih: Charge shake-up and electron density distortion.  Phys. Rev. A, 83:013411, Jan 2011.
  * [64] Alexander I Kuleff and Lorenz S Cederbaum.  Ultrafast correlation-driven electron dynamics.  Journal of Physics B: Atomic, Molecular and Optical Physics, 47(12):124002, jun 2014.
  * [65] Hans Jakob Wörner, Christopher A. Arrell, Natalie Banerji, Andrea Cannizzo, Majed Chergui, Akshaya K. Das, Peter Hamm, Ursula Keller, Peter M. Kraus, Elisa Liberatore, Pablo Lopez-Tarifa, Matteo Lucchini, Markus Meuwly, Chris Milne, Jacques-E. Moser, Ursula Rothlisberger, Grigory Smolentsev, Joël Teuscher, Jeroen A. van Bokhoven, and Oliver Wenger.  Charge migration and charge transfer in molecular systems.  Structural Dynamics, 4(6):061508, 2017.
  * [66] T. Kuś, B. Mignolet, R. D. Levine, and F. Remacle.  Pump and probe of ultrafast charge reorganization in small peptides: A computational study through sudden ionizations.  The Journal of Physical Chemistry A, 117(40):10513–10525, 2013\.  PMID: 24015765.
  * [67] Albert Stolow, Arthur E. Bragg, and Daniel M. Neumark.  Femtosecond time-resolved photoelectron spectroscopy.  Chemical Reviews, 104(4):1719–1758, 2004.  PMID: 15080710.
  * [68] Kai-Jun Yuan, Chuan-Cun Shu, Daoyi Dong, and André D. Bandrauk.  Attosecond dynamics of molecular electronic ring currents.  The Journal of Physical Chemistry Letters, 8(10):2229–2235, 2017\.  PMID: 28468499.
  * [69] A. Marciniak, V. Despré, V. Loriot, G. Karras, M. Hervé, L. Quintard, F. Catoire, C. Joblin, E. Constant, A. I. Kuleff, and F. Lépine.  Electron correlation driven non-adiabatic relaxation in molecules excited by an ultrashort extreme ultraviolet pulse.  Nature Communications, 10(1):337, Jan 2019.
  * [70] Deependra Jadoun and Markus Kowalewski.  Time-resolved photoelectron spectroscopy of conical intersections with attosecond pulse trains.  The Journal of Physical Chemistry Letters, 12(33):8103–8108, 2021\.  PMID: 34410134.
  * [71] Danielle Dowek and Piero Decleva.  Trends in angle-resolved molecular photoelectron spectroscopy.  Phys. Chem. Chem. Phys., 24:24614–24654, 2022.
  * [72] Stephan Hüfner.  Very high resolution photoelectron spectroscopy, volume 715.  Springer, 2007.
  * [73] B. Mignolet, R. D. Levine, and F. Remacle.  Localized electron dynamics in attosecond-pulse-excited molecular systems: Probing the time-dependent electron density by sudden photoionization.  Phys. Rev. A, 86:053429, Nov 2012.
  * [74] Aurelie Perveaux, David Lauvergnat, Fabien Gatti, Gábor J. Halász, Ágnes Vibók, and Benjamin Lasorne.  Monitoring the birth of an electronic wavepacket in a molecule with attosecond time-resolved photoelectron spectroscopy.  The Journal of Physical Chemistry A, 118(38):8773–8778, 2014.  PMID: 25167166.
  * [75] B Mignolet, R D Levine, and F Remacle.  Charge migration in the bifunctional penna cation induced and probed by ultrafast ionization: a dynamical study.  Journal of Physics B: Atomic, Molecular and Optical Physics, 47(12):124011, jun 2014.
  * [76] Peter Puschnig, Stephen Berkebile, Alexander J. Fleming, Georg Koller, Konstantin Emtsev, Thomas Seyller, John D. Riley, Claudia Ambrosch-Draxl, Falko P. Netzer, and Michael G. Ramsey.  Reconstruction of molecular orbital densities from photoemission data.  Science, 326(5953):702–706, 2009.
  * [77] P. Puschnig, E.-M. Reinisch, T. Ules, G. Koller, S. Soubatch, M. Ostler, L. Romaner, F. S. Tautz, C. Ambrosch-Draxl, and M. G. Ramsey.  Orbital tomography: Deconvoluting photoemission spectra of organic molecules.  Phys. Rev. B, 84:235427, Dec 2011.
  * [78] B. Stadtmüller, M. Willenbockel, E. M. Reinisch, T. Ules, F. C. Bocquet, S. Soubatch, P. Puschnig, G. Koller, M. G. Ramsey, F. S. Tautz, and C. Kumpf.  Orbital tomography for highly symmetric adsorbate systems.  EPL (Europhysics Letters), 100(2):26008, oct 2012.
  * [79] Per Aake. Malmqvist, Alistair. Rendell, and Bjoern O. Roos.  The restricted active space self-consistent-field method, implemented with a split graph unitary group approach.  The Journal of Physical Chemistry, 94(14):5477–5482, 1990.
  * [80] Francesco Aquilante, Jochen Autschbach, Rebecca K. Carlson, Liviu F. Chibotaru, Mickaël G. Delcey, Luca De Vico, Ignacio Fdez. Galván, Nicolas Ferré, Luis Manuel Frutos, Laura Gagliardi, Marco Garavelli, Angelo Giussani, Chad E. Hoyer, Giovanni Li Manni, Hans Lischka, Dongxia Ma, Per Åke Malmqvist, Thomas Müller, Artur Nenov, Massimo Olivucci, Thomas Bondo Pedersen, Daoling Peng, Felix Plasser, Ben Pritchard, Markus Reiher, Ivan Rivalta, Igor Schapiro, Javier Segarra-Martí, Michael Stenrup, Donald G. Truhlar, Liviu Ungur, Alessio Valentini, Steven Vancoillie, Valera Veryazov, Victor P. Vysotskiy, Oliver Weingart, Felipe Zapata, and Roland Lindh.  Molcas 8: New capabilities for multiconfigurational quantum chemical calculations across the periodic table.  Journal of Computational Chemistry, 37(5):506–541, 2016.
  * [81] Thom H. Dunning.  Gaussian basis sets for use in correlated molecular calculations. i. the atoms boron through neon and hydrogen.  The Journal of Chemical Physics, 90(2):1007–1023, 1989.
  * [82] Benjamin P. Pritchard, Doaa Altarawy, Brett Didier, Tara D. Gibson, and Theresa L. Windus.  New basis set exchange: An open, up-to-date resource for the molecular sciences community.  Journal of Chemical Information and Modeling, 59(11):4814–4820, 2019.  PMID: 31600445.
  * [83] E Heinecke, D Hartmann, R Müller, and A Hese.  Laser spectroscopy of free pentacene molecules (i): The rotational structure of the vibrationless s 1← s 0 transition.  The Journal of chemical physics, 109(3):906–911, 1998.
  * [84] Stefan Grimme and Maja Parac.  Substantial errors from time-dependent density functional theory for the calculation of excited states of large $\pi$ systems.  ChemPhysChem, 4(3):292–295, 2003.
  * [85] Pedro B. Coto, Sahar Sharifzadeh, Jeffrey B. Neaton, and Michael Thoss.  Low-lying electronic excited states of pentacene oligomers: A comparative electronic structure study in the context of singlet fission.  Journal of Chemical Theory and Computation, 11(1):147–156, Jan 2015\.
  * [86] Goran Kovačević and Valera Veryazov.  Luscus: molecular viewer and editor for molcas.  Journal of cheminformatics, 7(1):1–10, 2015.
  * [87] Koichi Momma and Fujio Izumi.  VESTA3 for three-dimensional visualization of crystal, volumetric and morphology data.  Journal of Applied Crystallography, 44(6):1272–1276, 2011.
  * [88] Daria Popova-Gorelova.  Imaging electron dynamics with ultrashort light pulses: A theory perspective.  Applied Sciences, 8(3), 2018.
  * [89] Caroline Arnold, Oriol Vendrell, and Robin Santra.  Electronic decoherence following photoionization: Full quantum-dynamical treatment of the influence of nuclear motion.  Phys. Rev. A, 95:033425, Mar 2017.
  * [90] Danylo T. Matselyukh, Victor Despré, Nikolay V. Golubev, Alexander I. Kuleff, and Hans Jakob Wörner.  Decoherence and revival in attosecond charge migration driven by non-adiabatic dynamics.  Nature Physics, 18(10):1206–1213, Oct 2022.
  * [91] Andrew J. Jenkins, Morgane Vacher, Rebecca M. Twidale, Michael J. Bearpark, and Michael A. Robb.  Charge migration in polycyclic norbornadiene cations: Winning the race against decoherence.  The Journal of Chemical Physics, 145(16):164103, 2016.
  * [92] V. Despré, A. Marciniak, V. Loriot, M. C. E. Galbraith, A. Rouzée, M. J. J. Vrakking, F. Lépine, and A. I. Kuleff.  Attosecond hole migration in benzene molecules surviving nuclear motion.  The Journal of Physical Chemistry Letters, 6(3):426–431, 2015.  PMID: 26261959.


# Near Optimal Per-Clip Lagrangian Multiplier Prediction in HEVC

Daniel J Ringis. François Pitié and Anil Kokaram  Sigmedia Group, Electronic
and Electrical Engineering Dept.
Trinity College Dublin
Dublin, Ireland
{ringisd, pitief, anil.kokaram} @tcd.ie

###### Abstract

The majority of internet traffic is video content. This drives the demand for
video compression to deliver high quality video at low target bitrates.
Optimising the parameters of a video codec for a specific video clip (per-clip
optimisation) has been shown to yield significant bitrate savings. In previous
work we have shown that per-clip optimisation of the Lagrangian multiplier
leads to up to 24% BD-Rate improvement. A key component of these algorithms is
modeling the R-D characteristic across the appropriate bitrate range. This is
computationally heavy as it usually involves repeated video encodes of the
high resolution material at different parameter settings. This work focuses on
reducing this computational load by deploying a NN operating on lower
bandwidth features. Our system achieves BD-Rate improvement in approximately
90% of a large corpus with comparable results to previous work in direct
optimisation.

###### Index Terms:

Video Compression, Video Codecs, Adaptive Encoding

##  I Introduction

With video content comprising over 80% of all internet traffic [1], there is
an ever growing need for video compression for user generated content (UGC).
The recent impact of the pandemic emphasises the importance of developing new
techniques. The modern compression schemes H.265/6 (HEVC) [2, 3, 4] and VP9
and AV1 [5, 6], have all been thrown into the spotlight.

Striking a balance between the rate and distortion of a video clip is a key
challenge in video coding. Since 1998 [7], it has been accepted practice to
represent this trade-off in terms of a cost $J$ as follows:

$J=D+\lambda R$ (1)

Here, a distortion, $D$ and a rate $R$ are combined through the action of a
Lagrangian multiplier, $\lambda$. The Lagrangian multiplier controls emphasis
on either rate minimisation or quality improvement. This has an impact
throughout the codec, as the idea is applied to many internal operations e.g.
motion vectors, block type (Skipped/Intra/Inter), and bit allocation at the
frame and clip levels [8].

The choice of $\lambda$ was determined experimentally in the early days of
codec research [9]. This parameter can be considered as a kind of
hyperparameter in a hybrid codec as it affects so many other decisions. This
was originally implemented for the reference implementation of the H.263
standard and then modified for the implementations of H.264 and H.265 (HEVC)
[10]. The approach was to select a value based on a cohort of examples such
that the performance was optimised on average across the set. That set was
quite small, less than 5 examples. While this small dataset may have been an
adequate representation of videos when developed, it may not be ideal with the
large range of user generated video content today.

It has been recognised that choosing coding parameters which are optimised for
a particular video lead to bitrate savings [11, 12]. This is because the
statistics of video clips varies greatly over any corpus. In our previous work
[13, 14], we have shown that direct optimisation methods can lead to bitrate
gains per clip of up to 20%. Unfortunately, this bitrate savings came at a
high computational cost, i.e. in excess of fifty video encodes per clip. At
scale, this cost is prohibitive.

This work therefore makes two principal contributions i) development of a more
computationally efficient optimiser for the Lagrangian based on Neural
Networks and ii) exploration of the gains achievable on a per clip basis.

##  II Previous Work on RD Optimisation

The rate distortion algorithm [15] establishes a balance between the quality
of the media and the transmission or storage capacities of the medium. As
mentioned above, seminal work of Sullivan and Wiegand [7] laid the foundation
for an experimental approach to choosing an appropriate $\lambda$. This work
established a relationship between the quantisation step size $Q$ and the
distortion $D$ in a frame. Through minimising $J$, this leads to a
relationship between $\lambda$ and $Q$ expressed as $\lambda=0.85\times
Q^{2}$. Updates to those experiments[10] then yielded similar relationships in
the most common implementations of H.264 and H.265 (HEVC). Introduction of bi-
directional (B) frames led to different constants and three different
relationships were established for each of the Intra (I),
$\lambda_{I}=(0.57)2^{(Q-12)/3}$, Predicted (P),
$\lambda_{P}=(0.85)2^{(Q-12)/3}$, and B frames,
$\lambda_{B}=(0.68)\max(2,\min(4,(Q-12)/6))2^{(Q-12)/3}$.

###  II-A Per Clip Optimisation

There exists a limited amount of work on adaptation of $\lambda$ in the rate
distortion equation. In all cases, results support the idea that adjusting
$\lambda$ leads to improvement in codec performance per clip. The idea is
typically to adjust $\lambda$ away from the codec default by using a constant
$k$ as follows.

$\lambda_{\textrm{new}}=k\times\lambda_{\textrm{orig}}$ (2)

where $\lambda_{\textrm{orig}}$ is the default Lagrangian multiplier estimated
in the video codec, and $\lambda_{\textrm{new}}$ is the updated Lagrangian.

Ma et al [16] used a Support Vector Machine to determine $k$. The perceptual
feature set included scalars representing Spatial Information and Temporal
Information, and a texture feature exploiting a Gray Level Concurrence Matrix.
Details can be found in [16]. Their focus was on Dynamic textures and they
used the DynTex dataset of 37 sequences. They reported up to 2dB improvement
in PSNR and 0.05 improvement in SSIM at equal bitrates. Hamza et al [17] also
take a classification approach but using gross scene classification into
indoor/outdoor/urban/non-urban classes. They then used the same $k$ for each
class. Their work used the Derfs dataset and reported up to 6% BD-Rate
improvement.

Zhang and Bull [18] used a single feature ${D}_{P}/{D}_{B}$, the ratio between
the MSE of P and B frames. This feature gives some idea of temporal
complexity. Experiments based on the DynTex database yielded up to 7%
improvement in BD-Rate. They modified $\lambda$ implicitly by adjusting the
quantiser parameter $Q$. Papadopoulos et al [19] exploited this and applied an
offset to Q, in HEVC, based on the ratio of the distortion in the P and B
frames. Each QP was updated from the previous Group of Pictures (GOP) using
$\textrm{QP}=a\times({D}_{P}/{D}_{B})-b$ where $a,b$ are constants determined
experimentally. This lead to an average BD-Rate improvement of 1.07% on the
DynTex dataset, with up to 3% BD-Rate improvement achieved for a single
sequence.

Yang et al [20] used a combination of features instead of just the MSE ratio
above. In their work, they used a perceptual content measurement $S$ to model
$k$ with a straight line fit $k=aS-b$. Here again $a,b$ were determined
experimentally using a corpus of the Derfs dataset. The loss in complexity of
the fit is compensated for by the increase in complexity of the feature. They
report a BD-Rate improvement of up to 6.2%. Recently, John et al[21] proposed
a machine learning method to classify Rate Distortion characteristics of a
clip in order to select the correct operating point for that clip. In this
work, they cluster videos based on their RD-Curve operating points and
determines the appropriate encoding parameters for each video based on the
model developed.

###  II-B Direct Optimisation of $k$

As shown in Equation 2, previous work attempts to adjust the rate control
parameter $\lambda$ adaptively in some way. In order to show that there is
something to be gained in varying $k$ per clip, Figure 1 shows BD-Rate vs $k$
for two different clips. There are two observations to be made. Firstly, the
relationship between $k$ and BD-Rate varies substantially between clips. This
reinforces the view that the traditional one size fits all approach is sub-
optimal. Secondly, the general shape of the curves shows a global minimum.
That minimum represents the maximum BD-Rate improvement available.

In recently published works [13, 14] we introduce the use of direct
optimisation to maximise BD-R and evaluated it on an expanded dataset,
primarily based on the YouTube-UGC dataset [22]. The dataset was complemented
with clips from other publicly available datasets, including the Netflix
dataset (Chimera and El Fuente)[23], DynTex dataset[24], MCL[25] and Derfs
dataset[26]. Multiple DASH segments (clips) of 5 seconds (150 frames) were
created from each sequence. The dataset comprises a total of 9,746 video clips
at varying resolutions with a wide range of video content, representative of
typical usage.

Figure 1: BD-Rate vs $k=[0.2:0.2:3]$ for two different clips
(LiveMusic_1080P-6d1a and NewsClip_720P-7e56). The best performing BD-Rate
gain is achieved at a different value for $k$ in each clip. Hence per-clip
optimisation is sensible. The curve shape shows a global minimum implying that
classic optimisation strategies would be successful.

In Figure 2, we report the BD-Rate improvement found in [13] to be achievable
by such a direct optimisation of the Lagrangian Multiplier. In this graph, we
see the BD-Rate improvement on the x-axis and the fraction of the dataset
which was able to achieve that BD-Rate improvement or better. Ideally, we
would want this curve to be as close to the top right as possible, as that
would indicate all clips achieved high BD-Rate improvements. We will be using
these results as our ground truth and upper bound for the deep learning
systems described later in this paper. The key result of this work is that 95%
of the clips had some BD-Rate improvement over the default Lagrangian
Multiplier, with 46% of them showing a BD-Rate improvement of 1% or better.

Figure 2: BD-Rate improvement vs fraction of dataset which achieves that
improvement or better. We can see from this curve that the direct optimisation
method was capable of achieving BD-Rate improvement in 95% of the clips in the
corpus. We can also read off of this graph what fraction of the clips had a 1%
or better BD-Rate improvement (47%) and what fraction of the clips had a 5% or
better improvement (3%).

###  II-C Machine-Learning Prediction of $k$

The direct optimisation method [13] outlined in the previous section, provides
positive results, however, it is computationally expensive. It takes on
average 11.7 iterations, and, as each BD-rate computation takes 5 video
encodes, finding the optimal $k$ takes $11.7\times 5\approx 60$ video encodes.
This is manageable at small resolutions, but is computationally intensive at
modern resolutions (720p and higher) and at scale.

Because of this, we have proposed in [14] to reduce the computational cost by
using proxies (low resolutions, older codecs) to calculate $k$ in practice. We
were able to get a reduction of $3.1\times$, while achieving 83% of the gain
found by direct optimisation. To attempt at further reducing the computational
cost, we also trained a machine learning system (Random Forest) based on
encoding information given by the x265 encoder at CRF 33. The results,
reported in the second row of Table II, show that the Random Forest model
leads to an average BD-Rate improvement of 0.19% and is still some way away
from the potential 1.87% improvement given by direct optimisation.

##  III Is the Current Lagrangian Multiplier Optimal Across a Modern Corpus?

Before exploring how to predict a better per-clip $k$, we propose first to
investigate if $k=1$, as set in the MPEG standard, is indeed optimal for our
corpus. What we propose here is to exhaustively calculate the BD-Rate over our
corpus for values of $k$ in the range $k$=0:0.001:3. The average BD-Rate
improvement for a given $k$ can be seen in Figure 3. The figure shows an
improvement where $k$ is approximately 0.7:0.8, with a best average
improvement coming at $k=0.782$. This also matches with the histogram of the
optimal $k$ values found with direct optimisation on the corpus videos (see
Figure 4). The following command invocation was used for all video encodes:

x265 --input SEQ.y4m --crf <XX> \--tune-PSNR --psnr --output OUT.mp4

where SEQ and OUT are the filenames for the raw input file and output encoded
video. Clips were encoded with <XX> in the range 22:5:42.

Figure 3: Average BD-Rate for a given k. Values of $k$ in the range
$[0.7,0.8]$ give a clear average improvement (negative BD-rate) across the
entire dataset.  Figure 4: Histogram of optimum $k$ (and by extension optimum
Lagrangian multiplier). The optimum is most frequently found in the
$k$=0.7-0.8 range.

For $k=0.782$, about two-thirds of the clips have an improvement but it is
worse for one-third of them (see Table II). Overall, we get an average gain of
0.32% across our corpus. This indicates that the Lagrangian Multiplier for
HEVC may not be the best. This would imply that for HEVC, we should adjust the
Lagrangian Multiplier to be $0.782\times$ its current value. However, this may
be specific to our corpus and the operating points which we are using.

Figure 5: CDF plots of the BD-Rate improvement across our corpus when using
$k$ = 0.7, 0.8 or 0.782. Ideally, we would want these plots to be as close as
possible to the top right and coincide with the optimal per-clip plot (in
black). We see for $k$ = 0.7, 0.8 or 0.782 positive BD-Rate improvements for
60-70% of the clips but worse performance for 30-40%.

##  IV Deep Learning for per clip Lagrangian Multiplier optimisation

We propose in this section to further explore the machine learning approach
proposed in our previous work [14]. The main contributions are 1) to extend
the feature set to per frame encoding statistics instead of global sequence
statistics, 2) to include semantic visual features, and lastly 3) to employ a
Deep Neural Network (DNN) instead of Random Forest.

###  IV-A Input Features

For this work, we collect two sets of features from each video from our
corpus. As with our previous work [14], our first set of features comes from
the statistics generated in the x265 encoder report.

TABLE I: List of Encoding Features used in Deep Learning system Features Per Clip | Features Per Frame
---|---
Average Bitrate | Bits
Average PSNR (Y, U, V, Global) | I/P cost Ratio
Average I, P and B frame count | PSNR (Y,U,V)
Bitrate for I, P and B frames | Avg Chroma Distortion
I Frame PSNR (Y,U,V) | Average Residual Energy
P Frame PSNR (Y,U,V) | Avg Luma Level
B Frame PSNR (Y,U,V) | Avg Cb Level
| Avg Cr Level

We expand from [14] by also including the per frame features (see full list in
Table I). This greatly expands the feature vector from 47 to 1523 features.
The features are collected for a clip encoded with no modifications to the
codec, at CRF 33.

In addition to these features, we also use the VGG16 network to generate
semantic visual features about a single frame from each video sequence. We
observed, indeed, that the nature of the clips content influenced the
Lagrangian optimisation. The 1000 features from the 3c7 layer of VGG16 are
used on a downsampled ($224\times 224$) median (75th) frame of the sequence.
These two feature vectors are then concatenated into a combined feature vector
of size 2523 for our model. Using the VGG features and NN architecture does
come at a cost of increased complexity, but this may be preferable to multiple
video encodes required for the direct optimisation method.

TABLE II:  Summary of BD-R Gains results. This shows the $\%$ of clips which have any improvement, $\>0.1\%$ and $\>1\%$ improvement, as well as the best and average positive BD Rate improvement  Method | Clips with BD-R Gain of | Best | Avg Final
---|---|---|---
$\geq 0\%$ | $>0.1\%$ | $>1\%$ | Gain | Gain
Direct Optimisation | 95% | 89% | 46% | 23.9% | 1.87%
Random Forest [14] (encoding features) | 67% | 28% | 2% | 8.5% | 0.19%
Random Forest (encoding new) | 68% | 41% | 11% | 12% | 0.23%
Random Forest (encoding + VGG features) | 69% | 65% | 28% | 18.3% | 0.82%
$k=0.7$ | 58% | 55% | 25% | 9.5% | 0.55%
$k=0.8$ | 62% | 56% | 23% | 16.9% | 0.58%
$k=0.782$ | 62% | 56% | 23% | 23.5% | 0.63%
DNN (encoding + VGG features) | 89% | 84% | 39% | 14.9% | 1.11%

###  IV-B Model & Training

TABLE III: Details of the Dense Layer sizes ($n$), dropout (0.1) and batch-normalisation (BN). All layers use the Gaussian Error Linear Unit (GELU) activation function. layer id | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
---|---|---|---|---|---|---|---|---|---|---|---
$n$ | 1000 | 800 | 600 | 200 | 100 | 64 | 32 | 16 | 8 | 4 | 1
BN | ✓ | ✓ |  |  |  |  |  |  |  |  | ✓
Dropout | ✓ | ✓ | ✓ |  |  |  |  | ✓ |  |  |

Our model consists in a sequence of fully connected layers with GELU
activation functions and Normalisation Layers. A full breakdown of the layers
can be found in table III. The corpus is split 70%/30% between training and
testing. The model was trained for 5000 epochs, using the SGD optimiser, a
learning rate of 0.001 and a momentum of 0.9. The loss function is the mean
absolute error between the predicted and directly optimised $k$.

##  V Results

Both our proposed deep learning model, and the Random Forest system outlined
in [14] (with the updated feature set) yield BD-Rate improvements comparable
to the direct optimisation. Figure 6 shows the portion of our test corpus
which have a BD-Rate improvement. In these graphs, as well as the results
presented in Table II, we see that roughly 95% of the clips had some BD-Rate
improvement, with 46% of the corpus having a 1% or better BD-Rate improvement
from direct optimisation. Our machine learning systems (Deep Learning and
Random Forest from [14]) give positive results with 90% and 70% of the corpus
getting a BD-Rate improvement but only 39% and 28% of the corpus getting a 1%
or better BD-Rate improvement. We can see that while all three random forest
implementations had a similar portion of the corpus show improvement, the
amount of improvement was best using both the expanded encoding features along
with the VGG features. The Random Forest and deep learning systems have a very
similar performance on clips which yield 5% or better BD-Rate improvements.
Both systems had roughly 1% of the corpus giving gains of 5% or more, but the
Direct Optimiser was able to achieve 5% or better gains in approximately 5% of
the clips. We believe that this is due to less training data available in the
”high improvement” range.

Figure 6: CDF plots of the BD-Rate improvement across our corpus for Random
Forest [14] and our proposed DNN method. We see that both per-clip approaches
outperform an overall adjustment of the Lagrangian multiplier ($k=0.782$) and
that our network reaches near optimal performance.

We also see that both machine learning systems drastically outperform the best
global Lagrangian multiplier adjustment ($k=0.782$), which confirms the
benefits of using a per clip approach.

Table II also reports the average positive BD-Rate improvement as our Average
Final BD-Rate Gain. This is the gain measured when our system is used as a
post-process or optimised re-run of the encoder given an initial encode with
default settings. The best BD-rate between the initial encode and the re-run
defines the final BD-Rate Gain. This is the most likely use of systems of this
kind, and can even be considered as a two-pass system. This allows us to take
advantage of any improvements in BD-Rate from Lagrangian multiplier
optimisation and default to the first default encode (unmodified Lagrangian
Multiplier) in the case of worse performance. In this situation, we can see
that our deep learning approach almost achieves what the direct optimiser
does, at a fraction of the computational cost. The direct optimiser would have
required on average sixty video encodes (12 RD-Curves) whereas our Deep
Learning approach only required ten (2 RD-Curves) to get near comparable
improvements.

##  VI Conclusion

In this paper, we have presented further data to support a per clip approach
for the Lagrangian Multiplier within the HEVC codec. We build upon our direct
optimisation results, by implementing a deep learning model capable of
comparable results in a fraction ($1/6$) of the computational cost. 90% of our
corpus shows some BD-Rate improvement with deep learning (compared to 95% of
the corpus with direct optimisation). We also see that we can get
approximately 1% BD-Rate savings across our entire corpus if we use the deep
learning system as a supplement to the existing codec. In summary, our deep
learning approach nears the direct optimiser results while using only
$1/6^{th}$ of the computational cost. Future work will include investigating
the deployment of this system for quality/distortion improvements at certain
bitrates instead of BD-Rate gain as well as expansion to newer codecs like
VVC.

## References

  * [1] S. Cass, “The age of the zettabyte cisco: the future of internet traffic is video [dataflow],” _IEEE Spectrum_ , vol. 51, no. 3, pp. 68–68, 2014.
  * [2] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” _IEEE Transactions on circuits and systems for video technology_ , vol. 22, no. 12, pp. 1649–1668, 2012\.
  * [3] T. Zhang and S. Mao, “An overview of emerging video coding standards,” _GetMobile: Mobile Computing and Communications_ , vol. 22, no. 4, pp. 13–20, 2019.
  * [4] B. Bross, J. Chen, S. Liu, and Y.-K. Wang, “Versatile video coding (draft 10),” _ITU-T and ISO/IEC JVET-S2001_ , 2020.
  * [5] D. Mukherjee, J. Bankoski, A. Grange, J. Han, J. Koleszar, P. Wilkins, Y. Xu, and R. Bultje, “The latest open-source video codec VP9-an overview and preliminary results,” in _2013 Picture Coding Symposium (PCS)_. IEEE, 2013, pp. 390–393.
  * [6] Y. Chen, D. Murherjee, J. Han, A. Grange, Y. Xu, Z. Liu, S. Parker, C. Chen, H. Su, U. Joshi _et al._ , “An overview of core coding tools in the av1 video codec,” in _2018 Picture Coding Symposium (PCS)_. IEEE, 2018, pp. 41–45.
  * [7] G. J. Sullivan and T. Wiegand, “Rate-distortion optimization for video compression,” _IEEE signal processing magazine_ , vol. 15, no. 6, pp. 74–90, 1998.
  * [8] T. Wiegand and B. Girod, “Lagrange multiplier selection in hybrid video coder control,” in _Proceedings 2001 International Conference on Image Processing_ , vol. 3. IEEE, 2001, pp. 542–545.
  * [9] A. Ortega and K. Ramchandran, “Rate-distortion methods for image and video compression,” _IEEE Signal processing magazine_ , vol. 15, no. 6, pp. 23–50, 1998.
  * [10] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the high efficiency video coding HEVC standard,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, no. 12, pp. 1649–1668, Dec 2012.
  * [11] I. Katsavounidis and L. Guo, “Video codec comparison using the dynamic optimizer framework,” in _Applications of Digital Image Processing XLI_ , vol. 10752. International Society for Optics and Photonics, 2018, p. 107520Q.
  * [12] M. Covell, M. Arjovsky, Y.-C. Lin, and A. Kokaram, “Optimizing transcoder quality targets using a neural network with an embedded bitrate model,” _Electronic Imaging_ , vol. 2016, no. 2, pp. 1–7, 2016.
  * [13] D. J. Ringis, F. Pitie, and A. Kokaram, “Per clip lagrangian multiplier optimisation for (HEVC),” _Electronic Imaging_ , vol. 2020, no. 12, 2020\. [Online]. Available: https://www.ingentaconnect.com/content/ist/ei/pre-prints/content-ei2020-ipas-136
  * [14] D. J. Ringis, F. Pitié, and A. Kokaram, “Per-clip adaptive lagrangian multiplier optimisation with low-resolution proxies,” in _Applications of Digital Image Processing XLIII_ , vol. 11510. International Society for Optics and Photonics, 2020, p. 115100E.
  * [15] T. Wiegand, M. Lightstone, D. Mukherjee, T. G. Campbell, and S. K. Mitra, “Rate-distortion optimized mode selection for very low bit rate video coding and the emerging H. 263 standard,” _IEEE Trans. on Circuits and Systems for Video Technology_ , vol. 6, no. 2, pp. 182–190, 1996.
  * [16] C. Ma, K. Naser, V. Ricordel, P. Le Callet, and C. Qing, “An adaptive lagrange multiplier determination method for dynamic texture in HEVC,” in _2016 IEEE International Conference on Consumer Electronics-China (ICCE-China)_. IEEE, 2016, pp. 1–4.
  * [17] A. M. Hamza, A. Abdelazim, and D. Ait-Boudaoud, “Parameter optimization in H. 265 rate-distortion by single frame semantic scene analysis,” _Electronic Imaging_ , vol. 2019, no. 11, pp. 262–1, 2019.
  * [18] F. Zhang and D. R. Bull, “Rate-distortion optimization using adaptive lagrange multipliers,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 3121–3131, Oct 2019.
  * [19] M. A. Papadopoulos, F. Zhang, D. Agrafiotis, and D. Bull, “An adaptive QP offset determination method for HEVC,” in _2016 IEEE International Conference on Image Processing (ICIP)_ , Sep. 2016, pp. 4220–4224.
  * [20] A. Yang, H. Zeng, J. Chen, J. Zhu, and C. Cai, “Perceptual feature guided rate distortion optimization for high efficiency video coding,” _Multidimensional Systems and Signal Processing_ , vol. 28, no. 4, pp. 1249–1266, 2017.
  * [21] S. John, A. Gadde, and B. Adsumilli, “Rate distortion optimization over large scale video corpus with machine learning,” _arXiv preprint arXiv:2008.12408_ , 2020.
  * [22] Y. Wang, S. Inguva, and B. Adsumilli, “Youtube UGC dataset for video compression research,” in _2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP)_ , Sep. 2019, pp. 1–5.
  * [23] Netflix, “Netflix Open Content,” https://opencontent.netflix.com/.
  * [24] B. Ghanem and N. Ahuja, “Maximum margin distance learning for dynamic texture recognition,” in _European Conference on Computer Vision_. Springer, 2010, pp. 223–236.
  * [25] J. Y. Lin, L. Jin, S. Hu, I. Katsavounidis, Z. Li, A. Aaron, and C.-C. J. Kuo, “Experimental design and analysis of jnd test on coded image/video,” in _Applications of digital image processing XXXVIII_ , vol. 9599. International Society for Optics and Photonics, 2015, p. 95990Z.
  * [26] Xiph.org, “Xiph.org Video Test Media,” https://media.xiph.org/video/derf/.