# A remark on 0-cycles as modules over algebras of finite correspondences

M.Rovinsky  HSE University (AG Laboratory, HSE, 6 Usacheva str., Moscow,
Russia, 119048) & Institute for Information Transmission Problems of Russian
Academy of Sciences<EMAIL_ADDRESS>
###### Abstract.

Given a smooth projective variety $X$ over a field, consider the
$\mathbb{Q}$-vector space $Z_{0}(X)$ of 0-cycles (i.e. formal finite
$\mathbb{Q}$-linear combinations of the closed points of $X$) as a module over
the algebra of finite correspondences. Then the rationally trivial 0-cycles on
$X$ form an absolutely simple and essential submodule of $Z_{0}(X)$.

Let $k$ be a field. There are several ways and their versions in which the
zero-cycles on $k$-schemes of finite type can be considered as a functor. In
each of these versions, we want this functor to be an object of an abelian
category, and we study its structure (“composition series”).

Consider a set $S$ of smooth projective varieties over a fixed field. Let
$Z_{0}(S)$ be the direct sum of the $\mathbb{Q}$-vector spaces of 0-cycles
(i.e. formal finite linear combinations of the closed points) on varieties in
$S$ with rational coefficients.

We consider $Z_{0}(S)$ as a module over the algebra of finite correspondences.

The aim of this note is to show that the rationally trivial 0-cycles form an
absolutely simple submodule $Z_{0}^{\mathrm{rat}}(S)$ of the module
$Z_{0}(S)$, which is contained in all non-zero submodules of $Z_{0}(S)$. To
some extent, this is analogous to the minimality of the rational equivalence
among all ‘adequate’ equivalence relations on algebraic cycles, cf. [12,
Proposition 8].

Assuming the Beilinson–Bloch motivic filtration conjecture, we show that the
radical filtration on $Z_{0}(S)/Z_{0}^{\mathrm{rat}}(S)$ is an evident
modification of the conjectural motivic filtration on Chow groups of 0-cycles.
This is checked unconditionally in the case of curves.

In the last section, a point of view on the 0-cycles on smooth, but not
necessarily proper, varieties as a cosheaf in an appropriate topology is
briefly discussed.

##  1\. Category algebras and non-degenerate modules

A category $\mathcal{C}$ is called preadditive if, for each pair of objects
$X,Y$, the morphism set $\mathrm{Hom}_{\mathcal{C}}(X,Y)$ is endowed with an
abelian group structure, while the morphism composition maps
$\circ_{X,Y,Z}:\mathrm{Hom}_{\mathcal{C}}(X,Y)\times\mathrm{Hom}_{\mathcal{C}}(Y,Z)\to\mathrm{Hom}_{\mathcal{C}}(X,Z)$
are bilinear for all objects $X,Y,Z$.

For any small preadditive category $\mathcal{C}$, set
$A_{\mathcal{C}}:=\bigoplus_{X,Y\in\mathcal{C}}\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(X,Y)$.

The composition pairings $\circ_{X,Y,Z}$ (and the zero pairings between the
groups $\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(W,X)$ and
$\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(Y,Z)$ for all quadruples
$W,X,Y,Z$ with $X\neq Y$) induce an associative ring structure on the abelian
group $A_{\mathcal{C}}$.

The ring $A_{\mathcal{C}}$ is unital if and only if there are only finitely
many objects in $\mathcal{C}$. However, even if $A_{\mathcal{C}}$ is not
unital, it is idempotented (in the sense of [2, Definition 4]), i.e. for every
finite collection $B$ of elements of $A_{\mathcal{C}}$ there is an idempotent
$e\in A_{\mathcal{C}}$ such that $ea=a$ for all $a\in B$. Namely the sums of
identities
$\mathrm{id}_{X}\in\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(X,X)\subseteq
A_{\mathcal{C}}$ for all objects $X$ in a finite set containing the union of
the supports of the elements of $B$. (By definition, the support of an element
$a$ is the smallest set $\mathrm{Supp}(a)$ such that
$a\in\bigoplus_{X,Y\in\mathrm{Supp}(a)}\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(X,Y)\subseteq
A_{\mathcal{C}}$.)

Recall, cf. e.g. [3, p.113], that a left module $M$ over an associative ring
$A$ is called non-degenerate if $AM=M$. Obviously, $A_{\mathcal{C}}$ is a non-
degenerate left $A_{\mathcal{C}}$-module.

Denote by $\mathrm{Mod}_{\mathcal{C}}$ the category of non-degenerate left
$A_{\mathcal{C}}$-modules.

Denote by $\mathcal{C}^{\vee}$ the category of additive functors from
$\mathcal{C}$ to the category of abelian groups.

######  Lemma 1.1 (Morita equivalence).

If $\mathcal{C}$ is a small preadditive category then $\mathcal{C}^{\vee}$ and
$\mathrm{Mod}_{\mathcal{C}}$ are equivalent abelian categories. In particular,
if two small preadditive categories $\mathcal{C}$ and $\mathcal{C}^{\prime}$
are equivalent then the categories $\mathrm{Mod}_{\mathcal{C}}$ and
$\mathrm{Mod}_{\mathcal{C}^{\prime}}$ are equivalent as well.

###### Proof.

We send any functor $\mathcal{F}$ from $\mathcal{C}$ to the category of
abelian groups to $\bigoplus_{X\in\mathcal{C}}\mathcal{F}(X)$, which is a non-
degenerate $A_{\mathcal{C}}$-module in an obvious way.

In the opposite direction, given an $A_{\mathcal{C}}$-module $M$ and an object
$X$, we set $\mathcal{F}(X):=\mathrm{id}_{X}(M)$. Any morphism
$f\in\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(X,X^{\prime})\subseteq
A_{\mathcal{C}}$ induces a map
$\mathcal{F}(X)=\mathrm{id}_{X}(M)\xrightarrow{f}f\circ\mathrm{id}_{X}(M)=\mathrm{id}_{X^{\prime}}\circ
f\circ\mathrm{id}_{X}(M)\subseteq\mathrm{id}_{X^{\prime}}(M)=\mathcal{F}(X^{\prime})$.

It is easy to see that these two functors are quasi-inverse equivalences. In
particular, we get a chain of equivalences:
$\mathrm{Mod}_{\mathcal{C}}\simeq\mathcal{C}^{\vee}\simeq(\mathcal{C}^{\prime})^{\vee}\simeq\mathrm{Mod}_{\mathcal{C}^{\prime}}$.
∎

The Yoneda embedding
$\mathcal{C}\to\mathcal{C}^{\vee}\simeq\mathrm{Mod}_{\mathcal{C}}$, $X\mapsto
h_{X}:=\mathop{\mathrm{Hom}}\nolimits_{\mathcal{C}}(X,-)$ is a fully faithful
functor. We are interested in the structure of the $A_{\mathcal{C}}$-module
$h_{X}$ for the ‘unit’ object $X$.

##  2\. Algebras of finite correspondences and their modules

Fix a field $k$. For each pair of smooth $k$-varieties $X$ and $Y$, define
$\mathrm{Cor}(X,Y)_{\mathbb{Q}}$ as the $\mathbb{Q}$-vector space with a basis
given by the irreducible closed subsets of $X\times_{k}Y$ whose associated
integral subschemes are finite, flat and surjective over a connected component
of $X$.

For each triple of smooth $k$-varieties $(X,Y,Z)$, define the bilinear pairing
$\mathrm{Cor}(X,Y)_{\mathbb{Q}}\times\mathrm{Cor}(Y,Z)_{\mathbb{Q}}\xrightarrow{\circ_{X,Y,Z}}\mathrm{Cor}(X,Z)_{\mathbb{Q}}$
in the standard way: $(\alpha,\beta)\mapsto\mathrm{pr}_{XZ*}(\alpha\times
Z\cap X\times\beta)$, see [5, Ch. 1].

These pairings as compositions, turn the category of smooth $k$-varieties with
morphisms $\mathrm{Cor}(-,-)_{\mathbb{Q}}$ into an additive category, denoted
$\mathrm{SmCor}_{k}$. Denote by $\mathrm{SmCor}_{k}^{\mathrm{proj}}$ the full
subcategory of projective $k$-varieties.

Given a set $S$ of smooth $k$-varieties, we may consider $S$ as a full
subcategory of $\mathrm{SmCor}_{k}$. As the category $S$ is preadditive, the
direct sum $A_{S}:=\bigoplus_{X,Y\in S}\mathrm{Cor}(X,Y)_{\mathbb{Q}}$ carries
a ring structure.

###  2.1. The socle of $Z_{0}(S)$

For each smooth variety $Y$ over $k$, let
$Z_{0}(Y):=\mathrm{Cor}(\mathrm{Spec}(k),Y)_{\mathbb{Q}}$ be the
$\mathbb{Q}$-vector space of 0-cycles on $Y$.

######  Lemma 2.1.

Let $X$ be a smooth quasiprojective variety over $k$, $F$ be a characteristic
zero field, and $\xi\in Z_{0}(Y)\otimes F$ be a non-zero 0-cycle. Then there
exists a correspondence $\vartheta\in\mathrm{Cor}(X,\mathbb{P}^{1}_{k})\otimes
F$, such that $\vartheta\xi=[0]-[\infty]\in Z_{0}(\mathbb{P}^{1}_{k})$.

###### Proof.

Let $\xi=\sum_{i=1}^{N}m_{i}[p_{i}]$ for non-zero $m_{i}\in F$ and closed
points $p_{i}\in X$.

By a refinement of the projective version of the Noether normalization lemma
proved in [9, Theorem 1], $X$ admits a morphism
$\varphi:X\to\mathbb{P}^{n}_{k}$, where $n:=\dim X$, which maps
$p_{2},\dots,p_{N}$ into a hyperplane $H\subset\mathbb{P}^{n}_{k}$ and maps
$p_{1}$ to the complement of $H$. Set $p_{i}^{\prime}:=\varphi(p_{i})$ for all
$i$, so $p_{2}^{\prime},\dots,p_{N}^{\prime}\in H$,
$p_{1}^{\prime}\in\mathbb{P}^{n}_{k}\smallsetminus H$. This means that
$\varphi_{*}\xi=\sum_{i=1}^{N}m_{i}[p_{i}^{\prime}]\neq 0$.

Let us show by induction on $N\geq 2$ that there exists a finite endomorphism
$\psi:\mathbb{P}^{n}_{k}\to\mathbb{P}^{n}_{k}$ sending the points
$p_{2}^{\prime},\dots,p_{N}^{\prime}$ to a single $k$-rational point
$p\in\mathbb{P}^{n}_{k}$ and sending the point $p_{1}^{\prime}$ to a distinct
$k$-rational point $q\in\mathbb{P}^{n}_{k}$, $q\neq p$. Let
$W_{0},\dots,W_{n}$ be homogeneous coordinates on $\mathbb{P}^{n}_{k}$ such
that $H$ is given by the equation $W_{0}=0$, while both $p_{2}^{\prime}$ and
$p_{3}^{\prime}$ do not lie on the hyperplane given by the equation $W_{1}=0$.

For each $2\leq i\leq n$, set $w_{i}:=W_{i}/W_{1}$, and let $P_{ij}$ be the
minimal polynomial of $w_{i}(p_{j}^{\prime})$ over $k$.

Set $d:=\max\limits_{2\leq i\leq n}\deg(P_{i2}P_{i3})$, and
$P_{i}:=P_{i2}(w_{i})P_{i3}(w_{i})w_{i}^{d-\deg(P_{i2}P_{i3})}W_{1}^{d}$. Then
the map

$g:(W_{0}:\ldots:W_{N})\mapsto(W_{0}^{d}:W_{1}^{d}:P_{1}:P_{2}:\ldots:P_{N})$

is a well-defined endomorphism of $\mathbb{P}^{n}_{k}$, $g$ preserves $H$, the
point $g(p_{2}^{\prime})=g(p_{3}^{\prime})$ is $k$-rational, and $g$
transforms $\varphi_{*}\xi$ to
$m_{1}[p_{1}^{\prime\prime}]+(m_{2}+m_{3})[p_{3}^{\prime\prime}]+\sum_{i=4}^{N}m_{i}[p_{i}^{\prime\prime}]$,
where $p_{3}^{\prime\prime},\dots,p_{N}^{\prime\prime}\in H$ and
$p_{1}^{\prime\prime}\notin H$.

Then $\psi_{*}\varphi_{*}\xi$ is a non-zero multiple of $[p]-[q]$.

Let $\Upsilon$ be an $n$-dimensional variety admitting a non-constant morphism
$h:\Upsilon\to\mathbb{P}^{1}_{k}$ (e.g.,
$\Upsilon=\mathbb{P}^{n-1}\times\mathbb{P}^{1}_{k}$ and
$h:\Upsilon\to\mathbb{P}^{1}_{k}$ is the projection). Fix a fibre $D$ of $h$,
and a hyperplane $H^{\prime}\subset\mathbb{P}^{n}_{k}$ containing $p$ but not
$q$. By the same [9, Theorem 1], there exists a finite morphism
$\pi:\Upsilon\to\mathbb{P}^{n}_{k}$ such that $\pi(D)=H^{\prime}$, so $D$
meets $\pi^{-1}(p)$ but not $\pi^{-1}(q)$, and therefore,
$h_{*}\pi^{*}\psi_{*}\varphi_{*}\xi\neq 0$. Then
$h_{*}({}^{t}\Gamma_{\pi})_{*}\psi_{*}\varphi_{*}\xi=h_{*}\pi^{*}\psi_{*}\varphi_{*}\xi$
is a non-zero divisor $E=\sum\limits_{i=0}^{n}a_{i}[q_{i}]$ on
$\mathbb{P}^{1}_{k}$ for some $a_{i}\neq 0$ and pairwise distinct $q_{i}$.

Choose a morphism $f:\mathbb{P}^{1}_{k}\to\mathbb{P}^{1}_{k}$ such that
$f(q_{0})=0$, $f(q_{i})=\infty$ for all $1\leq i\leq n$, so
$f_{*}h_{*}({}^{t}\Gamma_{\pi})_{*}\psi_{*}\varphi_{*}\xi=a_{0}([0]-[\infty])$.
∎

For each set $S$ of smooth varieties over $k$, consider
$Z_{0}(S):=\bigoplus_{X\in S}Z_{0}(X)$. Then the above pairings
$\circ_{\mathrm{Spec}(k),Y,Z}:Z_{0}(Y)\times\mathrm{Cor}(Y,Z)_{\mathbb{Q}}\to
Z_{0}(Z)$, given by $(\alpha,\beta)\mapsto\mathrm{pr}_{Z*}(\alpha\times
Z\cap\beta)$, induce an $A_{S}$-module structure on $Z_{0}(S)$.

Define the degree of a 0-cycle $\alpha=\sum_{i}m_{i}P_{i}$ on $X$ by
$\deg(\alpha):=\sum_{i}m_{i}[\varkappa(P_{i}):k]$, where $\varkappa(P_{i})$ is
the residue field of $P_{i}$.

For each smooth variety $Y$ over $k$, let $Z_{0}^{\circ}(Y)$ be the subspace
of 0-cycles of degree 0 on each connected component of $Y$.

Obviously, $Z_{0}^{\circ}(S):=\bigoplus\limits_{X\in S}Z_{0}^{\circ}(X)$ is an
$A_{S}$-submodule of $Z_{0}(S)$.

Recall ([12, §2], [5, Ch. 1]), that a cycle is called rationally equivalent to
zero (or rationally trivial) if it is a sum of divisors of rational functions
on subvarieties.

######  Theorem 2.2.

Let $S$ be a set of smooth varieties over $k$, and $F$ be a characteristic
zero field. Then

  1. (1)

any proper $(A_{S}\otimes F)$-submodule of $Z_{0}(S)\otimes F$ is contained in
the submodule $Z_{0}^{\circ}(S)\otimes F$;

  2. (2)

if $S$ consists of projective varieties then any non-zero $(A_{S}\otimes
F)$-submodule of $Z_{0}(S)\otimes F$ contains the $A_{S}$-submodule

$Z_{0}^{\mathrm{rat}}(S):=\bigoplus\limits_{X\in S}Z_{0}^{\mathrm{rat}}(X)$

of 0-cycles rationally equivalent to 0 on all $X\in S$.

###### Proof.

It is clear that if $S^{\prime}$ is the set of connected components of
varieties in $S$ then $A_{S^{\prime}}$ and $A_{S}$ are naturally isomorphic,
while $Z_{0}(S^{\prime})$ and $Z_{0}(S)$ coincide as $A_{S}$-modules. This
means that we may assume that all varieties in $S$ are connected. Given any
characteristic zero field $F$ and any non-zero element $\xi=(\xi_{X})_{X\in
S}\in Z_{0}(S)\otimes F$, there is $X\in S$ such that $\xi_{X}\neq 0$, so
$\xi^{\prime}:=\mathrm{id}_{X}\xi\neq 0$.

  1. (1)

For any $Y\in S$ and any closed point $y\in Y$, the finite correspondence
$[X\times_{k}y]\in\mathrm{Cor}(X,Y)_{\mathbb{Q}}$ maps $\xi_{X}$ to the
0-cycle $\deg(\xi_{X})\cdot[y]\in Z_{0}(Y)\otimes F$, so if $\deg(\xi_{X})\neq
0$ then $\xi^{\prime}$ (and therefore, $\xi$) generates the whole
$(A_{S}\otimes F)$-module $Z_{0}(S)\otimes F$, which is equivalent to (1).

  2. (2)

According to (1), we may further assume that $\deg(\xi_{X})=0$ and, as
$\xi_{X}\neq 0$, that $\dim X>0$.

By Lemma 2.1, there exists a correspondence
$\vartheta\in\mathrm{Cor}(X,\mathbb{P}^{1}_{k})\otimes F$, such that
$\vartheta\xi_{X}=[0]-[\infty]\in Z_{0}(\mathbb{P}^{1}_{k})$.

Finally, for each $Y\in S$, any 0-cycle on $Y$ rationally equivalent to 0 is a
linear combination of images of the cycle $[0]-[\infty]$ under finite
correspondences $\gamma$ from $\mathbb{P}^{1}_{k}$ to $Y$, i.e. of elements
$(\gamma\circ\vartheta)_{*}\xi_{X}$ for appropriate $\gamma$’s.

∎

######  Remark 2.3.

A module $M$ over a $\mathbb{Q}$-algebra $A$ is called absolutely simple if
$M\otimes F$ is a simple $(A_{S}\otimes F)$-module for any characteristic zero
field $F$. Equivalently, the $A$-module $M$ is simple and
$\mathrm{End}_{A}(M)=\mathbb{Q}$. In particular, in the setting of Theorem
2.2, the $A_{S}$-modules $Z_{0}(S)/Z_{0}^{\circ}(S)$ and
$Z_{0}^{\mathrm{rat}}(S)$ are absolutely simple, whenever $S$ is non-empty.

###  2.2. Motivic $A_{S}$-modules

By definition ([12]), an equivalence relation $\sim$ is adequate if it
satisfies the following conditions:

  * •

it is compatible with the addition of cycles, i.e. a subgroup $Z^{\sim}(X)$ of
cycles on each variety $X$ is fixed, and two cycles on $X$ are equivalent if
and only if their difference belongs to $Z^{\sim}(X)$;

  * •

for any variety $X$, any cycle $\alpha$ on $X$, and any subvariety $W$ on $X$,
there exists a cycle $\alpha^{\prime}\sim\alpha$ intersecting $W$ properly;

  * •

for any pair of smooth projective varieties $X$ and $Y$, a cycle $\beta\sim 0$
on $X$, and a cycle $\alpha$ on $X\times Y$ intersecting $\beta\times Y$
properly, the cycle $\alpha(\beta):=\mathrm{pr}_{Y}((\beta\times
Y)\cdot\alpha)$ is $\sim$-equivalent to 0 on $Y$.

######  Example 2.4 ([12], §2).

Besides the rational equivalence mentioned above, the following equivalence
relations are adequate.

  * •

A cycle $\alpha$ on a smooth projective variety $X$ is called algebraically
equivalent to zero if there exist a curve $C$, points $c,d\in C$ and a cycle
$\beta$ on $X\times C$, which is flat over $C$ such that
$\alpha=[\beta\cap(X\times\\{c\\})]-[\beta\cap(X\times\\{d\\})]$.

  * •

A cycle on a smooth projective variety is called homologically equivalent to
zero (with respect to a fixed Weil cohomology theory) if it is annihilated by
the cycle map.

  * •

A cycle $\alpha$ on a smooth projective variety is called numerically
equivalent to zero if $\deg(\alpha\cap W)=0$ for any subvariety $W$ of the
complementary dimension that meet $\alpha$ properly.

Recall (see, e.g., [11]), that a (homological) effective Grothendieck motive
over $k$ modulo an ‘adequate’ equivalence relation $\sim$ is defined as a pair
$(X,\pi)$ consisting of a smooth projective variety $X$ over $k$ and a
projector $\pi$ in the algebra of self-correspondences on $X$ of dimension
$\dim X$ with coefficients in $\mathbb{Q}$ modulo $\sim$. The morphisms
between pairs $(X,\pi)$ and $(X^{\prime},\pi^{\prime})$ are algebraic cycles
$\alpha$ on $X\times_{k}X^{\prime}$ of dimension $\dim X$ modulo $\sim$, and
such that $\alpha=\pi^{\prime}\circ\alpha\circ\pi$.

The motives over $k$ modulo an equivalence relation $\sim$ form a pseudo-
abelian category, denoted by $\mathcal{M}_{k,\mathrm{eff}}^{\sim}$. The
category $\mathcal{M}_{k,\mathrm{eff}}^{\sim}$ carries a tensor structure:
$(X,\pi)\otimes(X^{\prime},\pi^{\prime}):=(X\times_{k}X^{\prime},\pi\times\pi^{\prime})$.

Denote by
$\mathbb{M}^{\sim}:\mathrm{SmCor}_{k}^{\mathrm{proj}}\to\mathcal{M}_{k,\mathrm{eff}}^{\sim}$
the additive functor $X\mapsto(X,\Delta_{X})$, where $\Delta_{X}$ is the class
of the diagonal in $X\times_{k}X$. In particular,

$\mathbb{M}^{\sim}(\mathbb{P}^{1}_{k})\cong\mathbb{M}^{\sim}(\mathrm{Spec}(k))\oplus\mathbb{L},\quad\mbox{where
$\mathbb{L}=(\mathbb{P}^{1}_{k},[\\{q\\}\times\mathbb{P}^{1}_{k}])$}$

for any rational point $q\in\mathbb{P}^{1}(k)$. It is easy to see that the
natural map

$\mathrm{Hom}_{\mathcal{M}_{k,\mathrm{eff}}^{\sim}}(U,V)\to\mathrm{Hom}_{\mathcal{M}_{k,\mathrm{eff}}^{\sim}}(U\otimes\mathbb{L},V\otimes\mathbb{L})$

is bijective for all effective motives $U$ and $V$.

Denote by $\mathcal{M}_{k}^{\sim}$ the category of triples $(X,\pi,n)$, where
$(X,\pi)$ are as above and $n$ is an integer, while
$\mathrm{Hom}_{\mathcal{M}_{k}^{\sim}}((X,\pi,n),(X^{\prime},\pi^{\prime},n^{\prime})):=\mathrm{Hom}_{\mathcal{M}_{k,\mathrm{eff}}^{\sim}}((X,\pi)\otimes\mathbb{L}^{\otimes(m+n-n^{\prime})},(X^{\prime},\pi^{\prime})\otimes\mathbb{L}^{\otimes
m})$ for any integer $m>|n^{\prime}-n|$. We consider
$\mathcal{M}_{k,\mathrm{eff}}^{\sim}$ as a full subcategory of
$\mathcal{M}_{k}^{\sim}$ under the embedding $(X,\pi)\mapsto(X,\pi,0)$.

For each variety $Y$ and an integer $q$, denote by $CH_{q}(Y)$ the group of
dimension $q$ cycles on $Y$ modulo the rational equivalence.

######  Theorem 2.5.

The functor $\mathbb{M}^{\sim}$ is full. In other words, the natural ring
homomorphism $A_{S}\to\bigoplus_{X,Y\in S}CH_{\dim
X}(X\times_{k}Y)_{\mathbb{Q}}$ is surjective for any set $S$ of smooth
projective varieties over $k$.

###### Proof.

This is a particular case of [4, Theorem 7.1]. ∎

For any set $S$ of smooth projective vaieties over $k$, each Grothendieck
motive $N\in\mathcal{M}_{k}^{\sim}$ gives rise to an $A_{S}$-module
$\mathfrak{M}^{\sim}_{N}(S):=\bigoplus_{X\in
S}\mathop{\mathrm{Hom}}\nolimits_{\mathcal{M}_{k}^{\sim}}(N,\mathbb{M}^{\sim}(X))$.

We omit the symbol $\sim$ from the notation when $\sim=\sim_{\mathrm{num}}$ is
the numerical equivalence.

######  Corollary 2.6.

For any motive $N\in\mathcal{M}_{k}$, the $A_{S}$-module $\mathfrak{M}_{N}(S)$
is semisimple.

###### Proof.

The $A_{S}$-action on $\mathfrak{M}_{N}(S)$ factors through an action of the
algebra $A_{S}/\sim_{\mathrm{num}}$, while
$A_{S}/\sim_{\mathrm{rat}}\cong\bigoplus_{X,Y\in S}CH_{\dim
X}(X\times_{k}Y)_{\mathbb{Q}}$, so
$A_{S}/\sim_{\mathrm{num}}\cong\bigoplus_{X,Y\in S}CH_{\dim
X}(X\times_{k}Y)_{\mathbb{Q}}/\sim_{\mathrm{num}}$.

By [7], $\mathcal{M}_{k}$ is an abelian semisimple category, and therefore,
any non-degenerate
$(A_{S}/\sim_{\mathrm{num}})$-module is semisimple. In particular, so is the
$A_{S}$-module $\mathfrak{M}_{N}(S)$. ∎

##  3\. Loewy filtrations on $Z_{0}(S)$

Modifying slightly the standard definition (see, e.g. [6]), a filtration of a
module $M$ is called a Loewy filtration if it is finite, its successive
quotients are semisimple and its length is minimal under these assumptions.

Let $S$ be a set of smooth irreducible projective varieties over a field $k$.
We are interested in Loewy filtrations on the $A_{S}$-module $Z_{0}(S)$.

By Theorem 2.2, the socle (i.e. the maximal semisimple submodule) of the
$A_{S}$-module $Z_{0}(S)$ is $Z_{0}^{\mathrm{rat}}(S)$, while the radical
(i.e. the intersection of all maximal submodules) of the $A_{S}$-module
$Z_{0}(S)$ is $Z_{0}^{\circ}(S)$, and $Z_{0}^{\mathrm{rat}}(S)$ is an
essential submodule of $Z_{0}(S)$.

The $A_{S}$-action on the quotient
$CH_{0}(S)_{\mathbb{Q}}:=Z_{0}(S)/Z_{0}^{\mathrm{rat}}(S)$ factors through an
action of the quotient $A_{S}/\sim_{\mathrm{rat}}$ of $A_{S}$ by the rational
equivalence.

###  3.1. The case of curves

######  Proposition 3.1.

Let $S$ be a set of smooth projective curves over $k$.

Then $Z_{0}^{\mathrm{rat}}(S)\subset Z_{0}^{\circ}(S)\subset Z_{0}(S)$ is the
unique Loewy filtration on the $A_{S}$-module $Z_{0}(S)$.

###### Proof.

By Theorem 2.2, the socle of the $A_{S}$-module $Z_{0}(S)$ is simple and
coincides with $Z_{0}^{\mathrm{rat}}(S)$, while $Z_{0}^{\circ}(S)$ is the
unique maximal submodule of the $A_{S}$-module $Z_{0}(S)$. There remains only
to check the semisimplicity of $Z_{0}^{\circ}(S)/Z_{0}^{\mathrm{rat}}(S)$.

One has $A_{S}/\sim_{\mathrm{rat}}=\bigoplus_{X,Y\in
S}\mathrm{Pic}(X\times_{k}Y)_{\mathbb{Q}}$. Then the subgroup

$I:=\bigoplus_{X,Y\in S}\mathrm{Pic}^{\circ}(X\times_{k}Y)_{\mathbb{Q}}$

is an ideal in $A_{S}/\sim_{\mathrm{rat}}$ with $I^{2}=0$, while
$(A_{S}/\sim_{\mathrm{rat}})/I=\bigoplus_{X,Y\in
S}\mathrm{NS}(X\times_{k}Y)_{\mathbb{Q}}$ is a semisimple algebra. Here
$\mathrm{Pic}$ is the Picard group, $\mathrm{Pic}^{\circ}$ is the subgroup of
algebraically trivial elements,
$\mathrm{NS}:=\mathrm{Pic}/\mathrm{Pic}^{\circ}$ is the Néron–Severi group.

Then, for any $(A_{S}/\sim_{\mathrm{rat}})$-module $M$, the submodule $IM$ and
the quotient $M/IM$ can be considered as
$(A_{S}/\sim_{\mathrm{rat}})/I$-modules, and thus, they are semisimple.
Applying this to the module $M=Z_{0}(S)/Z_{0}^{\mathrm{rat}}(S)$, we see that
the $A_{S}$-module $IM=Z_{0}^{\circ}(S)/Z_{0}^{\mathrm{rat}}(S)$ is
semisimple. ∎

###  3.2. Consequences of the filtration conjecture

According to the Bloch–Beilinson motivic filtration conjecture (e.g., [8,
Conjecture 2.3], [10, Conjecture 33]), there should exist a neutral tannakian
$\mathbb{Q}$-linear category $\mathcal{MM}_{k}$ (of mixed motives over $k$)
containing the category $\mathcal{M}_{k}$ as the full subcategory of the
semisimple objects, covariant functors $H_{i}(-,\mathbb{Q}(j))$ (homology;
$i,j\in\mathbb{Z}$) from the category of varieties over $k$ to
$\mathcal{MM}_{k}$, and a functorial descending filtration
$\mathcal{F}^{\bullet}$ on the Chow groups $CH_{q}(X)_{\mathbb{Q}}$ for smooth
projective $k$-varieties $X$ such that
$\mathcal{F}^{0}CH_{q}(X)_{\mathbb{Q}}=CH_{q}(X)_{\mathbb{Q}}$ and

$gr^{i}_{\mathcal{F}}CH^{q}(X)_{\mathbb{Q}}=\mathrm{Ext}^{i}_{\mathcal{MM}_{k}}(\mathbb{Q}(0),H_{2q+i}(X,\mathbb{Q}(-q))).$

As a part of the filtration conjecture, it is natural to assume the
Grothendieck’s ‘semisimplicity conjecture’ on the coincidence of homological
$\otimes\mathbb{Q}$ and numerical equivalences, so that the motive
$H_{2q+i}(X,\mathbb{Q}(-q))$ is semisimple by U. Jannsen’s theorem, [7].

A simple effective motive $P\in\mathcal{M}_{k}$ is called primitive of weight
$-i\leq 0$ if, (i) $P\cong(X,\pi)$ for some $X,\pi$ with $\dim X=i$, and (ii)
$\mathrm{Hom}_{\mathcal{M}_{k}}(P,\mathbb{M}(Y\times\mathbb{P}^{1}))=0$ for
any smooth projective variety $Y$ of dimension $<i$.

In particular, when $q=0$ the Beilinson formula becomes

$gr^{i}_{\mathcal{F}}CH_{0}(X)_{\mathbb{Q}}=\mathrm{Ext}^{i}_{\mathcal{MM}_{k}}(\mathbb{Q}(0),H_{i}(X,\mathbb{Q}))\\\
=\bigoplus_{P}\mathrm{Ext}^{i}_{\mathcal{MM}_{k}}(\mathbb{Q}(0),P)\otimes_{\mathrm{End}_{\mathcal{M}_{k}}(P)}\mathrm{Hom}_{\mathcal{MM}_{k}}(P,H_{i}(X,\mathbb{Q})),$

where $P$ runs over the isomorphism classes of simple primitive motives of
weight $-i$, and we see that the spaces
$\mathcal{F}^{i}CH_{0}(X)_{\mathbb{Q}}$ should be covariant functorial.

For each set $S$ of smooth irreducible projective varieties over a field $k$,
and each integer $i\geq 0$, consider
$\mathcal{F}^{i}CH_{0}(S)_{\mathbb{Q}}:=\bigoplus_{X\in
S}\mathcal{F}^{i}CH_{0}(X)_{\mathbb{Q}}$. By the functoriality of
$\mathcal{F}^{\bullet}$, this is an $A_{S}$-submodule of
$CH_{0}(S)_{\mathbb{Q}}$.

The algebra $A_{S}$ acts on $gr^{i}_{\mathcal{F}}CH_{0}(S)_{\mathbb{Q}}$ via
its action on the motives $H_{i}(X,\mathbb{Q})$, so the $A_{S}$-action on
$gr^{i}_{\mathcal{F}}CH_{0}(S)_{\mathbb{Q}}$ factors through an action of the
quotient $A_{S}/\sim_{\mathrm{num}}$ of $A_{S}$, i.e. of the algebra
$B_{S}:=\bigoplus_{X,Y\in S}CH_{\dim
X}(X\times_{k}Y)_{\mathbb{Q}}/\sim_{\mathrm{num}}$. As the algebra $B_{S}$ is
semisimple, the $A_{S}$-module $gr^{i}_{\mathcal{F}}CH_{0}(S)_{\mathbb{Q}}$ is
semisimple as well.

In particular, if dimensions of the varieties in $S$ do not exceed $d$ then
the length $\ell(S)$ of any Loewy filtration of $CH_{0}(S)_{\mathbb{Q}}$ does
not exceed $d+1$. (More precisely, $\ell(S)-1$ does not exceed the number of
those $0\leq i\leq d$ for which $H_{i}(X,\mathbb{Q})$ is not a Tate twist of
an effective motive of weight $>-i$ for at least one $X\in S$.)

It seems that the radical filtration on $CH_{0}(S)_{\mathbb{Q}}$ (i.e. the
strictly descending sequence of the iterated radicals) is the motivic one, but
with the repeating terms omitted.

######  Remark 3.2.

Usually (e.g., [1] or [8, Conjecture 2.3], [10, §5.3]) one states the motivic
conjectures in the contravariant setting, i.e. instead of $\mathcal{M}_{k}$
one considers its dual category (which is in fact equivalent to
$\mathcal{M}_{k}$), while the homology functors from the category of varieties
over $k$ to $\mathcal{MM}_{k}$ are replaced by contravariant functors
$H^{i}(-,\mathbb{Q}(j))$. Then the homological object

$H_{i}(X,\mathbb{Q}):=H^{2\dim X-i}(X,\mathbb{Q}(\dim X))$

is the Poincaré dual of the cohomological object $H^{i}(X,\mathbb{Q})$, while
the Beilinson formula for codimension $q$ Chow groups of smooth projective
$k$-varieties $X$ can be rewritten as

$gr^{i}_{\mathcal{F}}CH^{q}(X)_{\mathbb{Q}}=\mathrm{Ext}^{i}_{\mathcal{MM}_{k}}(\mathbb{Q}(0),H^{2q-i}(X,\mathbb{Q}(q))).$

##  4\. Correspondences on non-proper varieties?

One could try to extend Theorem 2.2.2 to collections $S$ of smooth varieties
over $k$ that are not necessarily proper. However, as there are no non-
constant morphisms from projective varieties to affine ones, it seems that the
structure of the $A_{S}$-module $Z_{0}(S)$ may be quiet complicated.

On the other hand, if the set $S$ is considered as a preadditive category then
the $A_{S}$-modules become precosheaves with transfers (in analogy with the
terminology of V. Voevodsky). To restrict the category of precosheaves one can
pass to the category of cosheaves in such a non-trivial Grothendieck topology
where $Y\mapsto Z_{0}(Y)$ is a cosheaf.

In [14], a Grothendieck topology on the categories of schemes of finite type
over noetherian bases, called the $h$-topology, is defined, see also [13,
§10]. This topology is generated by a pretopology, where the coverings are
those finite families $(p_{i}:U_{i}\to X)_{i}$ of morphisms of finite type
that $\amalg p_{i}:\amalg U_{i}\to X$ are universal topological epimorphisms
(i.e. a subset of $X$ is open if and only if so is its preimage, and any base
change has the same property).

A precosheaf $\mathcal{F}$ of abelian groups on the category of schemes of
finite type over $k$ is an $h$-cosheaf if the sequence
$\mathcal{F}(U\times_{X}U)\xrightarrow{f_{*}\circ\mathrm{pr}_{1*}-f_{*}\circ\mathrm{pr}_{2*}}\mathcal{F}(U)\xrightarrow{f_{*}}\mathcal{F}(X)\to
0$ is exact for any $h$-covering $f:U\to X$. By an $h$-cosheaf on the category
of smooth varieties over $k$ we mean the restriction of an $h$-cosheaf on the
category of schemes of finite type over $k$.

The following lemma is related somehow to [15, Prop.3.1.3], where $f$ is a
Nisnevich cover.

######  Lemma 4.1.

If a quasi-compact morphism of schemes $Y\xrightarrow{f}X$ is surjective $($on
the sets of points$)$ then

  * •

it is surjective on the sets of closed points;

  * •

the sequence
$Z_{0}(Y\times_{X}Y)\xrightarrow{f_{*}\circ\mathrm{pr}_{1*}-f_{*}\circ\mathrm{pr}_{2*}}Z_{0}(Y)\xrightarrow{f_{*}}Z_{0}(X)\to
0$ is exact. In particular, $X\mapsto Z_{0}(X)$ is an $h$-cosheaf.

###### Proof.

Let $p$ be a closed point of $X$. Then $Y_{p}:=f^{-1}(p)$ is a non-empty
closed subset of $Y$, so it suffices to show the existence of a closed point
of $Y_{p}$. Suppose on the contrary that there are no closed points in
$Y_{p}$. As $Y_{p}$ is quasi-compact, it can be covered by a finite collection
$S$ of affine opens: $Y_{p}=\bigcup_{U\in S}U$. Let us construct recursively a
sequence of points $q_{i}\in X$ and a sequence $U_{1},U_{2},\dots$ of elements
of $S$ as follows: let $U_{1}$ be an arbitrary element of $S$, $q_{1}$ be an
arbitrary closed point of $U_{1}$; for $i>1$, if the closure
$\overline{\\{q_{i-1}\\}}$ of $q_{i-1}$ is not contained in
$U_{1}\cup\dots\cup U_{i-1}$, let (i) $q_{i}^{\prime}$ be a point of
$\overline{\\{q_{i-1}\\}}$ in the complement of $\bigcup_{j=1}^{i-1}U_{j}$,
(ii) $U_{i}$ be an element of $S$ containing $q_{i}^{\prime}$, (iii) $q_{i}$
be a closed point of $\overline{\\{q_{i}^{\prime}\\}}\cap U_{i}$:
$\overline{\\{q_{i}\\}}\cap U_{i}=\\{q_{i}\\}$.

Then $\overline{\\{q_{i}\\}}\cap(U_{1}\cup\dots\cup U_{j})$ is a subset of
$\\{q_{1},\dots,q_{j}\\}$ for any $j\leq i$. As $S$ is finite, there is some
$1\leq n\leq\\#S$ such that $\overline{\\{q_{n}\\}}$ is contained in
$U_{1}\cup\dots\cup U_{n}$. As the complement of $\bigcup_{j=1}^{n-1}U_{j}$ is
closed, the set
$\\{q_{n}\\}=\overline{\\{q_{n}\\}}\cap(X\smallsetminus(\bigcup_{j=1}^{n-1}U_{j}))\subseteq
U_{n}$ is closed as well.

The kernel of $f_{*}$ is spanned by the elements $q-q^{\prime}$ for all closed
points $q,q^{\prime}$ of $Y$ such that $f(q)=f(q^{\prime})$. But
$q-q^{\prime}$ is the image of any closed point of
$q\times_{X}q^{\prime}\subseteq Y\times_{X}Y$. ∎

######  Remark 4.2.

The proof of Lemma 4.1 can be obviously modified to show that the linear
combinations of $k$-rational points on $k$-schemes of finite type
($X\mapsto\mathbb{Q}[X(k)]$) form a Nisnevich subcosheaf without transfers
(i.e. functorial with respect to the morphisms of schemes, not with respect to
the finite correspondences) of the $h$-cosheaf with transfers $Z_{0}:X\mapsto
Z_{0}(X)$.

Lemma 4.1 suggests that, in the non-proper case, the category of $h$-cosheaves
is more appropriate than the much bigger category of $A_{S}$-modules. Then the
natural guess is that the socle $\mathrm{Soc}(Z_{0})$ of the $h$-cosheaf
$Z_{0}$ is simple and consists of those 0-cycles that become rationally
trivial on some smooth compactifications, while the radical filtration on
$Z_{0}/\mathrm{Soc}(Z_{0})$ is separable and coincides with the motivic one.

Acknowledgements. The study has been funded within the framework of the HSE
University Basic Research Program. Discussions with Vadim Vologodsky, Sergey
Gorchinskiy, Dmitry Kaledin, and especially with Ivan Panin, were very helpful
for me.

## References

  * [1] A.Beilinson, Remarks on $n$-motives and correspondences at the generic point, in Motives, polylogarithms and Hodge theory, Part I (Irvine, CA, 1998), volume 3 of Int. Press Lect. Ser. Int. Press, Somerville, MA, 2002, 35–46.
  * [2] J.Bernstein, Notes of lectures on Representations of $p$-adic groups, Harvard University, Fall 1992, written by Karl E. Rumelhart, http://www.math.tau.ac.il/~bernstei/
  * [3] P.Cartier, Representations of $\mathfrak{p}$-adic groups: A survey, 111–155. In Proc. of Symp. in Pure Math. 33, Part 1 Automorphic Forms, Representations, and $L$-functions, A.Borel, W.Casselman (Eds.) 1977.
  * [4] E.M.Friedlander, V.Voevodsky, Bivariant cycle cohomology, in V.Voevodsky, A.Suslin, E.M.Friedlander, Cycles, transfers, and motivic homology theories. Ann. Math. Studies 143 (2000), Princeton Univ. Press, 138–187.
  * [5] W.Fulton, Intersection Theory, Springer, 1984.
  * [6] R.S.Irving, The socle filtration of a Verma module, Ann. Sci. ENS, 4e série, 21 (1988), 47–65.
  * [7] U.Jannsen, Motives, numerical equivalence, and semi-simplicity, Invent. Math. 107 (1992), 447–452.
  * [8] U.Jannsen, Motivic sheaves and filtrations on Chow groups, in Motives, U.Jannsen, S.Kleiman,
J.-P.Serre (Eds.), Proc. of Symp. in Pure Math. 55 (1994), Part 1, 245–302.

  * [9] K.S.Kedlaya, More étale covers of affine spaces in positive characteristic, J. Algebraic Geom. 14 (2005), 187–192.
  * [10] M. Levine, Mixed Motives, in Handbook of K-Theory, Vol. 1, E.M. Friedlander, D.R. Grayson (Ed.), Springer-Verlag, Berlin Heidelberg, 2005, 429–521.
  * [11] Yu.I.Manin, Correspondences, motifs and monoidal transformations, Math. USSR-Sb., 6:4 (1968), 439–470.
  * [12] P.Samuel, Relations d’équivalence en géométrie algébrique, Proc. Internat. Congress Math. Edinburgh 1958, Cambridge University Press, Cambridge (1960), 470–487.
  * [13] A.Suslin, V.Voevodsky, Singular homology of abstract algebraic varieties, Invent. math. 123 (1996), 61–94.
  * [14] V.Voevodsky, Homology of schemes, Selecta Mathematica, New Series 2, No. 1 (1996), 111–153.
  * [15] V.Voevodsky, Triangulated categories of motives over a field, in V.Voevodsky, A.Suslin, E.M.Friedlander, Cycles, transfers, and motivic homology theories. Ann. Math. Studies 143 (2000), Princeton Univ. Press, 188–238.


# Breaking the Exascale Barrier for the Electronic Structure Problem in Ab-
Initio Molecular Dynamics

Robert Schade1, Tobias Kenter12, Hossam Elgabarty3, Michael Lass12, Thomas D.
Kühne134, Christian Plessl12  Paderborn University, Warburger Str. 100, 33098
Paderborn, Germany  1Paderborn Center for Parallel Computing 2Department of
Computer Science  3Department of Chemistry 4Corresponding author, Email:
<EMAIL_ADDRESS>
###### Abstract

The non-orthogonal local submatrix method applied to electronic-structure
based molecular dynamics simulations is shown to exceed 1.1 EFLOP/s in
FP16/FP32 mixed floating-point arithmetic when using 4,400 NVIDIA A100 GPUs of
the Perlmutter system. This is enabled by a modification of the original
method that pushes the sustained fraction of the peak performance to about
80%. Example calculations are performed for SARS-CoV-2 spike proteins with up
to 83 million atoms.

##  I Introduction

Electronic-structure based ab-initio molecular dynamics simulations (AIMD, [1,
2, 3]) are an important tool in solid-state physics, chemistry and material
science. The explicit treatment of quantum-mechanical effects in the
electronic structure is required in situations, where empirical model
potentials used in classical molecular dynamics fail to describe the relevant
physical or chemical phenomena.

To derive the forces acting on the atoms the electronic-structure problem has
to be solved in every time step during the propagation of the atoms. To make
this possible, linear-scaling methods have been developed, where the
computational complexity scales only linearly with the number of atoms in the
system [4, 5, 6, 7, 8]. We have proposed the non-orthogonal local submatrix
method (NOLSM, [9]) as a massively parallel method to solve the electronic-
structure problem via an approximate solution of the required matrix
functions. The local nature of the method avoids inter-node communication in
the solution phase and has been shown to scale extremely well to more than one
thousand GPUs, while efficiently using the mixed-precision tensor cores for
linear algebra operations. All of these developments together have made it
possible for the first time to break the exaflop barrier for a computational
science application. The only other instance we are aware of where more than
one exaflop of sustained performance have been achieved is for the real-time
simulation of a random quantum circuit [10].

This paper is building on the implementation described in [9], but since we
are focusing on improvements to increase the sustained peak performance,
aspects like the compensation of noise from numerical approximations with an
appropriately modified Langevin-type equation to obtain accurate
thermodynamical expectation values are not touched here, but have been
discussed in previous work [7, 11]. Instead, section II summarizes the tackled
problem, whereas Section III puts the achievement in relation to the
performance of related large-scale electronic-structure-based structure
relaxations or AIMD simulations. The innovations beyond those presented in [9]
are described in Section IV. In Section V-B, we discuss our evaluation and
define the performance measurements. Finally, Section VI discusses the
achieved performance.

##  II Overview of the Problem

Molecular dynamics calculations simulate the movement of atoms in molecules,
surfaces or solids by integrating Newtons equation of motion

$M_{i}\ddot{\bm{R}}_{i}=\bm{F}_{i},$ (1)

where $M_{i}$ is the mass of atom $i$, ${\bm{R}}_{i}$ its position in space
and $\bm{F}_{i}$ the force acting on it. Thus, the evaluation of the forces
acting on the atoms is required in every time step. In electronic-structure
based molecular dynamics these forces are evaluated on-the-fly from the total
energy $E$ of the system via

$\bm{F}_{i}=-\frac{\partial E}{\partial\bm{R}_{i}}.$ (2)

In turn, the total energy is not obtained from an empirical model as in
classical molecular dynamics, but directly from the quantum-mechanical problem
of electrons in the electrostatic field of the nuclei. The total energy of a
system can be written as

$E=E_{elec}+E_{dc}+E_{ion},$ (3)

where $E_{elec}$ is the electronic energy, $E_{dc}$ the double counting terms
and $E_{ion}$ the nuclear Coulomb repulsion energy of the atoms. The bulk of
the computational effort is required to obtain the electronic energy. While
this can be done efficiently for small and medium-sized systems by solving a
high-dimensional eigenvalue problem [12, 13], very large systems require
methods that scale at most linearly with the size of the system in terms of
number of atoms. Such linear-scaling electronic-structure methods have been
developed [4, 5, 6, 7] for example based on the one-particle reduced density
matrix $\bm{D}$ [14], which allows to obtain the electronic energy via

$E_{elec}=\text{Tr}(\bm{D}\bm{H}_{0}).$ (4)

At zero electronic temperature, the density matrix can be written as a matrix
function

$\bm{D}=\frac{1}{2}\left(\bm{I}-\text{sign}(\bm{S}^{-1}\bm{H}_{0}-\mu\bm{I})\right)\bm{S}^{-1},$
(5)

where $\bm{H}_{0}$ is the one-particle Hamiltonian matrix of the system and
$\bm{S}$ is the overlap matrix between the basis functions that are used to
describe the wave functions of the electrons. Furthermore, $\mu$ denotes the
chemical potential.

The evaluation of the matrix-sign function in Eq. 5 can be performed
iteratively for example with the Newton-Schulz iteration [15]

$\displaystyle\bm{X}_{0}$ $\displaystyle=\bm{A},\ \
\bm{X}_{k+1}=\frac{1}{2}\bm{X}_{k}(3\bm{I}-\bm{X}_{k}^{2})$ (6)
$\displaystyle\text{sign}(\bm{A})$
$\displaystyle=\lim_{k\rightarrow\infty}\bm{X}_{k},$ (7)

or other iteration schemes [16] that only require scalings, additions and
multiplications of matrices.

The submatrix method [17, 18] instead views the density matrix as a matrix
function to be evaluated. Therein, the evaluation of a matrix function
$f(\bm{A})$ is performed in three steps. The steps are performed for every
column $i$ of the matrix $\bm{A}$ independently and are schematically shown in
Fig. 1:

  1. 1.

In the first step a submatrix is generated for column $i$ of the input matrix
$\bm{A}$ by removing all rows and corresponding columns for which the matrix
$\bm{A}$ has vanishing or negligible elements in column $i$. The result is a
smaller and much denser submatrix $\mathcal{T}_{i}(\bm{A})$.

  2. 2.

The matrix function is applied to the submatrix $\mathcal{T}_{i}(\bm{A})$,
i.e., $f(\mathcal{T}_{i}(\bm{A}))$ is evaluated.

  3. 3.

The matrix elements of $f(\mathcal{T}_{i}(\bm{A}))$ that correspond to the
column $i$ are written to the matrix $\bm{B}$ in the sense that the submatrix
construction of step 1) is applied in a reverse way.

The resulting matrix $\bm{B}$ is an approximation of the matrix function
$f(\bm{A})$ and by construction has the same sparsity pattern as $\bm{A}$. The
non-orthogonal submatrix method, where $\bm{D}=\bm{D}(\bm{H}_{0},\bm{S})$ is a
matrix function of two matrices, combines the sparsity patterns $\bm{H}_{0}$
and $\bm{S}$ before the submatrix construction. Thus, it builds two
submatrices, $\mathcal{T}_{i}(\bm{H}_{0})$ and $\mathcal{T}_{i}(\bm{S})$ and
in step 2) the matrix function is

$\mathcal{T}_{i}(\bm{D})=\frac{1}{2}\left(\bm{I}-\text{sign}(\mathcal{T}_{i}(\bm{S})^{-1}\mathcal{T}_{i}(\bm{H}_{0})-\mu\bm{I})\right)\mathcal{T}_{i}(\bm{S})^{-1}.$
(8)

Please note, that the efficiency and accuracy of the submatrix method can be
improved by generating one submatrix for multiple columns instead of just a
single column as described in detail in [9] together with the GPU
implementation. Moreover, the use of the submatrix approximation and low-
precision numerics can be compensated by an modified Langevin-type equation
that replaces Newtons equation of motion so that exact ensemble-averaged
expectation values can be obtained [7, 11].

Figure 1: Schematic representation of the steps of the submatrix method for
the approximate calculation of a matrix function $f(\bm{A})$ of a large sparse
matrix $\bm{A}$. The first step is the construction of a submatrix
$\mathcal{T}_{i}(\bm{A})$ for every column of the matrix $\bm{A}$. Then the
matrix function is applied to the dense submatrices, i.e.,
$f(\mathcal{T}_{i}(A))$ and finally the relevant result columns are inserted
into the sparse result matrix. Figure from [9].

##  III Current State of the Art

Table I lists previous attempts to extend the boundaries of electronic-
structure based structure relaxation or molecular dynamics simulations.

Table I:  Performance of previously conducted electronic structure-based structure relaxation or AIMD simulations. Therein, the employed electronic structure method is abbreviated by DFT, NSC-DFT, LS-DFT and SS-DFT, which stands for density functional theory and its non-self-consistent, linear-scaling and subsystem variants, respectively. The corresponding basis set to represent the single-particle orbitals are denoted by PW for conventional plane waves, RMG-PW for real-space multigrid plane waves, GPW for Gaussian and plane waves, GTO for Gaussian-type orbitals, FD for finite difference, RS-FD for real-space finite difference, FEM for finite element method, NGWF for non-orthogonal generalized Wannier functions and PAO for polarized atomic orbitals. If the calculation was conducted involving trivial k-point parallelism, the total number of atoms is given as the product of number of independent instances time the number of atoms in anyone of them. The sustained efficiency is either given with respect to the corresponding peak performance, or estimated in terms of parallel efficiency and identified by the “$\approx$” sign. This table has been published previously in [9] and is included here with additional results for comparison. Code | Year | Method | Basis | System | # Atoms | # Cores | Machine |  Peak Performance | Efficiency
---|---|---|---|---|---|---|---|---|---
CPMD [19] | 2005 | DFT | PW | bulk SiC | 1k | 1.2k CPU | IBM p690 | 1.087 TFLOP/s |  $\approx$ 20%
Qbox [20] | 2006 | DFT | PW | bulk Mo | 8*1k | 128k CPU | IBM BlueGene/L | 207.3 TFLOP/s | 56.5%
LS3DF [21] | 2009 | DFT | PW | bulk ZnTeO | 36k | 147k CPU | Cray Jaguar | 442 TFLOP/s |  $\approx$ 33%
CONQUEST [22] | 2010 | NSC-DFT | PAO | bulk Si | 2.1M | 4k CPU | Cray XT4 |  |  $\approx$ 60%
CP2K [23] | 2012 | LS-DFT | GPW | bulk H2 | 1M | 47k CPU | Cray XT5 |  |
ONETEP [24] | 2014 | LS-DFT | NGWF |  amyloid fibril trimer | 42k | 115k CPU | IBM BlueGene/Q |  |
CONQUEST [25] | 2014 | LS-DFT | PAO | bulk Si | 786k | 200k CPU | K-Computer |  |
RSDFT [26] | 2014 | DFT | RS-FD | Si nanowire | 107k | 664k CPU | K-Computer | 5.48 PFLOP/s | 51.67%
CP2K [27] | 2016 | SS-DFT | GPW |  satellite tobacco mosaic virus | 1M | 20k CPU | Cray XC30 |  |
LDC-DFT [28] | 2014 | SS-DFT | RMG-PW | bulk SiC | 6.3M | 786k CPU | IBM BlueGene/Q | 5.08 PFLOP/s | 50.5%
OpenAtom [29] | 2016 | DFT | PW | periodic MOF | 32*424 | 262k CPU | IBM BlueGene/Q |  |  $\approx$ 52%
MGmol [30] | 2016 | LS-DFT | FD | bulk H2O | 1.2M | 1.6m CPU | IBM BlueGene/Q |  |  $\approx$ 39%
DFT-FE [31] | 2019 | DFT | FEM | Mg cluster | 10.5k |  159k CPU +22.8k GPUs | IBM Summit | 46 PFLOP/s | 27.8%
CP2K [9] | 2021 | LS-DFT | GTO | bulk water | 102M |  18.4k CPU +1.5k GPUs | JUWELS Booster | 206 PFLOP/s | 43%
CP2K [9] | 2021 | LS-DFT | GTO |  HIV-1 capsid in solution | 62.5M |  18.4k CPU +1.5k GPUs | JUWELS Booster | 324 PFLOP/s | 67.7%
CP2K, this work | 2022 | LS-DFT | GTO |  spike proteins in solution | 82.9M |  70.4k CPU +4.4k GPUs | NERSC Perlmutter | 1127 PFLOP/s | 82.1%

##  IV Innovations Realized

###  IV-A Summary of Contributions

This work uses the previously reported algorithmic innovations like the use of
approximate computing techniques, the non-orthogonal local submatrix method
and its realization with GPUs, while minimizing the communication, as well as
the heuristic combination of columns in the submatrix creation described in
[9]. A new development beyond the implementation innovations already shown in
[9] like the efficient iterative evaluation of matrix functions for dense
matrices on GPU tensor cores is introduced in section IV-B: The matrix-size
dependency of the GPU-performance is now also considered for the combination
of submatrices and yields an additional speedup.

###  IV-B Implementation Innovations

####  IV-B1 Submatrix Combination Heuristics

The combination of columns for the generation of submatrices introduced in [9]
used a cubic metric, i.e., the combination of two columns yields an
improvement if and only if

${(n_{i}+n_{j}-n_{i\land j})}^{3}<n_{i}^{3}+n_{j}^{3},$ (9)

where $n_{i}$ is the dimension of the submatrix for column $i$, $n_{j}$ for
column $j$ and $n_{i\land j}$ the number of overlapping columns. This cubic
metric represents the number of floating-point operations during the
evaluation of the matrix function for the submatrices, which is based on dense
matrix multiplications. We propose to modify the cubic metric by including the
performance characteristic of the used GPUs. For this purpose, the matrix
multiplication performance $p(n)$ of the GPUs is measured for different matrix
sizes $n$ and interpolated. The combination criterion then compares the
predicted runtime $p(n)\times n^{3}$ for the matrix functions of the submatrix
instead of the number of floating-point operations, i.e.,

$p(n_{i}+n_{j}-n_{i\land j})\times{(n_{i}+n_{j}-n_{i\land
j})}^{3}<p(n_{i})\times n_{i}^{3}+p(n_{j})\times n_{j}^{3}.$ (10)

This criterion effectively increases the dimension of the submatrices and the
achievable portion of the peak performance. Results are shown for the SARS-
CoV-2 Spike protein in aqueous solution with approx. 1.7 mio. atoms in Table
II and histograms for the submatrix dimensions in Fig. 2. The criterion of Eq.
10 approximately doubles the average submatrix dimension and slightly
increases the total number of floating point operations while drastically
increasing the estimated floating-point throughput and leading to an overall
estimated speedup compared to the previous criterion of Eq. 9.

Table II: Influence of the two different submatrix combination criteria, Eq. 9 and Eq. 10, on the submatrix sizes, number of floating-point operations for one matrix mutliplication of each submatrix, the estimated performance per NVIDIA A100 GPU and the estimated speedup considering the matrix-multiplication performance of an NVIDIA A100. Combination of submatrices | no | Eq. 9 | Eq. 10
---|---|---|---
Number of submatrices | 1693134 | 89784 | 28577
Smallest submatrix | 396 | 694 | 708
Largest submatrix | 10627 | 10637 | 10737
Average submatrix dimension | 715 | 1361 | 2282
FLOP count in PFLOP for one mult. | 2.02 | 1.54 | 1.66
Estimated performance in TFLOP/s | 103 | 224 | 270
Estimated speedup |  | 2.9 | 3.2
Figure 2: Histogram for the submatrix sizes in the case of the SARS-CoV-2
Spike protein in aqueous solution with approx. 1.7 mio. atoms: without
combining submatrices (blue), combination using criterion Eq. 9 (orange) and
using criterion Eq. 10 (green). A discussion of the further structure can be
found in [9] and also applies to the spike protein system.

##  V How Performance Was Measured

###  V-A Computational Details

####  V-A1 SARS-CoV-2 Spike Protein in Aqueous Solution

As our benchmark system, we have used the full-length SARS-CoV-2 spike protein
in the open state, anchored in a lipid bilayer (Reference PDB structure: 6VSB,
and pre-equilibrated with all-atom MD using NAMD [32, 33]. The system was
solvated in aqueous solution in a simulation cell with dimensions
204.7$\times$199.5$\times$408.5 Å, and including 1693134 atoms. The single
cell shown in Fig. 3 can be easily repeated in a two-dimensional grid of spike
proteins as a scalable benchmark system.

Figure 3: SARS-CoV-2 spike protein in aqueous solution: full cell (left) and
without hydrogen and oxygen atoms (right).

####  V-A2 Simulation Details

The electronic structure is simulated with the GFN-xTB approach in conjunction
with a London dispersion correction based on the rational Becke–Johnson
damping function [34]. Further details can be found in [9].

For the sake of benchmark resources, we have restricted each simulation run to
one SCF iteration in the spirit of the second-generation Car-Parrinello AIMD
method [12, 35], but included the iterations for finding an appropriate
chemical potential that produces a charge-neutral system.

###  V-B Measurements

The main measurements presented here are:

####  V-B1 Wall clock time of the NOLSM method $T_{\mathrm{NOLSM}}$

The wall clock time $T_{\mathrm{NOLSM},i}$ of the NOLSM method on node $i$ is
measured for each iteration of the chemical potential. Each iteration includes
all transfers between host and GPU. The overall wall clock time wall clock
time $T_{\mathrm{NOLSM}}$ is defined as the maximum over all node wall-clock
times.

####  V-B2 FLOPs in the NOLSM method $\mathrm{FLOPs}_{\mathrm{NOLSM},i}$

The per-node floating-point operations $\mathrm{FLOPs}_{\mathrm{NOLSM}}$ in
the FP16/FP32-mixed-precision matrix iterations in the NOLSM method are
estimated as $2n^{3}$ for a gemm-operation
$\bm{C}=\alpha\bm{A}\cdot\bm{B}+\beta\bm{C}$ with
$\bm{A},\bm{B},\bm{C}\in\mathbb{R}^{n\times n}$ for each iteration of the
chemical potential. The construction of the matrix elements of the submatrices
and other operations scaling like $\mathcal{O}(n^{2})$ are neglected in the
count.

####  V-B3 Node-Performance of NOLSM method $P_{\mathrm{NOLSM},i}$

The node performances of the NOLSM method are defined as
$P_{\mathrm{NOLSM},i}=\mathrm{FLOPs}_{\mathrm{NOLSM},i}/T_{\mathrm{NOLSM},i}$
for each node $i$.

####  V-B4 Performance of NOLSM method $P_{\mathrm{NOLSM}}$

The performance of the NOLSM method is defined as the sum of the node
performances.

###  V-C HPC System and Environment

The benchmark runs presented here have been performed on the Perlmutter
systsem at the National Energy Research Scientific Computing Center (NERSC).
The Perlmutter system consists of 1,536 GPU nodes with one AMD EPYC 7763
64-core CPU with 256 GB DDR4 memory and four NVIDIA A100 GPUs with 40 GB of
HBM2 memory each. The peak performance of the tensor cores in one NVIDIA A100
GPU is 312 TFLOP/s in FP16 with FP32-based accumulate [36]. The system uses
HPE Cray Slingshot as node interconnect.

The software environment used in this work consisted of GCC 11.2.0, Cray-MPICH
8.1.10, CUDA NVCC 11.5.119, and CUBLAS 11.5. One MPI-rank per node and 64 CPU-
threads per rank were used as well as four CUDA streams per GPU. Each stream
was controlled by a single CPU-thread.

##  VI Performance Measurements and Results

###  VI-A Performance of the NOLSM Method for the Spike Protein

We have performed calculations for three different grid sizes of spike
proteins: 6$\times$5 (51 mio. atoms), 6$\times$6 (61 mio. atoms) and
7$\times$7 (83 mio. atoms). All three example calculations have been performed
with 1,100 nodes of the Perlmutter system, i.e., 4,400 NVIDIA A100 GPUs.

The wall clock time of the NOLSM method $T_{\mathrm{NOLSM}}$ is shown in
Figure 4.

Figure 4: Wall time of the NOLSM method $T_{\mathrm{NOLSM}}$ for a grid of
SARS-CoV-2 spike proteins in aqueous solution on 1,100 nodes of the Perlmutter
system.

The distribution of the performances of individual nodes is shown in Figure 5
for 7$\times$7 spike proteins (83 mio. atoms) in relation to the peak
performance of the GPUs. The performances of the nodes with 4 NVIDIA A100 GPUs
mainly fall in the range between 1 PFLOP/s and 1.07 PFLOP/s with an average of
1.03 PFLOP/s. This represents about 80% of the peak performance of $1.248$
PFLOP/s$=4\cdot 0.312$ PFLOP/s per node.

Figure 5: Distribution of node performances for 83 mio. atoms (7$\times$7 grid
of SARS-CoV-2 spike proteins in aqueous solution) on 1,100 nodes of the
Perlmutter system.

Finally, Figure 6 shows the floating-point performance $P_{\mathrm{NOLSM}}$ in
mixed FP16/FP32 of the NOLSM method. The floating-point throughput of 1.106 to
1.127 EFLOP/s with 4,400 NVIDIA A100 GPUs achieving about 80% of the
theoretical peak performance of the tensor cores.

Figure 6: Distribution of node performances for 83 mio. atoms (7$\times$7 grid
of SARS-CoV-2 spike proteins in aqueous solution) on 1,100 nodes (4,400 NVIDIA
A100 GPUs) of the Perlmutter system.

##  VII Conclusion

To the best of our knowledge, the achieved $\sim$1.1 EFLOP/s in FP16/FP32
floating-point arithmetic positions electronic-structure based molecular
dynamics calculations with the non-orthogonal local submatrix method in CP2K
[37] as the first algorithm in computational natural science that has broken
the exaflop barrier within a scientific application. The massively parallel
nature of the method allows for an efficient use of many thousand GPUs. The
method can not only be applied to electronic-structure based molecular
dynamics, but also in other situations where a matrix function needs to be
evaluated for a large sparse matrix or problems that can be transformed to
such an operation.

## Acknowledgments

This research used resources of the National Energy Research Scientific
Computing Center (NERSC), a U.S. Department of Energy Office of Science User
Facility located at Lawrence Berkeley National Laboratory, operated under
Contract No. DE-AC02-05CH11231 using NERSC award DDR-ERCAP0022240.

Additionally, we would like to thank for funding of this project by computing
time provided by the Paderborn Center for Parallel Computing (PC2). This work
is partially funded by Paderborn University’s research award for “GreenIT”, as
well as the Federal Ministry of Education and Research (BMBF) and the state of
North Rhine-Westphalia as part of the NHR Program. T.D.K. received funding
from the European Research Council (ERC) under the European Union’s Horizon
2020 research and innovation program (Grant Agreement No. 716142).

## References

  * [1] R. Car and M. Parrinello  “Unified Approach for Molecular Dynamics and Density-Functional Theory”  In _Phys. Rev. Lett._ 55.22, 1985, pp. 2471–2474
  * [2] M.. Payne et al.  “Iterative minimization techniques for ab initio total-energy calculations: molecular dynamics and conjugate gradients”  In _Rev. Mod. Phys._ 64.4, 1992, pp. 1045–1097
  * [3] Thomas D. Kühne  “Second generation Car–Parrinello molecular dynamics”  In _WIREs Comput. Mol. Sci._ 4, 2014, pp. 391–406
  * [4] Stefan Goedecker  “Linear scaling electronic structure methods”  In _Rev. Mod. Phys._ 71.4, 1999, pp. 1085–1123
  * [5] Weitao Yang  “Direct calculation of electron density in density-functional theory”  In _Phys. Rev. Lett._ 66.11, 1991, pp. 1438–1441
  * [6] Giulia Galli and Michele Parrinello  “Large scale electronic structure calculations”  In _Phys. Rev. Lett._ 69.24, 1992, pp. 3547–3550
  * [7] Dorothee Richters and Thomas D. Kühne  “Self-consistent field theory based molecular dynamics with linear system-size scaling”  In _J. Chem. Phys._ 140.13, 2014, pp. 134109
  * [8] Anders M.. Niklasson et al.  “Graph-based linear scaling electronic structure theory”  In _The Journal of Chemical Physics_ 144.23, 2016, pp. 234101  DOI: 10.1063/1.4952650
  * [9] Robert Schade et al.  “Towards electronic structure-based ab-initio molecular dynamics simulations with hundreds of millions of atoms”  In _Parallel Computing_ 111, 2022, pp. 102920  DOI: https://doi.org/10.1016/j.parco.2022.102920
  * [10] Yong Alexander Liu et al.  “Closing the ”Quantum Supremacy” Gap: Achieving Real-Time Simulation of a Random Quantum Circuit Using a New Sunway Supercomputer”  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’21  St. Louis, Missouri: Association for Computing Machinery, 2021  DOI: 10.1145/3458817.3487399
  * [11] Varadarajan Rengaraj, Michael Lass, Christian Plessl and Thomas D. Kühne  “Accurate Sampling with Noisy Forces from Approximate Computing”  In _Computation_ 8.39, 2020, pp. 1–11
  * [12] Thomas D. Kühne, Matthias Krack, Fawzi R. Mohamed and Michele Parrinello  “Efficient and Accurate Car-Parrinello-like Approach to Born-Oppenheimer Molecular Dynamics”  In _Phys. Rev. Lett._ 98.6, 2007, pp. 066401
  * [13] Thomas D Kühne, Julian Heske and Emil Prodan  “Disordered crystals from first principles II: Transport coefficients”  In _Annals of Physics_ 421 Elsevier, 2020, pp. 168290
  * [14] Rev McWeeny  “Some recent advances in density matrix theory”  In _Reviews of Modern Physics_ 32.2 APS, 1960, pp. 335
  * [15] Günther Schulz  “Iterative Berechung der reziproken Matrix”  In _ZAMM - Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Mathematik und Mechanik_ 13.1, 1933, pp. 57–59
  * [16] Dorothee Richters et al.  “A General Algorithm to Calculate the Inverse Principal p-th Root of Symmetric Positive Definite Matrices”  In _Communications in Computational Physics_ 25.2 Global Science Press, 2019, pp. 564–585
  * [17] Michael Lass et al.  “A Massively Parallel Algorithm for the Approximate Calculation of Inverse p-th Roots of Large Sparse Matrices”  In _Proc. Platform for Advanced Scientific Computing (PASC) Conference_ ACM, 2018
  * [18] Michael Lass, Robert Schade, Thomas Kühne and Christian Plessl  “A Submatrix-Based Method for Approximate Matrix Function Evaluation in the Quantum Chemistry Code CP2K”  In _Proc. International Conference for High Performance Computing, Networking, Storage and Analysis (SC)_ IEEE Computer Society, 2020, pp. 1127–1140
  * [19] Jürg Hutter and Alessandro Curioni  “Dual-level parallelism for ab initio molecular dynamics: Reaching teraflop performance with the CPMD code”  In _Parallel Computing_ 31.1 Elsevier, 2005, pp. 1–17
  * [20] Francois Gygi  “Large-scale electronic structure calculations of high-Z metals on the BlueGene/L platform”  In _Proceedings of the 2006 ACM/IEEE conference on Supercomputing_ , 2006, pp. 45
  * [21] Zhengji Zhao  “The linearly scaling 3D fragment method for large scale electronic structure calculations”  In _Journal of Physics: Conference Series_ 180.1, 2009, pp. 012079  IOP Publishing
  * [22] David R Bowler and T Miyazaki  “Calculations for millions of atoms with density functional theory: linear scaling shows its potential”  In _Journal of Physics: Condensed Matter_ 22.7 IOP Publishing, 2010, pp. 074207
  * [23] Joost VandeVondele, Urban Borstnik and Jurg Hutter  “Linear scaling self-consistent field calculations with millions of atoms in the condensed phase”  In _Journal of chemical theory and computation_ 8.10 ACS Publications, 2012, pp. 3565–3573
  * [24] Karl A Wilkinson, Nicholas DM Hine and Chris-Kriton Skylaris  “Hybrid MPI-OpenMP parallelism in the ONETEP linear-scaling electronic structure code: Application to the delamination of cellulose nanofibrils”  In _Journal of chemical theory and computation_ 10.11 ACS Publications, 2014, pp. 4782–4794
  * [25] Michiaki Arita, Sergiu Arapan, David R Bowler and Tsuyoshi Miyazaki  “Large-scale DFT simulations with a linear-scaling DFT code CONQUEST on K-computer”  In _Journal of Advanced Simulation in Science and Engineering_ 1.1 Japan Society for Simulation Technology, 2014, pp. 87–97
  * [26] Yukihiro Hasegawa  “Performance evaluation of ultra-large-scale first-principles electronic structure calculation code on the K computer”  In _The International journal of high performance computing applications_ 28.3 Sage Publications Sage UK: London, England, 2014, pp. 335–355
  * [27] Samuel Andermatt, Jinwoong Cha, Florian Schiffmann and Joost VandeVondele  “Combining linear-scaling DFT with subsystem DFT in Born–Oppenheimer and Ehrenfest molecular dynamics Simulations: from molecules to a virus in solution”  In _Journal of chemical theory and computation_ 12.7 ACS Publications, 2016, pp. 3214–3227
  * [28] Ken-ichi Nomura  “Metascalable quantum molecular dynamics simulations of hydrogen-on-demand”  In _SC’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2014, pp. 661–673  IEEE
  * [29] Nikhil Jain  “Openatom: Scalable ab-initio molecular dynamics with diverse capabilities”  In _International Conference on High Performance Computing_ , 2016, pp. 139–158  Springer
  * [30] Jean-Luc Fattebert et al.  “Modeling dilute solutions using first-principles molecular dynamics: computing more than a million atoms with over a million cores”  In _SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2016, pp. 12–22  IEEE
  * [31] Sambit Das et al.  “Fast, scalable and accurate finite-element based ab initio calculations using mixed precision computing: 46 PFLOPS simulation of a metallic dislocation system”  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2019, pp. 1–11
  * [32] Daniel Wrapp et al.  “Cryo-EM structure of the 2019-nCoV spike in the prefusion conformation”  In _Science_ 367.6483, 2020, pp. 1260–1263  DOI: 10.1126/science.abb2507
  * [33] Lorenzo Casalino et al.  “AI-driven multiscale simulations illuminate mechanisms of SARS-CoV-2 spike dynamics”  In _The International Journal of High Performance Computing Applications_ 35.5, 2021, pp. 432–451  DOI: 10.1177/10943420211006452
  * [34] Stefan Grimme, Stephan Ehrlich and Lars Goerigk  “Effect of the damping function in dispersion corrected density functional theory”  In _Journal of computational chemistry_ 32.7 Wiley Online Library, 2011, pp. 1456–1465
  * [35] Thomas D Kühne and Emil Prodan  “Disordered crystals from first principles I: Quantifying the configuration space”  In _Annals of Physics_ 391 Elsevier, 2018, pp. 120–149
  * [36] URL: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper.pdf
  * [37] Thomas D. Kühne  “CP2K: An electronic structure and molecular dynamics software package - Quickstep: Efficient and accurate electronic structure calculations”  In _J. Chem. Phys._ 152.19, 2020, pp. 194103