# The classification of $n$-dimensional nilpotent non-Tortkara anticommutative
algebras with $(n-4)$-dimensional annihilator

Shun Xu
School of Mathematical Sciences, Tongji University, Shanghai, 200092, China
e-mail<EMAIL_ADDRESS>
###### Abstract

In this paper, we give a complete classification of $n$-dimensional nilpotent
non-Tortkara anticommutative algebras with $(n-4)$-dimensional annihilator
over $\mathbb{C}$.

Keywords: Anticommutative algebra, Tortkara algebra, Automorphism group.

##  1 Introduction

One of the classical problems in the theory of algebras is to classify (up to
isomorphism) the nilpotent algebras of dimension $n$ from a certain variety
defined by some family of polynomial equalities. There are many results
related to algebraic classification of small dimensional nilpotent algebras in
varieties of Jordan, Lie, Leibniz, Zinbiel and other algebras. Algebraic
classification of nilpotent Lie algebras of dimension 7 (over algebraically
closed field and $\mathbb{R}$) [4]; algebraic classification of five-
dimensional nilpotent Jordan algebras [6]; algebraic classification of four-
dimensional nilpotent Leibniz algebras [3]; algebraic classification of
complex 5-dimensional Zinbiel algebras [1].

There are also some results related to algebraic classification of
$n$-dimensional nilpotent algebras with $(n-s)$-dimensional annihilator, where
$s$ is small positive integer. The classification of $n$-dimensional
anticommutative algebras with $(n-3)$-dimensional annihilator [2]; The
classification of $n$-dimensional non-associative Jordan algebras with
$(n-3)$-dimensional annihilator [7]; The classification of $n$-dimensional
non-Lie Malcev algebras with $(n-4)$-dimensional annihilator [8].

In this paper, we give a complete classification of $n$-dimensional nilpotent
non-Tortkara anticommutative algebras with $(n-4)$-dimensional annihilator
over $\mathbb{C}$. An algebra $(A,\cdot)$ over $\mathbb{C}$ is anticommutative
if it satisfies: $xy+yx=0$ for all $x,y\in A$. Let $A$ be an anticommutative
algebra. The ideal $\mathrm{Ann}(A)=\\{x\in A:xA=0\\}$ is called the
annihilator of $A$. An anticommutative algebra is Tortkara algebra, if it
satisfies:

$(ab)(cb)=J(a,b,c)b,\text{where}\,J(a,b,c)=(ab)c+(bc)a+(ca)b.$

Our main result in this paper is the following theorem.

######  Theorem 1 (Main result).

Let $A$ be a $n$-dimensional nilpotent non-Tortkara anticommutative algebra
with $\dim\mathrm{Ann}(A)=n-4$ over $\mathbb{C}$. Then $A$ is isomorphic to
one of the followings:

  * •

$n=4$

    * –

$\mathrm{empty\,\,set};$

  * •

$n=5$

    * –

$A_{5,1}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5};$

  * •

$n=6$

    * –

$A_{6,1}=A_{5,1}\oplus\mathbb{C}e_{6};$

    * –

$A_{6,2}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6};$

    * –

$A_{6,3}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{3}=e_{6};$

    * –

$A_{6,4}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{1}e_{4}=e_{6};$

    * –

$A_{6,5}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{3}=e_{6},e_{1}e_{4}=e_{6};$

  * •

$n=7$

    * –

$A_{7,i}=A_{6,i}\oplus\mathbb{C}e_{7},i=1,\cdots,5;$

    * –

$A_{7,6}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{2}e_{3}=e_{7};$

    * –

$A_{7,7}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{1}e_{4}=e_{7};$

    * –

$A_{7,8}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{1}e_{4}=e_{6},e_{2}e_{3}=e_{7};$

    * –

$A_{7,9}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{1}e_{4}=e_{7},e_{2}e_{3}=e_{7};$

  * •

$n=8$

    * –

$A_{8,i}=A_{7,i}\oplus\mathbb{C}e_{8},i=1,\cdots,9;$

    * –

$A_{8,10}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{2}e_{3}=e_{7},e_{1}e_{4}=e_{8};$

  * •

$n\geqslant 9$

    * –

$A_{n,i}=A_{8,i}\oplus\mathbb{C}e_{9}\oplus\cdots\oplus\mathbb{C}e_{n},i=1,\ldots,10.$

The paper is organized as follows. In Section 2, we describe a
method(Skjelbred-Sund method) for classifying all anticommutative algebras of
dimension $n$ with $s$-dimensional annihilator given those algebras of
dimension $n-s$, which also appeared in [8]. In Section 3, the proof of
Theorem 1 is given.

Throughout the paper we use the following notation. All the vector spaces and
algebras will be assumed to be finite dimensional over $\mathbb{C}$. The
multiplication of an algebra is specified by giving only the nonzero products
among the basis elements.

##  2 The analogue of Skjelbred-Sund method for anticommutative algebras

Hegazi given the analogue of the Skjelbred–Sund method for Malcev algebras in
[8]. Inspired by [8], we give the analogue of the Skjelbred–Sund method for
anticommutative algebras in this section. Proofs for all the conclusions
mentioned in this section can be found in [8].

Let ${A}$ be an anticommutative algebra, ${V}$ be a vector space and
${Z}^{2}({A},{V})$ is defined to be the set of all skew-symmetric bilinear
maps $\theta:{A}\times{A}\longrightarrow{V}$. For $f\in\mathrm{Hom}({A},{V})$,
we define $\delta f\in{Z}^{2}({A},{V})$ by the equality $\delta f(x,y)=f(xy)$
and set ${B}^{2}({A},{V})=\\{\delta f\mid f\in\operatorname{Hom}({A},{V})\\}$.
One can easily check that ${B}^{2}({A},{V})$ is a linear subspace of
${Z}^{2}({A},{V})$. Let us define ${H}^{2}({A},{V})$ as the quotient space
${Z}^{2}({A},{V})/{B}^{2}({A},{V})$. The equivalence class of
$\theta\in{Z}^{2}({A},{V})$ in ${H}^{2}({A},{V})$ is denoted by $[\theta]$. As
usual, we call the elements of ${Z}^{2}({A},{V})$ cocycles, those of
${B}^{2}({A},{V})$ coboundaries, and ${H}^{2}({A},{V})$ is the corresponding
second cohomology space.

Suppose now that $\operatorname{dim}{A}=m<n$ and $\operatorname{dim}{V}=n-m$.
Let $\theta\in Z^{2}(A,V)$, we can define on the space
${A}_{\theta}:={A}\oplus{V}$ the anticommutative bilinear product by the
equality $(x+x^{\prime})(y+y^{\prime})=xy+\theta(x,y)$ for
$x,y\in{A},x^{\prime},y^{\prime}\in{V}$. $A_{\theta}$ is an anticommutative
algebra and ${A}_{\theta}$ is nilpotent if and only if ${A}$ is nilpotent. The
set $\theta^{\perp}=\\{x\in A:\theta(x,A)=0\\}$ is called the radical of
$\theta$. Then
$\mathrm{Ann}\left(A_{\theta}\right)=\left(\theta^{\perp}\cap\mathrm{Ann}(A)\right)\oplus
V$ by [8, Lemma 4]. The algebra ${A}_{\theta}$ is called a $(n-m)$-dimensional
central extension of ${A}$ by ${V}$.

######  Lemma 1.

[8, Lemma 5] Let $A$ be an anticommutative algebra with $\mathrm{Ann}(A)\neq
0$. Then there exists, up to isomorphism, a unique anticommutative algebra
$A^{\prime}$, and $\theta\in Z^{2}\left(A^{\prime},\mathrm{Ann}(A)\right)$
with $\theta^{\perp}\cap\mathrm{Ann}\left(A^{\prime}\right)=0$ such that
$A\cong A_{\theta}^{\prime}$ and $A/\mathrm{Ann}(A)\cong A^{\prime}.$

Let $e_{1},\ldots,e_{s}$ be a basis of $V$, and $\theta\in Z^{2}(A,V)$. Then
$\theta$ can be uniquely written as
$\theta(x,y)=\sum_{i=1}^{s}\theta_{i}(x,y)e_{i}$, where $\theta_{i}\in
Z^{2}(A,\mathbb{C}).$ Moreover,
$\theta^{\perp}=\theta_{1}^{\perp}\cap\theta_{2}^{\perp}\cap\cdots\cap\theta_{s}^{\perp}.$
Further, $\theta\in B^{2}(A,V)$ if and only if all $\theta_{i}\in
B^{2}(A,\mathbb{C}).$

Let $A$ be an anticommutative algebra with a basis $e_{1},e_{2},\ldots,e_{n}$.
Then by $\Delta_{ij}$ we denote the skew-symmetric bilinear form
$\Delta_{ij}:A\times A\longrightarrow\mathbb{C}$ with
$\Delta_{ij}\left(e_{i},e_{j}\right)=-\Delta_{ij}\left(e_{j},e_{i}\right)=1$
and $\Delta_{ij}\left(e_{l},e_{m}\right)=0$ if $\\{i,j\\}\neq\\{l,m\\}.$ Then
the set $\left\\{\Delta_{ij}:1\leq i<j\leq n\right\\}$ is a basis for the
linear space of skew-symmetric bilinear forms on $A$. Then every $\theta\in
Z^{2}(A,\mathbb{C})$ can be uniquely written as $\theta=\sum_{1\leq i<j\leq
n}c_{ij}\Delta_{i,j}$, where $c_{ij}\in\mathbb{C}.$

Let $\left\\{e_{1},e_{2},\ldots,e_{m}\right\\}$ be a basis of $A^{2}$. Then
the set $\left\\{\delta e_{1}^{*},\delta e_{2}^{*},\ldots,\delta
e_{m}^{*}\right\\}$ by [8, Lemma 6], where
$e_{i}^{*}\left(e_{j}\right)=\delta_{ij}$ and $\delta_{ij}$ is the Kronecker
delta, is a basis of $B^{2}(A,\mathbb{C})$. Let $\theta,\vartheta\in
Z^{2}(A,V)$ such that $[\theta]=[\vartheta].$ Then
$\theta^{\perp}\cap\mathrm{Ann}(A)=\vartheta^{\perp}\cap\mathrm{Ann}(A)$ or,
equivalently,
$\mathrm{Ann}\left(A_{\theta}\right)=\operatorname{\mathrm{\mathrm{Ann}}}\left(A_{\vartheta}\right)$
by [8, Lemma 7]. Furthermore, $A_{\theta}\cong A_{\vartheta}$.

Let $\mathrm{Aut}(A)$ be the automorphism group of an anticommutative algebra
$A.$ Let $\phi\in\mathrm{Aut}(A)$. For $\theta\in Z^{2}(A,V)$ define
$\phi\theta(x,y)=\theta(\phi(x),\phi(y)).$ Then $\phi\theta\in Z^{2}(A,V).$
So, $\mathrm{Aut}(A)$ acts on $Z^{2}(A,V)$. $\phi\theta\in B^{2}(A,V)$ if and
only if $\theta\in B^{2}(A,V)$ by [8, Lemma 8]. So, $\mathrm{Aut}(A)$ acts on
$H^{2}(A,V)$.

Let $\phi=\left(a_{ij}\right)\in Aut(A)$ and $\theta\in Z^{2}(A,\mathbb{C}).$
Let $C=\left(c_{ij}\right)$ be the matrix representing $\theta$ and
$C^{\prime}=\left(c_{ij}^{\prime}\right)$ be the matrix representing
$\phi\theta$. Then $C^{\prime}=\phi^{t}C\phi$.

######  Definition 1.

Let $A$ be an anticommutative algebra. If $A=I\oplus\mathbb{C}x$ is a direct
sum of two ideals, then $\mathbb{C}x$ is called an annihilator component of
$A.$

Let $\theta(x,y)=\sum_{i=1}^{s}\theta_{i}(x,y)e_{i}\in Z^{2}(A,V)$ and
$\theta^{\perp}\cap\mathrm{Ann}(A)=0.$ Then $A_{\theta}$ has an annihilator
component if and only if
$\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]$
are linearly dependent in $H^{2}(A,\mathbb{C})$ by [8, Lemma 13]. Let
$\vartheta(x,y)=\sum_{i=1}^{s}\vartheta_{i}(x,y)e_{i}$ be another element of
$Z^{2}(A,V).$ Suppose that $A_{\theta}$ has no annihilator components and
$\theta^{\perp}\cap\mathrm{Ann}(A)=\vartheta^{\perp}\cap\mathrm{Ann}(A)=0$.
Then $A_{\theta}\cong A_{\vartheta}$ if and only if there exists a map
$\phi\in\mathrm{Aut}(A)$ such that the set
$\left\\{\left[\phi\vartheta_{i}\right]:i=1,\ldots,s\right\\}$ spans the same
subspace of $H^{2}(A,\mathbb{C})$ as the set
$\left\\{\left[\theta_{i}\right]:i=1,\ldots,s\right\\}$ by [8, Lemma 14].

Let $V$ be a finite-dimensional vector space over a $\mathbb{C}$. The
Grassmannian $G_{k}(V)$ is the set of all $k$-dimensional linear subspaces of
$V$. Let $G_{s}\left(H^{2}(A,\mathbb{C})\right)$ be the Grassmannian of
subspaces of dimension $s$ in $H^{2}(A,\mathbb{C})$. There is a natural action
of $\mathrm{Aut}(A)$ on $G_{s}\left(H^{2}(A,\mathbb{C})\right)$. Let
$\phi\in\mathrm{Aut}(A)$. For
$W=\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle\in
G_{s}\left(H^{2}(A,\mathbb{C})\right)$ define $\phi
W=\left\langle\left[\phi\theta_{1}\right],\left[\phi\theta_{2}\right],\ldots,\left[\phi\theta_{s}\right]\right\rangle$.
Then $\phi W\in G_{s}\left(H^{2}(A,\mathbb{C})\right)$. We denote the orbit of
$W\in G_{s}\left(H^{2}(A,\mathbb{C})\right)$ under the action of
$\operatorname{\mathrm{Aut}}(A)$ by $\mathrm{Orb}(W)$.

Let
$W_{1}=\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle,W_{2}=\left\langle\left[\vartheta_{1}\right],\left[\vartheta_{2}\right],\ldots,\left[\vartheta_{s}\right]\right\rangle\in
G_{s}\left(H^{2}(A,\mathbb{C})\right)$. If $W_{1}=W_{2}$, then
$\bigcap_{i=1}^{s}\theta_{i}^{\perp}\cap\mathrm{Ann}(A)=\bigcap_{i=1}^{s}\vartheta_{i}^{\perp}\cap\mathrm{Ann}(A)$
by [8, Lemma 15]. This result allows us to define

$T_{s}(A)=\left\\{W=\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle\in
G_{s}\left(H^{2}(A,\mathbb{C})\right):\underset{i=1}{\bigcap}\theta_{i}^{\perp}\cap
Ann(A)=0\right\\}$

The set $T_{s}(A)$ is stable under the action of $\mathrm{Aut}(A)$ by [8,
Lemma 16].

Let $V$ be an $s$-dimensional vector space spanned by
$e_{1},e_{2},\ldots,e_{s}$. Given a anticommutative algebra $A$, let $E(A,V)$
denote the set of all anticommutative algebras without annihilator components
which are $s$-dimensional annihilator extensions of $A$ by $V$ and have
$s$-dimensional annihilator. Then $E(A,V)=$
$\left\\{A_{\theta}:\theta(x,y)=\sum_{i=1}^{s}\theta_{i}(x,y)e_{i}\right.$ and
$\left.\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle\in
T_{s}(A)\right\\}.$ Given $A_{\theta}\in E(A,V)$, let
$\left[A_{\theta}\right]$ denote the isomorphism class of $A_{\theta}.$ Let
$A_{\theta},A_{\vartheta}\in E(A,V)$. Suppose that
$\theta(x,y)=\sum_{i=1}^{s}\theta_{i}(x,y)e_{i}$ and
$\vartheta(x,y)=\sum_{i=1}^{s}\vartheta_{i}(x,y)e_{i}.$ Then
$\left[A_{\theta}\right]=\left[A_{\vartheta}\right]$ if and only if

$\mathrm{Orb}\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle=\mathrm{Orb}\left\langle\left[\vartheta_{1}\right],\left[\vartheta_{2}\right],\ldots,\left[\vartheta_{s}\right]\right\rangle$

by [8, Lemma 17].

######  Theorem 2.

[8, Theorem 18] There exists a one-to-one correspondence between the set of
$\mathrm{Aut}(A)$-orbits on $T_{s}(A)$ and the set of isomorphism classes of
$E(A,V)$. This correspondence is defined by

$\operatorname{\mathrm{Orb}}\left\langle\left[\theta_{1}\right],\left[\theta_{2}\right],\ldots,\left[\theta_{s}\right]\right\rangle\in\left\\{\operatorname{\mathrm{Orb}}(W):W\in
T_{s}(A)\right\\}\leftrightarrow\left[A_{\theta}\right]\in\left\\{\left[A_{\vartheta}\right]:A_{\vartheta}\in
E(A,V)\right\\}$

where $\theta(x,y)=\sum_{i=1}^{s}\theta_{i}(x,y)e_{i}$.

By this theorem, we may construct all anticommutative algebras of dimension
$n$ with $s$-dimensional annihilator, given those algebras of dimension $n-s$,
in the following way:

  1. 1.

For a given anticommutative algebra $A$ of dimension $n-s$, determine
$H^{2}(A,\mathbb{C})$, $\mathrm{\mathrm{Ann}}(A)$ and
$\operatorname{\mathrm{Aut}}(A)$.

  2. 2.

Determine the set of $\mathrm{Aut}(A)$-orbits on $T_{s}(A)$.

  3. 3.

For each orbit, construct the anticommutative algebra corresponding to a
representative of it.

This method gives all (Tortkara and non-Tortkara) anticommutative algebras
with non-trivial annihilator. We want to develop this method in such a way
that it only gives non-Tortkara anticommutative algebras. Clearly, any
annihilator extension of non-Tortkara anticommutative algebra is non-Tortkara.
So, we only have to study the central extensions of Tortkara algebras. Let $A$
be a Tortkara algebra and $\theta\in Z^{2}(A,\mathbb{C})$. Then $A_{\theta}$
is a Tortkara algebra if and only if $\theta((ab),(cb))=\theta(J(a,b,c),b)$
for all $a,b,c\in A$, where $J(a,b,c)=(ab)c+(bc)a+(ca)b.$ Define a subspace
$Z_{T}^{2}(A,\mathbb{C})$ of $Z^{2}(A,\mathbb{C})$ by

$Z_{T}^{2}(A,\mathbb{C})=\left\\{\theta\in
Z^{2}(A,\mathbb{C}):\theta((ab),(cb))=\theta(J(a,b,c),b)\,\forall a,b,c\in
A\right\\}$

Define $H_{T}^{2}(A,\mathbb{C})=Z_{T}^{2}(A,\mathbb{C})/B^{2}(A,\mathbb{C}).$
Therefore, $H_{T}^{2}(A,\mathbb{C})$ is a subspace of $H^{2}(A,\mathbb{C})$.
Define $R_{s}(A)=T_{s}(A)\cap G_{s}\left(H_{T}^{2}(A,\mathbb{C})\right)$. Then
$T_{s}(A)=R_{s}(A)\cup U_{s}(A)$ where $U_{s}(A)=T_{s}(A)-R_{s}(A).$ The sets
$R_{s}(A)$ and $U_{s}(A)$ are stable under the action of $\mathrm{Aut}(A)$.
Let $E_{T}(A,V)=\left\\{A_{\theta}\in
E(A,V):A_{\theta}\mathrm{\,\,is\,\,Tortkara\,\,algebra}\right\\}.$ Then
$E(A,V)=E_{T}(A,V)\cup E_{non-T}(A,V)$ where
$E_{{non-T}}(A,V)=E(A,V)-E_{T}(A,V).$

######  Theorem 3.

[8, Theorem 19] Let $A$ be a Tortkara algebra.

  1. 1.

There exists a one-to-one correspondence between the set of
$\mathrm{Aut}(A)$-orbits on $R_{s}(A)$ and the set of isomorphism classes of
$E_{T}(A,V)$.

  2. 2.

There exists a one-to-one correspondence between the set of
$\mathrm{Aut}(A)$-orbits on $U_{s}(A)$ and the set of isomorphism classes of
$E_{non-T}(A,V)$.

By this theorem and Theorem 3, we may construct all non-Tortkara
anticommutative algebras of dimension $n$ with $s$-dimensional annihilator,
given those algebras of dimension $n-s$, in the following way:

  1. 1.

For a given anticommutative algebra $A$ of dimension $n-s$, if $A$ is non-
Tortkara then do the following:

    1. (a)

Determine $H^{2}(A,\mathbb{C})$, $\mathrm{\mathrm{\mathrm{Ann}}}(A)$ and
$\mathrm{Aut}(A)$.

    2. (b)

Determine the set of $\mathrm{Aut}(A)$-orbits on $T_{s}(A)$.

    3. (c)

For each orbit, construct the anticommutative algebra corresponding to a
representative of it.

  2. 2.

Otherwise, do the following:

    1. (a)

Determine $H_{T}^{2}(A,\mathbb{C}),H^{2}(A,\mathbb{C}),\mathrm{Ann}(A)$ and
$\mathrm{Aut}(A)$.

    2. (b)

Determine the set of $\mathrm{Aut}(A)$-orbits on $U_{s}(A)$.

    3. (c)

For each orbit, construct the anticommutative algebra corresponding to a
representative of it.

##  3 The proof of Theorem 1

Thanks to [9], we have the classification of all nontrivial 4-dimensional
nilpotent anticommutative algebras.

$\begin{array}[]{|l|l|l|l|}\hline\cr{A}&\text{Multiplication table
}&{H}_{{T}}^{2}({A},\mathbb{C})&{H}^{2}({A},\mathbb{C})\\\
\hline\cr{A}_{1}&\rm{trivial\,\,algebra}&\begin{aligned}
&\left\langle\left[\Delta_{12}\right],\left[\Delta_{13}\right],\left[\Delta_{14}\right],\right.\\\
&\left.\left[\Delta_{23}\right],\left[\Delta_{24}\right],\left[\Delta_{34}\right]\right\rangle\end{aligned}&\begin{aligned}
&\left\langle\left[\Delta_{12}\right],\left[\Delta_{13}\right],\left[\Delta_{14}\right],\right.\\\
&\left.\left[\Delta_{23}\right],\left[\Delta_{24}\right],\left[\Delta_{34}\right]\right\rangle\end{aligned}\\\
\hline\cr{A}_{2}&e_{1}e_{2}=e_{3}&\left\langle\left[\Delta_{13}\right],\left[\Delta_{14}\right],\left[\Delta_{23}\right],\left[\Delta_{24}\right],\left[\Delta_{34}\right]\right\rangle&{H}_{{T}}^{2}\left({A}_{2},\mathbb{C}\right)\\\
\hline\cr{A}_{3}&e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4}&\left\langle\left[\Delta_{14}\right],\left[\Delta_{23}\right],\left[\Delta_{24}\right]\right\rangle&{H}_{{T}}^{2}\left({A}_{3},\mathbb{C}\right)\oplus\left\langle\left[\Delta_{34}\right]\right\rangle\\\
\hline\cr\end{array}$

In view of [5], all anticommutative central extensions of $A_{2}$ and of the
4-dimensional trivial algebra are Tortkara algebras, so we need only consider
central extensions of $A_{3}$. After direct calculation, we can get the form
of $\mathrm{Aut}(A_{3})$:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\ u&v&xz&0\\\
h&g&xv&x^{2}z\end{array}\right)$

where $xz\not=0$.

In the following subsections we will give the classification of all
$n$-dimensional nilpotent non-Tortkara anticommutative algebras with
$(n-4)$-dimensional annihilator.

###  3.1 $n=4$

There is no anticommutative algebra satisfies conditions of Theorem 1, because
the annihilator of $A_{1},A_{2}$ and $A_{3}$ are non-empty.

###  3.2 $n=5$

The anticommutative algebra satisfying conditions of Theorem 1 must have no
annihilator component. It is non-split non-Tortkara central extensions of
$A_{3}$. According to Theorem 3, we need to find the representatives of the
$\mathrm{Aut}(A_{3})$-orbits on $U_{1}(A_{3})$. Choose an arbitrary subspace
$W\in U_{1}(A_{3})$. Such a subspace is spanned by
$\theta=a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]+a_{4}[\Delta_{34}]$
with $a_{4}\not=0$. Let $\phi\in\mathrm{Aut}(A_{3})$ be the following
automorphism

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\
-(a_{1}x+a_{3}y)&-a_{3}z&xz&0\\\ h&a_{2}z&-a_{3}zx&x^{2}z\end{array}\right)$

where $xz\not=0$. Then $\phi W=<[\Delta_{34}]>$. Hence we get a representative
$<[\Delta_{34}]>$. This shows that $\mathrm{Aut}(A_{3})$ has only one orbit on
$U_{1}(A_{3})$. So we get the algebra:

$A_{5,1}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5}.$

###  3.3 $n=6$

First we classify nilpotent non-Tortkara anticommutative algebras with
annihilator component. We get the algebras
$A_{6,1}=A_{5,1}\oplus\mathbb{C}e_{6}$. Next we classify nilpotent non-
Tortkara anticommutative without any annihilator component. For this, choose
an arbitrary subspace $W\in U_{2}(A_{3})$. Such a subspace is spanned by
$[\theta_{1}]=a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]+a_{4}[\Delta_{34}]$
and
$[\theta_{2}]=b_{1}[\Delta_{14}]+b_{2}[\Delta_{23}]+b_{3}[\Delta_{24}]+b_{4}[\Delta_{34}]$
with $(a_{4},b_{4})\not=(0,0)$. By possibly swapping $[\theta_{1}]$ and
$[\theta_{2}]$, we may assume with out loss of generality that $a_{4}\not=0$.
Then, from Subsection 3.2, we may assume that $[\theta_{1}]=[\Delta_{34}]$.
Further, by subtracting scalar multiples of $[\theta_{1}]$ from
$[\theta_{2}]$, we may assume that
$[\theta_{2}]=a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]$. Hence
we may assume without loss of generality that

$W=<[\Delta_{34}],a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]>.$

Let us consider the following cases:

Case1: $a_{3}\not=0$, Let $\phi\in\mathrm{Aut}(A_{3})$ be as follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ -a_{1}xa_{3}^{-1}&z&0&0\\\
0&-a_{2}za_{3}^{-1}&xz&0\\\
h&v^{2}z^{-1}&-a_{2}za_{3}^{-1}x&x^{2}z\end{array}\right)$

where $xz\not=0$. Then $\phi
W=<vzx^{2}[\Delta_{24}]+x^{3}z^{2}[\Delta_{34}],a_{3}x^{2}z^{2}[\Delta_{24}]>=<[\Delta_{34}],[\Delta_{24}]>=W_{1}$

Case2: $a_{3}=a_{1}=0,a_{2}\not=0$, Let $\phi\in\mathrm{Aut}(A_{3})$ be as
follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $xz\not=0$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],a_{2}xz^{2}[\Delta_{23}]>=<[\Delta_{34}],[\Delta_{23}]>=W_{2}$

Case3: $a_{3}=a_{2}=0,a_{1}\not=0$, Let $\phi\in\mathrm{Aut}(A_{3})$ be as
follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $xz\not=0$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],a_{1}zx^{3}[\Delta_{14}]>=<[\Delta_{34}],[\Delta_{14}]>=W_{3}$

Case4: $a_{3}=0,a_{1}a_{2}\not=0$, Let $\phi\in\mathrm{Aut}(A_{3})$ be as
follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ 0&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $x=a_{2},z=a_{1}a_{2}$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],a_{1}^{2}a_{2}^{4}([\Delta_{23}]+[\Delta_{14}])>=<[\Delta_{34}],[\Delta_{23}]+[\Delta_{14}]>=W_{4}$

As shown we have three representatives, namely

$\displaystyle W_{1}:<[\Delta_{34}],[\Delta_{24}]>$ $\displaystyle
W_{2}:<[\Delta_{34}],[\Delta_{23}]>$ $\displaystyle
W_{3}:<[\Delta_{34}],[\Delta_{14}]>$ $\displaystyle
W_{4}:<[\Delta_{34}],[\Delta_{23}]+[\Delta_{14}]>$

Next, we claim that
$\mathrm{Orb}(W_{i})\cap\mathrm{Orb}(W_{j})=\emptyset,i\not=j$. Choose any
element of $\mathrm{Aut}(A_{3})$:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\ u&v&xz&0\\\
h&g&xv&x^{2}z\end{array}\right)$

where $xz\not=0$. Then

$\phi W_{1}=<x^{2}z^{2}[\Delta_{34}]-zg[\Delta_{23}]+x(uz-
yv)[\Delta_{14}],xz[\Delta_{24}]+v[\Delta_{23}]+xy[\Delta_{14}]>$

$[\Delta_{23}]$ is not in $\phi W_{1}$, otherwise there are $\lambda_{1}$ and
$\lambda_{2}$ such that

$[\Delta_{23}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]-zg[\Delta_{23}]+x(uz-
yv)[\Delta_{14}])+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}]+xy[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{2}=0$. Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{2})=\emptyset$. $[\Delta_{14}]$ is not
in $\phi W_{1}$, otherwise there are $\lambda_{1}$ and $\lambda_{2}$ such that

$[\Delta_{14}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]-zg[\Delta_{23}]+x(uz-
yv)[\Delta_{14}])+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}]+xy[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{2}=0$. Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{3})=\emptyset$.
$[\Delta_{14}]+[\Delta_{23}]$ is not in $\phi W_{1}$, otherwise there are
$\lambda_{1}$ and $\lambda_{2}$ such that

$[\Delta_{14}]+[\Delta_{23}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]-zg[\Delta_{23}]+x(uz-
yv)[\Delta_{14}])+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}]+xy[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{2}=0$. Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{4})=\emptyset$.

After $W_{2}$ is acted on by $\phi$, we can get

$\phi
W_{2}=<x^{2}z^{2}[\Delta_{34}]+zxv[\Delta_{24}]+uzx[\Delta_{14}],[\Delta_{23}]>$

$[\Delta_{14}]$ is not in $\phi W_{2}$, otherwise there are $\lambda_{1}$ and
$\lambda_{2}$ such that

$[\Delta_{14}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]+zxv[\Delta_{24}]+uzx[\Delta_{14}])+\lambda_{2}[\Delta_{23}]$

We have $\lambda_{1}=\lambda_{2}=0$, Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{3})=\emptyset$.
$[\Delta_{14}]+[\Delta_{23}]$ is not in $\phi W_{2}$, otherwise there are
$\lambda_{1}$ and $\lambda_{2}$ such that

$[\Delta_{14}]+[\Delta_{23}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]+zxv[\Delta_{24}]+uzx[\Delta_{14}])+\lambda_{2}[\Delta_{23}]$

We have $\lambda_{1}=0,\lambda_{2}=1$. Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{4})=\emptyset$.

After $W_{3}$ is acted on by $\phi$, we can get

$\phi
W_{3}=<x^{2}z^{2}[\Delta_{34}]+zxv[\Delta_{24}]+(v^{2}-zg)[\Delta_{23}],[\Delta_{14}]>$

$[\Delta_{14}]+[\Delta_{23}]$ is not in $\phi W_{3}$, otherwise there are
$\lambda_{1}$ and $\lambda_{2}$ such that

$[\Delta_{14}]+[\Delta_{23}]=\lambda_{1}(x^{2}z^{2}[\Delta_{34}]+zxv[\Delta_{24}]+(v^{2}-zg)[\Delta_{23}])+\lambda_{2}[\Delta_{14}]$

We have $\lambda_{1}=0,\lambda_{2}=1$. Obviously, the above equation cannot be
holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{3})\cap\mathrm{Orb}(W_{4})=\emptyset$. All in all, we have
$\mathrm{Orb}(W_{i})\cap\mathrm{Orb}(W_{j})=\emptyset,i\not=j$. So, the
corresponding anticommutative algebras are pairwise non-isomorphic. Hence we
get the algebras:

$\displaystyle
A_{6,2}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6};$
$\displaystyle
A_{6,3}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{3}=e_{6};$
$\displaystyle
A_{6,4}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{1}e_{4}=e_{6};$
$\displaystyle
A_{6,5}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{3}=e_{6},e_{1}e_{4}=e_{6}.$

###  3.4 $n=7$

First we classify the non-Tortkara anticommutative algebras with annihilator
component. We get the algebras
$A_{7,i}=A_{6,i}\oplus\mathbb{C}e_{7},i=1,\cdots,5$. Next we classify the non-
Tortkara anticommutative algebras without any annihilator component. Choose an
arbitrary subspace $W\in U_{3}(A_{3})$. Such a subspace is spanned by

$\displaystyle[\theta_{1}]=a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]+a_{4}[\Delta_{34}],$
$\displaystyle[\theta_{2}]=b_{1}[\Delta_{14}]+b_{2}[\Delta_{23}]+b_{3}[\Delta_{24}]+b_{4}[\Delta_{34}]$
$\displaystyle[\theta_{3}]=c_{1}[\Delta_{14}]+c_{2}[\Delta_{23}]+c_{3}[\Delta_{24}]+c_{4}[\Delta_{34}]$

with $(a_{4},b_{4},c_{4})\not=(0,0,0)$. In view of Subsection 3.3, we may
assume without loss of generality that $W\in\\{S_{1},S_{2},S_{3},S_{4}\\}$
where

$\displaystyle
S_{1}=<[\Delta_{34}],[\Delta_{24}],a_{1}[\Delta_{14}]+a_{2}[\Delta_{23}]>$
$\displaystyle
S_{2}=<[\Delta_{34}],[\Delta_{14}],a_{2}[\Delta_{23}]+a_{3}[\Delta_{24}]>$
$\displaystyle
S_{3}=<[\Delta_{34}],[\Delta_{23}],a_{1}[\Delta_{14}]+a_{3}[\Delta_{24}]>$
$\displaystyle
S_{4}=<[\Delta_{34}],[\Delta_{14}]+[\Delta_{23}],a_{1}[\Delta_{14}]+a_{3}[\Delta_{24}]>$

Case1: $W=S_{1}$

Case1.1: $a_{1}=0$ or $a_{2}=0$, then
$W=<[\Delta_{34}],[\Delta_{24}],[\Delta_{23}]>$ or
$W=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]>$.

Case1.2: $a_{1}a_{2}\not=0$. Let $\phi\in\mathrm{Aut}(A_{3})$ be as follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ 0&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $x=a_{2},z=a_{1}a_{2}$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],x^{2}z^{2}[\Delta_{24}],a_{1}^{2}a_{2}^{4}([\Delta_{14}]+[\Delta_{23}])>=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]+[\Delta_{23}]>$

Case2: $W=S_{2}$

Case2.1: $a_{2}=0$ or $a_{3}=0$. Then
$W=<[\Delta_{34}],[\Delta_{14}],[\Delta_{24}]>$ or
$W=<[\Delta_{34}],[\Delta_{14}],[\Delta_{23}]$.

Case2.2: $a_{2}a_{3}\not=0$. Let $\phi\in\mathrm{Aut}(A_{3})$ be as follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ 0&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $x=a_{3}^{2},z=a_{2}a_{3}^{-1}$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],zx^{3}[\Delta_{14}],a_{2}^{3}([\Delta_{23}]+[\Delta_{24}])>=<[\Delta_{34}],[\Delta_{14}],[\Delta_{23}]+[\Delta_{24}]>$

Case3: $W=S_{3}$

Case3.1: $a_{1}=0$ or $a_{3}=0$. Then
$W=<[\Delta_{34}],[\Delta_{23}],[\Delta_{24}]>$ or
$W=<[\Delta_{34}],[\Delta_{23}],[\Delta_{14}]>$.

Case3.2: $a_{1}a_{3}\not=0$. Let $\phi\in\mathrm{Aut}(A_{3})$ be as follows:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ 0&z&0&0\\\ 0&0&xz&0\\\
h&0&0&x^{2}z\end{array}\right)$

where $x=a_{3},z=a_{1}$. Then $\phi
W=<x^{3}z^{2}[\Delta_{34}],xz^{2}[\Delta_{23}],a_{1}^{2}a_{3}^{3}([\Delta_{14}]+[\Delta_{24}])>=<[\Delta_{34}],[\Delta_{23}],[\Delta_{14}]+[\Delta_{24}]>$.

Case4: $W=S_{4}$

Case4.1: $a_{1}=0$ or $a_{3}=0$. Then
$W=<[\Delta_{34}],[\Delta_{14}]+[\Delta_{23}],[\Delta_{24}]>$ or
$W=<[\Delta_{34}],[\Delta_{23}],[\Delta_{14}]>$.

Case4.2: $a_{1}a_{3}\not=0$. Let $\phi\in\mathrm{Aut}(A_{3})$ be as follows:

$\phi=\left(\begin{array}[]{cccc}1&0&0&0\\\ -a_{1}a_{3}^{-1}&1&0&0\\\
0&0&1&0\\\ 0&0&0&1\end{array}\right)$

Then $\phi
W=<[\Delta_{34}],[\Delta_{14}]+[\Delta_{23}],a_{3}[\Delta_{24}]>=<[\Delta_{34}],[\Delta_{14}]+[\Delta_{23}],[\Delta_{24}]>$

To summarize, we have the following representatives:

$\displaystyle W_{1}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{23}]>$
$\displaystyle W_{2}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]>$
$\displaystyle W_{3}=<[\Delta_{34}],[\Delta_{14}],[\Delta_{23}]>$
$\displaystyle
W_{4}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]+[\Delta_{23}]>$
$\displaystyle
W_{5}=<[\Delta_{34}],[\Delta_{14}],[\Delta_{23}]+[\Delta_{24}]>$
$\displaystyle
W_{6}=<[\Delta_{34}],[\Delta_{23}],[\Delta_{14}]+[\Delta_{24}]>$

Let us now determine the possible orbits among the representatives
$W_{1},\cdots,W_{6}$. Choose any element of $\mathrm{Aut}(A_{3})$:

$\phi=\left(\begin{array}[]{cccc}x&0&0&0\\\ y&z&0&0\\\ u&v&xz&0\\\
h&g&xv&x^{2}z\end{array}\right)$

where $xz\not=0$. Then

$\phi
W_{1}=<[\Delta_{23}],z[\Delta_{24}]+y[\Delta_{14}],xz[\Delta_{34}]+\left(u-vyz^{-1}\right)[\Delta_{14}]>$

Let $y=z,u=v$, we have $\phi W_{1}=W_{6}$. Then
$\mathrm{Orb}(W_{1})=\mathrm{Orb}(W_{6})$. $[\Delta_{14}]$ is not in $\phi
W_{1}$, otherwise there are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$
such that

$[\Delta_{14}]=\lambda_{1}[\Delta_{23}]+\lambda_{2}(z[\Delta_{24}]+y[\Delta_{14}])+\lambda_{3}(xz[\Delta_{34}]+\left(u-vyz^{-1}\right)[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{2}=\lambda_{3}=0$. Obviously, the above equation
cannot be holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{2})=\emptyset$,
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{3})=\emptyset$ and
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{5})=\emptyset$. We claim that
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{4})=\emptyset$, otherwise there are
$\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{34}]=\lambda_{1}[\Delta_{23}]+\lambda_{2}(z[\Delta_{24}]+y[\Delta_{14}])+\lambda_{3}(xz[\Delta_{34}]+\left(u-vyz^{-1}\right)[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{2}=0,u=vyz^{-1}$. There are $\lambda_{1}$ ,
$\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{24}]=\lambda_{1}[\Delta_{23}]+\lambda_{2}(z[\Delta_{24}]+y[\Delta_{14}])+\lambda_{3}(xz[\Delta_{34}]+\left(u-vyz^{-1}\right)[\Delta_{14}])$

We have $\lambda_{1}=\lambda_{3}=0,y=0$ and $u=y=0$. There are $\lambda_{1}$ ,
$\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{14}]+[\Delta_{23}]=\lambda_{1}[\Delta_{23}]+\lambda_{2}z[\Delta_{24}]+\lambda_{3}xz[\Delta_{34}]$

We have $\lambda_{1}=1,\lambda_{2}=\lambda_{3}=0$. Obviously, the above
equation cannot be holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{1})\cap\mathrm{Orb}(W_{4})=\emptyset$.

After $W_{2}$ is acted on by $\phi$, we can get

$\phi
W_{2}=<[\Delta_{14}],xz[\Delta_{24}]+v[\Delta_{23}],x^{2}z[\Delta_{34}]-g[\Delta_{23}]>$

Let $g=0,v=xz$, we have $\phi W_{2}=W_{5}$. Then
$\mathrm{Orb}(W_{2})=\mathrm{Orb}(W_{5})$. We claim that
$\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{3})=\emptyset$, otherwise there are
$\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{23}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}])+\lambda_{3}(x^{2}z[\Delta_{34}]-g[\Delta_{23}])$

We have $\lambda_{1}=\lambda_{2}=\lambda_{3}=0$. Obviously, the above equation
cannot be holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{3})=\emptyset$.

We claim that $\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{4})=\emptyset$,
otherwise there are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{34}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}])+\lambda_{3}(x^{2}z[\Delta_{34}]-g[\Delta_{23}])$

We have $g=0$. There are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such
that

$[\Delta_{24}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}(xz[\Delta_{24}]+v[\Delta_{23}])+\lambda_{3}(x^{2}z[\Delta_{34}]-g[\Delta_{23}])$

We have $v=0$. There are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such
that

$[\Delta_{14}]+[\Delta_{23}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}xz[\Delta_{24}]+\lambda_{3}x^{2}z[\Delta_{34}]$

We have $\lambda_{1}=\lambda_{2}=\lambda_{3}=0$. Obviously, the above equation
cannot be holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{2})\cap\mathrm{Orb}(W_{4})=\emptyset$.

After $W_{3}$ is acted on by $\phi$, we can get

$\phi W_{3}=<[\Delta_{14}],[\Delta_{23}],xz[\Delta_{34}]+v[\Delta_{24}]>$

We claim that $\mathrm{Orb}(W_{3})\cap\mathrm{Orb}(W_{4})=\emptyset$,
otherwise there are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such that

$[\Delta_{34}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}[\Delta_{23}]+\lambda_{3}(xz[\Delta_{34}]+v[\Delta_{24}])$

We have $v=0$. There are $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ such
that

$[\Delta_{24}]=\lambda_{1}[\Delta_{14}]+\lambda_{2}[\Delta_{23}]+\lambda_{3}xz[\Delta_{34}]$

We have $\lambda_{1}=\lambda_{2}=\lambda_{3}=0$. Obviously, the above equation
cannot be holded, it’s a contradiction. Then
$\mathrm{Orb}(W_{3})\cap\mathrm{Orb}(W_{4})=\emptyset$.

All representatives of $\mathrm{Aut}(A_{3})$-orbits are:

$\displaystyle W_{1}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{23}]>$
$\displaystyle W_{2}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]>$
$\displaystyle W_{3}=<[\Delta_{34}],[\Delta_{14}],[\Delta_{23}]>$
$\displaystyle
W_{4}=<[\Delta_{34}],[\Delta_{24}],[\Delta_{14}]+[\Delta_{23}]>$

So the corresponding anticommutative algebras are pairwise non-isomorphic.
Thus we get the algebras:

$\displaystyle
A_{7,6}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{2}e_{3}=e_{7};$
$\displaystyle
A_{7,7}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{1}e_{4}=e_{7};$
$\displaystyle
A_{7,8}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{1}e_{4}=e_{6},e_{2}e_{3}=e_{7};$
$\displaystyle
A_{7,9}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{1}e_{4}=e_{7},e_{2}e_{3}=e_{7}.$

###  3.5 $n=8$

First we classify the non-Tortkara anticommutative algebras with annihilator
component. We get the algebras
$A_{8,i}=A_{7,i}\oplus\mathbb{C}e_{8},i=1,\cdots,9$. Next we classify the non-
Tortkara anticommutative algebras without any annihilator component. Choose an
arbitrary subspace $W\in U_{4}(A_{3})$. Then $W=H^{2}(A_{3},\mathbb{C})$. So
we have only one orbit with a representative
$<[\Delta_{14}],[\Delta_{23}],[\Delta_{24}],[\Delta_{34}]>$. Hence we get the
algebra:

$A_{8,10}:e_{1}e_{2}=e_{3},e_{1}e_{3}=e_{4},e_{3}e_{4}=e_{5},e_{2}e_{4}=e_{6},e_{2}e_{3}=e_{7},e_{1}e_{4}=e_{8}.$

###  3.6 $n\geqslant 9$

From the results of Subsection 3.5, we get the algebras:

$A_{n,i}=A_{8,i}\oplus\mathbb{C}e_{9}\oplus\cdots\oplus\mathbb{C}e_{n},i=1,\ldots,10.$

##  4 Acknowledgment

I would like to express my sincere gratitude to professor Jianzhi Han from
Tongji University.

## References

  * [1] María Alejandra Alvarez, Renato Fehlberg Júnior, and Ivan Kaygorodov.  The algebraic and geometric classification of zinbiel algebras.  Journal of Pure and Applied Algebra, page 107106, 2022.
  * [2] Antonio Jesús Calderón, Amir Fernández Ouaridi, and Ivan Kaygorodov.  The classification of n-dimensional anticommutative algebras with (n-3)-dimensional annihilator.  Communications in Algebra, 47(1):173–181, 2019.
  * [3] Ismail Demir, Kailash C Misra, and Ernie Stitzinger.  On classification of four-dimensional nilpotent leibniz algebras.  Communications in Algebra, 45(3):1012–1018, 2017.
  * [4] Ming-Peng Gong.  Classification of nilpotent lie algebras of dimension 7 (over algebraically closed field and $\mathbb{R}$).  1998\.
  * [5] Ilya E Gorshkov, Ivan Kaygorodov, Alexey A Kytmanov, and Mohamed A Salim.  The variety of nilpotent tortkara algebras.  Journal of Siberian Federal University. Mathematics and Physics, 12(2):173–184, 2019.
  * [6] AS Hegazi and Hani Abdelwahab.  Classification of five-dimensional nilpotent jordan algebras.  Linear Algebra and its Applications, 494:165–218, 2016.
  * [7] AS Hegazi and Hani Abdelwahab.  The classification of n-dimensional non-associative jordan algebras with (n-3)-dimensional annihilator.  Communications in Algebra, 46(2):629–643, 2018.
  * [8] AS Hegazi, Hani Abdelwahab, and AJ Calderon Martin.  The classification of n-dimensional non-lie malcev algebras with (n-4)-dimensional annihilator.  Linear Algebra and its Applications, 505:32–56, 2016.
  * [9] Ivan Kaygorodov, Mykola Khrypchenko, and Samuel A Lopes.  The algebraic and geometric classification of nilpotent anticommutative algebras.  Journal of Pure and Applied Algebra, 224(8):106337, 2020.


# CXL Memory as Persistent Memory for Disaggregated HPC:
A Practical Approach

Yehonatan Fridman  Ben-Gurion University, NRCN, Israel<EMAIL_ADDRESS>,  Suprasad Mutalik Desai, Navneet Singh  Intel, India
<EMAIL_ADDRESS>,  Thomas Willhalm  Intel, Germany
<EMAIL_ADDRESS>and  Gal Oren  Technion, NRCN, Israel
<EMAIL_ADDRESS>
###### Abstract.

In the landscape of High-Performance Computing (HPC), the quest for efficient
and scalable memory solutions remains paramount. The advent of Compute Express
Link (CXL) introduces a promising avenue with its potential to function as a
Persistent Memory (PMem) solution in the context of disaggregated HPC systems.
This paper presents a comprehensive exploration of CXL memory’s viability as a
candidate for PMem, supported by physical experiments conducted on cutting-
edge multi-NUMA nodes equipped with CXL-attached memory prototypes. Our study
not only benchmarks the performance of CXL memory but also illustrates the
seamless transition from traditional PMem programming models to CXL,
reinforcing its practicality.

To substantiate our claims, we establish a tangible CXL prototype using an
FPGA card embodying CXL 1.1/2.0 compliant endpoint designs (Intel FPGA CXL
IP). Performance evaluations, executed through the STREAM and STREAM-PMem
benchmarks, showcase CXL memory’s ability to mirror PMem characteristics in
App-Direct and Memory Mode while achieving impressive bandwidth metrics with
Intel 4th generation Xeon (Sapphire Rapids) processors.

The results elucidate the feasibility of CXL memory as a persistent memory
solution, outperforming previously established benchmarks. In contrast to
published DCPMM results, our CXL-DDR4 memory module offers comparable
bandwidth to local DDR4 memory configurations, albeit with a moderate decrease
in performance. The modified STREAM-PMem application underscores the ease of
transitioning programming models from PMem to CXL, thus underscoring the
practicality of adopting CXL memory.

The sources of this work are available at: https://github.com/Scientific-
Computing-Lab-NRCN/STREAMer.

CXL, Memory disaggregation, Persistent Memory (PMem), Intel Optane DCPMM, HPC,
STREAM, STREAM-PMem, STREAMer

††copyright: none††conference: ; ; ††copyright: none

##  1\. Introduction

###  1.1. Current HPC Memory Solutions Limitations

As the era of Exa-Scale computing unfolds, the demand for analyzing,
manipulating, and storing massive amounts of data intensifies (Reed et al.,
2022). Exascale systems are designed to meet these demands and enable the
execution of a broad spectrum of computations, ranging from loosely to tightly
coupled tasks, including CFD simulations and deep learning optimizations
(Evans et al., 2022). Memory and storage resources play a crucial role in the
performance and scalability of these computations (Kogge and Dally, 2022).
Memory factors such as capacity, latency, and bandwidth are responsible for
successfully handling extensive tasks and delivering data to processing units
promptly (McKee, 2004). In scientific computing, storage devices hold
significance for preserving diagnostics throughout computations (Lockwood et
al., 2023). Notably, the growing frequency of failures in exascale machines
emphasizes the significance of storing vast data volumes to support recovery
and bolster fault tolerance (Bustos et al., 2023; Fridman et al., 2022).

However, the traditional memory and storage hierarchy in HPC systems reveals
notable gaps that impose critical constraints on scientific computations
(Kogge and Dally, 2022). From the vantage point of memory architecture, DRAM
has inherent limitations of bandwidth and capacity that impact performance and
prevent the processing of large-scale problems (Radulovic et al., 2015; Peng
et al., 2021). From the storage perspective, traditional devices (such as HDDs
and SSDs) provide large capacities but exhibit very slow access times, leading
to significant overheads for I/O-bound applications and fault tolerance
mechanisms (Lockwood et al., 2023). These gaps and limitations of traditional
hardware highlight the ongoing endeavors to expand the memory-storage
hierarchy and develop novel memory architectures and solutions. A notable
example is Non-Volatile RAM (Wang et al., 2023; Rai and Talawar, 2023) (on
which we elaborate in subsection 1.2).

While High-Bandwidth Memory (HBM) (Jun et al., 2017) has been introduced as a
solution to enhance memory performance, it doesn’t entirely alleviate the
problem (Jun et al., 2016). HBM memory modules are stacked vertically,
allowing for higher memory bandwidth due to their increased parallelism.
However, even with HBM, the memory capacity remains limited compared to
conventional DDR (Double Data Rate) memory modules (Shipman et al., 2022).
This limitation can still lead to constraints in memory-intensive applications
that require larger memory spaces (McCalpin, 2023). Moreover, while HBM
addresses the bandwidth issue to some extent, it doesn’t eliminate the
underlying problem of memory hierarchy (McCalpin, 2023). The processor still
needs to access different memory levels, and the latency of transferring data
between these levels can impact performance (McCalpin, 2023). HBM improves
bandwidth between the processor and certain memory modules, but the need to
access different levels of memory introduces latency that can affect the
execution of various tasks (McCalpin, 2023).

In general, it is possible to proclaim that the conventional approach of
locating memory modules directly on the board poses significant challenges in
the context of HPC systems (Ding et al., 2023). This arrangement restricts
memory bandwidth due to the limited number of connections between the
processor and these modules (Ding et al., 2023). As a result, the data
transfer rate between the processor and board-mounted memory becomes a
bottleneck, hindering the overall performance of the system (Ding et al.,
2023).

For an increase of memory capacity outside of the node, advanced communication
technologies such as the Remote Direct Memory Access (RDMA) based Message
Passing Interface (MPI) optimize inter-node communication (Liu et al., 2003).
However, these sophisticated frameworks are not devoid of challenges (Gropp,
2012): MPI, a cornerstone for distributed computing communication, contends
with latency and overhead issues during message transmission,
disproportionately affecting efficiency for applications requiring frequent
communication. Furthermore, the management complexity escalates with the
cluster’s scale due to heightened contention for network resources among a
larger node count (Bernholdt et al., 2020).

###  1.2. Persistent Memory in HPC

Property |  As a main memory extension |  As a direct access to persistent memory
---|---|---
Volatility |  Volatile in memory extension mode |  Non-volatile in direct access mode
Access |  Cache-coherent memory expansion |  Transactional byte-addressable object store
Capacity |  Higher than main memory volume |  Lower than storage volume
Cost |  Cheaper than the main memory |  More expansive than storage
Performance |  Several factors below main memory bandwidth |  High bandwidth compared to storage
Table 1. Properties of PMem modules, either as a memory extension (Memory
Mode) or as a direct access PMem (App-Direct).

A proposed solution aimed at bridging the gap between memory and storage is
Persistent Memory (PMem) (Lee et al., 2010; Mutlu, 2013). PMem implementations
such as BBU (battery backed up) DIMM or Non-Volatile RAM (NVRAM) aim to
deliver rapid byte-addressable data access alongside persistent data retention
across power cycles. PMem technologies establish a new tier within the memory-
storage hierarchy by combining memory and storage characteristics (Wang et
al., 2023; Rai and Talawar, 2023). Basic solutions include battery-backed DRAM
and have been accessible from diverse vendors over a significant timeframe,
representing an established concept (Narayanan and Hodson, 2012; Sainio et
al., 2016; Kateja et al., 2017; Malladi et al., 2016). However, these
solutions face challenges due to limited scalability and potential data loss
risks. The reliance on batteries introduces concerns regarding power failures,
leading to potential data corruption or loss if batteries deplete. Moreover,
the approach’s scalability is hampered by the need for individual batteries
for each module, impacting cost-effectiveness and overall system performance.

Yet, in recent years new PMem technologies have emerged, with 3D-Xpoint (Hady
et al., 2017) being the main technology and Intel Optane DCPMM (Weiland et
al., 2019; Hirofuchi and Takano, 2020) the prominent product on the market.
These modern PMem technologies offer byte-addressable memory in larger
capacities compared to DRAM while maintaining comparable access times
(Izraelevitz et al., 2019). Moreover, as these technologies are non-volatile
in nature, they enable data retrieval even in instances of power failures.
Moreover, PMem offers two configuration options based on these
characteristics: (1) It can be utilized as main memory expansion, providing
additional volatile memory, and (2) it can serve as a persistent memory pool
that can be accessed by applications via a PMem-aware file system (Wang et
al., 2023) or be managed and accessed directly by applications (Izraelevitz et
al., 2019). To simplify and streamline PMem programming and management, the
Persistent Memory Development Kit (PMDK) was created (Scargall, 2020).

During recent years, PMem has gained significant traction in HPC applications
(Patil et al., 2019; Mironov et al., 2019; Fridman et al., 2021; Rudoff,
2017), with two direct use cases of PMem for scientific applications that
require no (or minimal) changes to applications. The first use-case involves
PMem as memory expansion to support the execution of large scientific problems
(Mironov et al., 2019). The second use case involves leveraging PMem as a fast
storage device accessed by a PMem-aware file system (mainly based on the POSIX
API), primarily for application diagnostics and checkpoint restart (C/R)
mechanisms (Logan et al., 2023), but also for increasing the performance and
inherent fault tolerance of scientific applications (Fridman et al., 2022).

In addition to the direct use cases of PMem in scientific applications,
various frameworks and algorithms were developed to access and manage data
structures on PMem (Baldassin et al., 2021). Among these are primary methods
that are built on top of the PMDK library (Khan et al., 2020; Fridman et al.,
2022). For example, persistent memory object storage frameworks such as MOSIQS
(Khan et al., 2020) and the NVM-ESR recovery model for exact state
reconstruction of linear iterative solvers using PMem (Fridman et al., 2022).

Nevertheless, as HPC workloads advance, computing units evolve, and onboard
processing elements increase, the demand for heightened memory bandwidth
becomes essential (Kogge and Dally, 2022). Existing PMem solutions demonstrate
notable shortcomings in meeting these requirements, showing limitations in
scalability beyond a certain threshold (Fridman et al., 2021). Specifically,
PMem devices exhibit limited bandwidth. For instance, the bandwidth of Optane
DCPMM for reading and writing is multi-factor lower than that of DRAM
(Izraelevitz et al., 2019). This, in part, is connected with the hybrid and
in-between properties of a PMem module (Gugnani et al., 2020), as
schematically described in Table 1.

Adding to these challenges, a significant limitation arises from the physical
attachment of most PMem devices, like Optane DCPMM, to the CPU board through
memory DIMMs. This configuration restricts the potential for memory expansion,
as PMem contends for DIMM slots alongside conventional DRAM cards, presenting
a bottleneck to achieving optimal memory configurations (Tristian and Travis,
2019; Nathan Pham, 2019). The HPC community as a whole — both the super and
cloud computing (Ruan et al., 2023) — recognizes the drawbacks associated with
tight integrating memory and compute resources, particularly in relation to
capacity, bandwidth, elasticity, and overall system utilization (Peng et al.,
2020; Ding et al., 2023). PMem technologies that are tightly coupled with the
CPU inherit these limitations. Now, as prominent PMem technologies are phased
out (Optane DCPMM, for example, as announced in 2022 (Intel, 2022; Handy and
Coughlin, 2023)), there is an active and prominent pursuit for the adoption of
novel memory solutions in particular, and a strive to achieve more
disaggregated computing in general (Liu, 2023).

###  1.3. Dissagregated Memory with CXL

The emergence of discrete memory nodes housing DRAM and network interface
controllers (NICs) is anticipated to revolutionize conventional memory
paradigms, facilitating distributed and shared memory access and reshaping HPC
landscapes (Ding et al., 2023). This shift aligns with the concept of
disaggregation, where compute resources and memory units are decoupled for
optimized resource utilization, scalability, and adaptability.

The concept of memory disaggregation has been facilitated recently by the
development of advanced interconnect technologies, exemplified by Compute
Express Link (CXL) (Sharma et al., 2023). CXL is an open standard to support
cache-coherent interconnect between a variety of devices (Sharma et al.,
2023). After its introduction in 2019, the standard has evolved and continues
to be enhanced. CXL 1.1 defines the protocol for three major device types
(Sharma et al., 2023): Accelerators with cache-only (type 1), cache with
attached memory (type 2), and memory expansion (type 3). CXL 2.0 expands the
specification – among other capabilities – to memory pools using CXL switches
on a device level. CXL 3.0 introduces fabric capabilities and management,
improved memory sharing and pooling with dynamic capacity capability, enhanced
coherency, and peer-to-peer communication. Bandwidth-wise, CXL 1.1 and 2.0
employ PCIe 5.0, achieving 32 GT/s for transfers up to 64 GB/s in each
direction via a 16-lane link. On the other hand, CXL 3.0 utilizes PCIe 6.0,
doubling the speed to 64 GT/s, supporting 128 GB/s bi-directional
communication via an x16 link.

Since the market of CXL memory modules is emerging, several vendors have
announced products using the CXL protocol. For example, Samsung (Newsroom,
2022a) and SK Hynix (Newsroom, 2022b) introduce CXL DDR5 modules, AsteraLabs
(AsteraLabs, 2022) announced a CXL memory accelerator, and Montage Technology
(Technology, 2022) will offer a CXL memory expander controller.

Leveraging CXL, memory nodes will be interconnected through high-speed links,
enabling adaptive memory provisioning to compute nodes in real time (Wahlgren
et al., 2022). The practice of intra-rack disaggregation holds the potential
to effectively address the memory demands of applications while concurrently
ensuring an adequate supply of efficient remote memory bandwidth
(Michelogiannakis et al., 2022, 2023). Figure 1 demonstrates the expected
phase change from the processor’s point of view, from previous years’
DDR4+PMem memory access, equipped with NVMe SSDs via the PCIe Gen4, to the
upcoming future of DDR5 local memory equipped with local or remote NVMe SSDs
and CXL memory for memory expansion or persistency over the new generations of
PCIe.

Figure 1. The migration from PMem as hardware to CXL memory as PMem in future
systems.

Nevertheless, while the concept of memory disaggregation with technologies
like CXL holds significant promise, it is important to acknowledge that there
are still challenges and considerations that need to be addressed (Al Maruf
and Chowdhury, 2023; Geyer et al., 2023); challenges and considerations that
resemble the ones of persistent memory integration in HPC (Benson et al.,
2023). For example, software and programming models need to evolve to take
advantage of disaggregated memory fully; Applications and algorithms must be
designed or adapted to work seamlessly across distributed memory nodes; and
efficient data placement and movement strategies are crucial to minimize the
impact of network latencies and ensure that data-intensive workloads can
effectively utilize CXL-based disaggregated memory resources, especially when
cache-coherence or direct access is enabled. Notwithstanding, when comparing
CXL memory aspects to the ones of PMem as non-volatile RAM (NVRAM), in
general, it can be observed (Table 2) that from the disaggregated HPC usage
perspective, there should be a prevalence to CXL over NVRAM considering
bandwidth, data transfer, and scalability, but also considering memory
coherency, integration, pooling and sharing.

Aspect |  CXL Memory |  NVRAM
---|---|---
Bandwidth &
Data Transfer |  Significantly higher bandwidth enabling fast data transfers between processors and memory devices. |  Non-volatile storage with potential data transfer rate limitations due to underlying interface and technology.
Memory
Coherency |  Provides memory-coherent links, ensuring consistent data across different memory tiers. |  Requires additional mechanisms for memory coherency, except with local RAM, when integrated with other memory technologies.
Heterogeneous Memory
Integration |  Allows seamless integration of various memory technologies within a unified architecture. |  Effective for extending memory capacity, but integration may require additional considerations due to unique characteristics.
Memory Pooling
and Sharing |  Facilitates memory pooling and sharing, enabling efficient resource utilization and dynamic allocation based on workload requirements. |  Extends memory capacity, but inherent flexibility in memory sharing and pooling may be limited.
Industry
Standardization |  Open industry standard supported by major technology players, ensuring compatibility, interoperability, and broader adoption. |  Solutions may vary, potentially leading to compatibility challenges and limited integration options.
Scalability |  Architecture designed for scalability with multiple lanes and protocols, catering to evolving data center needs. |  Scalability may be constrained by underlying technology characteristics, such as DIMM count and RAM/NVRAM tradeoff.
Relevance to HPC |  Higher bandwidth, memory coherency, and memory pooling capabilities enhance HPC workload performance. Standardization compatibility in heterogeneous environments and scalability cater to evolving demands. |  Offers non-volatility but is constrained by limitations in bandwidth, coherency management, and scalability, affecting its applicability to complex HPC memory needs.
Table 2. General comparison between common aspects of CXL memory and NVRAM for
disaggregated HPC.

###  1.4. Contribution

In this work, based on actual physical experiments with multi-NUMA nodes and
multi-core high-performance SOTA hardware (subsection 2.1) and CXL-remote
memory (subsection 2.2), we claim that it is not only possible to exemplify
most persistent memory modules characteristics (as described in Table 1) with
CXL memory fully but also that in terms of performance, we can achieve much
better bandwidth than previously published Optane DCPMM ones (such in
(Izraelevitz et al., 2016), which, for a single Optane DCPMM, discovers that
its max read bandwidth is 6.6 GB/s, whereas its max write bandwidth is 2.3
GB/s). In fact, we show (Figure 9) that by approaching our CXL-DDR4 memory
module – much cheaper than DDR5 – we achieve comparable results to the local
DDR4 module and exhibit performance degradation of only about 60% in bandwidth
in comparison to local DDR5 module access (noting that DDR4 has about 50%
bandwidth of DDR5). Our tests were made in multiple configurations (subsection
3.2) in relation to the memory distance from the working threads using the
well-known STREAM benchmark (subsection 3.1).

In order to demonstrate the non-volatile properties of the memory as PMem, the
CXL memory was located outside of the node, in an FPGA device (subsection
2.2), potentially backed by battery, like previous battery-backed DIMMs. As
many nodes can approach the device, the battery-backed consideration is no
longer considered by us as a major overhead since it will be applied only once
for the memory modules and not in each compute node.

Moreover, besides the cache-coherent performance benchmarks with STREAM
(McCalpin, 1995a, 2007), we retested the memory bandwidth in an equivalent of
the App-Direct approach with a modified STREAM application, named STREAM-PMem
(Fagerheim, 2021) when all of the main arrays were allocated as a PMDK’s
pmemobj and manipulated accordingly (Scargall and Scargall, 2020). pmemobj
provides an assurance that the condition of objects will remain internally
consistent regardless of when the program concludes. Additionally, it offers a
transaction function that can encompass various modifications made to
persistent objects. This function ensures that either all of the modifications
are successfully applied or none of them take effect.

We stress that as our CXL memory module is located outside of the node and can
be backed by a battery, the ability to transactionally and directly access the
memory, exactly as previously done with Optane DCPMM, while achieving even
better performances, is a key to our practical approach, which consider CXL
memory as a persistent memory for the future of disaggregated HPC.

Finally, we open-sourced the entire benchmarking methodology as an easy-to-use
and automated tool named STREAMer for future CXL memory device evaluations for
HPC purposes.

##  2\. Physical Experimental Setup

###  2.1. HPC hardware

Our HPC hardware experimental environment is based on 2 setups:

  1. (1)

Node equipped with two Intel $4^{th}$ generation Xeon (Sapphire Rapids)
processors with a base frequency of 2.1GHz and 48 cores each, plus Hyper-
Threading. BIOS was updated to support only 10 cores per socket. Each
processor has one memory DIMM (64GB DDR5 4800MHz DIMM). The system is equipped
with a CXL prototype device, implemented as DDR4 memory on a PCIe-attached
FPGA (see Figure 4).

  2. (2)

Node equipped with two Intel Xeon Gold 5215 processors with a base frequency
of 2.5GHz and 10 cores each, plus Hyper-Threading. Each processor has total
96GB DRAM in 6 channels, 16GB DDR4 2666MHz DIMM per channel. (see Figure 4).

Figure 2. Setup #1 with DDR5 on-node memory and DDR4 CXL-attached memory.

Figure 3. Setup #2 with DDR4 on-node memory.

Figure 4. Overview of CXL IP for Intel® Agilex® 7 I-Series FPGA (Intel,
2023b), demonstrated in Figure 4 (setup #1).

###  2.2. CXL prototype

We provide an in-depth overview of our CXL prototype’s implementation on an
FPGA card (Intel, 2023c). Figure 4 and Figure 4 grant a more detailed view
into the implementation of our CXL memory pool on the FPGA card (while Figure
4 show the reference system, without any CXL attachment, with DDR4 main
memory). The prototype aims to harness the capabilities of the R-Tile Intel
FPGA IP for CXL, encompassing critical functionalities for CXL link
establishment and transaction layer management. This comprehensive solution
facilitates the construction of FPGA-based CXL 1.1/2.0 compliant endpoint
designs, including Type 1, Type 2, and Type 3 configurations. It’s built upon
a previously proven prototype, with necessary slight modifications for PMem
activity (Lee et al., 2023).

The architecture of our CXL implementation revolves around a synergistic
pairing of protocol Soft IP within the FPGA main fabric die and the Hard IP
counterpart, the R-Tile. This cohesive arrangement ensures effective
management of CXL link functions, which are pivotal for seamless
communication. Specifically, the R-Tile interfaces with a CPU host via a PCIe
Gen5x16 connection, delivering a theoretical bandwidth of up to 64GB/s. As a
key facet of our implementation, the FPGA device is duly enumerated as a CXL
endpoint within the host system.

Complementing this link management, the Soft IP assumes the mantle of
transaction layer functions, vital for the successful execution of different
CXL endpoint types. For Type 3 configurations, the CXL.mem transaction layer
adeptly handles incoming CXL.mem requests originating from the CPU host. It
orchestrates the generation of host-managed device memory (HDM) requests
directed toward an HDM subsystem. Simultaneously, the CXL.io transaction layer
undertakes the responsibility of processing CXL.io requests. These requests
encompass both configuration and memory space inquiries initiated from the CPU
host, seamlessly forwarding them to their designated control and status
registers. A noteworthy augmentation is the User Streaming Interface, offering
a conduit for custom CXL.io features that can be seamlessly integrated into
the user design.

Integral to our FPGA card is the inclusion of two onboard DDR4 memory modules,
each boasting a capacity of 8GB and operating at a clock frequency of 1333
MHz. These modules are accessible from the host system as conventional memory
resources. It is imperative to highlight a distinctive attribute of this
prototype configuration: the CXL link facilitates access to an identical
memory volume. In essence, this means that the same far memory segment can be
made available to two distinct NUMA nodes, eliminating any concerns of address
overlap. However, due to the absence of a unified cache-coherent domain, the
onus of maintaining coherency between the two NUMA nodes assigned to the
shared far memory rests with the applications leveraging this configuration.

Notably, the bandwidth attainable from this prototype configuration is subject
to current implementation constraints and does not reflect an intrinsic
limitation of the CXL standard. Potential avenues for enhancing bandwidth
include several considerations. First, transitioning to a higher-speed FPGA,
supporting DDR4 speeds of 3200 Mbps or even embracing the capabilities of DDR5
at 5600 Mbps, could appreciably enhance throughput. Additionally, scaling the
resources allocated to the CXL IP by increasing the number of slices is a
viable strategy. Furthermore, expanding the FPGA’s capacity to accommodate
multiple independent DDR channels, possibly transitioning from one channel to
four, holds promise in augmenting the prototype’s bandwidth potential.

In our discussion, the fact that the CXL memory device is DDR4 and not DDR5 is
key, as usually, PMem is slower and cheaper than the main memory. By using
DDR4 CXL memory and not DDR5, while main memory is DDR5, we keep on this
important relation.

##  3\. Performance Evaluation

###  3.1. STREAM and STREAM-PMem Benchmarks

The STREAM benchmark (McCalpin, 1995b) is a synthetic benchmark program that
measures sustainable memory bandwidth for simple vector kernels in high-
performance computers. STREAM was developed as a proxy for the basic
computational kernels in scientific computations (McCalpin, 1995c) and
includes Copy, Scale, Sum, and Triad kernels. STREAM has a dedicated version
to benchmark PMem modules by allocating and accessing PMem via PMDK (STREAM-
PMem (Fagerheim, 2021)).

The excerpt presented in 1 constitutes a portion of the initial codebase that
has since been extracted from the current version of the code.

Listing 1: Original STREAM benchmark code at line 175-181.

⬇

1#ifndef STREAM_TYPE

2#define STREAM_TYPE double

3#endif

4static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET],

5 b[STREAM_ARRAY_SIZE+OFFSET],

6 c[STREAM_ARRAY_SIZE+OFFSET];

The content represented in 1 has been substituted in STREAM-PMem (Fagerheim,
2021) with the code demonstrated in 2. The code commences by accessing the
memory pool. Furthermore, a function named initiate is employed to initialize
the three arrays. Following this initialization, the code proceeds to execute
the remaining segments of the STREAM benchmark code, mirroring the structure
of the original STREAM benchmark code.

Listing 2: Code that has replaced original code.

⬇

1PMEMobjpool *pop;

2POBJ_LAYOUT_BEGIN(array);

3POBJ_LAYOUT_TOID(array, double);

4POBJ_LAYOUT_END(array); //Declearing the arrays

5TOID(double) a, b, c;

6void initiate() { //Initiating the arrays.

7 POBJ_ALLOC(pop, &a, double, (STREAM_ARRAY_SIZE+OFFSET)*sizeof(STREAM_TYPE),
NULL, NULL); //Same for b and c.

8int main(){

9 const char path[] = ".../pool.obj";

10 pop = pmemobj_create(path, LAYOUT_NAME, 10737418240, 0666);

11 if (pop == NULL)

12 pop = pmemobj_open(path, LAYOUT_NAME);

13 if (pop == NULL) {

14 perror(path);

15 exit(1); }

16 initiate();

17 //The rest of the STREAM benchmark after this.

18 }

In this work, we employ STREAM in those two versions to showcase the shift
from PMem to CXL. Throughout this demonstration, we illustrate how programs
designed for PMem can seamlessly operate on CXL-enabled devices. Furthermore,
we provide performance assessments to anticipate the impact of CXL on
performance in relation to local RAM (DDR4 and DDR5) and local PMem-like
devices (emulation of remote sockets either for memory expansion or as a
direct access device, as done in (Foyer et al., 2023; Bergstrom, 2011)).

In contrast to previous research that primarily emphasizes demonstrating the
use of CXL memory for in-memory database queries or file system operations
(Ahn et al., 2022; Lee et al., 2023), STREAM memory access involves accessing
and manipulating large arrays, making it particularly applicable and
significant for scientific computations in HPC systems. Moreover, STREAM is
implemented with OpenMP threads, which is the common shared-memory paradigm in
scientific computing for parallelism (Dagum and Menon, 1998).

###  3.2. Test Configurations

The methodology of this work is to employ STREAM and STREAM-PMem in various
CPU and memory configurations, taking into account the availability of DRAM
and CXL memory available on the HPC setups, as will be described next. The
results presented in Figure 5, Figure 6, Figure 7, Figure 8 refer to STREAM
executions with 100M array elements for Scale, Add, Copy, and Triad operations
correspondingly. For each STREAM method, the results of our tests are
presented in 2 classes (and a total of 5 groups), divided conceptually for
unique comparisons. We sub-divide those 5 groups into two classes. The first
class (Class 1, (a)-(c)) refers to the equivalent of the App-Direct mode in
PMem in which we directly access the local or remote memory (either in the
alternative socket or in the CXL memory), and the second class (Class 2,
(a)-(b)) refers to the Memory Mode in PMem, in which we increase the available
memory using other CC-NUMA nodes:

Class 1 — App-Direct:

  1. (a)

Local memory access as PMem: Configurations within this group involve
accessing local memory (on-socket memory) in App-Direct mode (thus
benchmarking STREAM-PMem).

  2. (b)

Remote memory access as PMem: Configurations within this group involve
computing cores on a single socket that access remote memory in App-Direct
mode (thus benchmarking STREAM-PMem). The term ”remote memory” in this context
encompasses both CXL-attached memory and on-node memory accessed from the
alternative CPU socket (i.e., memory accessed through the UPI).

  3. (c)

Remote memory as PMem (thread affinity): Configurations within this group
involve computing cores in both sockets that access remote memory in App-
Direct mode (thus benchmarking STREAM-PMem) using two distinct thread affinity
methods: close and spread. The close method populates an entire socket first
and then adds cores from the second socket. The spread method, on the
opposite, adds cores alternately from both sockets.

Class 2 — Memory Mode:

  1. (a)

Remote CC-NUMA: Configurations within this group involve computing cores on a
single socket that access remote memory as CC-NUMA.

  2. (b)

Remote CC-NUMA (all cores): Configurations within this group involve cores on
both CPU sockets accessing remote memory as CC-NUMA. This includes
configurations where both sockets operate and access memory on one of them
since these workloads include remote accesses.

For better clarity, the data flow for each test configuration is demonstrated
in Figure 9. Each row in Figure 9 contains the data flow examinations of the
test groups of the two classes. Thus, in each of our test groups, for each of
the STREAM operations (Figures 5, 6, 7, 8), the way to understand each trend,
and its correspondence to the relevant dataflow, is given in the trend itself
by a combination of three: symbol, color and memory annotation. The symbol is
used to distinguish between accessing on-node DDR4 (▲), on-node DDR5 (⚫) or
CXL-attached DDR4 ($\times$). The color implies the active compute cores ––
either in socket0, socket1, or both. The annotations pmem$\\#\\{0,1,2\\}$ or
numa$\\#\\{0,1,2\\}$ accompanying each trend provide an explanation of the
accesses memory location: 0 for socket0; 1 for socket1; and 2 for CXL memory.
numa signifies STREAM accessing memory as NUMA memory expansion, while pmem
represents STREAM-PMem accessing memory using PMDK.

##  4\. Results and Analysis

(a) Class 1.a: Local memory access as PMem

(b) Class 1.b: Remote memory access as PMem

(c) Class 1.c: Remote memory as PMem (thread affinity)

(d) Class 2.a: Remote CC-NUMA

(e) Class 2.b: Remote CC-NUMA (all cores)

Figure 5. SCALE — Various STREAM test configurations. Refer to Section 3.2 for
definition of test groups 1.(a), 1.(b), 1.(c), 2.(a), 2.(b) and legend
clarifications.

(a) Class 1.a: Local memory access as PMem

(b) Class 1.b: Remote memory access as PMem

(c) Class 1.c: Remote memory as PMem (thread affinity)

(d) Class 2.a: Remote CC-NUMA

(e) Class 2.b: Remote CC-NUMA (all cores)

Figure 6. ADD — Various STREAM test configurations. Refer to Section 3.2 for
definition of test groups 1.(a), 1.(b), 1.(c), 2.(a), 2.(b) and legend
clarifications.

(a) Class 1.a: Local memory access as PMem

(b) Class 1.b: Remote memory access as PMem

(c) Class 1.c: Remote memory as PMem (thread affinity)

(d) Class 2.a: Remote CC-NUMA

(e) Class 2.b: Remote CC-NUMA (all cores)

Figure 7. COPY — Various STREAM test configurations. Refer to Section 3.2 for
definition of test groups 1.(a), 1.(b), 1.(c), 2.(a), 2.(b) and legend
clarifications.

(a) Class 1.a: Local memory access as PMem

(b) Class 1.b: Remote memory access as PMem

(c) Class 1.c: Remote memory as PMem (thread affinity)

(d) Class 2.a: Remote CC-NUMA

(e) Class 2.b: Remote CC-NUMA (all cores)

Figure 8. TRIAD — Various STREAM test configurations. Refer to Section 3.2 for
definition of test groups 1.(a), 1.(b), 1.(c), 2.(a), 2.(b) and legend
clarifications.

(Class 1.a)

(Class 1.b)

(Class 1.c)

(Class 2.a)

(Class 2.b)

Figure 9. Data flow demonstrations for the two classes (App Direct and Memory
Mode). Each test group is evaluated in corresponding to subfigures of Figure
5, Figure 6, Figure 7, Figure 8. Each row corresponds to a test group.

Figure 5, Figure 6, Figure 7 and Figure 8 present STREAM results for the
Scale, Add, Copy, and Triad operations correspondingly, and for the test
configurations defined in subsection 3.2 as will be described next. 5(a),
6(a), 7(a) and 8(a) through 5(e), 6(e), 7(e) and 8(e) present results for
Class 1.(a) group though Class 2.(b) group correspondingly.

The results explain the costs associated with memory access across varied
configurations distinguished by parameters such as memory type (on-node or
CXL-attached), memory placement (local to the socket, on the alternate CPU
socket, or the CXL-attached memory), access mode (App-Direct vs. Memory Mode),
and thread affinity (Close or Spread).

Next, we will examine and analyze the achieved results in relation to the
configuration classes and groups presented in subsection 3.2:

Class 1 — App-Direct:

  1. (a)

Local memory access as PMem: It is possible to observe that among all of the
STREAM actions, the App-Direct access using PMDK to the local DDR5 memory is
saturated around 20-22 GB/s. This test is a reference for the remote access
presented in the following group, either to a nearby remote socket or to the
CXL memory (with PMDK).

  2. (b)

Remote memory access as PMem: App-Direct access to the emulated remote PMem
(DDR5 on the alternate socket) results in a decrease of 30% ($\sim$15 GB/s) of
performance on average for all STREAM operations, in comparison to local App-
Direct access. In the case of App-Direct access to remote CXL memory (DDR4),
we experience 50% decrease in performance in comparison to the emulated PMem
on DDR5. However, we note that DDR5 inherently has about 50% higher bandwidth
than DDR4, meaning that the rest of the overhead – about 2-3 GB/s loss in
bandwidth – can be attributed to the CXL fabric.

  3. (c)

Remote memory as PMem (thread affinity): As observed in previous groups, local
App-Direct accesses result in higher bandwidth than remote accesses. In the
case of close thread affinity, after populating the entire socket, adding
remote accesses of compute cores to the workload negatively impacts the
bandwidth, whereas adding local accesses contributes positively. With spread
affinity, the performance demonstrates an average between local and remote
accesses due to the inclusion of alternating accesses. Eventually, when both
sockets are operating with the entire core count, the results converge for on-
node DDR5 and remote CXL memory, separately. Notably, accessing remote CXL
memory (DDR4) leads to a 50% observed degradation compared to on-node DDR5.

Class 2 — Memory Mode:

  1. (a)

Remote CC-NUMA: Evaluating DDR4 CC-NUMA, whether on the remote socket or CXL-
attached memory, yields comparable figures (with average gaps of up to 2-5
GB/s). However, following a small number of threads, a slight advantage is
observed for accessing CXL memory. This advantage can be attributed to the
larger caches in Setup #1 utilizing CXL (Shappire Rapids), as opposed to Setup
#2 (Xeon Gold) with on-node DDR4 (subsection 2.1). This indicates that the CXL
fabric overhead is constrained by the performance reduction when transitioning
back from Sapphire Rapids to Xeon Gold. Moreover, the gap between the CC-NUMA
to DDR5 and DDR4 (on-node or CXL-attached) stands on a factor of two, as
already observed in 1.(b) and 1.(c). In addition, in comparison to the results
of the App-Direct tests in 1.(b), it is observed that PMDK overheads over CC-
NUMA are 10%-15% (in all STREAM methods).

  2. (b)

Remote CC-NUMA (all cores): The observed gap between DDR4 and DDR5 repeats
here. Moreover, accessing on-node DDR4 using all cores converges to the same
results as accessing DDR4 CXL memory.

To conclude, the analysis reveals that direct access to local DDR5 memory
using PMDK saturates at 20-22 GB/s, while direct remote access to emulated
PMem and CXL memory results in 30% and 50% performance decreases,
respectively, with about 2-3 GB/s bandwidth loss attributed to CXL fabric. In
terms of memory expansion, accessing remote DDR4 CC-NUMA and DDR4 CXL-attached
memory exhibit similar performance gaps of 2-3 GB/s, while DDR5 CC-NUMA
maintains an advantage gap of a factor of 1.5 compared to DDR4, and on-node
DDR4 access converges with off-node DDR4 access under varying thread
affinities.

##  5\. Conclusions

In this study, we embarked on a comprehensive exploration of the potential of
CXL memory as a promising candidate for serving as a persistent memory
solution in the context of disaggregated HPC systems. By conducting physical
experiments on state-of-the-art multi-NUMA nodes equipped with high-
performance processors and CXL-attached memory prototypes, we have provided
empirical evidence that supports the feasibility of using CXL memory to
exhibit all the characteristics of persistent memory modules while achieving
impressive performance metrics.

Our findings demonstrate that CXL memory has the capability to outperform
previously published benchmarks for Optane DCPMM in terms of bandwidth.
Specifically, by employing a CXL-DDR4 memory module, which is a cost-effective
alternative to DDR5 memory, we achieved bandwidth results comparable to local
DDR4 memory configurations, with only a marginal decrease of around 50% when
compared to local DDR5 memory configurations. These results, attained across
various memory distances from the working threads, were assessed through the
well-established STREAM benchmark underscoring the reliability and versatility
of CXL memory in the HPC landscape.

The shift from PMem to CXL was not only demonstrated through performance
evaluations but was also highlighted through the modification of the STREAM
application into STREAM-PMem. We showcased the seamless transition of
programming models from PMem to CXL, leveraging the PMDK’s pmemobj to ensure
transactional integrity and consistency of operations on persistent objects.
Furthermore, the ability to access CXL memory directly and transactionally,
akin to Optane DCPMM, was underscored as a key advantage for practical
implementation.

Our study extends beyond theoretical considerations by implementing a
practical CXL prototype on an FPGA card. This prototype embodies CXL 1.1/2.0
compliant endpoint designs, demonstrating effective link establishment and
transaction layer management through a combination of Soft and Hard IP
components. The prototype’s performance, while constrained by current
implementation limitations, stands as a testament to the extensibility of this
solution and offers a blueprint for potential enhancements, including higher-
speed FPGAs and increased resources.

##  6\. Future Work

While this study provides valuable insights into the feasibility and potential
benefits of using CXL-enabled memory in HPC systems, several avenues for
future research and exploration remain:

  * •

Scalability and Performance Optimization: Further investigation is warranted
to explore the scalability of CXL-enabled memory in larger HPC clusters, with
more than one node accessing the CXL memory. Optimizing communication
protocols and memory access patterns can help maximize memory disaggregation
benefits.

  * •

Hybrid Architectures: Combining different memory technologies, such as DDR,
PMem, and CXL memory, in a hybrid memory architecture could offer a balanced
solution that leverages the strengths of each technology. Also, the CXL memory
could also use DDR5 and even Optane DCPMM, and as such, revisiting the results
with those CXL memories would be beneficial.

  * •

Real-World Applications: Extending the evaluation to real-world HPC
applications beyond benchmarks can provide a clearer understanding of how CXL
memory performs in practical scenarios.

  * •

Fault Tolerance and Reliability: Investigating fault tolerance mechanisms and
data reliability in the context of CXL-enabled memory is crucial, especially
in large-scale distributed environments. Specifically, code systems that have
previously been built upon PMDK and Optane DCPMM presence in the HPC system.

###### Acknowledgements.

This work was supported by Pazy grant 226/20, the Lynn and William Frankel
Center for Computer Science, and Intel Corporation (oneAPI Center of
Excellence program). Computational support was provided by the NegevHPC
project (Park, 2019) and Intel Developer Cloud (Intel, 2023a). The authors
would like to thank Gabi Dadush, Israel Hen, and Emil Malka for their hardware
support on NegevHPC. The authors also want to thank Jay Mahalingam and Guy
Tamir of Intel for their great help in forming this collaboration.

## References

  * (1)
  * Ahn et al. (2022) Minseon Ahn, Andrew Chang, Donghun Lee, Jongmin Gim, Jungmin Kim, Jaemin Jung, Oliver Rebholz, Vincent Pham, Krishna Malladi, and Yang Seok Ki. 2022\.  Enabling CXL memory expansion for in-memory database management systems. In _Proceedings of the 18th International Workshop on Data Management on New Hardware_. 1–5.
  * Al Maruf and Chowdhury (2023) Hasan Al Maruf and Mosharaf Chowdhury. 2023.  Memory Disaggregation: Open Challenges in the Era of CXL. In _Workshop on HotTopics in System Infrastructure_ , Vol. 18.
  * AsteraLabs (2022) AsteraLabs. 2022\.  _CXL Memory Accelerators_.  https://www.asteralabs.com/products/cxl-memory-platform/
  * Baldassin et al. (2021) Alexandro Baldassin, Joao Barreto, Daniel Castro, and Paolo Romano. 2021.  Persistent memory: A survey of programming support and implementations.  _ACM Computing Surveys (CSUR)_ 54, 7 (2021), 1–37.
  * Benson et al. (2023) Lawrence Benson, Marcel Weisgut, and Tilmann Rabl. 2023\.  What We Can Learn from Persistent Memory for CXL.  _BTW 2023_ (2023).
  * Bergstrom (2011) Lars Bergstrom. 2011\.  Measuring NUMA effects with the STREAM benchmark.  _arXiv preprint arXiv:1103.3225_ (2011).
  * Bernholdt et al. (2020) David E Bernholdt, Swen Boehm, George Bosilca, Manjunath Gorentla Venkata, Ryan E Grant, Thomas Naughton, Howard P Pritchard, Martin Schulz, and Geoffroy R Vallee. 2020\.  A survey of MPI usage in the US exascale computing project.  _Concurrency and Computation: Practice and Experience_ 32, 3 (2020), e4851.
  * Bustos et al. (2023) Andrés Bustos, Antonio Juan Rubio-Montero, Roberto Méndez, Sergio Rivera, Francisco González, Xandra Campo, Hernán Asorey, and Rafael Mayo-García. 2023\.  Response of HPC hardware to neutron radiation at the dawn of exascale.  _The Journal of Supercomputing_ (2023), 1–22.
  * Dagum and Menon (1998) Leonardo Dagum and Ramesh Menon. 1998.  OpenMP: An industry-standard API for shared-memory programming.  _Computing in Science & Engineering_ 1 (1998), 46–55.
  * Ding et al. (2023) Nan Ding, Pieter Maris, Hai Ah Nam, Taylor Groves, Muaaz Gul Awan, LeAnn Lindsey, Christopher Daley, Oguz Selvitopi, Leonid Oliker, and Nicholas Wright. 2023\.  Evaluating the Potential of Disaggregated Memory Systems for HPC applications.  _arXiv preprint arXiv:2306.04014_ (2023).
  * Evans et al. (2022) Thomas M Evans, Andrew Siegel, Erik W Draeger, Jack Deslippe, Marianne M Francois, Timothy C Germann, William E Hart, and Daniel F Martin. 2022\.  A survey of software implementations used by application codes in the Exascale Computing Project.  _The International Journal of High Performance Computing Applications_ 36, 1 (2022), 5–12.
  * Fagerheim (2021) Svein Gunnar Fagerheim. 2021\.  _Benchmarking Persistent Memory with Respect to Performance and Programmability_.  Master’s thesis.
  * Foyer et al. (2023) Clément Foyer, Brice Goglin, and Andrès Rubio Proaño. 2023\.  A survey of software techniques to emulate heterogeneous memory systems in high-performance computing.  _Parallel Comput._ (2023), 103023\.
  * Fridman et al. (2022) Yehonatan Fridman, Yaniv Snir, Harel Levin, Danny Hendler, Hagit Attiya, and Gal Oren. 2022\.  Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. In _2022 IEEE/ACM 12th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS)_. IEEE, 11–23.
  * Fridman et al. (2021) Yehonatan Fridman, Yaniv Snir, Matan Rusanovsky, Kfir Zvi, Harel Levin, Danny Hendler, Hagit Attiya, and Gal Oren. 2021\.  Assessing the use cases of persistent memory in high-performance scientific computing. In _2021 IEEE/ACM 11th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS)_. IEEE, 11–20.
  * Geyer et al. (2023) Andreas Geyer, Daniel Ritter, Dong Hun Lee, Minseon Ahn, Johannes Pietrzyk, Alexander Krause, Dirk Habich, and Wolfgang Lehner. 2023\.  Working with Disaggregated Systems. What are the Challenges and Opportunities of RDMA and CXL?  _BTW 2023_ (2023).
  * Gropp (2012) William Gropp. 2012\.  MPI 3 and beyond: why MPI is successful and what challenges it faces. In _European MPI Users’ Group Meeting_. Springer, 1–9.
  * Gugnani et al. (2020) Shashank Gugnani, Arjun Kashyap, and Xiaoyi Lu. 2020\.  Understanding the idiosyncrasies of real persistent memory.  _Proceedings of the VLDB Endowment_ 14, 4 (2020), 626–639.
  * Hady et al. (2017) Frank T Hady, Annie Foong, Bryan Veal, and Dan Williams. 2017\.  Platform storage performance with 3D XPoint technology.  _Proc. IEEE_ 105, 9 (2017), 1822–1833.
  * Handy and Coughlin (2023) Jim Handy and Tom Coughlin. 2023.  Optane’s Dead: Now What?  _Computer_ 56, 3 (2023), 125–130.
  * Hirofuchi and Takano (2020) Takahiro Hirofuchi and Ryousei Takano. 2020.  A prompt report on the performance of intel optane dc persistent memory module.  _IEICE TRANSACTIONS on Information and Systems_ 103, 5 (2020), 1168–1172.
  * Intel (2022) Intel. 2022.  Migration from Direct-Attached Intel Optane Persistent Memory to CXL-Attached Memory.  https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-11/optane-pmem-to-cxl-tech-brief.pdf.  [Online].
  * Intel (2023a) Intel. 2023a.  Intel Developer Cloud.  https://www.intel.com/content/www/us/en/developer/tools/devcloud/overview.html.  [Online].
  * Intel (2023b) Intel. 2023b.  Intel® FPGA Compute Express Link (CXL) IP.  https://www.intel.com/content/www/us/en/products/details/fpga/intellectual-property/interface-protocols/cxl-ip.html.  [Online].
  * Intel (2023c) Intel. 2023c.  Intel® FPGA Compute Express Link (CXL) IP.  https://www.intel.com/content/www/us/en/products/details/fpga/intellectual-property/interface-protocols/cxl-ip.html.  [Online].
  * Izraelevitz et al. (2016) Joseph Izraelevitz, Hammurabi Mendes, and Michael L Scott. 2016\.  Linearizability of persistent memory objects under a full-system-crash failure model. In _International Symposium on Distributed Computing_. Springer, 313–327.
  * Izraelevitz et al. (2019) Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R Dulloor, et al. 2019\.  Basic performance measurements of the intel optane DC persistent memory module.  _arXiv preprint arXiv:1903.05714_ (2019).
  * Jun et al. (2017) Hongshin Jun, Jinhee Cho, Kangseol Lee, Ho-Young Son, Kwiwook Kim, Hanho Jin, and Keith Kim. 2017.  Hbm (high bandwidth memory) dram technology and architecture. In _2017 IEEE International Memory Workshop (IMW)_. IEEE, 1–4.
  * Jun et al. (2016) Hongshin Jun, Sangkyun Nam, Hanho Jin, Jong-Chern Lee, Yong Jae Park, and Jae Jin Lee. 2016\.  High-bandwidth memory (HBM) test challenges and solutions.  _IEEE Design & Test_ 34, 1 (2016), 16–25.
  * Kateja et al. (2017) Rajat Kateja, Anirudh Badam, Sriram Govindan, Bikash Sharma, and Greg Ganger. 2017.  Viyojit: Decoupling battery and DRAM capacities for battery-backed DRAM.  _ACM SIGARCH Computer Architecture News_ 45, 2 (2017), 613–626.
  * Khan et al. (2020) Awais Khan, Hyogi Sim, Sudharshan S Vazhkudai, Jinsuk Ma, Myeong-Hoon Oh, and Youngjae Kim. 2020\.  Persistent memory object storage and indexing for scientific computing. In _2020 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)_. IEEE, 1–9.
  * Kogge and Dally (2022) Peter M Kogge and William J Dally. 2022.  Frontier vs the Exascale Report: Why so long? and Are We Really There Yet?. In _2022 IEEE/ACM International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)_. IEEE, 26–35.
  * Lee et al. (2010) Benjamin C Lee, Engin Ipek, Onur Mutlu, and Doug Burger. 2010\.  Phase change memory architecture and the quest for scalability.  _Commun. ACM_ 53, 7 (2010), 99–106.
  * Lee et al. (2023) Donghun Lee, Thomas Willhalm, Minseon Ahn, Suprasad Mutalik Desai, Daniel Booss, Navneet Singh, Daniel Ritter, Jungmin Kim, and Oliver Rebholz. 2023.  Elastic Use of Far Memory for In-Memory Database Management Systems. In _Proceedings of the 19th International Workshop on Data Management on New Hardware_. 35–43.
  * Liu et al. (2003) Jiuxing Liu, Jiesheng Wu, Sushmitha P Kini, Pete Wyckoff, and Dhabaleswar K Panda. 2003.  High performance RDMA-based MPI implementation over InfiniBand. In _Proceedings of the 17th annual international conference on Supercomputing_. 295–304.
  * Liu (2023) Ming Liu. 2023.  Fabric-Centric Computing. In _Proceedings of the 19th Workshop on Hot Topics in Operating Systems_. 118–126.
  * Lockwood et al. (2023) Glenn K Lockwood, Damian Hazen, Quincey Koziol, R Shane Canon, Katie Antypas, Jan Balewski, Nicholas Balthaser, Wahid Bhimji, James Botts, Jeff Broughton, et al. 2023\.  Storage 2020: A vision for the future of hpc storage.  (2023).
  * Logan et al. (2023) Luke Logan, Jay Lofstead, Xian-He Sun, and Anthony Kougkas. 2023\.  An Evaluation of DAOS for Simulation and Deep Learning HPC Workloads. In _Proceedings of the 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems_. 9–16.
  * Malladi et al. (2016) Krishna T Malladi, Manu Awasthi, and Hongzhong Zheng. 2016\.  DRAMPersist: Making DRAM Systems Persistent. In _Proceedings of the Second International Symposium on Memory Systems_. 94–95.
  * McCalpin (2007) John D. McCalpin. 1991-2007.  _STREAM: Sustainable Memory Bandwidth in High Performance Computers_.  Technical Report. University of Virginia, Charlottesville, Virginia.  http://www.cs.virginia.edu/stream/ A continually updated technical report. http://www.cs.virginia.edu/stream/.
  * McCalpin (1995a) John D. McCalpin. 1995a.  Memory Bandwidth and Machine Balance in Current High Performance Computers.  _IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter_ (Dec. 1995), 19–25.
  * McCalpin (1995b) John D. McCalpin. 1995b.  STREAM Benchmark.  https://www.cs.virginia.edu/stream/.  [Online].
  * McCalpin (1995c) John D McCalpin. 1995c.  Stream benchmark.  _Link: www. cs. virginia. edu/stream/ref. html# what_ 22, 7 (1995).
  * McCalpin (2023) John D. McCalpin. 2023\.  Bandwidth Limits in the Intel Xeon Max (Sapphire Rapids with HBM) Processors. In _ISC 2023 IXPUG Workshop_. 1–24.
  * McKee (2004) Sally A McKee. 2004\.  Reflections on the memory wall. In _Proceedings of the 1st conference on Computing frontiers_. 162.
  * Michelogiannakis et al. (2023) George Michelogiannakis, Yehia Arafa, Brandon Cook, Liang Yuan Dai, Abdel Hameed Badawy, Madeleine Glick, Yuyang Wang, Keren Bergman, and John Shalf. 2023\.  Efficient Intra-Rack Resource Disaggregation for HPC Using Co-Packaged DWDM Photonics.  _arXiv preprint arXiv:2301.03592_ (2023).
  * Michelogiannakis et al. (2022) George Michelogiannakis, Benjamin Klenk, Brandon Cook, Min Yee Teh, Madeleine Glick, Larry Dennison, Keren Bergman, and John Shalf. 2022.  A case for intra-rack resource disaggregation in HPC.  _ACM Transactions on Architecture and Code Optimization (TACO)_ 19, 2 (2022), 1–26.
  * Mironov et al. (2019) Vladimir Mironov, Igor Chernykh, Igor Kulikov, Alexander Moskovsky, Evgeny Epifanovsky, and Andrey Kudryavtsev. 2019.  Performance evaluation of the intel optane dc memory with scientific benchmarks. In _2019 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)_. IEEE, 1–6.
  * Mutlu (2013) Onur Mutlu. 2013\.  Memory scaling: A systems architecture perspective. In _2013 5th IEEE International Memory Workshop_. IEEE, 21–25.
  * Narayanan and Hodson (2012) Dushyanth Narayanan and Orion Hodson. 2012.  Whole-system persistence. In _Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems_. 401–410.
  * Nathan Pham (2019) Kevin Huang Nathan Pham. 2019\.  Analyzing the Performance of Intel Optane Persistent Memory 200 Series in Memory Mode with Lenovo ThinkSystem Servers.
  * Newsroom (2022a) Samsung Newsroom. 2022a.  _Samsung Electronics Introduces Industry’s First 512GB CXL Memory Module_.  https://news.samsung.com/global/samsung-electronics-introduces-industrys-first-512gb-cxl-memory-module
  * Newsroom (2022b) SK Hynix Newsroom. 2022b.  _SK hynix Develops DDR5 DRAM CXLTM Memory to Expand the CXL Memory Ecosystem_.  https://news.skhynix.com/sk-hynix-develops-ddr5-dram-cxltm-memory-to-expand-the-cxl-memory-ecosystem/
  * Park (2019) Rotem Industrial Park. 2019\.  NegevHPC Project.  https://www.negevhpc.com.  [Online].
  * Patil et al. (2019) Onkar Patil, Latchesar Ionkov, Jason Lee, Frank Mueller, and Michael Lang. 2019.  Performance characterization of a dram-nvm hybrid memory architecture for hpc applications using intel optane dc persistent memory modules. In _Proceedings of the International Symposium on Memory Systems_. 288–303.
  * Peng et al. (2021) Ivy Peng, Ian Karlin, Maya Gokhale, Kathleen Shoga, Matthew Legendre, and Todd Gamblin. 2021\.  A holistic view of memory utilization on HPC systems: Current and future trends. In _The International Symposium on Memory Systems_. 1–11.
  * Peng et al. (2020) Ivy Peng, Roger Pearce, and Maya Gokhale. 2020.  On the memory underutilization: Exploring disaggregated memory on hpc systems. In _2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)_. IEEE, 183–190.
  * Radulovic et al. (2015) Milan Radulovic, Darko Zivanovic, Daniel Ruiz, Bronis R de Supinski, Sally A McKee, Petar Radojković, and Eduard Ayguadé. 2015\.  Another trip to the wall: How much will stacked dram benefit hpc?. In _Proceedings of the 2015 International Symposium on Memory Systems_. 31–36.
  * Rai and Talawar (2023) Sadhana Rai and Basavaraj Talawar. 2023.  Nonvolatile Memory Technologies: Characteristics, Deployment, and Research Challenges.  _Frontiers of Quality Electronic Design (QED) AI, IoT and Hardware Security_ (2023), 137–173.
  * Reed et al. (2022) Daniel Reed, Dennis Gannon, and Jack Dongarra. 2022\.  Reinventing high performance computing: challenges and opportunities.  _arXiv preprint arXiv:2203.02544_ (2022).
  * Ruan et al. (2023) Chaoyi Ruan, Yingqiang Zhang, Chao Bi, Xiaosong Ma, Hao Chen, Feifei Li, Xinjun Yang, Cheng Li, Ashraf Aboulnaga, and Yinlong Xu. 2023\.  Persistent Memory Disaggregation for Cloud-Native Relational Databases. In _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_. 498–512.
  * Rudoff (2017) Andy Rudoff. 2017\.  Persistent memory: The value to hpc and the challenges. In _Proceedings of the Workshop on memory centric programming for hpc_. 7–10.
  * Sainio et al. (2016) Arthur Sainio et al. 2016\.  NVDIMM: changes are here so what’s next.  _Memory Computing Summit_ (2016).
  * Scargall (2020) Steve Scargall. 2020\.  Programming Persistent Memory: A Comprehensive Guide for Developers.  (2020).
  * Scargall and Scargall (2020) Steve Scargall and Steve Scargall. 2020.  libpmemobj: A Native Transactional Object Store.  _Programming Persistent Memory: A Comprehensive Guide for Developers_ (2020), 81–109.
  * Sharma et al. (2023) Debendra Das Sharma, Robert Blankenship, and Daniel S Berger. 2023.  An Introduction to the Compute Express Link (CXL) Interconnect.  _arXiv preprint arXiv:2306.11227_ (2023).
  * Shipman et al. (2022) Galen M Shipman, Sriram Swaminarayan, Gary Grider, Jim Lujan, and R Joseph Zerr. 2022.  Early Performance Results on 4th Gen Intel (R) Xeon (R) Scalable Processors with DDR and Intel (R) Xeon (R) processors, codenamed Sapphire Rapids with HBM.  _arXiv preprint arXiv:2211.05712_ (2022).
  * Technology (2022) Montage Technology. 2022\.  _Montage Technology Delivers the World’s First CXL™ Memory eXpander Controller_.  https://www.montage-tech.com/Press_Releases/20220506
  * Tristian and Travis (2019) TB Tristian and L Travis. 2019.  Analyzing the performance of Intel Optane DC persistent memory in app direct mode in Lenovo ThinkSystem servers.
  * Wahlgren et al. (2022) Jacob Wahlgren, Maya Gokhale, and Ivy B Peng. 2022\.  Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems. In _2022 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)_. IEEE, 11–20.
  * Wang et al. (2023) Ying Wang, Wen-Qing Jia, De-Jun Jiang, and Jin Xiong. 2023\.  A Survey of Non-Volatile Main Memory File Systems.  _Journal of Computer Science and Technology_ 38, 2 (2023), 348–372.
  * Weiland et al. (2019) Michèle Weiland, Holger Brunst, Tiago Quintino, Nick Johnson, Olivier Iffrig, Simon Smart, Christian Herold, Antonino Bonanni, Adrian Jackson, and Mark Parsons. 2019.  An early evaluation of intel’s optane dc persistent memory module and its impact on high-performance scientific applications. In _Proceedings of the international conference for high performance computing, networking, storage and analysis_. 1–19.