# Torsion and Chern-Simons gravity in 4D space-times from a Geometrodynamical
four-form

Patrick Das Gupta<EMAIL_ADDRESS>Department of Physics and
Astrophysics, University of Delhi, Delhi - 110 007 (India)

###### Abstract

In Hermann Minkowski’s pioneering formulation of special relativity, the
space-time geometry in any inertial frame is described by the line-element
$ds^{2}=\eta_{\mu\nu}dx^{\mu}dx^{\nu}$. It is interesting to note that not
only the Minkowski metric $\eta_{\mu\nu}$ is invariant under proper Lorentz
transformations, the totally antisymmetric Levi-Civita tensor
$e_{\mu\nu\alpha\beta}$ too is.

In Einstein’s general relativity (GR), $\eta_{\mu\nu}$ of the flat space-time
gets generalized to a dynamical, space-time dependent metric tensor
$g_{\mu\nu}$ that characterizes a curved space-time geometry. In the present
study, it is put forward that the flat space-time Levi-Civita tensor gets
elevated to a dynamical four-form field $\tilde{w}$ in curved space-time
manifolds, i.e. $e_{\mu\nu\alpha\beta}\rightarrow
w_{\mu\nu\alpha\beta}(x)=\phi(x)\ e_{\mu\nu\alpha\beta}$ so that
$\tilde{w}={1\over{4!}}\ w_{\mu\nu\rho\sigma}\
\tilde{d}x^{\mu}\wedge\tilde{d}x^{\nu}\wedge\tilde{d}x^{\rho}\wedge\tilde{d}x^{\sigma}$.
It is shown that this geometrodynamical four-form field, extends GR by leading
naturally to a torsion in the theory as well as to a Chern-Simons gravity.
Furthermore, it is demonstrated that the scalar-density $\phi(x)$ associated
with the geometrodynamical four form $\tilde{w}$ may be used to construct a
generalized exterior derivative that converts a p-form density to a (p+1)-form
density of identical weight.

In order to subject the hypothesized four-form field $\tilde{w}$ to
observational evidence, we first argue that the associated scalar-density
$\phi(x)$ corresponds to an axion-like pseudo-scalar field in the Minkowski
space-time, and that it can also masquerade as dark matter. Thereafter, we
provide a simple semi-classical analysis in which a self-gravitating Bose-
Einstein condensate of such ultra-light pseudo-scalars leads to the formation
of a supermassive black hole. A brief analysis of propagation of weak
gravitational waves in the presence of $\tilde{w}$ is also considered in this
article.

##  I I. Introduction

By envisaging a four dimensional manifold’s space-time geometry be described
by a line-element, $ds^{2}=\eta_{\mu\nu}dx^{\mu}dx^{\nu}$, Hermann Minkowski
had overturned the idea of treating space and time disjointly. Furthermore,
the cause and effect relation, in the realm of classical physics, attained a
significant elucidation in the Minkowskian framework - infinitesimally
separated events (assuming a signature (+ - - -)) for which $ds^{2}<0$
($ds^{2}>0$) are causally disconnected (causally connected). Of course, the
underlying physics involved in it is that an effect essentially follows a
cause due to transfer of energy from the latter to the former, and that the
rate of flow of energy is limited by the speed $c$. What is usually not
stressed is that, because of this causality criteria, the imaginary numbers
entered physics in a fundamental manner even before the advent of quantum
mechanics, i.e. the proper distance $ds$ between any two causally disconnected
events is always an imaginary number.

Although space and time coordinates get mixed up while transiting from one
inertial frame to another, the Minkowski metric $\eta_{\mu\nu}$ itself remains
invariant. In fact, this is utilized in the Minkowskian framework to define an
arbitrary Lorentz transformation $x^{\mu}\rightarrow
x^{{}^{\prime}\mu}=\Lambda^{\mu}_{\ \nu}x^{\nu}$ by the condition,

$\eta_{\mu\nu}=\Lambda^{\alpha}_{\ \mu}\Lambda^{\beta}_{\ \nu}\
\eta_{\alpha\beta}$ (1)

It must be pointed out that there is another tensor in the flat space-time
that is invariant under proper Lorentz transformations: the totally
antisymmetric Levi-Civita tensor $\epsilon_{\mu\nu\rho\sigma}$. Since, under
$x^{\mu}\rightarrow\Lambda^{\mu}_{\ \nu}x^{\nu}$,

$\epsilon_{\mu\nu\alpha\beta}\ \ \ \ \rightarrow\ \ \Lambda^{\sigma}_{\
\mu}\Lambda^{\tau}_{\ \nu}\Lambda^{\gamma}_{\ \alpha}\Lambda^{\kappa}_{\
\beta}\ \epsilon_{\sigma\tau\gamma\kappa}$ (2) $=det(\Lambda)\
\epsilon_{\mu\nu\alpha\beta}\ \ ,$ (3)

and $det(\Lambda)=1$ for proper Lorentz Transformations, it is clear that the
flat space-time Levi-Civita tensor does not change under arbitrary boosts and
rotations of inertial frames.

Given this special status that the Minkowski metric and the Levi-Civita tensor
enjoy in the Minkowskian space-time, as well as the fact that $\eta_{\mu\nu}$
metamorphoses into a dynamical space-time metric $g_{\mu\nu}(x)$ in GR, raises
a pertinent point of of elevating $\epsilon_{\mu\nu\alpha\beta}$ into a
dynamical field. In the present article, we explore such a possibility by
positing a geometrodynamical field, $w_{\mu\nu\rho\sigma}(x)$, totally
antisymmetric in its indices, that is independent of the metric tensor and
that influences space-time physics. Then, the interesting feature that
$\eta_{\mu\nu}$ and $\epsilon_{\mu\nu\alpha\beta}$ both share in the flat
space-time becomes relevant also in arbitrary curved space-times pdg-ref-20 .

We propose in this article that this geometrodynamical field represents a new
physical degree of freedom that couples universally to all matter. In the
following section, we discuss some of the differential geometric aspects of
$w_{\mu\nu\rho\sigma}(x)$ that may be significant in the context of space-time
physics.

##  II II. Geometrodynamical four-form field and Torsion

Now, in any 4D space-time manifold, because of its complete antisymmetry,
$w_{\mu\nu\rho\sigma}(x)$ has only one algebraically independent component
and, hence, renders itself to be expressed as,

$w_{\mu\nu\rho\sigma}=\phi(x)\epsilon_{\mu\nu\rho\sigma}\ \ ,$ (4)

where $\phi(x)$ is a scalar-density of weight +1, with
$\phi(x)\rightarrow\phi^{\prime}(x^{\prime})=\phi(x)/J(x,x^{\prime})$ under a
general coordinate transformation $x^{\mu}\rightarrow x^{\prime\mu}$,
$J(x,x^{\prime})$ being the corresponding Jacobian.

In a coordinate basis, this four-form field may be expressed as,

$\tilde{w}={1\over{4!}}\ w_{\mu\nu\rho\sigma}(x)\
\tilde{d}x^{\mu}\wedge\tilde{d}x^{\nu}\wedge\tilde{d}x^{\rho}\wedge\tilde{d}x^{\sigma}$
(5) $={1\over{4!}}\ \phi(x)\ \epsilon_{\mu\nu\rho\sigma}\
\tilde{d}x^{\mu}\wedge\tilde{d}x^{\nu}\wedge\tilde{d}x^{\rho}\wedge\tilde{d}x^{\sigma}$
(6)

Eq.(6) reflects the equivalence between four-forms and 0-forms in the case of
four dimensional manifolds.

Since we demand that this new dynamical field be independent of the metric
tensor, instead of raising the indices of $w_{\mu\nu\rho\sigma}$ using the
metric tensor, the totally antisymmetric components $w^{\mu\nu\rho\sigma}$
corresponding to the four-form $\tilde{w}$ are obtained from the associated
p-vector w, so that one obtains the relation, $w^{\mu\nu\rho\sigma}\
w_{\mu\nu\rho\sigma}=-4!$ Schutz80 . This implies,

$w^{\mu\nu\rho\sigma}=\epsilon^{\mu\nu\rho\sigma}/\phi(x)\ \ .$ (7)

We assume throughout that $\phi=w_{0123}$ has no physical dimension. As the
canonical volume-form in GR is given by,

$\tilde{V}={1\over{4!}}\sqrt{-g}\ \ \epsilon_{\mu\nu\rho\sigma}\
\tilde{d}x^{\mu}\wedge\tilde{d}x^{\nu}\wedge\tilde{d}x^{\rho}\wedge\tilde{d}x^{\sigma}\
,$ (8)

our geometrodynamical four-form field may be expressed in terms of a scalar
field $\chi(x)\equiv\frac{\phi(x)}{\sqrt{-g}}$ entailing,

$\tilde{w}=\chi(x)\tilde{V}$ (9)

A fundamental four-form field that varies with space and time may be utilized
to extend GR by having a dynamical torsion field in the theory. Introducing an
asymmetric affine connection (e.g. see Hehl ; Hehl1 ),

$\bar{\Gamma}^{\mu}_{\alpha\beta}\equiv\Gamma^{\mu}_{\alpha\beta}-a_{T}\
g^{\mu\nu}g^{\gamma\lambda}\frac{\partial\chi}{\partial x^{\lambda}}\
w_{\nu\gamma\alpha\beta}=\Gamma^{\mu}_{\alpha\beta}-a_{T}\ g^{\mu\nu}\chi^{\
;\gamma}\ w_{\nu\gamma\alpha\beta}\ ,$ (10)

where $\Gamma^{\mu}_{\alpha\beta}$ is the standard Christoffel-Levi Civita
connection while $a_{T}$ is a dimensionless constant that is a measure of the
coupling between the metric tensor and the four-form, we define a metric
compatible torsion field,

$S^{\ \ \
\mu}_{\alpha\beta}=\frac{1}{2}\bigg{(}\bar{\Gamma}^{\mu}_{\alpha\beta}-\bar{\Gamma}^{\mu}_{\beta\alpha}\bigg{)}=-a_{T}\
g^{\mu\nu}\chi^{\ ;\gamma}\ w_{\nu\gamma\alpha\beta}=-a_{T}\sqrt{-g}\
g^{\mu\nu}g^{\gamma\sigma}\ \chi\ \frac{\partial\chi}{\partial x^{\sigma}}\
\epsilon_{\nu\gamma\alpha\beta}$ (11)

In the language of differential geometry, since torsion is a vector valued
2-form, the above expression for it can be seen as arising from an inner
contraction (e.g. see Mukunda ; Schutz80 ) of $\tilde{w}$ with the vector
field $\chi^{;\gamma}\frac{\partial}{\partial x^{\gamma}}$ followed by inner
contractions with four vector fields $-g^{\mu\nu}\frac{\partial}{\partial
x^{\nu}}$, $\mu=0,1,..3$.

Covariant derivatives of tensor fields are obtained in the usual way,

$D_{\nu}A^{\mu}=\partial_{\nu}A^{\mu}+\bar{\Gamma}^{\mu}_{\nu\beta}A^{\beta}=A^{\mu}_{\
;\nu}-a_{T}\ g^{\mu\alpha}\chi^{\ ;\gamma}w_{\alpha\gamma\nu\beta}A^{\beta}\ \
,$ (12)
$D_{\nu}B_{\mu}=\partial_{\nu}B_{\mu}-\bar{\Gamma}^{\beta}_{\mu\nu}B_{\beta}=B_{\mu;\
\nu}+a_{T}g^{\beta\sigma}\chi^{;\gamma}w_{\sigma\gamma\mu\nu}B_{\beta}\ \ ,$
(13)

and,

$D_{\nu}C_{\mu\tau}=\partial_{\nu}C_{\mu\tau}-\bar{\Gamma}^{\beta}_{\mu\nu}C_{\beta\tau}-\bar{\Gamma}^{\beta}_{\tau\nu}C_{\mu\beta}=C_{\mu\tau;\
\nu}+a_{T}g^{\beta\sigma}\chi^{;\gamma}w_{\sigma\gamma\mu\nu}C_{\beta\tau}+a_{T}g^{\beta\sigma}\chi^{;\gamma}w_{\sigma\gamma\tau\nu}C_{\mu\beta}\
\ ,$ (14)

with $C_{\mu\tau;\ \nu}$ being the covariant derivative of $C_{\mu\tau}$ in
standard GR when there is no torsion (i.e. $a_{T}=0$ case). From eq.(14), the
metric compatibility can be easily proved by showing that
$D_{\alpha}g_{\mu\nu}=0$. Similarly, eq.(10) entails that,

$\bar{\Gamma}^{\mu}_{\alpha\beta}A^{\alpha}A^{\beta}={\Gamma}^{\mu}_{\alpha\beta}A^{\alpha}A^{\beta}\
$

implying that the torsion in our case is autoparallel compatible. In our case,
the contortion tensor $K^{\ \ \mu}_{\alpha\beta}$ and $S_{\alpha\beta\gamma}$
have the expressions,

$K^{\ \ \ \mu}_{\alpha\beta}\equiv-S^{\ \ \ \mu}_{\alpha\beta}+S^{\
\mu}_{\beta\ \alpha}-S^{\mu}_{\ \ \alpha\beta}=-S^{\ \ \ \mu}_{\alpha\beta}$
(15)

and,

$S_{\alpha\beta\gamma}=g_{\gamma\mu}S^{\ \ \ \mu}_{\alpha\beta}=a_{T}\ \chi^{\
;\sigma}w_{\sigma\alpha\beta\gamma}=a_{T}\sqrt{-g}\ g^{\gamma\sigma}\chi\
\frac{\partial\chi}{\partial x^{\gamma}}\epsilon_{\sigma\alpha\beta\gamma}$
(16)

which is simply an inner contraction of $\tilde{w}$ with the vector field
$\chi^{\ ;\sigma}\frac{\partial}{\partial x^{\sigma}}$.

The important point to note is that the torsion vanishes if either $a_{T}=0$
or if the scalar field $\chi(x)$ that corresponds to the geometrodynamical
four form is independent of both space as well as time. For the sake of
simplicity, in the remaining portions of the article we will concentrate on
the case $a_{T}=0$ while assuming $\frac{\partial\chi}{\partial x^{\alpha}}$
to be non-vanishing. It is straightforward to extend the analysis to the
$a_{T}\neq 0$ case in order to study the effects of torsion. For the rest of
the sections, we have chosen a unit system in which $\hbar=c=1$ and the metric
signature: (+ - - -).

##  III III. Action, Field Equations and Chern-Simons extensions

###  III.1 (i) Dynamical equations

As $\phi$ is a scalar-density of weight +1,

$\phi_{\ ;\beta}=\phi_{\ ,\beta}-\Gamma^{\alpha}_{\alpha\beta}\ \phi\ \ ,$

where we follow the standard notation,

$f_{\ ,\beta}\equiv\partial_{\beta}f\ \ .$

Using the above equation along with the relation $w^{\mu\nu\rho\sigma}\
w_{\mu\nu\rho\sigma}=-\ 4!$, the covariant derivatives of the four-form field
and its corresponding p-vector are easily shown to be,

$w_{\mu\nu\alpha\beta\
;\lambda}=[(\ln\phi)_{,\lambda}-\Gamma^{\sigma}_{\sigma\lambda}]w_{\mu\nu\alpha\beta}\
\ ,$ (17)

and,

$w^{\mu\nu\alpha\beta}_{\ \
;\lambda}=-[(\ln\phi)_{,\lambda}-\Gamma^{\sigma}_{\sigma\lambda}]w^{\mu\nu\alpha\beta}\
\ ,$ (18)

respectively.

The action $\mathcal{A}$ for the standard matter, geometry as well as the
dynamical four-form that is invariant under general coordinate transformations
may be expressed as,

$\mathcal{A}=-{{m^{2}_{Pl}}\over{16\pi}}\int{R\sqrt{-g}d^{4}x}+\int{L\sqrt{-g}d^{4}x}\
+$

$\ \ \ \ \ \ +{{a_{1}}\over{4!}}\int{\phi\ w^{\mu\nu\alpha\beta}_{\ \
;\lambda}w_{\mu\nu\alpha\beta}\ ^{;\lambda}\
d^{4}x}+a_{2}\int{\phi(x)d^{4}x},$ $None$

where $m_{Pl}\equiv\sqrt{\hbar c/G}$ and $L$ are the Planck mass and the
Lagrangian density of the matter fields, respectively, with $a_{1}$ and
$a_{2}$ being real valued constants of the theory with dimensions
$(\mbox{mass})^{2}$ and $(\mbox{mass})^{4}$, respectively. The part of the
action in eq.(19) that pertains to the four-form is by no means unique. For
instance, a term $\propto\int{R\ \phi d^{4}x}$ could be added to the above
action, but we limit ourselves to gravitational minimal coupling, at present.
Later, we shall discuss supplementing the above action with Chern-Simons terms
induced by the geometrodynamical four-form.

By extremizing $\mathcal{A}$ with respect to $g_{\mu\nu}$ and $\phi$,
respectively, we obtain the following equations of motion,

$R_{\mu\nu}-{1\over{2}}g_{\mu\nu}R={{8\pi}\over{m^{2}_{Pl}}}[T_{\mu\nu}+\Theta_{\mu\nu}],$
$None$

$\chi^{\ ;\alpha}_{\
;\alpha}\equiv{1\over{\sqrt{-g}}}(\sqrt{-g}g^{\alpha\beta}\chi_{,\alpha})_{,\beta}={1\over{2}}\bigg{[}g^{\mu\nu}{{\chi_{,\mu}\chi_{,\nu}}\over{\chi}}+{a_{2}\over{a_{1}}}\chi\bigg{]},$
$None$

where the scalar field $\chi(x)$ is defined, as earlier, to be
$\chi\equiv{\phi\over{\sqrt{-g}}}$, while $T_{\mu\nu}$ and $\Theta_{\mu\nu}$
are the energy-momentum tensors for the standard matter and the
geometrodynamical four-form, respectively. The expression for the latter is
given by,

$\Theta_{\mu\nu}=2a_{1}[{{\chi_{,\mu}\chi_{,\nu}}\over{\chi}}-g_{\mu\nu}\chi^{\
;\alpha}_{\ ;\alpha}].$ $None$

When $\chi$ satisfies the equation of motion given by eq.(21), the above
energy-momentum tensor gets simplified to,

$\Theta_{\mu\nu}=2a_{1}\bigg{[}{{\chi_{,\mu}\chi_{,\nu}}\over{\psi}}-{{g_{\mu\nu}}\over{2}}\bigg{(}g^{\alpha\beta}{{\chi_{,\alpha}\chi_{,\beta}}\over{\chi}}+{a_{2}\over{a_{1}}}\chi\bigg{)}\bigg{]}.$
$None$

###  III.2 (ii) Chern-Simons extensions

In this sub-section, we construct (3+1)-dimensional Chern-Simons (CS) terms.
Our CS extensions of GR follows closely the seminal paper of Jackiw and Pi
Jackiw , except for a crucial difference: they had introduced an external
fixed four-vector $v_{\mu}$ instead of a dynamical $\phi_{;\mu}$ in their
formulation. Because of this, their model had a manifest violation of Lorentz
invariance. In our case, both general as well as Lorentz covariances are
maintained throughout pdg-ref-20 .

First, we consider a Chern-Simons (CS) term that couples electromagnetic field
to the four-form $\tilde{w}$,

$\mathcal{A}_{CS}=\mbox{J}\int{w^{\mu\nu\alpha\beta}F_{\mu\nu}A_{\alpha}\phi_{;\beta}\
d^{4}x}$

$\ \ \
=\mbox{J}\int{\epsilon^{\mu\nu\alpha\beta}F_{\mu\nu}A_{\alpha}(\ln\chi)_{,\beta}\
d^{4}x}$ $None$

where J is a dimensionless constant and $F_{\mu\nu}=A_{\nu,\mu}-A_{\mu,\nu}$.
The action expressed in eq.(24) is invariant under diffeomorphism as well as
gauge transformations, $A_{\mu}\rightarrow A_{\mu}+\partial_{\mu}\xi(x)$.

The upshot of adding $\mathcal{A}_{CS}$ (eq.(24)) to the standard action for
an electromagnetic field interacting with charge particles in the presence of
gravitation $\mathcal{A}_{EM}$ is that, upon extremizing the full action
$\mathcal{A}_{EM}+\mathcal{A}_{CS}$ with respect to $A_{\mu}$, one obtains the
following modified Einstein-Maxwell equation,

$F^{\alpha\beta}_{\ \ ;\beta}=-4\pi j^{\alpha}+8\pi\mbox{J}\
w^{\mu\nu\alpha\beta}F_{\mu\nu}\chi_{,\beta}$ $None$

with $j^{\alpha}$ being the 4-current density associated with charge
particles.

Use of the covariant derivative to the above equation with respect to
$x^{\alpha}$ leads to the charge density continuity equation that entails
conservation of electric charge $Q$,

$(\sqrt{-g}j^{\alpha})_{,\alpha}=0\ \ \Rightarrow\ \ Q=\int{j^{0}\sqrt{-g}\
d^{3}x}\ \ .$ $None$

Application of the above CS formulation to study its effects on
magnetohydrodynamics as well as the $\vec{E}.\vec{B}\neq 0$ regions of a
pulsar magnetosphere as a source of propagating $\tilde{w}$ have been studied
earlier pdg-ref-47 .

We continue adopting the procedure laid out in the paper by Jackiw and Pi to
construct a Chern-Simons action for gravity in the 3+1-dimensional space-time.
Instead of their fixed Lorentz vector $v_{\mu}$, we use the covariant
derivative of $\phi(x)$ so that,

$\mathcal{A}_{GCS}=\mbox{H}\int{w^{\mu\nu\alpha\beta}[\Gamma^{\sigma}_{\nu\tau}\partial_{\alpha}\Gamma^{\tau}_{\beta\sigma}+{2\over{3}}\Gamma^{\sigma}_{\nu\tau}\Gamma^{\tau}_{\alpha\eta}\Gamma^{\eta}_{\beta\sigma}]\phi_{;\mu}\
d^{4}x}$

$\ \ \
=\mbox{H}\int{\epsilon^{\mu\nu\alpha\beta}[\Gamma^{\sigma}_{\nu\tau}\partial_{\alpha}\Gamma^{\tau}_{\beta\sigma}+{2\over{3}}\Gamma^{\sigma}_{\nu\tau}\Gamma^{\tau}_{\alpha\eta}\Gamma^{\eta}_{\beta\sigma}](\ln{\chi})_{,\mu}\
d^{4}x}\ \ \ ,$ $None$

where H is a dimensionless, real constant. After integrating by parts once,
eq.(27) can be expressed in terms of the Riemann tensor as,

$\mathcal{A}_{GCS}=-{\mbox{H}\over{2}}\int{\ln{\chi}\ {}^{*}RRd^{4}x}\ \ ,$
$None$

with,

${}^{*}RR\equiv{1\over{2}}\epsilon^{\mu\nu\alpha\beta}R^{\tau}_{\
\sigma\alpha\beta}R^{\sigma}_{\ \tau\mu\nu}=8\bigg{[}R^{\tau\sigma}_{\ \
01}R_{\sigma\tau 23}+R^{\tau\sigma}_{\ \ 12}R_{\sigma\tau
03}+R^{\tau\sigma}_{\ \ 13}R_{\sigma\tau 20}\bigg{]}$ $None$

and,

${}^{*}R^{\tau\rho\mu\nu}\equiv{1\over{2}}\epsilon^{\mu\nu\alpha\beta}R^{\tau\rho}_{\
\ \alpha\beta}$ $None$

We have used the following conventions pertaining to the Riemann curvature and
Ricci tensors,

$R^{\tau}_{\
\sigma\alpha\beta}=\partial_{\alpha}\Gamma^{\tau}_{\sigma\beta}-\partial_{\beta}\Gamma^{\tau}_{\sigma\alpha}+\Gamma^{\tau}_{\alpha\eta}\Gamma^{\eta}_{\beta\sigma}-\Gamma^{\tau}_{\beta\eta}\Gamma^{\eta}_{\alpha\sigma}$

and $R_{\alpha\beta}=R^{\tau}_{\ \alpha\tau\beta}$. From eq.(27) we note that
$\ln{\chi}$ acts like the parameter $\theta$ in Jackiw and Pi ’s paper in
which the external vector $v_{\mu}$ is set as $\theta_{,\mu}$. Adding
$\mathcal{A}_{CS}+\mathcal{A}_{GCS}$ to $\mathcal{A}$ of eq.(19) and then
extremizing the full action with respect to $\phi$ and $g_{\mu\nu}$ leads to,

$\chi^{\ ;\alpha}_{\
;\alpha}={1\over{2}}\bigg{[}g^{\mu\nu}{{\chi_{,\mu}\chi_{,\nu}}\over{\chi}}+{a_{2}\over{a_{1}}}\chi+{\mbox{J}\over{2a_{1}}}\chi\
w^{\mu\nu\alpha\beta}F_{\mu\nu}F_{\alpha\beta}-{\mbox{H}\over{4a_{1}}}\chi\
w^{\mu\nu\alpha\beta}R^{\tau}_{\ \sigma\alpha\beta}R^{\sigma}_{\
\tau\mu\nu}\bigg{]},$ $None$

$R_{\mu\nu}-{1\over{2}}g_{\mu\nu}R={{8\pi}\over{m^{2}_{Pl}}}[T_{\mu\nu}+\Theta_{\mu\nu}+C_{\mu\nu}],$
$None$

where the modified Cotton tensor $C^{\mu\nu}$ is defined as,

$C^{\mu\nu}\equiv-2{\mbox{H}\over{\sqrt{-g}}}\bigg{[}{1\over{4}}{}^{*}RRg^{\mu\nu}-(\ln\chi)_{;\alpha;\beta}\bigg{(}{}^{*}R^{\beta\mu\alpha\nu}+{}^{*}R^{\beta\nu\alpha\mu}\bigg{)}+(\ln\chi)_{,\alpha}\bigg{(}\epsilon^{\alpha\mu\sigma\tau}R^{\nu}_{\
\sigma;\tau}+\epsilon^{\alpha\nu\sigma\tau}R^{\mu}_{\
\sigma;\tau}\bigg{)}\bigg{]}.$ $None$

The first term in the RHS of Eq.(33) is new and is not present in the
expression for the Cotton tensor as delineated in Jackiw . It appears in our
work because under an infinitesmal variation $g_{\mu\nu}\rightarrow$
$g_{\mu\nu}+\delta g_{\mu\nu}$, the change in $(\ln\chi)_{,\mu}$ occurring in
eq.(28) is given by,

$\delta(\ln\chi)_{,\mu}=\delta(\phi_{;\mu}/\phi)=-\delta\Gamma^{\alpha}_{\alpha\mu}=-{1\over{2}}\delta(g^{\alpha\beta}g_{\alpha\beta,\mu})\
\ .$ $None$

However, when the equation of motion for the dynamical four-form given by
eq.(31) is substituted in eq.(22), its energy-momentum tensor in the presence
of CS-term takes the form,

$\Theta_{\mu\nu}=2a_{1}\bigg{[}{{\chi_{,\mu}\chi_{,\nu}}\over{\chi}}-{{g_{\mu\nu}}\over{2}}\bigg{(}g^{\alpha\beta}{{\chi_{,\alpha}\chi_{,\beta}}\over{\chi}}+{a_{2}\over{a_{1}}}\chi\bigg{)}\bigg{]}+{H\over{2\sqrt{-g}}}{}^{*}RRg_{\mu\nu}$
$None$

so that when eq.(35) is substituted in eq.(32) the first term in the RHS of
eq.(33) cancels with the last term in the RHS of eq.(35). In other words, the
term ${}^{*}RRg_{\mu\nu}$ does not appear in or contribute to the Einstein
equations given by eq.(32).

Assuming that the geometrodynamical four-form is a dark energy candidate, the
above dynamical equations were studied in an earlier work to address the
observed late time acceleration of the expansion rate of the universe (pdg-
ref-20  and the references therein). Our objective, in this article, is
somewhat different since we propose here that the particle associated with
$\tilde{w}$ is a pseudo-scalar which acts as a cold dark matter (CDM)
candidate aiding in the formation of supermassive black holes when the
universe was young. This is the subject of Section IV.

###  III.3 (iii) Generalized Exterior Derivative

Apart from the possibility of torsion and Chern-Simons extensions ensuing from
the four-form field $\tilde{w}$, there may also be another differential
geometric significance of this field. The scalar-density $\phi$ can be used to
have an antiderivation acting on antisymmetric tensor-densities of arbitrary
weights.

Suppose $\tilde{\alpha}$ is a p-form density with weight $w$ such that its
components $\alpha_{\nu_{1}\nu_{2}..\nu_{p}}$ transform to
$J^{-w}(x,x^{\prime})\ \alpha_{\nu_{1}\nu_{2}..\nu_{p}}$, under a general
coordinate transformation $x\rightarrow x^{\prime}$, $J(x,x^{\prime})$ being
the corresponding Jacobian. We define a generalized exterior derivative
$\tilde{d}_{w}$ acting on $\tilde{\alpha}$ in the following manner,

$\displaystyle\tilde{d}_{w}\tilde{\alpha}\equiv{1\over{p!}}\
\partial_{\mu}\alpha_{\nu_{1}\nu_{2}..\nu_{p}}\
\tilde{d}x^{\mu}\wedge\tilde{d}x^{\nu_{1}}\wedge...\wedge\tilde{d}x^{\nu_{p}}-$
$\displaystyle-w\partial_{\mu}(\ln\phi)\
\tilde{d}x^{\mu}\wedge\tilde{\alpha}=\tilde{d}\tilde{\alpha}-w\tilde{d}\
(\ln\phi)\ \wedge\tilde{\alpha}.$

It is easy to see that $\tilde{d}_{w}\tilde{\alpha}$ is a (p+1)-form density
of weight $w$. If $\alpha_{1}$ and $\alpha_{2}$ are scalar-densities of
weights $w_{1}$ and $w_{2}$, respectively, then the above equation leads to,

$\tilde{d}_{w}\alpha_{i}=\partial_{\mu}\alpha_{i}\
\tilde{d}x^{\mu}-w_{i}\alpha_{i}\ \partial_{\mu}(\ln\phi)\ \tilde{d}x^{\mu},\
\ \ i=1,2$ $None$

which are one-form densities and, furthermore, one can show that,

$\tilde{d}_{w}(\alpha_{1}\tilde{d}_{w}\alpha_{2})=\tilde{d}_{w}\alpha_{1}\wedge\tilde{d}_{w}\alpha_{2}.$
$None$

This generalized exterior derivative also satisfies (a)
$\tilde{d}_{w}\tilde{d}_{w}=0$ and (b)
$\tilde{d}_{w}(\tilde{\alpha}\wedge\tilde{\beta})=\tilde{d}_{w}\tilde{\alpha}\wedge\tilde{\beta}+(-1)^{p}\tilde{\alpha}\wedge\tilde{d}_{w}\tilde{\beta}$,
where $\tilde{\alpha}$ and $\tilde{\beta}$ are p- and q-form densities of
weights $w_{1}$ and $w_{2}$, respectively.

These properties are sufficient to qualify $\tilde{d}_{w}$ to be a well
defined antiderivation on differential form-densities Mukunda .

From eq.(36) it follows that,

$\tilde{d}_{w}\sqrt{-g}=-\sqrt{-g}\ \tilde{d}\ln({\phi\over{\sqrt{-g}}}),$
$None$

since $\sqrt{-g}$ is a scalar-density of weight +1. There are other physically
meaningful tensor densities e.g. the scalar density Dirac delta function and
antisymmetric tensor-densities, e.g. dual of $F^{\mu\nu}$, on which
$\tilde{d}_{w}$ can act.

In passing, we note that $\tilde{d}_{w}\phi=0$, which is analogous to
$g_{\mu\nu;\lambda}=0$.

##  IV IV. Dark bosons, Bose-Einstein Condensates and Formation of
Supermassive Black Holes

The wave-particle duality of quantum mechanics transcends to field-particle
duality in quantum field theory. Particles associated with quantum fields are
intimately linked with their Poincare’ algebra and special relativistic
covariance in the standard framework of quantum field theory (QFT). In the
context of the four-form field $\tilde{w}$, or equivalently the scalar-density
$\phi(x)$, we may ask what kind of particle is associated with it?

Now, $\phi(x)$ in the geometrically flat Minkowski space-time transforms to,

$\phi\rightarrow\phi^{\prime}=J(t,\vec{r};t,-\vec{r})\ \phi=-\phi\ \ ,$ $None$

under a space reversal $\vec{r}\rightarrow-\vec{r}$ as the Jacobian
$J(t,\vec{r};t,-\vec{r})=-1$. Since $\phi$ changes its sign under a parity
transformation, the quantum particle associated with the field $\tilde{w}$ is
a pseudo scalar, and thus is a boson.

Because such a pseudo scalar is likely to interact with matter with a strength
at the most comparable with that of gravity, we propose here that $\tilde{w}$
is the candidate for ultra-light cold dark matter (CDM), although such
ultralight CDM has been linked with axions in the literature (e.g. see
Fukuyama , and the references therein). Can this help us in solving the
intriguing problem of supermassive black holes (SMBHs)?

So far, around a million quasars have been observed, each powered by accretion
of matter onto SMBHs weighing $\gtrsim 10^{7}M_{\odot}$ pdg-ref-0 ; pdg-ref-1
; pdg-ref-2 ; Volon . Quasar J0100+2802 found at a redshift, $z=6.33$, is
associated with a SMBH of mass $\approx 10^{10}\ M_{\odot}$, while another
(J2157–3602) at a redshift, $z=4.7$, has a black hole weighing $\cong
2.4\times 10^{10}\ M_{\odot}$. Similarly, quasars J1120+0641 and J1342+0928
located at redshifts $z=7.09$ and $z=7.54$, respectively, are associated with
SMBHs with mass $\gtrsim 10^{8}\ M_{\odot}$. These findings pose a severe
challenge to the existing models that invoke seed black holes (BHs) growing
via accretion, since one needs extremely large seed BHs with mass $\gtrsim
10^{3}\ M_{\odot}$ at $z\gtrsim 40$ Volon ; pdg-ref-11 .

Recently, using the JWST observations, a very distant SMBH with mass $\sim
10^{6}\ M_{\odot}$ has been discovered at a redshift of 10.6 SMBHatz=10.6 . Of
course, even heavier SMBHs at lower redshifts, like with mass $\cong 4\times
10^{10}\ M_{\odot}$ at the centre of Holm 15A galaxy belonging to the galaxy
cluster Abell 85 have been detected Abell85 . At a redshift of 3.96, a SMBH
weighing $\cong 1.7\times 10^{10}\ M_{\odot}$, accreting matter at a rate one
solar mass per day has been seen 17BillionSolarMassBHz=3.962 .

In order to address the formation of SMBHs when the universe was $\lesssim
10^{9}$ yrs old, we had earlier put forward a scenario using the framework of
Gross-Pitaevskii equation in which ultra heavy BHs are created because of
gravitational contraction of Bose-Einstein condensates (BECs) of ultra-light
bosonic CDM PDGEklavya ; PDGFazlu . In this section, we provide a simple semi-
classical analysis that captures the essential physics of the problem.

Considering a galactic DM halo of size $R_{h}$ that is constituted of these
gravitationally bound dark bosons of mass $m$, a significant fraction of the
bosons that have sufficiently low momenta $p$ would form a BEC if their de
Broglie wavelength,

$\lambda_{DB}\sim\frac{h}{p}\gtrsim\bigg{(}\frac{3N}{4\pi
R^{3}_{h}}\bigg{)}^{-1/3}=R_{h}\bigg{(}\frac{3N}{4\pi}\bigg{)}^{-1/3}=R_{h}\bigg{(}\frac{3M}{4\pi
m}\bigg{)}^{-1/3}$ $None$

where $M$ and $N$ are the total mass of dark bosons and their number,
respectively, making up the BEC. Eq.(42) represents the condition for BEC
formation since it entails that the de Broglie wavelength be larger than the
mean separation between nearby bosons.

In the context of the CDM scenario and assuming a positive cosmological
constant, the energy $E_{b}$ of a typical dark boson is given by PDGCosmo ,

$E_{b}\sim\frac{p^{2}}{2m}-\frac{GMm}{R_{h}}+\frac{\Lambda\ r^{2}}{6}mc^{2}<0$
$None$

The condition $E_{b}<0$ follows from the fact that the DM halo is a
gravitationally bound structure. Hence,

$p^{2}<\frac{2GMm^{2}}{R_{h}}-\frac{\Lambda\ r^{2}}{3}m^{2}c^{2}$ $None$

Since a very weakly interacting dark boson can be anywhere within the DM halo,
application of the Heisenberg’s uncertainty principle implies that,

$\Delta p\sim p\gtrsim\frac{\hbar}{2R_{h}}$ $None$

From eqs.(42) and (45), we have an inequality,

$\frac{h}{4\pi p}\lesssim
R_{h}\lesssim\frac{h}{p}\bigg{(}\frac{3N}{4\pi}\bigg{)}^{1/3}$ $None$

that is self-consistent since $N\gg 1$.

Using the result $p\sim\frac{\hbar}{2R_{h}}$ from eq.(45) in eq.(43), we
obtain,

$E_{b}(R_{h})\sim\frac{\hbar^{2}}{8mR^{2}_{h}}-\frac{GMm}{R_{h}}+\frac{\Lambda\
R^{2}_{h}}{6}mc^{2}<0$ $None$

that leads to a condition,

$R_{h}\gtrsim\frac{\hbar^{2}}{8GMm^{2}}\bigg{(}1-\frac{R^{3}_{h}\Lambda
c^{2}}{6GM}\bigg{)}^{-1}$ $None$

In the context of the $\Lambda$CDM model, the numerical value of the
cosmological constant that ensues from the cosmological constant density
parameter, $\Omega_{\Lambda,0}\cong 0.7$ and the Hubble parameter,
$\mbox{H}_{0}\cong 68$ km/s/Mpc, is $\Lambda\cong 10^{-56}\ cm^{-2}$.

Even with a low dark matter halo mass $M\cong 10^{6}\ M_{\odot}$, one finds
$\Lambda c^{2}/6GM\cong 10^{-68}\ cm^{-3}$. Hence, for galactic scales
$R_{h}\lesssim 10^{22}\ cm$, $R^{3}_{h}\Lambda c^{2}/6GM\ll 1$ so that eq.(48)
can be approximated to,

$R_{h}\gtrsim\frac{\hbar^{2}}{8GMm^{2}}\bigg{(}1+\frac{R^{3}_{h}\Lambda
c^{2}}{6GM}\bigg{)}\approx\frac{\hbar^{2}}{8GMm^{2}}$ $None$

By minimizing $E_{b}$ with respect to $R_{h}$, one can obtain an estimate of
the BEC size,

$\frac{\partial E_{b}}{\partial R_{h}}=\frac{\Lambda\
R_{h}}{3}mc^{2}+\frac{GMm}{R^{2}_{h}}-\frac{\hbar^{2}}{4mR^{3}_{h}}=0$ $None$

which leads to a quartic equation to be solved,

$\Lambda R^{4}_{h}+\frac{3GM}{c^{2}}R_{h}-\frac{3\hbar^{2}}{4m^{2}c^{2}}=0$
$None$

The above equation can be solved exactly and, out of the four roots, only one
is real and positive. However, given the smallness of the quantity
$\lambda_{M}\equiv\Lambda c^{2}/6GM\cong 10^{-68}\ cm^{-3}$ in our case, even
up to the cubic expansion of this root, $R_{bec}$, in terms of $\lambda_{M}$,
the solution to eq.(51) corresponding to the minimum energy configuration of
the BEC is (upto $\lambda^{3}_{M}$ orders),

$R_{bec}\cong\frac{\hbar^{2}}{4GMm^{2}}\cong
22\bigg{(}\frac{10^{9}M_{\odot}}{M}\bigg{)}\bigg{(}\frac{10^{-22}\
\mbox{eV}}{m}\bigg{)}^{2}\ \mbox{pc}$ $None$

corresponding to a single boson energy,

$E_{min}=E_{b}(R_{bec})\cong-\frac{GMm}{2R_{bec}}=-2\
mc^{2}\bigg{(}\frac{m^{2}_{Pl}}{m\ M}\bigg{)}^{-2}$ $None$

Eqs.(52) and (53) are exact when $\Lambda=0$, ensuing trivially from eq.(51).
These equations entail that for larger BEC mass $M$ not only its size is
smaller, it is also more tightly bound since its energy is more negative. The
physical repercussion of these features is that for a sufficiently large mass
$M$, $R_{bec}$ can be very close to the corresponding Schwarzschild radius
$R_{s}\equiv 2GM/c^{2}$ so that even a slight perturbation would cause an
irreversible gravitational collapse leading to the formation of a BH.

The BEC would form a BH if its size $R_{bec}$ goes below the Schwarzschild
radius $R_{S}$,

$R_{S}=\frac{2GM}{c^{2}}=\frac{2M}{m^{2}_{Pl}}$ $None$

where $m_{Pl}$ is the Planck mass.

Using the criteria $R_{bec}\lesssim R_{S}$ along with eqs.(52) to (54), the
condition for the BEC to implode into a BH is given by the inequality,

$\bigg{(}\frac{m^{2}_{Pl}}{4M\ m}\bigg{)}^{2}\lesssim 1$ $None$ $\Rightarrow\
m\ M\gtrsim\ 0.25\ m^{2}_{Pl}=3\times 10^{9}\ \bigg{(}\frac{m}{10^{-20}\
\mbox{eV}}\bigg{)}^{-1}M_{\odot}$ $None$

A more rigorous analysis carried out earlier by employing a Gross-Pitaevskii
equation framework had shown that the dynamical evolution of ultra-light dark
bosons in the BEC phase leads to the formation of a BH on time scales of
$10^{8}$ years with a similar constraint given by eq.(56) PDGEklavya ;
PDGFazlu .

The Hawking temperature $T_{BH}$ of a BH of mass $M$ is given by Hawking ,

$k_{B}T_{BH}=\frac{m^{2}_{Pl}}{8\pi M}$ $None$

It is interesting to note that when eq.(56) is substituted in eq.(57), one
gets,

$k_{B}T_{BH}\lesssim\frac{m}{2\pi}$ $None$

implying that a SMBH created from the collapse of a BEC of dark bosons would
be predominantly radiating massless particles as well as these dark bosons
through Hawking evaporation!

##  V V. Propagation of Gravitational Waves in the presence of the
geometrodynamical four-form

We consider in this section the effect of $\tilde{w}$ on the propagation of
gravitational waves (GWs) by assuming that, except for a weak GW and the
geometrodynamical four form, there is no other matter present. Considering a
traceless-transverse (TT)-gauge for a plane GW travelling along the x-axis,

$h_{\oplus}\equiv h_{22}(t,x)=-h_{33}(t,x)\ \ \ \ \ h_{\otimes}\equiv
h_{23}(t,x)=h_{32}(t,x)$

we express eqs.(19) and (28), by making the weak field approximation
$g_{\mu\nu}=\eta_{\mu\nu}+h_{\mu\nu}$ and retaining only upto quadratic terms
in $h_{\oplus}$ and $h_{\otimes}$ for the gravitational part of the action
Alex ,

$\mathcal{A}\approx-{{m^{2}_{Pl}}\over{8\pi}}\int{[h_{\oplus}(\ddot{h}_{\oplus}-h^{\prime\prime}_{\oplus})+h_{\otimes}(\ddot{h}_{\otimes}-h^{\prime\prime}_{\otimes})]d^{4}x}+\mathcal{A}_{GCS}+\mathcal{A}_{\phi}$
$None$

with,

$\mathcal{A}_{GCS}\approx-H\int{\eta^{\tau\lambda}\bigg{(}\frac{\partial^{2}\ln\chi}{\partial
x\partial
x^{\tau}}(h_{\oplus}h_{\otimes,0\lambda}-h_{\otimes}h_{\oplus,0\lambda})+\frac{\partial^{2}\ln\chi}{\partial
t\partial
x^{\tau}}(h_{\otimes}h_{\oplus,1\lambda}-h_{\oplus}h_{\otimes,1\lambda})\bigg{)}d^{4}x}$
$\ \ \ \ \ \ \ \ \ -H\int{\frac{\partial\ln\chi}{\partial
x}\bigg{(}h_{\oplus}(\ddot{h}_{\otimes,0}-h^{\prime\prime}_{\otimes,0})-h_{\otimes}(\ddot{h}_{\oplus,0}-h^{\prime\prime}_{\oplus,0})\bigg{)}d^{4}x}$
$\ \ \ \ \ \ \ \ \ +H\int{\frac{\partial\ln\chi}{\partial
t}\bigg{(}h_{\oplus}(\ddot{h}_{\otimes,1}-h^{\prime\prime}_{\otimes,1})-h_{\otimes}(\ddot{h}_{\oplus,1}-h^{\prime\prime}_{\oplus,1})\bigg{)}d^{4}x}$

and,

$\mathcal{A}_{\phi}=\int{\bigg{[}\frac{a_{1}}{\chi}\eta^{\mu\nu}\chi_{,\mu}\chi_{,\nu}+a_{2}\chi-\frac{a_{1}}{\chi}\bigg{(}h_{\oplus}(\chi_{,2}\chi_{,2}-\chi_{,3}\chi_{,3})+2h_{\otimes}\chi_{,2}\chi_{,3}\bigg{)}\bigg{]}d^{4}x}$

After defining $\psi(x^{\mu})\equiv\ln{(\chi(x^{\mu}))}$, the dynamical
equations of motion that ensue from variations of $\mathcal{A}$ (eq.(60)) with
respect to $h_{\oplus}$, $h_{\otimes}$ and $\psi$ are given by,

$\ddot{h}_{\oplus}-h^{\prime\prime}_{\oplus}=-\frac{8\pi}{m^{2}_{Pl}}\bigg{[}H\bigg{\\{}\frac{\partial^{2}\psi}{\partial
t\partial
x}(\ddot{h}_{\otimes}+h^{\prime\prime}_{\otimes})-h_{\otimes,01}\bigg{(}\frac{\partial^{2}\psi}{\partial
t^{2}}+\frac{\partial^{2}\psi}{\partial
x^{2}}\bigg{)}+\frac{\partial\psi}{\partial
x}(\ddot{h}_{\otimes,0}-h^{\prime\prime}_{\otimes,0})-\frac{\partial\psi}{\partial
t}(\ddot{h}_{\otimes,1}-h^{\prime\prime}_{\otimes,1})\bigg{\\}}+$ $\ \ \ \ \ \
\ \ \ +a_{1}\exp{(\psi)}\bigg{\\{}\bigg{(}\frac{\partial\psi}{\partial
y}\bigg{)}^{2}-\bigg{(}\frac{\partial\psi}{\partial
z}\bigg{)}^{2}\bigg{\\}}\bigg{]}$ $None$

$\ddot{h}_{\otimes}-h^{\prime\prime}_{\otimes}=\frac{8\pi}{m^{2}_{Pl}}\bigg{[}H\bigg{\\{}\frac{\partial^{2}\psi}{\partial
t\partial
x}(\ddot{h}_{\oplus}+h^{\prime\prime}_{\oplus})-h_{\oplus,01}\bigg{(}\frac{\partial^{2}\psi}{\partial
t^{2}}+\frac{\partial^{2}\psi}{\partial
x^{2}}\bigg{)}+\frac{\partial\psi}{\partial
x}(\ddot{h}_{\oplus,0}-h^{\prime\prime}_{\oplus,0})-\frac{\partial\psi}{\partial
t}(\ddot{h}_{\oplus,1}-h^{\prime\prime}_{\oplus,1})\bigg{\\}}+$ $\ \ \ \ \ \ \
\ \ \ +2a_{1}\exp{(\psi)}\frac{\partial\psi}{\partial
y}\frac{\partial\psi}{\partial z}\bigg{]}$ $None$

and

$\partial^{\mu}\partial_{\mu}\psi+\frac{1}{2}\eta^{\mu\nu}\frac{\partial\psi}{\partial
x^{\mu}}\frac{\partial\psi}{\partial
x^{\nu}}-\frac{a_{2}}{4a_{1}}=h_{\oplus}\bigg{[}\frac{\partial^{2}\psi}{\partial
y^{2}}-\frac{\partial^{2}\psi}{\partial
z^{2}}+\frac{1}{2}\bigg{(}\frac{\partial\psi}{\partial
y}\bigg{)}^{2}-\frac{1}{2}\bigg{(}\frac{\partial\psi}{\partial
z}\bigg{)}^{2}\bigg{]}+2h_{\otimes}\bigg{[}\frac{\partial^{2}\psi}{\partial
y\partial z}+\frac{1}{2}\frac{\partial\psi}{\partial
y}\frac{\partial\psi}{\partial z}\bigg{]}\ \ .$ $None$

(The derivatives with respect to time $t$ and x-coordinate are denoted by dot
and prime, respectively.)

A simplification occurs if we assume $\psi$ to depend only on the x-coordinate
and time, like the GW amplitude itself. With $\psi=\psi(t,x)$, the last terms
in eqs.(61) and (62) drop out. Introducing the complex circularly polarized GW
amplitudes [21],

$h_{R}=\frac{1}{\sqrt{2}}(h_{\oplus}+ih_{\otimes})\ \ \ \ \ \
h_{L}=\frac{1}{\sqrt{2}}(h_{\oplus}-ih_{\otimes})$

one finds from eqs.(61) and (62) that,

$\ddot{h}_{R}-h^{\prime\prime}_{R}=-\frac{8\pi
Hi}{m^{2}_{Pl}}\bigg{[}\dot{\psi}\frac{\partial}{\partial
x}(\ddot{h}_{R}-h^{\prime\prime}_{R})-\psi^{\prime}\frac{\partial}{\partial
t}(\ddot{h}_{R}-h^{\prime\prime}_{R})+\dot{h}^{\prime}_{R}(\ddot{\psi}+\psi^{\prime\prime})-\dot{\psi}^{\prime}(\ddot{h}_{R}+h^{\prime\prime}_{R})\bigg{]}$
$None$ $\ddot{h}_{L}-h^{\prime\prime}_{L}=\frac{8\pi
Hi}{m^{2}_{Pl}}\bigg{[}\dot{\psi}\frac{\partial}{\partial
x}(\ddot{h}_{L}-h^{\prime\prime}_{L})-\psi^{\prime}\frac{\partial}{\partial
t}(\ddot{h}_{L}-h^{\prime\prime}_{L})+\dot{h}^{\prime}_{L}(\ddot{\psi}+\psi^{\prime\prime})-\dot{\psi}^{\prime}(\ddot{h}_{L}+h^{\prime\prime}_{L})\bigg{]}$
$None$

Since $\psi=\psi(t,x)$, the equation of motion for the four-form given by
eq.(63) simplifies to,

$\ddot{\psi}-\psi^{\prime\prime}+\frac{1}{2}(\dot{\psi}^{2}-\psi^{\prime\
2})+2\mu^{2}=0$ $None$

where $\mu\equiv\sqrt{-\frac{a_{2}}{4a_{1}}}$ acts as the mass of the four-
form field. Seeking exact solutions, involving monochromatic and circularly
polarized GWs, we substitute,

$h_{R}(t,x)=h\exp(i(\omega t-kx))$

in eq.(64), leading to a relation,

$\omega^{2}-k^{2}=-\beta[k(\omega^{2}-k^{2})\dot{\psi}+\omega(\omega^{2}-k^{2})\psi^{\prime}-i\omega
k(\ddot{\psi}+\psi^{\prime\prime})-i(\omega^{2}+k^{2})\dot{\psi}^{\prime}]$
$None$

where $\beta\equiv\frac{8\pi H}{m^{2}_{Pl}}$.

If $w=\pm k$, eq.(67) implies,

$2\dot{\psi}^{\prime}\pm(\ddot{\psi}+\psi^{\prime\prime})=0\ \ .$ $None$

The only self-consistent solution of eqs.(66) and (68) is,

$\psi(t,x)=(b_{0}-\frac{\mu^{2}}{b_{0}})t+(b_{0}+\frac{\mu^{2}}{b_{0}})x$
$None$

where $b_{0}$ is an integration constant. But the above solution is
unphysical, as it implies an exponentially growing $\psi$. On the other hand,
when $w^{2}>k^{2}$,

$k\dot{\psi}+\omega\psi^{\prime}+\frac{1}{\beta}=0\Rightarrow\psi^{\prime\prime}=\frac{k^{2}}{w^{2}}\
\ddot{\psi}$ $None$

and therefore, $\psi=\psi(x-\frac{\omega}{k}t)$, leading to an exact solution
for $b_{1}\geq\lambda$,

$\psi=\ln{\bigg{(}\sqrt{\frac{b_{1}}{\lambda}}\cos^{2}\bigg{(}\pm\sqrt{\frac{\lambda}{2}}(t-\frac{k}{\omega}x)+b_{2}\bigg{)}\bigg{)}}$
$None$

where $\lambda\equiv\frac{2\omega^{2}\mu^{2}}{\omega^{2}-k^{2}}$, while
$b_{1}$ and $b_{2}$ are integration constants.

Hence,
$\chi(t-\frac{k}{\omega}x)=\exp(\psi)\propto\cos^{2}\bigg{(}\pm\sqrt{\frac{\lambda}{2}}(t-\frac{k}{\omega}x)+b_{2}\bigg{)}$,
with $|\omega|>|k|$, is physically meaningful and is an acceptable solution.
Indeed, we find that it is possible to have circularly polarized,
monochromatic GWs with phase velocity exceeding the speed of light. The
caveat, however, is that eq.(66) being a nonlinear differential equation,
deriving exact solutions corresponding to GWs with superposed wave modes is
non-trivial.

##  VI VI. Conclusions

Based on the invariances of the Minkowski tensor and the flat space-time Levi-
Civita tensor under proper Lorentz transformations, the present study explores
the possibility of extending Einstein’s geometrical theory of gravitation by
including another geometrical degree of freedom $\tilde{w}$, which is simply a
generalisation of the Levi-Civita symbol, in the theory. This
geometrodynamical four-form $\tilde{w}$ leads not only to a dynamical torsion
it also generates Chern-Simon (CS) extensions in the 3+1-dimensional space-
times. Torsion, among several other interesting implications, is also
important in teleparallel gravity TelepGravity .

Pursuing closely the seminal work of Jackiw and Pi Jackiw  but avoiding the
use of an unphysical Lorentz vector, CS coupling between electromagnetic
fields and $\tilde{w}$ comes about naturally, bringing about a modification of
Einstein-Maxwell equations. Adopting once again the formulation of Jackiw , a
gravitational CS term is constructed that leads to a modified Cotton tensor.
However, when the dynamical equation of the four-form is used in the ensuing
Einstein equation, it is shown that it is the standard Cotton tensor that
affects the space-time dynamics.

The scalar-density $\phi$ associated with $\tilde{w}$ leads to a well-defined
exterior derivative that turns a differential p-form density into a (p+1)-form
density of same weight. Since the notion of an n-form and exterior derivative
in a differential manifold does not require either an affine connection or a
metric, further studies are required to investigate the role of $\tilde{w}$ in
situations where metric is degenerate as well as its impact on the manifold-
orientability.

The particle aspects of the geometrodynamical four-form field is considered
and is shown to be a pseudo-scalar. It is shown that the Bose-Einstein
condensates of such pseudo-scalars can give rise to formation of supermassive
black holes through an interplay of self-gravity and quantum mechanics.
Therefore, self-gravitating BECs of pseudo-scalar particles associated with
the geometrodynamical four-form field may solve the long standing problem
concerning frequent discovery of tens of billion solar mass SMBHs at epochs
when the universe was barely a billion years old.

It is also demonstrated that in the presence of the dynamical four-form,
gravitational waves can be decoupled using circularly polarized waveforms.
Moreover, exact, self-consistent monochromatic gravitational wave solutions of
the wave equation can be obtained. However, much more work is needed to obtain
realistic solutions of gravitational waves, with group velocities $\leq c$,
propagating in the background consisting of $\tilde{w}$.

##  VII Acknowledgements

###### Acknowledgements.

It is a pleasure to thank Prof. Vesselin Petkov for organising the Third
Minkowski Meeting that entailed stimulating discussions on the fundamental
aspects of space-time physics.

## References

  * (1)

##  VIII References

* (2) Das Gupta, P. (2009), arXiv:0905.1621
* (3) Schutz, B.F., 1980. Geometrical methods of mathematical physics. Cambridge university press.
* (4) Hehl, F.W., 1973. Spin and torsion in general relativity: I. Foundations. General relativity and gravitation, 4, pp.333-349.
* (5) Hehl, Friedrich W.; von der Heyde, Paul; Kerlick, G. David; Nester, James M., Reviews of Modern Physics, Volume 48, Issue 3, July 1976, pp.393-416
* (6) Mukunda, N, in Geometry, fields and cosmology: techniques and applications (Vol. 88), eds. Iyer, B.R. and Vishveshwara, C.V., Springer Science & Business Media 2013.
* (7) Jackiw, R. and Pi, S.Y., 2003. Chern-Simons modification of general relativity. Physical Review D, 68(10), p.104012.
* (8) Das Gupta, P. (2010), On Chern–Simons corrections to magnetohydrodynamics equations. Radiation Effects & Defects in Solids: Incorporating Plasma Science & Plasma Technology, 165(2), 106-113.
* (9) Fukuyama, T. (2024). Axion and the SuperMassive Black Holes at high z. International Journal of Modern Physics A, 2350191.
* (10) Ho, L. C. (1999), Observational Evidence for Black Holes in the Universe, (Springer Netherlands) pp. 157-186
* (11) Ferrarese, L., & Merritt, D. (2000), Ap J, 539, L9
* (12) Kormendy, J., & Ho, L. C. (2013), Ann. Rev. Astr. & Ap, 51, 511
* (13) Volonteri, M., Habouzit, M. and Colpi, M., 2021. The origins of massive black holes. Nature Reviews Physics, 3(11), pp.732-743.
* (14) Banados, E., Venemans, B. P., Mazzucchelli, C., Farina, E. P., Walter, F., Wang, F., … & Winters, J. M. (2018). An 800-million-solar-mass black hole in a significantly neutral Universe at a redshift of 7.5. Nature, 553(7689), 473-476.
* (15) Maiolino, R., Scholtz, J., Witstok, J. et al. A small and vigorous black hole in the early Universe. Nature (2024)
* (16) Mehrgan, K., Thomas, J., Saglia, R., Mazzalay, X., Erwin, P., Bender, R., Kluge, M. and Fabricius, M., 2019. A 40 billion solar-mass black hole in the extreme core of Holm 15A, the central galaxy of Abell 85. The Astrophysical Journal, 887(2), p.195.
* (17) Wolf, Christian, et al. ”The accretion of a solar mass per day by a 17-billion solar mass black hole.” Nature Astronomy (2024): 1-10.
* (18) Das Gupta, P. & Thareja, E. (2017). Supermassive black holes from collapsing dark matter Bose–Einstein condensates. Classical and Quantum Gravity, 34(3), 035006.
* (19) Das Gupta, P. , & Rahman, F. (2018). Aspects of Black Hole Physics and Formation of Super-massive Black Holes from Ultra-light Dark Bosons. The Physical Universe eds. S. M. Wagh, S. D. Maharaj & G. Chon (2018, Published by Central India Research Institute, India) arXiv:1801.02559.
* (20) Das Gupta, P. (1997). The Cosmological Constant: A Tutorial. in Geometry, Fields and Cosmology: Techniques and Applications (eds. B. R. Iyer & C. V. Vishveshwara), 525-548.
* (21) Hawking, S. W. (1974). Nature, 248, 30-31.
* (22) Alexander, S. and Martin, J., 2005. Phys. Rev. D 71, 063526.
* (23) Bahamonde, S., Dialektopoulos, K.F., Escamilla-Rivera, C., Farrugia, G., Gakis, V., Hendry, M., Hohmann, M., Said, J.L., Mifsud, J. and Di Valentino, E., 2023. Teleparallel gravity: from theory to cosmology. Reports on Progress in Physics, 86(2), p.026901.


# Linear Transformers with Learnable Kernel Functions are Better In-Context
Models

Yaroslav Aksenov, Nikita Balagansky∗, Sofia Maria Lo Cicero Vaina,
Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
Tinkoff

###### Abstract

Advancing the frontier of subquadratic architectures for Language Models (LMs)
is crucial in the rapidly evolving field of natural language processing.
Current innovations, including State Space Models, were initially celebrated
for surpassing Transformer performance on language modeling tasks. However,
these models have revealed deficiencies in essential In-Context Learning
capabilities – a domain where the Transformer traditionally shines. The Based
model emerged as a hybrid solution, blending a Linear Transformer with a
kernel inspired by the Taylor expansion of exponential functions, augmented by
convolutional networks. Mirroring the Transformer’s in-context adeptness, it
became a strong contender in the field. In our work, we present a singular,
elegant alteration to the Based kernel that amplifies its In-Context Learning
abilities evaluated with the Multi-Query Associative Recall task and overall
language modeling process, as demonstrated on the Pile dataset.

**footnotetext: Corresponding author<EMAIL_ADDRESS>
##  1 Introduction

Large Language Models (LLMs) are revolutionizing the field of natural language
processing and establishing new benchmarks across various tasks (Touvron et
al., 2023; Jiang et al., 2023). Nevertheless, despite their triumphs, most of
these models are built on Transformer frameworks that employ attention
mechanisms. These mechanisms scale poorly with long text sequences, leading to
impractical computational complexity for extending contextual processing
(Vaswani et al., 2017; Tay et al., 2021).

To address this constraint, several alternatives to Transformers were
proposed. Katharopoulos et al. (2020) suggested replacing the exponential
function in the attention mechanism with the kernel function to change the
order of computations and thus move away from quadratic complexity of the
sequence length. However, when compared to vanilla Transformers, this approach
leads to a drop in performance. Furthermore, the kernel function selection is
a topic still in need of consideration. An alternative way to define a linear
model is to utilize State Space Models (SSMs) (Gu et al., 2022; Smith et al.,
2023; Gu and Dao, 2023), which are capable of producing quality that is
comparable to Transformers when measured with perplexity on language modeling.

Notably, both Linear Transformers Katharopoulos et al. (2020) and SSMs can be
described as Recurrent Neural Networks (RNNs) (Chung et al., 2014; Hochreiter
and Schmidhuber, 1997), which have their limitations when it comes to managing
lengthy dependencies within texts since memory capacity can be overrun as the
volume of information increases. Additionally, while the hidden state of RNNs
is larger for Linear Transformers than for SSMs, the latter showed higher text
modeling quality. The introduction of the Based model (Arora et al., 2023)
attempted to address the abovementioned challenges by utilizing a hybrid
architecture (Fu et al., 2023a) based on a Linear Transformer with a novel
kernel function derived from a Taylor expansion of an exponential function.
Arora et al. (2023) demonstrated that the Based model was less prone to
performance issues when working with longer content than other models when
assessed on the Multi-Query Associative Recall (MQAR) task. Nonetheless, even
the Based model experiences a drop in performance when faced with extensive
contexts relative to the conventional transformer architecture.

Figure 1: Results on the MQAR dataset, designed to measure In-Context Learning
capabilities of an architecture Arora et al. (2023). ReBased outperforms all
baselines except Attention across different sequence lengths and model sizes.
See Section 5.2 for more details.

A profound comprehension of the processes occurring within the Based
architectures is essential for their advancement. Upon examining how attention
scores are distributed, we argue that the kernel function previously adopted
in Based cannot be considered optimal, resulting in limitations when dealing
with lengthy context and small model capacity. To address this issue, we
introduce ReBased (Revisited Based)111Source code is available by the link
https://github.com/corl-team/rebased, a novel variation of the Linear
Transformer model that improves the use of attention kernels. The crux of our
development lies in addressing the inability of Based to disregard specific
tokens with zero probability during the attention process. By refining the
kernel function and incorporating new architectural modifications, we have
created a model that improves accuracy on tasks involving retrieving
information from long sequences of tokens while simplifying the calculation of
the attention mechanism.

When testing our enhanced architecture on the MQAR task, we found that ReBased
surpasses the original Based model across a variety of contexts and model
sizes. Additionally, after training with the Pile dataset (Gao et al., 2020),
we observed that ReBased performs better than its predecessor at In-Context
Learning and excels at modeling associative dependencies measured through
improved perplexity metrics.

##  2 Recent Work

The Vanilla Transformer architecture (Vaswani et al., 2017), although widely
used in NLP (Radford et al., 2019; Touvron et al., 2023; Devlin et al., 2019;
Jiang et al., 2023), suffers from growing computational and memory demands
($\mathcal{O}(d*N^{2})$ as sequence lengths ($N$) and head size ($d$)
increase). While this is not much of a problem when it comes to shorter
sequences, it becomes a significant bottleneck when working with longer ones.

Several alternative architectures were proposed to address this issue.
Katharopoulos et al. (2020) suggested substituting the attention mechanism’s
exponential function, which is meant to measure the similarity between queries
and keys, with a product of kernel functions that can be separately evaluated
for queries and keys. This kernel-based approach reshapes the computation
within the attention mechanism, cutting the time and memory complexity to
$\mathcal{O}(d^{2}*N)$. Additionally, during inference, it supports sampling
sequences with linear complexity regarding length, similar to RNNs (Hochreiter
and Schmidhuber, 1997; Chung et al., 2014).

In a different approach, State Space Models (SSMs) borrow from control theory
to offer a simplified structure akin to RNNs, but without activation functions
across time (Gu et al., 2022; Smith et al., 2023; Gu et al., 2023). The Mamba
model, also known as S6 (Gu and Dao, 2023), stands out in this category,
displaying enhanced learning of short-term dependencies in texts compared to
existing pre-trained LLMs (Jiang et al., 2023; Touvron et al., 2023).

Despite these advancements, there is no standard way to fully evaluate these
innovative architectures to assess their performance limits. One standard
evaluation method is to pre-train a language model and assess its perplexity
with a given dataset, but this may not truly reflect the model’s ability to
manage long context dependencies. Another option is to use the Long Range
Arena (LRA) benchmark, which involves classification tasks with long input
sequences. Though some new models have outperformed Transformers in the LRA,
it is believed that the benchmark is capable of introducing bias in the
comparison (Amos et al., 2023).

One promising evaluation approach is to test an architecture’s In-Context
Learning abilities. Olsson et al. (2022) introduced the concept of Associative
Recall (AR), a task where the model learns to copy a token from a sequence
after a certain point. However, while in Fu et al. (2023a) the associative
recall task was implemented with a goal to retrieve only one token, Arora et
al. (2023) noted that this task could be considered overly simplistic. This
led to the creation of the Multi-Query Associative Recall (MQAR) task, which
requires retrieving multiple tokens from the context. Findings on MQAR
indicate that while newer models may compete with Transformer in terms of
perplexity, they can still struggle with long contexts at small model sizes
because of their limited In-Context Learning capabilities. Meanwhile,
Transformers remain robust against such factors. Lastly, Arora et al. (2023)
introduced Linear Transformer with a new kernel function (namely Based),
showcasing enhanced performance on the MQAR task when compared to Mamba.

Despite this improvement, compared to traditional Transformers, the problem of
decline in performance when handling long sequences with smaller models still
remains. Addressing this challenge is the primary goal of our paper.

##  3 Background

###  3.1 Linear Transformers

To fully grasp the Based architecture, it is vital to first discuss the
original Transformer model. The attention mechanism, which is central to the
Transformer’s functionality, evaluates the output $y_{i}$ for each position
$i$ as follows

$y_{i}=\frac{\sum_{j=0}^{i}\text{sim}(Q_{i},K_{j})V_{j}}{\sum_{n=0}^{i}\text{sim}(Q_{i},K_{n})},$

where the term
$\text{sim}(Q_{i},K_{j})=\exp\left(\frac{Q^{T}_{i}K_{j}}{\sqrt{d}}\right)$
represents the similarity between the query $Q_{i}$ and the key $K_{j}$ using
an exponential function. Despite its effectiveness, the original Transformer’s
reliance on this attention mechanism incurs a quadratic increase in both
computational time and memory use as the sequence length grows, which becomes
impractical for processing long sequences.

To address this scalability problem, Katharopoulos et al. (2020) suggested
replacing the direct computation of similarity between $Q$ and $K$ with a
transformation through a non-linear kernel function $\phi(\cdot)$. This allows
for the following approximation:
$\text{sim}(Q_{i},K_{j})\approx\phi^{T}(Q_{i})\phi(K_{j})$. By implementing
this kernel, the Linear Transformer computes $y_{i}$ as

$y_{i}=\frac{\sum_{j=0}^{i}\phi^{T}(Q_{i})\phi(K_{j})V_{j}}{\sum_{n=0}^{i}\phi(Q_{i})\phi^{T}(K_{n})}.$

By rearranging the operations, we can express the computation as

$y_{i}=\frac{\phi^{T}(Q_{i})\sum^{i}_{j=0}\phi(K_{j})V^{T}_{j}}{\phi^{T}(Q_{i})\sum^{i}_{n=0}\phi(K_{n})}.$

By calculating $\phi(K_{j})V^{T}_{j}\in\mathbb{R}^{d\times d}$ upfront, the
complexity of the attention mechanism transitions to linear with the sequence
length, addressing the inefficiencies of the original model.

###  3.2 Based

Selecting an appropriate kernel function $\phi(\cdot)$ is critical to a Linear
Transformer’s performance. Various kernel functions have been proposed (Peng
et al., 2021; Schlag et al., 2021; Qin et al., 2022), but on language modeling
tasks, none have surpassed the original attention mechanism. However, a
breakthrough was achieved by Arora et al. (2023), who introduced a novel
kernel function inspired by the Taylor series expansion of the exponential
function, defined as

$\text{sim}(q,k)=1+q^{T}k+\frac{(q^{T}k)^{2}}{2}.$

The choice of this kernel is motivated by its ability to approximate the
exponential function over a specific range of $q^{T}k$ values. In addition,
Arora et al. (2023) utilized a hybrid architecture by combining linear
attention with convolutional layers since doing so was shown to help models
handle short non-associative dependencies in the sequences (Fu et al., 2023a;
Poli et al., 2023; Fu et al., 2023b)

In doing so, when evaluated on the MQAR task, the Based model demonstrated
that it was capable of outperforming the Mamba model (Gu and Dao, 2023) under
circumstances of substantial context length and constrained model capacity due
to smaller sizes. Nevertheless, compared to the original Transformer, a
discernible drop-off in performance remains, indicating room for further
improvement.

##  4 Revisiting Based

Figure 2: Similarity between $q$ and $k$ with respect to scalar product. Note
that the Based model has a minimal $\text{sim}(q,k)$ value of 0.5, which can
lead to suboptimal performance. We propose to learn the scale and shift of the
parabola jointly with the model and make it possible to zero out the
similarity value. See Section 4 for more details and Section 5.1 for
experimental setup description.

In our study, we explore the fundamental requirements for kernel functions. We
examine the exponential function and its approximate representation, as
depicted in Figure 2. We observe a limitation in the approximation since its
minimal value is fixed at $0.5$. This is problematic for handling long
sequences, as it is difficult to assign a near-zero attention score to
specific token pairs. Ideally, we want to be able to diminish the attention
scores to zero, which would require significantly larger values elsewhere in
the normalization process with the Based model.

To rectify this issue, a straightforward approach would be to adjust the
lowest point of the kernel function to zero. However, this solution prompts us
to ask why the minimum value of the kernel function should occur precisely at
$q^{T}k=-1$. As used in the original Transformer, the traditional exponential
similarity function increases monotonically, but the quadratic kernel has an
optimal value to which it decreases and then ascends from. Therefore, to
decrease attention in the Transformer, one would aim to minimize $q^{T}k$. In
contrast, the ideal $q^{T}k$ should be exactly $-1$ for the Based method.
Otherwise, the attention score would increase. This condition may induce less-
than-ideal training outcomes and degrade the model’s accuracy.

These challenges lead us to conjecture that if the quadratic kernel is used to
calculate the similarity between $q$ and $k$, we must consider the range of
potential $q^{T}k$ values and create adjustable parameters for the parabolic
function to align with these values during training. Simplifying for clarity,
let us look at a one-dimensional scenario. We can express the trainable
parameters of the kernel function in relation to the affine transformation of
$q$ and $k$ as such

$\displaystyle q^{\prime}=$ $\displaystyle\gamma_{Q}\cdot q+\beta_{Q},\ \
k^{\prime}=\gamma_{K}\cdot k+\beta_{K};$
$\displaystyle\text{sim}(q^{\prime},k^{\prime})=\phi^{T}(q^{\prime})\phi(k^{\prime}).$

Here, $\phi(\cdot)$ represents a quadratic function. The model can learn any
quadratic function with a determined minimum value by adjusting its
parameters. We can, therefore, simplify the kernel function to

$\phi(x)={x}^{2}.$

Incorporating the affine transformation into the kernel function, we obtain

$\displaystyle\phi(q^{\prime})$ $\displaystyle=(\gamma_{Q}\cdot
q+\beta_{Q})^{2}=\gamma_{Q}^{2}q^{2}+2\gamma_{Q}\beta_{Q}q+\beta_{Q}^{2},$
$\displaystyle\phi(k^{\prime})$ $\displaystyle=(\gamma_{K}\cdot
k+\beta_{K})^{2}=\gamma_{K}^{2}k^{2}+2\gamma_{K}\beta_{K}k+\beta_{K}^{2}.$

where $q$ and $k$ have their unique parameters $\gamma_{Q}$, $\gamma_{K}$,
$\beta_{Q}$, and $\beta_{K}$, enabling the model to learn any quadratic
function that is non-negative and has a single real root.

Interestingly, our transformation resembles the application of Layer
Normalization (Ba et al., 2016), minus the normalization itself. We
hypothesize whether normalizing $q$ and $k$ before the kernel function could
improve the model’s performance. Our suspicion is confirmed when normalization
enhances results, as demonstrated in a later Ablation study. Consequently, our
refined ReBased model incorporates Layer Normalization.

Figure 3: Ablation study for the proposed modifications. For sequence length
$256$, the difference is not very significant. Nevertheless, the ReBased model
performs best on all model dimensions. With a sequence length of $2048$, the
difference becomes more evident. Unlike Based, the ReBased model retains
performance across long and short sequences. See Section 5.3 for the
experiment setup and extended description of our results and Section 5.1 for
experimental setup description.

In the following sections, we provide an in-depth analysis and conduct
comprehensive experiments to validate the effectiveness of these
modifications.

##  5 Experiments

###  5.1 Experimental Setup

We applied the first evaluation of our ReBased model on the MQAR task, for
which we trained a model to perform associative recall with varying numbers of
retrieved tokens. Arora et al. (2023) suggested that for a comprehensive
assessment, models need to be tested across different sequence lengths, model
sizes, and number of query-key pairs to be retrieved. However, those
experiments were limited, only exploring sequence lengths up to $512$. These
constraints resulted in the Based model displaying performance comparable to
the traditional attention mechanism.

Longer sequence lengths can be explored to gain a deeper understanding of how
improvements in associative recall are affected by changes in model
configurations. This is why we extended our training to include models capable
of handling sequence lengths of $[128,256,512,1024,2048]$. We tested a range
of hidden sizes from $64$ to $512$. For our ablation study to yield more
precise insights, we also employed smaller models with hidden sizes as modest
as $[16,24,32,48]$.

In order to tailor our approach to varied sequences, we used different query-
key (qk) pairs for each length. The specifics of these configurations are
detailed in Appendix A.

We also put other sub-quadratic architectures to the test, including Mamba
(SSM family) (Gu and Dao, 2023), Hyena (the long convolutions family) (Poli et
al., 2023), the vanilla attention method, and RWKV (Peng et al., 2023). By
comparing a diverse range of models, our goal was to present a well-rounded
evaluation of how our ReBased model stands out in the field. For Based, we
utilized Triton kernels published by Yang and Zhang (2024), and for ReBased,
we modified it so that $\phi(x)=x^{2}$.

We used a hybrid architecture with short convolution and kernel size 3 in the
first layer, and specified a mixer in the second. We found that this setup was
more stable on longer sequence lengths, especially when using an attention
mixer. However, we did not modify the Mamba model since convolutions were
already present inside the Mamba block. We put the full results and model
architecture details in Appendix A.

| Sequence Length 256 |  | Sequence Length 2048 |
---|---|---|---|---
Architecture | $16$ | $24$ | $32$ | $48$ | Mean | $16$ | $24$ | $32$ | $48$ | Mean
Based | $0.06\pm 0.02$ | $0.45\pm 0.15$ | $0.82\pm 0.06$ | $0.99\pm 0.00$ | 0.58 | $0.02\pm 0.02$ | $0.40\pm 0.18$ | $0.66\pm 0.08$ | $0.99\pm 0.01$ | 0.51
$x^{2}$ | $0.05\pm 0.05$ | $0.47\pm 0.08$ | $0.64\pm 0.42$ | $0.99\pm 0.00$ | 0.54 | $0.03\pm 0.03$ | $0.33\pm 0.21$ | $0.46\pm 0.42$ | $0.93\pm 0.09$ | 0.44
$norm(x)^{2}$ | $0.09\pm 0.04$ | $0.39\pm 0.24$ | $0.84\pm 0.09$ | $0.97\pm 0.02$ | 0.57 | $0.05\pm 0.05$ | $0.56\pm 0.10$ | $0.72\pm 0.17$ | $0.99\pm 0.00$ | 0.58
$(\gamma\cdot x)^{2}$ | $0.02\pm 0.02$ | $0.35\pm 0.22$ | $0.71\pm 0.09$ | $0.98\pm 0.03$ | 0.51 | $0.02\pm 0.03$ | $0.26\pm 0.45$ | $0.65\pm 0.37$ | $0.99\pm 0.01$ | 0.48
$(\gamma\cdot x+\beta)^{2}$ | $0.06\pm 0.01$ | $0.51\pm 0.08$ | $0.89\pm 0.03$ | $0.99\pm 0.00$ | 0.61 | $0.06\pm 0.03$ | $0.50\pm 0.08$ | $0.85\pm 0.04$ | $0.99\pm 0.01$ | 0.60
| ReBased
---
$(\gamma\cdot norm(x)+\beta)^{2}$
$0.09\pm 0.05$ | $0.59\pm 0.06$ | $0.86\pm 0.08$ | $0.99\pm 0.00$ | 0.63 | $0.04\pm 0.03$ | $0.58\pm 0.01$ | $0.83\pm 0.04$ | $0.99\pm 0.00$ | 0.61

Table 1: Ablation study for proposed modifications with standard deviation
across $5$ seeds. See Figure 3 for a visual presentation of the results,
Section 5.3 for experiment setup and extended result description and Section
5.1 for the description of our experimental setup.

In language modeling, our second experimental setup leveraged the extensive
Pile dataset (Gao et al., 2020) to train a language model (LM). We opted for a
sequence length of 4096, a slight increase from the standard value while still
ensuring the replication of the architectural framework as presented by Arora
et al. (2023)222The experiment details can be found in a blog post and WandB
report associated with the main paper.. Note that some hyperparameters such as
model dimension and the number of layers were set in order to match the number
of model parameters in the initial experiment. Detailed model configuration
can be found in Appendix B.

The MQAR task provided insights into In-Context Learning proficiencies across
various architectures, while the language modeling assessment allowed us to
appraise short-term dependency modeling capacities. Beyond traditional
perplexity metrics on validation data, we also scrutinized the Associative
(AR) and Non-Associative (Non-AR) variants of perplexity. Here, AR corresponds
to token positions necessitating associative recall, while Non-AR refers to
other tokens. When tokens recur within a text, the subsequent appearances are
categorized as AR, highlighting the model’s capability to recall from context.

###  5.2 MQAR experiment

In Figure 1, we present the capability of various models to handle the MQAR
task as the sequence length increases. One key observation is that, at a
sequence length of 2048, all models, except for the Attention model, struggled
to perform effectively when limited to a model dimension of 64. As we expanded
the model dimensions, the performance of the ReBased model matched or
surpassed the Based model. The RWKV and Mamba architectures failed on the MQAR
task across all tested model sizes.

This experiment highlights the significance of utilizing more sophisticated
setups, as the performance discrepancy between the Attention model and the
other models (Based and ReBased) becomes pronounced only when the sequence
length exceeds 512. These results suggest that the efficacy of attention
alternatives like ReBased becomes particularly important when processing long
sequences. Therefore, more consideration should be devoted to configurations
involving lengthy sequences to leverage the full potential of such models.

###  5.3 Ablation Study

We comprehensively examined the individual elements of our ReBased model to
understand how each of them contributes to its overall effectiveness, and
ensure the transparency of our findings. Our experiments were meticulously
designed to evaluate the model by assessing the influence of its separate
components on performance. The experimental configurations were as follows:

  * •

$x^{2}$ – substituting the original kernel function with a simple element-wise
squaring operation, $\phi(x)=x^{2}$.

  * •

$norm(x)^{2}$ – integrating a normalization step without an affine
transformation before applying the squaring operation.

  * •

$(\gamma\cdot x)^{2}$ – introducing an affine transformation solely in terms
of scaling (without bias) for the queries and keys.

  * •

$(\gamma\cdot x+\beta)^{2}$ – incorporating affine transformation with both
scaling and bias for the queries and keys.

  * •

ReBased $(\gamma\cdot norm(x)+\beta)^{2}$ – our comprehensive model, which
involves normalization and affine transformation, including bias, for queries
and keys.

Figure 4: Expected validation accuracy across different hyperparameters. The
ReBased model works best across all hyperparameters, budgets, and model
dimensions. See Section 5.3 for more details.

Note that for $q$ and $k$, there are different scaling parameters
$\gamma_{Q}$, $\beta_{Q}$, $\gamma_{K}$, and $\beta_{K}$ for each experiment
involving affine transformation.

Our goal is to highlight the effect of sequence length variability in the MQAR
task on model performance. For this evaluation, we standardized the number of
retrieval pairs to 32. Theoretically, no impact on performance should be
observed, as the amount of information required to be stored in the hidden
states is sequence-length agnostic. We investigated the effects on sequences
of lengths 256 and 2048 and illustrated our findings in Figure 3 (also
available in Table 1 with a standard deviation of accuracy across $5$ seeds).
We must emphasize the significance of long context setups evaluated in our
experiments. Its characteristics are vital, as successful performance on long
sequences highlights the capability of the model to make full use of its
architectural innovations. It also translates into notable practical
advantages in real-world applications where handling extensive context
efficiently can be crucial.

The proposed ReBased model performs better than every other modification.
Performance on the short $256$ length is less noticeable than on the long
$2048$ sequence length. We see a performance drop from simply replacing the
original kernel function with $x^{2}$. We presume that this is caused by
suboptimal scale of features, since by placing normalization before the kernel
function, we can notice a performance increase even in comparison to the Based
model. Affine transformations $(\gamma\cdot x)^{2}$ and $(\gamma\cdot
x+\beta)^{2}$ also show favorable performance compared to the $x^{2}$ model,
which does not significantly decrease with sequence length.

In our experiments, we found ReBased to be more stable during training with
various hyperparameters. To demonstrate this, we utilize an Expected
Validation Performance (EVP) plot (Dodge et al., 2021). We treat the average
across five seeds as the final accuracy. Our results are presented in Appendix
Figure 4. We noticed that even in cases where the model dimension is
sufficiently large to store all necessary information, our modifications lead
to $100\%$ accuracy for every hyperparameter set and every seed we use, in
contrast with the Based model, where we observe degradation for certain
learning rates.

###  5.4 Language Modeling

| Perplexity
---|---
Architecture | All | AR | Non-AR
Attention | 16.62 | 3.57 | 58.57
Based | 16.69 | 3.72 | 57.21
ReBased | 16.45 | 3.67 | 56.31

Table 2: Perplexity results on Pile Gao et al. (2020) dataset. ReBased
improves the result on AR tokens. However, there is still a small gap between
Attention and ReBased. See Section 5.4 for more details and Section 5.1 for
experimental setup description.

We conducted experiments with language modeling following the setup described
in Section 5.1. See Table 2 for the results.

When considering AR perplexity, we observe that there is still a gap between
the vanilla Transformer architecture and alternative models, which is aligned
with the results on the MQAR dataset. However, we note that ReBased still
performed better than Based. Regarding Non-AR perplexity, ReBased outperformed
both Based and Transformer architectures, leading to better overall perplexity
value.

These results suggest that, despite language modeling perplexity being lower
for an alternative to Transformer architectures (Arora et al., 2023; Gu and
Dao, 2023), this may be achieved due to better short-term dependency modeling,
which does not require learning associative operations necessary to perform
In-Context Learning (Olsson et al., 2022). The vanilla Transformer still
performs best in terms of its ability to attend to some token in-context.

###  5.5 Analysis

Figure 5: Attention matrix for the different models, and ground truth
positions for the query. We measure IoU between model’s attention and ground
truth matrix for $10000$ examples. Illustration of the experiment is described
in Section 5.5 Results are presented in Table 3.

In this section, we delve into the internal dynamics of the ReBased model by
examining attention matrices, which are commonly used to elucidate the
decision-making of models and the flow of information between tokens. Notably,
we can use the parallel mode with both Based and ReBased models to construct
these matrices.

For our analysis, we employ the MQAR dataset Arora et al. (2023) and train a
compact model configured with a sequence length of 128 and 32 retrieval pairs.
To ensure clear interpretation of the attention maps, we used fixed weights in
the first layer, which consists of a short convolution with a kernel that
attends to the previous token. Following the training phase, we compute the
Intersection over the Union (IoU) metric between the attention matrix and the
actual positions of the tokens that are to be retrieved. The correct positions
are crucial, as they represent the locations from which the model must copy
the hidden states in order to successfully resolve the task. This copying
mechanism is particularly vital and is implemented via focused attention in
the second layer of the network Olsson et al. (2022). Consequently, the IoU
provides a quantitative measure of how well our model has learned to replicate
this crucial pattern of token retrieval. A visualization of this phenomenon
using IoU on a randomly selected example from the dataset is shown in Figure
5. Note that we cropped attention matrix to examine only a region where qk-
pairs stored.

Architecture | IoU | Accuracy
---|---|---
Attention | 0.999 | 1
Based | $0.157$ | 0.956
ReBased | 0.173 | 0.957

Table 3: IoU with attention matrix and ground truth position to retrieve on
the MQAR task for $10000$ examples. Detailed experiment setup can be found in
5.5.

Our results are presented in Table 3. In our experiment, the Attention model
yielded a superior IoU score compared to both the Based and ReBased models.
However, the ReBased model shows promise in narrowing the performance divide
that exists between sub-quadratic methods and the attention-based model. This
suggests that, despite the relative simplicity of the method, it could serve
as an informative metric for the MQAR dataset, particularly when the accuracy
score is close to one, making it challenging to discern the performance
differences between models in more intricate testing scenarios.

##  6 Conclusion and Future Work

In this paper, we present ReBased, a novel architecture for sub-quadratic
attention computation. For our model, we analyzed the Base architecture and
proposed to develop it even further by using polynomial kernels with learnable
parameters and adding normalization before the kernel evaluation. While
incorporating layer normalization into model training was attempted previously
(Dehghani et al., 2023), our method integrates this normalization directly
into the kernel function. With this simple architectural change, we achieved
results that outperformed Based on MQAR and language modeling with the Pile
dataset tasks.

We analyzed the internal representations of ReBased, Based, and vanilla
attention modules, and concluded that ReBased resembles attention more than
Based. Notably, while Based uses a Taylor expansion of an exponential
function, a ReBased kernel function is different from the exponent but shows
better performance. Our research suggests that using a second-order polynomial
might be insufficient for the best performance, and indicates that more
sophisticated learnable kernels could be utilized to improve the performance
of trained models. Normalization could further improve various kernel
functions. This highlights a need for researchers to revisit classical kernel-
based methods with the goal of enhancing their adaptability and efficiency.

Our findings reveal a significant disparity in handling the MQAR task between
attention-based models and others such as Based, specifically as sequence
lengths increase. Attention models excel on longer sequences, significantly
outperforming their non-attention counterparts. These results highlight the
necessity of further research into strategies that could bridge this gap in
order to reach the performance of attention-based methods. Perhaps the
superior aspects of attention mechanisms could be matched or exceeded by other
models, especially on tasks that require associative recall, such as machine
translation Vardasbi et al. (2023). Future research could give insight into
this, leading to improved models for processing long sequences on various NLP
tasks.

##  7 Limitations

While our proposed method demonstrates applicability to a wide range of tasks
typically addressed by Transformers, its effectiveness in handling tasks
involving intensive copying or recalling previous context remains unclear (see
Table 2 and Jelassi et al. (2024)). Successfully addressing these tasks is
crucial for fully mitigating inference problems associated with attention
mechanisms.

It is also worth noting that our experiments are limited to academic-scale
models. This does pose certain limitations, particularly in extrapolating the
findings to larger models. However, given the resource constraints, our
results still provide valuable insights into the potential efficacy of our
method.

## References

  * Amos et al. (2023) Ido Amos, Jonathan Berant, and Ankit Gupta. 2023.  Never train from scratch: Fair comparison of long-sequence models requires data-driven priors.  _CoRR_ , abs/2310.02980.
  * Arora et al. (2023) Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. 2023.  Zoology: Measuring and improving recall in efficient language models.  _arXiv:2312.04927_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer normalization.  Cite arxiv:1607.06450.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  Cite arxiv:1412.3555Comment: Presented in NIPS 2014 Deep Learning and Representation Learning Workshop.
  * Dehghani et al. (2023) Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, J. Heek, J. Gilmer, A. Steiner, Mathilde Caron, Robert Geirhos, Ibrahim M. Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, M. Tschannen, Anurag Arnab, Xiao Wang, C. Riquelme, Matthias Minderer, J. Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, F. Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, A. Gritsenko, Vighnesh Birodkar, C. Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Paveti’c, Dustin Tran, Thomas Kipf, Mario Luvci’c, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and N. Houlsby. 2023.  Scaling vision transformers to 22 billion parameters.  _International Conference on Machine Learning_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dodge et al. (2021) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2021.  Expected validation performance and estimation of a random variable’s maximum.  _CoRR_ , abs/2110.00613.
  * Fu et al. (2023a) Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. 2023a.  Hungry Hungry Hippos: Towards language modeling with state space models.  In _International Conference on Learning Representations_.
  * Fu et al. (2023b) Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. 2023b.  Simple hardware-efficient long convolutions for sequence modeling.  _International Conference on Machine Learning_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gu and Dao (2023) Albert Gu and Tri Dao. 2023.  Mamba: Linear-time sequence modeling with selective state spaces.
  * Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Re. 2022.  Efficiently modeling long sequences with structured state spaces.  In _International Conference on Learning Representations_.
  * Gu et al. (2023) Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. 2023.  How to train your HIPPO: State space models with generalized orthogonal basis projections.  In _International Conference on Learning Representations_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Jelassi et al. (2024) Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. 2024.  Repeat after me: Transformers are better than state space models at copying.  _arXiv preprint arXiv: 2402.01032_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020.  Transformers are RNNs: Fast autoregressive transformers with linear attention.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5156–5165. PMLR.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022.  In-context learning and induction heads.  _Transformer Circuits Thread_.  Https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
  * Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. 2023.  RWKV: Reinventing RNNs for the transformer era.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14048–14077, Singapore. Association for Computational Linguistics.
  * Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021.  Random feature attention.  _ArXiv_ , abs/2103.02143.
  * Poli et al. (2023) Michael Poli, Stefano Massaroli, Eric Q. Nguyen, Daniel Y. Fu, Tri Dao, S. Baccus, Y. Bengio, Stefano Ermon, and Christopher Ré. 2023.  Hyena hierarchy: Towards larger convolutional language models.  _International Conference on Machine Learning_.
  * Qin et al. (2022) Zhen Qin, Weixuan Sun, Huicai Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022.  cosformer: Rethinking softmax in attention.  _ArXiv_ , abs/2202.08791.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Schlag et al. (2021) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. 2021.  Linear transformers are secretly fast weight programmers.  In _International Conference on Machine Learning_.
  * Smith et al. (2023) Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. 2023.  Simplified state space layers for sequence modeling.  In _The Eleventh International Conference on Learning Representations_.
  * Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024.  Roformer: Enhanced transformer with rotary position embedding.  _Neurocomputing_ , 568:127063.
  * Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021.  Long range arena : A benchmark for efficient transformers.  In _International Conference on Learning Representations_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, D. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, A. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Vardasbi et al. (2023) Ali Vardasbi, Telmo Pessoa Pires, Robin M. Schmidt, and Stephan Peitz. 2023.  State spaces aren’t enough: Machine translation needs attention.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Finland, 12-15 June 2023_ , pages 205–216. European Association for Machine Translation.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Yang and Zhang (2024) Songlin Yang and Yu Zhang. 2024.  Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism.

(a) Based architecture

(b) ReBased architecture.

Figure 6: Architectures visualization.

Model Dimension | Attention | ConvAttention | RWKV | ConvRWKV | Mamba | Based (Rebased)
---|---|---|---|---|---|---
64 | 623744 | 578752 | 623872 | 677120 | 655360 | 577984 $(+768)$
128 | 1313024 | 1149312 | 1313280 | 1395200 | 1413120 | 1179520 $(+768)$
256 | 2888192 | 2462464 | 2888704 | 2561024 | 3235840 | 2459392 $(+768)$
512 | 6824960 | 5580288 | 6825984 | 5777408 | 7847936 | 5307904 $(+768)$

Table 4: Number of model parameters in MQAR dataset. See Appendix A.

##  Appendix A Details for the MQAR dataset experiments.

In our experiments, we use the code from the official MQAR repository (Arora
et al., 2023)333https://github.com/HazyResearch/zoology. However, we modify
the attention model from the one reported in Arora et al. (2023), as we found
it more stable (see Figure 7). We can see that replacing the first attention
layer is beneficial for performance. RWKV performs better when we do not
replace the first layer, which is why we use we use two RWKV layers in our
main experiment (see Figure 1). We report the number of trainable parameters
in Table 4.

Figure 7: Performance of the hybrid architecture with convolutions on the
first layer and the vanilla architecture.

We also modify the data configs to be more challenging for the model. You can
see the adjusted parameters in Table 5.

seq_length | qk_pairs
---|---
128 | 16
256 | 64
512 | 128
1024 | 256
2048 | 512
Table 5: Sequence lengths and number of QK pairs in dataset.

We use a batch size of 512 for all experiments. In cases where there is not
enough GPU memory, we use the gradient accumulation technique. For the
learning rate, we use hyperparameter search with the following grid: 5e-4,
1e-3, 3e-3, 1e-2. We use five different seeds for all reported results.

##  Appendix B Pile Dataset Experiment Details

We train our model on the tokenized Pile dataset published on huggingface
hub444https://huggingface.co/datasets/EleutherAI. Note that this tokenization
differs from the one used in Based555See report. We also use our pipeline,
which we plan to release to the public soon. We do not use rotary positional
embeddings Su et al. (2024) or other tricks, as we copy our models from the
Based repository. Hyperparameters can be found in Table 6.

Hyper-Parameter | Value
---|---
warmup steps | 200
max grad norm | 1
num steps | 20000
seq len | 4096
lr | 1e-3
weight decay | 0.1
num heads | 12
d model | 768
effective batch size | 1024
Table 6: Hyperparameters used for training.

As in Arora et al. (2023), we use more hyperparameters in the Based/ReBased
models then in the Attention baseline. The number of layers and head dim
reported in Tables 7 and 8. We use a hybrid architecture for the Based/ReBased
models where we use short convolution as a mixer for every odd-numbered layer.

Hyperparameters | Value
---|---
layers | 12
head dim | 64
Table 7: Attention hyperparameters. Hyper-Parameter | Value
---|---
layers | 18
head dim | 16
Table 8: Based/ReBased hyperparameters. Figure 8: Attention matrix for the different models, and ground truth positions for the query. We measure IoU between the model’s attention and ground truth matrix for $10000$ examples. llustration of the experiment is described in Section 5.5 Results are presented in Table 3. Model | # Parameters
---|---
Attention | 126 807 552
Based | 156 955 392
ReBased | 156 992 256
Table 9: Parameters count for the pile experiment. See Appendix B.

##  Appendix C Additional Analysis

Figure 9: Attention scores for a random example. Based and Rebased scores are
noisy, while attention has one peak at the ground truth position. See Appendix
A.

In this section, we provide additional results and experiments to further
understand how the ReBased model learns dependencies. First, we offer more
examples for our experiments with attention matrices, as detailed in Section
5.5. Attention matrices for random examples from the test set are presented in
Figure 8. Generally, we can observe that attention "fires" more intensely at
retrieving tokens compared to Based/ReBased. This result suggests that there
may still be a flaw in our kernel function that distributes attention to
irrelevant tokens. We further investigate this phenomenon by analyzing the
distribution of attention on the last token, as shown in Figure 9. The number
of noisy tokens for the ReBased model is smaller compared to Based, but
Attention exhibits superior results.

Layer normalization is the main difference between the Based and ReBased
models. Therefore, it is important to analyze the parameters obtained during
the training process. We logged the mean and standard deviation of the
parameters across different sequence lengths. Our results can be found in
Figure 10. Notably, the final parameter value is independent of the training
sequence length, which can indicate that we may not need to train the model
for all possible lengths. Both $\gamma$ and $\beta$ parameters have high
standard deviation values compared to the mean absolute value. Consequently,
we can assume that it is important to provide features with different scales.

Figure 10: Analysis of the layer normalization parameters. Mean value of the
scale parameter (gamma) tends to the same value of about $0.13$, and the bias
parameter (beta) tends to $0$. See Section C.