${}^{*}Q_{1}$-equivariant and asymptotically ${}^{*}Q_{2}$-equivariant. Since
$G$ is boundedly generated by elements of $Q_{1}$ and $Q_{2}$, this implies
that
$\alpha\in\mathcal{L}_{b}^{\infty}(({}^{*}(G/P))^{2},\mathcal{V})^{\sim{}^{*}G}$.
∎

While the above theorem assumes that $\Gamma$ is a lattice in a semisimple
group $G$ that is simply connected, we can extend this as follows:

######  Theorem 6.2.6.

Let $\Gamma$ be a lattice in a semisimple group
$G=\prod^{k}_{i=1}{\mathbf{G}_{i}}(K_{i})$ (where for $1\leq i\leq k$, $K_{i}$
is a local field, and $\mathbf{G}_{i}$ is a connected almost $K_{i}$-simple
group) that has Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$. Then $\Gamma$
is uniformly $\mathfrak{U}$-stable with a linear estimate.

###### Proof.

Let $\hat{G}$ be the universal cover of $G$, which is a semisimple simply
connected group, and let $\hat{\Gamma}$ be the preimage of $\Gamma$ in
$\hat{G}$. Note that $\Gamma$ and $\hat{\Gamma}$ are commensurable, hence
Remark 3.0.3 tells us that it is sufficient to prove that $\hat{\Gamma}$ is is
uniformly $\mathfrak{U}$-stable with a linear estimate. Note that since the
fundamental group of $G$ is abelian, from Lemma 6.2.3 it follows that
$\hat{G}$ has Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$. Hence we can now
apply Theorem 6.2.5 to $\hat{\Gamma}\leq\hat{G}$ to conclude that
$\operatorname{H}_{a}^{2}(\hat{\Gamma},\mathcal{W})=0$, implying that
$\hat{\Gamma}$ is uniformly $\mathfrak{U}$-stable with a linear estimate. ∎

###  6.3 Groups with Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$

In this subsection, we shall list classes of groups that satisfy
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ used in the hypothesis of
Theorem 6.2.5. Since Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ involves
the existence of two proper parabolic subgroups with the 2½-property, we first
list classes of groups known to have this property.

#### Simple Groups with the 2½-property

We shall collect below the necessary statement from the existing literature,
and complement them by an additional argument in the non-Archimedan case, to
establish that a number of natural semisimple groups have the 2½-property.
Let us first consider simple groups over a non-archimedean field. Here, the
vanishing of $\operatorname{H}_{b}^{2}(G,\mathbf{R})$ was established in
[BM99]. More precisely, this reference establishes the injectivity of the
comparison map (and we recall that the case of trivial coefficients
$\mathbf{R}$ for the semisimple group $G$ was actually the easy part of this
result). On the other hand, the vanishing of the usual cohomology is well-
known in this setting, see e.g. [BW00].
Therefore, what we need to justify is the condition on
$\operatorname{H}_{b}^{3}(G,\mathbf{R})$. It turns out that this vanishes:
this result is established by the inductive method introduced in [Mon10], the
basis of the induction being provided by [BM19].

######  Theorem 6.3.1.

For $G=\mathbf{G}(K)$, where $K$ is a non-Archimedean local field and
$\mathbf{G}$ is a connected, simply connected, semisimple $K$-group, the
bounded cohomology $\operatorname{H}_{b}^{3}(G,\mathbf{R})$ vanishes (and is
therefore Hausdorff).

###### Proof.

The proof is by induction on the $K$-rank of $\mathbf{G}$. The case of rank
zero corresponds to $G=\mathbf{G}(K)$ being compact, and thus having trivial
bounded cohomology in all degrees. The induction really starts with rank one.
In that case, $G$ is an automorphism group of a Bruhat–Tits tree satisfying
the assumptions of the main result of [BM19], which states that
$\operatorname{H}_{b}^{n}(G,\mathbf{R})$ vanishes for all $n>0$.
We now perform the induction step for $\mathbf{G}$ of $K$-rank $r\geq 2$. We
use (a minor variation of) the spectral sequence introduced in [Mon10, §6.A]
as follows. We consider the Tits building $T$ of $\mathbf{G}$ over $K$ and
denote by $T^{(d)}$ its set of $d$-simplices. Thus $T^{(d)}$ is defined for
all $d\leq r-1$ and is topologized by identifying it with the union of
homogeneous spaces $G/P_{I}$, where $P_{I}$ ranges over standard parabolic
subgroups of semisimple rank $r-1-d$ (see [Mon10] for more details). By
convention, we take the _augmented_ simplicial complex, namely we also
consider the one-point space of negative simplices $T^{(-1)}=G/G$. In this
set-up, an appropriate version of the Solomon-Tits theorem implies that the
following sequence of spaces of continuous functions is exact:

$0\to C(T^{(-1)})\to C(T^{(0)})\to\cdots\to C(T^{(r-1)})$

see Theorems 2.7 and 3.9 in [Mon10]. We denote by $\mathrm{St}_{G}$ the
cokernel of the last map above, which is one version of the Steinberg
representation of $G$. Finally, the spectral sequence that we consider is the
first quadrant hypercohomology spectral sequence obtained by computing the
bounded cohomology of $G$ with coefficients in the following complex of Banach
$G$-modules

$0\to C(T^{(-1)})\to C(T^{(0)})\to\cdots\to C(T^{(r-1)})\to\mathrm{St}_{G}\to
0$ (6.1)

Concretely, the first page of the spectral sequence is by definition
$E^{p,q}_{1}=\operatorname{H}_{b}^{q}\left(G,C(T^{(p-1)})\right)$ when $p\leq
r$, for $p=r+1$ it is
$E^{r+1,q}_{1}=\operatorname{H}_{b}^{q}(G,\mathrm{St}_{G})$, and for $p>r+1$
it is zero. The only difference with [Mon10, §6.A] is that the complex was
truncated at $T^{(r-1)}$ there, not completing it with $\mathrm{St}_{G}$. A
crucial point to allow this definition is that $\mathrm{St}_{G}$ is Hausdorff,
which follows from an alternative description of the Steinberg representation
given by Borel–Serre, see Remark 2.8 in [Mon10].

This spectral sequence abuts to zero since the above complex is exact. By
cohomological induction, we have as in Theorem 6.1 of [Mon10] the
identifications

$E^{p,q}_{1}=\operatorname{H}_{b}^{q}\left(G,C(T^{(p-1)})\right)\cong\bigoplus\operatorname{H}_{b}^{q}(G_{I},\mathbf{R})\kern
19.91692pt(\forall\,p\leq r,\forall q)$ (6.2)

where $G_{I}$ is the semisimple part of the parabolic subgroup $P_{I}$ and the
sum is taken over all such subgroups of semisimple rank $r-p$. Notice that the
rank is indeed $r-p$ and not $r-1-p$ since we started with $T^{(-1)}$
corresponding to $p=0$.

Our goal is to prove $E^{0,3}_{1}=0$ since this is
$\operatorname{H}_{b}^{3}(G,\mathbf{R})$. We have $E^{0,3}_{1}=E^{0,3}_{2}$
since the right hand side is the kernel of the map $E^{0,3}_{1}\to
E^{1,3}_{1}$ which vanishes by Eq. 6.2 and the inductive hypothesis. Next,
$E^{0,3}_{2}=E^{0,3}_{3}$ because the right hand side is the kernel of the map
$E^{0,3}_{2}\to E^{2,2}_{2}$ and already $E^{2,2}_{1}$ vanishes by Eq. 6.2 and
the general vanishing of $\operatorname{H}_{b}^{2}$ mentioned earlier.

The next differential to consider is $E^{0,3}_{3}\to E^{3,1}_{3}$. If $r\geq
3$, we can still apply Eq. 6.2 and the general vanishing of
$\operatorname{H}_{b}^{1}$ to conclude $E^{0,3}_{3}=E^{0,3}_{4}$. If however
$r=2$, then we reach the same conclusion provided we justify that
$\operatorname{H}_{b}^{1}(G,\mathrm{St}_{G})$ vanishes. To this end, we first
observe that the comparison map to ordinary cohomology is always injective in
degree one. On the other hand, the ordinary first cohomology of $G$ with
values in the Steinberg representation is known to vanish if (and only if!)
the rank of $G$ is not one, see e.g. Theorem 4.12 p. 205 in [BW00]. Thus, in
either case we have established $E^{0,3}_{3}=E^{0,3}_{4}$.

The final differential is $E^{0,3}_{4}\to E^{4,0}_{4}$. But we have already
second page vanishing of all $E^{p,0}_{2}$. Indeed, by Eq. 6.2 this statement
amounts to the acyclicity of the subcomplex of $G$-invariants of Eq. 6.1.
Ignoring the last term $(\mathrm{St}_{G})^{G}$ at first, this comes from the
fact that the $G$-orbits in the Tits building form by definition a full
simplex of dimension $r-1$ (augmented in dimension $-1$), which is hence
acyclic. It remains to argue acyclicity of the last term

$C(T^{(r-1)})^{G}\to(\mathrm{St}_{G})^{G}\to 0$

which amounts to showing that $(\mathrm{St}_{G})^{G}$ vanishes. This follows
e.g. by realizing the Steinberg representation as a space of $L^{2}$ harmonic
maps on the Bruhat–Tits building, see [Kli04] for a concrete description of
this isomorphism (also due to Borel–Serre).

In conclusion, we have shown that $E^{0,3}_{1}=E^{0,3}_{4}=E^{0,3}_{\infty}$
holds. Since the spectral sequence converges to zero, this establishes as
desired the vanishing of $E^{0,3}_{1}=\operatorname{H}_{b}^{3}(G,\mathbf{R})$.
∎

Next, let us consider simple groups over $\mathbf{R}$ or $\mathbf{C}$.
Regarding $\operatorname{H}_{b}^{2}(G,\mathbf{R})$, the injectivity of the
comparison map was established for all connected semisimple Lie groups $G$ in
[BM99]. Therefore, the corresponding vanishing holds in all cases where the
usual second bounded cohomology vansihes, which is always the case for
$G=SL_{n}(\mathbf{C})$ and is the case for $G=SL_{n}(\mathbf{R})$ if and only
if $n\neq 2$. It therefore remains to collect the following results from the
existing literature:

######  Theorem 6.3.2.

Let $n\geq 2$.

  1. 1.

For $G=SL_{n}(\mathbf{R})$, the bounded cohomology
$\operatorname{H}_{b}^{3}(G,\mathbf{R})$ vanishes (and is therefore
Hausdorff).

  2. 2.

For $G=SL_{n}(\mathbf{C})$, the bounded cohomology
$\operatorname{H}_{b}^{3}(G,\mathbf{R})$ is one-dimensional and Hausdorff.

###### Proof.

The case of $SL_{n}(\mathbf{R})$ was established for $n=2$ in Theorem 1.5 of
[BM02] and for general $n$ in Theorem 1.2 of [Mon04].

Regarding $G=SL_{n}(\mathbf{C})$, these two references established that the
comparison map is an isomorphism from $\operatorname{H}_{b}^{3}(G,\mathbf{R})$
to $H^{3}(G,\mathbf{R})$, the latter being classically known to be one-
dimensional (see [BM02, Theorem 1.2] and [Mon04, Remark 3.5]). In our context,
the only relevant point (and the hard one anyway) is that the comparison map
is injective. Indeed, it is a general fact for any group and any degree $d$
that the injectivity of the comparison map $\operatorname{H}_{b}^{d}\to H^{d}$
implies that $\operatorname{H}_{b}^{d}$ is Hausdorff. This can be seen by
combining Theorems 2.3 and 2.8 in [MM85]. ∎

We now summarize the above results with the following list of simple groups
that we know to have the 2½-property:

######  Theorem 6.3.3.

The following groups $G$ have the 2½-property:

  1. 1.

$G=\mathbf{G}(k)$, where $k$ is a non-Archimedean local field and $\mathbf{G}$
is a connected semisimple $k$-group.

  2. 2.

$G=SL_{n}(\mathbf{R})$ for any $n\neq 2$.

  3. 3.

$G=SL_{n}(\mathbf{C})$ for any $n$.

#### From the 2½-property to Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$

We can now list out simple groups having
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ using the results discussed
earlier in this section. Again, we begin with simple groups, and then extend
the results to semisimple groups using Proposition 6.2.2.
Let us first consider the case of a simple group $G$ over a non-archimedean
field, and $Q\leq G$ be a parabolic subgroup. Note that, using Theorem 6.3.3,
Lemma 6.2.3 and Proposition 6.2.2, we can conclude that $Q$ has the
2½-property (since, modulo its amenable radical, it is a semisimple group).
Thus, since $G$ has rank at least $2$, we can always find two proper parabolic
subgroups, both having the 2½-property, generating $G$. Hence,

######  Proposition 6.3.4.

The group $G=\mathbf{G}(k)$, where $k$ is a non-Archimedean local field and
$\mathbf{G}$ is a connected, simply connected, semisimple $k$-group of rank at
least $2$, has Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$.

In the complex case, since we know that $SL_{n}(\mathbf{C})$ has the
2½-property for every $n$, we can use Dynkin diagrams to explicitly construct
proper parabolic subgroups (which, modulo their amenable radical, would
correspond to $SL_{n}(\mathbf{C})$ that we know has the 2½-property) that
together generate the group, to conclude that the simple group has
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$.

######  Proposition 6.3.5.

Let $G$ be a simple, simply connected, complex Lie group of rank $n\geq 2$.
Then $G$ has Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$.

###### Proof.

Recall that simple, simply connected Lie groups over $\mathbf{C}$ split, and
hence are classified by Dynkin diagrams, so let $X$ be the Dynkin diagram of
$G$. Any subdiagram $Y$ of $X$ corresponds to a parabolic subgroup $Q\leq G$
containing a fixed minimal parabolic subgroup $P\leq G$, such that $Q$ modulo
its amenable radical is a semisimple group corresponding to the subdiagram
$Y$. Furthermore, if the subdiagram $Y$ is of type $A_{n}$ (for $n\geq 1$),
then the parabolic subgroup $Q$ corresponding to $Y$, modulo its amenable
radical, is $SL_{n+1}(\mathbf{C})$, and so, by Theorem 6.3.3 and Lemma 6.2.3,
$Q$ has the 2½-property.
So constructing two proper parabolic subgroups $Q_{1}$ and $Q_{2}$, that both
contain $P$ and generate $G$, is equivalent to choosing two proper subdiagrams
$Y_{1}$ and $Y_{2}$ of $X$ that the union of the vertex sets of $Y_{1}$ and
$Y_{2}$ is the vertex set of $X$. Furthermore, if we can ensure both these
diagrams are of type $A_{n}$, then this gives us two proper parabolic
subgroups with the 2½-property, ensuring that $G$ has
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$. We claim that for any connected
Dynkin diagram $X$ corresponding to a simple complex Lie algebra of rank at
least $2$, we can find two such subdiagrams $Y_{1}$ and $Y_{2}$ both of type
$A_{n}$ (for $n\geq 1$). This can be seen by considering each case separately,
and is illustrated in the figure below. Observe that for the Dynkin diagram of
type $F_{4}$, we construct subdiagrams $Y_{1}$ and $Y_{2}$ both of type
$A_{2}$, while for a Dynkin diagram of any other type, we can always choose
$Y_{1}$ and $Y_{2}$ to be of type $A_{1}$ and $A_{m-1}$ (where $m$ is the rank
of the group $G$). ∎

We next consider the case of connected, simply connected groups over the
reals, where the situation is more involved. Firstly, note that even
$SL_{3}(\mathbf{R})$ (which, by Theorem 6.3.3, has the 2½-property) does not
have Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ even though it has rank
$2$, because any proper parabolic subgroup of $SL_{3}(\mathbf{R})$, modulo its
amenable radical, is $SL_{2}(\mathbf{R})$ for which
$\operatorname{H}_{b}^{2}(SL_{2}(\mathbf{R}),\mathbf{R})\neq 0$. However,
since $SL_{n}(\mathbf{R})$ has the 2½-property for $n\geq 3$, we can apply the
proof technique of Proposition 6.3.5 in the case of certain _split_ simple
real Lie groups to show that:

######  Proposition 6.3.6.

Let $G$ be a split simple real Lie group of type $A_{n}$, $D_{n}$ (for $n\geq
3$), $F_{4}$, $E_{6}$, $E_{7}$ or $E_{8}$.Then $G$ has
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$.

###### Proof.

As in the proof of Proposition 6.3.5, we construct two subdiagrams $Y_{1}$ and
$Y_{2}$ of the Satake diagram $X$ of the split simple real group $G$, such
that the subdiagrams whose vertex sets together cover the vertex set of $X$,
such that the simple real Lie groups corresponding to the subdiagrams are both
$SL_{n}(\mathbf{R})$ for some $n\geq 3$. This is illustrated in the figure
below, where the subdiagrams are encircled in grey. ∎

Note that our method will not work for the split simple real Lie groups of
type $C_{n}$ or $G_{2}$. In the case of $C_{n}$, for any two two proper
subdiagrams covering the vertices of the Satake diagram, at least one of them
must either itself be of type $C_{m}$, which corresponds to the simple group
$Sp(2m,\mathbf{R})$ which does _not_ have the 2½-property (as
$\operatorname{H}_{b}^{2}(Sp_{2m}(\mathbf{R}),\mathbf{R})\neq 0$), or of type
$A_{1}$, which corresponds to the simple group $SL_{2}(\mathbf{R})$. In the
case of $G_{2}$, any proper parabolic subgroup is of type $A_{1}$. Thus, split
simple real Lie groups of type $C_{n}$ or $G_{2}$ _do not_ have
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$.

##  7 Conclusions and Discussion

The current paper presents many questions and directions for futher research.
We highlight a few of them now.

  * •

Prove a complete Ulam-stability result for higher rank lattices in semisimple
Lie groups.
One could hope to extend our main results (specifically Theorem 6.2.5) by
considering a larger family of groups for which the
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ is known. However, this has its
limitations since there are groups for which
Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ is simply not true. An important
example of such a group is $SL_{3}(\mathbf{R})$, where for a maximal parabolic
subgroup $Q\leq SL_{3}(\mathbf{R})$,
$\operatorname{H}_{b}^{2}(Q,\mathbf{R})=\operatorname{H}_{b}^{2}(SL_{2}(\mathbf{R}),\mathbf{R})\neq
0$. We can ask if Property-$G(\mathcal{Q}_{1},\mathcal{Q}_{2})$ for the
ambient group is even necessary for Ulam stability, or if it just happens to
be an artifact of our proof technique.

  * •

Surjectivity/Injectivity of a comparison map
$\operatorname{H}_{a}^{2}(\Gamma,\mathcal{W})\mapsto\operatorname{H}_{b}^{2}(\Gamma,\tilde{\mathcal{W}})$.
There exists a natural forgetful map
$\operatorname{H}_{a}^{2}(\Gamma,\mathcal{W})\mapsto\operatorname{H}_{b}^{2}(\Gamma,\tilde{\mathcal{W}})$
(analogous to the comparison map $c:\operatorname{H}_{b}^{2}(\Gamma,W)\to
H^{2}(\Gamma,W)$ for a $\Gamma$-module $W$). It is not immediate if this map
is either surjective or injective. Suppose it were injective, then we could
truly reduce the question of uniform stability with a linear estimate to the
study of the second bounded cohomology group
$\operatorname{H}_{b}^{2}(\Gamma,\tilde{W})$ which is a well-studied notion.
For instance, it is known ([BM99]) that for a lattice $\Gamma$ in a higher
rank simple Lie group $G$, $\operatorname{H}_{b}^{2}(\Gamma,W)=0$ for every
dual separable Banach $\Gamma$-module $W$ (note, however, that the Banach
space ultraproduct $\tilde{W}$ is not separable unless it is finite
dimensional).

  * •

Uniform versus non-uniform stability
All along in this paper, we have dealt with the question of uniform stability
as opposed to non-uniform, or pointwise, stability. The connection between
pointwise stability and vanishing second cohomology is studied in [DCGLT20],
[LO20]. We stress that such stability results in the non-uniform setting is
far from being known for most lattices. So far such results are known for many
lattices in $p$-adic Lie groups (with respect to the Frobenius or $p$-Schatten
norms for $p<\infty$), but almost nothing is known for lattices in real (or
complex) Lie groups (see [BLSW22]). Moreover, when the family
$M_{n}(\mathbf{C})$ is endowed with the operator norm (i.e. the $p$-Schatten
norm for $p=\infty$) then it is known that the stability result is _not_ true
for most hgh rank lattices. This follows from the results in [Dad21] [CGM90]
that show that if $H^{2j}(\Gamma,\mathbf{R})\neq 0$ for some positive integer
$j$, then $\Gamma$ is not pointwise stable for the operator norm. Note that in
our setting of uniform stability, the case of $p=\infty$ and $p<\infty$ are
treated together without any problem.

  * •

Stability with respect to the (normalized) Hilbert-Schmidt norm.
In [BL20] it is shown that a lattice $\Gamma$ that has Property (T) is not
pointwise stable for the (normalized) Hilbert-Schmidt norm. In [AD22] it is
shown that if a residually finite group is uniformly stable with respect to
the (normalized) Hilbert-Schmidt norm, then it is virtually abelian. In
particular, this means that no lattice in a non-compact semisimple Lie group
is uniformly stable with respect to the (normalized) Hilbert-Schmidt norm.
In both cases, the results still leave the possibility that higher rank
lattices are _flexibly_ stable (pointwise or uniform) with respect to the
(normalized) Hilbert-Schmidt norm. For more on flexible stability, refer
[BC20] and [BL20].

  * •

Uniform stability with non-linear estimate.
Our machinery, whenever it can be applied, works to prove uniform stability
with a linear estimate. We do not know of examples of groups that are
uniformly stable, but without a linear estimate. It is interesting to compare
this with [BM21] where it is shown that $\mathbf{Z}^{2}$ exhibits pointwise
stability (with respect to Hamming metric on $Sym(n)$) but with a non-linear
estimate. However, in the case of uniform stability in our setting,
$\mathbf{Z}^{2}$ (being amenable) is uniformly stable for any
submultiplicative norm on $U(n)$. The quantitative aspects of stability are an
active line of current research.

  * •

Stability with respect to non-archimedean metrics.
A recent monograph of [FF21] studies stability with respect to $p$-adic
groups. An interesting feature in this non-archimedean setting is that the
ultrametric (strong triangle inequality) forces an equivalence between uniform
and pointwise stability (for finitely presented groups). One can ask if
further stability results in this setting too can be proved using the
framework of asymptotic cohomology.

## References

  * [ACH12] Leif O Arkeryd, Nigel J Cutland, and C Ward Henson.  Nonstandard analysis: Theory and applications, volume 493.  Springer Science & Business Media, 2012.
  * [AD22] Danil Akhtiamov and Alon Dogon.  On uniform Hilbert-Schmidt stability of groups.  Proceedings of the American Mathematical Society, 2022.
  * [ALM19] Nir Avni, Alexander Lubotzky, and Chen Meiri.  First order rigidity of non-uniform higher rank arithmetic groups.  Inventiones mathematicae, 217(1):219–240, 2019.
  * [AP15] Goulnara Arzhantseva and Liviu Păunescu.  Almost commuting permutations are near commuting permutations.  Journal of Functional Analysis, 269(3):745–757, 2015.
  * [BC20] Oren Becker and Michael Chapman.  Stability of approximate group actions: Uniform and probabilistic.  arXiv preprint arXiv:2005.06652, 2020.
  * [BFH20] Aaron Brown, David Fisher, and Sebastian Hurtado.  Zimmer’s conjecture for actions of $sl(m,\mathbf{Z})$.  Inventiones mathematicae, 221(3):1001–1060, 2020.
  * [BL20] Oren Becker and Alexander Lubotzky.  Group stability and property (T).  Journal of Functional Analysis, 278(1):108298, 2020.
  * [BLSW22] Uri Bader, Alex Lubotzky, Roman Sauer, and Shmuel Weinberger.  Stability and instability of lattices in semisimple Lie groups (in preparation).  2022\.
  * [BM99] Marc Burger and Nicolas Monod.  Bounded cohomology of lattices in higher rank Lie groups.  Journal of the European Mathematical Society, 1(2):199–235, 1999\.
  * [BM02] Marc Burger and Nicolas Monod.  On and around the bounded cohomology of $sl_{2}$.  Rigidity in dynamics and geometry (Cambridge, 2000), pages 19–37, 2002.
  * [BM19] Michelle Bucher and Nicolas Monod.  The bounded cohomology of $sl_{2}$ over local fields and $s$-integers.  International Mathematics Research Notices, 2019(6):1601–1611, 2019\.
  * [BM21] Oren Becker and Jonathan Mosheiff.  Abelian groups are polynomially stable.  International Mathematics Research Notices, 2021(20):15574–15632, 2021.
  * [BOT13] Marc Burger, Narutaka Ozawa, and Andreas Thom.  On Ulam stability.  Israel Journal of Mathematics, 193:109–129, 2013.
  * [BT65] Armand Borel and Jacques Tits.  Groupes réductifs.  Publications Mathématiques de l’IHÉS, 27:55–151, 1965.
  * [BW00] Armand Borel and Nolan Wallach.  Continuous cohomology, discrete subgroups, and representations of reductive groups.  Number 67. American Mathematical Soc., 2000.
  * [CDCMT15] Pierre-Emmanuel Caprace, Yves De Cornulier, Nicolas Monod, and Romain Tessera.  Amenable hyperbolic groups.  Journal of the European Mathematical Society, 17(11):2903–2947, 2015.
  * [CGM90] Alain Connes, Mikhaıl Gromov, and Henri Moscovici.  Conjecture de Novikov et fibrés presque plats.  CR Acad. Sci. Paris Sér. I Math, 310(5):273–277, 1990.
  * [Dad21] Marius Dadarlat.  Obstructions to matricial stability of discrete groups and almost flat k-theory.  Advances in Mathematics, 384:107722, 2021.
  * [DCGLT20] Marcus De Chiffre, Lev Glebsky, Alexander Lubotzky, and Andreas Thom.  Stability, cohomology vanishing, and non-approximable groups.  In Forum of Mathematics, Sigma, volume 8. Cambridge University Press, 2020.
  * [ES05] Gábor Elek and Endre Szabó.  Hyperlinearity, essentially free actions and $l^{2}$-invariants. The sofic property.  Mathematische Annalen, 332(2):421–441, 2005.
  * [FF21] Francesco Fournier-Facio.  Ultrametric analogues of Ulam stability of groups.  arXiv preprint arXiv:2105.00516, 2021.
  * [FFLM22] Francesco Fournier-Facio, Clara Löh, and Marco Moraschini.  Bounded cohomology and binate groups.  Journal of the Australian Mathematical Society, pages 1–36, 2022\.
  * [FFR23] Francesco Fournier-Facio and Bharatram Rangarajan.  Ulam stability of lamplighters and Thompson groups.  arXiv preprint arXiv:2301.03970, 2023.
  * [Fri17] Roberto Frigerio.  Bounded cohomology of discrete groups, volume 227.  American Mathematical Soc., 2017.
  * [Fuj98] Koji Fujiwara.  The second bounded cohomology of a group acting on a Gromov-hyperbolic space.  Proceedings of the London Mathematical Society, 76(1):70–94, 1998\.
  * [Gam11] Christoph Gamm.  $\varepsilon$-representations of groups and Ulam stability.  Master’s thesis, Georg-August-Universität Göttingen. 21, 2011\.
  * [Gol12] Robert Goldblatt.  Lectures on the hyperreals: An introduction to nonstandard analysis, volume 188.  Springer Science & Business Media, 2012.
  * [GW78] Alain Guichardet and David Wigner.  Sur la cohomologie réelle des groupes de Lie simples réels.  In Annales scientifiques de l’École Normale Supérieure, volume 11, pages 277–292, 1978.
  * [Hei80] Stefan Heinrich.  Ultraproducts in Banach space theory.  J. Reine Angew. Math., 313:77–104, 1980.
  * [Joh88] Barry E Johnson.  Approximately multiplicative maps between Banach algebras.  Journal of the London Mathematical Society, 2(2):294–316, 1988\.
  * [Kaz82] David Kazhdan.  On $\epsilon$-representations.  Israel Journal of Mathematics, 43(4):315–323, 1982.
  * [Kli04] Bruno Klingler.  Transformation de type Poisson relative aux groupes d’Iwahori.  Algebraic groups and Arithmetic, pages 321–337, 2004.
  * [LO20] Alexander Lubotzky and Izhar Oppenheim.  Non $p$-norm approximated groups.  Journal d’Analyse Mathématique, 141(1):305–321, 2020.
  * [Mar91] Gregori A Margulis.  Discrete subgroups of semisimple Lie groups, volume 17.  Springer Science & Business Media, 1991.
  * [Mit84] Yoshihiko Mitsumatsu.  Bounded cohomology and $\ell^{1}$-homology of surfaces.  Topology, 23(4):465–471, 1984.
  * [MM85] Shigenori Matsumoto and Shigeyuki Morita.  Bounded cohomology of certain groups of homeomorphisms.  Proceedings of the American Mathematical Society, 94(3):539–544, 1985.
  * [MN21] Nicolas Monod and Sam Nariman.  Bounded and unbounded cohomology of homeomorphism and diffeomorphism groups.  arXiv preprint arXiv:2111.04365, 2021.
  * [Mon01] Nicolas Monod.  Continuous bounded cohomology of locally compact groups.  Number 1758. Springer Science & Business Media, 2001.
  * [Mon04] Nicolas Monod.  Stabilization for $sl_{n}$ in bounded cohomology.  Contemporary Mathematics, 347:191–202, 2004.
  * [Mon06] Nicolas Monod.  An invitation to bounded cohomology.  In Proceedings of the International Congress of Mathematicians, pages 1183–1211. European Mathematical Society, 2006.
  * [Mon10] Nicolas Monod.  On the bounded cohomology of semi-simple groups, s-arithmetic groups and products.  J.reine angew.Math [Crelle’s J.], 640:167–202, 2010.
  * [Mon22] Nicolas Monod.  Lamplighters and the bounded cohomology of Thompson’s group.  Geometric and Functional Analysis, pages 1–14, 2022.
  * [MS04] Nicolas Monod and Yehuda Shalom.  Cocycle superrigidity and bounded cohomology for negatively curved spaces.  Journal of Differential Geometry, 67(3):395–455, 2004.
  * [MS06] Nicolas Monod and Yehuda Shalom.  Orbit equivalence rigidity and bounded cohomology.  Annals of mathematics, pages 825–878, 2006.
  * [PR93] Vladimir Platonov and Andrei Rapinchuk.  Algebraic groups and number theory.  Academic press, 1993.
  * [Sht13] A. I. Shtern.  Quasirepresentations of amenable groups: results, errors, and hopes.  Russian Journal of Mathematical Physics, 20(2):239–253, 2013.
  * [Spe50] Ernst Specker.  Additive Gruppen von folgen ganzer Zahlen.  Portugaliae Mathematica, 9(3):131–140, 1950.
  * [Tur38] A. M. Turing.  Finite approximations to Lie groups.  Ann. of Math. (2), 39(1):105–111, 1938.
  * [Ula60] Stanislaw M Ulam.  A collection of mathematical problems.  Interscience Publishers, 1960.
  * [vN29] J. von Neumann.  Beweis des Ergodensatzes und des H-Theorems in der neuen Mechanik.  Z. Phys., 57(1):30–70, 1929.
  * [Zim13] Robert J Zimmer.  Ergodic theory and semisimple groups, volume 81.  Springer Science & Business Media, 2013.

Lev Glebsky
Universidad Aut´onoma de San Luis Potosi, Mexico
E-mail address<EMAIL_ADDRESS>

Alexander Lubotzky
Faculty of Mathematics and Computer Science, The Weizmann Institute of
Science, 234 Herzl Street, Rehovot 7610001, ISRAEL
E-mail address<EMAIL_ADDRESS>

Nicolas Monod
École Polytechnique Fédérale de Lausanne (EPFL), CH–1015 Lausanne, Switzerland
E-mail address<EMAIL_ADDRESS>

Bharatram Rangarajan
Einstein Institute of Mathematics, Hebrew University of Jerusalem, Israel
E-mail address<EMAIL_ADDRESS>


# Sequence Generation using Deep Recurrent Networks and Embeddings: A study
case in music.

Sebastian Garcia-Valenciaa, Alejandro Betancourtb and Juan G. Lalinde-Pulidoa
Correspondence: Sebastian Garcia Valencia. Email<EMAIL_ADDRESS>aComputer Science Department, Universidad EAFIT, Medellin, Colombia; bDigital
Department, Ecopetrol, Bogota, Colombia

###### Abstract

Automatic generation of sequences has been a highly explored field in the last
years. In particular, natural language processing and automatic music
composition have gained importance due to the recent advances in machine
learning and Neural Networks with intrinsic memory mechanisms such as
Recurrent Neural Networks. This paper evaluates different types of memory
mechanisms (memory cells) and analyses their performance in the field of music
composition. The proposed approach considers music theory concepts such as
transposition, and uses data transformations (embeddings) to introduce
semantic meaning and improve the quality of the generated melodies. A set of
quantitative metrics is presented to evaluate the performance of the proposed
architecture automatically, measuring the tonality of the musical
compositions.

###### keywords:

Music Theory; Recurrent Neural Network; Generative Model; LSTM; NAS; UGRNN;
Automatic Evaluation; Embedding

##  1 Introduction

Since music origins, music composition has been considered an activity mainly
for musicians. By nature, music composition is a creative task and compromises
cognitive skills commonly developed exclusively by humans. These cognitive
requirements make music composition one of the most challenging tasks to
achieve automatically.

During recent years, academic and computer science community has been
interested in the use of algorithms to create musical pieces, but just
recently, advanced models started showing signs of creativity similar to those
of humans [Colombo et al., 2017, Johnson, 2017, Bahdanau et al., 2014]. These
signs of improvement can be explained by i) Recent revolution in computing
power and data availability, which facilitates the automatic training of
computer algorithms by using examples ii) Recent advances in Algorithmic
Structures allowing to include temporal consistency and memory to the
automatically composed pieces.

In the process of developing algorithms capable of composing natural and
appealing music, multiple strategies have been explored (e.g. Bayesian
Algorithms, Neural Networks, among others). In Particular, recent advances in
Recurrent Neural Networks (RNN), including memory mechanisms, show that is
possible to generate sequences by generalising patterns hidden in large data
sets, and have been applied in areas such as automatic music composition and
Natural Language Processing (NLP). In General, RNN is a family of algorithms
based on gated architectures composed by self-contained procedures denoted as
memory cells.

The training of the RNN is a process in which the memory cells are
automatically tuned by using human-made compositions. Once the training of the
network ends, it is intuitive to use their learned capabilities as
improvisation machines. This feature has different applications in arts, and
in particular, in music, it can be used for live play-answer acts, computer-
aided performance, and automated accompaniment. In addition to the algorithmic
challenges (the data, the algorithmic structure, and the training procedure)
automatic music composition must consider musical concepts such as:

  1. 1.

Music Representation: To make learning and automatic music composition
feasible, it is necessary to use an appropriate data structure (music
representation). In this paper, inspired by recent advances in NLP, the notes
are encoded as embeddings. The objective is to move the input data to a vector
space where the related outputs are closer. In this case, notes which tend to
appear in sequence.

  2. 2.

Long Term Consistency: A common problem in sequence generation is that, after
some time steps, the generated sequence loses sense and becomes random data,
this is commonly known as the problem of no long-term consistency. To
alleviate this issue, a common strategy is to use algorithms with memory
mechanisms (memory cells). This paper analyses the performance in automatic
music composition of different mechanisms, namely, Long Short Term Memory
(LSTM), Neural Architecture Search (NAS) and Update Gate Recurrent Neural
Network (UGRNN).

  3. 3.

Music Transposition: Music transposition makes two pieces of music to be
perceived in essence the same song despite having almost all notes different.
This concept, makes the learning of patterns in music sequences quite
particular, giving more importance to the changes between notes than to the
notes itself. The proposed approach considers three different strategies to
exploit the music transposition concept: i) A control case where the sequences
are the notes, ii) A modified case storing the pitch intervals, given the
first note, it is possible to reconstruct the original sequence, iii) The last
case on which each sequence is augmented to 12 through a shifting process with
the purpose of having all notes as starting point.

  4. 4.

Music Evaluation: Once the model generates some music sequences, it is
necessary to quantify the quality of the composition. We propose three metrics
considering music theory indications of tonality. The proposed metrics are
used to compare different RNN architectures and the impact of the Music
Transposition strategies.

The remaining part of this paper is organised as follows: chapter 2 introduces
existing bibliography about automatic music composition with computer science
techniques. Chapter 3 introduces our approach for the RNN architecture, the
training phase, and the music evaluation. Chapter 4 presents the experiments
to understand the selected architectures, the objective function, and finally
tune the hyperparameters of the proposed networks (e.g., Number of Layers,
Memory Cell). Chapter 5 analyses the models and the generated melodies from
the quantitative and musical perspective. Finally, chapter 6, concludes and
provides some future research lines of this work.

##  2 State of Art

This chapter summarises the main concepts supporting this work moving from
music theory to automatic music composition and machine learning. Section 2.1
briefly introduces the main musical concepts, for additional music theory
concepts see [MusicSales, 2000]. Section 2.2 presents existing strategies
including the proposed musical concepts in Machine Learning. Section 2.3
reviews recent architectures and training strategies. Lastly, section 2.4
introduces common ways to evaluate the quality of the generated pieces.

###  2.1 Music Theory

Figure 1: Musical basic concepts

A note is the basic element of a musical piece. A melody is a sequence of
notes which sounds at different time steps. There are 12 notes which form the
chromatic scale, and are divided into fundamental and altered notes. The
fundamental notes (C, D, E, F, G, A, B) form the C major scale and corresponds
to the notes of the white keys in a piano (Notes in bold in fig. 1.A). The
distance between notes is named interval, being the smallest, the semi-tone,
which is the distance between B-C or E-F, see fig. 1.B and 1.C respectively.
The semi-tone corresponds to the white keys in a piano next to each other. To
represent the altered notes, it is necessary to use accidentals which modify a
fundamental note. This modification can be adding (sharp ♯) or subtracting
(flat ♭) a semi-tone, these other notes correspond to the black keys in a
piano (Notes in italics in fig. 1.A).

As fig. 1.D shows, once the scale reaches B, the next note is again C but in a
different position in the sheet stave (second note in fig. 1.C). These groups
of 12 notes are known as octaves, the term is used also in psychics, to denote
a frequency which is a multiple of other. The frequency of a note, known as
pitch, from the next octave will be the double as the corresponding note in
the current octave. Figure 1.D begins with the C note of the fourth octave
(C4, 261.6Hz) and ends with the C note of the fifth octave (C5, 523.3Hz).

The 12 possible intervals between two notes (including the same note in
different octaves) receive the name of fundamental intervals, and once the
difference is bigger than 12 semitones, they become compound intervals. In
close relation with this concept are the tonality and dissonance. There is a
tendency for some intervals to be consonant and perceived pleasant (like 3
semitones or minor third) and others to be dissonant and perceived unpleasant
(6 semitones or augmented fourth).

Being explained the previous concepts, it is necessary to introduce musical
transposition, which is a characteristic of music that causes that the
perceived identity of a piece relates more to the relative changes than to the
notes. This feature causes that humans perceive C-D-E-F-D-C-C (fig. 2.A)and
G-A-B-C-A-G-G (fig. 2.B) as the same melody because they have the same pitch
interval series: 2 2 1 -3 -2 0. In section 3.2 3 strategies to capture this
patterns are presented.

After introducing the pitch, it is necessary to explain Time and rythm. For
this matter, there is a set of symbols, the reference value is the whole note
(fig. 1.E), equivalent to 2 half notes (fig. 1.F), an all the others are even
multiples and subdvisions of it (quarter, eighth and others).

The symbol does not define the duration of the note by itself; it is relative
to the Tempo (fig. 1.I). The quarter note (fig. 1.G) is frequently used as
reference to define the tempo as the number of quarter notes that sounds in
one minute, usually called Beats Per Minute (BPM). For example, in fig. 1 all
the sections (1.E, 1.F, 1. G and 1.H) which are known as bars have the same
duration of 2 seconds.

This paper only considers as song, melodies of at least 12 notes. All the
pieces have a tempo of 120 BPM and contain uniquely quarter notes. The
extension to harmonies (which consist of multiple notes at the same time) and
time variation is highlighted as future work.

Figure 2: Transposition in music

###  2.2 Representation

There are different approaches to represent the music for machine learning;
however, array-based representations are the most popular and can be divided
into direct representations and dimensional transformations. On the one hand,
direct representations consider various aspects of the notes or harmonies. The
authors in [Hörnel, 1998] define a 12-bit codification over motifs (notes
groups) which considers musical knowledge about intervals and harmonic
relationships. They also have an independent representation of the harmonies.
In [Johnson, 2017], the authors use a binary array for the note with the
position, pitch class, previous context, beat and previous vicinity. To define
vicinity, the authors use an idea taken from convolutional networks, replacing
the kernel of the convolution by the RNN, and the pixels by notes of other
time steps.

On the other hand, dimensional transformations is an alternative to represent
musical data as a more convenient array. Two examples of this approach are the
‘Dodecaphonic Trace Vector’ in [Kaliakatsos-Papakostas et al., 2010] and the
‘token’ of [De Prisco et al., 2017]. The former uses a normalised vector of
musical pieces transposed to the key of C major or A minor; this approach is
key tolerant and preserves the general characteristics of the song. The latter
is a 3-tuple: k1: the name of the chord, k2: the type of chord (maj7, m7,
etc.) and k3: the role of the note concerning the chord. The ‘Token’ approach
does not consider time in the representation but as an extra step in the
model.

An interesting vector representation called ‘word embedding’ [Mikolov et al.,
2013a] is commonly used in NLP, and is a strategy to transform words in a
corpus to points in a vectorial space. Embeddings popularity relies on its
ability to locate semantically nearby words in adjacent spacial segments. The
context can be as simple as the surrounding words or the actual syntactic
context [Levy and Goldberg, 2014]. Embeddings are highlighted as a powerful
dimensional reduction and enhance the capability of machine learning
algorithms to capture hidden patterns. In practice, the embedding
transformation can be learned simultaneously with the RNN.

Finally, there are also ‘Non-array’ representations such as the ‘abc language’
used in [Walshaw Chris, 2017, Oliwa and Wagner, 2008], on which notes and time
are represented with chars111conversion from midi is possible with the abcMIDI
package http://abc.sourceforge.net/abcMIDI/.

To the best of our knowledge, the concept of embedding has been only used in
music to extract some patterns [Madjiheurem et al., 2016, Herremans and Chuan,
2017] but not for automatic composition, which is usually highlighted as
future research. In this paper, the n-gram based model [Mikolov et al., 2013b]
is used to transform the notes and intervals to embeddings, and add semantic
meaning to the notes. In [Garcia-Valencia, 2020b], we analyse the resulting
embeddings.

###  2.3 Music composition with Algorithms

Three components are necessary to automatically generate music: the algorithm
architecture, the training data and the objective function to optimise.

Algortihms and Architectures:  An algorithm is denoted as generative when it
can produce information given an input. This input could be a set of numbers
which have a given distribution or an array of data containing hidden
parameters or weights. Some examples of generative models are Restricted
Boltzmann machine [Fischer and Igel, 2012] and Generative adversarial networks
[Goodfellow et al., 2014] commonly used for image generation. For automatic
sequence generation, the n-gram model [Brown et al., 1992] is widely used to
predict the next element in a sequence probabilistically.

An artificial neural network is a mathematical model inspired by nature. In
the biological context, a neuron has three main parts: i) An axon to send
signals to other neurons, ii) A dendrite to receive the signals from the axons
of others neurons, and iii) The soma, which is the neuron nucleus and is where
the signal can be modified [Farinella and Ros, 2014]. The points where
dendrites and axons communicates are called synapses and the way different
neurons connect with others in the brain determine what a living being learn,
think and do.

In the artificial version, a neuron receives a value and multiplies it by an
internal weight before transmitting it to the next neuron or layer of neurons.
The organic-dynamic interconnection is simulated by varying the weight of
every neuron until it satisfies an error.

The use of generative models for music has been a widespread problem. It has
been approached with different strategies such as neural networks [Colombo et
al., 2017, Hutchings and McCormack, 2017, Waite, 2016, Hörnel, 1998], Bayes
transform [Thornton, 2011], Bayesian networks with a hidden Markov model
[Raphael, 2002] and probabilistic inference using statistical models [Whorley
and Conklin, 2016, Allan and Williams, 2005]. As a consequence of the recent
advances in hardware and data availability, it is becoming popular to use more
complex Artificial Neural Networks (ANN) for tasks as computer vision with
Convolutional NN (CNN) [Krizhevsky et al., 2012], which basically make a
dimensional reduction of the input layer by layer, and sequence generative
models with Recurrent Neural Networks (RNN) [Schmidhuber, 2014].

Regarding CNNs, different adaptions have been tried, but the absence of memory
severely compromises the results. The authors in [Gehring et al., 2017] use a
combination of convolutions, gated linear units, and attention mechanisms for
language translation. The proposed CNN can not learn sequence positions, as
RNN does. The authors alleviate this issue by using positional embeddings. The
study concludes that CNNs are capable of discovering compositional structures
by building hierarchical representations. However, are less frequent in
sequence generation problems. Another interesting approach is proposed in
[Johnson, 2017], using a two folded architecture, the first part, considers
the relationship between notes using convolutions, while the second part
considers the temporal consistency. This approach allows the model to learn
patterns in both characteristics without losing invariance.

RNNs, in particular, use the output of the network as input in a feedback
process. There are different implementations like bidirectional RNN [Berglund
et al., 2015] which are used to predict lost values in the middle of
sequences. A special type of RNN with memory-like characteristics is the gated
RNN. These architectures are used in sequence modelling tasks and generative
models to address the problem of long-term consistency. In these gated RNN, in
addition to the coefficients (neural weights), there are a group of cells or
gates that must be trained with the past elements. In this paper three gated
architectures are explored:

  1. 1.

LSTM [Hochreiter and Schmidhuber, 1997]: The Long-Short-Term Memory NN is
composed of 4 elements: a cell state which determines the current context, a
forget gate which decides which information remove, an update gate to insert
new information and an output gate to decide the output [Olah, 2015].

  2. 2.

NAS: Neural Architecture Search is a type of cell which designs its internal
structure by exploring combinations of recurrent cells dispositions and
activation functions. NAS use reinforcement learning with the accuracy as
reward [Zoph and Le, 2016].

  3. 3.

UGRNN: the Update Gate RNN is a variation of a vanilla RNN with a single gate
added, which determines if the current state of the cell was updated or taken
from a past time step. Authors in [Collins et al., 2016] compare it against
LSTM and another gated RNN called GRU [Cho et al., 2014]. UGRNN shows to be as
good as GRU to learn difficult tasks but easier to train than LSTM and GRU for
deep architectures.

The authors in [Chung et al., 2014] compare LSTM and GRU and conclude that GRU
performs better in 3 out of 4 datasets. LSTM is used In [Colombo et al., 2017]
and [Hutchings and McCormack, 2017]. The former uses a deep LSTM with 2
different datasets (Irish folk music and Klezmer tunes). The trained algorithm
is able to keep the genre during the generation phase. In the later study, an
LSTM is the base for real time autonomous music generators. LSTM networks can
also extend to other problems like chord estimation [Deng and Kwok, 2018].

The magenta project (Google brain) studies two different approaches to address
long-term consistency: i) Look-back networks [Waite, 2016] using custom labels
and previous events from 1 and 2 musical bars in the input. ii) Attention
networks, improving the long-term memory by weighing the outputs of previous
steps. Attention networks were initially used for translation tasks [Bahdanau
et al., 2014].

Training Data:  Music datasets are usually the result of scraping music
repositories. These containers can be specific for symbolic music (Gutenberg
project [Project Gutenberg, 2006], Mutopia project [Project Mutopia, 2017]) or
general music files (Million song dataset [Bertin-Mahieux et al., 2011]). The
most popular container in recent years is the MuseScore sheet music archive
[Froment and Bonte, 2002], because there is a significant community supporting
and contributing to it.

Academically there are four open datasets commonly used for polyphonic music
(i.e. Nottingham, JSB Chorales, MuseData, Piano midi). These datasets use MIDI
format and consist of classical and folk music [Boulanger-Lewandowski et al.,
2012]. Other authors have used these datasets to test their models ([Gu et
al., 2015, Chung et al., 2014]). The authors in [Kaliakatsos-Papakostas et
al., 2010] use a similar dataset of Piano-midi but with fewer composers.

Objective Function:  Concerning the optimisation of the model in training, the
magenta project has an interesting approach based in reinforcement learning
[Jaques et al., 2016] on which a book of music theory is used to define the
rewards and train an LSTM based model called the RL tuner. This approach helps
to reduce unpleasant characteristics like notes out of key and improves good
aspects like compositions with only one top note.

Another common strategy is to use a defined metric as objective function in a
classic optimisation problem. Usually, cross-entropy is a popular choice for
classification and generative models [Johnson, 2017, Hutchings and McCormack,
2017, Whorley and Conklin, 2016, Kawthekar et al., 2017]. According to
information theory, the cross-entropy quantifies the differences between two
probability distributions [Shannon, 1948]. In machine learning, it is used to
compare the predicted output with the ground truth, and particularly in
generative models, measures the prediction capability [Kawthekar et al.,
2017]. Being the ground truth an unknown distribution, the negative average
log-likelihood is used as an estimator. Commonly, in NLP the metric chosen is
the perplexity, which is the exponentiation of the cross-entropy. For
optimisation purposes both are equivalent, but perplexity is more natural to
interpret due to its linear scale.

The base architecture for this paper is a multi-cell RNN with two
hyperparameters, namely the number of memory cells and their type (i.e. LSTM,
NAS, UGRNN). For training purposes we use the mono-midi-transposition-dataset,
a monophonic dataset created after applying a number of modifications to the
mono-musicxml-dataset proposed in [van der Wel and Ullrich, 2017], which is
the result of scraping musescore for monophonic music, this is an advantage
cause the most of the available data tend to be polyphonic. Finally, we use
cross-entropy as the objective function. In [Garcia-Valencia, 2020a] we
validate the use of cross-entropy showing the gradual reduction in the metric
loss, reflects a reduction of randomness of the generated melodies.

###  2.4 Music Composition Evaluation

According to the recent bibliography, music evaluation can be divided into two
groups. The first one uses mathematical formulations [Colombo et al., 2017,
Jaques et al., 2016, Tymoczko, 2011], while the second one uses musical
experiments to consider human judgments about the quality of the composition.

The quantitative evaluation of musical compositions defines mathematical
formulations to measure the quality of a musical piece according to music
theory. The authors in [Colombo et al., 2017] introduce a new metric called
novelty to quantify the musical creativity. The idea is to use a corpus of the
training to calculate how different are the generated songs with respect to
training dataset. In general, quantitative metrics such as novelty are useful
to judge the quality of musical pieces automatically and teach generative
algorithms. However, they ignore the subjective feeling that humans easily
detect.

To alleviate this issue, other authors propose human-based experiments. A good
example is a test made by [De Prisco et al., 2017], where professional
musicians are asked to score musical pieces by answering two questions with a
number between 1-7: (1) ‘How do you rate the quality of music?’, and (2) ‘How
do you rate the coherence of the music with the X style?’

Finally, there is also the option of using a combination of both methods. This
is the case of the study proposed in [Hutchings and McCormack, 2017] which
combine perplexity and human experts judgement. Perplexity assesses a part of
the model called the harmonic improviser. Then, the generated pieces are
scored by professional musicians in 6 dimensions like harmonic consistency,
thematic development, and melodic creativity.

This paper uses a combination of cross-entropy for training and quantitative
metrics for evaluation of music tonality. See section 3.3 for the mathematical
implementation of the metrics

##  3 Generating melodies

Following the ideas from section 2, our goal is to develop a network to
generate melodies. Section 3.1 describes the architecture and training of the
RNN. Section 3.2 illustrates the method to scrap the data from musescore,
preprocess it, clean it and create the two variations to introduce the
transposition principle. Finally, section 3.3 formalises the implementation of
the three quantitive metrics and shows the result of applying them to the
dataset. This result is used as the baseline for the performance analysis of
the results section 5.1.

###  3.1 RNN Architecture

The base for the proposed architecture of our RNN is char-rnn ([Karpathy,
2015]), originally introduced to generate text char by char. The main
variations with respect original char-rnn are: i) The embedding encoding and
decoding of music, ii) the use of 3 different cell types in the multi-cell
RNN.

The workflow through the network varies depending on the use, namely training
mode (fig. 3) or sampling mode (fig. 4). For training, the starting point is
the dataset. As explained in section 3.2, the three dataset variants have the
same two elements: the X and Y array, the later is the former shifted by one
position. First, the embedding component of the network produces a matrix
using all the distinct elements in the dataset, the dimension of this matrix
will be the size of the vocabulary multiplied by the embedding dimension.
Then, every sample goes first to the embedding component to become a vector.

Figure 3: RNN Architecture training mode

The embedding goes to the multicell RNN; this is the main component of the
network which can contain an array of cells of any type (LSTM, NAS or UGRNN).
Once the embedding flows through the multicell RNN, the internal state of the
cells changes and goes as feedback input for the next iteration (this is the
key of memory). After that, there is a layer with a classical linear
regression with the trainable variables W and b.

Using the output of the linear regression and the encoded Y, the network
calculates the cross-entropy and send it to a reduce sum layer, where it
calculates the final loss. Using an Adam optimiser, this last layer back-
propagates the error through all the layers (blue lines in fig. 3) until
reaching back the embeddings block. This procedure allows us to improve the
embedding encoder and decoder, the cells coefficients, and the linear
regression simultaneously.

In the case of sampling mode, the starting point is the seed, which is the
sequence that the network will use to initiate its internal state and then
generate new notes. From here, sampling mode has two phases. In the first one
(green lines in fig. 4), each sample of the seed flows through the embedding
encoder to become its vector representation and then change the internal state
of the cells in multicell RNN. Once it finishes with all the samples in seed
in iteration m, phase 2 begins (blue lines in fig. 4). The output of the
multicell flows through the linear regression and a softmax layer which
outputs the most probable next embedding. This output goes back as an input to
the multicell RNN in the next iteration, and simultaneously it is decoded and
added to the generated melody. This process repeats n times.

Figure 4: RNN Architecture sampling mode

###  3.2 Mono-midi-transposition-dataset

The dataset222The general dataset is available in
https://sebasgverde.github.io/mono-midi-transposition-dataset/ used in this
paper is a processed version of the mono-musicxml-dataset [van der Wel and
Ullrich, 2017]. The process consists of 3 steps333All the code, files and
final datasets to replicate the results of the whole paper are available in
https://sebasgverde.github.io/rnn-cells-music-paper/: i) Scraping: The
structure of the original dataset is used as the base to download the files
from the web. ii) Prepossessing: Each midi is transformed into an array
containing the sequence of notes (fig 5.2). iii) Cleaning: only songs with
more than 3 notes are kept (fig 5.3). The final list of arrays is used as the
base to create 3 datasets (fig 5.4 and 5.5):

Figure 5: Dataset workflow

Control Dataset: This is the base case, on which X is the concatenated array
of the original songs and Y is X shifted by one position.

DB12 Dataset: To construct this dataset, each song is transposed 12 times (on
each degree of the chromatic scale). Pseudocode 1 summarises the procedure,
defining a song as an ordered set of midi notes of the form
{$X_{0},X_{1},...,X_{|song|}$}, where $X_{i}$ is the note in the position i,
with $X_{i}\in[0,127]$. The general idea is to spread the song through the
midi range (0-127). The first part of the algorithm calculates the middle
point of the song and subsequently moves the song up and down by semitones
depending on how far is from central c4 (60). The algorithm intentionally
gives preference to high notes which are more common in melodies. The DB12
dataset is the equivalent of the data augmentation regularly done in computer
vision to increase the number of images by rotations, translations and
mirroring [Overett et al., 2008, Ciresan et al., 2013]

Algorithm 1 Converting one song in 12

1:procedure songToDB12($song$)

2: $centralC\leftarrow 60$

3: $minNote\leftarrow min(song)$

4: $maxNote\leftarrow max(song)$

5: $middlePoint\leftarrow\lfloor\frac{maxNote-minNote}{2}\rfloor+minNote$

6: $centralCgap\leftarrow centralC-middlePoint$

7: $remainingTransp\leftarrow 11-\left|centralCgap\right|$

8: if $remainingTransp\geq 0$ then

9: $up\leftarrow\left\lceil\frac{remainingTransp}{2}\right\rceil$

10: $down\leftarrow remainingTransp-up$

11: if $centralCgap<0$ then

12: $down\leftarrow down+\left|centralCgap\right|$

13: else

14: $up\leftarrow up+centralCgap$

15: end if

16: else

17: if $centralCgap\leq 0$ then $\text{down}=11$ else $\text{up}=11$

18: end if

19: $db12List\leftarrow song$

20: for $i\in[0,down]$ do

21: $newSong\leftarrow\emptyset$

22: for $X\in song$ do

23: $newSong[i]\leftarrow X-(i+1)$

24: end for

25: $db12List\leftarrow db12List\cup newSong$

26: end for

27: for $i\in[0,up]$ do

28: $newSong\leftarrow\emptyset$

29: for $X\in song$ do

30: $newSong[i]\leftarrow X+(i+1)$

31: end for

32: $db12List\leftarrow db12List\cup newSong$

33: end for

34: return $db12List$

35:end procedure

Intervals Dataset: In this case, we do not have a sequence of notes, but a
sequence of relative changes. The idea of having a representation of the
transitions between notes in a melodic context is not new ([Hörnel, 1998,
Kaliakatsos-Papakostas et al., 2010, Johnson, 2017]). In a classification
problem, the authors in [Chai and Vercoe, 2001] shows that using relative
changes improves the efficiency of their algorithm with respect to other
approaches. Pseudocode 2 shows our strategy to convert each song in an
interval representation.

Algorithm 2 Converting one song in an interval array

1:procedure songToInterval($song$)

2: $intervalArray\leftarrow\emptyset$

3: for $X_{i}\in song$ with $i\in[0,|song|-1]$ do

4: $intervalArray[i]\leftarrow X_{i+1}-X_{i}$

5: end for

6: return $intervalArray$

7:end procedure

###  3.3 Description of the Quantitative Evaluation

The quality of a musical piece is a subjective matter, and it is usually one
of the most challenging parts of automatic music composition, A common
strategy to quantify the quality of a composition is to use human evaluations.
However, the seminal work [Tymoczko, 2011] proposes five music characteristics
based on geometry (i.e. Conjunct Melody Motion, Acoustic Consonance, Harmonic
Consistency, Limited Macroharmony and Centricity). Together, these
characteristics contribute to the tonality and the perceived beauty of a piece
of music. Only 3 of this five features apply to melodies (i.e. Conjunct Melody
Motion, Limited Macroharmony and Centricity). Acoustic consonance and Harmonic
consistency apply exclusively to harmonies, and according to Tymoczko, these
are the most culturally specific. These two features are not considered in
this study. In the next subsections, the basic description of these features
and their implementation are briefly introduced444The metrics are proposed
only for songs with at least 12 notes..

Following the previously introduced notation of a song as an array, Let’s
define the $span_{i}$ as a subset of song from $X_{i}$ to $X_{i+n}$, where n =
$|$span$|$ is the size of the span (In this case the constant 12), and $Sq$
the quantity of spans given by equation 1.

$Sq=\begin{cases}1,&\text{if }|song|\leq n\\\
|song|-n+1,&\text{otherwise}\end{cases}$ (1)

####  3.3.1 Conjunct Melodic Motion (CMM)

CMM looks for smooth transitions through the melody, in other words, the
changes between notes must have short intervals. To capture this, we sum all
the changes in the notes and divide it by the number of notes of the song
minus 1. In our case the CMM is given by equation 2. In general, the CMM is 1
if every change is of 1 semitone, tends to be greater than one, if there are
changes of more than one semitone, and less than 1 for melodies with repeated
notes. The best CMM is one because it is a song in the chromatic scale and
sounds consonant by defect.

$CMM=\frac{1}{|song|-1}\sum_{i=1}^{|song|-1}\left|X_{i+1}-X_{i}\right|$ (2)

####  3.3.2 Limited Macroharmony (LM)

Macroharmony measures the diversity of notes in melodies. This is captured by
measuring the number of different notes in short slices of time (spans). For
explanatory purposes, lets divide the Limited Macroharmony into two concepts:

  * •

Local Limited Macroharmony ($llm$) quantifies the diversity of notes in a
particular span. According to Tymoczko, tonal melodies must have between 5 and
8 different notes since the number of alternatives a human can manage is $7\pm
2$ [Miller, 1956]. Using a span size of 12 causes that the maximal
penalisation for notes out the range (5,8) is the same if the span has either
1 or 12 different notes. It is important to remember that time is out of the
scope of this paper and all the songs are always quarter notes so in this
case, even when the book defines the span regarding musical time, in our case
it reduces to the number of notes.

The $llm$ is given by equation 3, where $lb$ is the lower bound, $ub$ is the
upper bound and $d_{j}$ is the number of distinct notes in the particular
span. In our case $lb$ = 5 and $ub$ = 8. The $llm$ is 1 if the number of
different notes in the span matches the appropriate range. Otherwise, it is
the difference between the number of different notes in the span and the
appropriate range plus 1. 555Since the same note in different octaves like D4
and D5 are different alternatives, we treat them just as different notes.

$llm=\begin{cases}1,&\text{if }lb\leq d_{j}\leq ub\\\ (lb-d_{j})+1,&\text{if
}dj<lb\\\ (d_{j}-ub)+1,&\text{otherwise}\end{cases}$ (3)

  * •

Limited Macroharmony ($LM$) measures the diversity in the complete melody. For
melodies longer than the span size, the $lm$ is the average of calculating the
$llm$ over a sliding window. The $lm$ is given by Equation 4, where the span
quantity $Sq$ is the number of sliding windows covering a melody. In this case
$lm$ is 1 if every span in a melody is within the appropriate limits, or
greater than one for songs with a large diversity of notes.

$LM=\frac{1}{Sq}\sum_{j=1}^{Sq}llm_{j}$ (4)

####  3.3.3 Centricity (CENTR)

Centricity claims that in short slices of time (spans), there must be a note
which appears with more frequency than the others. We calculate the frequency
of every note as the number of occurrences. Centricity is the max ratio
between each note frequency in a span and the total notes in that span. To
calculate the centricity of the complete song, we use the same strategy of
average over a sliding window form LM. CENTR is given by equation 5, where i
is defined in the interval $[0,...,d_{j}]$ and $f(X_{ij})$ is the frequency of
note i in span j. In general CENTR tends to 1 the more predominant is one of
the notes with respect to the others and tends to 0 the less frequent is the
most frequent note.

$CENTR=\frac{1}{Sq}\sum_{j=1}^{Sq}max_{i}(\frac{f(X_{ij})}{n})$ (5)

####  3.3.4 Geometry descriptive statistics for dataset

To properly evaluate the models, it is essential to define a baseline. The
natural step is to apply the proposed metrics to the original dataset. Since
the other 2 datasets are transformations of the original, the tonality in the
songs should not differ. As table 1 shows, for the CMM and the LM the average
is above the perfect score of 1 by 1.42 and 0.67 respectively, this evidence
that a score above 1 in the generated pieces is not necessary a bad result. In
the case of centricity, we can conclude that a tonal song should have a note
that is at least the 20% of the melody.

CMM | LM | CENTR
---|---|---
2.42 $\pm$ 0.98 | 1.67 $\pm$ 0.86 | 0.20 $\pm$ 0.12
Table 1: Dataset evaluation

##  4 Experiments

This section compares different types of RNN cells (3), with different depth
(5) and different transposition strategies (3). The combination of these
decisions gives 45 different models to be analysed. This section defines the
optimal number of layers, for every data-set and memory cell, reducing the
number of models to 9 (3 datasets and 3 memory cells). Looking for simplicity
the experiments are grouped by data-set and the convergence of the cross-
entropy used as evaluation metric.

Regarding the training conditions: i) In all the cases the batch size and the
sequence length is 50, which means that every training iteration uses 50
sequences of 50 notes simultaneously. ii) For the control and the interval
dataset, the model trains 300 epochs, what means that uses the complete
dataset 300 times. For the DB12 dataset, which is 12 times bigger than the
other 2, it trains only 50 epochs.

###  4.1 Control dataset experiment

The optimal number of layers for the three cell types is 3. The worst
performance is for 1 and then 5 layers, this is interesting since it shows
that this is not a simple matter of add as much layers as possible.

The five-layer model tends to outperform the one-layer one after some
iterations. This happens around 50k for LSTM (Fig. 6(a)) and 20k for NAS (Fig.
6(b)) and UGRNN (Fig. 6(c)). The four and five-layer models always begin with
a worse score, what means that the more layers, the more it takes to converge.

(a) LSTM

(b) NAS

(c) UGRNN

[•] 1 layer [•] 2 layers [•] 3 layers [•] 4 layers [•] 5 layers

Figure 6: Control Dataset Experiment Learning Curves

###  4.2 Interval dataset Experiment

Both, the four and five-layer models outperform the one and two-layer after
approximately 17k iterations with LSTM cells (Fig. 7(a)), in this case, 4 is
the optimal number of layers at the end.

For the NAS (Fig. 7(b)), the most in-depth models take a considerable number
of iterations to begin to converge, this is mainly evident for 5 layers, which
only improve after 40k iterations. These deep models, however, never
outperform the two-layer one, which is the best in this case.

(a) LSTM

(b) NAS

(c) UGRNN

[•] 1 layer [•] 2 layers [•] 3 layers [•] 4 layers [•] 5 layers

Figure 7: Interval dataset Experiment Learning Curves

The UGRNN experiment (Fig. 7(c)) shows an interesting behaviour. The five-
layer model outperforms the one-layer after 35k iterations, these two models
are at the end the fourth and fifth place respectively. The four-layer model
outperforms all the others gradually, the one-layer at 15k, two-layer at 19k
and three-layer at 40k, until it reaches the lowest loss at the end

###  4.3 DB12 Dataset Experiment

The LSTM (Fig. 8(a)) and NAS (Fig. 8(b)) have a similar behaviour since the
worst result in both cases is for 1 and 2 layers. The LSTM experiment (Fig.
8(a)) shows that 4 is the number of layers that reach the best result. In the
NAS experiment on the other hand (Fig. 8(b)), the five-layer model begins as
the worst, after iteration 50k, it gradually outperforms the others and
finally reaches the first place.

(a) LSTM

(b) NAS

(c) UGRNN

[•] 1 layer [•] 2 layers [•] 3 layers [•] 4 layers [•] 5 layers

Figure 8: DB12 Dataset Experiment Learning Curves

In UGRNN experiment (Fig. 8(c)), results are stable from the beginning until
the end. The five-layer model is the worst, while one and two layer models are
the best. This cell type is the only one with a good performance for 1 layer,
however, is the two-layer model which has the best loss.

###  4.4 Analysing the Network Depths

In general, the learning curves presented in previous sections show that the
deeper the network, the more iterations it needs to converge. Table 2
summarises the optimal number of layers for each dataset and memory mechanism
showing no consensus for all the cases and making necessary a more detailed
analysis. The following part of this section analyses the advantages and
disadvantages of each dataset (rows) and the performance of each memory-
mechanism (columns). The remaining part of this paper uses the number of
layers defined in Table 2.

| LSTM | NAS | UGRNN
---|---|---|---
CONTROL | 3 | 3 | 3
INTERVAL | 4 | 2 | 4
DB12 | 4 | 5 | 2
Table 2: Optimal number of layer per model

Regarding the datasets, it is interesting, that the Control dataset is the
only one with no variation in the optimal number of layers for the three
cells, suggesting it as the most stable dataset. It is the opposite case for
the DB12, which is the dataset where the LSTM and UGRNN (the similar memory
mechanism) diverge. Additionally, DB12 shows the higher variance concerning
the depth; it is the only dataset where the deepest models are the best option
(DB12-NAS models) and the only one with a good performance for a one layer
architecture according to the convergence curve (see fig. 8(c)). These facts
suggest the DB12 dataset as the less stable of the three datasets.

Regarding the type of memory mechanism, the LSTM and UGRNN (which have an
established and quite similar architecture) show a similar behavior, except
for the less stable DB12 dataset. The nature of the NAS cell, which does not
have a defined architecture but builds it while training, combined with the
representation of intervals seems to cause a divergence with LSTM and UGRNN.
Finally, this evolutionary nature of the NAS cell, combined with the less
stable DB12 dataset, can be the cause of the five layers result.

##  5 Results

This chapter introduces the evaluation and analysis of the 9 models of table
2. In section 5.1 the 3 metrics described in 3.3 test 100 songs for each of
the 9 models to quantify their composition capabilities. Section 5.2,
visualises the 9 most representative songs, and provide analysis from the
musical viewpoint.

###  5.1 Analysis of Metrics in the Models

To evaluate the 9 models from table 2, each model generates 100 songs with the
same parameters: the sequence C4-D4-E4-D4 as seed and 30 new notes. Table 3
shows the average and standard deviation for each metric and model. Concerning
the best model for each metric, it is not possible to conclude due to the
magnitudes of the standard deviations; However, under the light of each
metric, important insights can be obtained.

|  | LSTM | NAS | UGRNN
---|---|---|---|---
CMM | CONTROL | 2.14 $\pm$ 0.60 | 1.90 $\pm$ 0.49 | 2.11 $\pm$ 0.50
INTERVAL | 2.99 $\pm$ 1.08 | 2.44 $\pm$ 1.00 | 2.42 $\pm$ 0.74
DB12 | 2.49 $\pm$ 0.59 | 4.28 $\pm$ 2.71 | 1.66 $\pm$ 0.40
LM | CONTROL | 1.47 $\pm$ 0.46 | 1.77 $\pm$ 0.65 | 1.51 $\pm$ 0.46
INTERVAL | 2.37 $\pm$ 1.56 | 1.59 $\pm$ 0.75 | 1.66 $\pm$ 0.84
DB12 | 1.57 $\pm$ 0.64 | 2.56 $\pm$ 1.11 | 2.40 $\pm$ 0.65
CENTR | CONTROL | 0.20 $\pm$ 0.08 | 0.22 $\pm$ 0.09 | 0.21 $\pm$ 0.09
INTERVAL | 0.16 $\pm$ 0.08 | 0.16 $\pm$ 0.09 | 0.17 $\pm$ 0.12
DB12 | 0.16 $\pm$ 0.08 | 0.25 $\pm$ 0.26 | 0.23 $\pm$ 0.09
Table 3: Means and standard deviation for the 100 songs generated by the final
models with the 3 metrics.

According to the Conjunct Melodic Motion (CMM) the closest CMM to 1 is the
DB12-UGRNN (1.66), with a considerable advantage over Control-NAS(1.90), and
outperforming the average of the dataset (2.42). It is interesting how the
DB12-NAS model shows a significantly worse result, and the largest standard
deviation, positioning it as the least reliable model according to the CMM.

In the case of LM the control dataset have the best performance for this
metric with 3 of the best results. Regarding, to the cell type, each has a
good performance with 2 datasets and a bad one with the other, giving a result
of 3 wrong combinations: INTERVAL-LSTM, DB12-NAS and DB12-UGRNN. The model
with best LM is the CONTROL-LSTM with 1.47, what also outperforms the 1.67 of
the dataset (table 1), while the worse is the DB12-NAS.

Regarding the centricity, The model which shows higher centricity is the
DB12-NAS, however, the bad performance of this model in the other metrics
indicates a strong repetition of notes. Also, it has the highest standard
deviation, therefore, the DB12-UGRNN with only 0.09 of deviation and just 0.2
less centricity is the best candidate. Notice that this both models have poor
LM, this makes sense, while centricity measures the prevalence of a note, the
LM, makes the opposite.

In general, LSTM models are stable through the datasets for CMM and shows a
low centricity (tends to use more notes), what is evident in the good results
for LM. The NAS cell seems to be unstable with the DB12 dataset. Between the 2
strategies for transposition learning, the interval representation is the most
stable.

Respect to the best model for the 3 metrics, the CONTROL-NAS, model has a good
result for all the metrics. It is the only model that outperforms the
centricity of the dataset without affecting the LM considerably, which is the
weakness of the DB12-UGRNN model. In summary, concerning only the quantitative
analysis, the best models are i) DB12-UGRNN for CMM and CENTR, ii) CONTROL-
LSTM for LM and iii) CONTROL-NAS for general tonality.

###  5.2 Generated Melodies Musical Analysis

Using the average metrics for the 100 songs generated by each of the 9 models
(table 3), it is possible to select a representative song per model using the
Euclidean Distance to the average as selection criteria. Table 4 summarises
the metrics for sampled songs.

| LSTM | NAS | UGRNN
---|---|---|---
|  CMM |  LM |  CENTR |  CMM |  LM |  CENTR |  CMM |  LM |  CENTR
CONTROL | 2.21 | 1.52 | 0.17 | 1.85 | 1.78 | 0.20 | 2.06 | 1.39 | 0.26
INTERVAL | 2.73 | 2.22 | 0.10 | 2.36 | 1.65 | 0.24 | 2.42 | 1.70 | 0.18
DB12 | 2.45 | 1.61 | 0.08 | 4.48 | 2.70 | 0.29 | 1.58 | 2.39 | 0.25
Table 4: The 3 metrics for the most representative of the 100 songs generated
by the final models in each case.

Figures 9, 10 and 11 shows the musical sheet of the 9 songs666Listen to the 9
songs in https://youtu.be/FGUIEshh6WU from table 4. The CONTROL-LSTM model
(fig. 9(a)) has a balanced number of notes. It also shows an ascendant melody
which goes from C4 to A4 in bar 7. In the case of the NAS (fig. 9(b)) and
UGRNN (fig. 9(c)) variations, the first shows a descendant pattern in bar 3
going from G4 to D4. The second, as expected from the metrics for UGRNN, shows
a repetition of notes, but in this case, it forms a tonal pattern, repeating
groups of 3: G4 in bar 3, C4 in bar 5 and E4 in bar 6; these are the notes
that harmonise the C major chord. Under the light of current results it is not
clear if the 5 notes repeated at the end, are the result of a stagnant of the
model.

(a) LSTM Model

(b) NAS Model

(c) UGRNN Model

Figure 9: Generated Melodies Control Dataset

As expected by the metrics (Tables 3 and 4), the DB12-NAS model (fig. 10(b))
tends to repeat the notes, in this case, using the E4 19 times, and in the
middle of the song 5 times consecutively, this naturally favors the
centricity. The other model with good centricity DB12-UGRNN (fig. 10(c)) also
repeats, in this case, the note C4, but in this case more tonally. The good
performance of CMM is evident, unlike DB12-NAS which at the end has changes of
even 9 semitones (major sixth), this model keeps softer changes.

(a) LSTM Model

(b) NAS Model

(c) UGRNN Model

Figure 10: Generated Melodies DB12 Dataset

Finally, it is important to speak about the models which did not highlight in
the metrics evaluation. The DB12-LSTM (fig. 10(a)) shows the most interesting
ascendant patterns in the 9 songs, going from A3 to D4, then changing to G3,
to go to C4, and at the end goes from B3 to resolve in C5. Notice that there
are not accidentals in this melody, the model does not know by representation,
the difference between natural, sharp and flats, they are just midi numbers.
It could learn this patterns of the C major scale. In the case of the interval
dataset, the LSTM (Fig. 11(a)) shows many changes and never repeat
consecutively notes, this was expected for the low centricity and high CMM.
the UGRNN (Fig. 11(c)) seems not to explore many changes in the pattern, and
the NAS (Fig. 11(b)) even if at the beginning of bar 5 seems to be stagnant in
a pattern A4-F♯4, in bar 7 it resolves, and makes a melody with clear use of
A4 as pedal note.

(a) LSTM Model

(b) NAS Model

(c) UGRNN Model

Figure 11: Generated Melodies Interval Dataset

##  6 Conclusions and future research

About the 3 cell types, it is interesting that in the representative melodies
of the models, the consecutive repetition of notes in LSTM is almost null,
especially with the DB12 dataset, this cell shows an excellent learning
capacity to keep the scale. The NAS model shows in general a high standard
deviation for the metrics when trains with DB12, NAS models need many
iterations to begin to converge (Sec 4), the combination of this aspect with
the more significant size of DB12 dataset results in a less stable model.
Finally, melodies from UGRNN models shows a good capacity to use consecutive
notes as root pattern for the melodies.

In the musical context, the depth of the net is not a trivial problem of using
so many layers as possible, section 4 shows many examples where after 2 or 3
layers, the cost function is worse. Therefore, it is necessary to make always
tests with gradual increments in the layer number to check the optimal.

The automatic metrics are a useful tool to see the general behavior of a
model, they can describe it and potentially be the reward to train the models
using a strategy like reinforcement learning. However, the 3 metrics used in
the paper only describe tonality, they does not have into account other
aspects like rhythm, timbre, performance or rubato. They are not enough to
make a full judgment of a melody, the best proof of this are some of the songs
(fig. 11(b), 10(a)), which even when their models do not have a highlight
performance in the comparison table 3, sound tonal and have interesting
patterns.

With respect to the dataset variations, the control dataset is the most stable
for the LM metric, the interval dataset shows a low centricity, and even when
none of the interval models had a highlight result in the metrics, its scores
are stable among them. The db12 variation models on the other hand, show a
high centricity and tends to repeat the notes a lot, but also, the embeddings
shows a better understanding of the notes musical meaning, and in general
seems to capture better the patterns than the interval variation. A DB12
representation tends to be more complicated than an interval like one, as
result, interval has a more stable behaviour in the metrics (Table 3),
however, if the purpose is to have a transposition free representation of
musical patterns, section 5.2 shows DB12 in combination with embedding
encoding as the best alternative.

Finally, the general results of the trained models shows that cross-entropy is
as valid as the objective function in the musical context as it has been
through the years for machine learning models in all the areas where it has
been applied.

As future work, the natural step will be to extend the model to support rhythm
and harmonies. It is relevant to see the potential of embeddings in music, our
results in this research show learning of particular features of the notes as
pitch level and note nature (fundamental or altered). In this vein, a broader
study, including harmonies and time changes can show more appealing patterns,
potentially, a trained embedding can cluster the musical elements in genres,
having applications in tasks like genre classification. It would also be
interesting to see the performance of our optimal nine models when used to
generate sequences in other areas as i) Text: poetry or story writing, using
the words which a poem should have, or the main idea of the story as
initialisation. ii) Audio: audio synthesis and voice generation. Finally, the
implementation of the automatic metrics is a good starting point to make them
more robust and include others applicable to rhythm and harmony. The inclusion
of qualitative evaluation to compare and refine the metrics can also improve
its capacity to measure music quality. These metrics could be used not only to
measure the quality and tonality of the final generated songs but to train the
model using reinforcement learning to optimise the metrics directly.

##  7 Acknowledgement

Special thanks to the Center of Excellence and Appropriation on Big Data and
Data Analytics (Alianza CAOBA) and to the APOLO high performance computing
centre for the GPU hardware provided, which was very helpful to reduce the
time of training.

##  8 Funding

This research was supported by the Center of Excellence and Appropriation on
Big Data and Data Analytics (Alianza CAOBA).

##  9 Supplementary Materials

As support for the paper, there is a web page in the url:
https://sebasgverde.github.io/rnn-cells-music-paper/ with all the material
necessesary for research replication, which includes:

  1. 1.

Datasets

  2. 2.

Network Weights

  3. 3.

Midi songs of the final models

  4. 4.

Link to demo videos

  5. 5.

RNN model

  6. 6.

Library for midi manipulation

  7. 7.

Library for music evaluation

  8. 8.

Detailed instructions to run the scripts

Scripts available include:

  1. 1.

Environment setting

  2. 2.

Datasets and weights downloading

  3. 3.

Datasets unit tests

  4. 4.

Layer experiment

  5. 5.

Generating songs with trained models

  6. 6.

Music Evaluation of models, songs and datasets

  7. 7.

Embeddings experiment

  8. 8.

Cross-entropy Validation

## References

  * [Allan and Williams, 2005] Allan, M. and Williams, C. (2005).  Harmonising Chorales by Probabilistic Inference.  In Saul, L. K., Weiss, Y., and Bottou, L., editors, Advances in Neural Information Processing Systems 17, pages 25–32. MIT Press.
  * [Bahdanau et al., 2014] Bahdanau, D., Cho, K., and Bengio, Y. (2014).  Neural Machine Translation by Jointly Learning to Align and Translate.  ArXiv e-prints, abs/1409.0.
  * [Berglund et al., 2015] Berglund, M., Raiko, T., Honkala, M., Kärkkäinen, L., Vetek, A., and Karhunen, J. T. (2015).  Bidirectional Recurrent Neural Networks as Generative Models.  In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28, pages 856–864. Curran Associates, Inc.
  * [Bertin-Mahieux et al., 2011] Bertin-Mahieux, T., Ellis, D. P. W., Whitman, B., and Lamere, P. (2011).  The Million Song Dataset.  In Proceedings of the 12th International Conference on Music Information Retrieval ({ISMIR} 2011).
  * [Boulanger-Lewandowski et al., 2012] Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012).  Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription.  ArXiv e-prints.
  * [Brown et al., 1992] Brown, P. F., DeSouza, P. V., Mercer, R. L., Pietra, V. J. D., and Lai, J. C. (1992).  Class-based N-gram Models of Natural Language.  Comput. Linguist., 18(4):467–479.
  * [Chai and Vercoe, 2001] Chai, W. and Vercoe, B. (2001).  Folk music classification using hidden Markov models.  PROC. OF INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE.
  * [Cho et al., 2014] Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014).  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.  ArXiv e-prints.
  * [Chung et al., 2014] Chung, J., Gülçehre, Ç., Cho, K., and Bengio, Y. (2014).  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.  CoRR, abs/1412.3.
  * [Ciresan et al., 2013] Ciresan, D. C., Giusti, A., Gambardella, L. M., and Schmidhuber, J. (2013).  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks.  In Mori, K., Sakuma, I., Sato, Y., Barillot, C., and Navab, N., editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013: 16th International Conference, Nagoya, Japan, September 22-26, 2013, Proceedings, Part II, pages 411–418. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * [Collins et al., 2016] Collins, J., Sohl-Dickstein, J., and Sussillo, D. (2016).  Capacity and Trainability in Recurrent Neural Networks.  ArXiv e-prints.
  * [Colombo et al., 2017] Colombo, F., Seeholzer, A., and Gerstner, W. (2017).  Deep Artificial Composer: A Creative Neural Network Model for Automated Melody Generation.  In Correia, J., Ciesielski, V., and Liapis, A., editors, Computational Intelligence in Music, Sound, Art and Design: 6th International Conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, pages 81–96. Springer International Publishing, Cham.
  * [De Prisco et al., 2017] De Prisco, R., Malandrino, D., Zaccagnino, G., Zaccagnino, R., and Zizza, R. (2017).  A Kind of Bio-inspired Learning of mUsic stylE.  In Correia, J., Ciesielski, V., and Liapis, A., editors, Computational Intelligence in Music, Sound, Art and Design: 6th International Conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, pages 97–113. Springer International Publishing, Cham.
  * [Deng and Kwok, 2018] Deng, J. and Kwok, Y.-K. (2018).  Large vocabulary automatic chord estimation using bidirectional long short-term memory recurrent neural network with even chance training.  Journal of New Music Research, 47(1):53–67.
  * [Farinella and Ros, 2014] Farinella, M. and Ros, H. (2014).  Neurocomic.  Norma Editorial S.A.
  * [Fischer and Igel, 2012] Fischer, A. and Igel, C. (2012).  An Introduction to Restricted Boltzmann Machines, pages 14–36.  Springer Berlin Heidelberg, Berlin, Heidelberg.
  * [Froment and Bonte, 2002] Froment, N. and Bonte, T. (2002).  MuseScore.
  * [Garcia-Valencia, 2020a] Garcia-Valencia, S. (2020a).  Cross entropy as objective function for music generative models.  ArXiv e-prints, abs/2006.0.
  * [Garcia-Valencia, 2020b] Garcia-Valencia, S. (2020b).  Embeddings as representation for symbolic music.  ArXiv e-prints, abs/2005.0.
  * [Gehring et al., 2017] Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. (2017).  Convolutional Sequence to Sequence Learning.  ArXiv e-prints.
  * [Goodfellow et al., 2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative Adversarial Networks.  ArXiv e-prints.
  * [Gu et al., 2015] Gu, S., Ghahramani, Z., and Turner, R. E. (2015).  Neural Adaptive Sequential Monte Carlo.  In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28, pages 2629–2637. Curran Associates, Inc.
  * [Herremans and Chuan, 2017] Herremans, D. and Chuan, C.-H. (2017).  Modeling Musical Context with Word2vec.  CoRR, abs/1706.0.
  * [Hochreiter and Schmidhuber, 1997] Hochreiter, S. and Schmidhuber, U. (1997).  LONG SHORT-TERM MEMORY.  Neural Computation, 9(8):1735–1780.
  * [Hörnel, 1998] Hörnel, D. (1998).  MELONET I: Neural Nets for Inventing Baroque-Style Chorale Variations.  In Jordan, M. I., Kearns, M. J., and Solla, S. A., editors, Advances in Neural Information Processing Systems 10, pages 887–893. MIT Press.
  * [Hutchings and McCormack, 2017] Hutchings, P. and McCormack, J. (2017).  Using Autonomous Agents to Improvise Music Compositions in Real-Time.  In Correia, J., Ciesielski, V., and Liapis, A., editors, Computational Intelligence in Music, Sound, Art and Design: 6th International Conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, pages 114–127. Springer International Publishing, Cham.
  * [Jaques et al., 2016] Jaques, N., Gu, S., Turner, R. E., and Eck, D. (2016).  Tuning Recurrent Neural Networks with Reinforcement Learning.  Thesis, pages 410–420.
  * [Johnson, 2017] Johnson, D. D. (2017).  Generating Polyphonic Music Using Tied Parallel Networks.  In Correia, J., Ciesielski, V., and Liapis, A., editors, Computational Intelligence in Music, Sound, Art and Design: 6th International Conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, pages 128–143. Springer International Publishing, Cham.
  * [Kaliakatsos-Papakostas et al., 2010] Kaliakatsos-Papakostas, M. A., Epitropakis, M. G., and Vrahatis, M. N. (2010).  Musical Composer Identification through Probabilistic and Feedforward Neural Networks.  In Di Chio, C., Brabazon, A., Di Caro, G. A., Ebner, M., Farooq, M., Fink, A., Grahl, J., Greenfield, G., Machado, P., O’Neill, M., Tarantino, E., and Urquhart, N., editors, Applications of Evolutionary Computation: EvoApplications 2010: EvoCOMNET, EvoENVIRONMENT, EvoFIN, EvoMUSART, and EvoTRANSLOG, Istanbul, Turkey, April 7-9, 2010, Proceedings, Part II, pages 411–420. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * [Karpathy, 2015] Karpathy, A. (2015).  The Unreasonable Effectiveness of Recurrent Neural Networks. In Andrej Karpathy blog.
  * [Kawthekar et al., 2017] Kawthekar, P., Rewari, R., and Bhooshan, S. (2017).  Evaluating Generative Models for Text Generation.  (Working paper).
  * [Krizhevsky et al., 2012] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).  ImageNet Classification with Deep Convolutional Neural Networks.  In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA. Curran Associates Inc.
  * [Levy and Goldberg, 2014] Levy, O. and Goldberg, Y. (2014).  Dependency-Based Word Embeddings.  In 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014 - Proceedings of the Conference, volume 2, pages 302–308.
  * [Madjiheurem et al., 2016] Madjiheurem, S., Qu, L., and Walder, C. (2016).  Chord2Vec: Learning Musical Chord Embeddings.  (Working paper).
  * [Mikolov et al., 2013a] Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a).  Efficient Estimation of Word Representations in Vector Space.  CoRR, abs/1301.3.
  * [Mikolov et al., 2013b] Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013b).  Distributed Representations of Words and Phrases and their Compositionality.  CoRR, abs/1310.4.
  * [Miller, 1956] Miller, G. A. (1956).  The Magical Number Seven, Plus or Minus Two: Some Limitis on our Capacity for Processing Information.  The Psychological Review, 63(2):81–97.
  * [MusicSales, 2000] MusicSales (2000).  The Little Book of Music Theory: An Essential Introduction to the Language of Music.  Amsco Publications.
  * [Olah, 2015] Olah, C. (2015).  Understanding LSTM Networks.
  * [Oliwa and Wagner, 2008] Oliwa, T. and Wagner, M. (2008).  Composing Music with Neural Networks and Probabilistic Finite-State Machines.  In Giacobini, M., Brabazon, A., Cagnoni, S., Di Caro, G. A., Drechsler, R., Ekárt, A., Esparcia-Alcázar, A. I., Farooq, M., Fink, A., McCormack, J., O’Neill, M., Romero, J., Rothlauf, F., Squillero, G., Uyar, A. b., and Yang, S., editors, Applications of Evolutionary Computing: EvoWorkshops 2008: EvoCOMNET, EvoFIN, EvoHOT, EvoIASP, EvoMUSART, EvoNUM, EvoSTOC, and EvoTransLog, Naples, Italy, March 26-28, 2008. Proceedings, pages 503–508. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * [Overett et al., 2008] Overett, G., Petersson, L., Brewer, N., Andersson, L., and Pettersson, N. (2008).  A new pedestrian dataset for supervised learning.  In 2008 IEEE Intelligent Vehicles Symposium, pages 373–378.
  * [Project Gutenberg, 2006] Project Gutenberg (2006).  The Sheet Music Project - Gutenberg.
  * [Project Mutopia, 2017] Project Mutopia (2017).  The Mutopia Project.
  * [Raphael, 2002] Raphael, C. (2002).  A Bayesian Network for Real-Time Musical Accompaniment.  In Dietterich, T. G., Becker, S., and Ghahramani, Z., editors, Advances in Neural Information Processing Systems 14, pages 1433–1439. MIT Press.
  * [Schmidhuber, 2014] Schmidhuber, J. (2014).  Deep Learning in Neural Networks: An Overview.  CoRR, abs/1404.7.
  * [Shannon, 1948] Shannon, C. (1948).  A mathematical theory of communication.  Bell system technical journal, 27.
  * [Thornton, 2011] Thornton, C. (2011).  Generation of Folk Song Melodies using Bayes Transforms.  Journal of New Music Research, 40(4):293–312.
  * [Tymoczko, 2011] Tymoczko, D. (2011).  A geometry of music : harmony and counterpoint in the extended common practice.  Oxford University Press.
  * [van der Wel and Ullrich, 2017] van der Wel, E. and Ullrich, K. (2017).  Optical Music Recognition with Convolutional Sequence-to-Sequence Models.  CoRR, abs/1707.0.
  * [Waite, 2016] Waite, E. (2016).  Generating Long-Term Structure in Songs and Stories.
  * [Walshaw Chris, 2017] Walshaw Chris (2017).  abc notation.
  * [Whorley and Conklin, 2016] Whorley, R. P. and Conklin, D. (2016).  Music Generation from Statistical Models of Harmony.  Journal of New Music Research, 45(2):160–183.
  * [Zoph and Le, 2016] Zoph, B. and Le, Q. V. (2016).  Neural Architecture Search with Reinforcement Learning.  CoRR, abs/1611.0.