# Noncommutative $C^{*}$-algebra Net: Learning Neural Networks with Powerful
Product Structure in $C^{*}$-algebra

Ryuichiro Hataya∗ RIKEN Advanced Data Science Project, Tokyo, Japan  RIKEN
Advanced Intelligence Project, Tokyo, Japan  Yuka Hashimoto Equal
contribution. NTT Network Service Systems Laboratories, Tokyo, Japan  RIKEN
Advanced Intelligence Project, Tokyo, Japan

###### Abstract

We propose a new generalization of neural networks with noncommutative
$C^{*}$-algebra. An important feature of $C^{*}$-algebras is their
noncommutative structure of products, but the existing $C^{*}$-algebra net
frameworks have only considered commutative $C^{*}$-algebras. We show that
this noncommutative structure of $C^{*}$-algebras induces powerful effects in
learning neural networks. Our framework has a wide range of applications, such
as learning multiple related neural networks simultaneously with interactions
and learning invariant features with respect to group actions. We also show
the validity of our framework numerically, which illustrates its potential
power.

##  1 Introduction

Generalization of the parameter space of neural networks from real numbers to
others has attracted researchers for decades. Although using real-valued
parameters is standard and straightforward for real-valued data, it may be
more suitable to adopt parameters of complex numbers [Hirose, 1992, Nishikawa
et al., 2005, Amin et al., 2008, Yadav et al., 2005, Trabelsi et al., 2018,
Lee et al., 2022] or quaternion numbers Nitta [1995], Arena et al. [1997], Zhu
et al. [2018], Gaudet and Maida [2018] for data in signal processing, computer
vision, and robotics domains, among others. Clifford-algebra, the
generalization of these numbers, allows more flexible geometrical processing
of data, and thus is applied to neural networks Pearson and Bisset [1994],
Buchholz [2005], Buchholz and Sommer [2008] to handle rich geometric
relationships in data [Rivera-Rovelo et al., 2010, Zang et al., 2022,
Brandstetter et al., 2022]. Different from these approaches focusing on the
geometric perspective of parameter values, another direction of generalization
is to use function-valued parameters Rossi and Conan-Guez [2005], Thind et al.
[2022], which broadens the applications of neural networks to functional data.

Hashimoto et al. [2022] proposed generalizing neural network parameters to
$C^{*}$-algebra, which is called $C^{*}$-algebra net. They showed that
multiple related neural networks can be continuously combined into a single
$C^{*}$-algebra net. For example, networks for the same task with different
training datasets or different initial parameters can be combined
continuously, which enables the construction of infinitely many networks and
efficient learning using shared information among them. Such interaction among
networks is also applicable to learning from related tasks, such as ensemble
learning [Dong et al., 2020, Ganaie et al., 2022] and multitask learning
[Zhang and Yang, 2022]. However, because the product structure in the
$C^{*}$-algebra that Hashimoto et al. [2021] focused on is commutative, they
needed specially designed loss functions to induce the interaction.

In this paper, we propose a new generalization of neural networks with
noncommutative $C^{*}$-algebra, which also generalizes the framework of
Hashimoto et al. [2022]. $C^{*}$-algebra is a generalization of the space of
complex numbers Murphy [1990], Hashimoto et al. [2021]. Typical examples of
$C^{*}$-algebras include the space of diagonal matrices and the space of (not
necessarily diagonal) matrices. An important feature of $C^{*}$-algebra is the
product structure. Analogous to the case of complex numbers, for any two
elements $a$ and $b$ in a $C^{*}$-algebra, we can calculate the product
$a\cdot b$. However, unlike the case of complex numbers, the product is not
always commutative, i.e., $a\cdot b=b\cdot a$ is not always satisfied. For
example, whereas the product of two diagonal matrices commutes, the product of
two nondiagonal matrices does not commute. In the case of diagonal matrices,
the product is just described by the product of each diagonal element, and
there are no interactions with the other elements. On the other hand, the
product of two nondiagonal matrices is described by the sum of the products
between different elements in the matrices, which induces interactions with
the other elements.

For neural networks, the interactions induced by the noncommutative
$C^{*}$-algebras correspond to interactions among multiple neural networks.
Because the interactions are encoded in the structure of the network, we do
not need to design loss functions to make the networks interact. Instead, the
interactions required for them are implicitly and automatically learned
through the noncommutative product structure in $C^{*}$-algebras. Therefore,
our framework, which takes advantage of noncommutative $C^{*}$-algebras
enables us to go beyond existing frameworks of neural networks.

Our framework of the noncommutative $C^{*}$-algebra net is general and has a
wide range of applications, not limited to the above case of matrices. For
example, by setting the $C^{*}$-algebra as a group $C^{*}$-algebra, we can
construct a group equivariant neural network. In this case, we can naturally
generalize real-valued parameters of neural networks to those inducing group
convolution. We can also set the $C^{*}$-algebra as the $C^{*}$-algebra of
bounded linear operators on a function space, which can be applied to
analyzing functional data. Our main contributions are summarized as follows:

  * •

We generalize the commutative $C^{*}$-algebra net proposed by Hashimoto et al.
[2022] to noncommutative $C^{*}$-algebra, which enables us to take advantage
of the noncommutative product structure in the $C^{*}$-algebra when learning
neural networks.

  * •

We show a wide range of applications of our framework, inducing interactions
among networks and learning invariant features with respect to group actions.

  * •

We empirically illustrate the validity of noncommutative $C^{*}$-algebra nets,
including interactions among neural networks.

We emphasize that $C^{*}$-algebra is a powerful tool for neural networks, and
our work provides a lot of important perspectives about its application.

##  2 Background

In this section, we review the mathematical background of $C^{*}$-algebra
required for this paper and the existing $C^{*}$-algebra net. For more
theoretical details of the $C^{*}$-algebra, see, for example, Murphy [1990].

###  2.1 $C^{*}$-algebra

$C^{*}$-algebra is a generalization of the space of complex values. It has
structures of the product, involution ∗, and norm.

######  Definition 1 ($C^{*}$-algebra)

A set $\mathcal{A}$ is called a $C^{*}$-algebra if it satisfies the following
conditions:

  1. 1.

$\mathcal{A}$ is an algebra over $\mathbb{C}$ and equipped with a bijection
$(\cdot)^{*}:\mathcal{A}\to\mathcal{A}$ that satisfies the following
conditions for $\alpha,\beta\in\mathbb{C}$ and $c,d\in\mathcal{A}$:

     * •

$(\alpha c+\beta d)^{*}=\overline{\alpha}c^{*}+\overline{\beta}d^{*}$,

     * •

$(cd)^{*}=d^{*}c^{*}$,

     * •

$(c^{*})^{*}=c$.

  2. 2.

$\mathcal{A}$ is a normed space with $\|\cdot\|$, and for $c,d\in\mathcal{A}$,
$\|cd\|\leq\|c\|\,\|d\|$ holds. In addition, $\mathcal{A}$ is complete with
respect to $\|\cdot\|$.

  3. 3.

For $c\in\mathcal{A}$, $\|c^{*}c\|=\|c\|^{2}$ holds.

The product structure in $C^{*}$-algebras can be both commutative and
noncommutative.

######  Example 1 (Commutative $C^{*}$-algebra)

Let $\mathcal{A}$ be the space of continuous functions on a compact Hausdorff
space $\mathcal{Z}$. We can regard $\mathcal{A}$ as a $C^{*}$-algebra by
setting

  * •

Product: Pointwise product of two functions, i.e., for
$a_{1},a_{2}\in\mathcal{A}$, $a_{1}a_{2}(z)=a_{1}(z)a_{2}(z)$.

  * •

Involution: Pointwise complex conjugate, i.e., for $a\in\mathcal{A}$,
$a^{*}(z)=\overline{a(z)}$.

  * •

Norm: Sup norm, i.e., for $a\in\mathcal{A}$,
$\|a\|=\sup_{z\in\mathcal{Z}}|a(z)|$.

In this case, the product in $\mathcal{A}$ is commutative.

######  Example 2 (Noncommutative $C^{*}$-algebra)

Let $\mathcal{A}$ be the space of bounded linear operators on a Hilbert space
$\mathcal{H}$, which is denoted by $\mathcal{B}(\mathcal{H})$. We can regard
$\mathcal{A}$ as a $C^{*}$-algebra by setting

  * •

Product: Composition of two operators,

  * •

Involution: Adjoint of an operator,

  * •

Norm: Operator norm of an operator, i.e., for $a\in\mathcal{A}$,
$\|a\|=\sup_{v\in\mathcal{H},\|v\|_{\mathcal{H}}=1}\|av\|_{\mathcal{H}}$.

Here, $\|\cdot\|_{\mathcal{H}}$ is the norm in $\mathcal{H}$. In this case,
the product in $\mathcal{A}$ is noncommutative. Note that if $\mathcal{H}$ is
a $d$-dimensional space for a finite natural number $d$, then elements in
$\mathcal{A}$ are $d$ by $d$ matrices.

######  Example 3 (Group $C^{*}$-algebra)

The group $C^{*}$-algebra on a group $G$, which is denoted as $C^{*}(G)$, is
the set of maps from $G$ to $\mathbb{C}$ equipped with the following product,
involution, and norm:

  * •

Product: $(a\cdot b)(g)=\int_{G}a(h)b(h^{-1}g)\mathrm{d}\lambda(h)$ for $g\in
G$,

  * •

Involution: $a^{*}(g)=\Delta(g^{-1})\overline{a(g^{-1})}$ for $g\in G$,

  * •

Norm: $\|a\|=\sup_{[\pi]\in\hat{G}}\|\pi(a)\|$,

where $\Delta(g)$ is a positive number satisfying
$\lambda(Eg)=\Delta(g)\lambda(E)$ for the Haar measure $\lambda$ on $G$. In
addition, $\hat{G}$ is the set of equivalence classes of irreducible unitary
representations of $G$. Note that if $G$ is discrete, then $\lambda$ is the
counting measure on $G$. In this paper, we focus mainly on the product
structure of $C^{*}(G)$. For details of the Haar measure and representations
of groups, see Kirillov [1976]. If $G=\mathbb{Z}/p\mathbb{Z}$, then $C^{*}(G)$
is $C^{*}$-isomorphic to the $C^{*}$-algebra of circulant matrices [Hashimoto
et al., 2023]. Note also that if $G$ is noncommutative, then $C^{*}(G)$ can
also be noncommutative.

###  2.2 $C^{*}$-algebra net

Hashimoto et al. [2022] proposed generalizing real-valued neural network
parameters to commutative $C^{*}$-algebra-valued ones. Here, we briefly review
the existing (commutative) $C^{*}$-algebra net. Let
$\mathcal{A}=C(\mathcal{Z})$, the commutative $C^{*}$-algebra of continuous
functions on a compact Hausdorff space $\mathcal{Z}$. Let $H$ be the depth of
the network and $N_{0},\ldots,{N_{H}}$ be the width of each layer. For
$i=1,\ldots,H$, set $W_{i}:\mathcal{A}^{N_{i-1}}\to\mathcal{A}^{N_{i}}$ as an
Affine transformation defined with an $N_{i}\times N_{i-1}$
$\mathcal{A}$-valued matrix and an $\mathcal{A}$-valued bias vector in
$\mathcal{A}^{N_{i}}$. In addition, set a nonlinear activation function
$\sigma_{i}:\mathcal{A}^{N_{i}}\to\mathcal{A}^{N_{i}}$. The commutative
$C^{*}$-algebra net $f:\mathcal{A}^{N_{0}}\to\mathcal{A}^{N_{H}}$ is defined
as

$f=\sigma_{H}\circ W_{H}\circ\cdots\circ\sigma_{1}\circ W_{1}.$ (1)

By generalizing neural network parameters to functions, we can combine
multiple standard (real-valued) neural networks continuously, which enables
them to learn efficiently. We show an example of commutative $C^{*}$-nets
below. To simplify the notation, we focus on the case where the network does
not have biases. However, the same arguments are valid for the case where the
network has biases.

####  2.2.1 The case of diagonal matrices

If $\mathcal{Z}$ is a finite set, then $\mathcal{A}=\\{a\in\mathbb{C}^{d\times
d}\,\mid\,a\mbox{ is a diagonal matrix}\\}$. The $C^{*}$-algebra net $f$ on
$\mathcal{A}$ corresponds to $d$ separate real or complex-valued sub-models.
Indeed, denote by $x^{j}$ the vector composed of the $j$th diagonal elements
of $x\in\mathcal{A}^{N}$, which is defined as the vector in $\mathbb{C}^{N}$
whose $k$th element is the $j$th diagonal element of the $\mathcal{A}$-valued
$k$th element of $x$. Assume the activation function
$\sigma_{i}:\mathcal{A}^{N}\to\mathcal{A}^{N}$ is defined as
$\sigma_{i}(x)^{j}=\tilde{\sigma}_{i}(x^{j})$ for some
$\tilde{\sigma}_{i}:\mathbb{C}^{N}\to\mathbb{C}^{N}$. Since the $j$th diagonal
element of $a_{1}a_{2}$ for $a_{1},a_{2}\in\mathcal{A}$ is the product of the
$j$th element if $a_{1}$ and $a_{2}$, we have

$\displaystyle f(x)^{j}=\tilde{\sigma}_{H}\circ
W_{H}^{j}\circ\cdots\circ\tilde{\sigma}_{1}\circ W_{1}^{j},$ (2)

where $W_{i}^{j}\in\mathbb{C}^{N_{i}\times N_{i-1}}$ is the matrix whose
$(k,l)$-entry is the $j$th diagonal of the $(k,l)$-entry of
$W_{i}\in\mathcal{A}^{N_{i}\times N_{i-1}}$. Figure 1 (a) schematically shows
the $C^{*}$-algebra net over diagonal matrices.

(a) $C^{*}$-algebra net over diagonal matrices (commutative). Each diagonal
element corresponds to an individual sub-model.

(b) $C^{*}$-algebra net over nondiagonal matrices (noncommutative). The
nondiagonal elements induce the interactions among multiple sub-models.

Figure 1: Difference between commutative and noncommutative $C^{*}$-algebra
nets.

##  3 Noncommutative $C^{*}$-algebra Net

Although the existing $C^{*}$-algebra net provides a framework for applying
$C^{*}$-algebra to neural networks, it focuses on commutative
$C^{*}$-algebras, whose product structure is simple. Therefore, we generalize
the existing commutative $C^{*}$-algebra net to noncommutative
$C^{*}$-algebra. Since the product structures in noncommutative
$C^{*}$-algebras are more complicated than those in commutative
$C^{*}$-algebras, they enable neural networks to learn features of data more
efficiently. For example, if we focus on the $C^{*}$-algebra of matrices, then
the neural network parameters describe interactions between multiple real-
valued sub-models (see Section 3.1.1).

Let $\mathcal{A}$ be a general $C^{*}$-algebra and consider the network $f$ in
the same form as Equation 1. We emphasize that in our framework, the choice of
$\mathcal{A}$ is not restricted to a commutative $C^{*}$-algebra. We list
examples of $\mathcal{A}$ and their validity for learning neural networks
below.

###  3.1 Examples of $C^{*}$-algebras for neural networks

As mentioned in the previous section, we focus on the case where the network
does not have biases for simplification in this subsection.

####  3.1.1 Nondiagonal matrices

Let $\mathcal{A}=\mathbb{C}^{d\times d}$. Note that $\mathcal{A}$ is a
noncommutative $C^{*}$-algebra. In this case, unlike the network (2), the
$j$th diagonal element of $a_{1}a_{2}a_{3}$ for
$a_{1},a_{2},a_{3}\in\mathcal{A}$ depends not only on the $j$th diagonal
element of $a_{2}$, but also the other diagonal elements of $a_{2}$. Thus,
$f(x)^{j}$ depends not only on the sub-model corresponding to $j$th diagonal
element discussed in Section 2.2.1, but also on other sub-models. The
nondiagonal elements in $\mathcal{A}$ induce interactions between $d$ real or
complex-valued sub-models. In practice, to regard the nondiagonal elements as
factors of interactions, their values should be small compared to the diagonal
elements. We will see the effect of the nondiagonal elements in $\mathcal{A}$
numerically in Section 4.1. Figure 1 (b) schematically shows the
$C^{*}$-algebra net over nondiagonal matrices.

####  3.1.2 Block diagonal matrices

Let $\mathcal{A}=\\{a\in\mathbb{C}^{d\times
d}\,\mid\,a=\operatorname{diag}(\mathbf{a}_{1},\ldots,\mathbf{a}_{m}),\
\mathbf{a}_{i}\in\mathbb{C}^{d_{i}\times d_{i}}\\}$. The product of two block
diagonal matrices
$a=\operatorname{diag}(\mathbf{a}_{1},\ldots,\mathbf{a}_{m})$ and
$b=\operatorname{diag}(\mathbf{b}_{1},\ldots,\mathbf{b}_{m})$ can be written
as

$\displaystyle
ab=\operatorname{diag}(\mathbf{a}_{1}\mathbf{b}_{1},\ldots,\mathbf{a}_{m}\mathbf{b}_{m}).$

In a similar manner to Section 2.2.1, we denote by $\mathbf{x}^{j}$ the $N$ by
$d_{j}$ matrix composed of the $j$th diagonal blocks of $x\in\mathcal{A}^{N}$.
Assume the activation function $\sigma_{i}:\mathcal{A}^{N}\to\mathcal{A}^{N}$
is defined as
$\sigma_{i}(x)=\operatorname{diag}(\tilde{\bm{\sigma}}_{i}^{1}(\mathbf{x}^{1}),\ldots,\tilde{\bm{\sigma}}_{i}^{m}(\mathbf{x}^{m}))$
for some $\tilde{\bm{\sigma}}_{i,j}:\mathbb{C}^{N\times
d_{j}}\to\mathbb{C}^{N\times d_{j}}$. Then, we have

$\displaystyle\mathbf{f(x)}^{j}=\tilde{\bm{\sigma}}_{H}^{j}\circ\mathbf{W}_{H}^{j}\circ\cdots\circ\tilde{\bm{\sigma}}_{1}^{j}\circ\mathbf{W}_{1}^{j},$
(3)

where $\mathbf{W}_{i}^{j}\in(\mathbb{C}^{d_{j}\times d_{j}})^{N_{i}\times
N_{i-1}}$ is the block matrix whose $(k,l)$-entry is the $j$th block diagonal
of the $(k,l)$-entry of $W_{i}\in\mathcal{A}^{N_{i}\times N_{i-1}}$. In this
case, we have $m$ groups of sub-models, each of which is composed of
interacting $d_{j}$ sub-models mentioned in Section 3.1.1. Indeed, the block
diagonal case generalizes the diagonal and nondiagonal cases stated in
Sections 2.2.1 and 3.1.1. If $d_{j}=1$ for all $j=1,\ldots,m$, then the
network (3) is reduced to the network (2) with diagonal matrices. If $m=1$ and
$d_{1}=d$, then the network (3) is reduced to the network with $d$ by $d$
nondiagonal matrices.

####  3.1.3 Circulant matrices

Let $\mathcal{A}=\\{a\in\mathbb{C}^{d\times d}\,\mid\,a\mbox{ is a circulant
matrix}\\}$. Here, a circulant matrix $a$ is the matrix represented as

$\displaystyle a=\begin{bmatrix}a_{1}&a_{d}&\cdots&a_{2}\\\
a_{2}&a_{1}&\cdots&a_{3}\\\ &\ddots&\ddots&\\\
a_{d}&a_{d-1}&\cdots&a_{1}\end{bmatrix}$

for $a_{1},\ldots,a_{d}\in\mathbb{C}$. Note that in this case, $\mathcal{A}$
is commutative. Circulant matrices are diagonalized by the discrete Fourier
matrix as follows [Davis, 1979]. We denote by $F$ the discrete Fourier
transform matrix, whose $(i,j)$-entry is $\omega^{(i-1)(j-1)}/\sqrt{p}$, where
$\omega=\mathrm{e}^{2\pi\sqrt{-1}/d}$.

###### Lemma 1

Any circulant matrix $a$ is decomposed as $a=F\Lambda_{a}F^{*}$, where

$\Lambda_{a}=\operatorname{diag}\bigg{(}\sum_{i=1}^{d}a_{i}\omega^{(i-1)\cdot
0},\ldots,\sum_{i=1}^{d}a_{i}\omega^{(i-1)(d-1)}\bigg{)}.$

Since $ab=F\Lambda_{a}\Lambda_{b}F^{*}$ for $a,b\in\mathcal{A}$, the product
of $a$ and $b$ corresponds to the multiplication of each Fourier component of
$a$ and $b$. Assume the activation function
$\sigma_{i}:\mathcal{A}^{N}\to\mathcal{A}^{N}$ is defined such that
$(F^{*}\sigma_{i}(x)F)^{j}$ equals to
$\hat{\tilde{\sigma}}_{i}((FxF^{*})^{j})$ for some
$\hat{\tilde{\sigma}}_{i}:\mathbb{C}^{N}\to\mathbb{C}^{N}$. Then, we obtain
the network

$\displaystyle(F^{*}f(x)F)^{j}=\hat{\tilde{\sigma}}_{H}\circ\hat{W}_{H}^{j}\circ\cdots\circ\hat{\tilde{\sigma}}_{1}\circ\hat{W}_{1}^{j},$
(4)

where $\hat{W}_{i}^{j}\in\mathbb{C}^{N_{i}\times N_{i-1}}$ is the matrix whose
$(k,l)$-entry is $(Fw_{i,k,l}F^{*})^{j}$, where $w_{i,k,l}$ is the the
$(k,l)$-entry of $W_{i}\in\mathcal{A}^{N_{i}\times N_{i-1}}$. The $j$th sub-
model of the network (4) corresponds to the network of the $j$th Fourier
component.

###### Remark 1

The $j$th sub-model of the network (4) does not interact with those of other
Fourier components than the $j$th component. This fact corresponds to the fact
that $\mathcal{A}$ is commutative in this case. Analogous to the case in
Section 3.1.1, if we set $\mathcal{A}$ as noncirculant matrices, then we
obtain interactions between sub-models corresponding to different Fourier
components.

####  3.1.4 Group $C^{*}$-algebra on a symmetric group

Let $G$ be the symmetric group on the set $\\{1,\ldots,d\\}$ and let
$\mathcal{A}=C^{*}(G)$. Note that since $G$ is noncomuutative, $C^{*}(G)$ is
also noncommutative. Then, the output $f(x)\in\mathcal{A}^{N_{H}}$ is the
$\mathbb{C}^{N_{H}}$-valued map on $G$. Using the product structure introduced
in Example 3, we can construct a network that takes the permutation of data
into account. Indeed, an element $w\in\mathcal{A}$ of a weight matrix
$W\in\mathcal{A}^{N_{i-1}\times N_{i}}$ is a function on $G$. Thus, $w(g)$
describes the weight corresponding to the permutation $g\in G$. Since the
product of $x\in C^{*}(G)$ and $w$ is defined as $wx(g)=\sum_{h\in
G}w(h)x(h^{-1}g)$, by applying $W$, all the weights corresponding to the
permutations affect the input. For example, let $z\in\mathbb{R}^{d}$ and set
$x\in C^{*}(G)$ as $x(g)=g\cdot z$, where $g\cdot z$ is the action of $g$ on
$z$, i.e., the permutation of $z$ with respect to $g$. Then, we can input all
the patterns of permutations of $z$ simultaneously, and by virtue of the
product structure in $C^{*}(G)$, the network is learned with the interaction
among these permutations. Regarding the output, if the network is learned so
that the outputs $y$ become constant functions on $G$, i.e., $y(g)=c$ for some
constant $c$, then it means that $c$ is invariant with respect to $g$, i.e.,
invariant with respect to the permutation. We will numerically investigate the
application of the group $C^{*}$-algebra net to permutation invariant problems
in Section 4.2.

###### Remark 2

If the activation function $\sigma$ is defined as $\sigma(x)(g)=\sigma(x(g))$,
i.e., applied elementwisely to $x$, then the network $f$ is permutation
equivariant. That is, even if the input $x(g)$ is replaced by $x(gh)$ for some
$h\in G$, the output $f(x)(g)$ is replaced by $f(x)(gh)$. This is because the
product in $C^{*}(G)$ is defined as a convolution. This feature of the
convolution has been studied for group equivariant neural networks [Lenssen et
al., 2018, Cohen et al., 2019, Sannai et al., 2021, Sonoda et al., 2022]. The
above setting of the $C^{*}$-algebra net provides us with a design of group
equivariant networks from the perspective of $C^{*}$-algebra.

####  3.1.5 Bounded linear operators on a Hilbert space

For functional data, we can also set $\mathcal{A}$ as an infinite-dimensional
space. Using infinite-dimensional $C^{*}$-algebra for analyzing functional
data has been proposed [Hashimoto et al., 2021]. We can also adopt this idea
for neural networks. Let $\mathcal{A}=\mathcal{B}(L^{2}(\Omega))$ for a
measure space $\Omega$. Set $\mathcal{A}_{0}=\\{a\in\mathcal{A}\,\mid\,a\mbox{
is a multiplication operator}\\}$. Here, a multiplication operator $a$ is a
linear operator that is defined as $av=v\cdot u$ for some $u\in
L^{\infty}(\Omega)$. The space $\mathcal{A}_{0}$ is a generalization of the
space of diagonal matrices to the infinite-dimensional space. If we restrict
elements of weight matrices to $\mathcal{A}_{0}$, then we obtain infinitely
many sub-models without interactions. Since outputs are in
$\mathcal{A}_{0}^{N_{H}}$, we can obtain functional data as outputs. Similar
to the case of matrices (see Section 3.1.1), by setting elements of weight
matrices as elements in $\mathcal{A}$, we can take advantage of interactions
among infinitely many sub-models.

###  3.2 Approximation of functions with interactions by $C^{*}$-algebra net

We observe what kind of functions the $C^{*}$-algebra net can approximate. We
focus on the case of $\mathcal{A}=\mathbb{C}^{d\times d}$. Consider a shallow
network $f:\mathcal{A}^{N_{0}}\to\mathcal{A}$ defined as
$f(x)=W_{2}^{*}\sigma(W_{1}x+b)$, where $W_{1}\in\mathcal{A}^{N_{1}\times
N_{0}}$, $W_{2}\in\mathcal{A}^{N_{1}}$, and $b\in\mathcal{A}^{N_{1}}$. Let
$\tilde{f}:\mathcal{A}^{N_{0}}\to\mathcal{A}$ be the function in the form of
$\tilde{f}(x)=[\sum_{j=1}^{d}f_{kj}(x^{l})]_{kl}$, where
$f_{kj}:\mathbb{C}^{N_{0}d}\to\mathbb{R}$. Here, we abuse the notation and
denote by $x^{l}\in\mathbb{C}^{N_{0}d}$ the $l$th column of $x$ regarded as an
$N_{0}d$ by $d$ matrix. Assume $f_{kj}$ is represented as

$\displaystyle
f_{kj}(x)=\int_{\mathbb{R}}\int_{\mathbb{R}^{N_{0}d}}T_{kj}(w,b)\sigma(w^{*}x+b)\mathrm{d}w\,\mathrm{d}b$
(5)

for some $T_{kj}:\mathbb{R}^{N_{0}d}\times\mathbb{R}\to\mathbb{R}$. By the
theory of the ridgelet transform, such $T_{kj}$ exists for most realistic
settings [Candès, 1999, Sonoda and Murata, 2017]. For example, if $f_{kj}$ and
its Fourier transform is in $L^{1}(\mathbb{R}^{N_{0}d})$ and $\sigma$ is the
ReLU function, then $f_{kj}$ has a representation of Equation 5. We discretize
Equation 5 by replacing the Lebesgue measures with
$\sum_{i=1}^{N_{1}}\delta_{w_{ij}}$ and $\sum_{i=1}^{N_{1}}\delta_{b_{ij}}$,
where $\delta_{w}$ is the Dirac measure centered at $w$. Then, the
$(k,l)$-entry of $\tilde{f}(x)$ is written as

$\displaystyle\sum_{j=1}^{d}\sum_{i=1}^{N_{1}}T_{kj}(w_{ij},b_{ij})\sigma(w_{ij}^{*}x^{l}+b_{ij}).$

Setting the $i$-th element of $W_{2}\in\mathcal{A}^{N_{1}}$ as
$[T_{kj}(w_{ij},b_{ij})]_{kj}$, the $(i,m)$-entry of
$W_{1}\in\mathcal{A}^{N_{1}\times N_{0}}$ as $[(w_{i,j})_{md+l}]_{jl}$, the
$i$th element of $b\in\mathcal{A}^{N_{1}}$ as $[b_{j}]_{jl}$, we obtain

$\displaystyle\sum_{j=1}^{d}\sum_{i=1}^{N_{1}}T_{kj}(w_{ij},b_{ij})\sigma(w_{ij}^{*}x^{l}+b_{ij})=(W_{2}^{k})^{*}\sigma(W_{1}x^{l}+b^{l}),$

which is the $(k,l)$-entry of $f(x)$.

###### Remark 3

As we discussed in Sections 2.2.1 and 3.1.1, a $C^{*}$-algebra net over
matrices can be regarded as $d$ interacting sub-models. The above argument
insists the $l$th column of $f(x)$ and $\tilde{f}(x)$ depends only on $x^{l}$.
Thus, in this case, if we input data $x^{l}$ corresponding to the $l$th sub-
model, then the output is obtained as the $l$th column of the
$\mathcal{A}$-valued output $f(x)$. On the other hand, the weight matrices
$W_{1}$ and $W_{2}$ and the bias $b$ are used commonly in providing the
outputs for any sub-model, i.e., $W_{1}$, $W_{2}$, and $b$ are learned using
data corresponding to all the sub-models. Therefore, $W_{1}$, $W_{2}$, and $b$
induce interactions among the sub-models.

##  4 Experiments

In this section, we numerically demonstrate the abilities of noncommutative
$C^{*}$-algebra nets using nondiagonal $C^{*}$-algebra nets over matrices and
group $C^{*}$-algebra nets. We use $C^{*}$-algebra-valued multi-layered
perceptrons (MLPs) to simplify the experiments. However, they can be naturally
extended to other neural networks, such as convolutional neural networks.

The models were implemented with JAX [Bradbury et al., 2018]. Experiments were
conducted on an AMD EPYC 7543 CPU and an NVIDIA A-100 GPU with CUDA 11.7. See
Section 6.1 for additional information on experiments.

###  4.1 $C^{*}$-algebra nets over matrices

In a noncommutative $C^{*}$-algebra net over matrices consisting of
nondiagonal-matrix parameters, each sub-model is expected to interact with
others and thus improve performance compared with its commutative counterpart
consisting of diagonal matrices. We demonstrate the effectiveness of such
interaction using image classification and neural implicit representation
(NIR) tasks.

See Section 3.1.1 for the notations. When training the $j$th sub-model
($j=1,2,\dots,d$), an original $N_{0}$-dimensional input data point
$\bm{x}=[\bm{x}_{1},\ldots,\bm{x}_{N_{0}}]\in\mathbb{R}^{N_{0}}$ is converted
to its corresponding representation
$x\in\mathcal{A}^{N_{0}}=\mathbb{R}^{N_{0}\times d\times d}$ such that
$x_{i,j,j}=\bm{x}_{i}$ for $i=1,2,\dots,N_{0}$ and $0$ otherwise. The loss to
its $N_{H}$-dimensional output of a $C^{*}$-algebra net
$y\in\mathcal{A}^{N_{H}}$ and the target $t\in\mathcal{A}^{N_{H}}$ is computed
as $\ell(y_{:,j,j},t_{:,j,j})+\frac{1}{2}\sum_{k,(l\neq
j)}(y_{k,j,l}^{2}+y_{k,l,j}^{2})$, where $\ell$ is a certain loss function; we
use the mean squared error (MSE) for image classification and the Huber loss
for NIR. The second and third terms suppress the nondiagonal elements of the
outputs to $0$. In both examples, we use leaky-ReLU as an activation function
and apply it only to the diagonal elements of pre-activations.

####  4.1.1 Image classification

We conduct experiments of image classification tasks using MNIST [Le Cun et
al., 1998], Kuzushiji-MNIST [Clanuwat et al., 2018], and Fashion-MNIST [Xiao
et al., 2017], which are composed of 10-class $28\times 28$ gray-scale images.
Each sub-model is trained on a mutually exclusive subset sampled from the
original training data and then evaluated on the entire test data. Each subset
is sampled to be balanced, i.e., each class has the same number of training
samples. As a baseline, we use a commutative $C^{*}$-algebra net over diagonal
matrices, which consists of the same sub-models but cannot interact with other
sub-models. Both noncommutative and commutative models share hyperparameters:
the number of layers was set to 4, the hidden size was set to 128, and the
models were trained for 30 epochs.

Table 1 shows the average test accuracy over sub-models. As can be seen, a
noncommutative $C^{*}$-algebra net consistently outperforms its commutative
counterpart, which is significant, particularly when the number of sub-models
is 40. Note that when the number of sub-models is 40, the size of the training
dataset for each sub-model is 40 times smaller than the original one, and
thus, the commutative $C^{*}$-algebra net fails to learn. Nevertheless, the
noncommutative $C^{*}$-algebra net retains performance mostly. These results
suggest that sub-models share knowledge through interaction.

Additionally, Table 2 illustrates that related tasks help performance
improvement through interaction. Specifically, we prepare five sub-models per
dataset, one of MNIST, Kuzushiji-MNIST, and Fashion-MNIST, and train a total
of 15 sub-models simultaneously. In addition to the commutative
$C^{*}$-algebra net, where sub-models have no interaction, and the
noncommutative $C^{*}$-algebra net, where each sub-model can interact with any
other sub-models, we use a block-diagonal noncommutative $C^{*}$-algebra net
(see Section 3.1.2), where each sub-model can only interact with other sub-
models trained on the same dataset. Table 2 shows that the fully
noncommutative $C^{*}$-algebra net surpasses the block-diagonal one on
Kuzushiji-MNIST and Fashion-MNIST, implying that not only intra-task
interaction but also inter-task interaction helps performance gain. Note that
each dataset is subsampled so that every class has the same number of samples,
so it is not possible to compare the values of Tables 1 and 2.

Dataset |  # sub-models |  Commutative
$C^{*}$-algebra net (baseline) |  Noncommutative
$C^{*}$-algebra net
---|---|---|---
MNIST |  5 |  $0.963\pm 0.003$ |  $0.970\pm 0.003$
10 |  $0.937\pm 0.004$ |  $0.956\pm 0.002$
20 |  $0.817\pm 0.018$ |  $0.932\pm 0.003$
40 |  $0.107\pm 0.008$ |  $0.858\pm 0.004$
Kuzushiji-MNIST |  5 |  $0.839\pm 0.002$ |  $0.858\pm 0.003$
10 |  $0.770\pm 0.006$ |  $0.813\pm 0.006$
20 |  $0.577\pm 0.024$ |  $0.746\pm 0.008$
40 |  $0.101\pm 0.004$ |  $0.577\pm 0.010$
Fashion-MNIST |  5 |  $0.861\pm 0.002$ |  $0.868\pm 0.002$
10 |  $0.837\pm 0.002$ |  $0.852\pm 0.004$
20 |  $0.740\pm 0.007$ |  $0.829\pm 0.004$
40 |  $0.103\pm 0.010$ |  $0.782\pm 0.005$
Table 1: Average test accuracy over sub-models of commutative and noncommutative $C^{*}$-algebra nets over matrices on test datasets. Interactions between sub-models that the noncommutative $C^{*}$-algebra net introduces improve performance significantly when the number of sub-models is 40. Dataset |  Commutative
$C^{*}$-algebra net |  Block-diagonal
noncommutative
$C^{*}$-algebra net |  Noncommutative
$C^{*}$-algebra net
---|---|---|---
MNIST |  $0.957\pm 0.001$ |  $0.969\pm 0.002$ |  $0.970\pm 0.002$
K-MNIST |  $0.744\pm 0.005$ |  $0.773\pm 0.006$ |  $0.787\pm 0.008$
F-MNIST |  $0.761\pm 0.002$ |  $0.797\pm 0.008$ |  $0.811\pm 0.003$
Table 2: Average test accuracy over five sub-models simultaneously trained on
the tree datasets. The (fully) noncommutative $C^{*}$-algebra net outperforms
block-diagonal the noncommutative $C^{*}$-algbebra net on Kuzushiji-MNIST
(K-MNIST) and Fashion-MNIST (F-MNIST), indicating that the interaction can
leverage related tasks.

####  4.1.2 Neural implicit representation

In the next experiment, we use a $C^{*}$-algebra net over matrices to learn
implicit representations of 2D images that map each pixel coordinate to its
RGB colors [Sitzmann et al., 2020, Xie et al., 2022]. Specifically, an input
coordinate in $[0,1]^{2}$ is transformed into a random Fourier features in
$[-1,1]^{320}$ and then converted into its $C^{*}$-algebraic representation
over matrices as an input to a $C^{*}$-algebra net over matrices. Similar to
the image classification task, we compare noncommutative NIRs with commutative
NIRs, using the following hyperparameters: the number of layers to 6 and the
hidden dimension to 256. These NIRs learn $128\times 128$-pixel images of
ukiyo-e pictures from The Metropolitan Museum of
Art111https://www.metmuseum.org/art/the-collection and photographs of cats
from the AFHQ dataset [Choi et al., 2020].

Figure 2 (top) shows the curves of the average PSNR (Peak Signal-to-Noise
Ratio) of sub-models corresponding to the image below. Both commutative and
noncommutative $C^{*}$-algebra nets consist of five sub-models trained on five
ukiyo-e pictures (see also Figure 6). PSNR, the quality measure, of the
noncommutative NIR grows faster, and correspondingly, it learns the details of
ground truth images faster than its commutative version (Figure 2 bottom).
Noticeably, the noncommutative representations reproduce colors even at the
early stage of learning, although the commutative ones remain monochrome after
500 iterations of training. Along with the similar trends observed in the
pictures of cats (Figure 3), these results further emphasize the effectiveness
of the interaction. Longer-term results are presented in Figure 7.

This INR for 2D images can be extended to represent 3D models. Figure 4 shows
synthesized views of 3D implicit representations using the same
$C^{*}$-algebra MLPs trained on three 3D chairs from the ShapeNet dataset
[Chang et al., 2015]. The presented poses are unseen during training. Again,
the noncommutative INR reconstructs the chair models with less noisy
artifacts, indicating that interaction helps efficient learning. See Sections
6.1 and 6.2 for details and results.

Figure 2: Average PSNR of implicit representations of the image below (top)
and reconstructions of the ground truth image at every 100 iterations
(bottom). The noncommutative $C^{*}$-algebra net learns the geometry and
colors of the image faster than its commutative counterpart. Figure 3: Ground
truth images and their implicit representations of commutative and
noncommutative $C^{*}$-algebra nets after 500 iterations of training. The
noncommutative $C^{*}$-algebra net reproduces colors more faithfully. Figure
4: Synthesized views of 3D implicit representations of commutative and
noncommutative $C^{*}$-algebra nets after 5000 iterations of training. The
noncommutative $C^{*}$-algebra net can produce finer details. Note that the
commutative $C^{*}$-algebra net could not synthesize the chair on the left.

###  4.2 Group $C^{*}$-algebra nets

As another experimental example of $C^{*}$-algebra nets, we showcase group
$C^{*}$-algebra nets, which we introduced in Section 3.1.4. The group
$C^{*}$-algebra nets take functions on a symmetric group as input and return
functions on the group as output.

Refer to Section 3.1.4 for notations. A group $C^{*}$-algebra net is trained
on data $\\{(x,y)\in\mathcal{A}^{N_{0}}\times\mathcal{A}^{N_{H}}\\}$, where
$x$ and $y$ are $N_{0}$\- and $N_{H}$-dimensional vector-valued functions.
Practically, such functions may be represented as real tensors, e.g.,
$x\in\mathbb{C}^{N_{0}\times\\#G}$, where $\\#G$ is the size of $G$. Using
product between functions explained in Section 3.1.4 and element-wise
addition, a linear layer, and consequently, an MLP, on $\mathcal{A}$ can be
constructed. Following the $C^{*}$-algebra nets over matrices, we use leaky
ReLU for activations.

One of the simplest tasks for the group $C^{*}$-algebra nets is to learn
permutation-invariant representations, e.g., predicting the sum of given $d$
digits. In this case, $x$ is a function that outputs permutations of features
of $d$ digits, and $y(g)$ is an identity function that returns their sum for
all $g\in G$. In this experiment, we use features of MNIST digits of a pre-
trained CNN in 32-dimensional vectors. Digits are selected so that their sum
is less than 10 to simplify the problem, and the model is trained to classify
the sum of given digits using cross-entropy loss. We set the number of layers
to 4 and the hidden dimension to 32. For comparison, we prepare a permutation-
invariant DeepSet model [Zaheer et al., 2017], which uses sum pooling for
permutation invariance, containing the same number of parameters as floating-
point numbers with the group $C^{*}$net.

Table 3 displays the results of the task with various training dataset sizes
when $d=3$. What stands out in the table is that the group $C^{*}$-algebra net
consistently outperforms the DeepSet model by large margins, especially when
the number of training data is limited. Additionally, as can be found in
Figure 5, the group $C^{*}$-algebra net converges much faster than the DeepSet
model. These results suggest that the inductive biases implanted by the
product structure in the group $C^{*}$-algebra net are effective.

Dataset size | DeepSet | Group $C^{*}$-algebra net
---|---|---
1k | $0.413\pm 0.031$ | $0.777\pm 0.009$
5k | $0.807\pm 0.031$ | $0.921\pm 0.002$
10k | $0.878\pm 0.009$ | $0.944\pm 0.005$
50k | $0.904\pm 0.007$ | $0.971\pm 0.001$
Table 3: Average test accuracy of a DeepSet model and a group $C^{*}$-algebra
net on test data of the sum-of-digits task after 100 epochs of training. The
group $C^{*}$-algebra net can learn from fewer data. Figure 5: Average test
accuracy curves of a DeepSet model and a group $C^{*}$-algebra net trained on
10k data of the sum-of-digits task. The group $C^{*}$-algebra net can learn
more efficiently and effectively.

##  5 Conclusion and Discussion

In this paper, we have generalized the space of neural network parameters to
noncommutative $C^{*}$-algebras. Their rich product structures bring powerful
properties to neural networks. For example, a $C^{*}$-algebra net over
nondiagonal matrices enables its sub-models to interact, and a group
$C^{*}$-algebra net learns permutation-equivariant features. We have
empirically demonstrated the validity of these properties in various tasks,
image classification, neural implicit representation, and sum-of-digits tasks.

Although Section 4 experimentally showed that noncommutative $C^{*}$-algebra
nets outperformed the baselines, practical consideration of noncommutative
$C^{*}$-algebra nets may arise in their computational complexities. In
particular, the $C^{*}$-algebra net over matrices used in the experiments
requires $O(d^{2})$ space complexity for the number of sub-models $d$, which
limits the possible number of sub-models and their interactions. This
complexity could be alleviated by, for example, parameter sharing or
introducing structures to nondiagonal elements by an analogy between self-
attentions and their efficient variants. The design of such structures may be
data-dependent, and we leave it for future research.

Another important and interesting research direction is an application of
infinite-dimensional $C^{*}$-algebras. In this paper, we focused mainly on
finite-dimensional $C^{*}$-algebras. We showed that the product structure in
$C^{*}$-algebras is a powerful tool for neural networks, for example, learning
with interactions and group equivariance (or invariance) even for the finite-
dimensional case. Infinite-dimensional $C^{*}$-algebra allows us to analyze
functional data. Practical applications of our framework to functional data
with infinite-dimensional $C^{*}$-algebras are also our future work.

Our framework with noncommutative $C^{*}$-algebras has a wide range of
applications, and we believe that our framework opens up a new approach to
learning neural networks.

## Acknowledgement

We would like to thank Dr. Tomohiro Hayase for a helpful and constructive
discussion about the application of $C^{*}$-algebra to neural networks.

##  6 Supplemental Material

###  6.1 Implementation details

We implemented $C^{*}$-algebra nets using JAX [Bradbury et al., 2018] with
equinox [Kidger and Garcia, 2021] and optax [Babuschkin et al., 2020].
Throughout the experiments, we used the Adam optimizer [Kingma and Ba, ] with
a learning rate of $1.0\times 10^{-4}$, except for the 3D NIR experiment,
where Adam’s initial learning rate was set to $1.0\times 10^{-3}$. We set the
batch size to 32, except for the 2D NIR, where each batch consisted of all
pixels, and 3D NIR, where a batch size of 4 was used.

The implementation of 3D neural implicit representation (Section 4.1.2) is
based on a simple NeRF-like model and its renderer in Tancik et al. [2021].
For training, 25 views of each 3D chair from the ShapeNet dataset [Chang et
al., 2015] are adopted with their $64\times 64$ pixel reference images. The
same $C^{*}$-algebra MLPs with the 2D experiments were used, except for the
hyperparameters: the number of layers of four and the hidden dimensional size
of 128.

The permutation-invariant DeepSet model used in Section 4.2 processes each
data sample with a four-layer MLP with hyperbolic tangent activation, sum-
pooling, and a linear classifier. Although we tried leaky ReLU activation as
the group $C^{*}$-algebra net, this setting yielded sub-optimal results. The
hidden dimension of the MLP was set to 84 to match the number of floating-
point-number parameters equal to that of the group $C^{*}$-algebra net.

###  6.2 Additional results

Figures 6 and 7 present the additional figures of 2D INRs (Section 4.1.2).
Figure 6 is an ukiyo-e counterpart of Figure 3 in the main text. Again, the
noncommutative $C^{*}$-algebra net learns color details faster than the
commutative one. Figure 7 shows average PSNR curves over three NIRs of the
image initialized with different random states for 5,000 iterations. Although
it is not as effective as the beginning stage, the noncommutative
$C^{*}$-algebra net still outperforms the commutative one after the
convergence.

Figure 6: Ground truth images and their implicit representations of
commutative and noncommutative $C^{*}$-algebra nets after 500 iterations of
training. Figure 7: Average PSNR over implicit representations of the image of
commutative and noncommutative $C^{*}$-algebra nets trained on five cat
pictures (top) and reconstructions of the ground truth image at every 500
iterations (bottom).

Tables 4 and 8 show the additional results of 3D INRs (Section 4.1.2). Table 4
presents the average PSNR of synthesized views. As can be noticed from the
synthesized views in Figures 4 and 8, the noncommutative $C^{*}$-algebra net
produces less noisy output, resulting in a higher PSNR.

Figure 8: Synthesized views of implicit representations of a chair. Commutative $C^{*}$-algebra net | Noncommutative $C^{*}$-algebra net
---|---
$18.40\pm 4.30$ | $25.22\pm 1.45$
Table 4: Average PSNR over synthesized views. The specified poses of the views
are unseen during training.

## References

  * Amin et al. [2008] M. F. Amin, M. M. Islam, and K. Murase.  Single-layered complex-valued neural networks and their ensembles for real-valued classification problems.  In _IJCNN_ , 2008.
  * Arena et al. [1997] P. Arena, L. Fortuna, G. Muscato, and M. G. Xibilia.  Multilayer perceptrons to approximate quaternion valued functions.  _Neural Networks_ , 10(2):335–342, 1997.
  * Babuschkin et al. [2020] I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, T. Hennigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, T. Norman, J. Quan, G. Papamakarios, R. Ring, F. Ruiz, A. Sanchez, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, L. Wang, W. Stokowiec, and F. Viola.  The DeepMind JAX Ecosystem, 2020.  URL http://github.com/deepmind.
  * Bradbury et al. [2018] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brandstetter et al. [2022] J. Brandstetter, R. v. d. Berg, M. Welling, and J. K. Gupta.  Clifford neural layers for pde modeling.  _arXiv_ , 2022.
  * Buchholz [2005] S. Buchholz.  _A Theory of Neural Computation with Clifford Algebras_.  PhD thesis, 2005.
  * Buchholz and Sommer [2008] S. Buchholz and G. Sommer.  On clifford neurons and clifford multi-layer perceptrons.  _Neural Networks_ , 21(7):925–935, 2008.
  * Candès [1999] E. J. Candès.  Harmonic analysis of neural networks.  _Applied and Computational Harmonic Analysis_ , 6(2):197–218, 1999.
  * Chang et al. [2015] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical report, Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * Choi et al. [2020] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In _CVPR_ , 2020.
  * Clanuwat et al. [2018] T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha.  Deep learning for classical japanese literature.  _arXiv_ , 2018.
  * Cohen et al. [2019] T. S. Cohen, M. Geiger, and M. Weiler.  A general theory of equivariant CNNs on homogeneous spaces.  In _NeurIPS_ , 2019.
  * Davis [1979] P. J. Davis.  _Circulant Matrices_.  Wiley, 1979.
  * Dong et al. [2020] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma.  A survey on ensemble learning.  _Frontiers of Computer Science_ , 14:241–258, 2020.
  * Ganaie et al. [2022] M. Ganaie, M. Hu, A. Malik, M. Tanveer, and P. Suganthan.  Ensemble deep learning: A review.  _Engineering Applications of Artificial Intelligence_ , 115:105151, 2022.
  * Gaudet and Maida [2018] C. J. Gaudet and A. S. Maida.  Deep quaternion networks.  In _IJCNN_ , pages 1–8. IEEE, 2018.
  * Hashimoto et al. [2021] Y. Hashimoto, I. Ishikawa, M. Ikeda, F. Komura, T. Katsura, and Y. Kawahara.  Reproducing kernel Hilbert $C^{*}$-module and kernel mean embeddings.  _Journal of Machine Learning Research_ , 22(267):1–56, 2021.
  * Hashimoto et al. [2022] Y. Hashimoto, Z. Wang, and T. Matsui.  $C^{*}$-algebra net: a new approach generalizing neural network parameters to $C^{*}$-algebra.  In _ICML_ , 2022.
  * Hashimoto et al. [2023] Y. Hashimoto, M. Ikeda, and H. Kadri.  Learning in RKHM: a $C^{*}$-algebraic twist for kernel machines.  In _AISTATS_ , 2023.
  * Hirose [1992] A. Hirose.  Continuous complex-valued back-propagation learning.  _Electronics Letters_ , 28:1854–1855, 1992.
  * Kidger and Garcia [2021] P. Kidger and C. Garcia.  Equinox: neural networks in JAX via callable PyTrees and filtered transformations.  _Differentiable Programming workshop at Neural Information Processing Systems 2021_ , 2021.
  * [22] D. P. Kingma and J. L. Ba.  Adam: a Method for Stochastic Optimization.  In _ICLR_.
  * Kirillov [1976] A. A. Kirillov.  _Elements of the Theory of Representations_.  Springer, 1976.
  * Le Cun et al. [1998] Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee et al. [2022] C. Lee, H. Hasegawa, and S. Gao.  Complex-valued neural networks: A comprehensive survey.  _IEEE/CAA Journal of Automatica Sinica_ , 9(8):1406–1426, 2022.
  * Lenssen et al. [2018] J. E. Lenssen, M. Fey, and P. Libuschewski.  Group equivariant capsule networks.  In _NeurIPS_ , 2018.
  * Murphy [1990] G. J. Murphy.  _$C^{*}$ -Algebras and Operator Theory_.  Academic Press, 1990.
  * Nishikawa et al. [2005] I. Nishikawa, K. Sakakibara, T. Iritani, and Y. Kuroe.  2 types of complex-valued hopfield networks and the application to a traffic signal control.  In _IJCNN_ , 2005.
  * Nitta [1995] T. Nitta.  A quaternary version of the back-propagation algorithm.  In _ICNN_ , volume 5, pages 2753–2756, 1995.
  * Pearson and Bisset [1994] J. Pearson and D. Bisset.  Neural networks in the clifford domain.  In _ICNN_ , 1994.
  * Rivera-Rovelo et al. [2010] J. Rivera-Rovelo, E. Bayro-Corrochano, and R. Dillmann.  Geometric neural computing for 2d contour and 3d surface reconstruction.  In _Geometric Algebra Computing_ , pages 191–209. Springer, 2010\.
  * Rossi and Conan-Guez [2005] F. Rossi and B. Conan-Guez.  Functional multi-layer perceptron: a non-linear tool for functional data analysis.  _Neural Networks_ , 18(1):45–60, 2005.
  * Sannai et al. [2021] A. Sannai, M. Imaizumi, and M. Kawano.  Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces.  In _UAI_ , 2021.
  * Sitzmann et al. [2020] V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein.  Implicit neural representations with periodic activation functions.  In _NeurIPS_ , 2020.
  * Sonoda and Murata [2017] S. Sonoda and N. Murata.  Neural network with unbounded activation functions is universal approximator.  _Applied and Computational Harmonic Analysis_ , 43(2):233–268, 2017.
  * Sonoda et al. [2022] S. Sonoda, I. Ishikawa, and M. Ikeda.  Universality of group convolutional neural networks based on ridgelet analysis on groups.  In _NeurIPS_ , 2022.
  * Tancik et al. [2021] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P. Srinivasan, J. T. Barron, and R. Ng.  Learned Initializations for Optimizing Coordinate-Based Neural Representations.  In _CVPR_ , 2021.
  * Thind et al. [2022] B. Thind, K. Multani, and J. Cao.  Deep learning with functional inputs.  _Journal of Computational and Graphical Statistics_ , 0(0):1–10, 2022.
  * Trabelsi et al. [2018] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. J. Pal.  Deep complex networks.  In _ICLR_ , 2018.
  * Xiao et al. [2017] H. Xiao, K. Rasul, and R. Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv_ , 2017.
  * Xie et al. [2022] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar.  Neural fields in visual computing and beyond.  _Computer Graphics Forum_ , 2022.
  * Yadav et al. [2005] A. Yadav, D. Mishra, S. Ray, R. Yadav, and P. Kalra.  Representation of complex-valued neural networks: a real-valued approach.  In _ICISIP_ , 2005.
  * Zaheer et al. [2017] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola.  Deep sets.  In _NeurIPS_ , 2017.
  * Zang et al. [2022] D. Zang, X. Chen, J. Lei, Z. Wang, J. Zhang, J. Cheng, and K. Tang.  A multi-channel geometric algebra residual network for traffic data prediction.  _IET Intelligent Transport Systems_ , 16(11):1549–1560, 2022.
  * Zhang and Yang [2022] Y. Zhang and Q. Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12):5586–5609, 2022.
  * Zhu et al. [2018] X. Zhu, Y. Xu, H. Xu, and C. Chen.  Quaternion convolutional neural networks.  In _ECCV_ , 2018.