# In search of projectively equivariant networks

Georg Bökman, Fredrik Kahl
Chalmers University
Gothenburg, Sweden
bokman, fredrik.kahl (at) chalmers.se
&Axel Flinth
Umeå University
Umeå, Sweden
axel.flinth (at) umu.se

###### Abstract

Equivariance of linear neural network layers is well studied. In this work, we
relax the equivariance condition to only be true in a projective sense. We
propose a way to construct a projectively equivariant neural network through
building a standard equivariant network where the linear group representations
acting on each intermediate feature space are “multiplicatively modified
lifts” of projective group representations. By theoretically studying the
relation of projectively and linearly equivariant linear layers, we show that
our approach is the most general possible when building a network out of
linear layers. The theory is showcased in two simple experiments.

##  1 Introduction

Deep neural networks [21] have been successfully applied across a large number
of areas, including but not limited to computer vision [19], natural language
processing [8], game play [27] and biology [16]. In many of these areas the
data has geometric properties or contains symmetries that can be exploited
when designing neural networks. For instance, AlphaFold [16] models proteins
in $3\mathrm{D}$-space while respecting translational and rotational
symmetries. Much of the work on neural networks respecting geometry and
symmetry of the data can be boiled down to formulating the symmetries in terms
of _group equivariance_. Group equivariance of neural networks is a currently
very active area of research starting with [31] and brought in to the deep
networks era by [7]. Recent surveys include [4, 12].

Given a set of transformations $\mathcal{T}$ acting on two sets $X$, $Y$, a
function $f:X\to Y$ is called equivariant if applying $f$ commutes with the
transformations $t\in\mathcal{T}$, i.e., $t[f(x)]=f(t[x])$. Concretely,
consider rotations acting on point clouds and $f$ mapping point clouds to
point clouds. If applying $f$ and then rotating the output yields the same
thing as first rotating the input and then applying $f$, we call $f$ rotation
equivariant. In the most general scenario, $t\in\mathcal{T}$ could act
differently on $X$ and $Y$. If the set of transformations $\mathcal{T}$ form a
group, results from abstract algebra can be used to design equivariant
networks. A typical way of formulating an equivariance condition in deep
learning is to consider the neural network as a mapping from a vector space
$V$ to another vector space $W$ and requiring application of the network to
commute with group actions on $V$ and $W$. As $V,W$ are vector spaces the
setting is nicely framed in terms of representation theory (we will provide a
brief introduction in Section 2). What happens however when $V$ and/or $W$ are
not vector spaces? This work covers the case when $V$ and $W$ are projective
spaces, i.e., vector spaces modulo multiplication by scalars. This may seem
obscure and practically irrelevant, but already the utility of homogenous
coordinates in computer vision suggests the contrary. In Section 2, we will
describe more examples.

In this paper, we propose a projectively equivariant architecture. The main
idea is to build features which transform equivariantly, in the standard way,
up to multiplication by scalar valued group homomorphisms $\varepsilon$. This
approach may seem too restrictive, since the projective equivariance condition
at first glance looks quite weak. We can however prove (Theorem 2.14) that in
terms of restricting to certain linear layers, our approach is the most
general thing one can do.

Using well-known results from group theory, we subsequently describe the
structure of the resulting networks both in general and for several important
special cases. It turns out that for a few important special cases, including
the rotation group $\operatorname{SO}(3)$, the projectively equivariant linear
layers are exactly the same as the ordinarily equivariant ones. The
equivalence is slightly surprising, since projective equivariance is a weaker
condition and hence should allow more expressive architectures.

Our theoretical considerations are accompanied by some empirical proof-of-
concept experiments on modified MNIST data, and spinor valued data in Section
3.

##  2 Projective equivariance

111We have for convenience collected important notation used in this article
in Table 1 in the appendix. Appendix A also contains many more standard
mathematical definitions that did not fit into the main text.

In this work, we are concerned with networks that are _projectively
equivariant_. Towards giving a formal definition, let us first discuss how
equivariance can be formulated using the notion of a _representation_ of a
group.

A _representation_ of a group $G$ on a vector space $V$ over a field
$\mathbb{F}$ is a group homomorphism
$\widehat{\rho}:G\to\operatorname{GL}(V)$. That is, a map $\widehat{\rho}$
associating each group element $g$ with an isomorphism $\widehat{\rho}(g)$ in
a manner that respects the structure of the group, i.e.,

$\displaystyle\widehat{\rho}(g)\widehat{\rho}(h)=\widehat{\rho}(gh)\text{ for
all }g,h\in G.$ (1)

We also assume that $G$ is equipped with a topology, with respect to which
that the representations are continuous. As we will also consider _projective
representations_ in this paper, we will refer to ‘ordinary’ representations
(1) as _linear representations_. Given linear representations
$\widehat{\rho}_{0}$ and $\widehat{\rho}_{1}$ on spaces $V$ and $W$,
respectively, we say that a map $\Phi:V\to W$ is _equivariant_ if

$\displaystyle\widehat{\rho}_{1}(g)\Phi(v)=\Phi(\widehat{\rho}_{0}(g)v)\text{
for all }g\in G,v\in V.$ (2)

If in (2), $\widehat{\rho}_{1}(g)$ is the identity transformation for every
$g\in G$, we say that $\widehat{\rho}_{1}$ acts trivially and then $\Phi$ is
called _invariant_. Many symmetry conditions can be phrased in terms of (2),
and ways to design neural networks $\Phi$ satisfying (2) have been extensively
studied for e.g. rigid motions in $2\mathrm{D}$/$3\mathrm{D}$/$n\mathrm{D}$
[7, 29, 2, 28, 5], permutations of graph nodes [23, 32, 25] and for more
general groups $G$ [31, 17, 6, 1, 9]. The most famous example is probably the
translation equivariance of CNNs [10, 20].

In this work, we want to study networks that are _projectively in- and
equivariant_. To explain this notion, let us first introduce the notation
$\operatorname{P}(V)$ for the projective space associated to a vector space
$V$ over a field $\mathbb{F}$. That is, $\operatorname{P}(V)$ is the space of
equivalence classes of $V\setminus\\{0\\}$ under the equivalence relation
$v\sim w\Longleftrightarrow\exists\lambda\in\mathbb{F}\setminus\\{0\\}\text{
s.t. }v=\lambda w$. Furthermore, we will write $\Pi_{V}$ for the projection
that maps $v\in V$ to its equivalence class
$\Pi_{V}(v)\in\operatorname{P}(V)$.

######  Example 2.1.

The _pinhole camera model_ is well-known and popular in computer vision [15].
In essence, the idea is to identify a point $y\in\mathbb{R}^{2}$ in an image
with the line in $\mathbb{R}^{3}$ that of points $x$ that $y$ can be an image
of. Mathematically, this amounts to embedding $y\in\mathbb{R}^{2}$ to the
$3\mathrm{D}$-point $x=[y,1]\in\mathbb{R}^{3}$, and then considering it as a
point in $\operatorname{P}(\mathbb{R}^{3})$. A $2\mathrm{D}$-point-cloud can
in the same way be identified with a set of points in homogeneous coordinates,
say $x_{i}=[y_{i},1]\in\mathbb{R}^{3},i\in[m]$. In this manner, we can think
of point-clouds $X$ as elements of $V=(\mathbb{R}^{3})^{m}$. Of course,
multiplying all points in $X$ with a scalar $\lambda$ does not change the
image of the points, we can regard $X$ as only being defined modulo
$\mathbb{R}$, i.e., as an element of $\operatorname{P}(V)$.

######  Example 2.2.

In quantum mechanics, the state of a particle is determined via a
$\mathbb{C}$-valued _wave-function_ , that is an element $\psi$ of some
Hilbert space $\mathcal{H}$, with $\langle\psi,\psi\rangle=1$ [13]. Any
measurable property of particle is however determined through its _probability
density function_ $|\psi|^{2}$, which is not changed when $\psi$ is multiplied
with a normalized scalar $\theta\in\mathbb{C}$. Physical observations can
hence only reveal $\psi$ modulo such a scalar, and it should therefore be
interpreted as an element of the projective space
$\operatorname{P}(\mathcal{H})$.

The benefit of projective equivariance is imminent when trying to infer a
projective feature transforming in a certain manner from projective data –
think for example of inferring the center of mass of from a photo of a point
cloud (which is equivariant to rigid motions). Projective invariance can also
be applied in a slightly different and unexpected way, that we outline next.

######  Example 2.3.

Consider the problem of detecting ’T’-shapes in natural images. We may phrase
this as learning a function $p_{\text{T}}:\mathbb{R}^{n,n}\to[0,1]$ giving the
probability that the image $v\in\mathbb{R}^{n,n}$ contains a ’T’. This
probability is not changed by translating the image – the function is hence
linearly invariant to the standard action of $\mathbb{Z}_{n}^{2}$ on
$\mathbb{R}^{n,n}$. The same is true for the action of horizontally flipping
the image. The same is not true when flipping the image vertically – an image
containing a ’T’ will instead contain a ’$\perp$’ symbol. In other words,
$p_{\text{T}}(v)\approx 1\Leftrightarrow p_{\text{T}}(v_{\text{ver.
flip}})\approx 0$. The action of the two flips, which commute, can be modelled
with the help of _Klein’s Vierergroup_ 222’Vier’ is German for ’four’,
referring to the number of elements in the group. $\mathbb{Z}_{2}^{2}$ –
$(1,0)$ corresponds to the vertical flip, and $(0,1)$ to the horizontal one.

Figure 1: The group action of Example 2.3.

Let us write $p_{\text{T}}=p_{0}e^{q_{\text{T}}}$, where $p_{0}$ is some small
number. If $v$ is an image neither containing a ’T’ or a ’$\perp$’,
$p_{\text{T}}(v)$ should be small. $q_{T}(v)=0$ suffices in this case. If $v$
instead is an image containing a ’T’, $q_{\text{T}}(v)$ should be large, and
$q_{T}(v_{\text{ver. flip}})$ should be very small. This is satisfied by
$q(v_{\text{ver. flip}})=-q(v)$. Hence, $q_{\text{T}}$ should modulo a
multiplication with $-1$ not change by any flip – that is, $q_{\text{T}}$
should be projectively invariant. Now imagine that the dataset also contains
images of ’H”s. Here, the probability $p_{\text{H}}$ – or equivalently
$q_{\text{H}}$ – is linearly, and therefore projectively, invariant of either
flip. See also Figure 1. Since $q_{\text{H}}$ and $q_{\text{T}}$ from a linear
point of view have different transformation behaviour, we cannot use the same
linearly projective architecture to model them both. Since they are both
projectively equivariant, a projectively equivariant model should.

######  Remark 2.4.

A projectively invariant network could for $v$ being an image of a ‘T’ satisfy
$q_{T}(v_{\text{ver. flip}})=\lambda q_{T}(v)$ for arbitrary $\lambda\neq 0$,
which notably any function that is non-zero for both $v_{\text{ver. flip}}$
and $v$ satisfies. Such a model would be too general for the example given.
However, as we will see later, in our suggested network architecture,
$\lambda$ can in fact only be $\pm 1$ here, meaning that a projectively
invariant network makes sense for the task.

Motivated by these examples, we are interested in maps that fulfil the
equivariance relation (2) up to a multiplicative scalar. To formalise this
notion, let us begin by introducing some notation. We will use the shorthand
$\operatorname{PGL}(V)=\operatorname{P}(\operatorname{GL}(V))$ which is called
the _projective general linear group_. Note that an element
$M\in\operatorname{PGL}(V)$ defines a projective-linear map
$\operatorname{P}(V)\to\operatorname{P}(V)$.

######  Definition 2.5.

A _projective representation_ of a group $G$ on a projective space
$\operatorname{P}(V)$ is a group homomorphism
$\rho:G\to\operatorname{PGL}(V)$.

Put more concretely, a projective representation is a map associating each
$g\in G$ to an equivalence class $\rho(g)$ of invertible linear maps. $\rho$
respects the group structure in the sense of (1) – however, only in the sense
of elements in $\operatorname{PGL}(V)$, i.e., up to a multiplicative scalar.

######  Example 2.6.

The simplest form of a projective representation is a projected linear one.
That is, given a linear representation
$\widehat{\rho}:G\to\operatorname{GL}(V)$, we immediately obtain a projective
representation through $\rho=\Pi_{\operatorname{GL}(V)}\circ\widehat{\rho}$.

Not all projective representations are projections of linear ones, as shown by
the next example.

######  Example 2.7.

[13] For every $\ell=0,1/2,1,3/2,\dots$, there is a complex vector space
$\mathcal{V}_{\ell}$ of dimension $2\ell+1$ on which the group
$\mathrm{SU}(2)$ is acting. Together with the fact that $\mathrm{SU}(2)$ is a
global double cover of $\mathrm{SO}(3)$, this can be used to define a map
$\rho$ from $\mathrm{SO}(3)$ to $\operatorname{GL}(\mathcal{V}_{\ell}^{n})$ up
to a scalar, i.e., a projective representation. For integer $\ell$, this is a
projection of a linear representation of $\mathrm{SO(3)}$, but for half-
integers $1/2,3/2,\dots$, it is not. This corresponds to integer and half-
integer spins in quantum mechanics.

Given projective representations $\rho_{0}$, $\rho_{1}$ on
$\operatorname{P}(V)$ and $\operatorname{P}(W)$, respectively, we can now
state equivariance of a map $\Phi:\operatorname{P}(V)\to\operatorname{P}(W)$
exactly as before in (2):

$\displaystyle\rho_{1}(g)\Phi(v)=\Phi(\rho_{0}(g)v)\text{ for all $g\in G$,
$v\in P(V)$}.$ (3)

Note that we only demand that the equality holds in $\operatorname{P}(W)$ and
_not_ necessarily in $W$ which was the case in (2). We refer to $\Phi$
satisfying (3) as being _projectively equivariant_.

###  2.1 A projectively equivariant architecture

A canonical way to construct (linearly) equivariant neural networks is to
alternate equivariant activation functions and equivariant linear layers. This
method easily generalizes to projective equivariance, motivating the following
question: Which linear maps $A:V\to W$ define projectively equivariant
transformations $\operatorname{P}(V)\to\operatorname{P}(W)$? If the projective
representations $\rho_{0}$ and $\rho_{1}$ are projections of linear
representations $\widehat{\rho}_{0}$ and $\widehat{\rho}_{1}$ respectively, we
immediately see that every equivariant $A$ defines a projectively equivariant
transformation. These are however not the only ones – an $A$ satisfying

$\displaystyle
A\widehat{\rho}_{0}(g)=\varepsilon(g)\widehat{\rho}_{1}(g)A\quad,\quad g\in G$
(4)

for some function $\varepsilon:G\to\mathbb{F}$ will also suffice. It is not
hard to see that for such a relation to hold for generic $A$, $\varepsilon$
needs to be a (continuous) _group homomorphism_. The set of these, equipped
with pointwise multiplication, form the so-called _character group_.

######  Definition 2.8.

Let $G$ be a topological group. The set of continuous group homomorphisms
$\varepsilon:G\to\mathbb{F}$ is called the _character group_ of $G$, and is
denoted $G^{*}$.

Given an $\varepsilon\in G^{*}$ and a linear representation $\widehat{\rho}$
of $G$, we may define a new representation $\widehat{\rho}^{\varepsilon}$
through $\widehat{\rho}^{\varepsilon}(g)=\varepsilon(g)\widehat{\rho}(g)$.
Notice that (4) then means nothing more than that $A$ is linearly equivariant,
when its image is endowed with $\widehat{\rho}^{\epsilon}$.

This immediately, at least when $\rho$ is a projection of a linear
representation $\widehat{\rho}$, suggests a way to build a projectively
equivariant net. We write it as a composition of maps
$\operatorname{P}(V_{0})\to\operatorname{P}(V_{1})\to\cdots\to\operatorname{P}(V_{K})$.
For each $\varepsilon\in G^{*}$, we determine the space $U^{\varepsilon}$ of
maps from $V_{0}$ to $V_{1}$ fulfilling (4). We then multiply the input
features $v\in V_{0}$ with maps $A_{0}^{\varepsilon}\in U^{\varepsilon}$, to
form new features. By subsequently applying an equivariant non-linearity
$\sigma_{0}:V_{1}\to V_{1}$ commuting with multiplication by elements
$\theta\in\mathbb{F}$, i.e., $\sigma_{0}(\widehat{\rho}_{1}(g)\theta
v)=\theta\widehat{\rho}_{1}(g)\sigma_{0}(v)$, $g\in G$, $\theta\in\mathbb{F}$,
we obtain the $V_{1}$-valued features

$\displaystyle v_{1}^{\varepsilon}=\sigma_{0}(A_{0}^{\varepsilon}v),\quad
A_{0}^{\varepsilon}\in W_{0}^{\varepsilon},\varepsilon\in G^{*}.$ (5)

Note that we here get one feature for each $\varepsilon\in G^{*}$, so we map
from $V_{0}$ to a $|G^{*}|$-tuple of $V_{1}$’s. If the $\varepsilon$ map into
the unit circle, we only need $\sigma_{0}$ to commute with elements in the
unit circle of $\mathbb{F}$ (we will see later that for finite-dimensional
$V_{i}$ and compact groups, this will suffice).

The features $v_{1}^{\varepsilon}$ can be combined with new linear maps
$A_{1}^{\gamma}$, $\gamma\in G^{*}$ to form new features as follows:

$\displaystyle
v_{2}^{\varepsilon}=\sigma_{1}\bigg{(}\sum_{\begin{subarray}{c}\gamma,\delta\in
G^{*},\gamma\delta=\varepsilon\end{subarray}}A_{1}^{\gamma}v_{1}^{\delta}\bigg{)}.$
(6)

This process can continue for $k=3,4,\dots,K-1$ until we arrive at
projectively equivariant $V_{K}$-valued features. We record that as a theorem,
which we formally prove in Appendix B.

######  Theorem 2.9.

The features $v_{k}^{\varepsilon}$ are projectively equivariant.

Figure 2: The structure of a three-layer projectively equivariant net for
$G=\mathbb{Z}_{3}$. $\mathbb{Z}_{3}^{*}$ consists of three elements of the
form $\epsilon(k)=\theta^{k}$, where $\theta=1,\alpha,\alpha^{2}$,
$\alpha=e^{2\pi i/3}$ – we therefore write
$\mathbb{Z}_{3}^{*}=\\{1,\alpha,\alpha^{2}\\}$. The features $v^{\delta}$ are
combined with linear operators in $U^{\gamma}$ to produce new features
$v^{\varepsilon}$, with $\gamma,\delta,\epsilon\in\mathbb{Z}_{3}^{*}$. In a
final step, the features of the $\varepsilon$-indexed tuples can be linearly
combined with the help of a selector $p$ to yield one single output. Best
viewed in color.

The above construction inherently produces $|G^{*}|$ output features
$v_{K}^{\varepsilon}\in V_{K}$. If we want a single output feature that is
projectively equivariant, we propose to learn a ’selection-vector’
$p\in\mathbb{F}^{|G^{*}|}$, that linearly decides which feature(s) to ’attend’
to:

$\displaystyle w=\sum_{\varepsilon\in
H^{*}}p_{\varepsilon}v^{\varepsilon}_{K}.$

Technically, this does not yield an exactly equivariant function unless all
but one entry in $p$ is equal to zero. Still, we believe it is a reasonable
heuristic. We provide a graphical depiction of our approach for the (simple
but non-trivial) case of $G=\mathbb{Z}_{3}$ in Figure 2.

Relation to earlier work. As described above, once we fix $\varepsilon$, (4)
is a linear equivariance condition on the linear map $A$. Hence the
construction proposed here is equivalent to the following linear equivariance
approach. For each feature space $V_{k}$, $k>0$, duplicate it $|G^{*}|$ times
to form a new feature space $\tilde{V}_{k}=\bigoplus_{\varepsilon\in
G^{*}}V_{k}$ and select the linear representation $\bigoplus_{\varepsilon\in
G^{*}}\widehat{\rho}^{\varepsilon}$ to act on $\tilde{V}_{k}$. Then
parametrize the neural network by linearly equivariant maps
$\tilde{V}_{k}\to\tilde{V}_{k+1}$. Depending on the group $G$ and the spaces
$\tilde{V}_{k}$ this can be done by various methods found in the literature
[7, 9, 5, 29].

This connection may make it seem like our proposed approach is too
restrictive. (4) a priori seems like a stronger condition than the inherently
projective relation $\rho_{1}(g)A=A\rho_{0}(g),g\in G$, where $\rho_{i}$ are
projective representations. However, it turns out that _the two conditions are
essentially equivalent_ , and that our linear layers are as general as
possible. To prove this is the purpose of the next section.

###  2.2 Projectively equivariant linear maps

To make our considerations general enough to include examples like Example
2.7, we will not assume that $\rho$ is a projection of a $\widehat{\rho}$, but
only that we can ‘lift’ $\rho$ to a linear representation of a so-called
covering group of $G$.

######  Definition 2.10.

Let $\rho:G\to\operatorname{PGL}(V)$ be a projective representation, and $H$ a
covering group of $G$, with group covering $\varphi$. A _lift_ of $\rho$ is a
linear representation $\widehat{\rho}:H\to\operatorname{GL}(V)$ with
$\rho\circ\varphi=\Pi_{\operatorname{GL}(V)}\circ\widehat{\rho}$.

######  Example 2.11.

If $\rho:G\to\operatorname{PGL}(V)$ is a projection of a linear representation
$\widehat{\rho}:G\to\operatorname{GL}(V)$ as in Example 2.6, then
$\widehat{\rho}$ is a lift of $\rho$ (the covering group is simply $G$
itself).

######  Example 2.12.

The linear representations related to spin discussed in Example 2.7 are
defined on $\operatorname{SU}(2)$, which is a covering group of
$\operatorname{SO}(3)$. The linear representations are lifts of the projective
representations of $\operatorname{SO}(3)$ discussed there.

Let us first reformulate the equivariance problem slightly as is routinely
done in the linear case [29, 9]. Given projective representations $\rho_{0}$
and $\rho_{1}$ on $\operatorname{P}(V)$ and $\operatorname{P}(W)$, we may
define a projective representation
$\rho:G\to\operatorname{PGL}(\operatorname{Hom}(V,W))$ on the space
$\operatorname{Hom}(V,W)$ of linear maps between $V$ and $W$, through
$\rho(g)A=\rho_{1}(g)A\rho_{0}^{-1}(g)$. Just as in the linear case, we can
reformulate projective equivariance of an $A\in\operatorname{Hom}(V,W)$ as an
_invariance_ equation under $\rho$.

######  Lemma 2.13.

A linear map $A:V\to W$ is projectively equivariant if and only if $A$ is
invariant under $\rho$, that is $\rho(g)A=A\text{ for all $g\in G$. }$ These
equations are understood in $\operatorname{P}(\operatorname{Hom}(V,W))$.

The proof can be found in the appendix. The key is that the equivariance
condition for $A$ means that any $v$ solves the ’eigenvalue problem’
$\rho_{1}(g)Av=\lambda_{v,g}A\rho_{0}(g)v$, from which $\rho(g)A=A$ in
$\operatorname{P}(V)$ can be deduced. The lemma implies that we in the
following can concentrate on invariance equations

$\displaystyle\rho(g)v=v\text{ for all }g\in G,$ ($\mathrm{Proj}_{G}$)

which, importantly, are understood in $\operatorname{P}(V)$. With this
knowledge, the main result of this section is relatively easy to show. It says
that given a lift $\widehat{\rho}$ of $\rho$, the solutions of projective
invariance equations ($\mathrm{Proj}_{G}$) of $\rho$ are exactly the ones of
the $\varepsilon$-linear invariance equations of $\widehat{\rho}$,

$\displaystyle\widehat{\rho}(h)v=\varepsilon(h)v\text{ for all $h\in H$},$
($\mathrm{Lin}_{H}^{\varepsilon}$)

where $\varepsilon$ ranges the whole of $H^{*}$. Note in particular that
${\eqref{eq:liftinv}}$ is nothing but the standard linear invariance problem
for $\widehat{\rho}$. For a given $\varepsilon\in H^{*}$ we denote the space
of solutions to ($\mathrm{Lin}_{H}^{\varepsilon}$) by $U^{\varepsilon}$.

######  Theorem 2.14.

Let $G$ be a group and $H$ a covering group of $G$ with covering map
$\varphi$. Further, let $\rho:G\to\operatorname{PGL}(V)$ be a projective
representation of $G$ and $\widehat{\rho}:H\to\operatorname{GL}(V)$ a lift of
$\rho$. Then, the following are equivalent

$(i)\text{ $v$ solves \eqref{eq:inv}.}\qquad(ii)\text{ $v$ is in the
equivalence class of some $x\in U^{\varepsilon}$, \ $\varepsilon\in H^{*}.$ }$

###### Proof.

To prove that $(ii)\Rightarrow(i)$, note that
$\Pi_{V}(Mv)=\Pi_{\operatorname{GL}(V)}(M)\Pi_{V}(v)$ for
$M\in\operatorname{GL}(V)$ and $v\in V$. Let $x$ solve
($\mathrm{Lin}_{H}^{\varepsilon}$). For any $g\in G$, there is a $h$ such that
$\varphi(h)=g$ and hence

$\rho(g)\Pi_{V}(x)=\rho(\varphi(h))\Pi_{V}(x)=\Pi_{\operatorname{GL}(V)}(\widehat{\rho}(h))\Pi_{V}(x)=\Pi_{V}(\widehat{\rho}(h)x)=\Pi_{V}(\varepsilon(x)x)=\Pi_{V}(x),$

meaning that the equivalence class of $x$ solves ($\mathrm{Proj}_{G}$).

To prove that $(i)\Rightarrow(ii)$, let $\Pi_{V}(x)$ be a solution of
($\mathrm{Proj}_{G}$) for some $x\neq 0$. $\widehat{\rho}(h)x$ must then for
all $h$ lie in the subspace spanned by $x$ – in other words, there must exist
a map $\lambda:H\to\mathbb{F}$ with $\widehat{\rho}(h)x=\lambda(h)x$. Since
$\lambda(h)\lambda(k)x=\widehat{\rho}(h)\widehat{\rho}(k)x=\widehat{\rho}(hk)x=\lambda(hk)x$,
$\lambda$ must be a group homomorphism. Also, since $\widehat{\rho}$ is
continuous, $\lambda$ must also be. That is, $\lambda\in H^{*}$, and $x\in
U^{\lambda}$. ∎

###  2.3 The structure of the spaces $U^{\varepsilon}$

Let us study the spaces $U^{\varepsilon}$. Before looking at important special
cases, let us begin by stating two properties that hold in general.

######  Proposition 2.15.

(i) If $V$ is finite dimensional and $H$ is compact, $U^{\varepsilon}$ is only
non-trivial if $\varepsilon$ maps into the unit circle.

(ii) The $U^{\varepsilon}$ are contained in the space $U_{HH}$ of solutions of
the linear invariance problem

$\displaystyle\widehat{\rho}(h)v=v\text{ for all $h\in\\{H,H\\}$},$
($\mathrm{Lin}_{\\{H,H\\}}^{1}$)

of the commutator subgroup $\\{H,H\\}$ generated by commutators
$\\{h,k\\}=hkh^{-1}k^{-1},h,k\in H$. If $\mathbb{F}=\mathbb{C}$, $V$ is finite
dimensional and $H$ is compact, $U_{HH}$ is even the direct sum of the
$U^{\varepsilon}$, $\varepsilon\in H^{*}$.

The proofs require slightly more involved mathematical machinery than above.
The idea for $(i)$ is that the $\widehat{\rho}$ in this case can be assumed
unitary. The “$\subseteq$” part of $(ii)$ is a simple calculation, whereas we
for the “direct sum” part need to utilize the simultaneous diagonalization
theorem for commuting families of unitary operators. The details are given in
Appendix B.4.

This proposition has two important consequences for networks with finite-
dimensional intermediate spaces and compact symmetry groups (which will be
true for all examples we consider in this paper). First, $(i)$ means that the
nonlinearity we use in our networks can be restricted to only commute with
elements $\theta$ in the unit circle, instead of all $\theta\in\mathbb{F}$.
This opens up for e.g. the use of the $\tanh$-nonlinearity for real-valued
networks. Second, $(ii)$ implies that $U^{\varepsilon}$ is nontrivial only for
finitely many $\varepsilon$, and further that the sum of their dimensions does
not exceed $\dim(U_{HH})$. Hence, each tuple
$(A_{k}^{\varepsilon})_{\varepsilon\in H^{*}}$ can be represented with no more
than $\dim(U_{HH})\leq\dim(\operatorname{Hom}(V_{k},V_{k+1}))$ scalars.

We move on to discuss the structure of the spaces $U^{\varepsilon}$ for three
relevant cases333A few well-known group theory results will be useful. For
convenience, we give proofs of them in B.7.

The rotation group. $\operatorname{SO}(3)$ is a _perfect_ group, meaning that
$\operatorname{SO}(3)$ is equal to its commutator subgroup
$\\{\operatorname{SO}(3),\operatorname{SO}(3)\\}$ generated by commutators
$\\{R,S\\}=RSR^{-1}S^{-1}$. This has the consequence that
$\operatorname{SO}(3)^{*}$ only contains the trivial character $1$, which in
turn shows that _for projections of linear representations_
$\widehat{\rho}:\operatorname{SO}(3)\to\operatorname{GL}(V)$, _(
$\mathrm{Proj}_{G}$) is equivalent to the linear equivariance problem
($\mathrm{Lin}_{H}^{\varepsilon}$)_. Consequently, the projective nets we have
proposed will in this case necessarily be the standard linearly equivariant
ones. This has the consequence that all projectively equivariant net related
to the pinhole camera model (Example 2.1) we design using our approach are
necessarily linearly equivariant.

The pinhole camera representation is a projection of a linear representation
of $\operatorname{SO}(3)$. Not all projective representations are, though –
they can also be linear projections of representations of
$\operatorname{SU}(2)$. Since $\operatorname{SU}(2)$ also is perfect, there is
again an equivalence, but only on the $\operatorname{SU}(2)$-level. Hence, in
general _the projective invariance problem for the
$\operatorname{SO}(3)$-representation is not necessarily equivalent to a
linear equivariance problem ($\mathrm{Lin}_{H}^{\varepsilon}$), but surely
always to one of the form ($\mathrm{Lin}_{H}^{\varepsilon}$)_.

The permutation group. The character group $S_{n}^{*}$ of the permutation
group contains two elements: $1$ and the sign function $\mathrm{sgn}$. This
means that the solutions to ${\eqref{eq:inv}}$ form two subspaces $U^{1}$ and
$U^{\mathrm{sgn}}$. However, in the important case of $\rho$ being a
projection of the standard representation $\widehat{\rho}$ of $S_{n}$ on the
tensor space $(\mathbb{F}^{n})^{\otimes k}$ , something interesting happens:
$U^{\mathrm{sgn}}$ is empty for $n\geq k+2$. We prove this in Section B.5.
This means that at least for large sets $(n\geq k+2)$, the projective and
linear invariance problems are again equivalent.

The translation group. $\mathbb{Z}_{n}^{*}$ is isomorphic to the set of $n$:th
unit roots: every unit root $\omega$ defines a character through
$\varepsilon_{\omega}(k)=\omega^{k},k\in\mathbb{Z}_{n}$. Furthermore, for the
standard linear representation on $\mathbb{F}^{n}$, each of the spaces
$U^{\varepsilon_{\omega}}$ is non trivial – it contains the element defined
through $v_{k}=\omega^{-k}$:
$(\rho(\ell)v)_{k}=v_{k-\ell}=\omega^{\ell-k}=\varepsilon_{\omega}(\ell)v_{k}$.
In the case $\mathbb{F}=\mathbb{C}$, this leaves us with $n$ linear spaces of
protectively invariant elements – which are not hard to identify as the
components of the Fourier transform of $v$. If $\mathbb{F}=\mathbb{R}$, the
number of unit roots depend on $n$ – if $n$ is odd, there are two roots
$\\{+1,-1\\}$, but if $n$ is even, the only root is $+1$. In the latter case,
we thus again have an equivalence between ($\mathrm{Proj}_{G}$) and
($\mathrm{Lin}_{H}^{\varepsilon}$).

##  3 Experiments

We perform two experiments to illustrate the theory. Code is provided at
https://github.com/usinedepain/projectively_equivariant_deep_nets

###  3.1 Spinor Field Networks

In this section we consider a regression task which is equivariant under a
projective representation of $\operatorname{SO}(3)$ corresponding to a linear
representation of $\operatorname{SU}(2)$ as was briefly described in Example
2.7. Our main result (Theorem 2.14) and discussion in Section 2.3 show that we
for projective $\operatorname{SO}(3)$ equivariance should build a net using
linearly $\operatorname{SU}(2)$ equivariant layers. We give an introduction to
the representation theory of $\operatorname{SU}(2)$ and the spinor spaces
$\mathcal{V}_{\ell}$, $\ell=0,1/2,1,3/2,\dots$ in Appendix C.

Figure 3:  Data used for the regression task with
$\operatorname{SO}(3)$-equivariance. We define four classes, each class being
a point cloud with three points in 3D. To each 3D point we attach a spinor
feature $s\in\mathbb{C}^{2}$. These spinor features are plotted inside the
frames next to each 3D point, the frames and points are color coded.
$\mathrm{Re}(s)$ is plotted in red and $\mathrm{Im}(s)$ in blue. For every
class a different spinor should be regressed, which is shown in the black
frame. Left: The four classes. They are defined such that if one looks only at
the spatial locations of the points, the first and third classes are
equivalent as are the second and fourth. If one instead looks only at the
spinor features, the first and second classes are equivalent as are the third
and fourth. A network that regresses the output spinor correctly for all
classes must take into account both the locations and the spinor features of
each point. Right: The regression should be equivariant under rotations of the
point clouds. When a point cloud is rotated by $R\in\operatorname{SO}(3)$, the
spinor features and labels are transformed by a corresponding matrix
$U(R)\in\operatorname{SU}(2)$. A random rotation of the leftmost point cloud
is shown in the center column. Given a rotation $R$, the corresponding $U(R)$
is however only defined up to sign. E.g., the identity rotation corresponds to
both the identity matrix $I\in\operatorname{SU}(2)$ and $-I$. The rightmost
column illustrates a correct regression even though all spinor input features
have been multiplied by $-I$, while the output spinor has not.

Data. The data in the experiment consists of point clouds equipped with spinor
features and regression targets that are spinors. That is, one data sample is
given by $\left(\\{(x_{i},s_{i})\\}_{i=1}^{n},t\right)$ where
$x_{i}\in\mathbb{R}^{3}$ are the 3D positions of the points,
$s_{i}\in\mathcal{V}_{1/2}\sim\mathbb{C}^{2}$ are the attached spinor features
and $t\in\mathcal{V}_{1/2}$ is the regression target. We define an action of
$\operatorname{SO}(3)$ on these samples by

$R\cdot\left(\\{(x_{i},s_{i})\\}_{i=1}^{n},t\right):=\left(\\{(Rx_{i},U(R)s_{i})\\}_{i=1}^{n},U(R)t\right).$
(7)

Here $U(R)$ is an element of $\operatorname{SU}(2)$ that corresponds to $R$.
We note that since $\operatorname{SU}(2)$ double covers
$\operatorname{SO}(3)$, $U(R)$ is only defined up to sign. Hence, $s_{i}$ and
$t$ can be thought of as $\mathbb{R}$-projective, and the action of
$\operatorname{SO}(3)$ on them as a projective representation.

The task is to regress the target in a way that is equivariant to (7) and
invariant to translations of the point clouds. The data is defined by four
prototype samples that are shown in Figure 3. To make the task more difficult
we also add Gaussian noise to the spatial coordinates $x_{i}$, varying the
noise level (i.e., std.) from $0$ to $2/5$. During training, a network sees
only the four prototypes + noise, while evaluation is done on rotated
prototypes + noise. According to Theorem 2.14, and the discussion regarding
the rotation group in Section 2.3, the projectively
$\operatorname{SO}(3)$-equivariant network architectures are in this case
exactly the standard linearly $\operatorname{SU}(2)$-equivariant networks. We
illustrate the data and how it transforms under $\operatorname{SO}(3)$ in
Figure 3.

Models. We will tackle the problem by building an equivariant net in the
spirit of Tensor Field Networks (TFNs) [28]. Let us first recall how TFNs
work. Indata are point clouds $\\{x_{i}\\}_{i=1}^{n}$ in $\mathbb{R}^{3}$ with
features $f_{i}$ in some vector space $V$ attached to each point, on which
$\operatorname{SO}(3)$ is acting through $\widehat{\rho}_{V}$. We map the
inputs to point clouds with features $f_{i}^{\prime}$ in some vector space $W$
(on which $\operatorname{SO}(3)$ acts through $\widehat{\rho}_{W}$) via
convolution with a filter function
$\Psi:\mathbb{R}^{3}\to\operatorname{Hom}(W,V)$ through

$\displaystyle f_{i}^{\prime}=\sum_{j\neq i}\Psi(x_{j}-x_{i})f_{j}.$ (8)

Provided the filter $\Psi$ satisfies the invariance condition
$\Psi(Rx)~{}=~{}\widehat{\rho}_{W}(R)\Psi(x)\widehat{\rho}_{V}(R)^{-1}$,
$R\in\operatorname{SO}(3)$, this defines a layer invariant to translations and
rotations of $\mathbb{R}^{3}$, and the $\operatorname{SO}(3)$-action on $V$
and $W$.

The idea of a TFN is to define $\Psi$ using a tensor product,
$\Psi(x)v=\psi(x)\otimes v$, $v\in V$. Here, $\psi(x)$ lives in some third
vector space $U$, and satisfies the condition $\psi(Rx)=\rho_{U}(R)\psi(x)$,
$R\in\operatorname{SO}(3)$. $\Psi$ maps equivariantly into the space
$W=U\otimes V$, which can be decomposed into spaces isomorphic to the irreps
$\mathcal{V}_{0},\mathcal{V}_{1}{},\mathcal{V}_{2},\dots$ of the action of
$\operatorname{SO}(3)$. In this manner, we may transform features in different
irreps into new ones living in other irreps. For instance, if we combine
features and filters living in the space $\mathcal{V}_{1}\sim\mathbb{R}^{3}$
(i.e., vectors), we end up with one scalar component $\mathcal{V}_{0}$, one
vector component in $\mathcal{V}_{1}$ and a traceless symmetric $3\times
3$-matrix component in $\mathcal{V}_{2}$. The higher order component
$\mathcal{V}_{2}$ can be discarded to keep the dimensions under control.
Another type of layer are self-interaction layers where features of the same
irrep at each point are linearly combined using learnt weight matrices.

The idea of our approach is now to let $U$ and $V$ be spaces on which
$\operatorname{SU}(2)$ is acting, i.e., also consider features and vectors of
half-integer spin. We limit ourselves to $\ell=0,1/2$ and $1$, i.e., scalars,
spinors and vectors, and call the resulting nets Spinor Field Networks. The
hyperparameter choices for a Spinor Field Network in our implementation are
the number of scalar, spinor and vector features output by each layer. We use
sigmoid-gated nonlinearities [30] for non-scalar features and GeLU for scalar
features.

Figure 4:  Results of the spinor regression experiments for varying
architectures and spatial noise levels. We plot the mean over 30 runs, showing
the standard deviation with errorbars.

We consider in total five different models. (i) _Spinors as scalars._ A non-
equivariant baseline TFN that treats input and output spinors as four real
scalars. It has the correct equivariance for the 3D structure of the locations
but not for the spinors. (ii) _Spinors as features._ An
$\operatorname{SU}(2)$-equivariant net as described above, with intermediate
scalar,vector and spinor features. (iii) _Spinors squared as vector features._
Prior to the first layer, we tensor square the spinor features $s_{i}\mapsto
s_{i}\otimes s_{i}$ and extract a vector feature at each point, which we then
process $\operatorname{SO}(3)$-equivariantly. This allows the network to only
use real valued (scalar and vector) features in the intermediate layers, which
is likely an advantage in most modern machine learning frameworks (including
PyTorch which we use). In the last layer, the scalar and vector features are
tensored with the input spinors to produce an output spinor.

The last network types have filters that consist of not only
$\mathcal{V}_{\ell}$ valued functions, but also the input spinors themselves.
That is, we define layers $f_{i}^{\prime}=\sum_{j\neq
i}(\psi(x_{j}-x_{i})\oplus s_{j})\otimes f_{j}$. This could be generalized to
tensor spherical harmonics valued filters which we explain in C.1. We consider
two such types: (vi) _Spinors as filters_ works as just described. (v)
_Spinors squared as vector filters_ analogously defines vector valued filters
by tensor squaring the input spinors as in the _Spinors squared as vector
features_ case. This again allows for a mostly real valued network.

Each network consists of three layers and the hyperparameters are chosen to
make the number of parameters approximately equal between all networks. In all
networks the input feature at a location $x_{i}$ consists of the average of
the vectors $x_{i}-x_{j}$ for $j\neq i$ and for all except the two last
networks also the spinor feature $s_{i}$ (interpreted as either 4 scalars, a
spinor or a vector after squaring). The final output is the mean over all
locations of the final features, meaning that the last layer maps to a single
spinor feature – except in the non-equivariant variant in which the last layer
maps to 4 scalar features. We describe the training setup and further details
in Appendix C.2.

Results. All networks are trained on just the four point clouds shown in the
left part of Figure 3 (plus added spatial noise), except one version of the
non-equivariant net which is trained on randomly rotated versions of these
four point clouds as data augmentation. The evaluation is always on randomly
rotated versions of the data. We present results in Figure 4. Notably, the
networks that tensor square the spinors before the first layer to then work on
real-valued data until the last layer perform the best. At low noise levels,
the networks that work directly on complex-valued spinors also perform well,
but their performance degrades quickly with added noise. The network without
correct equivariance unsurprisingly performs quite poorly with, and extremely
poorly without, data augmentation. The networks with filters defined using the
input spinors outperform the networks where the inputs are only fed in once at
the start of the network. This is likely due to the fact that inserting the
inputs in each layer makes the learning easier for the network.

###  3.2 ViererNet

We consider a toy example alluding to Example 2.3, i.e., an image
classification problem which is linearly invariant to translations, and
projectively invariant to flips along the horisontal and vertical axes, i.e.,
the Vierergroup $\mathbb{Z}_{2}^{2}$. Its character group also contains four
elements – the values in $(1,0)$ and $(0,1)$ can both be either $+1$ or $-1$.

Data. We modify the MNIST dataset [20], by adding an additional class, which
we will refer to as ’NaN’. When an image $v\in\mathbb{R}^{n,n}$ is loaded, we
randomly (with equal probability) either use the image as is, flip it
horizontally or flip it vertically. When flipped, the labels are changed, but
differently depending on the label: if it is either $0$, $1$ or $2$, it stays
the same. If it $3$, $4$ or $5$, we change the label to NaN if it flipped
horizontally, but else not. If it is $6$ or $7$, we instead change the label
if it is flipped vertically, but else not. The labels of $8$ and $9$ are
changed regardless which flip is applied.

Models. Since we operate on images, we build a linearly translation-
equivariant architecture by choosing layers in the space of $3\times
3$-convolutional layers. We create ViererNet – a four-layer projectively
equivariant network according to the procedure described in Section 2.1 – and
a four-layer baseline CNN with comparable number of parameters. Details about
the models can be found in Appendix D.

Figure 5: Training(left) and test (right) accuracy for the MNIST experiments.
The lines report the median performance for each epoch, wheras the errorbars
depict confidence intervals of 80$\%$.

Results. We train 30 models of each type. The evolution of the test and
training accuracy is depicted in Figure 5, along with confidence intervals
containing $80\%$ of the runs. We clearly see that the ViererNet is both
better at fitting the data, and is more stable and requires shorter training
to generalize to the test-data. To give a quantitative comparison, we for each
model determine which epoch gives the best median performance on the test
data. The ViererNet then achieves a median accuracy of $92.8\%$, whereas the
baseline only achives $90.2\%$. We subsequently use those epochs to test the
hypothesis that the ViererNet outperforms the baseline. Indeed, the
performance difference is significant ($p<0.025$).

In Appendix D, similar modifications of the CIFAR10 dataset are considered.
This dataset is less suited for this experiment (since one of the flips often
leads to a plausible CIFAR10-image), and the results are less clear: The
baseline outperforms the ViererNet slightly, but not significantly so. To some
extent, the trend that the ViererNet requires less training to generalize
persists.

##  4 Conclusion

In this paper, we proposed a neural network model for capturing projective
equivariance. A theoretical study of the relation between projective and
linear equivariance for neural network linear layers revealed that our
proposed approach is the most general one. We in particular found examples
(including $\operatorname{SO}(3)$, or rather $\operatorname{SU}(2)$) for which
the two problems are, somewhat surprisingly, equivalent. Finally, we presented
two simple experiments to illustrate our theory.

### Acknowledgement

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
The computations were/was enabled by resources provided by the National
Academic Infrastructure for Supercomputing in Sweden (NAISS) at Chalmers
Centre for Computational Science and Engineering (C3SE) partially funded by
the Swedish Research Council through grant agreement no. 2022-06725.

## References

  * Aronsson [2021] Jimmy Aronsson.  Homogeneous vector bundles and G-equivariant convolutional neural networks.  _arXiv:2105.05400 [cs, math, stat]_ , May 2021.  URL http://arxiv.org/abs/2105.05400.  arXiv: 2105.05400.
  * Bekkers et al. [2018] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits.  Roto-Translation Covariant Convolutional Networks for Medical Image Analysis.  _arXiv:1804.03393 [cs, math]_ , June 2018.  URL http://arxiv.org/abs/1804.03393.  arXiv: 1804.03393.
  * Biedenharn and Louck [1984] L. C. Biedenharn and James D. Louck.  _Angular Momentum in Quantum Physics: Theory and Application_.  Encyclopedia of Mathematics and Its Applications. Cambridge University Press, Cambridge, 1984.  ISBN 978-0-521-30228-9.  doi: 10.1017/CBO9780511759888.
  * Bronstein et al. [2021] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković.  Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.  _arXiv:2104.13478 [cs, stat]_ , May 2021.  URL http://arxiv.org/abs/2104.13478.  arXiv: 2104.13478.
  * Cesa et al. [2021] Gabriele Cesa, Leon Lang, and Maurice Weiler.  A Program to Build E(N)-Equivariant Steerable CNNs.  _ICLR_ , September 2021.  URL https://openreview.net/forum?id=WE4qe9xlnQw.
  * Cohen et al. [2020] Taco Cohen, Mario Geiger, and Maurice Weiler.  A General Theory of Equivariant CNNs on Homogeneous Spaces.  _arXiv:1811.02017 [cs, stat]_ , January 2020.  URL http://arxiv.org/abs/1811.02017.  arXiv: 1811.02017.
  * Cohen and Welling [2016] Taco S. Cohen and Max Welling.  Group Equivariant Convolutional Networks.  _arXiv:1602.07576 [cs, stat]_ , June 2016.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Finzi et al. [2021] Marc Finzi, Max Welling, and Andrew Gordon Wilson.  A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 3318–3328. PMLR, 2021\.  URL https://proceedings.mlr.press/v139/finzi21a.html.
  * Fukushima and Miyake [1982] Kunihiko Fukushima and Sei Miyake.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and cooperation in neural nets_ , pages 267–285. Springer, 1982.
  * [11] Mario Geiger and Tess Smidt.  E3nn: Euclidean Neural Networks.  URL http://arxiv.org/abs/2207.09453.
  * Gerken et al. [2021] Jan E. Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson.  Geometric Deep Learning and Equivariant Neural Networks.  _arXiv:2105.13926 [hep-th]_ , May 2021.  URL http://arxiv.org/abs/2105.13926.  arXiv: 2105.13926.
  * Hall [2013] Brian C. Hall.  _Quantum Theory for Mathematicians_.  Graduate Texts in Mathematics. Springer, 2013.
  * Hall [2015] Brian C. Hall.  _Lie Groups, Lie Algebras, and Representations: An Elementary Introduction_ , pages 49–76.  Springer International Publishing, Cham, 2015.  ISBN 978-3-319-13467-3.  URL https://arxiv.org/pdf/math-ph/0005032.pdf.
  * Hartley and Zisserman [2003] Richard Hartley and Andrew Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, Cambridge, UK ; New York, 2nd ed edition, 2003.  ISBN 978-0-521-54051-3.
  * Jumper et al. [2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.  Highly accurate protein structure prediction with AlphaFold.  _Nature_ , July 2021.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/s41586-021-03819-2.  URL http://www.nature.com/articles/s41586-021-03819-2.
  * Kondor and Trivedi [2018] Risi Kondor and Shubhendu Trivedi.  On the generalization of equivariance and convolution in neural networks to the action of compact groups.  In _International Conference on Machine Learning_ , pages 2747–2755. PMLR, 2018.
  * [18] Risi Kondor, Zhen Lin, and Shubhendu Trivedi.  Clebsch-Gordan Nets: A Fully Fourier Space Spherical Convolutional Neural Network.  URL http://arxiv.org/abs/1806.09231.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Lee [2018] Gregory T. Lee.  _Abstract Algebra - a first course_.  Springer Undergraduate Mathematics Series. Springer, 2018.
  * Maron et al. [2019] Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman.  Invariant and equivariant graph networks.  In _International Conference on Learning Representations_ , 2019.
  * Ore [1951] Oystein Ore.  Some remarks on commutators.  _Proceedings of the American Mathematical Society_ , 2(2):307–314, 1951.
  * Qi et al. [2017] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  _arXiv:1612.00593 [cs]_ , April 2017.  URL http://arxiv.org/abs/1612.00593.  arXiv: 1612.00593.
  * Rudin [1962] Walter Rudin.  _Fourier Analysis on Groups_.  Interscience publishers, 1962.
  * Silver et al. [2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Thomas et al. [2018] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds.  _arXiv:1802.08219 [cs]_ , May 2018.  URL http://arxiv.org/abs/1802.08219.  arXiv: 1802.08219.
  * Weiler and Cesa [2019] Maurice Weiler and Gabriele Cesa.  General e(2)-equivariant steerable cnns.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/45d6637b718d0f24a237069fe41b0db4-Paper.pdf.
  * [30] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen.  3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data.  URL http://arxiv.org/abs/1807.02547.
  * Wood and Shawe-Taylor [1996] Jeffrey Wood and John Shawe-Taylor.  Representation theory and invariant neural networks.  _Discrete Applied Mathematics_ , 69(1-2):33–60, August 1996.  ISSN 0166218X.  doi: 10/c3qmr6.  URL https://linkinghub.elsevier.com/retrieve/pii/0166218X95000753.
  * Zaheer et al. [2018] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola.  Deep Sets.  _arXiv:1703.06114 [cs, stat]_ , April 2018.  URL http://arxiv.org/abs/1703.06114.  arXiv: 1703.06114.
  * Zee [2016] A. Zee.  _Group Theory in a Nutshell for Physicists_.  In a Nutshell. Princeton University Press, 2016.  ISBN 978-0-691-16269-0.

$\operatorname{P}(V)$ |  Projective space of $V$, i.e., $V/(\mathbb{F}\setminus\\{0\\})$ | $\Pi_{V}$ |  Projection map from $V$ to $\operatorname{P}(V)$
---|---|---|---
$\operatorname{Hom}(V,W)$ |  Space of linear maps from $V$ to $W$ | $\operatorname{GL}(V)$ |  General linear group of $V$
$\operatorname{PGL}(V)$ |  Projective general linear group of $V$ | $\varphi$ | Group covering map
$\rho$ | Projective representation | $\widehat{\rho}$ | Linear representation
$[n]$ | $\\{0,1,\dots,n-1\\}$ | $\mathbb{Z}_{n}$ |  Additive group of integers modulo $n$
$S_{n}$ |  Permutation group of $n$ elements | $A_{n}$ |  Permutations of signature $1$
$\operatorname{SO}(3)$ |  Rotation group in $3\mathrm{D}$ | $\operatorname{SU}(2)$ |  Unitary matrices in $\mathbb{C}^{2,2}$
$\mathrm{id}$ | The identity matrix | $\mathrm{sgn}$ |  The signum function on $S_{n}$.
Table 1: Symbol glossary

##  Appendix A Technical definitions

Table 1 contains a symbol glossary for various symbols used in this work. The
remainder of this section contains definitions that did not have room in the
main text.

######  Definition A.1 (Group homomorphism).

Given two groups $G,H$, a group homomorphism from $G$ to $H$ is a map $a:G\to
H$ that respects the group structure, i.e., such that
$a(g_{1}g_{2})=a(g_{1})a(g_{2})$ for all $g_{1},g_{2}$ in $G$.

######  Definition A.2 (General linear group).

The general linear group $\operatorname{GL}(V)$ of a vector space $V$ is the
group of all invertible linear maps from $V$ to itself.

######  Definition A.3 (Topological group).

A topological group is a set $G$ that is simultaneously a topological space
and a group such that

  1. 1.

The group operation is continuous w.r.t. the topology.

  2. 2.

The inversion map $g\mapsto g^{-1}$ is continuous w.r.t. the topology.

######  Definition A.4 (Homeomorphism).

A homeomorphism is an invertible, continuous map between two topological
spaces whose inverse is also continuous. A function maps a set
homeomorphically if the restriction of the function to that set is a
homeomorphism.

######  Definition A.5 (Tensor product of finite dimensional vector spaces).

Given two vector spaces $U$ with basis $\\{u_{i}\\}_{i=1}^{n}$ and $V$ with
basis $\\{v_{i}\\}_{i=1}^{m}$, their tensor product $U\otimes V$ is a vector
space with one basis vector for each pair $(u_{i},v_{j})$ of one basis vector
from $U$ and one from $V$. The basis vectors of $U\otimes V$ are typically
denoted $\\{u_{i}\otimes v_{j}\\}_{i=1,j=1}^{n,m}$.

######  Definition A.6 (Tensor product of vectors).

The tensor product of an element $u=\sum_{i}\alpha_{i}u_{i}$ in $U$ and an
element $v=\sum_{j}\beta_{j}v_{j}$ in $V$ is the element $u\otimes
v=\sum_{i,j}\alpha_{i}\beta_{j}u_{i}\otimes v_{j}$ of $U\otimes V$. This
defines a bilinear map $(u,v)\mapsto u\otimes v$.

######  Definition A.7 (Tensor power).

The tensor power $U^{\otimes k}$ is simply the tensor product of $U$ with
itself $k$ times.

######  Definition A.8 (Covering group).

[14, Def. 5.36] Let $G$ be a topological group. A group $H$ is called a
_covering_ of $G$ if there exists a _group covering_ $\varphi:H\to G$, i.e., a
surjective continuous group homomorphism which maps some neighbourhood of the
unit element $e_{H}\in H$ to a neighbourhood of the unit element $e_{G}\in G$
homeomorphically.

######  Definition A.9 (Irreducability).

A linear representation $\widehat{\rho}:G\to\operatorname{GL}(V)$ is called
_irreducible_ if there is no subspace $U\notin\\{\\{0\\},V\\}$ which is
invariant under all $\widehat{\rho}(g)$, $g\in G$.

##  Appendix B Omitted proofs

In this section, we collect some further proofs we left out in the main text.

###  B.1 Induced projected representations

We begin, for completeness, by remarking and proving a small statement that we
use implicitly throughout the entire article.

######  Proposition B.1.

Let $G$ be a group, $V$ and $W$ vector spaces, $\widehat{\rho}_{V}$ and
$\widehat{\rho}_{W}$ linear representations of $G$ on $V$ and $W$,
respectively, and $\rho_{V}$ and $\rho_{W}$ their corresponding projected
representations. Then, the induced projective representation $\rho$ on
$\mathrm{Hom}(V,W)$

$\displaystyle\rho(g)A=\rho_{W}(g)\circ A\circ\rho_{V}(g)^{-1}$

is the projection of the corresponding induced linear projection

$\displaystyle\widehat{\rho}(g)A=\widehat{\rho}_{W}(g)\circ
A\circ\widehat{\rho}_{V}(g)^{-1}.$

###### Proof.

What we need to prove is that if $M$ is equivalent to $\widehat{\rho}_{W}(g)$
and $N$ is equivalent to $\widehat{\rho}_{V}(g)$, the linear map

$\displaystyle
R_{MN}:\operatorname{Hom}(V,W)\to\operatorname{Hom}(V,W),A\mapsto MAN^{-1}$

is equivalent to $\widehat{\rho}(g)$. However, the stated equivalences mean
that $M=\lambda\widehat{\rho}_{W}(g)$ and $N=\mu\widehat{\rho}_{V}(g)$ for
some non-zero $\mu,\lambda$. This in turn shows that

$\displaystyle
R_{MN}(A)=\lambda\widehat{\rho}_{W}(g)A(\mu\widehat{\rho}_{V}(g))^{-1}=\lambda\mu^{-1}\widehat{\rho}(g)A,$

i.e., that $R_{MN}$ is equivalent to $\widehat{\rho}(g)$. The claim has been
proven. ∎

###  B.2 Theorem 2.9

###### Proof of Theorem 2.9.

We will prove that

$v_{k}^{\varepsilon}(\widehat{\rho}_{0}(g))=\varepsilon(g)\widehat{\rho}_{k}(g)v_{k}^{\varepsilon}(v)$

for $g\in G$, $v\in V$ for all $k$. This in particular means
$v_{k}^{\varepsilon}(\widehat{\rho}_{0}(g))=\widehat{\rho}_{k}(g)v_{k}^{\varepsilon}(v)$
as elements in $\operatorname{P}(V_{k})$, which is to be proven.

We proceed with induction starting with $k=1$. We have for $g\in G$ and
$\varepsilon\in G^{*}$ arbitrary

$\displaystyle v_{1}^{\varepsilon}(\widehat{\rho}_{0}(g)v)$
$\displaystyle=\sigma_{0}(\widehat{\rho}_{1}(g)\widehat{\rho}_{1}(g)^{-1}A_{0}^{\varepsilon}\widehat{\rho}_{0}(g)v)\stackrel{{\scriptstyle\eqref{eq:epsilonequivariance}}}{{=}}\widehat{\rho}_{1}(h)\sigma_{0}(\varepsilon(g)A_{0}^{\varepsilon}v)=\varepsilon(g)\widehat{\rho}_{1}(g)v_{1}^{\varepsilon}(v),$

where we in the final step used our assumption on $\sigma_{0}$.

To prove the induction step $k\to k+1$, let us first concentrate on each
expression $A_{k}^{\gamma}v_{k}^{\delta}$ in (6). Due to the induction
assumption, we have
$v_{k}^{\delta}(\widehat{\rho}_{0}(g)v)=\delta(g)\widehat{\rho}_{k}(g)v_{1}^{\delta}(g)$
for $g\in G$, $\delta\in G^{*}$. Consequently

$\displaystyle A_{k}^{\gamma}v_{k}^{\delta}(\widehat{\rho}_{0}(g)v)$
$\displaystyle=\delta(g)A_{k}^{\gamma}\widehat{\rho}_{k}(g)v_{1}^{\delta}(v)=\delta^{1}(t)\widehat{\rho}_{k+1}(g)\widehat{\rho}_{k+1}(g)^{-1}A_{k}^{\gamma}\widehat{\rho}_{k}(h)v_{1}^{\delta}(v)$
$\displaystyle\stackrel{{\scriptstyle\eqref{eq:epsilonequivariance}}}{{=}}\delta(g)\gamma(g)\widehat{\rho}_{k+1}(g)A_{k}^{\gamma}v_{1}^{\delta}(v)$

Summing over all $\gamma,\delta\in G^{*}$ with $\gamma\cdot\delta=\varepsilon$
yields a feature that transforms the claimed way. This is not changed through
an application of the equivariant $\sigma_{k}$. The claim has been proven. ∎

###  B.3 Lemma 2.13

###### Proof of Lemma 2.13.

Equivariance of the map $A$ means that for all $g\in G$ and
$v\in\operatorname{P}(V)$, we have $\rho_{1}(g)Av=A\rho_{0}(g)v$. Since all
$\rho_{0}(g)$ are invertible, this is equivalent to

$\displaystyle\rho_{1}(g)A\rho_{0}(g)^{-1}v=Av\text{ for all $v\in V$.}$

Now, the above equality is not an equality of elements in $W$, but rather of
elements in $\operatorname{P}(W)$. That is, it says that all vectors $v\in V$
are solutions of the generalized eigenvalue problem
$\rho_{1}(g)A\rho_{0}(g)^{-1}v=\lambda Av$. The aim is to show that this
implies that $\rho_{1}(g)A\rho_{0}(g)^{-1}=\lambda A$ for some
$\lambda\in\mathbb{F}$. To show this, we proceed in two steps.

Claim Let $M\in\operatorname{GL}(V)$. If every vector $v\in V$ is an
eigenvector of $M$, $M$ is a multiple of the identity.

Proof Suppose not. Then, since all vectors are eigenvectors, there exists
$v\neq w\neq 0$ and $\lambda\neq\mu$ with $Mv=\lambda v$, $Mw=\mu v$. Now,
again since all vectors are eigenvectors of $M$, there must exist a third
scalar $\sigma$ with $M(v+w)=\sigma(v+w)$. Now,

$\displaystyle\sigma(v+w)=M(v+w)=Mv+Mw=\lambda v+\mu w\ \Longleftrightarrow\
(\sigma-\mu)w=(\mu-\sigma)v.$

Now, the final equation can only be true if $\lambda-\mu=\mu-\sigma=0$, i.e.,
$\lambda=\sigma=\mu$, which is a contradiction.

Claim Let $E,F\in\operatorname{Hom}(V,W)$. If every $v\in V$ is a solution of
the generalized eigenvalue problem $Ev=\lambda Fv$, $E$ is a multiple of $F$.

Proof First, $E$ restricted to $\ker F$ must be the zero map, since for all
$v\in\ker F$, $Ev=\lambda Fv=\lambda\cdot 0=0$. Secondly, $F^{\circ}:\ker
F^{\perp}\to\operatorname{ran}F$ is an isomorphism, so that the generalized
eigenvalue problem of $Ev=\lambda Fv$ on $\ker F^{\perp}$ is equivalent to the
eigenvalue problem of
$E(F^{\circ})^{-1}:\operatorname{ran}F\to\operatorname{ran}F$. By the previous
claim, $E(F^{\circ})^{-1}=\lambda\operatorname{id}$ for some $\lambda\in F$,
i.e., $E=\lambda F^{\circ}$ on $\ker F^{\perp}$. Since both $E$ and $F$
additionally are equal to the zero map on $\ker F$, the relation $E=\lambda F$
remains true also there. The claim has been proven. ∎

###  B.4 Proposition 2.15

###### Proof of Proposition 2.15.

Ad $(i):$ If $V$ is finite-dimensional and $H$ is compact, it is well known
that we (by modifying the inner product on $V$) may assume that
$\widehat{\rho}$ is unitary – for convenience, a proof is given in Section
B.7. This means that $|v|=|\widehat{\rho}(g)v|$ for all $h\in H$. If now $v\in
U^{\varepsilon}$ is non trivial, we have
$0\neq|v|=|\widehat{\rho}(h)v|=|\varepsilon(h)v|$, i.e., $|\varepsilon(h)|=1$
for all $h\in H$.

Ad $(ii)$: Let $\varepsilon\in H^{*}$, $x\in U^{\varepsilon}$ $k,h\in H$ be
arbitrary. Then

$\displaystyle\widehat{\rho}(\\{k,h\\})x=\widehat{\rho}(k)\widehat{\rho}(h)\widehat{\rho}(k)^{-1}\widehat{\rho}(h)^{-1}x=\varepsilon(k)\varepsilon(h)\varepsilon(k)^{-1}\varepsilon(h)^{-1}x=x.$

Since $\widehat{\rho}$ is a group isomorphism, this relation extends to
$\widehat{\rho}(\ell)x=x$ for all $\ell\in\\{H,H\\}$. This proves the first
part.

To prove the second claim, we will show that the operators
${\widehat{\rho}(h),\ h\in H}$ restricted to $U_{HH}$ are commuting operators
$U_{HH}\to U_{HH}$. Since we are in the case $\mathbb{F}=\mathbb{C}$, and we
WLOG can assume that they are unitary, this means that they are simultaneously
diagonalizable, i.e.,

$\displaystyle\widehat{\rho}(h)x_{i}=\lambda_{i}(h)x_{i},$

for some basis $x_{i}$ of $U_{HH}$ and ’eigenvalue functions’
$\lambda_{i}:H\to\mathbb{C}$. With the exact same argument as in the proof of
the main theorem, we show that $\lambda_{i}\in H^{*}$. Hence, the $x_{i}$ are
elements of subspaces $U^{\varepsilon}$, which shows that
$U_{HH}\subseteq\mathrm{span}(U^{\varepsilon},\varepsilon\in H^{*})$. Since
the other inclusion holds in general by what we just proved, the claim
follows.

So let us prove that the $\widehat{\rho}(h)$ are commuting operators
$U_{HH}\to U_{HH}$. Both of this follows from the following equality:

$\displaystyle\widehat{\rho}(k)\widehat{\rho}(h)=\widehat{\rho}(kh)=\widehat{\rho}(khk^{-1}h^{-1}hk)=\widehat{\rho}(h)\widehat{\rho}(k)\widehat{\rho}(\\{k^{-1},h^{-1}\\})$

If now $x\in{H,H}$, and $k\in\\{H,H\\}$ and $h\in H$ are arbitrary, it follows

$\displaystyle\widehat{\rho}(k)\widehat{\rho}(h)x=\widehat{\rho}(h)\widehat{\rho}(k)\widehat{\rho}(\\{k^{-1},h^{-1}\\})x\stackrel{{\scriptstyle
x\in U_{HH}}}{{=}}\widehat{\rho}(h)x$

which means that $\widehat{\rho}(h)x\in U_{HH}$. To prove that the maps
commute on $U_{HH}$ is even easier: for $x\in U_{HH}$ and $k,h\in H$
arbitrary, we immediately deduce

$\displaystyle\widehat{\rho}(k)\widehat{\rho}(h)x=\widehat{\rho}(h)\widehat{\rho}(k)\widehat{\rho}(\\{k^{-1},h^{-1}\\})x=\widehat{\rho}(h)\widehat{\rho}(k)x.$

The proof is finished. ∎

###  B.5 $U^{\mathrm{sgn}}$ for the standard action on
$(\mathbb{F}^{n})^{\otimes k}$

######  Proposition B.2.

Let $\widehat{\rho}$ be the standard representation of $S_{n}$ on
$(\mathbb{F}^{n})^{\otimes k}$. For $n\geq k+2$, $U^{\varepsilon}$ is
nontrivial only for $\varepsilon=1$. The bound is tight.

###### Proof.

Let $T\in(\mathbb{F}^{n})^{\otimes k}$ be an element of $U^{\mathrm{sgn}}$,
and let $I=(i_{0},\dots,i_{k-1})$ be an arbitrary $k$-index. Since $n\geq
k+2$, there are at least two elements
$(i,j)\notin\\{(i_{0},\dots,i_{k-1})\\}$. Consider the transposition $\tau$ of
$i$ and $j$. Then, $\tau(i_{\ell})=i_{\ell}$, $\ell\in[k]$, and
$\mathrm{sgn}(\tau)=-1$. Consequently,

$\displaystyle
T_{I}=T_{\tau(I)}=(\widehat{\rho}(\tau)T)_{I}=\mathrm{sgn}(\tau)T_{I}=-T_{I},$

since $T$ was in $U^{\mathrm{sgn}}$. Consequently, $T_{I}=0$. Since $I$ was
arbitrary, $T$ must be trivial, and first part of the claim follows.

To show that the bound is tight, it is enough to construct a non-zero element
in $(\mathbb{F}^{k+1})^{\otimes k}\cap U^{\mathrm{sgn}}$. Let
$I_{0}=(0,1,2,\dots,k)\in[k+1]^{k}$. For each multi-index $I$, there are
either multiple indices in $I$, or there exists a unique permutation
$\pi_{I}\in S_{n+1}$ so that $I=\pi_{I}^{-1}(I_{0})$. Due to the uniqueness,
it is not hard to show that $\pi_{\sigma^{-1}(I)}=\pi_{I}\circ\sigma$,
$\sigma\in S_{n}$. Let us now define a tensor $S$ through

$\displaystyle S_{I}=\begin{cases}0&\text{ if $I$ contains multiple indices
}\\\ \mathrm{sgn}(\pi_{I})&\text{ else.}\end{cases}$

We claim that $S$ is the sought element. To show this, let $I\in[k+1]^{k}$ and
$\sigma\in S_{n}$ be arbitrary. We have

$\displaystyle(\widehat{\rho}(\sigma)S)_{I}=S_{\sigma^{-1}(I)}$

Now, if $I$ contains multiple indices, $\sigma^{-1}(I)$ must also, so
$S_{\sigma^{-1}(I)}=0$, and $(\widehat{\rho}(\sigma)S)_{I}=0$ for such
indices. If $I$ does not contain multiple indices, we deduce

$\displaystyle
S_{\sigma^{-1}(I)}=\mathrm{sgn}(\pi_{\sigma^{-1}(I)})=\mathrm{sgn}(\pi_{I}\circ\sigma)=\mathrm{sgn}(\pi_{I})\mathrm{sgn}(\sigma)=\mathrm{sgn}(\sigma)S_{I}.$

Since $I$ was arbitrary, we have shown that
$\widehat{\rho}(\sigma)S=\mathrm{sgn}(\sigma)S$, which was the claim. ∎

######  Remark B.3.

In an earlier version of the manuscript, we proved the above proposition in a
different, arguably more clumsy, manner. Since it provides a nice connection
to the results in [23], we include a sketch of that argument also here.

First, we realise that $U^{\mathrm{sgn}}$ being trivial is the same as to say
that ($\mathrm{Proj}_{G}$) is equivalent to the linear invariance problem
($\mathrm{Lin}_{H}^{\varepsilon}$). The solutions of the latter are surely
included in the former (by e.g. Theorem 2.14) However, Proposition 2.15(ii)
also tells us that the solutions of ($\mathrm{Proj}_{G}$) are included in the
solution space of ($\mathrm{Lin}_{\\{H,H\\}}^{1}$). The commutator
$\\{S_{n},S_{n}\\}$ is the alternating group $A_{n}$, i.e., the group of
permutations of sign $1$ (see Section B.7). Now, the linear invariance problem
($\mathrm{Lin}_{H}^{\varepsilon}$) was treated in [23] – it was shown that for
$n\geq k+2$, the set of solutions of ($\mathrm{Lin}_{H}^{\varepsilon}$) is
exactly equal to the one of ($\mathrm{Lin}_{H}^{\varepsilon}$) (in fact, the
proof of that builds on the same idea as our proof of Proposition B.2). Hence,
we conclude

$\displaystyle\text{Sol. of }{\eqref{eq:liftinv}}\subseteq\text{Sol. of
}{\eqref{eq:inv}}\subseteq\text{Sol. of }{\eqref{eq:redinv}}=\text{Sol. of
}{\eqref{eq:liftinv}}=\text{Sol. of }{\eqref{eq:liftinv}}.$

###  B.6 The Vierergroup

Let us begin by describing the character group of $\mathbb{Z}_{2}^{2}$.
$\mathbb{Z}_{2}$ is an abelian group generated by two elements, $(1,0)$ and
$(0,1)$. Any group homomorphism $\varepsilon$ is hence uniquely determined by
its values $\varepsilon((1,0))$ and $\varepsilon((0,1))$. Since
$(1,0)+(1,0)=0=(0,1)+(0,1)$, the values must be square roots of one, i.e.,
either $+1$ or $-1$. Hence, $\mathbb{Z}_{n}^{*}$ contains four elements
$\epsilon_{{\scriptsize++}}$, $\epsilon_{{\scriptsize+-}}$,
$\epsilon_{{\scriptsize-+}}$, $\varepsilon_{--}$, defined through the
following table:

The appearance of the spaces $U^{\varepsilon}$ will depend on
$\widehat{\rho}$. They are however simple to compute – given a vector $v\in V$
and an $\varepsilon\in(\mathbb{Z}_{2}^{2})^{*}$, the element

$\displaystyle
v_{\varepsilon}=\sum_{h\in\mathbb{Z}_{2}^{2}}\varepsilon(h)^{-1}\widehat{\rho}(h)v$

is in $U^{\varepsilon}$444This strategy is generally applicable. It will be
quite inefficient if the group has many elements, though.

$\displaystyle\widehat{\rho(k)}v_{\varepsilon}=\sum_{h\in\mathbb{Z}_{2}^{2}}\varepsilon(h)^{-1}\widehat{\rho}(kh)v=\lceil
kh=\ell\rceil=\sum_{\ell\in\mathbb{Z}_{2}^{2}}\varepsilon(k^{-1}\ell)^{-1}\widehat{\rho}(\ell)v=\varepsilon(k)v_{\varepsilon}.$

Consequently, given any basis of $V$, we may construct a spanning set of
$U^{\varepsilon}$ via calculating the above vector for each basis vector, and
subsequently extract a basis. Applying this strategy to the example of the
Vierergroup acting through flipping on the spaces of $3\times 3$ convolutional
filters yields the bases depicted in Figure 6.

Figure 6: The bases for the spaces $U^{\varepsilon}$ for the action of
$\mathbb{Z}_{2}^{2}$ on the space $C$ of $3\times 3$ filters.

###  B.7 Group theoretic facts

Here we, out of convenience for the reader, present proofs of some well-known
group theoretical facts.

######  Lemma B.4.

Let $H$ be a compact group and $\widehat{\rho}$ a representation of $H$ on a
finite-dimensional space $V$. Then there exist an scalar product on $V$ with
respect to which all $\widehat{\rho}(h)$, $h\in H$, are unitary.

###### Proof.

Since $H$ is compact, there exists a unique normalized Haar measure $\mu$ on
$H$. That is, $\mu$ is a measure with the property that $\mu(h^{-1}A)=\mu(A)$
for all $h\in H$ and Borel measureable $A\subseteq H$. Given any scalar
product $\langle\,\cdot\,,\,\cdot\,\rangle$ on $V$, we may now define

$\displaystyle\langle
v,w\rangle_{H}=\int_{H}\langle\widehat{\rho}(h)v,\widehat{\rho}(h)w\rangle\mathrm{d}\mu(h).$

This is a well-defined expression due to $V$ being finite-dimensional and $H$
compact (and hence,
$h\mapsto\langle\widehat{\rho}(h)v,\widehat{\rho}(h)w\rangle$ is a continuous,
bounded function). It is easy to show that it is sesqui-linear. To show that
it is definite, note that if $0=\langle v,v\rangle_{H}$,
$\langle\widehat{\rho}(h)v,\widehat{\rho}(h)v\rangle$ is a continuous function
in $h$ zero almost everywhere, i.e., zero. This means that
$\langle\widehat{\rho}(h)v,\widehat{\rho}(h)v\rangle=0$ for all $h$, which
immediately implies that $v$ is zero.

We now claim that any $\widehat{\rho}(k)$ is unitary with respect to this
inner product. We have

$\displaystyle\langle\widehat{\rho}(k)v,\widehat{\rho}(k)w\rangle_{H}$
$\displaystyle=\int_{H}\langle\widehat{\rho}(h)\widehat{\rho}(k)v,\widehat{\rho}(h)\widehat{\rho}(k)w\rangle\mathrm{d}\mu(h)=\int_{H}\langle\widehat{\rho}(hk)v,\widehat{\rho}(hk)w\rangle\mathrm{d}\mu(h)$
$\displaystyle=\lceil
h^{\prime}=hk\rceil\int_{H}\langle\widehat{\rho}(h^{\prime})v,\widehat{\rho}(h^{\prime})w\rangle\mathrm{d}\mu(h^{\prime})=\langle
v,w\rangle_{H},$

where we used the invariance of the Haar measure in the substitution step. ∎

######  Lemma B.5.

$S_{n}^{*}=\\{1,\mathrm{sgn}\\}$.

###### Proof.

Let $\varepsilon\in S_{n}^{*}$ be arbitrary. Since $S_{n}$ is generated by
transpositions $\tau=(ij)$, $\varepsilon$ is uniquely determined by its values
of them. Since $\tau^{2}=1$, we must have $\varepsilon(\tau)^{2}=1$, meaning
that each value $\varepsilon(\tau)$ is either $+1$ or $-1$. If we can prove
that they must all have the same value, we are done: if $\varepsilon$ maps the
transpositions to $+1$, $\varepsilon$ is the character $1$, and if it maps
them to $-1$, it is $\mathrm{sgn}$.

So let $\tau$ and $\tau^{\prime}$ be to transpositions. We distinguish two
cases.

Case 1: $\tau$ and $\tau^{\prime}$ share an element, i.e., $\tau=(ij)$ and
$\tau^{\prime}=(ik)$. Since $(ik)=(ij)(jk)(ij)$, we must however have
$\varepsilon(\tau)=\varepsilon(\tau^{\prime})\cdot\varepsilon(ij)^{2}=\varepsilon(\tau^{\prime})$.

Case 2: $\tau$ and $\tau^{\prime}$ do not share an element, i.e., $\tau=(ij)$
and $\tau^{\prime}=(k\ell)$. Since $(k\ell)=(ik)(j\ell)(ij)(ik)(j\ell)$, we
have
$\varepsilon(\tau^{\prime})=\varepsilon(\tau)\varepsilon(ik)^{2}\varepsilon(j\ell)^{2}=\varepsilon(\tau)$.
∎

######  Lemma B.6.

$\\{S_{n},S_{n}\\}=A_{n}$.

###### Proof.

This proof can be found in e.g. [24].

That $\\{S_{n},S_{n}\\}\subseteq A_{n}$ follows from the fact that all
commutators have signature $1$:

$\displaystyle\sigma(\tau\circ\pi\circ\tau^{-1}\circ\pi^{-1})=\sigma(\tau)\sigma(\pi)\sigma(\tau)^{-1}\sigma(\pi^{-1})=1.$

To prove the converse, let us first note that the cases $n=1,2$ are trivial –
in both those cases, $A_{n}=\\{\operatorname{id}\\}$, but $S_{n}$ is also
abelian, so that $\\{S_{n},S_{n}\\}=\\{\operatorname{id}\\}$. For $n\geq 3$,
we use the well-known fact that $A_{n}$ is generated by $3$-cycles [22, Cor.
6.3] $(ijk)$. If we show that they are commutators, we are hence done. But

$\displaystyle(ijk)=(jk)(ij)(jk)(ij),$

so that the claim follows. ∎

######  Lemma B.7.

$\operatorname{SU}(2)$ is perfect.

###### Proof.

For this proof, we are going to use that elements in $\operatorname{SU}(2)$
can be thought of as unit quaternions $q=(\alpha,v)$, where
$\alpha\in\mathbb{R}$ is the real part of the quaternion, and
$v\in\mathbb{R}^{3}$ its vector part. $q$ being of unit norm means that
$\alpha^{2}+|v|^{2}=1$ The group action is given by the quaternion
multiplication, i.e.,

$\displaystyle(\alpha,v)\cdot(\beta,w)=(\alpha\beta-v\cdot w,\alpha v+\beta
w+v\times w),$

where $\cdot$ and $\times$ are the standard dot and cross products of
$\mathbb{R}^{3}$. We need to show that all unit quaternions can be generated
by commutators. We will do this in three steps.

Step 1. Each unit norm quaternion $q$ has a square root $r$, i.e., an $r$ with
$r^{2}=q$. This can be seen through direct calculation: Each unit norm
quaternion can be written as $(\cos(2\theta),\sin(2\theta)u)$ for a
$\theta\in[0,\pi]$ and a unit norm $u$. If we then define
$r=(\cos(\theta),\sin(\theta)u)$, we have

$\displaystyle r^{2}$
$\displaystyle=(\cos^{2}(\theta)-\sin^{2}(\theta),\cos(\theta)\sin(\theta)u+\cos(\theta)\sin(\theta)u+\sin^{2}(\theta)u\times
u)$ $\displaystyle=(\cos(2\theta),\sin(2\theta)u)=q.$

Step 2. Write $r=(\alpha,\beta u)$, let $v$ be a unit norm ector orthogonal to
$u$ and $w=u\times v$. If we define $s=(0,v)$, we have

$\displaystyle sr^{-1}s^{-1}$ $\displaystyle=(0,v)(\alpha,-\beta
u)(0,-v)=(0+\beta v\cdot u,\alpha v-\beta v\times u)(0,-v)=(0,\alpha v+\beta
w)(0,-v)$ $\displaystyle=(\alpha|v|^{2},-\alpha v\times v-\beta w\times
v)=(\alpha,\beta u)=r,$

i.e., $r=sr^{-1}s^{-1}$

Step 3. We now combine the above steps

$\displaystyle q=r^{2}=rsr^{-1}s^{-1}=\\{r,s\\}.$

Hence, $q$ is a commutator, and the claim has been proven. ∎

######  Corollary B.8.

$\operatorname{SO}(3)$ is perfect.

###### Proof.

The perfectness follows from the perfectness of its cover $SU(2)$. If
$\varphi:\operatorname{SU}(2)\to\operatorname{SO}(3)$ is a covering map, write
any $R\in\operatorname{SO}(3)$ as $\varphi(r)$ for an
$r\in\operatorname{SU}(2)$. We just proved that there exists
$s,t\in\operatorname{SU}(2)$ with $r=\\{s,t\\}$. But this implies

$\displaystyle R=\varphi(r)=\varphi(\\{s,t\\})=\\{\varphi(s),\varphi(t)\\}.$

∎

######  Lemma B.9.

$\mathbb{Z}_{n}^{*}$ is isomorphic to the set of $n$-th unit roots.

###### Proof.

$\mathbb{Z}_{n}$ is an abelian group, generated by the element $1$.
Consequently, any group homomorphism is uniquely determined by the value
$\epsilon(1)$. Since $1=\epsilon(0)=\epsilon(\underbrace{1+\dots+1}_{n\text{
times}})=\epsilon(1)^{n}$, $\epsilon(1)$ must be a unit root. Hence, all
elements of $Z_{n}^{*}$ is of the form $\epsilon_{\omega}(k)=\omega^{k}$ with
$\omega$ a unit root. Since that surely defines a group homomorphism,
$\mathbb{Z}_{n}^{*}=\\{\varepsilon_{\omega},\omega^{n}=1\\}$. As for the
isomorphy, we simply need to remark that
$\varepsilon_{\omega}\varepsilon_{\omega^{\prime}}=\varepsilon_{\omega\omega^{\prime}}$
by direct calculation. ∎

######  Lemma B.10.

If $G$ is perfect, $G^{*}$ only contains the trivial character $1$.

###### Proof.

Let $k=\\{h,\ell\\}$ be a commutator and $\varepsilon\in G^{*}$. Then

$\displaystyle\varepsilon(k)=\varepsilon(h\ell
h^{-1}\ell^{-1})=\varepsilon(h)\varepsilon(\ell)\varepsilon(h)^{-1}\varepsilon(\ell)^{-1}=1$

Since $\varepsilon$ is a group homomorphism, this shows that $\varepsilon$ is
equal to $1$ on the entire commutator subgroup $\\{G,G\\}$, and thus, since
$G$ is perfect, on the entirety of $G$. ∎

In fact, to some extent, the converse statement of the last lemma holds. We
include the proof for completeness.

######  Lemma B.11.

Let $\mathbb{F}=\mathbb{C}$. If $G$ is a compact group with $G^{*}=\\{1\\}$,
$G$ is perfect.

###### Proof.

$\\{G,G\\}$ is a _normal_ subgroup of $G$, meaning that if $h\in G$ and
$g\in\\{G,G\\}$, the conjugation $hgh^{-1}$ is also
($\\{h,g\\}=hgh^{-1}g^{-1}$ is surely in $\\{G,G\\}$, and therefore also
$hgh^{-1}=\\{h,g\\}g$). This implies that we can define the quotient group
$A=G/\\{G,G\\}$, which is defined via identifying elements $h,k\in G$ for
which $hk^{-1}\in\\{G,G\\}$. The group operation is inherited from $G$: For
two equivalence classes $[h]$ and $[k]$, its product is defined as
$[h][k]=[hk]$. Also, an $\varepsilon\in G^{*}$ defines an element of $A^{*}$:
Any $\varepsilon$ must by the proof of Lemma B.10 be identically equal to $1$
on $\\{G,G\\}$. This has the consequence that if $h$,$k$ are in the same
equivalence class,

$\displaystyle\varepsilon(h)=\varepsilon(hk^{-1}k)=\varepsilon(hk^{-1})\varepsilon(k)=\varepsilon(k),$

so that $\varepsilon$ is well-defined on $A$.

The group $A$ is called the _Abelianization_ of $G$. The naming stems from the
fact that it is an abelian group – for any elements $[h]$, $[k]$ of $A$,
$\\{[h],[k]\\}=[\\{h,k\\}]=[e]$, due to the definition of $A$. Since $G$ is
compact, $A$ surely also is.

Since $G^{*}$ only contains the trivial element, $A^{*}$ also does. Since $A$
is an abelian compact group, we may now apply the _Pontryagin duality theorem_
[26, p.28], which states that $A$ is isomorphic to the _bidual_
$A^{**}=\\{1\\}^{*}=\\{e\\}$. Hence, $A$ only contains the unity element,
which in turn means that $k=ke^{-1}\in\\{G,G\\}$ for every $k\in G$. That is,
$G$ is perfect. ∎

##  Appendix C Details of the Spinor Field Networks experiments

As advertised, we begin with a review of spinors and the representation theory
of $\operatorname{SU}(2)$. The reader is referred to e.g. [3, 13, 33] for in
depth treatments. $\operatorname{SU}(2)$ has one so-called irreducible
representation (irrep) for each dimension $n>0$. Rather than labelling them by
$n$, they are typically labelled by $\ell=0,1/2,1,3/2,\ldots$, where the
$\ell$’th irrep (or rather the space it is acting on) has dimension $2\ell+1$.
$\ell=0$ corresponds to scalar values that don’t transform under
$\operatorname{SU}(2)$ (i.e., $\operatorname{SU}(2)$ acts as the identity on
them), $\ell=1$ corresponds to vector values on which $\operatorname{SU}(2)$
acts as 3D rotation matrices and $\ell=1/2$ corresponds to spinor values on
which $\operatorname{SU}(2)$ acts as the ordinary $2\times 2$ complex matrix
representation of $\operatorname{SU}(2)$. For integer $\ell$, the
$\operatorname{SU}(2)$-irrep can be taken to consist of only real valued
matrices and hence can be interpreted as acting on $\mathbb{R}^{2\ell+1}$.
These irreps are also irreps of $\operatorname{SO}(3)$. For non-integer $\ell$
on the other hand, it is not possible to let the representation consist of
only real valued matrices. Thus these irreps act on $\mathbb{C}^{2\ell+1}$.
Furthermore, these irreps do not correspond to linear representations of
$\operatorname{SO}(3)$, but only to projective representations of
$\operatorname{SO}(3)$.

Let us denote the irreps of $\operatorname{SU}(2)$ by
$\widetilde{\rho}_{\ell}$, $\ell=0,1/2,1,3/2,\ldots$ and the spaces that they
act on by $\mathcal{V}_{\ell}\sim\mathbb{C}^{2\ell+1}$ (or
$\sim\mathbb{R}^{2\ell+1}$ for integer $\ell$). All other finite dimensional
representations of $\operatorname{SU}(2)$ can be decomposed into these irreps,
meaning that if $\operatorname{SU}(2)$ acts linearly on $\mathbb{C}^{m}$, we
can decompose the space as

$\mathbb{C}^{m}\sim\bigoplus_{\ell\in L}\mathcal{V}_{\ell}$

for some specific multiset $L$ of indices such that $\sum_{\ell\in
L}(2\ell+1)=m$. One particular decomposition is the decomposition of the
tensor product of two $\mathcal{V}_{\ell}$’s. Given $\ell\geq\ell^{\prime}$ we
have that

$\mathcal{V}_{\ell}\otimes\mathcal{V}_{\ell^{\prime}}\sim\mathcal{V}_{\ell-\ell^{\prime}}\oplus\mathcal{V}_{\ell-\ell^{\prime}+1}\oplus\cdots\oplus\mathcal{V}_{\ell+\ell^{\prime}-1}\oplus\mathcal{V}_{\ell+\ell^{\prime}}.$
(9)

In particular, there exists a change of basis matrix that transforms canonical
coordinates on $\mathcal{V}_{\ell}\otimes\mathcal{V}_{\ell^{\prime}}$ into
coordinates which can be split into blocks, each representing coordinates for
one $\mathcal{V}_{j}$. We denote this matrix from the left hand side of (9) to
the right hand side by $C$. $C$ consists of so-called Clebsch-Gordan
coefficients, which are known and hence it is feasible to perform this
decomposition.

The basic idea for our $\operatorname{SU}(2)$-equivariant neural network
architecture is now to have layers that combine features and filters using the
tensor product. This is a standard approach to constructing
$\operatorname{SO}(3)$-equivariant neural networks [28, 11, 18] which we here
generalize slightly to $\operatorname{SU}(2)$. Say that we have a feature $f$
that transforms according to $\widetilde{\rho}_{\ell}$ and a filter $\psi$
that transforms according to $\widetilde{\rho}_{\ell^{\prime}}$. Their tensor
product will transform according to
$\widetilde{\rho}_{\ell}\otimes\widetilde{\rho}_{\ell^{\prime}}$. Note that
(9) means that $C(f\otimes\psi)$ transforms according to

$C(f\otimes\psi)\xmapsto{\operatorname{SU}(2)}C(\widetilde{\rho}_{\ell}\otimes\widetilde{\rho}_{\ell^{\prime}})(f\otimes\psi)=(\widetilde{\rho}_{\ell-\ell^{\prime}}\oplus\widetilde{\rho}_{\ell-\ell^{\prime}+1}\oplus\cdots\oplus\widetilde{\rho}_{\ell+\ell^{\prime}-1}\oplus\widetilde{\rho}_{\ell+\ell^{\prime}})C(f\otimes\psi).$

This means that $C(f\otimes\psi)$ is a concatenation of quantities that
transform according to known irreps. Since the tensor product is the most
general bilinear map, a good way to look at it is that $C$ defines all
possible multiplications of $f$ and $\psi$ that are $\operatorname{SU}(2)$
equivariant. If we are only interested in a particular output type, say the
part of $C(f\otimes\psi)$ that transforms according to
$\widetilde{\rho}_{\ell-\ell^{\prime}}$, then we can simply let $\tilde{C}$
consist of the corresponding rows of $C$, i.e., the first
$2(\ell-\ell^{\prime})+1$ rows, and the quantity $\tilde{C}(f\otimes\psi)$
will be the output with correct behaviour. $C$ typically contains many zeros
and so we don’t actually have to compute the complete $f\otimes\psi$ if we are
only interested in some particular output irreps, but can instead only compute
those values that will be multiplied by non-zero values of $C$. If our network
layers consist of tensor products like this, it is easy to keep track of how
the features in the network transform under $\operatorname{SU}(2)$, and hence
ensure that the output of the network transforms correctly. In our experiments
we have outputs that transform according to $\widetilde{\rho}_{1/2}$ and so
build networks that guarantee such output.

######  Example C.1.

Let us walk through the example of tensoring a vector feature $f$ by a vector
filter $\psi$. Both the feature and filter transform according to
$\widetilde{\rho}_{1}$ which we can take to consist of real-valued 3D rotation
matrices. This means that according to (9), the output should be decomposable
into $\mathcal{V}_{0}\oplus\mathcal{V}_{1}\oplus\mathcal{V}_{2}$, i.e., a
scalar, a vector and a “higher order” 5-dim. feature. The reader already knows
how to combine two vectors to form a scalar and a vector, we use the scalar
product and the cross product! Indeed, the tensor product has the following
form:

$(f_{1},f_{2},f_{3})^{T}\otimes(\psi_{1},\psi_{2},\psi_{3})^{T}=(f_{1}\psi_{1},f_{1}\psi_{2},f_{1}\psi_{3},f_{2}\psi_{1},f_{2}\psi_{2},f_{2}\psi_{3},f_{3}\psi_{1},f_{3}\psi_{2},f_{3}\psi_{3})^{T}.$

The first row of $C$ selects those values that yield the scalar product of $f$
and $\psi$:

$C_{1}=\begin{pmatrix}1&0&0&0&1&0&0&0&1\\\ \end{pmatrix},\quad
C_{1}(f\otimes\psi)=f\cdot\psi.$

The second to fourth rows of $C$ select those values that yield the cross
product of $f$ and $\psi$:

$C_{2:4}=\begin{pmatrix}0&0&0&0&0&1&0&-1&0\\\ 0&0&-1&0&0&0&1&0&0\\\
0&1&0&-1&0&0&0&0&0\\\ \end{pmatrix},\quad C_{2:4}(f\otimes\psi)=f\times\psi.$

The last rows of $C$ select values that yield a $5$-dim. product of $f$ and
$\psi$ transforming according to the irrep $\widetilde{\rho}_{2}$, we don’t
write it out here as it is not too illuminating.

###  C.1 Tensor spherical harmonics filters

We defined the layers in our Spinor Field Networks like

$f_{i}^{\prime}=\sum_{j\neq i}(\psi(x_{j}-x_{i})\oplus s_{j})\otimes f_{j}.$
(10)

A tensor spherical harmonic (see [3]) is the tensor product of a spherical
harmonic and some other quantity that transforms under $\operatorname{SU}(2)$
according to some representation $\widehat{\rho}$. Then (10) could become for
instance

$f_{i}^{\prime}=\sum_{j\neq i}(\psi(x_{j}-x_{i})\otimes s_{j})\otimes f_{j}.$
(11)

Technically our filters in (10) are tensor spherical harmonics where the
spherical harmonic tensored by $s_{j}$ is the trivial $Y^{0}$.

###  C.2 Training/implementation details

| Layer 1
---|---
| Input | Filter
| Scalars | Spinors | Vectors | Scalars | Spinors | Vectors
Spinors as scalars | 4 | 0 | 1 | 1 | 0 | 1
Spinors as features | 0 | 1 | 1 | 1 | 0 | 1
Spinors as filters | 0 | 0 | 1 | 1 | 1 | 1
Spinors squared as vector features | 0 | 0 | 3 | 1 | 0 | 1
Spinors squared as vector filters | 0 | 0 | 1 | 1 | 0 | 3
| Layer 2
| Input | Filter
| Scalars | Spinors | Vectors | Scalars | Spinors | Vectors
Spinors as scalars | 32 | 0 | 8 | 1 | 0 | 1
Spinors as features | 32 | 12 | 12 | 1 | 0 | 1
Spinors as filters | 32 | 4 | 4 | 1 | 1 | 1
Spinors squared as vector features | 32 | 0 | 8 | 1 | 0 | 1
Spinors squared as vector filters | 32 | 0 | 8 | 1 | 0 | 3
| Layer 3
| Input | Filter
| Scalars | Spinors | Vectors | Scalars | Spinors | Vectors
Spinors as scalars | 32 | 0 | 8 | 1 | 0 | 1
Spinors as features | 0 | 12 | 0 | 1 | 0 | 1
Spinors as filters | 32 | 4 | 4 | 1 | 1 | 1
Spinors squared as vector features | 32 | 0 | 8 | 0 | 1 | 0
Spinors squared as vector filters | 32 | 0 | 8 | 0 | 1 | 0
Table 2: Number of scalars, spinors and vectors in each layer of the five
network types. The number of outputs of each type is the same as the number of
inputs to the next layer. The final layer always outputs one spinor, except
for the _Spinors as scalars_ net which outputs four scalars.

The implementation is inspired by the e3nn package [11], but differs in some
crucial aspects (apart from the obvious fact of using spinors).

A tensor product layer in our framework works as follows. We define the number
of input, filter and output scalars, spinors and vectors. For each output
requested, certain input types can be used. For instance, to produce an output
scalar, we can multiply

  1. (i)

an input scalar and a filter scalar, or

  2. (ii)

an input spinor and a filter spinor, or

  3. (iii)

an input vector and a filter vector.

So for each output scalar requested, we compute a weighted sum of all input
scalars and multiply it by a weighted sum of all filter scalars, and similarly
for spinors and vectors. These weighted sums contain the learnable parameters
in the net. We do not use learnt radial functions as in Tensor Field Networks,
as that did not seem to be necessary for our data.

When mapping from spinors to scalars/vectors, we get complex valued scalars
and vectors. These are split into real and imaginary parts, hence if 10 output
scalars are requested we only compute 5 complex output scalars yielding 10
real output scalars. Note that the operation of taking real and imaginary
parts of a vector is rotation equivariant as we work in the basis where
rotation matrices are real valued.

The loss used is $L_{2}$-loss on the regressed spinor, but accounting for
arbitrary sign due to projective ambiguity, i.e.

$\text{loss}(s_{\text{pred}},s)=\min(\|s_{\text{pred}}-s\|,\|s_{\text{pred}}+s\|).$

The networks contain between 2500 and 3000 parameters and are small enough to
train on CPU in tens of seconds per net. All nets were trained using the Adam
optimiser with default PyTorch settings, for 300 epochs with learning rate
$10^{-2}$.

The number of features of each type in the layers and filters of the Spinor
Field Networks is summarised in Table 2.

##  Appendix D Details and further results for ViererNet experiments

We begin by providing some details about the model used in the experiments in
the main paper.

The layers of the projectively equivariant ViererNet used in the main paper
has, in succession, $32$, $32$, $32$ and $11$ intermediate
$\mathbb{R}^{n,n}$-valued feature tuples, each of size
$4=(\mathbb{Z}_{2}^{2})^{*}$. These are subsequently average-pooled to $11$
$4$-tuples. To transform these to scalars, we use $11$ selector vectors, and
send it through first a batch-norm and then a softmax layer to end up with a
final output distribution $p\in\mathbb{R}^{11}$. Before sending the data to
the first layer, we normalize it to have zero mean. The base-line model is a
three-layer standard CNN, with $32$, $32$, $32$ and $11$
$\mathbb{R}^{n,n}$-valued intermediate features, which after average pooling
simply feeds their $11$ output features through first a batch-norm and then a
softmax layer. Both models use $\tanh$ non-linearities. The two models have
different memory footprints, due to the ViererNet having to handle $4$
features in each layer, but each input-output channel pair can be described
the same number of parameters (due to Proposition 2.15(ii)). The selector
vectors do result in 44 additional ViererNet parameters, but this is miniscule
compared to the about 30K in total. The architecture is schematically
presented in Figure 7

We train for 100 epochs using the Adam algorithm, with the learning rate
parameter set to $10^{-4}$.

###  D.1 Additional experiments on CIFAR10

To complement the MNIST experiments in the paper, we tested ViererNet to
classify a similarly modified version of the CIFAR10 dataset.

#### Data

We modify the CIFAR10 dataset in two ways. First, we modify the dataset
exactly the way MNIST is modified in the main paper: I.e., we add a class ’not
an image’ which we put (or not, depending on the class) images after they are
flipped horizontally or vertically: The images in the classes ’airplane’,
’automobile’ and ’bird’ are never put in the ’not an image’-class, the ’cat’
and ’deer’ and ’dog’ images are put there after a horizontal flip, ’frog’ and
’horse’ after a vertical flip and ’ship’ and ’truck’ after both types of flip.
We refer to this version of the dataset as the ’both flip’ dataset.

The above modification has a problem, and that is that since flipping a
CIFAR10-image vertically, in contrast to the MNIST images, most often results
in a plausible image. The horizontal flips however results in objects that are
upside down, and hence are different from the ’normal’ CIFAR images. For this
reason, we also consider a different type of modification of the dataset. In
this version, we still perform flips with the same probabilities as before,
but choose to not change the labels after vertical axis flips at all. We refer
to this version of the data set as the ’single flip’ datasets.

Figure 7: The ViererNet architechtures. $V$ is the input space. For the MNIST
experiment, $M=32$, and for CIFAR, $M=64$. Best viewed in color

#### Models

We use the same ViererNet and baseline architectures as in the main paper,
with the only difference that the number of intermediate features are $64$,
$64$, $64$, $11$, instead of $32$, $32$, $32$, $11$.

#### Results

Using the same training algorithm and learning rate (e.g. Adam with
$\texttt{lr}=10^{-4}$) as for the MNIST experiments, we train each model on
each dataset for 100 epochs. The experiments are repeated 30 times. In Figure
8, the evolutions of the median training and test losses are depicted, along
with errorbars encapsulating $80\%$ of the experiments. In comparison to the
experiments in the main paper, the models are performing more similar. When
comparing the models at the epochs with best median performance, the baseline
outperforms the ViererNet slightly on both datasets – $51.0\%$ vs. $49.9\%$ on
the both flip dataset and $59.0\%$ vs. $57.0\%$ on the single flip dataset.
The performance difference is however not as significant as in the main paper
– the $p$-values for the baseline outperforming ViererNet is only $0.26$ for
the both flip dataset, and $.11$ for the single flip dataset.

Figure 8: Results for the CIFAR10 datasets. From left to right: Training
accuracy on the ’double flip’ version, test accuracy for the ’double flip’
version, training accuracy on the ’single flip’ version and test accuracy on
the ’single flip’ version. The errorbars depict confidence intervals of
$80\%$. Generally, the baseline performs slightly better at the end of the
training, but well within the margin of error. The ViererNet is quicker to
generalize than the baseline.

#### Performance vs. number of iterations

Figures 8 and 5 both suggest that the ViererNet can generalize to the test
data faster, since it outperforms the baseline in early iterations. To test
this quantitatively, we perform the same statistical tests as above, but
choose the best performing epoch among the 25, 50 and 75 first iterations
instead out of all 100 iterations. We report the median accuracies for all
experiments in Table 3. Statistically better figures are highlighted in bold.
We see that the ViererNet outperforms the baseline in the MNIST experiments by
wider margins, and with high statistical confidence, earlier in the training.
Looking at the median values in Table 3, the trend that the ViererNet is
quicker to train apparently persists to some extent for the CIFAR experiments.
However, a closer looks reveals that we can only with high statistical
confidence say that the ViererNet is better than the baseline after 25
iterations on the single flip dataset.

Dataset | MNIST | CIFAR (double flip) | CIFAR10 (single flip)
---|---|---|---
Max it. | 25 | 50 | 75 | 100 | 25 | 50 | 75 | 100 | 25 | 50 | 75 | 100
Baseline | 80.3% | 86.6% | 89.4% | 90.2% | 43.2% | 48% | 49.4% | 51.0% | 47.3% | 53.7% | 57.2% | 59.0%
ViererNet | 89.6% | 91.7% | 92.5% | 92.8% | 45.1% | 48.1% | 49.4% | 49.9% | 52.1% | 55.2% | 56.5% | 57.0%
Table 3: Median accuracies (over 30 runs) in all experiments for different
upper iterations bounds. Statistically significantly ($p<.05$) better
performing models for each dataset and upper iteration limits are marked in
bold font. (In fact, using the higher $p$-value $0.1$ would not change the
appearance of the table).

#### Discussion

Although the CIFAR experiments are in general not statistically conclusive, we
can conclude that the ViererNet in comparison to the MNIST experiments in the
main paper perform worse on the CIFAR set. We believe the main reason behind
this is simply that the property of projective
$\mathbb{Z}_{2}^{2}$-equivariance is less well suited for these tasks. For the
both flips version of the dataset, as we discussed above, flipping vertically
will most often lead to a new image which is plausible to come from the same
CIFAR-class, and it will be less helpful to apply the strategy of either
changing or maintaining the class when flipping the images of the class. As
for the single flip version of the dataset, the same argument does not apply.
However, in this case, it is ultimately only a subgroup of the Vierergroup
that is changing the images in the dataset. Hence, a ViererNet is not the best
suited architecture – a $\mathbb{Z}_{2}$-equivariant would instead be.

A more thorough empirical comparison of the effectiveness of projectively
equivariant architectures for classification tasks would be very interesting,
but out of scope for this article. We leave it for future work.

###  D.2 Training details

We used cross-entropy-loss in our experiments, to account for the different
sizes of the classes: the weight vector vectors were chosen as

$\displaystyle
w_{\mathrm{MNIST}}=[1,1,1,1.5,1.5,1.5,1.5,1.5,3,3,1],w_{\mathrm{CIFAR}}=[1,1,1,1.5,1.5,1.5,1,1,1.5,1.5,1]$

for the MNIST and CIFAR10 experiments, respectively. A batch size of 32 was
used for all experiments. The models were trained, on single A40 GPUs, in
parallell on a high performance computing cluster. The MNIST experiments
presented in the main paper used in total about 110 GPU hours, whereas the
CIFAR10-experiments presented in the appendix used about 170 GPU hours each.

###  D.3 Implementational details

Since all of the basis elements, as depicted in Figure 6 are sparse, we
implement the ViererNet layers i PyTorch by saving nine parameters per in-
layer-out-layer pair, which we turn into four $3\times 3$ filters through
multiplying the $\mathbb{R}^{9}$ vector with a sparse matrix. These are then
used in padded standard PyTorch convolution layers, and combined in the manner
described in Section 2.1.