# Discrete neural nets and polymorphic learning

Charlotte Aten  Department of Mathematics
University of Denver
Denver 80208
USA https://aten.cool<EMAIL_ADDRESS>
###### Abstract.

Theorems from universal algebra such as that of Murskiĭ from the 1970s have a
striking similarity to universal approximation results for neural nets along
the lines of Cybenko’s from the 1980s. We consider here a discrete analogue of
the classical notion of a neural net which places these results in a unified
setting. We introduce a learning algorithm based on polymorphisms of
relational structures and show how to use it for a classical learning task.

###### Key words and phrases:

Neural nets, polymorphisms, clones, universal algebra, machine learning

###### 2020 Mathematics Subject Classification:

68T07, 08A70, 05C60

This project was supported by NSF Grant US NSF HDR TRIPODS 1934962. Thanks to
Nick Cimaszewski and Andrey Yao for their participation in the corresponding
REU program. Thanks to Rachel Dennis for the code she contributed as part of
her senior thesis project[5].

###### Contents

  1. 1 Introduction
  2. 2 Universal algebra and clones
  3. 3 Polymorphisms
  4. 4 Discrete neural nets
  5. 5 Learning algorithm
  6. 6 Example application: Polymorphisms for binary images

##  1\. Introduction

This work has its genesis in the observation that a class of theorems from
universal algebra (exemplified by Murskiĭ’s Theorem from the 1970s[1, Section
6.2]) were discrete forerunners of a class of theorems from machine learning
(exemplified by Cybenko’s work from the 1980s[4]). The similarity between
these two types of results motivates the mathematical analysis of a discrete
analogue of the usual (continuous) treatment of neural nets. Unsurprisingly,
this analogue fits neatly into the established theory of universal algebra,
and in particular the theory of clones. Although our work has no immediate
predecessor to our knowledge, we direct the reader to a similar perspective in
evolutionary computation[3] as well as the successful use of polymorphisms of
relational structures in the resolution of the Dichotomy Conjecture for the
Constraint Satisfaction Problem by Bulatov in 2017[2]. See [6] for an
introduction to such applications of universal algebra and clones to
combinatorial problems. The equivariant maps of equivariant neural nets[7] can
be viewed as polymorphisms, although existing work on such neural nets does
not appear to have moved in the direction we describe here.

We use Shalev-Shwartz and Ben-David as a general reference for the
mathematical treatment of machine learning[8] and we use Bergman as a
reference for universal algebra and clone theory[1].

The structure of the paper is as follows: We give some relevant background on
universal algebra, clones, and polymorphisms, after which we introduce our
concept of a discrete neural net. The reader need not be familiar with the
established theory of neural nets nor their application in order to follow
this section, although one who is will note that our concept of neural net
both mathematically subsumes the established (continuous) one and also is, in
a sense, closer to neural nets as they are actually implemented in digital
computers. We then describe a learning algorithm which makes use of
polymorphisms pertaining to the task at hand. Finally, we illustrate how this
algorithm may be used for image classification and transformation.

In this paper we adopt the convention that the natural numbers are
$\mathbb{N}\coloneqq\left\\{\,\;\middle|\;0,1,2,\dots\,\right\\}$ and the
positive integers are
$\mathbb{P}\coloneqq\left\\{\,\;\middle|\;1,2,3,\dots\,\right\\}$. We write
$\operatorname{Sb}(A)$ the indicate the power set of a set $A$, we denote the
cardinality of a set $A$ by $\lvert A\rvert$, and we define

$\operatorname{Sb}_{\mathbb{P}}(A)\coloneqq\left\\{\,U\subset
A\;\middle|\;\lvert U\rvert\in\mathbb{P}\,\right\\}.$

Given $n\in\mathbb{P}$ we set
$[n]\coloneqq\left\\{\,\;\middle|\;0,1,\dots,n-1\,\right\\}$. We denote by
$\operatorname{Mat}_{n}(\mathbb{F})$ the set of $n\times n$ matrices over a
field $\mathbb{F}$. Our matrices are $0$-indexed, so the entries of a matrix
$a\in\operatorname{Mat}_{2}(\mathbb{F})$ are $a_{00}$, $a_{01}$, $a_{10}$, and
$a_{11}$. This is to say that

$\operatorname{Mat}_{n}(\mathbb{F})=\mathbb{F}^{[n]^{2}}.$

We indicate by $\Sigma_{n}$ the set of permutations of $[n]$ and we indicate
by $\mathbf{\Sigma}_{n}$ the corresponding permutation group. We define
$\mathbf{D}_{n}$ to the the group of diheadral symmetries of an $n$-gon and
write $D_{n}$ to indicate the set of such symmetries. Note that $\lvert
D_{n}\rvert=2n$.

##  2\. Universal algebra and clones

We will make use of some of the language of universal algebra and the
associated theory of clones. Algebras in the universal algebra sense are built
from operations, which we will use as activation functions in our neural nets.

######  Definition 1 (Operation, arity).

Given a set $A$ and some $n\in\mathbb{N}$, we refer to a function $f\colon
A^{n}\to A$ as an _operation_ on the set $A$. We sometimes say that such a
function is an _$n$ -ary operation_ on $A$ or that $f$ has _arity_ $n$.

Although we won’t make extensive use of the notion here, the basic objects of
study for universal algebra are defined as follows.

######  Definition 2 (Algebra, universe, signature, basic operation).

An algebra $\mathbf{A}\coloneqq(A,F)$ consists of a set $A$ (the _universe_ or
_underlying set_ of $\mathbf{A}$) along with a collection
$F\coloneqq\\{f_{i}\\}_{i\in I}$ of operations on $A$ indexed by a set $I$.
There exists a well-defined function $\rho\colon I\to\mathbb{N}$ such that
$f_{i}$ has arity $\rho(i)$. This function $\rho$ is called the _signature_ of
$\mathbf{A}$. Each operation $f_{i}$ is said to be a _basic operation_ of
$\mathbf{A}$.

As usual in the categorical perspective on mathematics, one actually wants to
study the morphisms between the objects under consideration. The relevant
notion of morphism follows.

######  Definition 3 (Homomorphism (of algebras)).

Given algebras $\mathbf{A}\coloneqq(A,F)$ where $F\coloneqq\\{f_{i}\\}_{i\in
I}$ and $\mathbf{B}\coloneqq(B,G)$ where $G\coloneqq\\{g_{i}\\}_{i\in I}$,
both of signature $\rho\colon I\to\mathbb{N}$, we say that a function $h\colon
A\to B$ is a _homomorphism_ from $\mathbf{A}$ to $\mathbf{B}$ when for each
$i\in I$ we have for all $a_{1},\dots,a_{\rho(i)}\in A$ that

$h(f(a_{1},\dots,a_{\rho(i)}))=g(h(a_{1}),\dots,h(a_{\rho(i)})).$

When $h$ is a homomorphism from $\mathbf{A}$ to $\mathbf{B}$ we write
$h\colon\mathbf{A}\to\mathbf{B}$.

These homomorphisms will be generalized in section 3 and we will spend much of
the rest of the paper discussing that generalization.

Given an algebra $\mathbf{A}\coloneqq(A,F)$, one might notice that the basic
operations could be composed in a way analogous to that of functions (of a
single variable). This is captured in the next definition.

######  Definition 4 (Generalized composite).

Given a set $A$, a $k$-ary operation $f$ on $A$ and a collection
$\\{g_{i}\\}_{i\in\left\\{\,\;\middle|\;1,2,\dots,k\,\right\\}}$ of $n$-ary
operations on $A$ the _generalized composite_ of these operations is the
$n$-ary operation

$f[g_{1},\dots,g_{k}]\colon A^{n}\to A$

where for any $a\in A^{n}$ we set

$f[g_{1},\dots,g_{k}](a)\coloneqq f(g_{1}(a),\dots,g_{k}(a)).$

Just as the notion of a monoid may arise from studying sets of functions
closed under composition, so we arrive at the notion of a clone by studying
sets of operations closed under generalized composition. Just as we would like
the include the identity function in any monoid of functions under
composition, we would also like the include the following operations in any
clone.

######  Definition 5 (Projection).

Given $k,n\in\mathbb{P}$ with $k\leq n$ we define the _$(n,k)$ -projection_
operation $\pi_{n,k}\colon A^{n}\to A$ by

$\pi_{n,k}(a_{1},\dots,a_{n})\coloneqq a_{k}.$

We are now ready to define a clone.

######  Definition 6 (Clone).

Given a set $A$ and a set of operations $\mathcal{C}$ on $A$, we say that
$\mathcal{C}$ is a _clone_ when $\mathcal{C}$ is closed under generalized
composition and $\mathcal{C}$ contains all the projection operations on $A$.

There are a couple natural examples of clones.

######  Example 1.

Let $\operatorname{Op}_{n}(A)\coloneqq A^{A^{n}}$ be the set of all $n$-ary
operations on the set $A$. The largest clone on $A$ is

$\operatorname{Op}(A)\coloneqq\bigcup_{n\in\mathbb{N}}\operatorname{Op}_{n}(A),$

which merely consists of all possible operations on $A$.

######  Example 2.

The clone

$\operatorname{Proj}(A)\coloneqq\left\\{\,\pi_{n,k}\colon A^{n}\to
A\;\middle|\;k,n\in\mathbb{P}\text{ and }k\leq n\,\right\\}$

of all projection operations is the smallest clone on the set $A$.

Intermediate between these two examples are the clones of polymorphisms we
will see in the proceeding section.

##  3\. Polymorphisms

We need a notion which simultaneously generalizes homomorphism and operation.
That is, we would like functions which are simultaneously homomorphisms and
operations in an appropriate way. If we choose such operations for our
activation functions, we will find that we can ensure our neural net is at
every step representing a function which obeys some pre-defined structure
related to our learning task.

These ideas can be formulated in a more abstract categorical context, but for
the purposes of this paper we will introduce polymorphisms only for the kinds
of relational structures studied in model theory, as this will suffice for our
examples.

Relational structures are like algebras in the previous section, but they can
also carry the following type of basic object.

######  Definition 7 (Relation, arity).

Given a set $A$ and $n\in\mathbb{P}$ we say that $\theta\subset A^{n}$ is an
_$n$ -ary relation_ on $A$. We also say that $\theta$ has _arity_ $n$.

The general model-theoretic concept of a structure is as follows.

######  Definition 8 (Structure, universe, basic operations/relations).

A _structure_ $\mathbf{A}\coloneqq(A,F,\Theta)$ conisists of a set $A$ (the
_universe_ or _underlying set_ of the structure) as well as indexed
collections $F\coloneqq\\{f_{i}\\}_{i\in I}$ and
$\Theta\coloneqq\\{\theta_{j}\\}_{j\in J}$ of operations on $A$ (the _basic
operations_ of $\mathbf{A}$) and of relations on $A$ (the _basic relations_ of
$\mathbf{A}$). We require that the index sets $I$ and $J$ be disjoint.

Observe that such a structure consists of an algebra $(A,F)$ along with a
collection of relations on a the underlying set $A$. We have a notion of
signature for structures.

######  Definition 9 (Signature).

Given a structure $(A,F,\Theta)$ there exists a well-defined function
$\rho\colon I\cup J\to\mathbb{N}$ such that each basic operation $f_{i}$ has
arity $\rho(i)$ and each basic relation $g_{j}$ has arity $\rho(j)$.

There is again a notion of morphism for structures of a given signature.

######  Definition 10 (Homomorphism (of structures)).

Given structures $\mathbf{A}\coloneqq(A,F,\Theta)$ and
$\mathbf{B}\coloneqq(B,G,\Psi)$ where $F\coloneqq\\{f_{i}\\}_{i\in I}$,
$G\coloneqq\\{g_{i}\\}_{i\in I}$, $\Theta\coloneqq\\{\theta_{j}\\}_{j\in J}$,
and $\Psi\coloneqq\\{\psi_{j}\\}_{j\in J}$, both of the same signature
$\rho\colon I\cup J\to\mathbb{N}$, we say that a function $h\colon A\to B$ is
a _homomorphism_ from $\mathbf{A}$ to $\mathbf{B}$ when $h\colon(A,F)\to(B,G)$
is a homomorphism of algebras and for each $j\in J$ we have for all
$a_{1},\dots,a_{\rho(j)}\in A$ that if

$(a_{1},\dots,a_{\rho(j)})\in\theta_{j}$

then

$(h(a_{1}),\dots,h(a_{\rho(j)}))\in\psi_{j}.$

Given a signature $\rho\colon I\cup J$ we have a category whose objects are
such structures with signature $\rho$ are whose morphisms are the
aforementioned homomorphisms. In this category has all products, so in
particular we have that the $n^{\text{th}}$ direct power of any structure
$\mathbf{A}$ exists for any $n\in\mathbb{N}$.

We can now define one of the titular concepts of this paper.

######  Definition 11 (Polymorphism).

Given a structure $\mathbf{A}$ we say that a homomorphism
$f\colon\mathbf{A}^{n}\to\mathbf{A}$ is a _polymorphism_ of $\mathbf{A}$.

If $\mathbf{A}\coloneqq(A,F,\Theta)$ and $f\colon\mathbf{A}^{n}\to\mathbf{A}$
is a polymorphism of $\mathbf{A}$ we have that $f\colon A^{n}\to A$ is an
$n$-ary operation in the manner previously discussed. The polymorphisms of a
given structure form a clone.

######  Definition 12 (Clone of polymorphisms).

Given a structure $\mathbf{A}$ the _clone of polymorphisms_ of $\mathbf{A}$ is

$\operatorname{Poly}(\mathbf{A})\coloneqq\left\\{\,f\colon\mathbf{A}^{n}\to\mathbf{A}\;\middle|\;f\text{
is a polymorphism and }n\in\mathbb{N}\,\right\\}.$

Since the set of polymorphisms of a structure $\mathbf{A}$ is closed under
generalized composition, we will see that neural nets with such operations as
their activation functions always model polymorphisms of $\mathbf{A}$.

##  4\. Discrete neural nets

In order to apply results and theory pertaining to finite algebras we will
consider a discrete, finite analogue of neural networks. We begin with a
description of neural nets which encompasses both the traditional, continuous
variant and our new variant.

######  Definition 13 (Neural net).

A _neural net_ $(V_{1},\dots,V_{r},E,\Phi,\leq)$ with $r$ layers on a set $A$
consists of

  1. (1)

a finite digraph $(V,E)$ (the _architecture_ of the neural net),

  2. (2)

for each $v\in V\setminus V_{1}$ a function $\Phi(v)\colon A^{\rho(v)}\to A$
(the _activation function_ of $v$),

  3. (3)

for each $i\in[r]$ a total ordering $\leq_{i}$ of $V_{i}$ (the _vertex
ordering_ of layer $i$)

where

  1. (1)

$V\coloneqq\bigcup_{i=1}^{r}V_{i}$,

  2. (2)

$V_{i}\cap V_{j}=\varnothing$ when $i\neq j$,

  3. (3)

the only edges in $E$ are from vertices in $V_{i}$ to vertices in $V_{i+1}$
for $i<r$,

  4. (4)

$\rho(v)$ is the indegree of $v$ in $(V,E)$, and

  5. (5)

if $i\neq r$ then every vertex $v\in V_{i}$ has nonzero outdegree.

The set $A$ in the preceding definition is also referred to as the _universe_
or the _underlying set_ of the neural net in question. The case where
$A=\mathbb{R}$ is that of a traditional neural net. Our treatment of
activation functions is a bit different from that in the existing literature
in the sense that we allow for nullary (constant) activation functions
$\Phi(v)\colon A^{0}\to A$ and do not assume that each activation function is
of the form $f(w\cdot x)$ where $x=(x_{1},\dots,x_{\rho(v)})$ is the tuple of
arguments of $\Phi(v)$ and $w=(w_{1},\dots,w_{\rho(v)})$ is a tuple of
constant weights. Indeed, there is no relevant analogue of a dot product for
an arbitrary set. Should one wish to separate out the weights in this
presentation of a neural net the weights $w_{i}$ may be taken to be the output
values of nullary activation functions.

By the total ordering $\leq_{i}$ on each $V_{i}$ (which is necessarily finite
since we assume that $(V,E)$ is a finite digraph) we can list the vertices of
$V_{i}$ as

$v_{i,1}<v_{i,2}<\cdots<v_{i,\lvert V_{i}\rvert}.$

Similarly, we will use the notation $\phi_{i,j}\coloneqq\Phi(v_{i,j})$.
Naturally we will write $v_{ij}$ and $\phi_{ij}$ rather than $v_{i,j}$ and
$\phi_{i,j}$ when there is no chance of confusion.

######  Example 3.

Consider a neural net

$N\coloneqq(V_{1},V_{2},V_{3},E,\Phi,\leq)$

with $3$ layers on the set
$\mathbb{F}_{5}=\left\\{\,\;\middle|\;0,1,2,3,4\,\right\\}$ where

$V_{1}\coloneqq\left\\{\,\;\middle|\;v_{11},v_{12},v_{13}\,\right\\},$
$V_{2}\coloneqq\left\\{\,\;\middle|\;v_{21},v_{22},v_{23},v_{24}\,\right\\},$
$V_{3}\coloneqq\left\\{\,\;\middle|\;v_{31},v_{32}\,\right\\},$ $\displaystyle
E\coloneqq\\{$
$\displaystyle(v_{11},v_{21}),(v_{11},v_{22}),(v_{12},v_{22}),(v_{13},v_{22}),(v_{13},v_{23}),$
$\displaystyle(v_{21},v_{31}),(v_{22},v_{31}),(v_{22},v_{32}),(v_{23},v_{32}),(v_{24},v_{32})\\},$
$\phi_{21}\colon\mathbb{F}_{5}\to\mathbb{F}_{5}$

is given by

$\phi_{21}(x)\coloneqq-x,$
$\phi_{22}\colon\mathbb{F}_{5}^{3}\to\mathbb{F}_{5}$

is given by

$\phi_{22}(x,y,z)\coloneqq x-y+z,$
$\phi_{23}\colon\mathbb{F}_{5}\to\mathbb{F}_{5}$

is given by

$\phi_{23}(x)\coloneqq-x,$
$\phi_{24}\colon\mathbb{F}_{5}^{0}\to\mathbb{F}_{5}$

is given by

$\phi_{24}()\coloneqq 3,$ $\phi_{31}\colon\mathbb{F}_{5}^{2}\to\mathbb{F}_{5}$

is given by

$\phi_{31}(x,y)\coloneqq xy^{2},$
$\phi_{32}\colon\mathbb{F}_{5}^{3}\to\mathbb{F}_{5}$

is given by

$\phi_{32}(x,y,z)\coloneqq xy-z,$

and $\leq\coloneqq(\leq_{1},\leq_{2},\leq_{3})$ is the triple of total orders
determined by

$v_{11}<v_{12}<v_{13},$ $v_{21}<v_{22}<v_{23}<v_{24},$

and

$v_{31}<v_{32}.$

The architecture $(V,E)$ of $N$ is pictured in figure 1.

size(120); dotfactor=10; label(”$v_{11}$”,(0,0)); draw(circle((0,0),1));
draw((1,0)–(4,0),arrow=Arrow); draw((1,0)–(4,-3),arrow=Arrow);
label(”$v_{12}$”,(0,-3)); draw(circle((0,-3),1));
draw((1,-3)–(4,-3),arrow=Arrow); label(”$v_{13}$”,(0,-6));
draw(circle((0,-6),1)); draw((1,-6)–(4,-3),arrow=Arrow);
draw((1,-6)–(4,-6),arrow=Arrow); label(”$v_{21}$”,(5,0));
draw(circle((5,0),1)); draw((6,0)–(9,0),arrow=Arrow);
label(”$v_{22}$”,(5,-3)); draw(circle((5,-3),1));
draw((6,-3)–(9,0),arrow=Arrow); draw((6,-3)–(9,-3),arrow=Arrow);
label(”$v_{23}$”,(5,-6)); draw(circle((5,-6),1));
draw((6,-6)–(9,-3),arrow=Arrow); label(”$v_{24}$”,(5,-9));
draw(circle((5,-9),1)); draw((6,-9)–(9,-3),arrow=Arrow);
label(”$v_{31}$”,(10,0)); draw(circle((10,0),1)); label(”$v_{32}$”,(10,-3));
draw(circle((10,-3),1));

Figure 1. The architecture of a neural net

While all neural nets as we have defined them must have a finite number of
nodes and edges in their architectures, the preceding example is of a neural
net on a finite set, which is the type of neural net we will primarily examine
here.

######  Definition 14 (Discrete neural net).

We say that a neural net $N$ on a set $A$ is _discrete_ when $A$ is a finite
set.

Our general definition also encompasses neural nets on infinite sets, such as
$\mathbb{R}$.

######  Example 4.

The neural nets that Cybenko considered in his work on universal
approximation[4] would, in our scheme, have $r=5$ layers (one input, three
hidden, and one output). We would take $A=\mathbb{R}$ and allow our activation
functions to be either constants (the weights), the identity map
$\operatorname{Id}_{\mathbb{R}}\colon\mathbb{R}^{1}\to\mathbb{R}$ (for
carrying a value forward to the next layer unchanged), a sigmoid function

$\sigma(t)\coloneqq\frac{1}{1+e^{-t}},$

or a dot product. Other characterizations are also possible, depending on
which activation functions one allows.

Note that we count layers $1$ and $r$ (the input and output layers,
respectively) in the total number of layers in the neural net. Thus, a neural
net with no hidden layers has $2$ layers by this definition, while a neural
net with one hidden layer has $3$ layers by this definition, and so on.

Neural nets represent functions by way of generalized composition.

######  Definition 15 (Function represented by a neural net).

Given a neural net

$N_{r}\coloneqq(V_{1},\dots,V_{r},E,\Phi,\leq)$

on a set $A$ the _function represented_ by $N_{r}$ is

$g_{r}\colon A^{\lvert V_{1}\rvert}\to A^{\lvert V_{r}\rvert}$

where

  1. (1)

$g_{r}=\operatorname{Id}_{A^{\lvert V_{1}\rvert}}$ when $r=1$ and

  2. (2)

when $r>1$ we set

$(g_{r}(x))_{j}\coloneqq\phi_{r,j}((g_{r-1}(x))_{j_{1}},\dots,(g_{r-1}(x))_{j_{\rho(v_{r,j})}})$

where $x=(x_{1},\dots,x_{\lvert V_{1}\rvert})$, $g_{r-1}$ is the function
represented by the neural net
$N_{r-1}\coloneqq(V_{1},\dots,V_{r-1},E^{\prime},\Phi^{\prime},\leq^{\prime})$
obtained by deleting the $r^{\text{th}}$ layer of $N_{r}$, and the in-
neighborhood of $v_{j}$ in $(V,E)$ is

$\left\\{\,\;\middle|\;v_{r-1,j_{1}},v_{r-1,j_{2}},\dots,v_{r-1,j_{\rho(v_{r,j})}}\,\right\\}$

where

$v_{r-1,j_{1}}<v_{r-1,j_{2}}<\cdots<v_{r-1,j_{\rho(v_{r,j})}}.$

Our previous examples of neural nets also give us examples of functions
represented by neural nets.

######  Example 5.

Let $N$ be the neural net from example 3. The function
$g\colon\mathbb{F}_{5}^{3}\to\mathbb{F}_{5}^{2}$ represented by the neural net
$N$ is given by

$\displaystyle g(x_{1},x_{2},x_{3})$
$\displaystyle\coloneqq(\phi_{31}(\phi_{21}(x_{1}),\phi_{22}(x_{1},x_{2},x_{3})),\phi_{32}(\phi_{22}(x_{1},x_{2},x_{3}),\phi_{23}(x_{3}),\phi_{24}()))$
$\displaystyle=(\phi_{31}(-x_{1},x_{1}-x_{2}+x_{3}),\phi_{32}(x_{1}-x_{2}+x_{3},-x_{3},3))$
$\displaystyle=(-x_{1}(x_{1}-x_{2}+x_{3})^{2},(x_{1}-x_{2}+x_{3})(-x_{3})-3).$

It can be helpful to view the architecture of the neural net with each node
labeled by its corresponding argument variable (for the input nodes in layer
$1$) or activation function (for the nodes in subsequent layers), as shown in
figure 2.

size(120); dotfactor=10; label(”$x_{1}$”,(0,0)); draw(circle((0,0),1));
draw((1,0)–(4,0),arrow=Arrow); draw((1,0)–(4,-3),arrow=Arrow);
label(”$x_{2}$”,(0,-3)); draw(circle((0,-3),1));
draw((1,-3)–(4,-3),arrow=Arrow); label(”$x_{3}$”,(0,-6));
draw(circle((0,-6),1)); draw((1,-6)–(4,-3),arrow=Arrow);
draw((1,-6)–(4,-6),arrow=Arrow); label(”$\phi_{21}$”,(5,0));
draw(circle((5,0),1)); draw((6,0)–(9,0),arrow=Arrow);
label(”$\phi_{22}$”,(5,-3)); draw(circle((5,-3),1));
draw((6,-3)–(9,0),arrow=Arrow); draw((6,-3)–(9,-3),arrow=Arrow);
label(”$\phi_{23}$”,(5,-6)); draw(circle((5,-6),1));
draw((6,-6)–(9,-3),arrow=Arrow); label(”$\phi_{24}$”,(5,-9));
draw(circle((5,-9),1)); draw((6,-9)–(9,-3),arrow=Arrow);
label(”$\phi_{31}$”,(10,0)); draw(circle((10,0),1));
label(”$\phi_{32}$”,(10,-3)); draw(circle((10,-3),1));

Figure 2. Labeled architecture of a neural net

##  5\. Learning algorithm

If discrete neural nets are to have some utility, there ought to be a learning
algorithm which tells us how to take a given neural net and modify its
activation functions in such a way as to obtain a new neural net which
performs better on a given learning task. Traditionally this was done by using
differentiable activation functions on a neural net with universe $\mathbb{R}$
so that a loss function could be differentiated and weights could be adjusted
in the direction that would most quickly reduce the empirical loss.

It is immediately apparent that we cannot do something identical, as we have
thrown out the notion of continuity, much less differentiability. We can
recover some of what we lost by specifying which operations on our universe
are “close to” each other, as captured in the following definition.

######  Definition 16 (Neighbor function, neighbor).

Given a clone $\mathcal{C}$ we say that a function
$\eta\colon\mathcal{C}\to\operatorname{Sb}_{\mathbb{P}}(\mathcal{C})$ is a
_neighbor function_ on $\mathcal{C}$ when

  1. (1)

if $g\in\operatorname{Op}_{n}(\mathcal{C})$ then
$\eta(g)\subset\operatorname{Op}_{n}(\mathcal{C})$ and

  2. (2)

for all $g\in\mathcal{C}$ we have $g\in\eta(g)$.

We refer to a member of $\eta(g)$ as a _neighbor_ of $g$ (with respect to
$\eta$).

This definition refers to an arbitrary clone $\mathcal{C}$ rather than just
the clone of all operations $\operatorname{Op}(A)$. The reason for this is
that using the clone of all operations will allow us to learn any function
given a large enough architecture, but this is exactly what allows
overfitting. We will give an example of a sensible family of choices for the
clone $\mathcal{C}$ in the following section.

Given a choice of a neighbor function, we now have an idea of which activation
functions are “close to” each other but we still don’t have an analogue of
differentiation which would allow us to pick the best way to improve our
neural net’s performance by changing an activation function. Here we proceed
by simply trying each of the neighbor functions to a given activation function
at a particular node, checking whether each one reduces the empirical loss,
and then switching the activation function at that node to the one which
reduces the empirical loss the most.

A single iteration of our learning algorithm is given in pseudocode in
algorithm 1.

Algorithm 1 Our learning algorithm

0: A neural net $N\coloneqq(V_{1},\dots,V_{r},E,\Phi,\leq)$ on a set $A$ whose
activation functions all belong to a fixed clone $\mathcal{C}$.

0: A loss function $\ell\colon A^{\lvert V_{r}\rvert}\times A^{\lvert
V_{r}\rvert}\to\mathbb{R}_{\geq 0}$.

0: A neighbor function $\eta$ on $\mathcal{C}$.

0: A set of training pairs $T\subset A^{\lvert V_{1}\rvert}\times A^{\lvert
V_{r}\rvert}$.

1: Select a random node $v_{ij}$ from $\bigcup_{s=2}^{r}V_{s}$.

2: Create a dictionary $D$ whose keys are the members of $\eta(\phi_{ij})$.

3: for all $g\in\eta(\phi_{ij})$ do

4: Set the value of $D[g]$ to be $\frac{1}{\lvert T\rvert}\sum_{(x,y)\in
T}\ell(f(x),y)$ where $f$ is the function represented by the neural net
$N_{g}$ obtained from $N$ by redefining $\Phi(v_{ij})$ to be $g$.

5: end for

6: Fix $g$ to be a member of $\eta(\phi_{ij})$ so that
$D[g]=\min(\left\\{\,D{[g^{\prime}]}\;\middle|\;g^{\prime}\in\eta(\phi_{ij})\,\right\\})$.

6: The neural net $N_{g}$ obtained from $N$ by redefining $\Phi(v_{ij})$ to be
$g$.

Note that because the clone $\mathcal{C}$ is closed under generalized
composition, the function represented by $N$ and the function represented by
$N_{g}$ in algorithm 1 will both always belong to $\mathcal{C}$.

We give a few examples of neighbor functions. Our first example generally
leads to overfitting in practice and is also very computationally expensive as
each operation of arity $n$ has $\lvert A\rvert^{\lvert A\rvert^{n}}$
neighbors.

######  Example 6.

In that case that $\mathcal{C}=\operatorname{Op}(A)$ we can define a neighbor
function $\eta$ on $\mathcal{C}$ by setting
$\eta(g)\coloneqq\operatorname{Op}_{n}(A)$ when
$g\in\operatorname{Op}_{n}(A)$.

Our next example is a bit more structured.

######  Example 7.

Take $\mathcal{C}\coloneqq\operatorname{Poly}(\mathbb{Z}/p\mathbb{Z})$ where
$\mathbb{Z}/p\mathbb{Z}$ is the cyclic group of prime order $p$. Since
$\operatorname{Poly}_{n}(\mathbb{Z}/p\mathbb{Z})=(\mathbb{F}_{p}^{n})^{*}$ we
have that each member $g$ of $\operatorname{Poly}_{n}(\mathbb{Z}/p\mathbb{Z})$
may be represented as a vector
$\tilde{g}\coloneqq(g_{1},g_{2},\dots,g_{n})\in\mathbb{F}_{p}^{n}$ where
$g(x)=\tilde{g}\cdot x$. We define a neighborhood function $\eta$ on
$\mathcal{C}$ by

$\eta(g)\coloneqq\left\\{\,h\in(\mathbb{F}_{p}^{n})^{*}\;\middle|\;\tilde{h}=(g_{1}+u_{1},\dots,g_{n}+u_{n})\text{
where }u_{i}\in\left\\{\,\;\middle|\;-1,0,1\,\right\\}\,\right\\}$

when $g\in\operatorname{Poly}_{n}(\mathbb{Z}/p\mathbb{Z})$.

While it is often difficult to parametrize the polymorphism clone of a
structure in the manner of the previous example, we can obtain a practical
family of neighbor functions by knowing some collection of endomorphisms of
the given structure.

######  Example 8.

Let $\mathbf{A}$ be a relational structure, let
$H\subset\operatorname{End}(\mathbf{A})$ be a collection of endomorphisms of
$\mathbf{A}$, and let $G\subset\operatorname{Poly}(\mathbf{A})$ be a
collection of polymorphisms of $\mathbf{A}$. We define a neighbor function
$\eta$ on $\operatorname{Poly}(\mathbf{A})$ by setting

$\eta(g)\coloneqq\left\\{\,h_{n+1}\circ
g{[h_{1},\dots,h_{n}]}\;\middle|\;h_{i}\in H\,\right\\}\cup G.$

In order to actually make use of the preceding neighbor function in our
algorithm we need to know some collection
$G\subset\operatorname{Poly}(\mathbf{A})$ of higher-arity polymorphisms of
$\mathbf{A}$ in order to set the initial activation functions for higher-arity
nodes in our neural net. Finding such polymorphisms is not a tractable task in
general, but we can do this in some specific cases of interest.

##  6\. Example application: Polymorphisms for binary images

We consider a special case of the neighbor function given in example 8. Fix
$n\in\mathbb{N}$ and take
$A_{n}\coloneqq\operatorname{Mat}_{n}(\mathbb{F}_{2})$. We refer to $a\in
A_{n}$ as a _binary image_ and we say that $a\in A_{n}$ has _size_ $n$. We
refer to $(i,j)\in[n]^{2}$ as a _pixel_ in this context and say that $a\in
A_{n}$ has an entry of $a_{ij}$ at pixel $(i,j)$.

There is a natural notion of distance between two binary images.

######  Definition 17 (Hamming distance).

Given binary images $a_{1},a_{2}\in A_{n}$ the _Hamming distance_ between
$a_{1}$ and $a_{2}$ is

$d(a_{1},a_{2})\coloneqq\left\lvert\left\\{\,(i,j)\in{[n]}^{2}\;\middle|\;(a_{1})_{ij}\neq(a_{2})_{ij}\,\right\\}\right\rvert.$

We can use these distances to create a combinatorial graph whose vertices are
binary images.

######  Definition 18 (Hamming graph).

Given $n\in\mathbb{N}$ we define the _$n$ -Hamming graph_ to be

$\operatorname{\mathbf{Ham}}_{n}\coloneqq(A_{n},\left\\{\,(a_{1},a_{2})\in
A_{n}^{2}\;\middle|\;d(a_{1},a_{2})\leq 1\,\right\\}).$

That is, given binary images $a_{1},a_{2}\in A_{n}$ we say that $a_{1}\sim
a_{2}$ in $\operatorname{\mathbf{Ham}}_{n}$ when either $a_{1}=a_{2}$ or
$a_{1}$ and $a_{2}$ have the same values at all pixels except one. Thus,
$a_{1}\sim a_{2}$ when the binary images $a_{1}$ and $a_{2}$ differ in at most
one pixel.

Of course there are many possible variations on the definition of a Hamming
graph given here. Note that if we had required that $d(a_{1},a_{2})=1$ then we
would have the $n^{2}$-cube graph instead. It follows that
$\operatorname{\mathbf{Ham}}_{n}$ is isomorphic to the graph whose nodes are
the vertices of the $n^{2}$-dimensional cube and whose edges are those of the
cube along with a single loop at each vertex.

The loops at each vertex will be useful to us in what follows. In order to
apply the ideas of example 8 in the context where the relational structure in
question is $\operatorname{\mathbf{Ham}}_{n}$ we must find a set of
endomorphisms
$H_{n}\subset\operatorname{End}(\operatorname{\mathbf{Ham}}_{n})$ and a set of
higher-arity polymorphisms
$G_{n}\subset\operatorname{Poly}(\operatorname{\mathbf{Ham}}_{n})$.

###  6.1. Endomorphisms of the Hamming graph

We give several families of endomorphisms of the Hamming graph
$\operatorname{\mathbf{Ham}}_{n}$.

######  Definition 19 (Dihedral endomorphism).

Identify $\mathbf{D}_{4}$ with the group of isometries of the plane generated
by

$\begin{bmatrix}0&-1\\\ 1&0\end{bmatrix}\text{ and }\begin{bmatrix}1&0\\\
0&-1\end{bmatrix},$

and define a map $\gamma_{n}\colon[n]^{2}\to U_{n}$ where

$U_{n}\coloneqq\left(\left[-\left\lfloor\frac{n}{2}\right\rfloor,\left\lfloor\frac{n}{2}\right\rfloor\right]\cap\mathbb{Z}\right)^{2}$

when $n$ is odd and

$U_{n}\coloneqq\left\\{\,(x,y)\in U_{n+1}\;\middle|\;x\neq 0\text{ and }y\neq
0\,\right\\}$

when $n$ is even by setting

$\gamma_{n}(i,j)\coloneqq\left(-\left\lfloor\frac{n}{2}\right\rfloor+j,\left\lfloor\frac{n}{2}\right\rfloor-i\right)$

when $n$ is odd and setting

$\gamma_{n}(i,j)\coloneqq\begin{cases}\left(-\lfloor\frac{n}{2}\rfloor+j,\lfloor\frac{n}{2}\rfloor-i\right)&\text{when
}i,j<\frac{n}{2}\\\
\left(-\lfloor\frac{n}{2}\rfloor+j,\lfloor\frac{n}{2}\rfloor-i-1\right)&\text{when
}i\geq\frac{n}{2}\text{ and }j<\frac{n}{2}\\\
\left(-\lfloor\frac{n}{2}\rfloor+j+1,\lfloor\frac{n}{2}\rfloor-i\right)&\text{when
}i<\frac{n}{2}\text{ and }j\geq\frac{n}{2}\\\
\left(-\lfloor\frac{n}{2}\rfloor+j+1,\lfloor\frac{n}{2}\rfloor-i-1\right)&\text{when
}i,j\geq\frac{n}{2}\end{cases}$

when $n$ is even. Given $\sigma\in D_{4}$ define the _dihedral endomorphism_
$h_{\sigma}\colon A_{n}\to A_{n}$ by

$(h_{\sigma}(a))_{ij}\coloneqq a_{\gamma_{n}^{-1}\sigma\gamma_{n}(i,j)}.$

Observe that these dihedral endomorphisms are actually automorphisms. We write
$\operatorname{Dihed}_{n}$ to indicate the set of dihedral endomorphisms
$\operatorname{\mathbf{Ham}}_{n}$. The class of dihedral automorphisms belongs
to the larger class of automorphisms obtained by permuting the pixels of an
image according to an arbitrary permutation of $[n]^{2}$, but we will restrict
ourselves to $\operatorname{Dihed}_{n}$ as these automorphisms are easier to
store and compute with than a general permutation.

Another class of automorphisms are given by adding a fixed binary image
pointwise.

######  Definition 20 (Swapping endomorphism).

Given $b\in A_{n}$ the _swapping endomorphism_ for $b$ is the map
$h_{b}^{+}\colon A_{n}\to A_{n}$ which is given by $h_{b}^{+}(a)\coloneqq a+b$
where the sum $a+b$ is the usual componentwise sum of matrices over
$\mathbb{F}_{2}$.

Note that swapping endomorphisms are also automorphisms. We write
$\operatorname{Swap}_{n}$ to indicate the set of swapping endomorphisms of
$\operatorname{\mathbf{Ham}}_{n}$.

Finally, we have endomorphisms which are not automorphisms. These are obtained
by taking the Hadamard product with a fixed binary image.

######  Definition 21 (Blanking endomorphisms).

Given $b\in A_{n}$ the _blanking endomorphism_ for $b$ is the map
$h_{b}^{\odot}\colon A_{n}\to A_{n}$ which is given by
$h_{b}^{\odot}(a)\coloneqq a\odot b$ where $a\odot b$ is the Hadamard product
of matrices over $\mathbb{F}_{2}$.

We denote by $\operatorname{Blank}_{n}$ the set of all blanking endomorphisms
of $\operatorname{\mathbf{Ham}}_{n}$.

By setting

$H_{n}\coloneqq\operatorname{Dihed}_{n}\cup\operatorname{Swap}_{n}\cup\operatorname{Blank}_{n}$

we take our given set of endomorphisms to consist of all those from the three
above-defined classes.

###  6.2. Polymorphisms of the Hamming graph

In this subsection we will make use of a notion related to that of Hamming
distance.

######  Definition 22 (Hamming weight).

Given a binary image $a\in A_{n}$ the _Hamming weight_ of $a$ is

$\left\lVert a\right\rVert\coloneqq d(a,0)$

where $0$ is the image whose pixels are all $0$.

Thus, $\left\lVert a\right\rVert$ is the number times $1$ appears as an entry
of $a$. The following set will be useful.

######  Definition 23 (Standard basis).

The _standard basis_ of the space of size $n$ binary images is

$B_{n}\coloneqq\left\\{\,a\in A_{n}\;\middle|\;\left\lVert
a\right\rVert=1\,\right\\}.$

This is to say that $B_{n}$ consists of all images where exactly one pixel is
given the value $1$. Note that $a_{1}\sim a_{2}$ in
$\operatorname{\mathbf{Ham}}_{n}$ if and only if $a_{1}+a_{2}\in
B_{n}\cup\left\\{\,\;\middle|\;0\,\right\\}$.

Any function $h\colon A_{n}^{k}\to A_{n}$ with
$\operatorname{Im}(h)=\left\\{\,\;\middle|\;a_{1},a_{2}\,\right\\}$ with
$a_{1}\sim a_{2}$ is a polymorphism. These are unwieldy to work with and store
in general so we restrict our attention to a special class of such
polymorphisms.

######  Definition 24 (Multi-linear indicator).

Given $b\in B_{n}$ and $c\in A_{n}^{k}$ the _multi-linear indicator
polymorphism_ for $(b,c)$ is the map $g_{b,c}\colon A_{n}^{k}\to A_{n}$ given
by

$g_{b,c}(a_{1},\dots,a_{k})\coloneqq\left(\prod_{i=1}^{k}a_{i}\cdot
c_{i}\right)b$

where $x\cdot y\coloneqq\sum_{i,j}x_{ij}y_{ij}$ denotes the standard dot
product in $\mathbb{F}_{2}^{[n]^{2}}$.

We denote the class of multi-linear indicator polymorphisms of
$\operatorname{\mathbf{Ham}}_{n}$ by $\operatorname{MultInd}_{n}$.

Our final, most interesting, class of polymorphisms has members whose images
are not generally a pair of adjacent vertices in the Hamming graph. In order
to define these we need to set up some machinery.

######  Definition 25 (Hamming weight map).

The _$k$ -ary Hamming weight map_

$\psi_{k}\colon A_{n}^{k}\to[n^{2}+1]^{k}$

is given by

$\psi_{k}(a_{1},\dots,a_{k})\coloneqq(\left\lVert
a_{1}\right\rVert,\dots,\left\lVert a_{k}\right\rVert).$

This map induces a relation on the members of $A_{n}^{k}$, which is
$\ker(\psi_{k})\subset(A_{n}^{k})^{2}$. We consider the quotient of
$\operatorname{\mathbf{Ham}}_{n}^{k}$ induced by this relation.

######  Definition 26 (Hamming weight graph).

We refer to the graph $\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi_{k})$ as
the _$k$ -ary Hamming weight graph of size $n$_.

Observe that $\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi_{k})$ is
isomorphic to the graph whose vertex set is $[n^{2}+1]^{k}$ and whose edges
are between $k$-tuples $(u_{1},\dots,u_{k})$ and $(v_{1},\dots,v_{k})$ such
that for all $i$ we have $\lvert u_{i}-v_{i}\rvert\leq 1$. We will suppress
this isomorphism in what follows. We denote the canonical homomorphism of
graphs from $\operatorname{\mathbf{Ham}}_{n}^{k}$ to the quotient
$\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi)$ by $\tilde{\psi}$.

Our strategy for finding polymorphisms of $\operatorname{\mathbf{Ham}}_{n}$ is
as follows. We will give a procedure for finding graph homomorphisms
$f\colon\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi_{k})\to\operatorname{\mathbf{Ham}}_{n}$.
Given such a homomorphism we will obtain a polymorphism
$f\circ\tilde{\psi}\colon\operatorname{\mathbf{Ham}}_{n}^{k}\to\operatorname{\mathbf{Ham}}_{n}$
by composing with the quotient map $\tilde{\psi}$. Since the Hamming weight
graph is simpler to understand than powers of this original Hamming graph this
will simplify our search.

In order to define such homomorphisms
$f\colon\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi_{k})\to\operatorname{\mathbf{Ham}}_{n}$
we introduce the following combinatorial objects.

######  Definition 27.

Basic cube A _basic cube_ $C_{u}$ of the $k$-ary Hamming weight graph of size
$n$ is a set of vertices

$\left\\{\,(v_{1},\dots,v_{k})\in{[n^{2}+1]}^{k}\;\middle|\;(\forall
i)(v_{i}-u_{i}\in\left\\{\,\;\middle|\;0,1\,\right\\})\,\right\\}$

where $u=(u_{1},\dots,u_{k})\in[n^{2}]^{k}$ is called the _top corner_ of the
basic cube.

We will denote by $\mathbf{K}_{L}$ the complete graph on the vertex set $L$
with a loop at each vertex. Our (slightly nonstandard) definition of an _$L$
-coloring_ of a graph $\mathbf{G}$ is then a homomorphism from $\mathbf{G}$ to
$\mathbf{K}_{L}$. Note that under this weakened definition any function from
the vertex set of $\mathbf{G}$ to $L$ would constitute an $L$-coloring of
$\mathbf{G}$.

######  Definition 28 (Dominion).

Given a set of labels $L$ a _$(k,n,L)$ -dominion_ is an $L$-coloring of the
$k$-ary Hamming weight graph of size $n$ such that given any vertex
$u\in[n^{2}]^{k}$ the basic cube $C_{u}$ is colored using at most $2$ members
of $L$.

We will often refer to a $(k,n,L)$-dominion as the corresponding function

$D\colon\operatorname{\mathbf{Ham}}_{n}^{k}/\ker(\psi_{k})\to L.$

There is a graph whose vertices are the labels $L$ induced by such a coloring.

######  Definition 29 (Minimum constraint graph).

Given a $(k,n,L)$-dominion $D$ the _minimum constraint graph_
$\operatorname{\mathbf{MinC}}(D)$ is the simple graph whose vertices are $L$
and whose adjacency relation is given by $\ell_{1}\sim\ell_{2}$ when there is
a basic cube $C_{u}$ of $\operatorname{\mathbf{Ham}}_{n}/\ker(\psi_{k})$ such
that the labels $\ell_{1}$ and $\ell_{2}$ are both used to color at least one
vertex from $C_{u}$.

Note that the minimum constraint graph of a $(k,n,L)$-dominion $D$ is the
subgraph of $\mathbf{K}_{L}$ whose edges consist only of those joining pairs
of distinct vertices in $\operatorname{Im}(D)$.

Homomorphisms from the minimum constraint graph of a dominion to
$\operatorname{\mathbf{Ham}}_{n}$ yield polymorphisms of
$\operatorname{\mathbf{Ham}}_{n}$.

######  Definition 30 (Dominion polymorphism).

Given a $(k,n,L)$-dominion $D$ and a homomorphism
$\alpha\colon\operatorname{\mathbf{MinC}}(D)\to\operatorname{\mathbf{Ham}}_{n}$
the _dominion polymorphism_
$g_{\alpha}\colon\operatorname{\mathbf{Ham}}_{n}^{k}\to\operatorname{\mathbf{Ham}}_{n}$
is given by

$g_{\alpha}(a_{1},\dots,a_{k})\coloneqq\alpha(D(\psi_{k}(a))).$

Given a typical dominion $D$ it may be challenging to find homomorphisms
$\alpha\colon\operatorname{\mathbf{MinC}}(D)\to\operatorname{\mathbf{Ham}}_{n}$.
If we assume that our minimum constraint graph is a subgraph of a tree we can
find such homomorphisms more easily.

Given a $(k,n,L)$-dominion $D$ and any graph $\mathbf{L}$ which contains
$\operatorname{\mathbf{MinC}}(D)$ as a subgraph, we know that there is an
inclusion homomorphism
$\iota\colon\operatorname{\mathbf{MinC}}(D)\hookrightarrow\mathbf{L}$. It
follows that if we can find a homomorphism
$\alpha^{\prime}\colon\mathbf{L}\to\operatorname{\mathbf{Ham}}_{n}$ then we
can take

$\alpha\coloneqq\alpha^{\prime}\circ\iota\colon\operatorname{\mathbf{MinC}}(D)\to\operatorname{\mathbf{Ham}}_{n}$

as our homomorphism for creating a dominion polymorphism

$g_{\alpha}\colon\operatorname{\mathbf{Ham}}_{n}^{k}\to\operatorname{\mathbf{Ham}}_{n}.$

Code for this example can be found on
GitHub111https://github.com/caten2/Tripods2021UA. At the time of this writing
this repo is still under active development. As one can see in the version of
the code from this date, we have tested this polymorphic learning algorithm
with the MNIST dataset available from TensorFlow. Nothing else from TensorFlow
was used in our testing and we will remove this dependency soon.

We found that neural nets equipped with polymorphisms of the Hamming graph as
activation functions would converge quickly to a minimum possible empirical
loss. It is important to note that this final empirical loss was not $0$, as
such overfitting is made impossible by our choice of activation function.

## References

  * [1] Clifford Bergman  “Universal algebra” Fundamentals and selected topics 301, Pure and Applied Mathematics (Boca Raton)  CRC Press, Boca Raton, FL, 2012, pp. xii+308
  * [2] Andrei A. Bulatov  “A dichotomy theorem for nonuniform CSPs”  In _58th Annual IEEE Symposium on Foundations of Computer Science—FOCS 2017_ IEEE Computer Soc., Los Alamitos, CA, 2017, pp. 319–330
  * [3] David M. Clark  “Evolution of algebraic terms 1: Term to term operation continuity”  In _Internat. J. Algebra Comput._ 23.5, 2013, pp. 1175–1205
  * [4] G. Cybenko  “Approximation by superpositions of a sigmoidal function”  In _Math. Control Signals Systems_ 2.4, 1989, pp. 303–314
  * [5] Rachel Dennis  “Polymorphisms and Neural Networks for Image Classification”, 2023  URL: https://www.sas.rochester.edu/mth/undergraduate/honorspaperspdfs/r_dennis23.pdf
  * [6] Peter Jeavons  “On the algebraic structure of combinatorial problems”  In _Theoret. Comput. Sci._ 200.1-2, 1998, pp. 185–204
  * [7] Lek-Heng Lim and Bradley J. Nelson  “What is an equivariant neural network?”  In _arXiv e-prints_ , 2022, pp. arXiv:2205.07362  DOI: 10.48550/arXiv.2205.07362
  * [8] Shai Shalev-Shwartz and Shai Ben-David  “Understanding Machine Learning: From Theory to Algorithms”  32 Avenue of the Americas, New York, NY 10013-2473, USA: Cambridge University Press, 2014