# Fine-Tuning Surrogate Gradient Learning for Optimal Hardware Performance in
Spiking Neural Networks

Ilkin Aliyev and Tosiron Adegbija
Department of Electrical and Computer Engineering
The University of Arizona, Tucson, AZ, USA
Email: {ilkina<EMAIL_ADDRESS>
###### Abstract

The highly sparse activations in Spiking Neural Networks (SNNs) can provide
tremendous energy efficiency benefits when carefully exploited in hardware.
The behavior of sparsity in SNNs is uniquely shaped by the dataset and
training hyperparameters. This work reveals novel insights into the impacts of
training on hardware performance. Specifically, we explore the trade-offs
between model accuracy and hardware efficiency. We focus on three key
hyperparameters: surrogate gradient functions, beta, and membrane threshold.
Results on an FPGA-based hardware platform show that the fast sigmoid
surrogate function yields a lower firing rate with similar accuracy compared
to the arctangent surrogate on the SVHN dataset. Furthermore, by cross-
sweeping the beta and membrane threshold hyperparameters, we can achieve a
$48\%$ reduction in hardware-based inference latency with only $2.88\%$ trade-
off in inference accuracy compared to the default setting. Overall, this study
highlights the importance of fine-tuning model hyperparameters as crucial for
designing efficient SNN hardware accelerators, evidenced by the fine-tuned
model achieving a $1.72\times$ improvement in accelerator efficiency (FPS/W)
compared to the most recent work.

###### Index Terms:

Surrogate Gradient Learning, Sparsity-aware SNN, Neuromorphic Computing.

##  I Introduction

Recent studies have demonstrated significant benefits to considering sparsity
in improving hardware efficiency in Spiking Neural Networks (SNNs) [1, 2]. For
example, Yin et al. [1] showed that by explicitly exploiting the sparse
gradients (as high as $93\%$ for some datasets) in hardware, training SNNs can
consume up to $5.58\times$ less energy compared to training on the same
hardware without considering sparsity. Similarly, Wang et al. [2] achieved a
$2.1\times$ improvement in inference efficiency by exploiting sparsity in
hardware compared to sparsity-oblivious hardware. The primary driving factor
in the formation of the sparsity characteristic is the input coding scheme of
the dataset. Recognizing that this is an active area of research and various
encoding approaches are being proposed to reduce the firing rate of the
network, in this study, we explore, for the first time, how training
hyperparameters can influence the sparsity of SNN models and, in effect, the
hardware performance of SNN accelerators.

Surrogate gradients [3] are typically employed to train SNN models due to
their ability to overcome the non-linear nature of spiking neurons (i.e.,
binary activations/spikes instead of linear continuous activations). The
surrogate gradient approximates the true derivative, thereby effectively
providing state-of-the-art accuracy in classification-oriented machine
learning (ML) tasks. Nonetheless, the choice of a surrogate function (or the
scaling factor of its derivative) is not standardized. This raises the
question of how its implementation affects the network’s classification
accuracy and firing intensity.

To investigate our hypothesis, we systematically study two well-known
surrogate functions: arctangent and fast sigmoid, empirically evaluating each
function’s optimal trade-off points for learning performance versus sparsity
under a range of derivative factors. We also rigorously evaluate the impact of
two critical hyperparameters: beta ($\beta$) and threshold ($\theta$). We
leverage our in-house hardware platform to conduct these hardware experiments.
This platform allows efficient “model-to-hardware” mapping by accounting for
the model’s layer-wise workload characteristics. To the best of our knowledge,
this work is the first to present a holistic evaluation of SNN hardware from a
training perspective.

##  II Background

###  II-A Spiking Neuron Model

The spiking neuron model used is a leaky integrate-and-fire (LIF) neuron,
whose characteristics are shown in Equations 1 and 2.

$u_{j}[t+1]=\beta u_{j}[t]+\sum_{i}w_{ij}s_{i}[t]-s_{j}[t]\theta$ (1)

$s_{j}[t]=\begin{cases}1,&\text{if }u_{j}[t]>\theta\\\
0,&\text{otherwise}\end{cases}$ (2)

Here, $\beta$ represents the decay or leak factor of the neuron’s membrane
potential, typically ranging between 0 and 1. This parameter influences how
the previous potential $u_{j}[t]$ affects the current potential $u_{j}[t+1]$.
A higher $\beta$ value implies less decay, enabling the neuron to retain more
of its previous state, which can increase the likelihood of firing. $\theta$,
on the other hand, represents the threshold value that the neuron’s membrane
potential must surpass to trigger a firing event, as indicated by a spike
$s_{j}[t]$. A lower $\theta$ value reduces the potential required for firing,
thereby increasing the neuron’s firing frequency.

###  II-B Surrogate Approximation Functions

Surrogate gradients have emerged as a competitive solution to effectively
approximate the step function [3]. Equations 4 and 3 show the approximation
formulas used for arctangent and fast sigmoid, respectively.

$S\approx\frac{1}{\pi}\arctan\left(\pi U\frac{\alpha}{2}\right)$ (3)

$S\approx\frac{U}{1+k|U|}$ (4)

where $U$ is the membrane potential and $\alpha$ and $k$ are derivative
scaling factors for each surrogate gradients.

##  III Surrogate Gradient Fine-Tuning and Hyperparameter Search Results

###  III-A Experimental Setup

We used snnTorch [4] to construct the spiking neuron models and PyTorch for
training the network using the Street View House Numbers (SVHN) dataset. The
network is a convolutional SNN with the following structure:
32C3-P2-32C3-MP2-256-10 (where $X$C$Y$ stands for $X$ filters with size
$Y\times Y$ and MP$Z$ for maxpooling with size of $Z\times Z$). We employ
cosine annealing [5] for the learning rate scheduler (with epochs set to 25)
during training due to its ability to rapidly converge to optimal accuracy.
The trained model was mapped to an in-house hardware platform111Publicly
available at https://github.com/githubofaliyev/SNN-DSE, developed in
SystemVerilog, and implemented on a Xilinx Kintex® UltraScale+™ FPGA. This
hardware efficiently allocates platform resources for the model by leveraging
the model’s layer sizes and layer-wise sparsity characteristics to achieve an
ultra-low power resource allocation scheme. In addition, the hardware operates
in a layer-wise lock-step manner to save memory resources and achieve high
throughput.

Figure 1: Cross-comparison results for arctangent and fast sigmoid surrogate
functions over varying derivative scaling factors

###  III-B Results

Surrogate functions: To systematically evaluate each surrogate function, we
performed a parameter sweep over the derivative scaling factors $k$ and
$\alpha$ while leaving both $\beta$ and $\theta$ set to their default values
($0.25$ and $1.0$ respectively). Figure 1 shows the variation in accuracy and
accelerator efficiency (FPS/W) for each surrogate gradient. We set both $k$
and $\alpha$to the value range of $0.5$ to $32$, beyond which the accuracy for
the arctangent surrogate drops below $20\%$. We observe that while both
surrogate gradients follow a similar trend in accuracy and efficiency, the
fast sigmoid yields lower firing activity (i.e., higher sparsity) compared to
the arctangent, resulting in higher accelerator efficiency. Moreover, in both
surrogate gradients, through hyperparameter tuning, our network yields higher
accuracy than previous work [6] (as highlighted by the horizontal green line
in Figure 1), using the same network architecture and dataset, while the fast
sigmoid achieves $11\%$ better accelerator efficiency.

Beta-threshold cross sweep: Given its high sparsity, we chose the fast sigmoid
surrogate with a slope scaling factor of $0.25$ for the following experiments.
In Figure 2, we cross-sweep $\beta$ (leakage factor) and $\theta$ (threshold
potential). Our analysis identifies the optimal balance at a $\beta$ value of
$0.5$ and a $\theta$ value of $1.5$. This configuration significantly reduced
the inference latency by $48\%$, while only incurring a minor accuracy loss of
$2.88\%$, compared to the best accuracy configuration. Compared to prior work
[6], this fine-tuning—with $\beta$ set to $0.7$ and $\theta$ to $1.5$—
achieved $1.72\times$ greater hardware efficiency without degrading the
accuracy.

Figure 2: Cross-sweep results for $\beta$ and $\theta$ parameters.

##  IV Conclusion

This study sheds new light on a previously unexplored aspect of SNN hardware
accelerator design. While previous research has primarily focused on non-
hardware-related factors like dataset encoding, our work pioneers the
evaluation of the training hyperparameter space in relation to hardware
efficiency. We show that fine-tuning surrogate gradient hyperparameters can
provide significant benefits to hardware efficiency and should be considered
carefully in the design of efficient SNN accelerators. In future work, we aim
to broaden our analysis by exploring additional datasets and the hardware
efficiency impacts of other hyperparameters like loss functions.

## References

  * [1] R. Yin, A. Moitra, A. Bhattacharjee, Y. Kim, and P. Panda, “Sata: Sparsity-aware training accelerator for spiking neural networks,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022.
  * [2] Z. Wang, Y. Zhong, X. Cui, Y. Kuang, and Y. Wang, “A spiking neural network accelerator based on ping-pong architecture with sparse spike and weight,” in 2023 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, IEEE, 2023.
  * [3] E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019.
  * [4] J. K. Eshraghian, M. Ward, E. O. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu, “Training spiking neural networks using lessons from deep learning,” Proceedings of the IEEE, 2023.
  * [5] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983, 2016.
  * [6] W. Ye, Y. Chen, and Y. Liu, “The implementation and optimization of neuromorphic hardware for supporting spiking neural networks with mlp and cnn topologies,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 42, no. 2, pp. 448–461, 2022.


# Narrowing the Gap between Combinatorial and Hyperbolic Knot Invariants via
Deep Learning

Daniel Grünbaum  Fakultät für Mathematik, Universität Regensburg, 93040
Regensburg<EMAIL_ADDRESS>
###### Abstract.

We present a statistical approach for the discovery of relationships between
mathematical entities that is based on linear regression and deep learning
with fully connected artificial neural networks. The strategy is applied to
computational knot data and empirical connections between combinatorial and
hyperbolic knot invariants are revealed.

##  1\. Introduction

A central goal in mathematical research is the discovery of theoretical
relationships between different entities. After a relationship is noticed,
mathematicians try to formalize all involved concepts by definitions, the
exact nature of the relationship by theorems and the precise reasoning for the
existence of the relationship by proofs. The first step of finding possible
connections that warrant further investigation, however, is often left to a
mixture of the researcher’s intuition and domain knowledge.

In disciplines other than mathematics, for example in biology or in the social
sciences, a routinely performed exploratory step is gathering data for
different entities and plotting them against each other. In the case of an
existing relationship, a visual inspection may reveal a non-random pattern to
the researcher and the strength of the association can subsequently be made
more precise through statistical evaluations. For a mathematician following
this approach, several obstacles are bound to occur: Firstly, data collection
is traditionally not a prominent part of pure mathematics. Many entities are
hard to calculate by hand and computational aid in the form of specialized
software is not always available. Secondly, mathematical objects can be of
infinite or at least prohibitively high dimension, rendering a visualization
impossible. Thirdly, a quantitative description of the strength of a
relationship through statistical methods does not constitute a formal theorem,
let alone a proof that shows the exact nature of the connection in rigorous
mathematical terms. The goal of this article is to argue that a data-driven
exploratory step can still be a valuable tool for knot theorists. With respect
to the first of the above obstacles, scarcity of data, the field of knot
theory fares better than most other areas of mathematics due to an active
community of computational knot theorists that provide databases containing
many knot invariants or suitable tools for calculating them [1, 2, 3, 4].
Given the amount of data that can be generated, data-hungry algorithms of
high-dimensional statistical learning, with deep learning as its most
prominent subfield [5], become viable options for replacing the visual
inspection. The last objection still holds, but the presented investigation of
different knot invariants will show that methods of data analysis can at least
provide concrete research questions, which can then be addressed by more
traditional mathematical methods.

This article builds on the work of other researchers. A strong correlation
between the determinant and the hyperbolic volume of a knot has been
established by Dunfield through visual inspection in an unpublished note [6].
The subject was revisited by Friedl and Jackson [7], who provided a
quantitative analysis of the relationship between the hyperbolic volume and
several invariants derived from the Alexander polynomial, employing linear
regression as a statistical method. Deep learning has been introduced to knot
theory only recently: Jejjala, Kar and Parrikar [8] reported that a standard
version of an artificial neural network could predict the hyperbolic volume of
a knot with high precision from the coefficients of its Jones polynomial. They
suggested incorporating invariants from Khovanov homology for a possible
improvement of the predictive power. Following up on this discovery, Craven,
Jejjala and Kar [9] used layer-wise relevance propagation to further analyze
and simplify the complex neural network. They were able to distill a
performant approximate formula for the hyperbolic volume that inputs only a
single evaluation of the Jones polynomial at a root of unity. Hughes [10]
successfully used deep learning techniques to predict from a braid word
representation whether a given knot is quasipositive. Joining forces, Craven,
Hughes, Jejjala and Kar [11] used a similar methodology to unveil empirical
connections between the Jones polynomial, Khovanov homology, smooth slice
genus and Rasmussen’s $s$-invariant. In an independent stream of research,
Davies et al. [12] combined deep learning with attribution techniques to
detect a simple relationship between the knot signature, hyperbolic volume,
meridian and longitude translation. The empirical result lead to the
formulation of a conjecture that could be proven rigorously by adding the
injectivity radius as a last ingredient [13].

In this article, we will use techniques from deep learning as an exploratory
tool to gather evidence for the existence of theoretical relationships between
combinatorial and hyperbolic knot invariants. The experimental evidence will
by no means obliterate the necessity for a formal mathematical treatment of
the underlying abstract causes. On the contrary, it indicates only a starting
point for mathematical research of a more traditional nature.

The article is structured as follows: Section 2 presents linear regression and
deep learning as empirical methods of scientific discovery. Section 3 briefly
introduces combinatorial and hyperbolic knot invariants. The conducted
experiments and their results are documented and discussed in Section 4.
Section 5 uses the discovered empirical relationships to formulate a list of
questions for subsequent theoretical research.

##  2\. Statistical Methods

This section explains basic concepts of linear regression and deep learning,
before arguing how prediction errors of artificial neural networks can be
interpreted as a more evolved version of visual inspection methods and
correlation coefficients.

###  2.1. Linear Regression

If we want to investigate the existence of a relationship between two
$1$-dimensional entities, the easiest way is to plot them against each other
and inspect the resulting data cloud visually. An example would be the
aforementioned plotting of the hyperbolic volume against the knot determinant
for different classes of knots. If a relationship exists, the data shows a
systematic pattern that can be recognized by the human eye. In several fields
of science, such as biology or the social sciences, it is customary to assess
the strength of a conjectured linear relationship quantitatively via the
Pearson correlation coefficient.

######  Definition.

Let $x=(x_{i})_{i=1,...,n}$ be a finite family of real numbers.

  * •

The _sample mean_ of $x$ is given by $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i}$.

  * •

The _sample variance_ of $x$ is given by
$\sigma^{2}_{x}=\frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}$.

  * •

The _sample standard deviation_ of $x$ is given by
$\sigma_{x}=\sqrt{\sigma^{2}_{x}}$.

  * •

Let $y=(y_{i})_{i=1,...,n}$ be another finite family of real numbers. The
_sample covariance_ of $x$ and $y$ is given by
$\operatorname{Cov}\left({x,y}\right)=\frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})$.

  * •

For $\sigma_{x},\sigma_{y}\neq 0$, the _Pearson correlation coefficient_ of
$x$ and $y$ is given by

(1)
$r_{x,y}=\frac{\operatorname{Cov}\left({x,y}\right)}{\sigma_{x}\sigma_{y}}.$

Note that we have $r_{x,y}\in[-1,1]$ and $r_{x,y}\in\\{-1,1\\}$ if and only if
$y$ is an affine function of $x$. Visually, $|r_{x,y}|$ describes how well the
data points $(x_{i},y_{i})$ fit on a straight line. A related way of
quantifying the strength of the association between $x$ and $y$ is _linear
regression_ , i.e. fitting the optimal linear approximation

(2) $y\approx f(x)=a_{*}x+b_{*}$

to the data according to

(3)
$(a_{*},b_{*})=\operatorname*{arg\,min}_{(a,b)\in\mathbb{R}^{2}}\frac{1}{n}\sum_{i=1}^{n}\|y_{i}-(ax_{i}+b)\|^{2}$

and using the expression in the minimum, which is called the _mean squared
error_ $\Delta_{\text{MSE}}$, as a figure of merit. An additional relative
error measurement is given by the _mean absolute percentage error_

(4)
$\Delta_{\text{MAPE}}(x,y,a_{*},b_{*})=\frac{1}{n}\sum_{i=1}^{n}\frac{|y_{i}-(a_{*}x_{i}+b_{*})|}{|y_{i}|}$

if $y$ has only non-zero entries. Two main adjustments to this procedure are
possible:

  * •

If the visual data inspection shows a non-linear pattern, we can transform $x$
by applying a suitable function $t\colon\mathbb{R}\to\mathbb{R}$ in such a way
that the relationship between $t(x)$ and $y$ is linear. After the
transformation, the same measures of dependence as above can still be used for
an evaluation. A common example for this technique is the use of logarithmic
scaling to quantify the strength of exponential relationships.

  * •

If $x\in\mathbb{R}^{m}$ is a multidimensional entity and we suspect a linear
relationship, we can simply fit a multilinear model $y\approx\beta\cdot x+b$
for $\beta\in\mathbb{R}^{m}$ that minimizes the mean squared error as above.
Note that the ideal coefficients can be found analytically without any
numerical optimization techniques.111Extensions to the case where $y$ is also
multidimensional can be formulated in a similar way, but it is not necessary
for our purposes.

###  2.2. Deep Learning

The above adjustments are not sufficient if we want to investigate
relationships that are non-linear and involve high-dimensional data at the
same time. Since we are looking for the existence of any, possibly non-linear,
relationship between $x$ and $y$, it is not sufficient to calculate the
Pearson correlation between $x$ and $y$ and give up if the absolute value is
low. Contrary to the $1$-dimensional case, we cannot simply plot the data and
identify an ideal transformation $T\colon\mathbb{R}^{m}\to\mathbb{R}^{k}$ for
some $k\in\mathbb{N}$ that ensures a linear relationship between $T(x)$ and
$y$. Deep learning circumvents this problem by choosing a wide family of
possible parameterized transformations and optimizing not only the parameters
of the multilinear model but also those of the transformations. If the family
of allowed transformations provides enough expressive power and a suitable
optimization strategy is chosen, we can be confident that any possible
functional relationship between $x$ and $y$ will be detected [14]. A common
and well-studied form of deep learning is via fully connected artificial
neural networks, which we will define after a short notational digression.

######  Definition.

Let $k,l\in\mathbb{N}$. For a map $\phi\colon\mathbb{R}^{k}\to\mathbb{R}^{l}$
and a function $\psi\colon\mathbb{R}\to\mathbb{R}$, we define the
_componentwise composition of $\phi$ and $\psi$_ to be the map

(5) $\begin{split}\psi\odot\phi\colon\mathbb{R}^{k}&\to\mathbb{R}^{l}\\\
x&\mapsto\left(\psi(\phi(x)_{1}),...,\psi(\phi(x)_{l})\right).\end{split}$

The componentwise composition allows us to formulate the definition of an
artificial neural network in a compact way:

######  Definition.

Let $m,p\in\mathbb{N}$ and let $(k_{0},...,k_{p+1})$ be a family of natural
numbers with $k_{0}=m$ and $k_{p+1}=1$.

  * •

A _fully connected artificial neural network (ANN) with $p$ hidden layers_ is
a function $f\colon\mathbb{R}^{m}\to\mathbb{R}$ that can be written as a
composition

(6)
$f=\phi_{p}\circ(\psi_{p-1}\odot\phi_{p-1})\circ...\circ(\psi_{1}\odot\phi_{1})\circ(\psi_{0}\odot\phi_{0})$

for a family of affine maps

(7)
$\left(\phi_{t}\colon\mathbb{R}^{k_{t}}\to\mathbb{R}^{k_{t+1}}\right)_{t\in\\{0,...,p\\}}$

and a family of functions

(8)
$\left(\psi_{t}\colon\mathbb{R}\to\mathbb{R}\right)_{t\in\\{0,...,p-1\\}}.$

For $t=1,...,p$, we call $k_{t}$ the _number of hidden neurons in the $t$-th
layer_.

  * •

For each $t\in\\{0,...,p\\}$, there is a pair $(W_{t},b_{t})$ of a
$(k_{t+1}\times k_{t})$-dimensional matrix $W_{t}$ with real entries and a
vector $b_{t}\in\mathbb{R}^{k_{t+1}}$ such that we have

(9) $\phi_{t}(x)=W_{t}x+b_{t}$

for all $x\in\mathbb{R}^{k_{t}}$. We call $W_{t}$ the _$t$ -th weight matrix_
and $b_{t}$ the _$t$ -th bias vector_. The entries of the weight matrices are
called _weights_.

  * •

For each $t\in\\{0,...,p-1\\}$, we call $\psi_{t}$ the _$t$ -th activation
function_.

As in the case of a multilinear model, we can view the entries of the weight
matrices and bias vectors as free model parameters of the artificial neural
network and obtain a parameterized model of the form

(10)
$\mathbb{R}^{m}\times\mathbb{R}^{(k_{0}k_{1})+(k_{1}k_{2})+...+(k_{p}k_{p+1})}\times\mathbb{R}^{k_{1}+...+k_{p+1}}\to\mathbb{R}$

whose mean squared error on the data can be minimized with respect to the
model parameters. Finding the optimal pair $(W_{p},b_{p})$ for the last
transformation $\phi_{p}$ is precisely a multilinear regression. Since we
optimize not only the weights for the regression function, but also the
weights in the previous matrices, the goal of optimizing the parameters of an
ANN can be formulated as follows: We want to find a transformation
$T\colon\mathbb{R}^{m}\to\mathbb{R}^{k_{p}}$ that is optimal for a prediction
of the target variable $y$ by multilinear regression and that can be obtained
by the application of affine transformations and activation functions.

The activation functions enable non-linear transformations. Historically,
_sigmoidal_ activation functions, such as the _hyperbolic tangent_
$\psi(z)=\tanh(z)$ and the _logistic function_ $\psi(z)=\frac{1}{1+e^{-z}}$
have been popular [15]. Named after the characteristic S-shape of their
graphs, these functions are monotonic and bounded. Their use is justified from
a biological viewpoint where neurons in the artificial neural network
correspond to actual neurons of the human brain. Monotonicity of the
activation function mirrors a type of biological neurons that fire electrical
pulses with a higher frequency if they receive stronger incoming signals.
Bounded functions are employed since biochemical limitations impose an upper
bound on the firing frequency of the biological neurons [15]. In modern feed-
forward neural networks, however, the _Rectified Linear Unit (ReLU)_
$\psi(z)=\max(z,0)$ is the most widely-used activation function [16]. The
simple functional form of the ReLU is advantageous not only from a design
principle of minimalism, but also for differentiating the whole neural network
with respect to its parameters in the search for an optimal configuration:
Computing the derivative of the ReLU is trivial and the vanishing gradient
problem in the near constant regions of sigmoidal activation functions is
avoided in the active region of the ReLU, which prevents the gradient-based
optimization methods in the subsequent paragraph from stalling [17]. A more
thorough discussion of activation functions can be found in [5].

There is a downside of ANNs in comparison to a simple linear regression:
Firstly, it is no longer possible to find the optimal model parameters by
analytic means. Therefore, computationally efficient numerical optimization
methods, such as stochastic gradient descent, must be employed in the search
for sufficiently performant approximate solutions [5]. Although these
algorithms offer a practical alternative to analytic solution strategies for
ANNs of reasonable size, the need for numerical optimization adds another
layer of complexity to the task of constructing regression models. Secondly,
an ANN with enough parameters can always arrive at very low errors by
memorizing overly specific characteristics of the given data [18]. As a
remedy, the data is split into a _training set_ and a _test set_. The
parameters of the ANN are optimized with respect to the training set, which is
called _training_ , and the performance of the trained model is evaluated only
with respect to the data in the test set. If the ANN performs well on the test
set, we can assume that it has learned a general rule that tells us how the
input variable is related to the target variable. Although the multitude of
parameters makes it hard for humans to understand this general rule, we know
that it exists. In that sense, a low error on the test set for an ANN gives us
the same type of evidence for the existence of a relationship between two
entities, as a visual pattern or the Pearson correlation do in the
$1$-dimensional case.

##  3\. Knot Invariants

The techniques from the previous section were employed to probe possible
relationships between several knot invariants. The investigated knot
invariants can be divided into two groups: a group of combinatorial invariants
is derived from the knot diagram, whereas a group of geometric invariants is
derived from the hyperbolic structure on the knot complement. For the
remainder of the article, let $K$ denote a hyperbolic knot in $S^{3}$.

###  3.1. Combinatorial Knot Invariants

The considered combinatorial invariants in the first group can be understood
as compressed or generalized versions of the well-known Jones polynomial (see
[19] for the definition):

  * •

The Jones polynomial of $K$ is denoted by $J_{K}$.

  * •

The _knot determinant_ of $K$ is given by $\det(K)=J_{K}(-1)$.

  * •

The _Mahler measure_ of $J_{K}$ is given by
$m(J_{K})=\exp\left(\frac{1}{2\pi}\int_{0}^{2\pi}\ln\left|J_{K}\left(e^{i\theta}\right)\right|d\theta\right)$.

  * •

As a slight variation of the knot determinant, we will also evaluate the Jones
polynomial at the root of unity $\zeta=e^{2\pi i3/5}$.

  * •

The _Khovanov polynomial_ of $K$ is denoted by $\mathcal{KH}_{K}$. It is a
Laurent polynomial in two variables whose graded Euler characteristic equals
the Jones polynomial, in the sense that setting one of the variables to $-1$
recovers $J_{K}$. Mikhail Khovanov introduced it as a more refined version of
the Jones polynomial [20]. For alternating knots, the diagonal coefficients
equal the coefficients of the Jones polynomial and the off-diagonal
coefficients vanish.

###  3.2. Hyperbolic Knot Invariants

The second group of knot invariants is derived by exploiting the geometric
structure of the knot complement.

  * •

A _hyperbolic knot_ is a knot whose complement $S^{3}\setminus K$ admits a
hyperbolic structure. Mostow rigidity [21] shows that all such structures have
the same volume and it is therefore a topological invariant, which is called
the _hyperbolic volume_ $\operatorname{vol}(K)$.

  * •

Witten [22] describes how the Jones polynomial can be defined using only
intrinsic $3$-dimensional properties of a knot. In this context, the Chern-
Simons form from Chern-Simons theory can be used to assign a complex number to
each hyperbolic knot whose modulus coincides up to a normalization factor with
the hyperbolic volume. The argument of the complex number is called _Chern-
Simons invariant_ $\mathcal{CS}(K)\in\mathbb{R}/\mathbb{Z}$ and it follows
again from Mostow rigidity that it is a topological invariant of a hyperbolic
knot.

  * •

Another group of hyperbolic knot invariants is presented by Adams in [23].
Hyperbolic manifolds are seen as quotients $\mathbb{H}^{3}/\Gamma$ of the
hyperbolic $3$-space by a discrete group of fixed point free isometries
$\Gamma$. If one considers tubular neighborhoods $T\times(0,t]$ of the deleted
knot $T\times\\{0\\}$ in the knot complement, small neighborhoods lift to a
disjoint set of horoballs in the covering space $\mathbb{H}^{3}$. Increasing
the tubular neighborhood of the knot until the first two horoballs touch in
the covering space, we obtain a maximal tubular neighborhood
$T\times(0,t_{\max}]$ of the missing knot in the knot complement. This maximal
tubular neighborhood is called the _maximal cusp_ and several hyperbolic knot
invariants can be derived from it:

    * –

The _maximal cusp volume_ $\mathcal{C}(K)$ is the hyperbolic volume of the
maximal cusp.

    * –

The _longitude length_ $|\lambda(K)|$ is the minimal length of a longitude in
the boundary of the maximal cusp.

    * –

The _meridian length_ $|\mu(K)|$ is the minimal length of a meridian in the
boundary of the maximal cusp.

    * –

The boundary torus of the maximal cusp has $\mathbb{R}^{2}$ as its universal
cover and longitudes and meridians are non-trivial elements of the fundamental
group $\pi_{1}(T\times\\{t_{max}\\})$. Therefore, we can go around the minimal
length meridian and longitude once and observe the difference between the
starting point and end point of a corresponding path in the cover
$\mathbb{R}^{2}$. The respective differences are called _longitude
translation_ $(\lambda_{x},\lambda_{y})$ and _meridian translation_
$(\mu_{x},\mu_{y})$. In order to compare the data for several knots, the
orientations can be chosen such that we have $\lambda_{y}=0$, which means that
$\lambda_{x}$ is fully determined by $|\lambda(K)|$.

###  3.3. Known Connections

To the best of our knowledge, theoretical research in knot theory has
struggled to establish connections between the combinatorial and geometric
properties of knots. An exception is given by the above-mentioned work of
Witten that connects the Jones polynomial to the hyperbolic volume and the
Chern-Simons invariant via Chern-Simons theory [22]. Another example of
speculative nature is the long standing _volume conjecture_ , which predicts
that the N-colored Jones polynomials determine the volume of a hyperbolic knot
[24, 25]. This conjecture is one of the most well-known and mysterious
conjectures in low-dimensional topology and it has been extensively studied.
Its connection to Chern-Simons theory has been established in [26]. We do not
know of any conjecture relating the invariants derived from the maximal cusp
to the presented combinatorial ones.

Data-driven approaches have recently started to renew the interest in
connections between combinatorial and hyperbolic knot invariants: Jejjala, Kar
and Parrikar [8] reported that a standard version of an artificial neural
network could predict the hyperbolic volume of a knot from the coefficients of
its Jones polynomial with a mean absolute percentage error of under 3%. Their
results for knots with up to 15 crossings compared favorably to prediction
models based on volume-ish bounds and the Khovanov homology rank,
respectively, and were robust to shrinking the size of the training set. A
spectral analysis of the weight matrices revealed a regularity in the largest
eigenvalues over several training runs, suggesting that these matrices encode
a stable relationship between the Jones polynomial and the hyperbolic volume.
The authors proposed incorporating invariants from Khovanov homology for a
possible improvement of the predictive power, which is why we have included
this aspect in our research. A subsequent study conducted by Craven, Jejjala
and Kar [9] focussed on disentangling the approximate prediction formula
encoded in the ANN from [8]. This was a necessary step for a further
understanding of the precise relationship, given that the original ANN was
using over 10000 trainable parameters. In order to extract a more compact and
interpretable version of the formula that showed similar prediction
performance, layer-wise relevance propagation was employed to identify the
features that were mainly responsible for the successful prediction. They were
able to distill a performant approximate formula of the form

(11) $\operatorname{vol}(K)\approx a\log(|J_{K}(e^{3\pi i/4})|+b)-c,$

for real numbers $a,b,c$, which predicts the hyperbolic volume from a single
evaluation of the Jones polynomial at a root of unity. The simplicity of the
formula is remarkable when compared to the original formulation of the volume
conjecture, which involves a limit over all the N-colored Jones polynomials.
The success of these data-driven methods leads us to the the next section
where we apply similar techniques to analyze further connections between
combinatorial and hyperbolic knot invariants.

##  4\. Experiments

In this section, the conducted experiments and their results are documented
and discussed.

###  4.1. Data Preparation

All of the experiments could not have been performed without the helpful
software packages provided by the computational knot theory community. The
following sources were used:

  * •

The Jones polynomials of all hyperbolic knots with up to 14 crossings (almost
60000 knots) were computed by a combination of SnapPy [1] and SageMath [2].
The $1$-dimensional derived invariants were computed from the Jones
polynomials with NumPy [27].

  * •

The Khovanov polynomials for all hyperbolic knots with up to 12 crossings
(almost 3000 knots) were computed with the Software package KhoHo [3].

  * •

The hyperbolic volumes of all hyperbolic knots with up to 14 crossings were
computed by SnapPy.

  * •

Further hyperbolic knot invariants, namely the longitude length, the meridian
length and translation, the maximal cusp volume and the Chern-Simons
invariant, were taken from the online database KnotInfo [4]. The database
includes several knot invariants for all prime knots with up to 12 crossings.

The coefficients of the Jones polynomials and the Khovanov polynomials were
vectorized and padded with zeros to obtain a consistent data format. The three
single-variable invariants $\det(K),m(J_{K})$ and $J_{K}(\zeta)$ were
transformed to a logarithmic scale and divided by $\ln(\deg(J_{K})))$, in
order to obtain the rescaled invariants
$\underline{\det}(K),\underline{m}(J_{K})$ and
$\underline{J}_{K}(\zeta)$.222The _degree_ $\deg(J_{K})$ of a Jones polynomial
$J_{K}(t)=c_{0}t^{n_{0}}+...+c_{k}t^{n_{k}}$ with $c_{0}\neq 0$ and $c_{k}\neq
0$ is given by $n_{k}-n_{0}.$ All hyperbolic invariants were left unchanged.

###  4.2. Linear Regression

The first part of our experiments focussed on using linear regression to
detect possible connections between the $1$-dimensional knot invariants.

####  4.2.1. Model Design

Separate models were constructed for alternating knots $\mathcal{K}_{a}$, non-
alternating knots $\mathcal{K}_{n}$ and all knots $\mathcal{K}$. For each pair
$(\Phi,\Psi)$ with

(12)
$\Phi\in\\{\underline{\det}(K),\underline{m}(J_{K}),\underline{J}_{K}(\zeta)\\}$

and

(13)
$\Psi\in\\{\operatorname{vol}(K),|\lambda(K)|,|\mu(K)|,\mu_{x}(K),\mu_{y}(K),\mathcal{C}(K),\mathcal{CS}(K)\\},$

a least-squares linear regression was fit using NumPy and the Pearson
correlation was computed.

####  4.2.2. Experimental Results

| $r$
---|---
| $\mathcal{K}$ | $\mathcal{K}_{a}$ | $\mathcal{K}_{n}$
$\underline{\det}(K)$ | 0.74 | 0.98 | 0.66
$\underline{m}(J_{K})$ | 0.78 | 0.71 | 0.77
$\underline{J}_{K}(\zeta)$ | 0.94 | 0.96 | 0.92
Table 1. Pearson correlation coefficients between three combinatorial knot
invariants and the hyperbolic volume for knots with up to 14 crossings.

For hyperbolic knots with up to 14 crossings, the approximately linear
relationships between $\operatorname{vol}(K)$ on the one hand and
$\underline{\det}(K),\underline{m}(J_{K})$ and $\underline{J}_{K}(\zeta)$ on
the other are shown qualitatively in Fig. 1 and quantitatively in Table 1.

  * •

The first row shows how the relationship between the knot determinant and the
hyperbolic volume varies drastically depending on the knots being alternating
or not ($r=0.98$ vs. $r=0.66$), as was already observed by Dunfield [6] and
Friedl and Jackson [7]. The correlation on all knots is $r=0.74$.

  * •

The second row shows the more stable behavior for the Mahler measure
(${r=0.71}$ vs. $r=0.77$) and the characteristic formation of two clusters.
Given that a linear regression fits only one line through the data, the
overall correlation of $r=0.77$ barely surpasses the knot determinant. Fitting
two separate lines through the data from all knots yields intracluster
correlations of $r=0.87$ and $0.82$, respectively. The separation into two
clusters did not occur for the Mahler measure of the Alexander polynomial in
[7].

  * •

The third row shows that the evaluation of the Jones polynomial at the complex
root of unity performs well in both cases ($r=0.96$ vs. $r=0.92$) without
splitting the knots into two groups, leading to the best overall performance
of $r=0.94$. The correlation is still weaker than for the twisted Alexander
polynomial that was proposed by Friedl and Jackson ($r=0.98$ for knots with up
to 15 crossings), but the simplicity of evaluating the Jones polynomial is
remarkable in comparison from a theoretical and computational point of view.

Figure 1. Plots of the hyperbolic volumes corresponding to the linear
regression evaluations from Table 1.

Table 2 shows the correlation coefficients between combinatorial and
hyperbolic invariants for hyperbolic knots with up to 12 crossings. It is not
surprising that the first column again shows high values, given that the used
knot data is a subset of the knot data that was used to create Table 1.
Furthermore, it is important to keep in mind that being linearly related is an
equivalence relation, meaning that high correlations between the combinatorial
invariants and hyperbolic invariants other than $\operatorname{vol}(K)$
already imply high correlations between the hyperbolic invariant and
$\operatorname{vol}(K)$. This is most prominent for the maximal cusp volume
$\mathcal{C}(K)$, whose high correlation with $\operatorname{vol}(K)$ could be
confirmed by plotting both invariants against each other. However, observing
these implied correlations with $\operatorname{vol}(K)$ can be interesting in
its own right.

|  $r$ for $\mathcal{K}$
---|---
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\underline{\det}(K)$ | 0.80 | 0.26 | -0.41 | -0.38 | 0.27 | 0.76 | -0.03
$\underline{m}(J_{K})$ | 0.70 | 0.27 | -0.17 | -0.22 | 0.22 | 0.66 | -0.02
$\underline{J}_{K}(\zeta)$ | 0.94 | 0.25 | -0.34 | -0.44 | 0.39 | 0.90 | -0.03
|  $r$ for $\mathcal{K}_{a}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\underline{\det}(K)$ | 0.99 | 0.28 | 0.02 | -0.37 | 0.47 | 0.91 | -0.05
$\underline{m}(J_{K})$ | 0.65 | 0.26 | 0.05 | -0.15 | 0.25 | 0.61 | -0.02
$\underline{J}_{K}(\zeta)$ | 0.95 | 0.23 | -0.03 | -0.40 | 0.49 | 0.89 | -0.05
|  $r$ for $\mathcal{K}_{n}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\underline{\det}(K)$ | 0.73 | 0.15 | -0.47 | -0.35 | 0.23 | 0.71 | 0.00
$\underline{m}(J_{K})$ | 0.73 | 0.20 | -0.31 | -0.25 | 0.21 | 0.70 | 0.00
$\underline{J}_{K}(\zeta)$ | 0.93 | 0.16 | -0.46 | -0.42 | 0.34 | 0.88 | 0.00
Table 2. Pearson correlation coefficients between combinatorial and hyperbolic
invariants for knots with up to 12 crossings.

  * •

The Chern-Simons invariant is the only invariant that shows only very low
values. In light of the already established theoretical connection between the
Jones polynomial and the Chern-Simons invariant via Chern-Simons theory, the
negative results are somewhat surprising. The least-squares optimization might
have been inappropriate in this case, given that the invariant takes values in
$\mathbb{R}/\frac{1}{2}\mathbb{Z}$ for the data from KnotInfo and we used the
Euclidean distance between the representatives without taking into account the
cyclic structure.

  * •

All of the other invariants show moderate connections to
$\operatorname{vol}(K)$, with absolute values of most coefficients ranging
from $0.2$ to $0.5$.

  * •

Notable exceptions are the meridian length $|\mu(K)|$, where the correlation
coefficient nearly vanishes for alternating knots, and the longitude length
$|\lambda(K)|$, where the correlation weakens for non-alternating knots.

###  4.3. Deep Learning

For the multidimensional input invariants, we replaced the linear regression
models by fully connected ANNs and proceeded similarly to the previously
presented experiments.

####  4.3.1. Model Design

As in the linear case above, separate models were constructed for alternating
knots $\mathcal{K}_{a}$, non-alternating knots $\mathcal{K}_{n}$ and all knots
$\mathcal{K}$.

  * •

For each pair of invariants $(\Phi,\Psi)$ with input

(14) $\Phi\in\\{J_{K},\mathcal{KH}_{K}\\}$

and target

(15)
$\Psi\in\\{\operatorname{vol}(K),|\lambda(K)|,|\mu(K)|,\mu_{x}(K),\mu_{y}(K),\mathcal{C}(K),\mathcal{CS}(K)\\},$

a fully connected ANN with two hidden layers, consisting of 100 hidden neurons
each and using ReLU activation functions, was set up in the deep learning
framework TensorFlow [28]. The model was trained on randomly selected 80% of
the data and evaluated on the remaining 20%, with respect to the prediction
errors $\Delta_{\text{MAPE}}$ and $\Delta_{\text{MSE}}$.

  * •

For each pair of invariants $(\Phi,\Psi)$ with input

(16)
$\Phi\in\\{\underline{\det}(K),\underline{m}(J_{K}),\underline{J}_{K}(\zeta)\\}$

and target $\Psi$ as above, a least-squares linear regression was fit on
randomly selected 80% of the data. Subsequently, the prediction errors
$\Delta_{\text{MAPE}}$ and $\Delta_{\text{MSE}}$ were computed on the
remaining 20% of the data, in order to compare the predictive power of the
linear models and the ANNs.

####  4.3.2. Experimental Results

The prediction errors of the ANNs and linear regressions are reported in Table
3 for mean absolute percentage errors and Table 4 for mean squared errors,
respectively. The last row of the tables shows the errors of a model that
always predicts the mean value of the hyperbolic target invariant. This serves
as a base line that allows a comparison of the predictive power of the
optimized statistical models and a simple educated guess. Prediction errors
are grouped by horizontal lines according to the model type used (ANN, linear
regression or base line model). The mean squared errors are reported as
fractions of the mean squared errors of the base line model, in order to
facilitate comparison.

|  $\Delta_{\text{MAPE}}$ for $\mathcal{K}$
---|---
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 12.1 | 8.9 | 6.6 | - | 12.1 | 14.7 | -
$J_{K}$ | 3.9 | 15.0 | 6.3 | - | 11.7 | 8.8 | -
$\underline{\det}(K)$ | 13.4 | 23.4 | 6.7 | - | 22.8 | 14.9 | -
$\underline{m}(J_{K})$ | 15.1 | 23.4 | 7.1 | - | 23.1 | 16.7 | -
$\underline{J}_{K}(\zeta)$ | 7.2 | 22.9 | 6.9 | - | 21.4 | 9.5 | -
base line | 24.2 | 25.7 | 7.1 | - | 24.6 | 24.2 | -
|  $\Delta_{\text{MAPE}}$ for $\mathcal{K}_{a}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 5.8 | 5.5 | 6.0 | - | 7.3 | 10.0 | -
$J_{K}$ | 2.7 | 5.9 | 5.9 | - | 8.3 | 7.1 | -
$\underline{\det}(K)$ | 3.0 | 17.5 | 6.5 | - | 19.4 | 7.9 | -
$\underline{m}(J_{K})$ | 14.3 | 18.0 | 6.5 | - | 21.5 | 14.2 | -
$\underline{J}_{K}(\zeta)$ | 5.6 | 18.0 | 6.5 | - | 18.7 | 8.3 | -
base line | 20.0 | 19.1 | 6.5 | - | 22.8 | 19.1 | -
|  $\Delta_{\text{MAPE}}$ for $\mathcal{K}_{n}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 12.5 | 13.5 | 6.9 | - | 13.0 | 15.9 | -
$J_{K}$ | 3.9 | 15.0 | 7.0 | - | 14.0 | 9.2 | -
$\underline{\det}(K)$ | 13.8 | 28.9 | 5.9 | - | 25.9 | 16.1 | -
$\underline{m}(J_{K})$ | 13.2 | 28.2 | 6.4 | - | 26.4 | 14.6 | -
$\underline{J}_{K}(\zeta)$ | 6.9 | 28.8 | 6.0 | - | 23.8 | 9.3 | -
base line | 20.7 | 30.5 | 6.4 | - | 27.6 | 22.7 | -
Table 3. Mean absolute percentage errors for predictions of hyperbolic knot invariants of knots with up to 12 crossings. Within each subtable, errors are grouped by horizontal lines according to the model type used (ANN, linear regression or base line model) and printed in bold if they are less than half of the error of the base line model. Two columns could not be calculated because the data contained zeros. | Relative $\Delta_{\text{MSE}}$ for $\mathcal{K}$
---|---
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 0.31 | 0.24 | 1.00 | 0.22 | 0.34 | 0.46 | 1.19
$J_{K}$ | 0.03 | 0.39 | 0.88 | 0.36 | 0.46 | 0.19 | 0.92
$\underline{\det}(K)$ | 0.35 | 0.89 | 0.85 | 0.87 | 0.94 | 0.43 | 1.00
$\underline{m}(J_{K})$ | 0.47 | 0.92 | 0.96 | 0.94 | 0.93 | 0.55 | 1.00
$\underline{J}_{K}(\zeta)$ | 0.10 | 0.89 | 0.89 | 0.83 | 0.88 | 0.20 | 1.00
base line | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
| Relative $\Delta_{\text{MSE}}$ for $\mathcal{K}_{a}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 0.13 | 0.10 | 0.96 | 0.08 | 0.17 | 0.32 | 1.01
$J_{K}$ | 0.02 | 0.14 | 1.10 | 0.10 | 0.20 | 0.18 | 0.55
$\underline{\det}(K)$ | 0.03 | 0.96 | 1.00 | 0.88 | 0.80 | 0.20 | 0.99
$\underline{m}(J_{K})$ | 0.55 | 0.96 | 0.99 | 0.98 | 0.92 | 0.59 | 1.00
$\underline{J}_{K}(\zeta)$ | 0.09 | 0.99 | 0.99 | 0.85 | 0.76 | 0.21 | 0.99
base line | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
| Relative $\Delta_{\text{MSE}}$ for $\mathcal{K}_{n}$
| $\operatorname{vol}(K)$ | $|\lambda(K)|$ | $|\mu(K)|$ | $\mu_{x}(K)$ | $\mu_{y}(K)$ | $\mathcal{C}(K)$ | $\mathcal{CS}(K)$
$\mathcal{KH}_{K}$ | 0.45 | 0.35 | 1.12 | 0.28 | 0.37 | 0.59 | 1.13
$J_{K}$ | 0.05 | 0.32 | 1.09 | 0.46 | 0.56 | 0.22 | 1.16
$\underline{\det}(K)$ | 0.52 | 0.94 | 0.71 | 0.89 | 1.05 | 0.54 | 1.01
$\underline{m}(J_{K})$ | 0.55 | 0.91 | 0.89 | 0.94 | 1.01 | 0.61 | 1.00
$\underline{J}_{K}(\zeta)$ | 0.13 | 0.95 | 0.80 | 0.81 | 0.96 | 0.19 | 1.00
base line | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Table 4. Relative mean squared errors for predictions of hyperbolic knot
invariants of knots with up to 12 crossings. Within each subtable, errors are
grouped by horizontal lines according to the model type used (ANN, linear
regression or base line model) and printed in bold if they are less than half
of the error of the base line model.

  * •

As was already shown by Jejjala, Kar and Parrikar in [8], the hyperbolic
volume can be predicted with high precision by an ANN that takes the
coefficients of the Jones polynomial as an input. Unfortunately, using the
coefficients of the Khovanov polynomials did not improve the predictive power
of the model. On the contrary, the observed performance decreased
considerably, even though all the information from the Jones polynomials is
contained in the Khovanov polynomials. This may indicate that the additional
information in the Khovanov polynomials does not add any valuable input to the
prediction. Since the coefficients of the Khovanov polynomials are two-
dimensional matrices, the dimension of the input increases drastically when
compared to the prediction based on the Jones polynomials. This makes the
optimization problem in the training process harder, as more parameters in the
input layer need to be optimized, and it can lead to a worse resulting model.
Further evidence for this hypothesis is given by the fact that the difference
between the prediction errors of the two models was smaller for alternating
knots, where most of the Khovanov coefficients are zero. The general statement
that high-dimensional problems require more data for accurate predictions is
known as the _curse of dimensionality_ [29].

  * •

For the prediction of the longitude length, the situation changed and the
Khovanov ANN was the best model by a large margin, even though the Jones ANN
was also clearly better than the base line.

  * •

None of the models could predict the meridian length significantly better than
the base line.

  * •

For the meridian translation in both directions, the Khovanov ANNs were again
the best predictors, followed by the Jones ANNs, which also clearly beat the
linear models and the base line.

  * •

The maximal cusp volume shows similar results to the hyperbolic volume, which
is not surprising in light of the above discussion about their strong linear
association. It is worth noting that the Jones ANN performed considerably
worse than in the case of predicting the hyperbolic volume, which could be due
to optimization problems in the network.

  * •

The predictions for the Chern-Simons invariant were the least accurate ones,
with only one small success: In the case of alternating knots, the Jones ANN
beat the base line by a large margin. As discussed above, the poor results for
the other input variables could be due to the neglected cyclic nature of the
Chern-Simons invariant.

###  4.4. Varying Regression Inputs and Network Size

Apart from evaluating the Jones polynomial at $\zeta=e^{2\pi i3/5}$, other
roots of unity of the form $e^{2\pi ik/n}$ were tested. It turned out that
choosing $k/n$ close, but not equal to $1/2$ provided the best results, and
$k=3$, $n=5$ was chosen for simplicity reasons. Inspired by the success of
$\underline{J}_{K}(\zeta)$, ANNs were tested that input the values of $J_{K}$
at regularly spaced roots of unity instead of the coefficients of $J_{K}$. The
results did not differ significantly. Most experiments were repeated with
smaller ANNs in order to probe the necessary representative power of the
models. The hyperbolic volume could be predicted with under 5% error from an
ANN with only one hidden layer and 5 hidden neurons, reducing the number of
parameters in the weight matrices from over 10000 to only 80. In principle, it
is possible to read off a formula relating the Jones polynomial to the
hyperbolic volume with high accuracy from these smaller networks, but the
number of parameters is still far too high to allow a conceptual
interpretation of the resulting expression. As mentioned above, Craven,
Jejjala and Kar [9] have recently solved this very problem by using layer-wise
relevance propagation and distilled the compact formula

(17) $\operatorname{vol}(K)\approx 6.20\log(|J_{K}(e^{3\pi i/4})|+6.77)-0.94,$

which needs only one evaluation of the Jones polynomial at a root of unity to
predict the hyperbolic volume with an error of 2.86% on all hyperbolic knots
up to and including 16 crossings. Note the similarity of this formula to the
linear regression using $\underline{J}_{K}(\zeta)$ that was discussed above.
Our way of reasoning was to replace the root of unity $-1$ in the definition
of the knot determinant by another root of unity, whereas (17) was extracted
in a purely data-driven way from the complex inner workings of an ANN with
several thousands of parameters. Given that the numerical constants in (17)
are derived from a least-squares optimization on a finite dataset, it is
natural that it is not an exact formula. Apart from the trainable parameters,
the phase in the exponential is a hyperparameter that can be varied to some
degree without losing much predictive power, which is investigated in more
detail in [9]. These observations raise the question how the above approximate
formula can be turned into a theorem with meaningful and interpretable
constants. Do we need to include additional variables, as was the case for
Davies et al. in [12]? Are there multiple phases that lead to different exact
formulas or is there a single viable choice? Similar questions can be asked
about the empirical findings in our own work, which leads us to the last
section.

##  5\. Open Questions

In summary, we have seen that statistical methods, such as linear regression
and deep learning with artificial neural networks, can be valuable tools for
the discovery of relationships between knot invariants. Now that we know of
some empirical connections between combinatorial and hyperbolic knot
invariants, further mathematical research could address the following
questions:

  * •

How can the strong correlation between $\underline{J}_{K}(\zeta)$ and
$\operatorname{vol}(K)$ be explained?

  * •

Why is $\underline{\det}(K)$ only effective for predicting
$\operatorname{vol}(K)$ in the case of alternating knots?

  * •

What is a theoretical reason for the formation of the two clusters when
comparing $\underline{m}(J_{K})$ to $\operatorname{vol}(K)$ in Fig. 1?

  * •

What is an exact formula that predicts $\operatorname{vol}(K)$ from $J_{K}$?
Are additional invariants needed for a perfect prediction?

  * •

What is the theoretical relationship between $\mathcal{KH}_{K}$ and $\mu(K)$?
Are additional invariants needed for an exact prediction?

  * •

Why is there a clear correlation between $\operatorname{vol}(K)$ and
$|\mu(K)|$ only for non-alternating knots?

  * •

How can $\mathcal{CS}(K)$ be related to $J_{K}$ in the alternating case? What
other invariants are needed for an exact formula? Is there a relationship in
the non-alternating case?

  * •

What happens if we repeat the experiments with links instead of knots?

Clearly, most of these questions cannot be answered by the presented
statistical methods alone and need to be tackled by mathematical research of a
more traditional nature. ANNs are simply not made to be interpretable, as was
already discussed by Hughes [10]: they are designed to achieve a maximum of
predictive power based on many non-linear transformations of the input
variables. Perhaps the growing interest in explainable artificial intelligence
[30] will produce algorithms that can support mathematicians even further in
the quest for detecting and deciphering relationships between mathematical
entities.

## Acknowledgements

This article presents work that was conducted as part of my master thesis
_Deep Learning of Hyperbolic Knot Invariants_ [31] at Universität Regensburg.
The thesis provides a more detailed exposition of all the concepts that are
discussed in this article and it can be accessed online. I would like to thank
Stefan Friedl, the advisor of the thesis, for many helpful conversations and
suggestions. I am also grateful to Lukas Lewark for his input on topics in
computational knot theory. Maike Stern and Heribert Wankerl deserve special
credit for proofreading the paper.

## References

  * [1] M. Culler, N. M. Dunfield, M. Goerner, and J. R. Weeks.  SnapPy, a computer program for studying the geometry and topology of $3$-manifolds.  http://snappy.computop.org, 2009.
  * [2] The Sage Developers.  Sagemath, the Sage Mathematics Software System (Version 8.9).  https://www.sagemath.org, 2005.
  * [3] A. Shumakovitch.  KhoHo - a program for computing and studying Khovanov homology.  https://github.com/AShumakovitch/KhoHo, 2008.
  * [4] C. Livingston and A. H. Moore.  KnotInfo: Table of knot invariants.  http://www.indiana.edu/~knotinfo, 2004.
  * [5] I. Goodfellow, Y. Bengio, and A. Courville.  Deep Learning.  The MIT Press, 2016.
  * [6] N. Dunfield.  An interesting relationship between the jones polynomial and hyperbolic volume.  http://www.math.uiuc.edu/, 1999.
  * [7] S. Friedl and N. Jackson.  Approximations to the volume of hyperbolic knots.  arXiv, 1102.3742, 2011.
  * [8] V. Jejjala, A. Kar, and O. Parrikar.  Deep learning the hyperbolic volume of a knot.  Physics Letters B, 799:135033, 2019.
  * [9] Jessica Craven, Vishnu Jejjala, and Arjun Kar.  Disentangling a deep learned volume formula.  Journal of High Energy Physics, 2021(6), Jun 2021.
  * [10] M. C. Hughes.  A neural network approach to predicting and computing knot invariants.  Journal of Knot Theory and Its Ramifications, 29(03):2050005, 2020\.
  * [11] Jessica Craven, Mark Hughes, Vishnu Jejjala, and Arjun Kar.  Learning knot invariants across dimensions.  arXiv, 2112.00016, 2021.
  * [12] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al.  Advancing mathematics by guiding human intuition with ai.  Nature, 600(7887):70–74, 2021.
  * [13] Alex Davies, András Juhász, Marc Lackenby, and Nenad Tomasev.  The signature and cusp geometry of hyperbolic knots.  arXiv, 2111.15323, 2021.
  * [14] K. Hornik, M. Stinchcombe, and H. White.  Multilayer feedforward networks are universal approximators.  Neural Netw., 2(5):359–366, 1989.
  * [15] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 315–323, Fort Lauderdale, FL, USA, 11–13 Apr 2011\. PMLR.
  * [16] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions.  arXiv, 1710.05941, 2018.
  * [17] Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall.  Activation functions: Comparison of trends in practice and research for deep learning.  arXiv, 1811.03378, 2018.
  * [18] Igor V. Tetko, David J. Livingstone, and Alexander I. Luik.  Neural network studies. 1. comparison of overfitting and overtraining.  Journal of Chemical Information and Computer Sciences, 35(5):826–833, 1995.
  * [19] W. B. R. Lickorish.  An Introduction to Knot Theory.  Springer New York, New York, NY, 1997.
  * [20] M. Khovanov.  A categorification of the jones polynomial.  Duke Math. J., 101(3):359–426, 2000.
  * [21] G. D. Mostow.  Quasi-conformal mappings in $n$-space and the rigidity of hyperbolic space forms.  Publications Mathématiques de l’IHÉS, 34:53–104, 1968.
  * [22] E. Witten.  Quantum field theory and the jones polynomial.  Comm. Math. Phys., 121(3):351–399, 1989.
  * [23] C. Adams.  Chapter 1 - hyperbolic knots.  In W. Menasco and M. Thistlethwaite, editors, Handbook of Knot Theory, pages 1–18. Elsevier Science, Amsterdam, 2005.
  * [24] R. M. Kashaev.  The hyperbolic volume of knots from quantum dilogarithm.  arXiv, q-alg/9601025, 1996.
  * [25] Hitoshi Murakami and Jun Murakami.  The colored jones polynomials and the simplicial volume of a knot.  arXiv, math/9905075, 1999.
  * [26] Hitoshi Murakami, Jun Murakami, Miyuki Okamoto, Toshie Takata, and Yoshiyuki Yokota.  Kashaev’s conjecture and the chern-simons invariants of knots and links.  arXiv, math/0203119, 2002.
  * [27] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Pauli Virtanen, D. Cournapeau, E. Wieser, J Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Rio, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, 2020.
  * [28] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems.  https://www.tensorflow.org/, 2015.
  * [29] Richard Bellman.  Dynamic programming.  Science, 153(3731):34–37, 1966.
  * [30] A. Adadi and M. Berrada.  Peeking inside the black-box: A survey on explainable artificial intelligence (xai).  IEEE Access, 6:52138–52160, 2018.
  * [31] D. Grünbaum.  Deep learning of hyperbolic knot invariants.  https://sites.google.com/view/gruenbaum/theses, 2020.