# Hyperbolic Geometry in Computer Vision: A Novel Framework for Convolutional
Neural Networks

Ahmad Bdeir
University of Hildesheim
<EMAIL_ADDRESS>
&Kristian Schwethelm∗
University of Hildesheim
<EMAIL_ADDRESS>
Niels Landwehr
University of Hildesheim
<EMAIL_ADDRESS>
denotes equal contribution

###### Abstract

Real-world visual data exhibit intrinsic hierarchical structures that can be
represented effectively in hyperbolic spaces. Hyperbolic neural networks
(HNNs) are a promising approach for learning feature representations in such
spaces. However, current methods in computer vision rely on Euclidean
backbones and only project features to the hyperbolic space in the task heads,
limiting their ability to fully leverage the benefits of hyperbolic geometry.
To address this, we present HCNN, the first fully hyperbolic convolutional
neural network (CNN) designed for computer vision tasks. Based on the Lorentz
model, we generalize fundamental components of CNNs and propose novel
formulations of the convolutional layer, batch normalization, and multinomial
logistic regression (MLR). Experimentation on standard vision tasks
demonstrates the effectiveness of our HCNN framework and the Lorentz model in
both hybrid and fully hyperbolic settings. Overall, we aim to pave the way for
future research in hyperbolic computer vision by offering a new paradigm for
interpreting and analyzing visual data. Our code is publicly available at
https://github.com/kschwethelm/HyperbolicCV.

##  1 Introduction

As machine learning models become more complex and datasets grow larger, the
ability to accurately represent and manipulate data becomes increasingly
important. Traditionally, Euclidean geometry has been used to represent data
in machine learning as it offers a convenient, intuition-friendly environment
with a defined vector space and stable closed-form formulas. However, recent
research has shown that hyperbolic geometry may be better suited for capturing
hierarchical structures in data. Specifically, the negative curvature of
hyperbolic manifolds allows for distances to scale exponentially with respect
to the radius, which matches the exponential scaling of tree distances between
graph nodes [43]. This can then better retain graph information while
preventing spatial distortion. Additionally, even the natural spatial
representations in the human brain have shown to exhibit a hyperbolic geometry
[48].

Leveraging this better representative capacity, Hyperbolic Neural Networks
(HNNs) have demonstrated superior performance compared to Euclidean models in
many natural language processing (NLP) and graph embedding tasks [38].
However, hierarchical structures are not limited to textual and graph-
structured data but can also be found in images. The notion of hierarchy in
images is particularly established through the concept of part-whole
relationships within object representations and classes. In addition, Khrulkov
et al. [20] have found high hyperbolicity in image datasets; where the
hyperbolicity measure represents the degree of innate hierarchy between
semantic embeddings and can highlight the potential benefits of using HNNs.

In light of these findings, recent works have begun integrating hyperbolic
geometry into vision architectures. Specifically, they rely on the Poincaré
ball and the Lorentz model as descriptors of hyperbolic space, and formalize
hyperbolic translations of Euclidean machine learning operations. Currently,
most HNN components are only available in the Poincaré ball as it supports the
gyrovector space with basic vector operations. However, due to its hard
numerical constraint, the Poincaré ball is much more susceptible to numerical
instability than the Lorentz model [32]. Moreover, HNNs in computer vision
have been limited to a mixture of Euclidean backbones and hyperbolic task
heads due to the lack of required operations and the high computation cost
entailed. Until now, fully hyperbolic architectures have mainly been designed
for NLP and graph applications [38].

In this work, we propose the first fully hyperbolic framework for vision
tasks. We generalize the ubiquitous convolutional neural network (CNN)
architecture to the Lorentz model and present novel hyperbolic formulations of
the convolutional layer, batch normalization, and multinomial logistic
regression (MLR). Our methodology is general, and we show that our components
can be easily integrated into existing architectures. This is done through a
series of experiments based on direct translations of Euclidean architectures.
Our contributions then become three-fold:

  1. 1.

We extend vision HNNs to the fully hyperbolic setting by proposing the first
fully hyperbolic CNN framework dubbed HCNN.

  2. 2.

We bridge the gap between HNNs in the Lorentz model and the Poincaré ball by
providing novel Lorentzian formulations of the convolutional layer, batch
normalization, and multinomial logistic regression (MLR), which can also be
applied outside computer vision.

  3. 3.

We empirically demonstrate the effectiveness of HCNN in experiments on
standard vision tasks, including image classification and generation.

We believe our contributions provide a foundation for the development of more
powerful HNNs that can better represent the complex hierarchical structures
found in data.

##  2 Related works

#### Hyperbolic image embeddings

Previous research on HNNs has mainly focused on combining Euclidean backbones
and hyperbolic embeddings. This approach involves projecting Euclidean
features onto the hyperbolic space in the task heads and designing loss
functions based on hyperbolic geometry. Such simple hybrid architectures have
been proven effective in various computer vision tasks like recognition [20,
28, 15], segmentation [18, 1], reconstruction/generation [31, 34, 36, 25, 40],
and metric learning [10, 46]. However, Guo et al. [15] have shown that
learning a mixture of Euclidean and hyperbolic features can exacerbate
gradient vanishing, and it remains unclear whether these hybrid models can
fully exploit the properties of hyperbolic geometry. In contrast, our HCNN
naturally learns latent hyperbolic feature representations in every layer,
mitigating these issues. We also forgo the typically used Poincaré ball in
favor of the Lorentz model, as it is considered to offer better properties for
stability and optimization [32].

#### Fully hyperbolic neural networks

Designing fully hyperbolic neural networks requires generalizing Euclidean
network components to hyperbolic geometry. Notably, Ganea et al. [12] and
Shimizu et al. [45] utilized the Poincaré ball with the gyrovector space and
generalized various layers, including the fully-connected, convolutional, and
attention layers, as well as operations like split, concatenation, and
multinomial logistic regression (MLR). Researchers have also designed
components in the Lorentz model [4, 11, 5, 40], but some crucial components
are still missing, including the standard convolutional layer and MLR.
Following the hyperbolic layer definitions, fully hyperbolic neural networks
have dominated various tasks in natural language processing (NLP) and graph
applications [38]. However, no fully hyperbolic architecture has yet been
utilized in computer vision. Our work provides formulations for missing
components in the Lorentz model, allowing for the first fully hyperbolic
vision CNNs.

#### Normalization in HNNs

Standard normalization layers of Euclidean neural networks have found little
application in HNNs. To the best of our knowledge, there is only a single
hyperbolic normalization layer, i.e., the general Riemannian batch
normalization [30]. However, this method is not ideal due to the slow
iterative computation of the Fréchet mean and the arbitrary re-scaling
operation that is not based on hyperbolic geometry. We propose an efficient
batch normalization algorithm founded in the Lorentz model, which utilizes the
Lorentzian centroid [24] and a mathematically motivated re-scaling operation.

#### Numerical stability of HNNs

The exponential growth of the Lorentz model’s volume with respect to the
radius can introduce numerical instability and rounding errors in floating-
point arithmetic. This requires many works to rely on 64-bit precision at the
cost of higher memory and runtime requirements. To mitigate this, researchers
have introduced feature clipping and Euclidean reparameterizations [32, 15,
31]. We adopt these approaches in HCNN, allowing us to run under float32
arithmetic and reducing the computational cost.

##  3 Background

This section shortly summarizes the mathematical background of hyperbolic
geometry [3, 41]. The $n$-dimensional hyperbolic space $\mathbb{H}^{n}_{K}$ is
a smooth Riemannian manifold ($\mathcal{M}^{n},\mathfrak{g}_{\bm{x}}^{K}$)
with constant negative curvature $K<0$, where $\mathcal{M}^{n}$ and
$\mathfrak{g}_{\bm{x}}^{K}$ represent the manifold and the Riemannian metric,
respectively. There are multiple isometrically equivalent models describing
hyperbolic geometry. We build our HCNN upon the Lorentz model because of its
numerical stability and simple exponential/logarithmic maps and distance
functions. Additionally, we use the Poincaré Ball as a baseline. Both
hyperbolic models provide closed-form formulae for standard Riemannian
operations, including distance measures, exponential/logarithmic maps, and
parallel transportation. They are detailed in Appendix A.

Figure 1: Hyperbolic models.

#### Lorentz model

The $n$-dimensional Lorentz model
$\mathbb{L}^{n}_{K}=(\mathcal{L}^{n},\mathfrak{g}_{\bm{x}}^{K})$ models
hyperbolic geometry on the upper sheet of a two-sheeted hyperboloid
$\mathcal{L}^{n}$, with origin
$\overline{\bm{0}}=[\sqrt{-1/K},0,\cdots,0]^{T}$ and embedded in
$(n+1)$-dimensional Minkowski space. Based on the Riemannian metric
$\mathfrak{g}_{\bm{x}}^{K}=\mathrm{diag}(-1,1,\dots,1)$, the manifold is
defined as

$\displaystyle\mathcal{L}^{n}:=\\{\bm{x}\in\mathbb{R}^{n+1}~{}|~{}\langle\bm{x},\bm{x}\rangle_{\mathcal{L}}=\frac{1}{K},~{}x_{t}>0\\},$
(1)

with the Lorentzian inner product

$\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}:=-x_{t}y_{t}+\bm{x}_{s}^{T}\bm{y}_{s}=\bm{x}^{T}\mathrm{diag}(-1,1,\cdots,1)\bm{y}.$
(2)

When describing points in the Lorentz model, we inherit the terminology of
special relativity and call the first dimension the time component $x_{t}$ and
the remaining dimensions the space component $\bm{x}_{s}$, such that
$\bm{x}\in\mathbb{L}^{n}_{K}=[x_{t},\bm{x_{s}}]^{T}$ and
$x_{t}=\sqrt{||\bm{x}_{s}||^{2}-1/K}$.

#### Poincaré ball

The n-dimensional Poincaré ball
$\mathbb{B}^{n}_{K}=(\mathcal{B}^{n},\mathfrak{g}_{\bm{x}}^{K})$ is defined by
the Riemannian metric
$\mathfrak{g}_{\bm{x}}^{K}=(\lambda^{K}_{\bm{x}})^{2}\mathrm{\textbf{I}}_{n}$,
where $\lambda_{\bm{x}}^{K}=2(1+K||\bm{x}||^{2})^{-1}$. It describes the
hyperbolic space by an open ball of radius $\sqrt{-1/K}$ as follows

$\mathcal{B}^{n}=\\{\bm{x}\in\mathbb{R}^{n}~{}|~{}-K||\bm{x}||^{2}<1\\}.$ (3)

##  4 Fully Hyperbolic CNN (HCNN)

Our HCNN framework aims to give way to building vision models that can
effectively extract features in hyperbolic spaces, fully leveraging the
advantages of hyperbolic geometry. For this, we generalize Euclidean CNN
components to the Lorentz model, yielding one-to-one replacements that can be
integrated into existing architectures. In the following, we first define the
cornerstone of HCNNs, i.e., the Lorentz convolutional layer, including its
transposed variant. Then, we introduce the Lorentz batch normalization
algorithm and the MLR classifier. Finally, we generalize the residual
connection and non-linear activation.

###  4.1 Lorentz convolutional layer

#### Hyperbolic feature maps

The convolutional layer applies vector operations to an input feature map
containing the activations of the previous layer. In Euclidean space,
arbitrary numerical values can be combined to form a vector. Therefore, it is
not required to strictly determine which feature map dimension holds feature
vectors. However, in the Lorentz model, not all possible value combinations
represent a point that can be processed with hyperbolic operations
($\mathbb{L}_{K}^{n}\subset\mathbb{R}^{n+1}$). We propose using channel-last
feature map representations throughout the model and adding the time component
as an additional channel dimension. This defines a feature map as an ordered
set of $n$-dimensional hyperbolic vectors, where every spatial position
contains a vector that can be combined with its neighbors. Additionally, it
offers a nice interpretation where an image is an ordered set of color
vectors, each describing a pixel.

#### Formalization of the convolutional layer

We define the convolutional layer as an affine transformation between a
linearized kernel and a concatenation of the values in its receptive field,
following Shimizu et al. [45]. Then, we generalize this definition by
replacing the Euclidean operators with their hyperbolic counterparts in the
Lorentz model.

Given an input feature map
$\mathrm{\textbf{x}}=\\{\bm{x}_{h,w}\in\mathbb{L}^{n}_{K}\\}_{h,w=1}^{H,W}$ as
an ordered set of $n$-dimensional hyperbolic feature vectors each describing
image pixels, the features within the receptive field of the kernel
$\mathrm{\textbf{K}}\in\mathbb{R}^{m\times n\times\tilde{H}\times\tilde{W}}$
are
$\\{\bm{x}_{{h^{\prime}+\delta\tilde{h}},{w^{\prime}+\delta\tilde{w}}}\in\mathbb{L}^{n}_{K}\\}_{\tilde{h},\tilde{w}=1}^{\tilde{H},\tilde{W}}$,
where $(h^{\prime},w^{\prime})$ denotes the starting position and $\delta$ is
the stride parameter. Now, we define the Lorentz convolutional layer as

$\bm{y}_{h,w}=\text{LFC}(\text{HCat}(\\{\bm{x}_{{h^{\prime}+\delta\tilde{h}},{w^{\prime}+\delta\tilde{w}}}\\}_{\tilde{h},\tilde{w}=1}^{\tilde{H},\tilde{W}}\\})),$
(4)

where HCat denotes the concatenation of hyperbolic vectors and LFC denotes a
Lorentz fully-connected layer performing the affine transformation and
parameterizing the kernel and bias, respectively. Additionally, we propose
padding with origin vectors, the analog of zero vectors in hyperbolic space.

Our formulation generalizes arbitrary dimensional convolutional layers with
little modification to the 2-dimensional case presented here. However, it
remains essential to predetermine the structure of hyperbolic feature maps.

#### Extension to the transposed setting

The transposed convolutional layer is usually used in encoder-decoder
architectures for up-sampling. It is straightforward to generalize to
hyperbolic space, as it can be defined using a standard convolutional layer. A
convolutional layer carries out a transposed convolution when the correct
local connectivity is established by inserting zeros at certain positions.
Specifically, when stride $s>1$, then $s-1$ zero vectors are inserted between
the features. The input feature map is always implicitly padded on each side.
We refer to Dumoulin and Visin [9] for illustrations. Under this relationship,
the Lorentz transposed convolutional layer is a Lorentz convolutional layer
with changed connectivity through origin padding.

###  4.2 Lorentz batch normalization

Given a batch $\mathcal{B}$ of $m$ features $\bm{x}_{i}$, the traditional
batch normalization algorithm [19] calculates the mean
$\bm{\mu}_{\mathcal{B}}=\frac{1}{m}\sum_{i=1}^{m}\bm{x}_{i}$ and variance
$\bm{\sigma}^{2}_{\mathcal{B}}=\frac{1}{m}\sum_{i=1}^{m}(\bm{x}_{i}-\bm{\mu}_{\mathcal{B}})^{2}$
across the batch dimension. Then, the features are re-scaled and re-centered
using a parameterized variance $\bm{\gamma}$ and mean $\bm{\beta}$ as follows

$\text{BN}(\bm{x}_{i})=\bm{\gamma}\odot\frac{\bm{x}_{i}-\bm{\mu}_{\mathcal{B}}}{\sqrt{\bm{\sigma}^{2}_{\mathcal{B}}}}+\bm{\beta}.$
(5)

At test time, running estimates approximate the batch statistics. They are
calculated iteratively during training:
$\bm{\mu}_{t}=(1-\eta)\bm{\mu}_{t-1}+\eta\bm{\mu}_{\mathcal{B}}$ and
$\bm{\sigma}^{2}_{t}=(1-\eta)\bm{\sigma}^{2}_{t-1}+\eta\bm{\sigma}^{2}_{\mathcal{B}}$,
with $\eta$ and $t$ denoting momentum and the current iteration, respectively.
We generalize batch normalization to the Lorentz model using the Lorentzian
centroid and the parallel transport operation for re-centering, and the
Fréchet variance and straight geodesics at the origin’s tangent space for re-
scaling.

#### Re-centering

To re-center hyperbolic features, it is necessary to compute a notion of mean.
Usually, the Fréchet mean is used [30], which minimizes the expected squared
distance between a set of points in a metric space [39]. Generally, the
Fréchet mean must be solved iteratively, massively slowing down training. To
this end, we propose to use the centroid with respect to the squared
Lorentzian distance, which can be calculated efficiently in closed form. The
weighted Lorentzian centroid, which solves
$\min_{\bm{\mu}\in\mathbb{L}^{n}_{K}}\sum_{i=1}^{m}\nu_{i}d^{2}_{\mathcal{L}}(\bm{x}_{i},\bm{\mu})$,
with $\bm{x}_{i}\in\mathbb{L}^{n}_{K}$ and $\nu_{i}\geq
0,\sum_{i=1}^{m}\nu_{i}>0$, is given by

$\bm{\mu}=\frac{\sum_{i=1}^{m}\nu_{i}\bm{x}_{i}}{\sqrt{-K}\left|||\sum_{i=1}^{m}\nu_{i}\bm{x}_{i}||_{\mathcal{L}}\right|}.$
(6)

In batch normalization, the mean is not weighted, which gives
$\nu_{i}=\frac{1}{m}$. Now, we shift the features from the batch’s mean
$\bm{\mu}_{\mathcal{B}}$ to the parameterized mean $\bm{\beta}$ using the
parallel transport operation
$\mathrm{PT}^{K}_{{\bm{\mu}_{\mathcal{B}}}\rightarrow{\bm{\beta}}}\left({\bm{x}}\right)$.
Parallel transport does not change the variance, as it is defined to preserve
the distance between all points. Finally, the running estimate is updated
iteratively using the weighted centroid with $\nu_{1}=(1-\eta)$ and
$\nu_{2}=\eta$.

#### Re-scaling

For re-scaling, we rely on the Fréchet variance $\sigma^{2}\in\mathbb{R}^{+}$,
which can be defined as the expected squared Lorentzian distance between a
point $\bm{x}_{i}$ and the mean $\bm{\mu}$, given by
$\sigma^{2}=\frac{1}{m}\sum_{i=1}^{m}d^{2}(\bm{x}_{i},\bm{\mu})$. To re-scale
the batch, features must be moved along the geodesics connecting them to their
centroid, which is generally infeasible to compute. However, geodesics
intersecting the origin are very simple, as they can be represented by
straight lines in tangent space
$\mathcal{T}_{\overline{\bm{0}}}\mathbb{L}_{K}^{n}$. This is reflected by the
equality between the distance of a point to the origin and the length of its
corresponding tangent vector
($d_{\mathcal{L}}(\bm{x},\overline{\bm{0}})=||\log^{K}_{\overline{\bm{0}}}(\bm{x})||$).
Using this property, we propose to re-scale features by first parallel
transporting them towards the origin
$\mathrm{PT}^{K}_{{\mu_{\mathcal{B}}}\rightarrow{\overline{\bm{0}}}}\left({\bm{x}}\right)$,
making the origin the new mean. Then, a simple multiplication re-scales the
features in tangent space. Finally, parallel transporting to the parameterized
mean completes the algorithm and yields the normalized features. The final
algorithm of Lorentz batch normalization can be formalized as

$\text{LBN}(\bm{x})=\mathrm{PT}^{K}_{{\overline{\bm{0}}}\rightarrow{\bm{\beta}}}\left({{\gamma}\cdot\frac{\mathrm{PT}^{K}_{{\bm{\mu}_{B}}\rightarrow{\overline{\bm{0}}}}\left({\bm{x}}\right)}{{\sqrt{{\sigma}^{2}_{\mathcal{B}}}}}}\right).$
(7)

In our definition of hyperbolic feature maps, the centroid must be computed
across multiple dimensions. For example, given a feature map
$\\{\bm{x}_{h,w}\in\mathbb{L}^{n}_{K}\\}_{h,w=1}^{H,W}$ for all $m$ instances,
we first compute the centroid of all hyperbolic vectors per instance and then
the centroid across the batch dimension, i.e., the centroid of instance
centroids. Similarly, the Fréchet variance is computed for separate instances
and then averaged across the batch dimension as it is in $\mathbb{R}^{+}$.
Furthermore, the running estimate of the Fréchet variance is computed using a
standard running average.

Notice that in Lorentz batch normalization, the features must be parallel
transported twice, which is computationally slower than directly transporting
to the new mean, as in [30]. However, transporting towards and from the origin
are special cases that can be computed efficiently, reducing the overhead
significantly. That is why our algorithm still obtains a similar runtime to
methods using a single parallel transport operation.

###  4.3 Lorentz MLR classifier

In this section, we consider the problem of classifying instances that are
represented in the Lorentz model. A standard method for multi-class
classification is multinomial logistic regression (MLR). Inspired by the
generalization of MLR to the Poincaré ball [12, 45] based on the distance to
margin hyperplanes, we derive a formulation in the Lorentz model.

#### Hyperplane in the Lorentz model

Analogous to Euclidean space, hyperbolic hyperplanes split the manifold into
two half-spaces, which can then be used to separate instances into classes.
The hyperplane in the Lorentz model is defined by a geodesic that results from
the intersection of an n-dimensional hyperplane with the hyperboloid in the
ambient space $\mathbb{R}^{n+1}$ [6]. Specifically, for
$\bm{p}\in\mathbb{L}^{n}_{K}$ and
$\bm{w}\in\mathcal{T}_{\bm{p}}\mathbb{L}^{n}_{K}$, the hyperplane passing
through $\bm{p}$ and perpendicular to $\bm{w}$ is given by

$\displaystyle
H_{\bm{w},\bm{p}}=\\{\bm{x}\in\mathbb{L}^{n}_{K}\>|\>\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}=0\\}.$
(8)

This formulation comes with the non-convex optimization condition
$\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}>0$, which is undesirable in machine
learning. To eliminate this condition, we use the Euclidean reparameterization
of Mishne et al. [32], which we extend to include the curvature $K$ in
Appendix B.1. In short, $\bm{w}$ is parameterized by a vector
$\bm{\overline{z}}\in\mathcal{T}_{\overline{\bm{0}}}\mathbb{L}_{K}^{n}=[0,a{\bm{z}/}{||\bm{z}||}]$,
where $a\in\mathbb{R}$ and $\bm{z}\in\mathbb{R}^{n}$. As
$\bm{w}\in\mathcal{T}_{\bm{p}}\mathbb{L}_{K}^{n}$, $\bm{\overline{z}}$ is
parallel transported to $\bm{p}$, which gives

$\bm{w}:=\mathrm{PT}^{K}_{{\overline{\bm{0}}}\rightarrow{\bm{p}}}\left({\overline{\bm{z}}}\right)=[\sinh(\sqrt{-K}a)||\bm{z}||,\cosh(\sqrt{-K}a)\bm{z}].$
(9)

Inserting Eq. 9 into Eq. 8, the formula of the Lorentz hyperplane becomes

$\tilde{H}_{\bm{z},a}=\\{\bm{x}\in\mathbb{L}^{n}_{K}~{}|~{}\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a)~{}||\bm{z}||~{}x_{t}=0\\}.$
(10)

Finally, we need the distance to the hyperplane to quantify the model’s
confidence. It is formulated by the following theorem, proven in Appendix B.2.

###### Theorem 1

Given $a\in\mathbb{R}$, $\bm{z}\in\mathbb{R}^{n}$, the minimum hyperbolic
distance from a point $\bm{x}\in\mathbb{L}^{n}_{K}$ to the hyperplane
$\tilde{H}_{\bm{z},a}$ defined in Eq. 10 is given by

$d_{\mathcal{L}}(\bm{x},\tilde{H}_{\bm{z},a})=\frac{1}{\sqrt{-K}}\left|\sinh^{-1}\left(\sqrt{-K}\frac{\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a)~{}||\bm{z}||~{}x_{t}}{\sqrt{||\cosh(\sqrt{-K}a)\bm{z}||^{2}-(\sinh(\sqrt{-K}a)||\bm{z}||)^{2}}}\right)\right|.$
(11)

#### MLR in the Lorentz model

Lebanon and Lafferty [27] formulated the logits of the Euclidean MLR
classifier using the distance from instances to hyperplanes describing the
class regions. Specifically, given input $\bm{x}\in\mathbb{R}^{n}$ and $C$
classes, the output probability of class $c\in\\{1,...,C\\}$ can be expressed
as

$p(y=c\>|\>\bm{x})\propto\exp(v_{\bm{w}_{c}}(\bm{x})),~{}~{}v_{\bm{w}_{c}}(\bm{x})=\text{sign}(\langle\bm{w}_{c},\bm{x}\rangle)||\bm{w}_{c}||d(\bm{x},H_{\bm{w}_{c}}),~{}~{}\bm{w}_{c}\in\mathbb{R}^{n},$
(12)

where $H_{\bm{w}_{c}}$ is the decision hyperplane of class $c$.

We define the Lorentz MLR without loss of generality by inserting the
Lorentzian counterparts into Eq. 12. The MLR logits are given in the following
theorem, proven in Appendix B.3.

###### Theorem 2

Given parameters $a_{c}\in\mathbb{R}$, $\bm{z}_{c}\in\mathbb{R}^{n}$, the
Lorentz MLR’s output logit corresponding to class $c$ and input
$\bm{x}\in\mathbb{L}^{n}_{K}$ is given by

$v_{\bm{z}_{c},a_{c}}(\bm{x})=\frac{1}{\sqrt{-K}}~{}\mathrm{sign}(\alpha)\beta\left|\sinh^{-1}\left(\sqrt{-K}\frac{\alpha}{\beta}\right)\right|,$
(13)

with

$\alpha=\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a),$
$\beta=\sqrt{||\cosh(\sqrt{-K}a)\bm{z}||^{2}-(\sinh(\sqrt{-K}a)||\bm{z}||)^{2}}.$

###  4.4 Lorentz residual connection and activation

#### Residual connection

The residual connection is a crucial component when designing deep CNNs. As
vector addition is ill-defined in the Lorentz model, we add the vector’s space
components and concatenate a corresponding time component. This is possible as
a point $\bm{x}\in\mathbb{L}^{n}_{K}$ can be defined by an arbitrary space
component $\bm{x}_{s}\in\mathbb{R}^{n}$ and a time component
$x_{t}=\sqrt{||\bm{x}_{s}||^{2}-1/K}$. Our method is straightforward and
provides the best empirical performance compared to other viable methods we
implemented, i.e., tangent space addition [35], parallel transport addition
[4], Möbius addition (after projecting to the Poincaré ball) [12], and fully-
connected layer addition [5].

#### Non-linear activation

Prior works use non-linear activation in tangent space [11], which weakens the
model’s stability due to frequent logarithmic and exponential maps. We propose
to simplify the operation for the Lorentz model by applying the activation
function to the space component and concatenating a time component. For
example, the Lorentz ReLU activation is given by

$\bm{y}=\left[\begin{array}[]{c}\sqrt{||\text{ReLU}(\bm{x}_{s})||^{2}-1/K}\\\
\text{ReLU}(\bm{x}_{s})\end{array}\right].$ (14)

This operation can be interpreted similarly to Euclidean activations that
break linearity with heuristic mathematical projections.

##  5 Experiments

We evaluate HCNN models on image classification and generation tasks and
compare them against Euclidean and hybrid HNN counterparts. To ensure a fair
comparison, in every task, we directly translate a Euclidean baseline to the
hyperbolic setting by using hyperbolic modules as one-to-one replacements. All
experiments are implemented in PyTorch [37], and we optimize hyperbolic models
using adaptive Riemannian optimizers [2] provided by Geoopt [22], with
floating-point precision set to 32 bits. We provide detailed experimental
configurations and results in Appendix C. To facilitate reproducibility and
further exploration, we make the code of our experiments publicly
available111https://github.com/kschwethelm/HyperbolicCV.

###  5.1 Image classification

#### Experimental setup

In this experiment, we evaluate the performance of HNNs on standard image
classification tasks using ResNet-18 [16] and three benchmark datasets:
CIFAR-10 [23], CIFAR-100 [23], and Tiny-ImageNet [26]. We test two hybrid
HNNs, which retain the Euclidean model for embedding images, project the final
embeddings onto hyperbolic space, and apply a hyperbolic classification layer.
The first hybrid HNN employs the state-of-the-art Poincaré MLR classifier
[45], while the second model employs our novel Lorentz MLR. Additionally, we
must use feature clipping on embeddings to stop gradient issues that arise
from the hybrid architecture [15]. Furthermore, we create an HCNN model by
replacing all Euclidean components in the ResNet architecture with our
proposed modules. Finally, for all models, we adopt the training procedure and
hyperparameters of DeVries and Taylor [8].

#### Main results

Table 1 summarizes the classification accuracies on all datasets. Our HCNN-
ResNet achieves competitive results compared to the Euclidean baseline, even
though the hyperparameters are optimized for the Euclidean model. This
suggests that much performance can still be gained through optimized
architectures and novel hyperbolic training procedures. For example, Euclidean
ResNet has greatly benefited from a tighter training regime, with CIFAR-100
performance increasing from 74.84% for the original ResNet-110 [47] to 77.72%
we obtained for the much smaller ResNet-18.

Comparing performance between HNNs, we find that both Lorentz HNNs exhibit
superior stability and performance compared to the Poincaré baseline.
Remarkably, our hybrid Lorentzian model is the first HNN to outperform the
Euclidean baseline in standard image classification. These findings highlight
the potential of our HCNN framework in improving image classification tasks
and suggest that the Lorentz model has superior properties for HNNs than the
Poincaré ball.

Table 1: Classification accuracy (%) of manifold ResNets. We report mean and
standard deviation across five runs. The best performance is highlighted in
bold (higher is better).

CIFAR-10 CIFAR-100 Tiny-ImageNet Euclidean $\bm{95.14_{\pm 0.12}}$ $77.72_{\pm
0.15}$ $65.09_{\pm 0.12}$ Hybrid Poincaré $95.04_{\pm 0.13}$ $77.19_{\pm
0.50}$ $64.93_{\pm 0.38}$ Hybrid Lorentz (Ours) ${94.98_{\pm 0.12}}$
$\bm{78.03_{\pm 0.21}}$ $\bm{65.63_{\pm 0.10}}$ HCNN Lorentz (Ours)
$94.93_{\pm 0.11}$ $77.52_{\pm 0.10}$ $64.97_{\pm 0.21}$

###  5.2 Image generation

#### Experimental setup

Variational autoencoders (VAEs) [21, 42] have been widely adopted in HNN
research to model latent embeddings in hyperbolic spaces [34, 31, 36, 18].
However, these works rely on hybrid architectures and mainly focus on learning
strong embeddings. In this experiment, we extend the hyperbolic VAE to the
fully hyperbolic setting using our proposed HCNN framework and, for the first
time, evaluate its performance on image generation using the standard Fréchet
Inception Distance (FID) metric [17]. Building on the experimental settings of
Ghosh et al. [13], we test vanilla VAEs and assess generative performance on
CIFAR-10 [23], CIFAR-100 [23], and CelebA [29] datasets. We compare our HCNN-
VAE against the Euclidean and two hybrid models. Like most previous works, the
hybrid models only include a hyperbolic latent distribution and no hyperbolic
layers. Furthermore, we employ the wrapped normal distributions in the Lorentz
model [34] and the Poincaré ball [31], respectively.

#### Main results

The results in Table 2 show that our HCNN-VAE outperforms all baselines. Also,
the hybrid models improve performance over the Euclidean model, indicating
that learning the latent embeddings in hyperbolic spaces is beneficial.
However, our HCNN model is better at leveraging the advantages of hyperbolic
geometry due to its fully hyperbolic architecture. These results suggest that
our method is a promising approach for modeling latent structures in image
data and has the potential to generate high-quality images when employing more
complex architectures.

Table 2: Reconstruction and generation FID of manifold VAEs. We report mean
and standard deviation across five runs. The best performance is highlighted
in bold (lower is better).

CIFAR-10 CIFAR-100 CelebA Rec. FID Gen. FID Rec. FID Gen. FID Rec. FID Gen.
FID Euclidean $61.21_{\pm 0.72}$ $92.40_{\pm 0.80}$ $63.81_{\pm 0.47}$
$103.54_{\pm 0.84}$ $54.80_{\pm 0.29}$ $79.25_{\pm 0.89}$ Hybrid Poincaré
$59.85_{\pm 0.50}$ $90.13_{\pm 0.77}$ $62.64_{\pm 0.43}$ $98.19_{\pm 0.57}$
$54.62_{\pm 0.61}$ $81.30_{\pm 0.56}$ Hybrid Lorentz $59.29_{\pm 0.47}$
$90.91_{\pm 0.84}$ $62.14_{\pm 0.35}$ $98.34_{\pm 0.62}$ $54.64_{\pm 0.34}$
$82.78_{\pm 0.93}$ HCNN Lorentz (Ours) $57.83_{\pm 0.44}$ $89.23_{\pm 0.80}$
$61.91_{\pm 0.59}$ $101.09_{\pm 0.92}$ $54.13_{\pm 0.63}$ $78.28_{\pm 1.01}$

###  5.3 Ablation studies

In this section, we conduct ablation studies to obtain additional insights
into HCNN. Unless otherwise stated, all ablation experiments consider image
classification on CIFAR-100 and no Poincaré HNN.

Table 3: Runtime improvement using PyTorch’s compile function on ResNet-18.
Duration of a training epoch in seconds.

Eucl. Hybrid HCNN Default 7.4 9.2 103.3 Compiled 7.1 8.0 62.1 Difference -4.1%
-13.0% -39.9%

#### Runtime

Currently, two major drawbacks of HNNs are relatively high runtime and memory
requirements. This is partly due to custom pythonic implementations of
hyperbolic network components introducing significant computational overhead.
To study this, we use PyTorch’s compile function, which automatically builds a
more efficient computation graph. We compare the runtime of our Lorentz
ResNets with the Euclidean baseline under compiled and default settings in
Table 3. The results show that hybrid HNNs only add little overhead compared
to the significantly slower HCNN. This makes scaling HCNNs challenging and
requires special attention in future works. However, we also see that
hyperbolic models gain much more performance from the automatic compilation
than the Euclidean model. This indicates greater room for improvement in terms
of implementation optimizations.

#### Curvature

The curvature $K$ of hyperbolic space is an additional hyperparameter in HNNs.
We investigate the effect of $K$ in Lorentzian ResNets by evaluating
classification accuracy with fixed curvatures from 0.2 to 1.4, and a learnable
curvature initialized with 1.0. The results are shown in Figure 2. While in
the fully hyperbolic model, a curvature deviating from $K=-1$ deteriorates
accuracy, the hybrid model is stable against fixed changes in curvature.
Furthermore, learning $K$ does not show to be effective in our experimental
setting.

#### Low embedding dimension

HNNs have shown to be most effective in lower-dimensional spaces. To this end,
we reduce the dimensionality of the final ResNet block and evaluate
classification accuracy. The results in Figure 2 verify the effectiveness of
hyperbolic spaces with low dimensions, where our HCNN can leverage this
property best. This suggests that HCNNs offer great opportunities for
designing smaller models with fewer parameters.

0.20.40.60.81.01.21.4LC$75$$76$$77$$78$$79$Curvature $-K$Accuracy
(%)HybridHCNN

8163264$72$$74$$76$$78$Embedding dimensionAccuracy (%)EuclideanHybridHCNN

Figure 2: Results of ablation experiments on CIFAR-100 classification using a
varied curvature $K$ (left, "LC" denotes a learnable curvature) and low
dimensionality in the final ResNet block (right).

##  6 Conclusion

In this work, we proposed HCNN, a generalization of the convolutional neural
network that learns latent feature representations in hyperbolic spaces. To
this end, we formalized the necessary modules in the Lorentz model and derived
novel formulations of the convolutional layer, batch normalization, and
multinomial logistic regression. We demonstrated that ResNet and VAE models
based on our HCNN framework achieve competitive performance on standard vision
tasks compared to Euclidean and hybrid baselines. Additionally, we showed that
using the Lorentz model in HNNs leads to better stability and performance than
the Poincaré ball.

However, HCNNs are still in their early stages, and it remains to be seen to
which extent they can replace Euclidean networks as they introduce
mathematical complexity and computational overhead. Moreover, our HCNN
framework relies on generalizations of networks that were designed for
Euclidean geometry and might not fully capture the unique properties of
hyperbolic geometry. Further research is needed to fully understand the
properties of HCNNs and address open questions such as optimization,
scalability, and performance on other machine learning problems. We hope our
work will inspire future research and development in this exciting and rapidly
evolving field.

## Acknowledgments and Disclosure of Funding

This work was performed on the HoreKa supercomputer funded by the Ministry of
Science, Research and the Arts Baden-Württemberg and by the Federal Ministry
of Education and Research. Ahmad Bdeir was funded by the European Union’s
Horizon 2020 research and innovation programme under the SustInAfrica grant
agreement No 861924. Kristian Schwethelm was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) - project number
225197905.

## References

  * Atigh et al. [2022] M. G. Atigh, J. Schoep, E. Acar, N. van Noord, and P. Mettes.  Hyperbolic image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4453–4462, June 2022.
  * Bécigneul and Ganea [2018] G. Bécigneul and O.-E. Ganea.  Riemannian adaptive optimization methods, 2018.  URL https://arxiv.org/abs/1810.00760.
  * Cannon et al. [2006] J. W. Cannon, W. J. Floyd, R. Kenyon, and W. R. Parry.  _Hyperbolic Geometry_ , volume 31.  MSRI Publications, 2006.
  * Chami et al. [2019] I. Chami, R. Ying, C. Ré, and J. Leskovec.  Hyperbolic graph convolutional neural networks, 2019.  URL https://arxiv.org/abs/1910.12933.
  * Chen et al. [2021] W. Chen, X. Han, Y. Lin, H. Zhao, Z. Liu, P. Li, M. Sun, and J. Zhou.  Fully hyperbolic neural networks.  _CoRR_ , abs/2105.14686, 2021.  URL https://arxiv.org/abs/2105.14686.
  * Cho et al. [2019] H. Cho, B. DeMeo, J. Peng, and B. Berger.  Large-margin classification in hyperbolic space.  In K. Chaudhuri and M. Sugiyama, editors, _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , volume 89 of _Proceedings of Machine Learning Research_ , pages 1832–1840. PMLR, 16–18 Apr 2019.  URL https://proceedings.mlr.press/v89/cho19a.html.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * DeVries and Taylor [2017] T. DeVries and G. W. Taylor.  Improved regularization of convolutional neural networks with cutout, 2017\.  URL https://arxiv.org/abs/1708.04552.
  * Dumoulin and Visin [2016] V. Dumoulin and F. Visin.  A guide to convolution arithmetic for deep learning, 2016.  URL https://arxiv.org/abs/1603.07285.
  * Ermolov et al. [2022] A. Ermolov, L. Mirvakhabova, V. Khrulkov, N. Sebe, and I. Oseledets.  Hyperbolic vision transformers: Combining improvements in metric learning, 2022.  URL https://arxiv.org/abs/2203.10833.
  * Fan et al. [2022] X. Fan, C.-H. Yang, and B. C. Vemuri.  Nested hyperbolic spaces for dimensionality reduction and hyperbolic nn design.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 356–365, 2022.  doi: 10.1109/CVPR52688.2022.00045.
  * Ganea et al. [2018] O. Ganea, G. Becigneul, and T. Hofmann.  Hyperbolic neural networks.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper/2018/file/dbab2adc8f9d078009ee3fa810bea142-Paper.pdf.
  * Ghosh et al. [2019] P. Ghosh, M. S. M. Sajjadi, A. Vergari, M. Black, and B. Schölkopf.  From variational to deterministic autoencoders, 2019.  URL https://arxiv.org/abs/1903.12436.
  * Grosek [2008] J. Grosek.  Fundamental reflection domains for hyperbolic tesselations.  In _Rose-Hulman Undergraduate Mathematics Journal_ , volume 9, 2008\.  URL https://scholar.rose-hulman.edu/rhumj/vol9/iss1/4.
  * Guo et al. [2022] Y. Guo, X. Wang, Y. Chen, and S. X. Yu.  Clipped hyperbolic classifiers are super-hyperbolic classifiers.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–10, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.  doi: 10.1109/CVPR52688.2022.00010.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00010.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.  URL http://arxiv.org/abs/1512.03385.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6629–6640, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Hsu et al. [2020] J. Hsu, J. Gu, G.-H. Wu, W. Chiu, and S. Yeung.  Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations, 2020.  URL https://arxiv.org/abs/2012.01644.
  * Ioffe and Szegedy [2015] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ICML’15, page 448–456. JMLR.org, 2015.
  * Khrulkov et al. [2020] V. Khrulkov, L. Mirvakhabova, E. Ustinova, I. Oseledets, and V. Lempitsky.  Hyperbolic image embeddings.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6417–6427, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.  doi: 10.1109/CVPR42600.2020.00645.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.00645.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes, 2013.  URL https://arxiv.org/abs/1312.6114.
  * Kochurov et al. [2020] M. Kochurov, R. Karimov, and S. Kozlukov.  Geoopt: Riemannian optimization in pytorch, 2020.
  * Krizhevsky [2009] A. Krizhevsky.  Learning multiple layers of features from tiny images.  pages 32–33, 2009.  URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  * Law et al. [2019] M. Law, R. Liao, J. Snell, and R. Zemel.  Lorentzian distance learning for hyperbolic representations.  In K. Chaudhuri and R. Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3672–3681. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/law19a.html.
  * Lazcano et al. [2021] D. Lazcano, N. F. Franco, and W. Creixell.  Hgan: Hyperbolic generative adversarial network.  _IEEE Access_ , 9:96309–96320, 2021.  doi: 10.1109/ACCESS.2021.3094723.
  * Le and Yang [2015] Y. Le and X. S. Yang.  Tiny imagenet visual recognition challenge.  2015\.
  * Lebanon and Lafferty [2004] G. Lebanon and J. Lafferty.  Hyperplane margin classifiers on the multinomial manifold.  In _Proceedings of the Twenty-First International Conference on Machine Learning_ , ICML ’04, page 66, New York, NY, USA, 2004. Association for Computing Machinery.  ISBN 1581138385.  doi: 10.1145/1015330.1015333.  URL https://doi.org/10.1145/1015330.1015333.
  * Liu et al. [2020] S. Liu, J. Chen, L. Pan, C.-W. Ngo, T.-S. Chua, and Y.-G. Jiang.  Hyperbolic visual embedding learning for zero-shot recognition.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9270–9278, 2020.  doi: 10.1109/CVPR42600.2020.00929.
  * Liu et al. [2015] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 3730–3738, 2015.  doi: 10.1109/ICCV.2015.425.
  * Lou et al. [2020] A. Lou, I. Katsman, Q. Jiang, S. Belongie, S.-N. Lim, and C. De Sa.  Differentiating through the fréchet mean, 2020.  URL https://arxiv.org/abs/2003.00335.
  * Mathieu et al. [2019] E. Mathieu, C. L. Lan, C. J. Maddison, R. Tomioka, and Y. W. Teh.  Continuous hierarchical representations with poincaré variational auto-encoders, 2019.  URL https://arxiv.org/abs/1901.06033.
  * Mishne et al. [2022] G. Mishne, Z. Wan, Y. Wang, and S. Yang.  The numerical stability of hyperbolic representation learning, 2022.  URL https://arxiv.org/abs/2211.00181.
  * Moretti [2002] V. Moretti.  The interplay of the polar decomposition theorem and the lorentz group, 2002.  URL https://arxiv.org/abs/math-ph/0211047.
  * Nagano et al. [2019] Y. Nagano, S. Yamaguchi, Y. Fujita, and M. Koyama.  A wrapped normal distribution on hyperbolic space for gradient-based learning, 2019.  URL https://arxiv.org/abs/1902.02992.
  * Nickel and Kiela [2018] M. Nickel and D. Kiela.  Learning continuous hierarchies in the lorentz model of hyperbolic geometry, 2018.  URL https://arxiv.org/abs/1806.03417.
  * Ovinnikov [2019] I. Ovinnikov.  Poincaré wasserstein autoencoder.  2019\.  doi: 10.48550/ARXIV.1901.01427.  URL https://arxiv.org/abs/1901.01427.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Peng et al. [2022] W. Peng, T. Varanka, A. Mostafa, H. Shi, and G. Zhao.  Hyperbolic deep neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(12):10023–10044, 2022.  doi: 10.1109/TPAMI.2021.3136921.
  * Pennec [2006] X. Pennec.  Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements.  _Journal of Mathematical Imaging and Vision_ , 25(1):127–154, 2006.
  * Qu and Zou [2022] E. Qu and D. Zou.  Lorentzian fully hyperbolic generative adversarial network, 2022.  URL https://arxiv.org/abs/2201.12825.
  * Ratcliffe [2006] J. G. Ratcliffe.  _Foundations of Hyperbolic Manifolds_.  Springer, 2 edition, 2006.
  * Rezende et al. [2014] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models, 2014.  URL https://arxiv.org/abs/1401.4082.
  * Sarkar [2012] R. Sarkar.  Low distortion delaunay embedding of trees in hyperbolic plane.  In M. van Kreveld and B. Speckmann, editors, _Graph Drawing_ , pages 355–366, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.  ISBN 978-3-642-25878-7.
  * Seitzer [2020] M. Seitzer.  pytorch-fid: FID Score for PyTorch.  https://github.com/mseitzer/pytorch-fid, August 2020.  Version 0.2.1.
  * Shimizu et al. [2020] R. Shimizu, Y. Mukuta, and T. Harada.  Hyperbolic neural networks++, 2020.  URL https://arxiv.org/abs/2006.08210.
  * Yue et al. [2023] Y. Yue, F. Lin, K. D. Yamada, and Z. Zhang.  Hyperbolic contrastive learning, 2023.  URL https://arxiv.org/abs/2302.01409.
  * Zagoruyko and Komodakis [2017] S. Zagoruyko and N. Komodakis.  Wide residual networks, 2017.
  * Zhang et al. [2023] H. Zhang, P. D. Rich, A. K. Lee, and T. O. Sharpee.  Hippocampal spatial representations exhibit a hyperbolic geometry that expands with experience.  _Nature Neuroscience_ , 26(1):131–139, 2023.

##  Appendix A Operations in hyperbolic geometry

###  A.1 Lorentz model

(a) Geodesic

(b) Tangent space

(c) Parallel Transport

Figure 3: Illustrations of geometrical operations in the 2-dimensional Lorentz
model. (a) The shortest distance between two points is represented by the
connecting geodesic (red line). (b) The red line gets projected onto the
tangent space of the origin resulting in the green line. (c) The green line
gets parallel transported to the tangent space of the origin.

(a) Lorentz rotation

(b) Lorentz boost

Figure 4: Illustration of the Lorentz transformations in the 2-dimensional
Lorentz model.

In this section, we describe essential geometrical operations in the Lorentz
model. Most of these operations are defined for all Riemannian manifolds and
thus introduced for the general case first. However, the closed-form formulae
are only given for the Lorentz model. We also provide visual illustrations in
Figure 3.

#### Distance

Distance is defined as the length of the shortest path between a pair of
points on a surface. While in Euclidean geometry, this is a straight line, in
hyperbolic space, the shortest path is represented by a curved geodesic
generalizing the notion of a straight line. In the Lorentz model, the distance
is inherited from Minkowski space. Let $\bm{x},\bm{y}\in\mathbb{L}^{n}_{K}$
denote two points in the Lorentz model. Then, the length of the connecting
geodesic and, thereby, the distance is given by

$d_{\mathcal{L}}(\bm{x},\bm{y})=\frac{1}{\sqrt{-K}}\cosh^{-1}(K\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}),$
(15)

and the squared distance [24] by

$d^{2}_{\mathcal{L}}(\bm{x},\bm{y})=||\bm{x}-\bm{y}||_{\mathcal{L}}^{2}=\frac{2}{K}-2\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}.$
(16)

When calculating the distance of any point $\bm{x}\in\mathbb{L}^{n}_{K}$ to
the origin $\overline{\bm{0}}$, the equations can be simplified to

$d_{\mathcal{L}}(\bm{x},\overline{\bm{0}})=||\log^{K}_{\overline{\bm{0}}}(\bm{x})||,$
(17)

$d_{\mathcal{L}}^{2}(\bm{x},\overline{\bm{0}})=\frac{2}{K}(1+\sqrt{-K}x_{t}).$
(18)

#### Tangent space

The space around each point $\bm{x}$ on a differentiable manifold
$\mathcal{M}$ can be linearly approximated by the tangent space
$\mathcal{T}_{\bm{x}}\mathcal{M}$. It is a first-order approximation bridging
the gap to Euclidean space. This helps performing Euclidean operations, but it
introduces an approximation error, which generally increases with the distance
from the reference point. Let $\bm{x}\in\mathbb{L}^{n}_{K}$, then the tangent
space at point $\bm{x}$ can be expressed as

$\mathcal{T}_{\bm{x}}\mathbb{L}_{K}^{n}:=\\{\bm{y}\in\mathbb{R}^{n+1}~{}|~{}\langle\bm{y},\bm{x}\rangle_{\mathcal{L}}=0\\}.$
(19)

#### Exponential and logarithmic maps

Exponential and logarithmic maps are mappings between the manifold
$\mathcal{M}$ and the tangent space $\mathcal{T}_{\bm{x}}\mathcal{M}$ with
$\bm{x}\in\mathcal{M}$. The exponential map
$\exp^{K}_{\bm{x}}(\bm{z}):\mathcal{T}_{\bm{x}}\mathbb{L}_{K}^{n}\rightarrow\mathbb{L}^{n}_{K}$
maps a tangent vector $\bm{z}\in\mathcal{T}_{\bm{x}}\mathbb{L}_{K}^{n}$ on the
Lorentz manifold by

$\displaystyle\exp_{\bm{x}}^{K}(\bm{z})=\cosh(\alpha)\bm{x}+\sinh(\alpha)\frac{\bm{z}}{\alpha},\text{
with
}\alpha=\sqrt{-K}||\bm{z}||_{\mathcal{L}},~{}~{}||\bm{z}||_{\mathcal{L}}=\sqrt{{\langle\bm{z},\bm{z}\rangle}_{\mathcal{L}}}.$
(20)

The logarithmic map is the inverse mapping and maps a vector
$\bm{y}\in\mathbb{L}^{n}_{K}$ to the tangent space of $\bm{x}$ by

$\displaystyle\log_{\bm{x}}^{K}(\bm{y})=\frac{\cosh^{-1}(\beta)}{\sqrt{\beta^{2}-1}}\cdot(\bm{y}-\beta\bm{x}),\text{
with }\beta=K\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}.$ (21)

In the special case of working with the tangent space at the origin
$\overline{\bm{0}}$, the exponential map simplifies to

$\displaystyle\exp_{\overline{\bm{0}}}^{K}(\bm{z})=\frac{1}{\sqrt{-K}}\left[\cosh(\sqrt{-K}||\bm{z}||),~{}\sinh(\sqrt{-K}||\bm{z}||)\frac{\bm{z}}{||\bm{z}||}\right].$
(22)

#### Parallel transport

The parallel transport operation
$\mathrm{PT}^{K}_{{\bm{x}}\rightarrow{\bm{y}}}\left({\bm{v}}\right)$ maps a
vector $\bm{v}\in\mathcal{T}_{\bm{x}}\mathcal{M}$ from the tangent space of
$\bm{x}\in\mathcal{M}$ to the tangent space of $\bm{y}\in\mathcal{M}$. It
preserves the local geometry around the reference point by moving the points
along the geodesic connecting $\bm{x}$ and $\bm{y}$. The formula for the
Lorentz model is given by

$\displaystyle\mathrm{PT}^{K}_{{\bm{x}}\rightarrow{\bm{y}}}\left({\bm{v}}\right)$
$\displaystyle=\bm{v}-\frac{\langle\log^{K}_{\bm{x}}(\bm{y}),\bm{v}\rangle_{\mathcal{L}}}{d_{\mathcal{L}}(\bm{x},\bm{y})}(\log^{K}_{\bm{x}}(\bm{y})+\log^{K}_{\bm{y}}(\bm{x}))$
(23)
$\displaystyle=\bm{v}+\frac{\langle\bm{y},\bm{v}\rangle_{\mathcal{L}}}{\frac{1}{-K}-\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}}(\bm{x}+\bm{y}).$
(24)

#### Lorentzian centroid [24]

The weighted centroid with respect to the squared Lorentzian distance, which
solves
$\min_{\bm{\mu}\in\mathbb{L}^{n}_{K}}\sum_{i=1}^{m}\nu_{i}d^{2}_{\mathcal{L}}(\bm{x}_{i},\bm{\mu})$,
with $\bm{x}_{i}\in\mathbb{L}^{n}_{K}$ and $\nu_{i}\geq
0,\sum_{i=1}^{m}\nu_{i}>0$, is given by

$\bm{\mu}=\frac{\sum_{i=1}^{m}\nu_{i}\bm{x}_{i}}{\sqrt{-K}\left|||\sum_{i=1}^{m}\nu_{i}\bm{x}_{i}||_{\mathcal{L}}\right|}.$
(25)

#### Lorentz transformations

The set of linear transformations in the Lorentz model are called Lorentz
transformations. A transformation matrix
$\mathrm{\textbf{A}}^{(n+1)\times(n+1)}$ that linearly maps
$\mathbb{R}^{n+1}\rightarrow\mathbb{R}^{n+1}$ is called Lorentz transformation
if and only if
$\langle\mathrm{\textbf{A}}\bm{x},\mathrm{\textbf{A}}\bm{y}\rangle_{\mathcal{L}}=\langle\bm{x},\bm{y}\rangle_{\mathcal{L}}~{}\forall~{}\bm{x},\bm{y}\in\mathbb{R}^{n+1}$.
The set of matrices forms an orthogonal group $\bm{O}(1,n)$ called the Lorentz
group. As the Lorentz model only uses the upper sheet of the two-sheeted
hyperboloid, the transformations under consideration here lie within the
positive Lorentz group
$\bm{O}^{+}(1,n)=\\{\mathrm{\textbf{A}}\in\bm{O}(1,n):a_{11}>0\\}$, preserving
the sign of the time component $x_{t}$ of $\bm{x}\in\mathbb{L}^{n}_{K}$.
Specifically, here, the Lorentz transformations can be formulated as

$\bm{O}^{+}(1,n)=\\{\mathrm{\textbf{A}}\in\mathbb{R}^{(n+1)\times(n+1)}~{}|~{}\forall\bm{x}\in\mathbb{L}^{n}_{K}:\langle\mathrm{\textbf{A}}\bm{x},\mathrm{\textbf{A}}\bm{x}\rangle_{\mathcal{L}}=\frac{1}{K},(\mathrm{\textbf{A}}\bm{x})_{0}>0)\\}.$
(26)

Any Lorentz transformation can be decomposed into a Lorentz rotation and
Lorentz boost by polar decomposition
$\mathrm{\textbf{A}}=\mathrm{\textbf{R}}\mathrm{\textbf{B}}$ [33]. The former
rotates points around the time axis, using matrices given by

$\mathrm{\textbf{R}}=\left[\begin{array}[]{cc}1&\bm{0}^{T}\\\
\bm{0}&\tilde{\mathrm{\textbf{R}}}\end{array}\right],$ (27)

where $\bm{0}$ is a zero vector,
$\tilde{\mathrm{\textbf{R}}}^{T}\tilde{\mathrm{\textbf{R}}}=\mathrm{\textbf{I}}$,
and $\det(\tilde{\mathrm{\textbf{R}}})=1$. This shows that the Lorentz
rotations for the upper sheet lie in a special orthogonal subgroup
$\bm{SO}^{+}(1,n)$ preserving the orientation, while
$\tilde{\mathrm{\textbf{R}}}\in\bm{SO}(n)$. On the other side, the Lorentz
boost moves points along the spatial axis given a velocity
$\bm{v}\in\mathbb{R}^{n},||\bm{v}||<1$ without rotating them along the time
axis. Formally, the boost matrices are given by

$\mathrm{\textbf{B}}=\left[\begin{array}[]{cc}\gamma&-\gamma\bm{v}^{T}\\\
-\gamma\bm{v}&\mathrm{\textbf{I}}+\frac{\gamma^{2}}{1+\gamma}\bm{v}\bm{v}^{T}\end{array}\right],$
(28)

with $\gamma=\frac{1}{\sqrt{1-||\bm{v}||^{2}}}$. See Figure 4 for
illustrations of the Lorentz rotation and Lorentz boost.

#### Lorentz fully-connected layer

Recently, Chen et al. [5] showed that the linear transformations performed in
the tangent space [12, 35] can not apply all Lorentz transformations but only
a special rotation and no boost. They proposed a direct method in pseudo-
hyperbolic space222Chen et al. [5] note that their general formula is not
fully hyperbolic, but a relaxation in implementation, while the input and
output are still guaranteed to lie in the Lorentz model., which can apply all
Lorentz transformations. Specifically, let $\bm{x}\in\mathbb{L}^{n}_{K}$
denote the input vector and $\mathrm{\textbf{W}}\in\mathbb{R}^{m\times n+1}$,
$\bm{v}\in\mathbb{R}^{n+1}$ the weight parameters, then the transformation
matrix is given by

$f_{\bm{x}}(\mathrm{\textbf{M}})=f_{\bm{x}}\left(\left[\begin{array}[]{c}\bm{v}^{T}\\\
\mathrm{\textbf{W}}\end{array}\right]\right)=\left[\begin{array}[]{c}\frac{\sqrt{||\mathrm{\textbf{W}}\bm{x}||^{2}-1/K}}{\bm{v}^{T}\bm{x}}\bm{v}^{T}\\\
\mathrm{\textbf{W}}\end{array}\right].$ (29)

Adding other components of fully-connected layers, including normalization,
the final definition of the proposed Lorentz fully-connected layer becomes

$\bm{y}=\left[\begin{array}[]{c}\sqrt{||\phi(\mathrm{\textbf{W}}\bm{x},\bm{v})||^{2}-1/K}\\\
\phi(\mathrm{\textbf{W}}\bm{x},\bm{v})\end{array}\right],$ (30)

with operation function

$\phi(\mathrm{\textbf{W}}\bm{x},\bm{v})=\lambda\sigma(\bm{v}^{T}\bm{x}+\bm{b}^{\prime})\frac{\mathrm{\textbf{W}}\psi(\bm{x})+\bm{b}}{||\mathrm{\textbf{W}}\psi(\bm{x})+\bm{b}||},$
(31)

where $\lambda>0$ is a learnable scaling parameter and
$\bm{b}\in\mathbb{R}^{n},\psi,\sigma$ denote the bias, activation, and sigmoid
function, respectively.

In this work, we simplify the layer definition by removing the internal
normalization, as we use batch normalization. This gives following formula for
the Lorentz fully connected layer

$\bm{y}=\text{LFC}(\bm{x})=\left[\begin{array}[]{c}\sqrt{||\psi(\mathrm{\textbf{W}}\bm{x}+\bm{b})||^{2}-1/K}\\\
\psi(\mathrm{\textbf{W}}\bm{x}+\bm{b})\end{array}\right].$ (32)

#### Lorentz direct concatenation [40]

Given a set of hyperbolic points
$\\{\bm{x}_{i}\in\mathbb{L}^{n}_{K}\\}_{i=1}^{N}$, the Lorentz direct
concatenation is given by

$\bm{y}=\mathrm{HCat}(\\{\bm{x}_{i}\\}_{i=1}^{N})=\left[\sqrt{\sum_{i=1}^{N}x^{2}_{i_{t}}+\frac{N-1}{K}},\bm{x}^{T}_{1_{s}},\dots,\bm{x}^{T}_{N_{s}}\right]^{T},$
(33)

where $\bm{y}\in\mathbb{L}^{nN}_{K}\subset\mathbb{R}^{nN+1}$.

###  A.2 Poincaré ball

The n-dimensional Poincaré ball
$\mathbb{B}^{n}_{K}=(\mathcal{B}^{n},\mathfrak{g}_{\bm{x}}^{K})$ is defined by
the Riemannian metric
$\mathfrak{g}_{\bm{x}}^{K}=(\lambda^{K}_{\bm{x}})^{2}\mathrm{\textbf{I}}_{n}$,
where $\lambda_{\bm{x}}^{K}=2(1+K||\bm{x}||^{2})^{-1}$ and by the open ball

$\mathcal{B}^{n}=\\{\bm{x}\in\mathbb{R}^{n}~{}|~{}-K||\bm{x}||^{2}<1\\}.$ (34)

###  A.3 Mapping between models

Because of the isometry between models of hyperbolic geometry, points in the
Lorentz model can be mapped to the Poincaré ball by the following
diffeomorphism

$p_{\mathbb{L}^{n}_{K}\rightarrow\mathbb{B}^{n}_{K}}(\bm{x})=\frac{\bm{x_{s}}}{x_{t}+\frac{1}{\sqrt{-K}}}.$
(35)

##  Appendix B Proofs

###  B.1 Proofs for Lorentz hyperplane

This section contains the proof for the Euclidean reparameterization of the
Lorentz hyperplane proposed by Mishne et al. [32]. Unfortunately, the authors
only provided proof for the unit Lorentz model, i.e., assuming a curvature of
$K=-1$. However, in general, the curvature can be different $K<0$. That is why
we reproduce their proof for the general case.

#### Proof for Eq. 9

Let $a\in\mathbb{R}$, $\bm{z}\in\mathbb{R}^{n}$, and
$\bm{\overline{z}}\in\mathcal{T}_{\overline{\bm{0}}}\mathbb{L}_{K}^{n}=[0,a{\bm{z}/}{||\bm{z}||}]$.
Then, Mishne et al. [32] parameterize a point in the Lorentz model as follows

$\displaystyle\bm{p}\in\mathbb{L}^{n}_{K}$
$\displaystyle:=\exp_{\overline{\bm{0}}}\left(a\frac{\bm{z}}{||\bm{z}||}\right)$
(36)
$\displaystyle=\left[\frac{1}{\sqrt{-K}}\cosh(\alpha),~{}\sinh(\alpha)\frac{a\frac{\bm{z}}{||\bm{z}||}}{\alpha}\right].$
(37)

Now, with
$\alpha=\sqrt{-K}||a\frac{\bm{z}}{||\bm{z}||}||_{\mathcal{L}}=\sqrt{-K}a$ we
get

$\displaystyle\bm{p}$
$\displaystyle=\left[\cosh(\sqrt{-K}a)\frac{1}{\sqrt{-K}},~{}\sinh(\sqrt{-K}a)\frac{a\frac{\bm{z}}{||\bm{z}||}}{\sqrt{-K}a}\right]$
(38)
$\displaystyle=\frac{1}{\sqrt{-K}}\left[\cosh(\sqrt{-K}a),~{}\sinh(\sqrt{-K}a)\frac{\bm{z}}{||\bm{z}||}\right].$
(39)

This definition gets used to reparameterize the hyperplane parameter $\bm{w}$
as follows

$\displaystyle\bm{w}$
$\displaystyle:=\mathrm{PT}^{K}_{{\overline{\bm{0}}}\rightarrow{\bm{p}}}\left({\overline{\bm{z}}}\right)$
$\displaystyle=\overline{\bm{z}}+\frac{\langle\bm{p},\overline{\bm{z}}\rangle_{\mathcal{L}}}{\frac{1}{-K}-\langle\overline{\bm{0}},\bm{p}\rangle_{\mathcal{L}}}(\overline{\bm{0}}+\bm{p})$
$\displaystyle=[0,\bm{z}]^{T}+\frac{\langle\bm{p},[0,\bm{z}]^{T}\rangle_{\mathcal{L}}}{\frac{1}{-K}-\langle\overline{\bm{0}},\bm{p}\rangle_{\mathcal{L}}}(\overline{\bm{0}}+\bm{p})$
$\displaystyle=[0,\bm{z}]^{T}+\frac{\frac{1}{\sqrt{-K}}\sinh(\sqrt{-K}a)||\bm{z}||}{\frac{1}{-K}+\frac{1}{-K}\cosh(\sqrt{-K}a)}\cdot\frac{1}{\sqrt{-K}}\left[1+\cosh(\sqrt{-K}a),~{}\sinh(\sqrt{-K}a)\frac{\bm{z}}{||\bm{z}||}\right]$
$\displaystyle=[0,\bm{z}]^{T}+\frac{\sinh(\sqrt{-K}a)||\bm{z}||}{1+\cosh(\sqrt{-K}a)}\cdot\left[1+\cosh(\sqrt{-K}a),~{}\sinh(\sqrt{-K}a)\frac{\bm{z}}{||\bm{z}||}\right]$
$\displaystyle=\left[\sinh(\sqrt{-K}a)||\bm{z}||,~{}\bm{z}+\frac{\sinh^{2}(\sqrt{-K}a)}{1+\cosh(\sqrt{-K}a)}\bm{z}\right]$
$\displaystyle=\left[\sinh(\sqrt{-K}a)||\bm{z}||,~{}\bm{z}+\frac{\cosh^{2}(\sqrt{-K}a)-1}{1+\cosh(\sqrt{-K}a)}\bm{z}\right]$
$\displaystyle=[\sinh(\sqrt{-K}a)||\bm{z}||,~{}\cosh(\sqrt{-K}a)\bm{z}].$

#### Proof for Eq. 10

After inserting Eq. 9 into Eq. 8 and solving the inner product, the hyperplane
definition becomes

$\tilde{H}_{\bm{z},a}=\\{\bm{x}\in\mathbb{L}^{n}_{K}~{}|~{}\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a)~{}||\bm{z}||~{}x_{t}=0\\}.$
(40)

###  B.2 Proof for distance to Lorentz hyperplane

#### Proof for Theorem 1

To proof the distance to a hyperplane in the Lorentz model, we follow the
approach of Cho et al. [6] and utilize the hyperbolic reflection. The idea is,
that a hyperplane defines a reflection that interchanges two half-spaces.
Therefore, the distance from a point $\bm{x}\in\mathbb{L}^{n}_{K}$ to the
hyperplane $H_{\bm{w},\bm{p}}$ can be calculated by halving the distance to
its reflection in the hyperplane $\bm{x}\rightarrow\bm{y}_{\bm{w}}$

$d_{\mathcal{L}}(\bm{x},H_{\bm{w},\bm{p}})=\frac{1}{2}d_{\mathcal{L}}(\bm{x},\bm{y}_{\bm{w}}).$
(41)

The hyperbolic reflection is well-known in the literature [14] and can be
formulated as

$\bm{y}_{\bm{w}}=\bm{x}+\frac{2\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}\bm{w}}{\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}},$
(42)

where $\bm{w}$ is the perpendicular vector to the hyperplane and
$\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}>0$. Now, inserting Equation 42 into
Equation 41 we can compute the distance to the hyperplane as follows

$\displaystyle d_{\mathcal{L}}(\bm{x},H_{\bm{w},\bm{p}})$
$\displaystyle=\frac{1}{2\sqrt{-K}}\cosh^{-1}(K\langle\bm{x},\bm{y}_{\bm{w}}\rangle_{\mathcal{L}})$
$\displaystyle=\frac{1}{2\sqrt{-K}}\cosh^{-1}(K\langle\bm{x},\bm{x}+\frac{2\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}\bm{w}}{\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}}\rangle_{\mathcal{L}})$
$\displaystyle=\frac{1}{2\sqrt{-K}}\cosh^{-1}(2K\langle\bm{x},\bm{x}\rangle_{\mathcal{L}}+K\langle\bm{x},\frac{\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}\bm{w}}{\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}}\rangle_{\mathcal{L}})$
$\displaystyle=\frac{1}{2\sqrt{-K}}\cosh^{-1}\left(K\frac{1}{K}+2K\left(\frac{\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}}{\sqrt{\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}}}\right)^{2}\right)$
$\displaystyle=\frac{1}{2\sqrt{-K}}\cosh^{-1}\left(1+2\left(\sqrt{-K}\frac{\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}}{\sqrt{\langle\bm{w},\bm{w}\rangle_{\mathcal{L}}}}\right)^{2}\right)$
$\displaystyle=\frac{1}{\sqrt{-K}}\left|\sinh^{-1}\left(\sqrt{-K}\frac{\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}}{||\bm{w}||_{\mathcal{L}}}\right)\right|,$

which gives the final formula:

$d_{\mathcal{L}}(\bm{x},H_{\bm{w},\bm{p}})=\frac{1}{\sqrt{-K}}\left|\sinh^{-1}\left(\sqrt{-K}\frac{\langle\bm{w},\bm{x}\rangle_{\mathcal{L}}}{||\bm{w}||_{\mathcal{L}}}\right)\right|$
(43)

Comparing Eq. 43 to the formula of Cho et al. [6] shows that the distance
formula of hyperplanes in the unit Lorentz model can be extended easily to the
general case by inserting the curvature parameter $K$ at two places.

Finally, defining $\bm{w}$ with the aforementioned reparameterization

$\bm{w}:=\mathrm{PT}^{K}_{{\overline{\bm{0}}}\rightarrow{\bm{p}}}\left({\overline{\bm{z}}}\right)=[\sinh(\sqrt{-K}a)||\bm{z}||,\cosh(\sqrt{-K}a)\bm{z}],$
(44)

and solving the inner products, gives our final distance formula

$\displaystyle
d_{\mathcal{L}}(\bm{x},\tilde{H}_{\bm{z},a})=\frac{1}{\sqrt{-K}}\left|\sinh^{-1}\left(\sqrt{-K}\frac{\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a)~{}||\bm{z}||~{}x_{t}}{\sqrt{||\cosh(\sqrt{-K}a)\bm{z}||^{2}-(\sinh(\sqrt{-K}a)||\bm{z}||)^{2}}}\right)\right|.$
(45)

###  B.3 Proof for logits in the Lorentz MLR classifier

#### Proof for Theorem 2

Following Lebanon and Lafferty [27], given input $\bm{x}\in\mathbb{R}^{n}$ and
$C$ classes, the Euclidean MLR logits of class $c\in\\{1,...,C\\}$ can be
expressed as

$v_{\bm{w}_{c}}(\bm{x})=\text{sign}(\langle\bm{w}_{c},\bm{x}\rangle)||\bm{w}_{c}||d(x,H_{\bm{w}_{c}}),~{}~{}\bm{w}_{c}\in\mathbb{R}^{n},$
(46)

where $H_{\bm{w}_{c}}$ is the decision hyperplane of class $c$.

Replacing the Euclidean operations with their counterparts in the Lorentz
model yields logits for $\bm{x}\in\mathbb{L}^{n}_{K}$ as follows

$v_{\bm{w}_{c},\bm{p}_{c}}(\bm{x})=\text{sign}(\langle\bm{w}_{c},\bm{x}\rangle_{\mathcal{L}})||\bm{w}_{c}||_{\mathcal{L}}d_{\mathcal{L}}(x,H_{\bm{w}_{c},\bm{p}_{c}}),$
(47)

with
$\bm{w}_{c}\in\mathcal{T}_{\bm{p}_{c}}\mathbb{L}_{K}^{n},\bm{p}_{c}\in\mathbb{L}^{n}_{K}$,
and $\langle\bm{w}_{c},\bm{w}_{c}\rangle_{\mathcal{L}}>0$.

Inserting Eq. 43 into Eq. 47 gives a general formula without our
reparameterization

$v_{\bm{w}_{c},\bm{p}_{c}}(\bm{x})=\frac{1}{\sqrt{-K}}~{}\text{sign}(\langle\bm{w}_{c},\bm{x}\rangle_{\mathcal{L}})||\bm{w}_{c}||_{\mathcal{L}}\left|\sinh^{-1}\left(\sqrt{-K}\frac{\langle\bm{w}_{c},\bm{x}\rangle_{\mathcal{L}}}{||\bm{w}_{c}||_{\mathcal{L}}}\right)\right|.$
(48)

Now, we reparameterize $\bm{w}$ with Eq. 44 again, which gives

$\displaystyle\alpha:=\langle\bm{w}_{c},\bm{x}\rangle_{\mathcal{L}}=\cosh(\sqrt{-K}a)\langle\bm{z},\bm{x}_{s}\rangle-\sinh(\sqrt{-K}a),$
(49)

$\displaystyle\beta:=||\bm{w}_{c}||_{\mathcal{L}}=\sqrt{||\cosh(\sqrt{-K}a)\bm{z}||^{2}-(\sinh(\sqrt{-K}a)||\bm{z}||)^{2}},$
(50)

with $a\in\mathbb{R}$ and $\bm{z}\in\mathbb{R}^{n}$. Finally, we obtain the
equation in Theorem 2:

$v_{\bm{z}_{c},a_{c}}(\bm{x})=\frac{1}{\sqrt{-K}}~{}\mathrm{sign}(\alpha)\beta\left|\sinh^{-1}\left(\sqrt{-K}\frac{\alpha}{\beta}\right)\right|.$
(51)

##  Appendix C Additional experimental details

###  C.1 Classification

#### Datasets

For classification, we employ the benchmark datasets CIFAR-10 [23], CIFAR-100
[23], and Tiny-ImageNet [26]. The CIFAR-10 and CIFAR-100 datasets each contain
60,000 $32\times 32$ colored images from 10 and 100 different classes,
respectively. We use the dataset split implemented in PyTorch, which includes
50,000 training images and 10,000 testing images. Tiny-ImageNet is a small
subset of the ImageNet [7] dataset, with 100,000 images of 200 classes
downsized to $64\times 64$ images. Here, we use the official validation split
for testing our models.

#### Settings

Table 4 summarizes the hyperparameters we adopt from DeVries and Taylor [8] to
train all ResNet classification models. Additionally, we use standard data
augmentation methods in training, i.e., random mirroring and cropping.
Regarding the feature clipping in hybrid HNNs, we tune the feature clipping
parameter $r$ between 1.0 and 5.0 and find that, for most experiments, the
best feature clipping parameter is $r=1$. Only the Lorentz hybrid ResNet
performs best with $r=4$ on Tiny-ImageNet, and $r=2$ on CIFAR-100 with lower
embedding dimensions (see ablation experiment). Overall, we observe that the
hybrid Lorentz ResNet has fewer gradient issues, allowing for higher clipping
values.

Table 4: Summary of hyperparameters used in training classification models. Hyperparameter | Value
---|---
Epochs | 200
Batch Size | 128
Learning Rate (LR) | 1e-1
Drop LR epochs | 60, 120, 160
Drop LR gamma | 0.2
Weight Decay | 5e-4
Optimizer | (Riemannian)SGD
Floating point Precision | 32 bit
GPU Type | RTX A5000
Hyperbolic curvature $K$ | -1

###  C.2 Generation

#### Datasets

For image generation, we use the aforementioned CIFAR-10 [23] and CIFAR-100
[23] datasets again. Additionally, we employ the CelebA [29] dataset, which
includes colored $64\times 64$ images of human faces. Here, we use the PyTorch
implementation, containing 162,770 training images, 19,867 validation images,
and 19,962 testing images.

#### Settings

For hyperbolic and Euclidean models, we use the same architecture (see Table
5) and training hyperparameters (see Table 6). We employ a vanilla VAE similar
to Ghosh et al. [13] as the baseline Euclidean architecture (E-VAE). For the
hybrid model, we replace the latent distribution of the E-VAE with the
hyperbolic wrapped normal distribution in the Lorentz model [34] and the
Poincaré ball [31], respectively. Replacing all layers with our proposed
hyperbolic counterparts yields the fully hyperbolic model. Furthermore, we set
the curvature $K$ for the Lorentz model to $-1$ and for the Poincaré ball to
$-0.1$.

We evaluate the VAEs by employing two versions of the FID using the PyTorch
implementation of Seitzer [44]:

  1. 1.

The reconstruction FID gives a lower bound on the generation quality. It is
calculated by comparing test images with reconstructed validation images. As
the CIFAR datasets have no official validation set, we exclude a fixed random
portion of 10,000 images from the training set.

  2. 2.

The generation FID measures the generation quality by comparing random
generations from the models’ latent space with the test set.

Table 5: Vanilla VAE architecture employed in all image generation experiments. Convolutional layers have a kernel size of $3\times 3$ and transposed convolutional layers of $4\times 4$. $s$ and $p$ denote stride and zero padding, respectively. The MLR in the Euclidean model is mimicked by a $1\times 1$ convolutional layer. Layer | CIFAR-10/100 | CelebA
---|---|---
Encoder: |  |
$\rightarrow$ Proj${}_{\mathbb{R}^{n}\rightarrow\mathbb{L}^{n}_{K}}$ | 32$\times$32$\times$3 | 64$\times$64$\times$3
$\rightarrow$ Conv64,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 16$\times$16$\times$64 | 32$\times$32$\times$64
$\rightarrow$ Conv128,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 8$\times$8$\times$128 | 16$\times$16$\times$128
$\rightarrow$ Conv256,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 4$\times$4$\times$256 | 8$\times$8$\times$256
$\rightarrow$ Conv512,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 2$\times$2$\times$512 | 4$\times$4$\times$512
$\rightarrow$ Flatten | 2048 | 8192
$\rightarrow$ FC-Meand | 128 | 64
$\rightarrow$ FC-Vard $\rightarrow$ Softplus | 128 | 64
Decoder: |  |
$\rightarrow$ Sample | 128 | 64
$\rightarrow$ FC32768 $\rightarrow$ BN $\rightarrow$ ReLU | 32768 | 32768
$\rightarrow$ Reshape | 8$\times$8$\times$512 | 8$\times$8$\times$512
$\rightarrow$ ConvTr256,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 16$\times$16$\times$256 | 16$\times$16$\times$256
$\rightarrow$ ConvTr128,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | 32$\times$32$\times$128 | 32$\times$32$\times$128
$\rightarrow$ ConvTr64,s2,p1 $\rightarrow$ BN $\rightarrow$ ReLU | - | 64$\times$64$\times$64
$\rightarrow$ Conv64,s1,p1 | 32$\times$32$\times$64 | 64$\times$64$\times$64
$\rightarrow$ MLR | 32$\times$32$\times$3 | 64$\times$64$\times$3
Table 6: Summary of hyperparameters used in training image generation models. Hyperparameter | CIFAR-10/100 | CelebA
---|---|---
Epochs | 100 | 70
Batch Size | 100 | 100
Learning Rate | 5e-4 | 5e-4
Weight Decay | 0 | 0
KL Loss Weight | 0.024 | 0.09
Optimizer | (Riem)Adam | (Riem)Adam
Floating point Precision | 32 bit | 32 bit
GPU Type | RTX A5000 | RTX A5000