# Atlas flow : compatible local structures on the manifold

Taejin Paik  ,  Jaemin Park  and  Jung Ho Park

###### Abstract.

In this paper, we focus on the intersections of a manifold’s local structures
to analyze the global structure of a manifold. We obtain local regions on data
manifolds such as the latent space of StyleGAN2, using Mapper, a tool from
topological data analysis. We impose gluing compatibility conditions on
overlapping local regions, which guarantee that the local structures can be
glued together to the global structure of a manifold. We propose a novel
generative flow model called Atlas flow that uses compatibility to reattach
the local regions. Our model shows that the generating processes perform well
on synthetic dataset samples of well-known manifolds with noise. Furthermore,
we investigate the style vector manifold of StyleGAN2 using our model.

##  1\. Introduction

A popular method to improve data-driven models such as deep neural networks
improves the data representation using manifold learning and density
estimation.

Various methods have been proposed for manifold learning assuming the manifold
hypothesis [1]. In statistics, dimension reduction methods, such as PCA,
Isomap [2], and tSNE [3], have been widely used to infer the latent manifold.
Furthermore, various deep neural networks have been studied to apply manifold
learning techniques such as autoencoder [4, 5, 6, 7, 8].

On the other hand, density estimation is as important as manifold learning.

Generative models are practical examples of methods that learn density either
implicitly [9, 10, 11, 12, 5], or explicitly [13, 14].

Recently, a flow-based framework[15] was proposed, called manifold-learning
flow to perform both manifold learning and density estimation. In this
setting, there are two flow-based maps: one for manifold learning, and one for
density estimation. Using these two maps, one can often identify the full data
manifold and generate sample points on this manifold using the density
function. To improve this philosophy, we studied a new flow-based model for a
general manifold which includes spaces such as spheres.

In order to analyze a manifold with a complex shape, theoretically, we have to
describe the manifold with the atlas [16]. An atlas refers to a collection of
coordinate charts, which are local structures consisting of local regions and
maps. A coordinate map is a diffeomorphism from a local region of a manifold
to a Euclidean space preserving topological and geometrical properties. Using
this approach, several studies [17, 18] trained models to find the local parts
of a manifold. However, these studies used partitions as multi-charts, so
their components trained on local regions are independent of each other. In
short, these studies investigated different non-overlapping components
yielding independent local structures.

Hence, we study our method to match the concept of the manifold with
coordinate charts that have overlapping local regions appropriately. By
allowing overlap of regions, trained local structures can be combined and
hence used to cover the entire data manifold. Thus, it should be noted that
allowing overlapping regions is a more general approach than partitioning
(even if they both consider local geometry). We adopt an idea of topological
data analysis (TDA), namely Mapper, in order to select appropriate local
regions. When a prior data manifold is unknown, Mapper is used to infer the
global topological structure of a manifold by considering the connectivity of
local regions.

We suggest the term “compatibility" in order to describe the smoothness of
overlaps of mutually trained and distinct local structures. Through a training
process for coordinate maps, we intend reconstruction points to coincide for
compatible regions. Moreover, we also study how to decompose densities on
overlapping regions using disintegration, which is a notion of probability
theory. In section 3, we explain our story of study and propose a novel method
Atlas flow. In section 4, we conduct two types of experiments.

We first apply our method to synthetic datasets which consist of sample points
on manifolds embedded in $\mathbb{R}^{3}$, and then compare to the original
datasets. Moreover, we apply our method to the latent space of StyleGAN2 which
was pre-trained on FFHQ dataset. Our contributions are as follows:

  1. (1)

Our method is a generative model working under the assumption that the dataset
is lying on a manifold. In principle, we can handle even complicated manifolds
using our Atlas flow model.

  2. (2)

This includes a new manifold learning algorithm that works for a wider class
of manifolds. Compatibility ensures successful inference of the overall
manifold by combining local regions.

  3. (3)

Furthermore, we also obtain a measure-theoretic decomposition method of a
density function using the concept of disintegration. Each decomposed density
is supported on a local region obtained from manifold learning. Compatibility
ensures successful inference of the overall density function by integrating
decomposed densities.

##  2\. Related works

###  2.1. Flow-based models

Normalizing flow (NF) [13, 19, 14, 20] is one of the most elegant frameworks
in generative models. NF can be considered as an approach to transforming a
trivial density into a non-trivial one. The essence of NF is that its encoder
is a diffeomorphism. A diffeomorphism means a smooth bijection whose inverse
is also smooth. By the definition of diffeomorphism, the decoder of NF becomes
the inverse map of the encoder. Thus NF can preserve geometrical information
between a data manifold and corresponding latent space. Moreover,
diffeomorphisms have the nice property that their compositions are also
diffeomorphisms. This property enhances the ability of NF to learn non-trivial
density.

Since the first NF assumes the data manifold is a Euclidean space, a branch of
study [19, 21, 22] generalizes the latent manifold from a Euclidean space to
Riemannian manifolds. Thus, one can estimate the appropriate density on a
manifold when prior knowledge of the data manifold exists. The manifold-
learning flow [15] is based on these ideas and combines manifold learning and
density learning. Manifold-learning flow consists of two parts: outer and
inner networks. The outer network is an autoencoder based on NF and learns the
manifold structure. The inner network learns the density which is tractable in
this setting.

###  2.2. Multi-Chart Manifold Learning

We need to consider a collection of local domains covering the manifold; we
allow overlap in this covering. Hence the knowledge of such a collection is of
significant interest to many researchers. Estimating the latent dimension has
been actively studied by many researchers. Nowadays, some research [17, 18]
suggests that studying a collection of local parts is worthwhile.
Topologically-informed atlas learning [17] splits a manifold to preserve
topological information such as $k$-dimensional holes. Multi-chart flows [18]
which is similar to our work divides a manifold into local partitions and
learns each coordinate map using the Manifold-learning flow [15]. We propose
an algorithm to analyze a manifold by using a collection of local domains, not
partitions. This makes our work different from Multi-chart flows and necessary
to deal with overlapping charts.

Figure 1. Framework scheme of Atlas flow. A data manifold is covered by
coordinate domains obtained from Mapper. Each coordinate domain
$\mathcal{M}\cap U_{k}$ maps to $V_{k}\subset\mathbb{R}^{n}$ and reconstructed
points are exactly on the submanifold. Finally, $V_{k}$ maps to $W_{k}$ with
tractable density (standard normal). Generating process is in reverse order:
sampling points from $W_{k}$ and mapping to $\mathcal{M}$ by
$\varphi_{k}^{-1}\circ\gamma_{k}^{-1}$.

##  3\. Method

###  3.1. Motivation

(a)

(b)

(c)

(d)

(e)

Figure 2. (A) The original torus dataset. (B) Estimation of a torus by using
single chart $\mathcal{M}$-flow. (C) Estimation of density on torus without
constraints densities of overlapping regions. (D) The original Trefoil knot
dataset. (E) Estimation of a trefoil knot by using multiple charts with
overlapping regions without constraints.

The Flow layer corresponds to a diffeomorphism which is a invertible
differentiable function preserving smooth structure. In many manifolds, it can
be proved mathematically that there are no diffeomorphisms from the manifold
to any subset of Euclidian space $\mathbb{R}^{n}$ where $n$ is the dimension
of the manifold . For instance, there are no diffeomorphisms from the
$2-$dimensional torus to any subset of $\mathbb{R}^{2}$ since their
topological structures are different. Therefore, it can be very difficult for
a flow model to learn a whole manifold if the manifold is topologically
complicated. We conducted several experiments of manifold learning using a
flow model on the $2-$dimensional torus and confirmed that the quality of
sampling using the model was not good. (see Figure 2(b).)

By the definition of a differentiable manifold, there are open subsets $U_{i}$
of a manifold $\mathcal{M}$ and homeomorphisms $\varphi_{i}$ from $U_{i}$ to
an open subset of some Euclidean space $\mathbb{R}^{n}$, such that
$\varphi_{i}\circ\varphi_{j}^{-1}$ is diffeomorphism for any $i$ and $j$.
Given this idea, we think it is natural to use multiple flow models for the
manifold learning task. Multi-chart flows [18] uses multiple flow models after
dividing data points. It was confirmed in our individual experiments that
using more than one flow model for manifold learning improves the results a
lot in the sense that the sampled points usually do not escape largely from
the original manifold in our toy example experiments. However, we also found
that if we use partitions that do not allow overlap, the model does not work
well, especially on the boundaries of the partitions. (see Figure 2(b).) In
some cases, sampling quality itself is not good, or more commonly, there is a
difference between sampled points in other partitions, resulting in the
disconnected or not smooth manifold. (see Figure 2(b).)

We solve this problem using an open cover, a family of open subsets
overlapping each other. We add a loss term to make the parts of a manifold
attached together smoothly using the data points on the overlapping regions.
To get an open cover of a manifold, we used Mapper. [23] One can also use soft
clustering assignment methods to get an open cover. However, additional work
is needed such as setting a threshold for probability.

After obtaining an open cover, manifold learning is done for each open set. We
could confirm that allowing overlap between open sets can improve the manifold
learning task. However, if the density estimation is done in each open set,
then the density on overlapping regions becomes higher than what it should be
(see Figure 2(c)). We took the disintegration concept from probability theory
so that the probability is well estimated in the overlapping regions also.

###  3.2. Mapper

Mapper [23] is a tool for the representation of a data manifold as a network
while preserving some topological features. The nodes of a Mapper network
correspond to a simpler subset of the data manifold and the connection between
the nodes describes the global structure of the data manifold. Based on this
property, we developed a method obtaining global geometry and density on the
data manifold via local inferences.

We consider a topological manifold $\mathcal{M}$ and a continuous function
$h:\mathcal{M}\to\mathbb{R}$ called a lens function. Given an interval in
$h(\mathcal{M})\subset\mathbb{R}$, we extract a subset of $\mathcal{M}$ whose
values for $h$ lie in the interval. Using a clustering method (e.g.,
hierarchical linkage clustering, DBSCAN, etc.), this subset is decomposed to
form clusters, which represent nodes of the Mapper network. By repeating this
process for intervals covering $h(\mathcal{M})$, we can obtain various nodes.
The nodes are connected if corresponding clusters share some data points.

Theoretically, nodes in the Mapper network correspond to critical values if
$h$ satisfies some regularity conditions in the sense of Morse Theory [24].
Moreover, the original manifold $\mathcal{M}$ and the Mapper network are
topologically equivalent if a family of clusters forms a good cover [25]. For
more detailed descriptions, see Appendix A.1.

###  3.3. Atlas flow

Consider a data-generating process on an $n$-dimensional manifold
$\mathcal{M}\subset\mathbb{R}^{d}$ with probability $p^{\mathcal{M}}$ on
$\mathcal{M}$ and assume that we have a finite sampled dataset with some
noise. We introduce a novel generative model that contains the geometric and
topological information of $\mathcal{M}$ and the probabilistic/statistical
information of $p^{\mathcal{M}}$. We call this model Atlas flow. Figure 1
illustrates framework of our model.

Given a dataset, we infer the shape of $\mathcal{M}$ and the density of
$p^{\mathcal{M}}$ under the assumption that $\mathcal{M}$ is $n$-dimensional
compact Riemannian manifold embedded in $\mathbb{R}^{d}$.

####  3.3.1. Manifold learning

Differential manifold theory ensures that there exists a local coordinate
chart system $\\{U_{k},\varphi_{k}\\}_{k=1}^{L}$ of a subset of
$\mathbb{R}^{d}$ such that the coordinate domains $\\{U_{k}\\}$ cover
$\mathcal{M}$, i.e.

(1) $\mathcal{M}\subset\bigcup_{k=1}^{L}U_{k}\subset\mathbb{R}^{d}\;\text{ and
}\;\mathcal{M}\cap U_{k}\neq\phi$

and $\varphi_{k}:U_{k}\to
V_{k}\times\tilde{V}_{k}\subset\mathbb{R}^{n}\times\mathbb{R}^{d-n}$ is a
coordinate map with

(2) $\varphi_{k}(\mathcal{M}\cap U_{k})=V_{k}\times\\{0\\}^{d-n}$

for each $k=1,\dots,L$. (see Appendix A.2.) We note that $\varphi_{k}$ is a
diffeomorphism between coordinate domains $U_{k}\cap\mathcal{M}$ and a latent
space $V_{k}\subset\mathbb{R}^{n}$. Also, $\mathcal{M}$ can be covered by the
finite number of coordinate domains since we assume $\mathcal{M}$ is compact.

Each node of the Mapper construction corresponds to a coordinate domain
$\\{U_{k}\\}_{k=1}^{L}$. Combining all nodes in this correspondence, we get a
covering of $\mathcal{M}$. Then we use a flow-based model to train
$\varphi_{k}$ on $U_{k}$ in several steps.

Since a flow-based model largely depends on an initial guess, an appropriate
initial guess is required to stabilize the model. We first pre-trained
$\varphi_{k}$ with a dimension reduction method such that the image of
$\varphi_{k}$ is equal to the result of the dimension reduction (Algorithm 1).
We found that Isomap fits our flow model well, presumably because Isomap
preserves the simple structure of the coordinate domain well.

We set the reconstruction loss for $x\in U_{k}$ as

$\displaystyle\mathcal{L}_{\text{recon}}(x)$
$\displaystyle=\|x-\mathrm{Recon}_{k}(x)\|_{2}$ (3)
$\displaystyle=\|x-(\varphi_{k}^{-1}\circ\mathrm{Proj}\circ\varphi_{k})(x)\|_{2},$

where
$\mathrm{Proj}:\mathbb{R}^{n}\times\mathbb{R}^{d-n}\to\mathbb{R}^{n}\times\\{0\\}^{d-n}$
is the orthogonal projection map (Algorithm 2). We define the pairwise
distance loss defined for a mini-batch $\mathcal{B}$ as

(4)
$\mathcal{L}_{\text{dist}}=\mathbb{E}\left[(D_{ij}-\|x_{i}-x_{j}\|_{2})^{2}\right],$

where $x_{i},x_{j}\in\mathcal{B}$ and $D=[D_{ij}]$ is the pairwise distance
matrix whose entries are geodesic distances on the $k$nn graph of Isomap
(Algorithm 3). Then we combined the two losses with loss weight hyperparameter
$\lambda$ as follows

(5)
$\mathcal{L}=\lambda\mathcal{L}_{\text{dist}}+(1-\lambda)\mathcal{L}_{\text{recon}}.$

The reason for using pairwise distance is that we noticed that the embedded
Isomap result is sometimes far from ideal embedding to the low dimensional
space. This can be caused by the initial guess or by an embedding into a too
low-dimensional space. Since the embedding of Isomap is determined by the
pairwise distance, we decided to use the pairwise distance directly. We also
observed that sudden change of the loss may confuse the model, and that’s why
we slowly decrease the $\lambda$. With pre-training by Isomap, this process
remarkably stabilizes the training process of our flow model.

We note that coordinate domains may overlap. In differential manifold theory,
compatibility between coordinate maps on the overlapping regions is necessary.
The compatibility makes it possible for local coordinates to extend to the
global structure of a manifold. Thus, it is required to fine-tune
$\varphi_{k}$ for compatibility. Since data points are sampled on the manifold
with noise and $\varphi_{k}$ is an approximated coordinate map, the
reconstructions of a point $x$ by distinct coordinate maps may differ. We
denote the expected point of $x$ by $\hat{x}$, which is obtained as the
average of the projections of $x$ to the estimated data manifold in each
$U_{k}$. In the final step of manifold learning, we adjusted $\varphi_{k}$ so
that reconstructions become equal on the overlapping regions by reducing
compatibility loss

(6) $\mathcal{L}_{\text{comp}}(x)=\|x-\hat{x}\|_{2}$

(See Figure 3, Algorithm 4). The process for compatibility is necessary since
the reconstructed coordinate domains may cross each other. (see Figure 2(e).)

Figure 3. Reconstruction points $\mathrm{Recon}_{k}(x)$ are exactly on the
reconstructed coordinate domain $\mathrm{Recon}_{k}(U_{k})$. These may not
coincide initially. Thus we set expected point
$\hat{x}=(\mathrm{Recon}_{1}(x)+\mathrm{Recon}_{2}(x))/2$ and adjust
coordinate maps $\varphi_{k}$ so that $\mathrm{Recon}_{k}(x)=\hat{x}$.

A naive approach for the compatibility loss is adding a loss term based on the
distance between the reconstructed sample points. However, this approach
requires heavy memory since more than two flow models should be used and
updated at the same time. Instead, we use only one model while calculating
expected points, processing inference, and updating the model. Using this
method, we can save a lot of memory while calculating compatibility loss.

####  3.3.2. Density estimation

From now, we estimate the density on $\mathcal{M}$ in terms of
$p^{\mathcal{M}}$. Since we trained the manifold structure by local coordinate
charts in the previous subsection, the density should be estimated on each
coordinate domain and then combined into the whole density on $\mathcal{M}$.
Thus we needed a appropriate way to decompose density into local densities on
coordinate domains. Decomposition of density function was difficult to apply
in practice even if it is well established theoretically. Thus we decomposed
probability measure instead of density function using disintegration of which
concept is based on probability theory[26]. The decomposition by
disintegration is a novel approach that has never been before.

######  Theorem 3.1 (Disintegration theorem).

For a partition $\\{U_{k^{\prime}}^{\prime}\\}_{k^{\prime}=1}^{L^{\prime}}$ of
$\mathcal{M}$, the probability $p^{\mathcal{M}}$ can be decomposed as

(7)
$p^{\mathcal{M}}=\sum_{k^{\prime}=1}^{L^{\prime}}p^{U^{\prime}}_{k^{\prime}}\nu(k^{\prime}),$

where $p^{U^{\prime}}_{k^{\prime}}$ is a probability on $\mathcal{M}$ with
$p^{U^{\prime}}_{k^{\prime}}(U^{\prime}_{k^{\prime}})=1$ and $\nu$ is the
push-forward measure of $p^{\mathcal{M}}$ by the canonical quotient map
$\mathcal{M}\to\\{1,\dots,L^{\prime}\\}$.

Using disintegration theorem, we can decompose $p^{\mathcal{M}}$ into

(8) $p^{\mathcal{M}}=\sum_{k=1}^{L}c_{k}p^{U}_{k},$

where $c_{k}$ is a scaling constant and $p^{U}_{k}$ is a probability on
$\mathcal{M}$ with $p^{U}_{k}(U_{k})=1$.

We state more details of disintegration in Appendix A.3.

For each $k=1,\dots,L$, consider learnable diffeomorphisms
$\gamma_{k}:V_{k}\to W_{k}$ called density transformation map, where
$W_{k}\subset\mathbb{R}^{n}$ adopts probability $p^{W}_{k}$ according to the
standard normal distribution, i.e. another latent space with tractable
density. We then train $\gamma_{k}$ by negative log-likelihood loss
$\mathcal{L}_{\text{density}}$ of $p^{W}_{k}$. The value of $p^{W}_{k}$ at the
latent variable from the sampled data point is calculated from the following
formula (Algorithm 5). For $x\in U_{k}$,

$\displaystyle p^{U}_{k}(x)$
$\displaystyle=p^{V}_{k}(\varphi_{k}(x))\left|\det[J_{\varphi_{k}}^{T}(\varphi_{k}(x))J_{\varphi_{k}}(\varphi_{k}(x))]\right|^{\frac{1}{2}}$
$\displaystyle=p^{W}_{k}(\gamma_{k}(\varphi_{k}(x)))\left|\det
J_{\gamma_{k}}(\gamma_{k}(\varphi_{k}(x)))\right|$ (9)
$\displaystyle\qquad\times\left|\det[J_{\varphi_{k}}^{T}(\varphi_{k}(x))J_{\varphi_{k}}(\varphi_{k}(x))]\right|^{\frac{1}{2}},$

where $J_{*}$ is the Jacobian matrix of a function $*$.

The last term of (9) only depends on the coordinate map, not on the density
transforming map. Calculating the last term of (9) is also heavy, so we omit
that term in (Algorithm 5).

We first trained $p^{U}_{k}$ by data samples in $U_{k}$. However, this process
provoked serious problem that the density on the overlapping regions becomes
higher since the densities are also overlapped. Therefore we noticed that each
$p^{U}_{k}$ should have a lower density in the overlapping regions in $U_{k}$
by the construction of $p^{U}_{k}$. (Appendix A.3.) In order to consider this
property to learning $p^{U}_{k}$, we used a special batch technique. We denote
by $m_{x}$ such that $x$ is contained in $m_{x}$ coordinate domains and let
$m_{k}$ be the least common multiplier of $\\{m_{x}:x\in U_{k}\\}$. Then we
made mini-batches for training $\gamma_{k}$ by bootstrapping according to a
density on $x\in U_{k}$ as $m_{x}/m_{k}$. This batch technique allows that the
density of $p^{U}_{k}$ gets lower in the overlapping region inversely
proportional to the number of overlapping coordinate domains.

##  4\. Experiment

We used our method on some datasets, ranging from the well-known manifolds
such as trefoil knot and torus in $\mathbb{R}^{3}$, to a not well-known space
such as $\mathcal{W}$ space of StyleGAN2. For the well-known manifolds, we
consider generated sample points to evaluate the performance. For the
$\mathcal{W}$ space, we sampled points on it and generated images from those
points. Furthermore, we demonstrated that the compatibility of coordinate maps
is well considered in our Atlas flow model from the observation that
overlapping coordinate domains are topologically the same.

Every flow model in our experiments is based on rational-quadratic splines.
The number of layers depends on the datasets. Usually, we choose $11-13$
layers per one flow model. For more experimental detail, please refer to
Appendix A.4.

###  4.1. Trefoil knot

First, We want to apply the method to complicated $1$-manifold in
$\mathbb{R}^{3}$, trefoil knot with noise. Obviously, this manifold cannot be
covered by a single chart. Therefore, using several coordinate domains is
meaningful and necessary. From the Mapper algorithm, we can get four
coordinate domains on the trefoil knot. We use 11 coupling layers for manifold
learning, and 11 autoregressive layers[27] for density estimation.

The overlapping regions are well attached to each other thanks to
compatibility loss, and density is also well estimated.

(a)

(b)

(c)

Figure 4. (A) Synthetic dataset. The colors of points represent densities
estimated by the KDE algorithm. (B) Four coordinate domains. (C) Sampling
points by Atlas flow model. The colors of points represent densities estimated
by the KDE algorithm.

###  4.2. Torus

(a)

(b)

(c)

Figure 5. (A) Synthetic dataset. The colors of points represent densities
estimated by the KDE algorithm. (B) Six coordinate domains. (C) Sampling
points by Atlas flow model. The colors of points represent densities estimated
by the KDE algorithm.

For a more complicated manifold, we apply the Atlas flow to the
$2-$dimensional torus in $\mathbb{R}^{3}$ with noise. Getting six coordinate
domains from Mapper, we trained the model for each cover. We use 13 coupling
layers both for the manifold learning and the density estimation. The density
functions on the overlapping regions are well estimated in this experiment.
This experiment shows that our density adjustment technique is well applied.

###  4.3. StyleGAN2

Figure 6. Generated images from a submanifold in the $\mathcal{W}$ space.

We analyze the style vector space $\mathcal{W}$ of StyleGAN2 [11] using the
pre-trained model on the FFHQ dataset. In the StyleGAN2 model, the style
vector in $\mathcal{W}$ can be obtained using the mapping network of StyleGAN2
from the standard normal distribution in $\mathbb{R}^{512}$, the $\mathcal{Z}$
space.

The next step of the generator is obtaining the image using the synthesis
network of the model from the style vector. In our experiment, we noticed that
the generated image is mainly determined by the direction of the vector in
$\mathcal{Z}$ space. Hence, to analyze the $\mathcal{W}$ space, we took
$4\times 10^{4}$ samples from $S^{511}\subset\mathbb{R}^{512}$ on the
$\mathcal{Z}$ space, which represents every direction of $\mathbb{R}^{512}$,
and observed the data points which come from the synthesis network of the
generator model.

From a geometric viewpoint, the manifold can be defined as a collection of
coordinate charts and transition maps between the coordinate charts. Using our
method, we present a new geometric method to analyze the $\mathcal{W}$ space
by training coordinate maps from the manifold. We checked that the quality of
images from our generating model is plausible (Figure 6).

Next, we show that the local coordinate systems in the overlapping region
between coordinate domains are well matched. In this experiment, we first
fixed a vector from the overlapping region in $\mathcal{W}$ space. Using a
coordinate map and a density transforming map, we can perturb the vector in
the $W$ space for each coordinate chart. Since the coordinate domains cover a
small neighborhood of the vector, the images generated from the neighborhood
should match each other even though the scaling and direction of the axis can
be different and skewed.

For the experiment, we first sampled from $S^{2}$ using the first $3$
coordinates of $\mathcal{Z}$ space and multiplied an orthogonal matrix of size
$512\times 512$ so that various directions on $\mathbb{R}^{512}$ can be
regarded while it preserves the dimension of the manifold. Figure 7
illustrates the images generated from the vectors in the perturbation
neighborhood in each coordinate domain. From the result, we demonstrate the
fact that the small neighborhood of the vector can be matched up to a skewed
axis and different scales.

In our experiments using the StyleGAN2 model, we use 13 coupling layers both
for the manifold learning and the density estimation after getting two
coordinate domains from the Mapper algorithm.

Figure 7. Generated images from perturbation vectors in each coordinate
domain. The Center image with a red dotted border is from the fixed vector on
the overlapping region. The images with the other color dotted borders
represent different coordinate axis on two coordinate domains.

###  4.4. Comparing with the partitions

We designed an experiment that shows the advantage of using a cover instead of
partition[18] in the training process of the Atlas flow.

The experiment shows the reconstructed data points of boundaries of
partition/cover of the manifold. We sampled $10,000$ points of the torus with
noise in $\mathbb{R}^{3}$. We define a data label for each data point by a
partition/cover label in which the point is contained.

The experiment process is following:

  1. (1)

The data label is illustrated in Figure 8.

  2. (2)

We took data points in $30\%$ bands of overlapping regions around the
boundaries from partition.

  3. (3)

We used models which are trained by using partition-wise and cover-wise
datasets. For fairness of comparison, the models were trained by the same
hyper-parameters.

  4. (4)

For the points on the boundary of the cover/partition, reconstruction was
performed with a model trained, not on the cover containing the points, but on
the cover adjacent to the cover containing the points.

  5. (5)

Compute and compare the MSE losses between the original data and its
reconstructions by models from partition and cover.

(a)

(b)

Figure 8. (A) is the partition label and (B) is the cover label.

(a)

(b)

(c)

Figure 9. (A) The boundary points in the original torus dataset. (B) The
reconstructed points that is estimated by using models trained by partition.
(C) The reconstructed points that is estimated by using models trained by
cover.

Through the quantitative (See the Tables 1 and 2) and qualitative (See the
Figure 9) results of this experiment, we justified that the model from cover
is better than the model from partition in the training process of Atlas flow.

| Average Reconstruction Errors
---|---
Partition | 0.1176
Covers | 0.0181
Table 1.  The table is the average reconstruction error of points in the boundaries. $1,344$ points is selected out of $10,000$ data points. (Data label, Model label) | $(1,2)$ | $(2,3)$ | $(2,4)$ | $(3,5)$ | $(4,5)$ | $(5,6)$
---|---|---|---|---|---|---
Partitions | 0.1011 | 0.1147 | 0.0287 | 0.0507 | 0.0329 | 0.0803
Covers | 0.0198 | 0.0200 | 0.0245 | 0.0209 | 0.0129 | 0.0239
Number of points | 168 | 89 | 79 | 93 | 75 | 168
(Data label, Model label) | $(2,1)$ | $(3,2)$ | $(4,2)$ | $(5,3)$ | $(5,4)$ | $(6,5)$
Partitions | 0.3283 | 0.1905 | 0.1689 | 0.0902 | 0.0453 | 0.0657
Covers | 0.0138 | 0.0175 | 0.0187 | 0.0111 | 0.0128 | 0.0183
Number of points | 168 | 94 | 74 | 84 | 84 | 168

Table 2. For each pair of data label and model label, the model from cover is
better than the model from partition.

###  4.5. Comparing with the single chart

Figure 10. The light blue region shows the minimum/maximum of the MSE loss for
each epoch when we used a single chart, and the blue line means the average of
the loss for each epoch. Similarly, we presented the MSE loss for each epoch
using a light orange region and red line when we used the chart.

We designed an experiment to identify the difference in reconstruction error
between a single chart[15] and multi charts by the cover. The $x$-axis of
Figure 10 means the number of epochs as the training process progress, and the
$y$-axis means the average of MSE loss on the training set. We ran exactly the
same $10$ experiments using noisy torus data and briefly presented them in
Figure 10.

In the training process, we confirmed that the MSE loss decreased more rapidly
on the model from cover. Furthermore, we found that after MSE losses converge,
the final loss is lower on the model from cover.

##  5\. Conclusion

In this study, we mainly focused on the reconstructibility of a manifold with
a possibly complicated topological structure. Thanks to the development of
flow-based models, we can take concepts from differential manifold theory, and
use them in machine learning theory. A key point of our Atlas flow model is
that we used local coordinate charts that are compatible with each other. Only
by using the local coordinate charts can the complex structure of a manifold
be accurately identified. Compatibility between the local charts should be
considered to extend local structures to the a global structure.

The structure of a manifold $\mathcal{M}$ is determined by local coordinate
charts. A coordinate map identifies the corresponding coordinate domain with
Euclidean space. That is, it allows us to regard a manifold as a union of
Euclidean spaces which has the advantage of using a coordinate system on the
manifold. Although coordinates do not coincide on overlapping regions,
compatibility between coordinate maps ensures the local coordinate systems are
patched together to form a global structure. To ensure the existence of a
coordinate system, the coordinate domain should be open subsets in Euclidean
space.

One limitation of the flow-based generative model is that the training process
is unstable. In our Atlas flow model, the training process becomes more stable
by using simple coordinate domains. However, it is not enough so we added some
steps to stabilize the training process. For this reason, the training process
becomes heavy and slow. We intend to overcome this limitation by improving the
flow model in future work.

In topological data analysis, it is a crucial problem to take a cover for a
given manifold. A cover consisting of simple regions is ideal but it is almost
impossible to take such a cover without any information of the manifold. We
use Mapper to overcome this problem. However, even though a cover from Mapper
has a big advantage in that it gives topological information of the manifold,
it mainly depends on the lens function and cannot guarantee that it is good
enough. Thus we expect that further research on how to take an ideal cover
enhances our Atlas flow model.

## References

  * [1] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan.  Testing the manifold hypothesis.  Journal of the American Mathematical Society, 29(4):983–1049, 2016\.
  * [2] Joshua B Tenenbaum, Vin De Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  science, 290(5500):2319–2323, 2000.
  * [3] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [4] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In Proceedings of ICML workshop on unsupervised and transfer learning, pages 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * [5] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  stat, 1050:1, 2014.
  * [6] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  Advances in neural information processing systems, 29:4743–4751, 2016.
  * [7] Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff.  Lstm-based encoder-decoder for multi-sensor anomaly detection.  arXiv preprint arXiv:1607.00148, 2016.
  * [8] Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber.  Stacked convolutional auto-encoders for hierarchical feature extraction.  In International conference on artificial neural networks, pages 52–59. Springer, 2011.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [10] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [11] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [12] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [13] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  * [14] Diederik P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  arXiv preprint arXiv:1807.03039, 2018.
  * [15] Johann Brehmer and Kyle Cranmer.  Flows for simultaneous manifold learning and density estimation.  arXiv preprint arXiv:2003.13913, 2020.
  * [16] John M Lee.  Smooth manifolds.  In Introduction to Smooth Manifolds, pages 1–31. Springer, 2013\.
  * [17] Thomas Cohn, Nikhil Devraj, and Odest Chadwicke Jenkins.  Topologically-informed atlas learning.  arXiv preprint arXiv:2110.00429, 2021.
  * [18] Dimitris Kalatzis, Johan Ziruo Ye, Jesper Wohlert, and Søren Hauberg.  Multi-chart flows.  arXiv preprint arXiv:2106.03500, 2021.
  * [19] Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed.  Normalizing flows on riemannian manifolds.  arXiv preprint arXiv:1611.02304, 2016.
  * [20] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  Neural spline flows.  Advances in Neural Information Processing Systems, 32:7511–7522, 2019.
  * [21] Danilo Jimenez Rezende, George Papamakarios, Sébastien Racanière, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer.  Normalizing flows on tori and spheres.  In International Conference on Machine Learning, pages 8083–8092. PMLR, 2020.
  * [22] Emile Mathieu and Maximilian Nickel.  Riemannian continuous normalizing flows.  arXiv preprint arXiv:2006.10605, 2020.
  * [23] Gurjeet Singh, Facundo Mémoli, Gunnar E Carlsson, et al.  Topological methods for the analysis of high dimensional data sets and 3d object recognition.  PBG@ Eurographics, 2, 2007.
  * [24] John Milnor.  Morse Theory.(AM-51), Volume 51.  Princeton university press, 2016.
  * [25] Herbert Edelsbrunner and John Harer.  Computational topology: an introduction.  American Mathematical Soc., 2010.
  * [26] Jan K Pachl.  Disintegration and compact measures.  Mathematica Scandinavica, pages 157–168, 1978.
  * [27] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville.  Neural autoregressive flows.  In International Conference on Machine Learning, pages 2078–2087. PMLR, 2018.
  * [28] Monica Nicolau, Arnold J Levine, and Gunnar Carlsson.  Topology based data analysis identifies a subgroup of breast cancers with a unique mutational profile and excellent survival.  Proceedings of the National Academy of Sciences, 108(17):7265–7270, 2011.
  * [29] Li Li, Wei-Yi Cheng, Benjamin S Glicksberg, Omri Gottesman, Ronald Tamler, Rong Chen, Erwin P Bottinger, and Joel T Dudley.  Identification of type 2 diabetes subgroups through topological analysis of patient similarity.  Science translational medicine, 7(311):311ra174–311ra174, 2015\.
  * [30] Yoshihisa Shinagawa, Tosiyasu L Kunii, and Yannick L Kergosien.  Surface coding based on morse theory.  IEEE Computer Graphics and Applications, 11(05):66–78, 1991.
  * [31] Robert W Ghrist.  Elementary applied topology, volume 1.  Createspace Seattle, WA, 2014.
  * [32] Allen Hatcher.  Algebraic topology.  Cambridge University Press, Cambridge, 2002.

##  Appendix A Appendix

###  A.1. Topological background

Mapper [23] is the most useful method for dimension reduction and exploratory
data analysis based on the topological data analysis (TDA) [28, 29]. The
result of Mapper represents a dataset as a network, which preserve the
topological features of the original data point cloud. Using dimension
reduction (for a filter function) and clustering, the resulting network
captures the local and global structure of the dataset simultaneously. More
precisely, the node of the network corresponds to a connected component of
some level set which is the local simplest region of a data manifold. The
edges connecting these region shows the global feature of a data manifold,
such as topological "holes" (homological feature, precisely) or "shape". These
advantages of Mapper are why we applied Mapper to take a local chart.

In our theoretical view, a bundle of local information of data manifold can be
expanded to the global information and these processes will give more precise
information of data manifold than trying to take information on whole data
manifold at once.

Theoretically, Mapper can be interpreted as two mathematical objects.

  * •

A discrete version of Reeb graph

  * •

An 1-skeleton of a nerve of a refined pull-back cover.

The rest of this section gives some theoretical meaning of Mapper with some
regularity assumptions.

####  A.1.1. Reeb graph

Given a topological manifold $\mathcal{M}$ and a continuous function
$h:\mathcal{M}\to\mathbb{R}$, the Reeb graph $R_{h}(\mathcal{M})$ of
$\mathcal{M}$ with $h$ is the quotient space $\mathcal{M}/\sim$ such that for
$x,y\in\mathcal{M}$, $x\sim y$ if and only if they are contained in the same
connected component of some level set $h^{-1}(a)$ [30]. We note that strictly
speaking, a Reeb graph is not a graph object in a mathematical sense but it
can be converted to a graph structure. Furthermore, we think of a Reeb graph
as a directed graph by assigning a direction to each edge associated with the
gradient descent direction of $h$. With some regularity conditions, thanks to
the Morse theory [24], we can reconstruct $\mathcal{M}$ from
$R_{h}(\mathcal{M})$ up to topological equivalence. That is, the Reeb graph
$R_{h}(\mathcal{M})$ contains the topological information of the original
space $\mathcal{M}$ even though $R_{h}(\mathcal{M})$ is the simpler object.

####  A.1.2. Nerve

Consider a topological sapce $\mathcal{M}$ and a cover
$\mathcal{U}=\\{U_{1},\dots,U_{L}\\}$ on $\mathcal{M}$. The nerve
$\mathcal{N}(\mathcal{U})$ of $\mathcal{U}$ is a simplicial complex consisting
of all simplex of the form $\\{U_{i_{1}},\dots U_{i_{k}}\\}$ such that
$U_{i_{1}}\cap\cdots\cap U_{i_{k}}\neq\phi$ [25]. For a simplex
$\\{U_{i_{1}},\dots U_{i_{k}}\\}$, $k$ is a dimension of the simplex. The
1-skeleton of $\mathcal{N}(\mathcal{U})$ is the subcomplex of
$\mathcal{N}(\mathcal{U})$ consisting of simplex of dimension $\leq 1$. We
note that any 1-skeleton is a graph.

From Mapper section in the main context, we call pre-images of intervals
$\\{h^{-1}(I)|I\subset h(\mathcal{M})\\}$ a pull-back cover and
$\\{U_{i}\\}_{k=1}^{L}$ a refined pull-back cover. Then, by definition, Mapper
is equal to a nerve of an 1-skeleton of a nerve of a refined pull-back cover.

A nerve is a useful object in topology and geometry fields and its importance
is guaranteed by the nerve theorem [31].

######  Theorem A.1 (Nerve theorem).

If $\mathcal{U}$ is a good cover, that is, any intersection
$U_{i_{1}}\cap\cdots\cap U_{i_{k}}$ of elements $\mathcal{U}$ has no holes
(contractible), then $\mathcal{N}(\mathcal{U})$ is topologically equivalent to
$\mathcal{M}$. (Precisely, homotopic equivalent.)

We do not give a detailed proof or description of mathematical terminologies,
such as good cover or homotopic equivalent. (For more details, see [32].)
However, the nerve theorem means that if we choose an appropriate cover for a
data manifold, then Mapper is the most useful method for visualizing and
analyzing the topological aspect of the ambient data space.

###  A.2. Geometric background

####  A.2.1. Manifold

A $n$-dimensional differential manifold $\mathcal{M}$ is a topological space
adapting differential structure on it [16]. One important property of
$\mathcal{M}$ is the locally Euclidean, that is, $\mathcal{M}$ is covered by
open subsets, which are the same to Euclidean space. More precisely, the
differential structure of $\mathcal{M}$ consists of open subsets $\\{U_{k}\\}$
and invertible maps $\\{\varphi_{k}:U_{k}\to\mathbb{R}^{n}\\}$ with following
properties.

  * •

$\\{U_{k}\\}$ covers $\mathcal{M}$, i.e. $\mathcal{M}=\cup_{k}U_{k}$.

  * •

$\varphi_{k}$ is a homeomorphism, i.e. the domain and image of $\varphi_{k}$
have the exectly same in the topological sense.

  * •

For any $(U_{k},\varphi_{k})$ and $(U_{l},\varphi_{l})$, the transition map
$\varphi_{l}\circ\varphi_{k}^{-1}:\varphi_{k}(U_{k}\cap
U_{l})\to\varphi_{l}(U_{k}\cap U_{l})$ is a diffeomorphism.

We call such a pair $(U_{k},\varphi_{k})$ as a local coordinate chart. We note
that if $\mathcal{M}$ is compact, then the structure of $\mathcal{M}$ can
consist of the finite number of charts.

A submanifold of $\mathcal{M}$ is the topological subspace of $\mathcal{M}$
with differential structure induced by the restriction. Thus we can regard a
submanifold as a manifold itself. If we consider a manifold $\mathcal{M}$ as a
subamanifold of $\mathbb{R}^{d}$, then we may think that each $\varphi_{k}$ is
a restriction of a diffeomorphism on $\mathbb{R}^{d}$.

####  A.2.2. Rank theorem

A differential structure of a manifold has special property that charts are
not uniquely determined and geometric properties are invariant to choices of
charts if they are compatible. Moreover, we can take charts so that induced
submanifold structure become simpler in view of the chart map. This property
is summarized by following theorem.

######  Theorem A.2.

(Submanifold property) For a $n$-dimensional submanifold $\mathcal{M}$ in
$\mathbb{R}^{d}$, there are coordinate charts $\\{(U_{k},\varphi_{k})\\}$ of
$\mathbb{R}^{d}$ such that

(10) $\varphi_{k}(\mathcal{M}\cap
U_{k})\subset\mathbb{R}^{n}\times\\{0\\}^{d-n}.$

The submanifold property is comes from rank theorem stated generally below.

######  Theorem A.3 (Rank Theorem).

Given manifolds $\mathcal{M}$ and $\mathcal{N}$ of dimension $n,d$,
respectively, and a differentiable function $F$ of constant rank $r$ between
them, then for every $p\in\mathcal{M}$, there are charts $(V_{k},\psi_{k})$ on
$\mathcal{M}$ and $(U_{k},\varphi_{k})$ on $\mathcal{N}$ such that $p\in
V_{k}$, $F(p)\in U_{k}$ such that $F(V_{k})\subset U_{k}$ and

(11) $\varphi_{k}\circ
F\circ\psi_{k}^{-1}\,(x_{1},\dots,x_{r},x_{r+1},\dots,x_{n})=(x_{1},\dots,x_{r},0,\dots,0).$

In the above statement, if we consider $\mathcal{N}$ as $\mathbb{R}^{d}$ and
$F$ as an inclusion map, then the rank $r$ is equal to $n$. Also, we may
ragard $(\mathcal{M}\cap U_{k},\varphi_{k}|_{\mathcal{M}\cap U_{k}})$ as a
chart of $\mathcal{M}$ since it is compatible with $\\{(V_{k},\psi_{k})\\}$
that provided by rank theorem. This proves the submanifold property.

###  A.3. Probabilistic background

By a refined partition
$\\{U^{\prime}_{k^{\prime}}\\}_{k^{\prime}=1}^{L^{\prime}}$ of cover
$\\{U_{k}\\}_{k=1}^{L}$, we mean that it forms a partition of $\mathcal{M}$
itself and each $U^{\prime}_{k^{\prime}}$ is either a proper subset of some
$U_{k}$ or intersection of $U_{k}$’s. Here a proper subset of $U_{k}$ means a
subset $U_{k}\setminus\cup\\{U_{k_{0}}:k_{0}\neq k\\}$. Intuitively,
$\\{U^{\prime}_{k^{\prime}}\\}_{k^{\prime}=1}^{L^{\prime}}$ is obtained from
$\mathcal{M}$ by cutting along all boundaries of $U_{k}$’s. We define the
quotient map $\pi:\mathcal{M}\to\\{1,\dots,L^{\prime}\\}$ such that if $x$ is
in $U_{k^{\prime}}$ then $\pi(x)=k^{\prime}$. We let $\nu$ be a probability on
$\\{1,\dots,L^{\prime}\\}$ which is defined by a push-forward measure of
$p^{\mathcal{M}}$ by $\pi$.

######  Definition A.4.

In the above setting, there is a disintegration of $p^{\mathcal{M}}$ given by

(12)
$p^{\mathcal{M}}=\int_{\\{1,\dots,L^{\prime}\\}}p^{U^{\prime}}_{k^{\prime}}\;d\nu(k^{\prime})=\sum_{k^{\prime}=1}^{L^{\prime}}\nu(k^{\prime})p^{U^{\prime}}_{k^{\prime}},$

where $p^{U^{\prime}}_{k^{\prime}}$ is the probability on
$U^{\prime}_{k^{\prime}}$ that can be considered as the probability on
$\mathcal{M}$ with support in $U^{\prime}_{k^{\prime}}$.

Since $p^{\mathcal{M}}$ is unknown, we cannot calculate its push-forward
measure $\nu$. Therefore, for each $k^{\prime}=1,\dots,L^{\prime}$, we
estimate the value of $\nu(k^{\prime})$ to the normalized number of sampled
data points in $U^{\prime}_{k^{\prime}}$. We note that
$\nu(k^{\prime})=p^{\mathcal{M}}(U^{\prime}_{k^{\prime}})$ by the definition
of the push-forward measure. And expectations of the numbers of sampled data
points in $U^{\prime}_{k^{\prime}}$’s are proportional to
$p^{\mathcal{M}}(U^{\prime}_{k^{\prime}})$, which is equal to
$\nu(k^{\prime})$. Hence our estimation is suitable for sufficiently sampled
datasets.

In order to estimate $p^{\mathcal{M}}$ on each cover $U_{k}$, we decompose
$p^{\mathcal{M}}$ as

(13) $p^{\mathcal{M}}=\sum_{k=1}^{L}c_{k}p^{U}_{k},$

where $c_{k}$ is a scaling contant and $p^{U}_{k}$ is a probability on
$\mathcal{M}$ with $p^{U}_{k}(U_{k})=1$. For the decomposition, we define
$p^{U}_{k}$ as

(14) $p^{U}_{k}=\frac{1}{c_{k}}\sum_{k^{\prime}:U_{k^{\prime}}\subset
U_{k}}\frac{1}{n(k^{\prime})}\nu(k^{\prime})p^{U^{\prime}}_{k^{\prime}},$

where $U_{k^{\prime}}$ consists of $n(k^{\prime})$-intersection of $U_{k}$’s
and

(15) $c_{k}=\sum_{k^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}}\frac{1}{n(k^{\prime})}\nu(k^{\prime})$

is the scaling constant that total measure of $p_{\alpha}$ is equal to 1.

In geometric terms, we can think of the decomposition of $p^{\mathcal{M}}$ to
$p^{U}_{k}$’s in terms of "modified partition of unity". We denote the
probability density functions for $p^{\mathcal{M}}$ and
$p^{U^{\prime}}_{k^{\prime}}$ by $\textbf{PDF}_{\mathcal{M}}$ and
$\textbf{PDF}_{k^{\prime}}$, respectively. We define the probability density
function $\textbf{PDF}_{k}$ supported on $U_{k}$ as

(16)
$\textbf{PDF}_{k}=\frac{1}{|\\{k^{\prime}=1,\dots,L^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}\\}|}\sum_{k^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}}\textbf{PDF}_{k^{\prime}}.$

######  Definition A.5.

Given a cover $\\{U_{k}\\}_{k=1}^{L}$ on $\mathcal{M}$, a modified partition
of unity is a family $\\{\rho_{k}:\mathcal{M}\to\mathbb{R}\\}$ of functions
such that for each $k=1,\dots,L$, the value of $\rho_{k}$ outside $U_{k}$ is
equal to 0 and

(17) $\int_{-\infty}^{\infty}\sum_{k=1}^{L}\rho_{k}(x)\;dx=1.$

The original definition of a partition of unity is that the left hand-side of
(17) does not have the integral. For applying its concept to a probability
density function, we modified the definition. Then if we set a partition of
unity as

(18) $\rho_{k}=\sum_{k^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}}\frac{1}{n(k^{\prime})}\nu(k^{\prime})\mathbf{1}_{U^{\prime}_{k^{\prime}}},$

where $\mathbb{1}$ is an indicator function, we have

(19)
$\sum_{k=1}^{L}\rho_{k}\textbf{PDF}_{k}=\sum_{k=1}^{L}\sum_{k^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}}\frac{1}{n(k^{\prime})}\nu(k^{\prime})\textbf{PDF}_{k^{\prime}}=\sum_{k^{\prime}:U^{\prime}_{k^{\prime}}\subset
U_{k}}\nu(k^{\prime})\textbf{PDF}_{k^{\prime}}=\textbf{PDF}_{\mathcal{M}}$

which yields the same result with (13).

###  A.4. Experiment details

We specify the details of our experiments in this section. we use python of
version $3.8$ for overall experiments. For the essential packages in our
algorithm, We use Kepler Mapper of version $2.0.1$, PyTorch of version
$1.10.0$, and scikit-learn of version $1.0.1$. We use Isomap and PCA from the
scikit-learn package.

####  A.4.1. Gaussian mixture model on a trefoil knot

The trefoil knot without noise is parametrized by the following equations. For
$t\in[0,2\pi]$,

(20) $\begin{cases}x=\sin(t)+3\sin(2t)\\\ y=\cos(t)-3\cos(2t)\\\
z=-\sin(3t).\end{cases}$

We add Gaussian noise with a mean of 0 and a standard deviation of $0.1$. The
Gaussian mixture model consists of two Gaussian probability densities whose
means are $0,\pi$, and standard deviations are all $\pi/6$. We train the
manifold learning flow and density estimation flow with $10^{4}$ data samples.

For the lens function of the Mapper algorithm, we use $1$-dimensional PCA
values. In the Mapper algorithm, Single-linkage clustering is used for
clustering algorithm with distance threshold 1. For the other hyperparameter
for Mapper, n_cubes is $2$, and perc_overlap is $0.2$.

During every epoch, we use Adam optimizer with an initial rate of $0.0015$
decreasing the learning rate with cosine annealing with weight decay of
$10^{-4}$. For gradient clipping, we set the norm $5$. We set the calculating
step for expected points as $C_{s}=2$. In the (Algorithm 8), the number of
epochs are $e_{1},e_{2},e_{3},e_{4},e_{5}=15,30,60,60,60$ respectively. The
factors for weights are
$\lambda_{m},\lambda_{p},\lambda_{o},\lambda_{d}=100,0.01,100,0.1$,
respectively. We set the batch sizes $b=256$.

####  A.4.2. Gaussian mixture model on a torus

The parametric equations for the torus are as follows. For $t,s\in[0,2\pi]$,

(21) $\begin{cases}x=(\cos(t)+3)\cos(s)\\\ y=(\cos(t)+3)\sin(s)\\\
z=\sin(t).\end{cases}$

We add Gaussian noise with a mean of $0$ and a standard deviation of $0.1$.
The Gaussian mixture model consists of four Gaussian probability densities
whose means are randomly chosen on $[-\pi,\pi]$ and standard deviations are
all $\pi/3$. The Gaussian mixture noise is added through the parameters $t,s$.
We train the models with $10^{4}$ data points from the noisy torus.

For the lens function of the Mapper algorithm, we use $1$-dimensional PCA
values. In Mapper algorithm, Single-linkage clustering is used for clustering
algorithm with distance threshold 1. For the other hyperparameter for Mapper,
n_cubes is $5$, and perc_overlap is $0.45$.

During every epoch, we use Adam optimizer with an initial rate of $0.0015$
decreasing the learning rate with cosine annealing with weight decay of
$10^{-4}$. For gradient clipping, we set the norm $5$. We set the calculating
step for expected points as $C_{s}=2$. In the (Algorithm 8), the number of
epochs are $e_{1},e_{2},e_{3},e_{4},e_{5}=60,30,60,60,60$ respectively. The
factors for weights are
$\lambda_{m},\lambda_{p},\lambda_{o},\lambda_{d}=100,0.1,25,0.01$,
respectively. We set the batch sizes $b=256$.

####  A.4.3. StyleGAN2

For both $S^{2}$ and $S^{511}$ experiments, we use $1$ dimensional PCA lens
function in the Mapper algorithm. For the other hyperparameter, the
overlapping percentage is 0.33, and n_cube is $2$.

During every epoch, we use Adam optimizer with an initial rate of $0.0001$
decreasing the learning rate with cosine annealing with weight decay of
$10^{-4}$. For gradient clipping, we set the norm $1$. We set the calculating
step for expected points as $C_{s}=5$. We noticed that the pretraining loss is
not necessary. In the 8, the number of epochs are
$e_{1},e_{2},e_{3},e_{4},e_{5}=0,0,40,80,60$ respectively. The factors for
weights are $\lambda_{m},\lambda_{p},\lambda_{o},\lambda_{d}=100,0,100,0.01$,
respectively. We set the batch sizes $b=512$.

###  A.5. Algorithms

Algorithm 1 Pretraining loss

1:Input: Flow function $\varphi$. Data points $\\{x_{i}\\}_{i=1}^{b}$ on
$\mathbb{R}^{d}$. Reference data points on $\mathbb{R}^{n}$,
$R=\\{r_{i}\\}_{i=1}^{b}$.

2:Output: Pretraining loss $\mathcal{L}_{\text{pre}}$,

3:for $i=1,\dots,b$ do

4: $v_{i},\tilde{v}_{i}\leftarrow\varphi(x_{i})$

5:end for

6:$\mathcal{L}_{\text{pre}}$ = $\frac{1}{b}\sum_{i}\|v_{i}-r_{i}\|_{2}^{2}$

Algorithm 2 Reconstruction loss

1:Input: Flow function $\varphi$. Data points $\\{x_{i}\\}_{i=1}^{b}$ on
$\mathbb{R}^{d}$.

2:Output: Reconstruction loss $\mathcal{L}_{\text{recon}}$

3:for $i=1,\dots,b$ do

4: $v_{i},\tilde{v}_{i}\leftarrow\varphi(x_{i})$

5: $x_{i}^{\prime}\leftarrow\varphi^{-1}(v_{i},0)$

6:end for

7:$\mathcal{L}_{\text{recon}}$ =
$\frac{1}{b}\sum_{i}\|x_{i}-x_{i}^{\prime}\|_{2}^{2}$

Algorithm 3 Pairwise distance loss

1:Input: Flow function $\varphi$. Data points $\\{x_{i}\\}_{i=1}^{b}$ on
$\mathbb{R}^{d}$. Reference pairwise distance matrix, $A=[a_{ij}]^{b\times
b}$.

2:Output: Pairwise distance $\mathcal{L}_{\text{dist}}$

3:for $i=1,\dots,b$ do

4: $v_{i},\tilde{v}_{i}\leftarrow\varphi(x_{i})$

5:end for

6:$T\leftarrow 0$

7:for $i=1,\dots b$ do

8: for $j=1,\dots b$ do

9: $T\leftarrow T+(a_{ij}-\|v_{i}-v_{j}\|_{2})^{2}$

10: end for

11:end for

12:$\mathcal{L}_{\text{dist}}$ = $\frac{1}{b(b-1)}T$

Algorithm 4 Compatibility loss

1:Input: Cover-wise coordinate maps, $\\{\varphi_{k}\\}_{k=1}^{L}$. Cover-wise
data points $X_{k}=\\{x_{i}^{k}\\}_{i=1}^{b}$ on $\mathbb{R}^{d}$ and expected
points $\\{\hat{x}_{i}\\}_{i=1}^{b}$.

2:Output: Compatibility loss $\mathcal{L}_{\text{comp}}$

3:$T\leftarrow 0$

4:$j\leftarrow 0$

5:for $i=1,\dots,b$ do

6: if $x_{i}^{k}$ is in overlapping region then

7: $T\leftarrow T+\|x_{i}^{k}-\hat{x}_{i}\|_{2}^{2}$

8: $j\leftarrow j+1$

9: end if

10:end for

11:$\mathcal{L}_{\text{comp}}$ = $\frac{1}{j}T$

Algorithm 5 Density learning loss

1:Input: Flow function $\gamma$. Data points $\\{v_{i}\\}_{i=1}^{b}$.
Probability density function on $\mathbb{R}^{n}$, $p^{W}$.

2:Output: $\mathcal{L}_{\text{density}}$, density loss

3:for $i=1,\dots,b$ do

4: $w_{i}\leftarrow\gamma(v_{i})$

5:end for

6:$\mathcal{L}_{\text{density}}\leftarrow-\frac{1}{b}\sum_{i}[\log
p^{W}(w_{i})-\log\det J_{\gamma^{-1}}(w_{i})]$

Algorithm 6 Calculate Expected Points

1:Input: All data points, $X=\\{x_{i}\\}_{k=1}^{N}$. Cover-wise data points
$\\{X_{k}\\}_{k=1}^{L}$ .

2:Output: Expected points $\\{\hat{x}_{i}\\}_{i=1}^{N}$

3:for $x_{i}$ in $X$ do

4: $\hat{x}_{i}\leftarrow 0$

5: $j\leftarrow 0$

6: for $k=1,\dots,L$ do

7: if $x_{i}$ in $X_{k}$ then

8: $\hat{x}_{i}\leftarrow\hat{x}_{i}+x_{i}$

9: $j\leftarrow j+1$

10: end if

11: end for

12: $\hat{x}_{i}\leftarrow\hat{x}_{i}/j$

13:end for

Algorithm 7 Manifold Learning Loss

1:Input: Graph-based pairwise distance matrix of all data points, $D$. Flow
function $\varphi$. Data points $\\{x_{i}\\}$. Factor weighting terms in the
loss function, $\lambda_{t}$.

2:Output: Manifold Learning Loss $\mathcal{L}_{\text{mfd}}$

3:$D_{b}\leftarrow$pairwise distance matrix corresponding the batch data
$\\{x_{i}\\}$.

4:$\mathcal{L_{\text{dist}}\leftarrow}\operatorname{PairwiseDistanceLoss}(\varphi,\\{x_{i}\\},D_{b})$

5:$\mathcal{L_{\text{recon}}\leftarrow}\operatorname{Reconstructionloss}(\varphi,\\{x_{i}\\})$

6:$\mathcal{L_{\text{mfd}}}\leftarrow\lambda_{t}\mathcal{L}_{\text{dist}}+(1-\lambda_{t})\mathcal{L}_{\text{recon}}$

Algorithm 8 Training process

1:Input: Cover-wise coordinate maps $\\{\varphi_{k}\\}_{k=1}^{L}$. Cover-wise
density transforming map $\\{\gamma_{k}\\}_{k=1}^{L}$. Weights of
$\\{\varphi_{k}\\}_{k=1}^{L}$, $\\{\theta_{k}\\}_{k=1}^{L}$. Weights of
$\\{\gamma_{k}\\}_{k=1}^{L}$, $\\{\omega_{k}\\}_{k=1}^{L}$. The learning rate,
$\alpha$. The batch size, $b$. All point cloud, $X$. Cover-wise point clouds,
$\\{X_{k}\\}_{k=1}^{L}$. Cover-wise Isomap results of $\\{X_{k}\\}_{k=1}^{L}$,
$\\{R_{k}\\}_{k=1}^{L}$. Cover-wise graph-based pairwise distance matrix of
$\\{X_{k}\\}_{k=1}^{L}$, $\\{D_{k}^{N_{k}\times N_{k}}\\}_{k=1}^{L}$. Pair-
wise probability density function on space $W$, $\\{p_{k}^{W}\\}_{k=1}^{L}$.
Factors weighting terms in the loss functions,
$\lambda_{m},\lambda_{p},\lambda_{o}$ and $\lambda_{d}$. The number of epochs
for each step, $e_{1},e_{2},e_{3},e_{4}$ and $e_{5}$. The number of steps for
calculating expected points, $C_{s}$.

2:for $k=1,\dots L$ do

3: for $j=1,\dots,e_{1}$ do

4: for mini-batch data $\\{x_{i}\\}_{i=1}^{b},\\{r_{i}\\}_{i=1}^{b}$ from
$X_{k}$ and $R_{k}$ respectively do

5:
$\mathcal{L_{\text{pre}}\leftarrow}\operatorname{Pretrainingloss}(\varphi_{k},\\{x_{i}\\},\\{r_{i}\\})$

6:
$\theta_{k}\leftarrow\theta_{k}-\alpha\lambda_{m}\nabla_{\theta_{k}}\mathcal{L}_{\text{pre}}$

7: end for

8: end for

9: for $j=1,\dots,e_{1}$ do

10: for mini-batch data $\\{x_{i}\\}_{i=1}^{b}$ from $X_{k}$ do

11:
$L_{\text{density}}\leftarrow\operatorname{DensityLearningLoss}(\gamma_{k},\\{\varphi_{k}(x_{i})\\},p_{k}^{W})$

12:
$\omega_{k}\leftarrow\omega_{k}-\alpha\lambda_{d}\nabla_{\omega_{k}}\mathcal{L}_{\text{density}}$

13: end for

14: end for

15: for $j=1,\dots,e_{2}+e_{3}$ do

16: if $j\leq e_{2}$ then

17:
$\lambda_{t}\leftarrow\frac{\lambda_{p}-1}{e_{2}-1}j+1-\frac{\lambda_{p}-1}{e_{2}-1}$

18: end if

19: for mini-batch data $\\{x_{i}\\}_{i=1}^{b}$ from $X_{k}$ do

20:
$\mathcal{L_{\text{mfd}}}\leftarrow\operatorname{ManifoldLearningLoss}(D,\varphi_{k},\\{x_{i}\\},\lambda_{t})$

21:
$\theta_{k}\leftarrow\theta_{k}-\alpha\lambda_{m}\nabla_{\theta_{k}}\mathcal{L_{\text{mfd}}}$

22:
$L_{\text{density}}\leftarrow\operatorname{DensityLearningLoss}(\gamma_{k},\\{\varphi_{k}(x_{i})\\},p_{k}^{W})$

23:
$\omega_{k}\leftarrow\omega_{k}-\alpha\lambda_{d}\nabla_{\omega_{k}}\mathcal{L}_{\text{density}}$

24: end for

25: end for

26:end for

27:for $j=1,\dots,e_{4}$ do

28: if $j\,\%\,C_{s}$ is $1$ then

29:
$\\{\hat{x}_{i}\\}\leftarrow\operatorname{CalculateExpectedPoints}(X,\\{X_{k}\\}_{k=1}^{L})$

30: end if

31: for $k=1,\dots L$ do

32: for mini-batch data $\\{x_{i}\\}_{i=1}^{b}$ from $X_{k}$ do

33:
$\mathcal{L_{\text{mfd}}}\leftarrow\operatorname{ManifoldLearningLoss}(D,\varphi_{k},\\{x_{i}\\},\lambda_{p})$

34:
$\mathcal{L_{\text{comp}}\leftarrow}\operatorname{CompatibilityLoss}(\\{\varphi_{k}\\},\\{X_{k}\\},\\{\hat{x}_{i}\\})$

35:
$\mathcal{L}\leftarrow\mathcal{L}_{\text{mfd}}+\frac{1}{e_{4}}j\lambda_{o}\mathcal{L}_{\text{comp}}$

36:
$\theta_{k}\leftarrow\theta_{k}-\alpha\lambda_{m}\nabla_{\theta_{k}}\mathcal{L}$

37:
$L_{\text{density}}\leftarrow\operatorname{DensityLearningLoss}(\gamma_{k},\\{\varphi_{k}(x_{i})\\},p_{k}^{W})$

38:
$\omega_{k}\leftarrow\omega_{k}-\alpha\lambda_{d}\nabla_{\omega_{k}}\mathcal{L}_{\text{density}}$

39: end for

40: end for

41:end for

42:for $k=1,\dots,L$ do

43: for $j=1,\dots,e_{5}$ do

44: for mini-batch data $\\{x_{i}\\}_{i=1}^{b}$ from $X_{k}$ do

45:
$L_{\text{density}}\leftarrow\operatorname{DensityLearningLoss}(\gamma_{k},\\{\varphi_{k}(x_{i})\\},p_{k}^{W})$

46:
$\omega_{k}\leftarrow\omega_{k}-\alpha\lambda_{d}\nabla_{\omega_{k}}\mathcal{L}_{\text{density}}$

47: end for

48: end for

49:end for