\inf_{\enc \in \mathcal{F}} \E_{X \sim \trueMeas}\left[c\left(X, \dec_{\param_1}\left(\enc(X)\right)\right)\right],
\end{equation}

where $\mathcal{F}=\{\enc: \dataspace \rightarrow \latentspace \mid \enc \text{ is measurable}\}$, involves an infimum over measurable functions (which is then minimized over $\param_1$ during the first step). In the nonparametric regime (<ref>) we assume that neural networks are flexible enough to approximate any continuous function arbitrarily well, but this property need not a priori extend to measurable functions. Intuitively this should however not be a problem thanks to Lusin's theorem – which, informally, states that in certain settings any measurable function can be approximated by a continuous one. It is nonetheless pertinent to show that the infimum in <ref> can actually be replaced by a corresponding infimum over continuous functions, which we do below.

propositionwasserstein  Let $\trueMeas$ be a probability measure on $\dataspace$, $\dec_{\param_1}: \latentspace \rightarrow \dataspace$ be measurable, and $c: \dataspace \times \dataspace \rightarrow \R$ be measurable and such that there exists $C > 0$ such that

\begin{equation}\label{eq:assumption_th2}
\sup_{(x, y) \in \dataspace \times \dataspace} |c(x,y)| < C.
\end{equation}

\begin{equation}
\inf_{\enc \in \mathcal{F}} \E_{X \sim \trueMeas}\left[c\left(X, \dec_{\param_1}\left(\enc(X)\right)\right)\right] = \inf_{\enc \in \mathcal{C}} \E_{X \sim \trueMeas}\left[c\left(X, \dec_{\param_1}\left(\enc(X)\right)\right)\right],
\end{equation}

where $\mathcal{C} = \{\enc:\dataspace \rightarrow \latentspace \mid \enc \text{ is continuous}\}$.

See [app:prop1]Appendix B.2.

This result allows us to formally interpret two-step models as minimizers of an upper bound of the Wasserstein distance in the nonparametric regime when the assumption in <ref> holds. We point out that this is a mild regularity condition, as it is always satisfied in the common case where $\dataspace$ is compact and $c$ is continuous (since continuous functions always achieve their supremums over compact sets).

§.§.§ Latent Diffusion Models

Latent diffusion models [Rombach et al., 2022, Peebles & Xie, 2023, Zhang et al., 2024] are another class of two-step models (<ref>). They first train a regularized autoencoder, which combines a Gaussian variational autoencoder objective (<ref>) with various potential regularizers [Larsen et al., 2016, Higgins et al., 2017, van den Oord et al., 2017].
Once this autoencoder-based model is trained and the corresponding low-dimensional representations obtained, a diffusion model (<ref>) $s_{\param_2}^Z: \latentspace \times (0, T] \rightarrow \latentspace$ is trained on them as the second step model.

As previously discussed,  diffusion models can learn manifolds, but their score function must diverge to infinity at the end of the backward process (<ref>) as a consequence of the mismatch between the intrinsic and ambient dimensions of the data. To test that latent diffusion models are not as sensitive to this numerical pathology, we trained a diffusion model and a latent diffusion model on the CIFAR-10 dataset [Krizhevsky & Hinton, 2009], with all experimental details provided in <ref>. We plot the average squared Euclidean norm of the score functions, normalized by their dimension,[Note that normalizing squared Euclidean norm by dimension is the most natural way of enabling comparisons across dimensions. To see this consider a constant vector with all entries equal to $1$, which has a squared $\ell_2$ norm equal to its dimension; or consider a standard Gaussian vector, whose expected squared Euclidean norm also matches its dimension.] along generated paths in <ref>: it is evident that the score function of diffusion models on latent space exhibits much better numerical behaviour than when these models are trained on ambient space. This result is suggested by theory, and to the best of our knowledge, we are the first to empirically confirm it.

Currently, latent diffusion models are amongst the best performing DGMs empirically, and the manifold lens provides a convincing explanation for this: $(i)$ they can learn manifolds; $(ii)$ they alleviate the numerical issues of diffusion models; and $(iii)$ they are robust to misspecification of the dimension of the latent space, in the sense that even if $\mDim < \lDim$ (i.e. the latent dimension is specified as larger than the true intrinsic dimension), they still learn their target distribution – albeit with an exploding score function. To see this, simply note that setting $\mDim < \lDim$ results in a second step model whose target distribution $q^Z_{\paramAlt^*}$ is still supported on a manifold $\enc_{\paramAlt^*}(\M)$ of lower-than-ambient dimension, which the diffusion model from the second step can still learn. Although in this case the score function would also diverge to infinity, the fact that the difference between the ambient and intrinsic dimensions for the latent model ($\lDim - \mDim$) remains much smaller than for a model on ambient space ($\aDim - \mDim$) intuitively suggests that the numerical issues should nonetheless be diminished for latent diffusion models. Indeed, despite the latent diffusion model shown in <ref> using $d = 256$, which is likely larger than the true intrinsic dimension $\mDim$ of CIFAR-10 [Pope et al., 2021], it has a numerically much better behaved score function than the diffusion model on ambient space.

Average squared norm of the learned score function on CIFAR-10 over $100$ generated paths from <ref>, normalized by dimension (i.e. $\texttt{dim}=D=3072$ for diffusion models, and $\texttt{dim}=d=256$ for latent diffusion models); the shaded area corresponds to one standard deviation. The paths are stopped at time $T-\varepsilon$, with $\varepsilon=0.001$.

§.§.§ Injective Normalizing Flows

Ordinary normalizing flows (<ref>) are full-dimensional density models with $\aDim$-dimensional latent spaces, conflicting with the $\mDim$-dimensional nature of $\trueDens$. To correct this mismatch, a line of research started by Kumar et al., 2020 has proposed to shrink the NF's latent space dimensionality to $\lDim < \aDim$, allowing the model to represent densities on a low-dimensional submanifold of $\dataspace$. Whereas standard NF architectures must be bijective, $\dec_{\param_1}$ now cannot be bijective because it maps from $\lDim$ to $\aDim$ dimensions. The most one can ask is that $\dec_{\param_1}$ be injective, resulting in the injective normalizing flow (INF).

Pathology of naïvely maximizing <ref>. In this case, the true density is a standard Gaussian along $\M = \{(0, x_2) \in \R^2 \mid x_2 \in \R\}$, here shown in blue with darker values indicating higher density. The model can maximize likelihood by aligning $g_{\theta_1}(\latentspace)$ perpendicular to $\M$, and then learning a density along $g_{\theta_1}(\latentspace)$ that becomes infinitely peaked at the projection $\hat x_{\param_1}(x)$ onto $\M$; in this example the projection will always lie at the origin for any $x \in \R^2$. In the figure, we show $\dec_{\param_1}(\latentspace) = \{(x_1, 0) \in \R^2 \mid x_1 \in \R\}$ with the black line, the projection with the red dot, and increasing peakedness of $\modelDens$ with increasing opacity.

Brehmer & Cranmer, 2020 enforce injectivity architecturally, by constructing $\dec_{\param_1}$ as a zero-padding operation followed by a $\aDim$-dimensional NF, in which case the left inverse $\enc_{\param_1}$ is given by inverting this NF and applying a projection operation; this is the same construction as the one used by denoising NFs (<ref>).[Like in standard NFs, here the encoder $\enc_{\param_1}$ is determined by the decoder $\dec_{\param_1}$, and it is thus parameterized by the same parameters (i.e. $\param_1$), so no auxiliary parameters $\paramAlt$ are needed to parameterize it.] Injectivity enables density evaluation through the injective change-of-variables formula (<ref>):

\begin{equation}\label{eq:change_variable_inj2}
\ambientDens_{\param}(x) = \latentDens_{\param_2}(z)\left| \det \left( \nabla_z \dec_{\param_1}(z)^\top \nabla_z \dec_{\param_1}(z) \right) \right|^{-\frac{1}{2}},
\end{equation}

where $z = \enc_{\param_1}(x)$.
Unlike standard NFs, INFs cannot be naïvely trained through maximum-likelihood for two main reasons: $(i)$ the involved determinant is much more computationally challenging to compute and optimize than in standard NFs, and more importantly $(ii)$ Brehmer & Cranmer, 2020 showed that doing so would result in pathological solutions.
To understand why, recall that <ref> is only valid when $x \in \dec_{\param_1}(\latentspace)$. When $x$ lies outside $\dec_{\param_1}(\latentspace)$, the right hand side of <ref> evaluates to $\modelDens(\hat{x}_{\param_1}(x))$, where $\hat{x}_{\param_1}(x) = \dec_{\param_1}(\enc_{\param_1}(x))$ can be thought of as a projection of $x$ onto $\dec_{\param_1}(\latentspace)$. Since $\dec_{\param_1}(\latentspace)$ need not perfectly match the data manifold $\M$, attempting to maximize $\E_{X \sim \trueDens}[\log \latentDens_{\param_2}(\enc_{\param_1}(X)) - \tfrac{1}{2}\log |\det ( \nabla_z \dec_{\param_1}(\enc_{\param_1}(X))^\top \nabla_z \dec_{\param_1}(\enc_{\param_1}(X)))|]$ over $\param$ would thus result in maximizing the likelihood of projected data. As illustrated in <ref>, this objective can admit pathological solutions where the projections collapse onto a single point whose likelihood is sent to infinity.

To circumvent this issue, Brehmer & Cranmer, 2020 propose to train INFs as two-step models (<ref>), where $\dec_{\param_1}$ and $\enc_{\param_1}$ are trained to minimize an $\ell_2$ reconstruction error as in <ref>; this training procedure also obviates the need to optimize through the determinant in <ref>. Brehmer & Cranmer, 2020 then instantiate $\latentDens_{\param_2}$ as a $\lDim$-dimensional NF, which is trained on encoded data $\enc_{\param_1^*}(X)$, where $X \sim \trueDens$. Kothari et al., 2021 follow up on this work by proposing a more efficient architecture for $\dec_{\param_1}$. Kumar et al., 2020 originally proposed relaxed INFs, for which the encoder $\encoder$ is parameterized separately (and encouraged to invert the decoder on $\M$ through a reconstruction error), and where injectivity is instead encouraged by regularizing the singular values of $\nabla_z \dec_{\param_1}(\encoder(X))$ for $X \sim \trueDens$. They then take the second step density $\latentDens_{\param_2}$ as a Gaussian mixture model. By virtue of being two-step models, all these DGMs are manifold-aware.

End-to-end training As mentioned in <ref>, it is intuitively desirable to find an end-to-end objective to train two-step models, and INFs provide particularly interesting opportunities. To circumvent the pathological behaviour of naïve maximum-likelihood training of INFs outlined above, several works encourage perfect reconstructions to ensure that the model manifold matches the true data manifold:

\begin{align}
\begin{split}
& \max_\param \E_{X \sim \trueDens}\left[ \log \latentDens_{\param_2}(\enc_{\param_1}(X)) - \tfrac{1}{2}\log \left|\det \left( \nabla_z \dec_{\param_1}(\enc_{\param_1}(X))^\top \nabla_z \dec_{\param_1}(\enc_{\param_1}(X)) \right) \right| \right] \\
& \text{subject to } \E_{X \sim \trueDens}\left[\Vert X - \dec_{\param_1}(\enc_{\param_1}(X))\Vert_2^2\right]=0.
\end{split}
\end{align}

In practice, the constraint can be encouraged by adding an $\ell_2$ regularization term to the likelihood:

\begin{equation}\label{eq:obj_rnfs}
\max_\param \E_{X \sim \trueDens}\left[ \log \latentDens_{\param_2}(\enc_{\param_1}(X)) - \tfrac{1}{2}\log \left|\det \left( \nabla_z \decoder(\enc_{\param_1}(X))^\top \nabla_z \dec_{\param_1}(\enc_{\param_1}(X)) \right) \right| - \beta \Vert X - \dec_{\param_1}(\enc_{\param_1}(X))\Vert_2^2 \right],
\end{equation}

where $\beta > 0$ is a hyperparameter. Much like how normalizing flows focus on tractability of the log-det-Jacobian term, a central theme of end-to-end injective flows is tractability of the second term in <ref>, which we will refer to as the “log-det-$J^\top J$” term.
One approach is to impose structural constraints on $\dec_{\theta_1}$ to make the log-det-$J^\top J$ term easily computable, albeit at the cost of expressiveness, for example using conformal embeddings [Ross & Cresswell, 2021].
A concurrent approach, pursued by Caterini et al., 2021, is to approximate the gradient of the log-det-$J^\top J$ term with respect to $\theta_1$ through a combination of Hutchinson's estimator [Hutchinson, 1989] and various tricks from linear algebra and automatic differentiation [Baydin et al., 2018].
Despite these approximations, tractability remains an issue with this technique and it struggles to scale to datasets of higher ambient dimensionality than CIFAR-10 [Krizhevsky & Hinton, 2009].

Denoising NFs [Horvat & Pfister, 2021]  – detailed in <ref>
– perform single-step training of a flow with an injective component for generation, but with a full-dimensional model $p^{X_\sigma}_\theta$ of a noised-out data density $p_*^{X_\sigma} = \trueDens \circledast \mathcal N(\, \cdot \,; 0, I_\aDim)$.
The objective for denoising NFs (<ref>) ends up quite similar to <ref>, although with two important differences: $(i)$ the expectation is over $ p_*^{X_\sigma}$ rather than $\trueDens$, and $(ii)$ the formulation of $p^{X_\sigma}_\theta$ as a model for $p_*^{X_\sigma}$ eliminates the need to optimize over the costly log-det-$J^\top J$ term.
However, the computational benefit comes at the cost of introducing a disconnect between the injective generator and the full-dimensional density model. In particular, the numerical instabilities described in <ref> do not apply to INFs trained through <ref> since no full-dimensional densities are involved, whereas they do apply denoising NFs.

Cunningham et al., 2022 and Flouris & Konukoglu, 2023 both propose injective variants of the flow models described in <ref> using similar objectives to Caterini et al., 2021. Cunningham et al., 2022 in particular use a regularizer that, with a certain hyperparameter setting, cancels out the log-det-$J^\top J$ term in the likelihood, making likelihood-based optimization of injective flows more efficient.

Meanwhile, several works parameterize the encoder $\encoder$ separately, as Kumar et al., 2020 did.
Zhang et al., 2020 proposed a similar objective to <ref> in the context of variational autoencoders (<ref>). Sorrenson et al., 2024 argue that the objective in <ref> is subject to similar pathologies to those outlined by Brehmer & Cranmer, 2020 for naïve maximum-likelihood training if the encoder and decoder are flexible enough.
This is interesting as it highlights that despite being directly motivated to account for the low-dimensional structure of the data, INFs trained through <ref> can nonetheless still fail to learn manifolds. Sorrenson et al., 2024 propose an intuitively well-motivated but theoretically ad-hoc modification to the objective, along with an improved gradient estimator over that of Caterini et al., 2021, for end-to-end training of INFs. Their model no longer falls under the umbrella of two-step models, and hence its ability to learn the manifold is unclear, but it does inherit computational benefits as it circumvents calculation of the challenging log-det-$J^\top J$ term [Brehmer & Cranmer, 2020].
Finally, Nazari et al., 2023 proposed an autoencoder for manifold learning (not generative modelling) which penalizes the variance of the log-det-$J^\top J$ term during training, and which results in a smoother and more interpretable latent space as compared to standard autoencoders.

§.§ Overcoming Topological Obstacles to Manifold Learning

Models for $\M = \{(x_1, x_2) \mid x_1^2 + x_2^2 = 1\}$, the unit circle in $\dataspace = \R^2$. (a) Using a single encoder-decoder pair to model the circle. The pair approaches numerical non-invertibility since the encoder $\encoder$ must map two nearby points in $\M$ (in black) to two distant latent points in $\latentspace$ (also in black).
(b) Illustration of implicit manifolds.
Here, $\M$ is characterized as the $0$-level set of $F: \R^2 \rightarrow \R$. Specifically, $F(x_1, x_2) = 1-(x_1^2 + x_2^2)$ is shown in red, and the grey plane corresponds to $\{x \in \R^3 \mid x_3=0\}$. Neural implicit manifolds use no autoencoders, and instead attempt to learn $F_\param$ so that its $0$-level set, $F_\param^{-1}(\{0\})$, matches $\M$.
(c) The problem depicted in (a) can also be circumvented by employing multiple encoder-decoder pairs, each one using its own latent space and covering a different part of $\M$.

As mentioned in <ref>, one might want to set the latent dimension $\lDim$ of an autoencoder to $\mDim$.
Yet, as discussed in <ref>, when $\lDim=\mDim$, perfect manifold learning is not always achievable through bottleneck methods for topological reasons. We illustrate this problem, which can cause downstream issues with density estimation, in <ref>. In particular, if $\M$ has any non-Euclidean topological properties such as holes or disconnected components, any attempt to model $\M$ as the image of a decoder $\decoder$ will cause numerical instability in $\decoder$ [Cornish et al., 2020, Salmona et al., 2022].
This problem was originally identified in the context of normalizing flows (<ref>), with a line of work that first appends additional dimensions to $\dataspace$, and then trains a normalizing flow on the augmented space [Dupont et al., 2019, Chen et al., 2020, Huang et al., 2020]. While these techniques indeed increase the stability and expressiveness of the flow itself, they still produce full-support densities that can never truly model non-trivial topological structures in data.
Alternatively, to more faithfully tackle topological issues, some works have proposed to restructure the manifold-learning step to better reflect the ways manifolds are defined in theory; we cover these approaches in detail below.

§.§.§ Neural Implicit Manifolds

Under some conditions, manifolds with non-trivial topologies can be defined using level sets of functions. In particular, a level set of a smooth function $\mdf: \R^{\aDim} \to \R^{\aDim - \mDim}$ represents a $\mDim$-dimensional manifold if its Jacobian has full rank on that level set [Lee, 2012].
We illustrate such an implicitly defined manifold – which cannot be characterized with a single encoder-decoder pair – in [fig:topology2]Figure 10(b).

Neural implicit manifold learning [Ross et al., 2023] operationalizes this fact by modelling $\M$ as the zero set of a neural network $\neuralMdf: \dataspace \to \R^{\aDim - \lDim}$. The network $\neuralMdf$ is trained to align its zero set $\neuralMdf^{-1}(\{0\})$ with $\M$ while being regularized to have full rank on $\M$. The following loss is used,

\begin{equation}\label{eq:mdf_loss}
\min_{\param_1} \E_{X \sim \trueDens, Y \sim q_{\param_1'}^X, V \sim \mathcal{U}(\, \cdot \, ;\mathcal{S}^{\aDim-\lDim-1})}\Big[\Vert \neuralMdf(X)\Vert_2-\alpha\Vert \neuralMdf(Y)\Vert_2+\beta\!\left(\eta -\Vert V^\top\! \nabla_x\neuralMdf(X)\Vert_2\right)_{+}^2\Big],
\end{equation}

where: $Y$ is independent of $X$; $q^X_{\param_1'}(y) \propto e^{-\Vert F_{\param_1'}(y) \Vert_2^2}$ with $\param_1' = \mathtt{stopgrad}(\param_1)$ (as in energy-based models, see <ref>);
$\mathcal{U}(\, \cdot \, ;\mathcal{S}^{\aDim - \lDim -1})$ is the uniform distribution over $\mathcal{S}^{\aDim - \lDim -1} \coloneqq \{y \in \R^{\aDim - \lDim} \mid \Vert y \Vert_2=1\}$, the $(\aDim{-}\lDim{-}1)$-sphere in $\R^{\aDim-\lDim}$; $\alpha > 0$, $\beta > 0$, and $\eta > 0$ are hyperparameters; and $(\cdot)_+=\max(\, \cdot \, , 0)$. In this loss, the first term ensures that $\neuralMdf^{-1}(\{0\})$ contains $\M$, the second term prevents $\neuralMdf^{-1}(\{0\})$ from containing off-manifold samples, and the third term regularizes the Jacobian of $\neuralMdf$ to have full rank on $\M$.

Importantly, $\neuralMdf$ here is not an encoder; it is best interpreted as defining $\aDim - \lDim$ non-linear constraints on the data, thus leaving $\lDim$ degrees of freedom for the data manifold. The full-rank requirement can then be interpreted as ensuring none of these constraints is redundant, thereby ensuring the learned manifold has the correct dimensionality. This constraint-based learning procedure for $\M$ makes implicit manifold learning unusual among two-step models (<ref>) in that it is not autoencoder-based. The lack of encoder in this method means there is no latent space, which presents a challenge for learning the distribution on the model manifold, $\neuralMdfOpt^{-1}(\{0\})$.

[Ross et al., 2023] propose to learn this distribution with the constrained energy-based model, which represents an EBM constrained to the learned manifold, $E: \neuralMdfOpt^{-1}(\{0\}) \to \R$. This is parameterized in practice by a neural network $E_{\param_2}: \dataspace \to \R$, for which values are ignored outside of the learned manifold $\neuralMdfOpt^{-1}(\{0\})$.
To sample from constrained EBMs, constrained Langevin dynamics [Brubaker et al., 2012] is used to generate samples from $E_{\param_2}$ constrained to the manifold. This allows for likelihood maximization via <ref>, as with ordinary EBMs – except constrained EBMs are manifold-supported.

§.§.§ Multi-Chart Manifolds

Typical techniques model the manifold globally using a single encoder-decoder pair. In general, however, manifolds can consist of a patchwork of many charts: mathematical objects that each serve roughly the same function as a single encoder-decoder pair (for a formal definition, please see Lee, 2012). Some manifolds may thus require many encoder-decoder pairs $(\encoder^{(i)}, \decoder^{(i)})_{i=1}^n$ – each using its latent space $\latentspace_i$ to locally describe some subset of the manifold – rather than a single one: we illustrate this fact in [fig:topology3]Figure 10(c).

Several works have taken this route to learn the data manifold. Schonsheck et al., 2019 first proposed a multi-chart latent space using multiple encoder-decoder pairs in a non-generative modelling context. For each incoming datapoint, the correct chart is selected dynamically using a prediction head for the encoder-decoder pair with the smallest reconstruction error. Kalatzis et al., 2021 propose multi-chart flows, in which likelihoods are defined using a mixture of injective normalizing flows (<ref>), with mixture weights again computed using a prediction head. On the other hand, Sidheekh et al., 2022 propose a mixture of INFs in which chart membership for an incoming datapoint is computed using discrete latent assignments.

Modelling a manifold with multiple charts imposes drawbacks. For one, each encoder-decoder pair has its own latent space, so for a given datapoint $x \in \M$, choosing the correct encoder can be a challenge. This makes it unclear how to perform tasks involving the manipulation of latent representations, such as interpolation. In many cases, there is no single correct encoder, as the images of various decoders need to overlap to correctly define topologically complex manifolds (such as in [fig:topology3]Figure 10(c)). The ambiguity of choosing the correct encoder is underlined by how differently each of the aforementioned methods attempt to do so.

§.§.§ Disconnected Manifolds

Another common source of topological complexity in the data manifold is when it consists of more than one connected component. This situation occurs, for example, in datasets with multiple disjoint classes. In this context, theoretical analyses have shown  that any decoder-based model will suffer from training instability [Salmona et al., 2022] and poor sample quality [Luzi et al., 2020].[Note that the theoretical analysis of Salmona et al., 2022 shows numerical instability when $\trueDens$ is multimodal, in which case its support $\M$ can be considered as numerically disconnected.]
One technique to improve sample quality is to avoid sampling from latent regions where the network $\decoderOpt$ is unstable; Tanielian et al., 2020 propose to reject samples $X = \dec_{\param^*}(Z)$, where $Z \sim \latentDens$, for which $\nabla_z \dec_{\param^*}(Z)$ has a high Frobenius norm, which they show indicates an off-manifold sample. Other work, discussed below, seeks to avoid instability entirely during training.

A few general techniques have been proposed for modelling disconnected manifolds. One way is to use disconnected (or near-disconnected) latent distributions, which aims to match the topology of the support of $\latentDens_\param$ with that of $\M$. This is typically done with a Gaussian mixture model for $\latentDens_\param$ and has been proposed for multiple classes of generative model [Nalisnick et al., 2016, Dilokthanakul et al., 2016, Jiang et al., 2017, Ben-Yosef & Weinshall, 2018, Izmailov et al., 2020].

A related approach is to use multiple decoder networks in a similar manner to the aforementioned multi-chart methods from <ref>. The model then becomes a mixture $\modelDens(x) = \sum_{i=1}^n \pi_i p_{\param, i}^X(x)$ of generative submodels $p_{\param, i}^X$, where $\pi_1, \ldots, \pi_n$ are the mixture weights (sometimes trainable). For example, Arora et al., 2017 propose to directly train a mixture of generative adversarial networks (<ref> and <ref>) to stabilize training, wherein the entire mixture is learned with backpropagation.
Cornish et al., 2020 use a hierarchical continuously-indexed mixture of NFs. Other work uses techniques based on expectation-maximization [Dempster et al., 1977] to train the mixture [Banijamali et al., 2017, Locatello et al., 2018]. A key challenge in this line of work is to encourage different submodels to model distinct parts of the distributions. This can be done by partitioning the data beforehand, by class [Luzi et al., 2020] or through unsupervised clustering [Brown et al., 2023], and training a model on each partition. A more flexible approach is to backpropagate through an ancillary classification model to encourage each submodel to generate data from distinct manifolds [Hoang et al., 2018, Khayatkhoei et al., 2018, Ghosh et al., 2018].

While all the models mentioned above can properly account for some topological features of $\M$ such as disconnectedness, we highlight that most are nonetheless manifold-unaware. For example, full-dimensional models trained through maximum-likelihood remain exposed to the corresponding pathologies (<ref>), even when they are mixture models.

§ DISCRETE DEEP GENERATIVE MODELS

Since this survey's focus is on the manifold hypothesis, all of the models presented thus far are for continuous distributions. Nonetheless, many DGMs assume that the ambient space $\dataspace$ is discrete. For example, images can be modelled as having pixels which take only finitely many different values, rather than a continuum of them. In this case, $\trueDens$ and $\modelDens$ are both probability mass functions over $\dataspace$, and $\M \subset \dataspace$ denotes the support of $\trueDens$. Formally, in this setting $\dataspace$ is a $0$-dimensional manifold, so that even when $\M$ is a strict subset of $\dataspace$, it remains a $0$-dimensional submanifold. In other words, there can be no dimensionality mismatch for discrete data since $\aDim=\mDim=0$. In turn, this implies that mathematically, discrete likelihood-based DGMs are not exposed to problems such as manifold overfitting (<ref>) which arise from dimensionality mismatch. This view of discrete DGMs through the manifold lens is useful, since it suggests that whenever a manifold-unaware DGM admits a straightforward discrete analogue, the latter should be preferred as it will be unaffected by manifold-related woes. Indeed, as mentioned in <ref>, discrete variational autoencoders [Gulrajani et al., 2017, Vahdat & Kautz, 2020, Vahdat et al., 2021] empirically outperform their continuous variants. Similarly, discrete incarnations of likelihood-based autoregressive DGMs [Germain et al., 2015, van den Oord et al., 2016, Salimans et al., 2017, Parmar et al., 2018] outperform continuous ones [Uria et al., 2013]. In contrast, discrete versions of diffusion models [Austin et al., 2021, Campbell et al., 2022, Meng et al., 2022] do not outperform their manifold-aware continuous counterparts (<ref>) when modelling images.

Discrete and continuous DGMs nonetheless have similarities, despite the differences outlined above. As discussed in <ref>, a key motivation behind the manifold hypothesis is to capture the intuition that $\M$, the support of $\trueDens$, is somehow sparse within $\dataspace$. This intuition often remains true in the discrete case: using images as an example once again, there are $256^\aDim$ possible discrete images (assuming each pixel entry takes one of $256$ possible values), yet the subset of natural images is vanishingly small in comparison and contains orders of magnitude fewer elements. The main idea of continuous two-step models (<ref>), namely to first approximate the support of $\trueDens$ and then learn the distribution within, remains equally sensible in the discrete case. van den Oord et al., 2017 proposed an autoencoder which recovers discrete representations over which they train a discrete DGM; this idea that has been further developed, with strong empirical results [Razavi et al., 2019, Esser et al., 2021, Ramesh et al., 2021, Chang et al., 2022]. We finish by pointing out that the discussion in <ref> applies to all these discrete two-step models, so that they can be interpreted as minimizing a potentially regularized upper bound of the Wasserstein distance between $\trueDens$ and $\modelDens$ which becomes tight at optimality, because in the discrete case, perfect reconstructions are always achievable given enough capacity of the encoder and decoder.

§ CONCLUSIONS AND FUTURE OUTLOOK

[Ackley et al., 1985]
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski.
A learning algorithm for Boltzmann machines.
Cognitive Science, 90 (1):0 147–169, 1985.

[Albergo & Vanden-Eijnden, 2023]
Michael S Albergo and Eric Vanden-Eijnden.
Building normalizing flows with stochastic interpolants.
In International Conference on Learning Representations, 2023.

[Alemi et al., 2017]
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy.
Deep variational information bottleneck.
In International Conference on Learning Representations, 2017.

[Anderson, 1982]
Brian DO Anderson.
Reverse-time diffusion equation models.
Stochastic Processes and their Applications, 120
(3):0 313–326, 1982.

[Arbel et al., 2021]
Michael Arbel, Liang Zhou, and Arthur Gretton.
Generalized energy based models.
In International Conference on Learning Representations, 2021.

[Arjovsky et al., 2017]
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein generative adversarial networks.
In International Conference on Machine Learning, pp.  214–223, 2017.

[Arora et al., 2017]
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang.
Generalization and equilibrium in generative adversarial nets
In International Conference on Machine Learning, pp.  224–232, 2017.

[Austin et al., 2021]
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van
Den Berg.
Structured denoising diffusion models in discrete state-spaces.
In Advances in Neural Information Processing Systems, 2021.

[Bac et al., 2021]
Jonathan Bac, Evgeny M Mirkes, Alexander N Gorban, Ivan Tyukin, and Andrei
Scikit-dimension: A python package for intrinsic dimension
Entropy, 230 (10), 2021.

[Banijamali et al., 2017]
Ershad Banijamali, Ali Ghodsi, and Pascal Poupart.
Generative mixture of networks.
In International Joint Conference on Neural Networks (IJCNN),
pp.  3753–3760, 2017.

[Baydin et al., 2018]
Atılım Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul,
and Jeffrey Mark Siskind.
Automatic differentiation in machine learning: a survey.
Journal of Machine Learning Research, 18:0 1–43,

[Beals et al., 1968]
Richard Beals, David H Krantz, and Amos Tversky.
Foundations of multidimensional scaling.
Psychological Review, 750 (2):0 127, 1968.

[Behrmann et al., 2021]
Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Jörn-Henrik
Understanding and mitigating exploding inverses in invertible neural
In International Conference on Artificial Intelligence and
Statistics, pp.  1792–1800, 2021.

[Ben-Hamu et al., 2022]
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel,
Aditya Grover, Ricky TQ Chen, and Yaron Lipman.
Matching normalizing flows and probability paths on manifolds.
In International Conference on Machine Learning, pp.  1749–1763, 2022.

[Ben-Yosef & Weinshall, 2018]
Matan Ben-Yosef and Daphna Weinshall.
Gaussian mixture generative adversarial networks for diverse
datasets, and the unsupervised clustering of images.
arXiv:1808.10356, 2018.

[Bengio et al., 2013]
Yoshua Bengio, Aaron Courville, and Pascal Vincent.
Representation learning: A review and new perspectives.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 350 (8):0 1798–1828, 2013.

[Berenfeld & Hoffmann, 2021]
Clément Berenfeld and Marc Hoffmann.
Density estimation on an unknown submanifold.
Electronic Journal of Statistics, 150 (1):0
2179 – 2223, 2021.

[Berenfeld et al., 2022]
Clément Berenfeld, Paul Rosa, and Judith Rousseau.
Estimating a density near an unknown manifold: A Bayesian
nonparametric approach.
arXiv:2205.15717, 2022.

[Billingsley, 2012]
Patrick Billingsley.
Probability and Measure.
Wiley, 2012.

[Birrell et al., 2022]
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Luc
$(f, \gamma)$-Divergences: Interpolating between $f$-divergences
and integral probability metrics.
The Journal of Machine Learning Research, 23:0
1816–1885, 2022.

[Boehm & Seljak, 2022]
Vanessa M Boehm and Uros Seljak.
Probabilistic autoencoder.
Transactions of Machine Learning Research, 2022.

[Bogachev, 2007]
Vladimir Igorevich Bogachev.
Measure Theory, volume 2.
Springer, 2007.

[Bond-Taylor et al., 2022]
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks.
Deep generative modelling: A comparative review of VAEs, GANs,
normalizing flows, energy-based and autoregressive models.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 440 (11):0 7327–7347, 2022.

[Bonet et al., 2024]
Clément Bonet, Lucas Drumetz, and Nicolas Courty.
Sliced-Wasserstein distances and flows on Cartan-Hadamard
arXiv:2403.06560, 2024.

[Borji, 2019]
Ali Borji.
Pros and cons of GAN evaluation measures.
Computer Vision and Image Understanding, 179:0 41–65,

[Bose et al., 2020]
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton.
Latent variable modelling with hyperbolic normalizing flows.
In International Conference on Machine Learning, pp.  1045–1055, 2020.

[Boyda et al., 2021]
Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez
Rezende, Michael S Albergo, Kyle Cranmer, Daniel C Hackett, and Phiala E
Sampling using SU(N) gauge equivariant flows.
Physical Review D, 1030 (7):0 074504, 2021.

[Brehmer & Cranmer, 2020]
Johann Brehmer and Kyle Cranmer.
Flows for simultaneous manifold learning and density estimation.
In Advances in Neural Information Processing Systems, 2020.

[Brock et al., 2019]
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale GAN training for high fidelity natural image synthesis.
In International Conference on Learning Representations, 2019.

[Brown et al., 2023]
Bradley CA Brown, Anthony L Caterini, Brendan Leigh Ross, Jesse C Cresswell,
and Gabriel Loaiza-Ganem.
Verifying the union of manifolds hypothesis for image data.
In International Conference on Learning Representations, 2023.

[Brubaker et al., 2012]
Marcus Brubaker, Mathieu Salzmann, and Raquel Urtasun.
A family of MCMC methods on implicitly defined manifolds.
In International Conference on Artificial Intelligence and
Statistics, pp.  161–172, 2012.

[Campbell et al., 2022]
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George
Deligiannidis, and Arnaud Doucet.
A continuous time framework for discrete denoising models.
In Advances in Neural Information Processing Systems, 2022.

[Caterini et al., 2021]
Anthony L Caterini, Rob Cornish, Dino Sejdinovic, and Arnaud Doucet.
Variational inference with continuously-indexed normalizing flows.
In Uncertainty in Artificial Intelligence, pp.  44–53,

[Caterini et al., 2021]
Anthony L Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John P Cunningham.
Rectangular flows for manifold learning.
In Advances in Neural Information Processing Systems,

[Chadebec & Allassonnière, 2022]
Clément Chadebec and Stéphanie Allassonnière.
A geometric perspective on variational autoencoders.
In Advances in Neural Information Processing Systems, 2022.

[Chae et al., 2023]
Minwoo Chae, Dongha Kim, Yongdai Kim, and Lizhen Lin.
A likelihood approach to nonparametric estimation of a singular
distribution using deep generative models.
Journal of Machine Learning Research, 240
(77):0 1–42, 2023.

[Chakraborty & Bartlett, 2024]
Saptarshi Chakraborty and Peter Bartlett.
A statistical analysis of Wasserstein autoencoders for
intrinsically low-dimensional data.
In International Conference on Learning Representations,

[Chakraborty & Bartlett, 2024]
Saptarshi Chakraborty and Peter L Bartlett.
On the statistical properties of generative adversarial models for
low intrinsic data dimension.
arXiv:2401.15801, 2024b.

[Chang et al., 2022]
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.
MaskGIT: Masked generative image transformer.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  11315–11325, 2022.

[Che et al., 2020]
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull,
Yuan Cao, and Yoshua Bengio.
Your GAN is secretly an energy-based model and you should use
discriminator driven latent sampling.
In Advances in Neural Information Processing Systems, 2020.

[Chen et al., 2020]
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian.
VFlow: More expressive generative flows with variational data
In International Conference on Machine Learning, pp.  1660–1669, 2020.

[Chen et al., 2023]
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang.
Score approximation, estimation and distribution recovery of
diffusion models on low-dimensional data.
In International Conference on Machine Learning, pp.  4672–4712, 2023.

[Chen & Lipman, 2024]
Ricky TQ Chen and Yaron Lipman.
Flow matching on general geometries.
In International Conference on Learning Representations, 2024.

[Chen et al., 2018]
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.
In Advances in Neural Information Processing Systems, 2018.

[Chen et al., 2017]
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John
Schulman, Ilya Sutskever, and Pieter Abbeel.
Variational lossy autoencoder.
In International Conference on Learning Representations, 2017.

[Child, 2021]
Rewon Child.
Very deep VAEs generalize autoregressive models and can outperform
them on images.
In International Conference on Learning Representations, 2021.

[Cornish et al., 2020]
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet.
Relaxing bijectivity constraints with continuously indexed
normalising flows.
In International Conference on Machine Learning, pp.  2133–2143, 2020.

[Cresswell et al., 2022]
Jesse C Cresswell, Brendan Leigh Ross, Gabriel Loaiza-Ganem, Humberto
Reyes-Gonzalez, Marco Letizia, and Anthony L Caterini.
CaloMan: Fast generation of calorimeter showers with density
estimation on learned manifolds.
arXiv:2211.15380, 2022.

[Cunningham et al., 2022]
Edmond Cunningham, Adam D Cobb, and Susmit Jha.
Principal component flows.
In International Conference on Machine Learning, pp.  4492–4519, 2022.

[Dai & Wipf, 2019]
Bin Dai and David Wipf.
Diagnosing and enhancing VAE models.
In International Conference on Learning Representations, 2019.

[Dao et al., 2023]
Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran.
Flow matching in latent space.
arXiv:2307.08698, 2023.

[De Bortoli, 2022]
Valentin De Bortoli.
Convergence of denoising diffusion models under the manifold
Transactions on Machine Learning Research, 2022.

[De Bortoli et al., 2022]
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton,
Yee Whye Teh, and Arnaud Doucet.
Riemannian score-based generative modeling.
In Advances in Neural Information Processing Systems, 2022.

[Dempster et al., 1977]
Arthur P Dempster, Nan M Laird, and Donald B Rubin.
Maximum likelihood from incomplete data via the EM algorithm.
Journal of the royal statistical society: series B
(methodological), 390 (1):0 1–22, 1977.

[Dieudonné, 1973]
Jean Dieudonné.
Treatise on Analysis, volume 3.
Academic Press, 1973.

[Dilokthanakul et al., 2016]
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh
Salimbeni, Kai Arulkumaran, and Murray Shanahan.
Deep unsupervised clustering with Gaussian mixture variational
arXiv:1611.02648, 2016.

[Dinh et al., 2014]
Laurent Dinh, David Krueger, and Yoshua Bengio.
NICE: Non-linear independent components estimation.
arXiv:1410.8516, 2014.

[Dinh et al., 2017]
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.
Density estimation using Real NVP.
In International Conference on Learning Representations, 2017.

[Divol, 2021]
Vincent Divol.
Reconstructing measures on manifolds: An optimal transport approach.
arXiv:2102.07595, 2021.

[Donahue et al., 2017]
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.
Adversarial feature learning.
In International Conference on Learning Representations, 2017.

[Du & Mordatch, 2019]
Yilun Du and Igor Mordatch.
Implicit generation and modeling with energy based models.
In Advances in Neural Information Processing Systems, 2019.

[Dupont et al., 2019]
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.
Augmented neural ODEs.
In Advances in Neural Information Processing Systems, 2019.

[Durkan et al., 2019]
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.
Neural spline flows.
In Advances in Neural Information Processing Systems, 2019.

[Dziugaite et al., 2015]
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani.
Training generative neural networks via maximum mean discrepancy
In Uncertainty in Artificial Intelligence, pp.  258–267,

[Esser et al., 2021]
Patrick Esser, Robin Rombach, and Bjorn Ommer.
Taming transformers for high-resolution image synthesis.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  12873–12883, 2021.

[Facco et al., 2017]
Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio.
Estimating the intrinsic dimension of datasets by a minimal
neighborhood information.
Scientific Reports, 70 (1):0 12140, 2017.

[Flouris & Konukoglu, 2023]
Kyriakos Flouris and Ender Konukoglu.
Canonical normalizing flows for manifold learning.
In Advances in Neural Information Processing Systems, 2023.

[Gao & Zhu, 2024]
Xuefeng Gao and Lingjiong Zhu.
Convergence analysis for general probability flow ODEs of diffusion
models in Wasserstein distances.
arXiv:2401.17958, 2024.

[Gemici et al., 2016]
Mevlana C Gemici, Danilo Jimenez Rezende, and Shakir Mohamed.
Normalizing flows on Riemannian manifolds.
arXiv:1611.02304, 2016.

[Germain et al., 2015]
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle.
MADE: Masked autoencoder for distribution estimation.
In International Conference on Machine Learning, pp.  881–889, 2015.

[Ghojogh et al., 2023]
Benyamin Ghojogh, Mark Crowley, Fakhri Karray, and Ali Ghodsi.
Elements of Dimensionality Reduction and Manifold Learning.
Springer Nature, 2023.

[Ghosh et al., 2018]
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K
Multi-agent diverse generative adversarial networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  8513–8521, 2018.

[Ghosh et al., 2020]
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard
From variational to deterministic autoencoders.
In International Conference on Learning Representations, 2020.

[Goodfellow et al., 2014]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in Neural Information Processing Systems, 2014.

[Grathwohl et al., 2019]
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David
FFJORD: Free-form continuous dynamics for scalable reversible
generative models.
In International Conference on Learning Representations, 2019.

[Gretton et al., 2006]
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and
Alex Smola.
A kernel method for the two-sample-problem.
In Advances in Neural Information Processing Systems, 2006.

[Gulrajani et al., 2017]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C
Improved training of Wasserstein GANs.
In Advances in Neural Information Processing Systems,

[Gulrajani et al., 2017]
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin,
David Vazquez, and Aaron Courville.
PixelVAE: A latent variable model for natural images.
In International Conference on Learning Representations,

[Haussmann & Pardoux, 1986]
UG Haussmann and E Pardoux.
Time reversal of diffusions.
The Annals of Probability, 140 (4):0
1188–1205, 1986.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  770–778, 2016.

[Heusel et al., 2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
GANs trained by a two time-scale update rule converge to a local
Nash equilibrium.
In Advances in Neural Information Processing Systems, 2017.

[Higgins et al., 2017]
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot,
Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.
beta-VAE: Learning basic visual concepts with a constrained
variational framework.
In International Conference on Learning Representations, 2017.

[Ho et al., 2019]
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.
Flow++: Improving flow-based generative models with variational
dequantization and architecture design.
In International Conference on Machine Learning, pp.  2722–2730, 2019.

[Ho et al., 2020]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In Advances in Neural Information Processing Systems, 2020.

[Hoang et al., 2018]
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung.
MGAN: Training generative adversarial nets with multiple
In International Conference on Learning Representations, 2018.

[Hornik, 1991]
Kurt Hornik.
Approximation capabilities of multilayer feedforward networks.
Neural Networks, 40 (2):0 251–257, 1991.

[Horvat & Pfister, 2021]
Christian Horvat and Jean-Pascal Pfister.
Denoising normalizing flow.
In Advances in Neural Information Processing Systems, 2021.

[Horvat & Pfister, 2023]
Christian Horvat and Jean-Pascal Pfister.
Density estimation on low-dimensional manifolds: An
inflation-deflation approach.
Journal of Machine Learning Research, 240
(61):0 1–37, 2023.

[Horvat & Pfister, 2024]
Christian Horvat and Jean-Pascal Pfister.
On gauge freedom, conservativity and intrinsic dimensionality
estimation in diffusion models.
In International Conference on Learning Representations, 2024.

[Hu et al., 2023]
Tianyang Hu, Fei Chen, Haonan Wang, Jiawei Li, Wenjia Wang, Jiacheng Sun, and
Zhenguo Li.
Complexity matters: Rethinking the latent space for generative
In Advances in Neural Information Processing Systems, 2023.

[Huang et al., 2020]
Chin-Wei Huang, Laurent Dinh, and Aaron Courville.
Augmented normalizing flows: Bridging the gap between generative
flows and latent variable models.
arXiv preprint arXiv:2002.07101, 2020.

[Huang et al., 2022]
Jian Huang, Yuling Jiao, Zhen Li, Shiao Liu, Yang Wang, and Yunfei Yang.
An error analysis of generative adversarial networks for learning
Journal of Machine Learning Research, 230
(1):0 5047–5089, 2022.

[Hurewicz & Wallman, 1948]
Witold Hurewicz and Henry Wallman.
Dimension Theory (PMS-4).
Princeton University Press, 1948.

[Hutchinson, 1989]
M F Hutchinson.
A stochastic estimator of the trace of the influence matrix for
Laplacian smoothing splines.
Communications in Statistics - Simulation and Computation,
180 (3):0 1059–1076, 1989.

[Hyvärinen, 2005]
Aapo Hyvärinen.
Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 60
(24):0 695–709, 2005.

[Izmailov et al., 2020]
Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson.
Semi-supervised learning with normalizing flows.
In International Conference on Machine Learning, pp.  4615–4630, 2020.

[Jayasiri & Wijerathne, 2020]
Varuna Jayasiri and Nipun Wijerathne.
Annotated paper implementations, 2020.
URL <https://nn.labml.ai/>.

[Jiang et al., 2017]
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou.
Variational deep embedding: An unsupervised and generative approach
to clustering.
In Proceedings of the Twenty-Sixth International Joint
Conference on Artificial Intelligence (IJCAI), 2017.

[Johnsson et al., 2014]
Kerstin Johnsson, Charlotte Soneson, and Magnus Fontes.
Low bias local intrinsic dimension estimation from expected simplex
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 370 (1):0 196–202, 2014.

[Kalatzis et al., 2021]
Dimitris Kalatzis, Johan Ziruo Ye, Alison Pouplin, Jesper Wohlert, and Søren
Density estimation on smooth manifolds with normalizing flows.
arXiv:2106.03500, 2021.

[Kamkari et al., 2024]
Hamidreza Kamkari, Brendan Leigh Ross, Jesse C Cresswell, Anthony L Caterini,
Rahul G Krishnan, and Gabriel Loaiza-Ganem.
A geometric explanation of the likelihood OOD detection paradox.
arXiv:2403.18910, 2024.

[Kanwar et al., 2020]
Gurtej Kanwar, Michael S Albergo, Denis Boyda, Kyle Cranmer, Daniel C Hackett,
Sébastien Racaniere, Danilo Jimenez Rezende, and Phiala E Shanahan.
Equivariant flow-based sampling for lattice gauge theory.
Physical Review Letters, 1250 (12):0 121601,

[Karras et al., 2018]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability, and
In International Conference on Learning Representations, 2018.

[Karras et al., 2019]
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  4401–4410, 2019.

[Karras et al., 2020]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and
Timo Aila.
Analyzing and improving the image quality of StyleGAN.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  8110–8119, 2020.

[Katsman et al., 2021]
Isay Katsman, Aaron Lou, Derek Lim, Qingxuan Jiang, Ser Nam Lim, and
Christopher M De Sa.
Equivariant manifold flows.
In Advances in Neural Information Processing Systems, 2021.

[Khalil, 2002]
Hassan K Khalil.
Nonlinear Systems.
Prentice Hall, 2002.

[Khayatkhoei et al., 2018]
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal.
Disconnected manifold learning for generative adversarial networks.
In Advances in Neural Information Processing Systems, 2018.

[Kim et al., 2022]
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon.
Soft truncation: A universal training technique of score-based
diffusion model for high precision score estimation.
In International Conference on Machine Learning, pp.  11201–11228, 2022.

[Kim et al., 2020]
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim.
Softflow: Probabilistic framework for normalizing flow on manifolds.
In Advances in Neural Information Processing Systems, 2020.

[Kingma & Ba, 2015]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations, 2015.

[Kingma & Dhariwal, 2018]
Diederik P Kingma and Prafulla Dhariwal.
Glow: Generative flow with invertible 1x1 convolutions.
In Advances in Neural Information Processing Systems, 2018.

[Kingma & Gao, 2023]
Diederik P Kingma and Ruiqi Gao.
Understanding diffusion objectives as the ELBO with simple data
Advances in Neural Information Processing Systems, 2023.

[Kingma & Welling, 2014]
Diederik P Kingma and Max Welling.
Auto-encoding variational Bayes.
In International Conference on Learning Representations, 2014.

[Kingma et al., 2016]
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and
Max Welling.
Improved variational inference with inverse autoregressive flow.
In Advances in Neural Information Processing Systems, 2016.

[Kobyzev et al., 2020]
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker.
Normalizing flows: An introduction and review of current methods.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 430 (11):0 3964–3979, 2020.

[Koehler et al., 2021]
Frederic Koehler, Viraj Mehta, and Andrej Risteski.
Representational aspects of depth and conditioning in normalizing
In International Conference on Machine Learning, pp.  5628–5636, 2021.

[Koehler et al., 2022]
Frederic Koehler, Viraj Mehta, Chenghui Zhou, and Andrej Risteski.
Variational autoencoders in the presence of low-dimensional data:
Landscape and implicit bias.
In International Conference on Learning Representations, 2022.

[Kolouri et al., 2018]
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde.
Sliced Wasserstein auto-encoders.
In International Conference on Learning Representations, 2018.

[Kothari et al., 2021]
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić.
Trumpets: Injective flows for inference and inverse problems.
In Uncertainty in Artificial Intelligence, pp.  1269–1278,

[Köthe, 2023]
Ullrich Köthe.
A review of change of variable formulas for generative modeling.
arXiv:2308.02652, 2023.

[Kramer, 1991]
Mark A Kramer.
Nonlinear principal component analysis using autoassociative neural
AIChE Journal, 370 (2):0 233–243, 1991.

[Krizhevsky & Hinton, 2009]
Alex Krizhevsky and Geoffrey Hinton.
Learning multiple layers of features from tiny images.
Technical report, University of Toronto, 2009.

[Kruskal, 1964]
Joseph B Kruskal.
Multidimensional scaling by optimizing goodness of fit to a nonmetric
Psychometrika, 290 (1):0 1–27, 1964.

[Kumar et al., 2020]
Abhishek Kumar, Ben Poole, and Kevin Murphy.
Regularized autoencoders via relaxed injective probability flow.
In International Conference on Artificial Intelligence and
Statistics, pp.  4292–4301, 2020.

[Kwon et al., 2022]
Dohyun Kwon, Ying Fan, and Kangwook Lee.
Score-based generative modeling secretly minimizes the Wasserstein
Advances in Neural Information Processing Systems, 2022.

[Larsen et al., 2016]
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and
Ole Winther.
Autoencoding beyond pixels using a learned similarity metric.
In International Conference on Machine Learning, pp.  1558–1566, 2016.

[Lee, 2012]
John M Lee.
Introduction to Smooth Manifolds.
Springer, 2nd edition, 2012.

[Lee, 2018]
John M Lee.
Introduction to Riemannian Manifolds.
Springer, 2nd edition, 2018.

[Lehmann & Casella, 2006]
Erich L Lehmann and George Casella.
Theory of Point Estimation.
Springer Science & Business Media, 2006.

[Levina & Bickel, 2004]
Elizaveta Levina and Peter Bickel.
Maximum likelihood estimation of intrinsic dimension.
In Advances in Neural Information Processing Systems, 2004.

[Li et al., 2017]
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás
MMD GAN: Towards deeper understanding of moment matching network.
In Advances in Neural Information Processing Systems, 2017.

[Li et al., 2015]
Yujia Li, Kevin Swersky, and Richard Zemel.
Generative moment matching networks.
In International Conference on Machine Learning, pp.  1718–1727, 2015.

[Lipman et al., 2023]
Yaron Lipman, Ricky T Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.
Flow matching for generative modeling.
In International Conference on Learning Representations, 2023.

[Liu et al., 2023]
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with
rectified flow.
In International Conference on Learning Representations, 2023.

[Loaiza-Ganem et al., 2022]
Gabriel Loaiza-Ganem, Brendan Leigh Ross, Jesse C Cresswell, and Anthony L
Diagnosing and fixing manifold overfitting in deep generative models.
Transactions on Machine Learning Research, 2022a.

[Loaiza-Ganem et al., 2022]
Gabriel Loaiza-Ganem, Brendan Leigh Ross, Luhuan Wu, John Patrick Cunningham,
Jesse C Cresswell, and Anthony L Caterini.
Denoising deep generative models.
In Proceedings on "I Can't Believe It's Not Better! -
Understanding Deep Learning Through Empirical Falsification" at NeurIPS 2022
Workshops, 2022b.

[Locatello et al., 2018]
Francesco Locatello, Damien Vincent, Ilya Tolstikhin, Gunnar Rätsch,
Sylvain Gelly, and Bernhard Schölkopf.
Competitive training of mixtures of independent deep generative
arXiv:1804.11130, 2018.

[Lou et al., 2023]
Aaron Lou, Minkai Xu, and Stefano Ermon.
Scaling Riemannian diffusion models.
In Advances in Neural Information Processing Systems, 2023.

[Lu et al., 2023]
Yubin Lu, Zhongjian Wang, and Guillaume Bal.
Mathematical analysis of singularities in the diffusion model under
the submanifold assumption.
arXiv:2301.07882, 2023.

[Luzi et al., 2020]
Lorenzo Luzi, Randall Balestriero, and Richard G Baraniuk.
Ensembles of generative adversarial networks for disconnected data.
arXiv:2006.14600, 2020.

[MacKay & Ghahramani, 2005]
David JC MacKay and Zoubin Ghahramani.
Comments on “Maximum likelihood estimation of intrinsic dimension'
by E. Levina and P. Bickel (2004).
The Inference Group Website, Cavendish Laboratory, Cambridge
University, 2005.

[Mardia et al., 2016]
Kanti V Mardia, John T Kent, and Arnab K Laha.
Score matching estimators for directional distributions.
arXiv:1604.08470, 2016.

[Mathieu & Nickel, 2020]
Emile Mathieu and Maximilian Nickel.
Riemannian continuous normalizing flows.
In Advances in Neural Information Processing Systems, 2020.

[McInnes et al., 2018]
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger.
UMAP: Uniform manifold approximation and projection.
Journal of Open Source Software, 30 (29):0
861, 2018.

[Meng et al., 2022]
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon.
Concrete score matching: Generalized score matching for discrete
In Advances in Neural Information Processing Systems, 2022.

[Miyato et al., 2018]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
Spectral normalization for generative adversarial networks.
In International Conference on Learning Representations, 2018.

[Munkres, 2014]
James R Munkres.
Pearson Education, 2014.

[Murphy, 2012]
Kevin P Murphy.
Machine Learning: A Probabilistic Perspective.
MIT Press, 2012.

[Nalisnick et al., 2016]
Eric Nalisnick, Lars Hertel, and Padhraic Smyth.
Approximate inference for deep latent Gaussian mixtures.
In NeurIPS Workshop on Bayesian Deep Learning, volume 2, pp.  131, 2016.

[Narayanan & Mitter, 2010]
Hariharan Narayanan and Sanjoy Mitter.
Sample complexity of testing the manifold hypothesis.
In Advances in Neural Information Processing Systems, 2010.

[Narayanan & Niyogi, 2009]
Hariharan Narayanan and Partha Niyogi.
On the sample complexity of learning smooth cuts on a manifold.
In Conference on Learning Theory, 2009.

[Nazari et al., 2023]
Philipp Nazari, Sebastian Damrich, and Fred A Hamprecht.
Geometric autoencoders - what you see is what you decode.
In International Conference on Machine Learning, pp.  25834–25857, 2023.

[Nichol et al., 2022]
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela
Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.
GLIDE: Towards photorealistic image generation and editing with
text-guided diffusion models.
In International Conference on Machine Learning, pp.  16784–16804, 2022.

[Nowozin et al., 2016]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.
f-GAN: Training generative neural samplers using variational
divergence minimization.
In Advances in Neural Information Processing Systems, 2016.

[Oko et al., 2023]
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki.
Diffusion models are minimax optimal distribution estimators.
In International Conference on Machine Learning, pp.  26517–26582, 2023.

[Øksendal, 2003]
Bernt Øksendal.
Stochastic Differential Equations, pp.  65–84.
Springer Science & Business Media, 2003.

[Ozakin & Gray, 2009]
Arkadas Ozakin and Alexander Gray.
Submanifold density estimation.
In Advances in Neural Information Processing Systems, 2009.

[Pang et al., 2020]
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu.
Learning latent space energy-based prior model.
In Advances in Neural Information Processing Systems, 2020.

[Papamakarios et al., 2017]
George Papamakarios, Theo Pavlakou, and Iain Murray.
Masked autoregressive flow for density estimation.
In Advances in Neural Information Processing Systems, 2017.

[Papamakarios et al., 2021]
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed,
and Balaji Lakshminarayanan.
Normalizing flows for probabilistic modeling and inference.
Journal of Machine Learning Research, 220
(57):0 1–64, 2021.

[Parmar et al., 2018]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer,
Alexander Ku, and Dustin Tran.
Image transformer.
In International Conference on Machine Learning, pp.  4055–4064, 2018.

[Patrini et al., 2020]
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth
Bhargav, Max Welling, Tim Genewein, and Frank Nielsen.
Sinkhorn autoencoders.
In Uncertainty in Artificial Intelligence, pp.  733–743,

[Pearson, 1901]
Karl Pearson.
LIII. On lines and planes of closest fit to systems of points in
The London, Edinburgh, and Dublin Philosophical Magazine and
Journal of Science, 20 (11):0 559–572, 1901.

[Peebles & Xie, 2023]
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  4195–4205, 2023.

[Pennec, 2006]
Xavier Pennec.
Intrinsic statistics on Riemannian manifolds: Basic tools for
geometric measurements.
Journal of Mathematical Imaging and Vision, 250
(1):0 127–154, 2006.

[Peyré & Cuturi, 2019]
Gabriel Peyré and Marco Cuturi.
Computational optimal transport.
Foundations and Trends in Machine Learning, 110
(5-6):0 355–607, 2019.

[Pidstrigach, 2022]
Jakiw Pidstrigach.
Score-based generative models detect manifolds.
In Advances in Neural Information Processing Systems, 2022.

[Polyanskiy & Wu, 2022]
Yury Polyanskiy and Yihong Wu.
Information Theory: From Coding to Learning.
Cambridge University Press, 2022.

[Pope et al., 2021]
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.
The intrinsic dimension of images and its impact on learning.
In International Conference on Learning Representations, 2021.

[Postels et al., 2022]
Janis Postels, Martin Danelljan, Luc Van Gool, and Federico Tombari.
Maniflow: Implicitly representing manifolds with normalizing flows.
In International Conference on 3D Vision, pp.  84–93, 2022.

[Puthawala et al., 2022]
Michael Puthawala, Matti Lassas, Ivan Dokmanic, and Maarten De Hoop.
Universal joint approximation of manifolds and densities by simple
injective flows.
In International Conference on Machine Learning, pp.  17959–17983, 2022.

[Radford et al., 2015]
Alec Radford, Luke Metz, and Soumith Chintala.
Unsupervised representation learning with deep convolutional
generative adversarial networks.
In International Conference on Learning Representations, 2015.

[Ramesh et al., 2021]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec
Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation.
In International Conference on Machine Learning, pp.  8821–8831, 2021.

[Ramesh et al., 2022]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with CLIP latents.
arXiv:2204.06125, 2022.

[Razavi et al., 2019]
Ali Razavi, Aäron van den Oord, and Oriol Vinyals.
Generating diverse high-fidelity images with VQ-VAE-2.
In Advances in Neural Information Processing Systems, 2019.

[Rezende & Mohamed, 2015]
Danilo Jimenez Rezende and Shakir Mohamed.
Variational inference with normalizing flows.
In International Conference on Machine Learning, pp.  1530–1538, 2015.

[Rezende et al., 2014]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.
Stochastic backpropagation and approximate inference in deep
generative models.
In International Conference on Machine Learning, pp.  1278–1286, 2014.

[Rezende et al., 2020]
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael
Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer.
Normalizing flows on tori and spheres.
In International Conference on Machine Learning, pp.  8083–8092, 2020.

[Robbins, 1956]
Herbert Robbins.
An empirical Bayes approach to statistics.
In Third Berkeley Symposium on Mathematical Statistics and
Probability, volume 1, pp.  157–163, 1956.

[Robbins & Monro, 1951]
Herbert Robbins and Sutton Monro.
A stochastic approximation method.
The Annals of Mathematical Statistics, 220
(3):0 400–407, 1951.

[Rombach et al., 2022]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  10684–10695, 2022.

[Ronneberger et al., 2015]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-Net: Convolutional networks for biomedical image segmentation.
In Medical Image Computing and Computer-Assisted Intervention
– MICCAI 2015, pp.  234–241. Springer, 2015.

[Ross & Cresswell, 2021]
Brendan Leigh Ross and Jesse C Cresswell.
Tractable density estimation on learned manifolds with conformal
embedding flows.
In Advances in Neural Information Processing Systems, 2021.

[Ross et al., 2023]
Brendan Leigh Ross, Gabriel Loaiza-Ganem, Anthony L Caterini, and Jesse C
Neural implicit manifold learning for topology-aware generative
Transactions on Machine Learning Research, 2023.

[Roweis & Saul, 2000]
Sam T Roweis and Lawrence K Saul.
Nonlinear dimensionality reduction by locally linear embedding.
Science, 2900 (5500):0 2323–2326, 2000.

[Rozen et al., 2021]
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman.
Moser flow: Divergence-based generative modeling on manifolds.
In Advances in Neural Information Processing Systems, 2021.

[Ruan et al., 2021]
Yangjun Ruan, Karen Ullrich, Daniel S Severo, James Townsend, Ashish Khisti,
Arnaud Doucet, Alireza Makhzani, and Chris Maddison.
Improving lossless compression rates via Monte Carlo bits-back
In International Conference on Machine Learning, pp.  9136–9147, 2021.

[Rudin, 1987]
Walter Rudin.
Real and Complex Analysis.
McGraw-Hill, Inc., 3rd edition, 1987.

[Rumelhart et al., 1988]
David E Rumelhart, Geofrrey E Hinton, and Ronald J Williams.
Learning Internal Representations by Error Propagation, pp.  673–695.
MIT Press, 1988.

[Saharia et al., 2022]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L
Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim
Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.
Photorealistic text-to-image diffusion models with deep language
In Advances in Neural Information Processing Systems, 2022.

[Salimans et al., 2017]
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.
PixelCNN++: Improving the PixelCNN with discretized logistic
mixture likelihood and other modifications.
In International Conference on Learning Representations, 2017.

[Salman et al., 2018]
Hadi Salman, Payman Yadollahpour, Tom Fletcher, and Kayhan Batmanghelich.
Deep diffeomorphic normalizing flows.
arXiv:1810.03256, 2018.

[Salmona et al., 2022]
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnès Desolneux.
Can push-forward generative models fit multimodal distributions?
In Advances in Neural Information Processing Systems, 2022.

[Saremi & Hyvärinen, 2019]
Saeed Saremi and Aapo Hyvärinen.
Neural empirical Bayes.
Journal of Machine Learning Research, 200
(181):0 1–23, 2019.

[Sauer et al., 2023]
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.
StyleGAN-t: Unlocking the power of GANs for fast large-scale
text-to-image synthesis.
In International Conference on Machine Learning, pp.  30105–30118, 2023.

[Schilling & Kühn, 2021]
René L Schilling and Franziska Kühn.
Counterexamples in Measure and Integration.
Cambridge University Press, 2021.

[Schonsheck et al., 2019]
Stefan Schonsheck, Jie Chen, and Rongjie Lai.
Chart auto-encoders for manifold structured data.
arXiv:1912.10094, 2019.

[Schreuder et al., 2021]
Nicolas Schreuder, Victor-Emmanuel Brunel, and Arnak Dalalyan.
Statistical guarantees for generative models without domination.
In Algorithmic Learning Theory, pp.  1051–1071, 2021.

[Schuhmann et al., 2022]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson,
Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.
LAION-5b: An open large-scale dataset for training next generation
image-text models.
In Advances in Neural Information Processing Systems, 2022.

[Schölkopf et al., 1998]
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller.
Nonlinear component analysis as a kernel eigenvalue problem.
Neural Computation, 100 (5):0 1299–1319,

[Sidheekh et al., 2022]
Sahil Sidheekh, Chris B Dock, Tushar Jain, Radu Balan, and Maneesh K Singh.
VQ-Flows: Vector quantized local normalizing flows.
In Uncertainty in Artificial Intelligence, pp.  1835–1845.
PMLR, 2022.

[Simon-Gabriel & Schölkopf, 2018]
Carl-Johann Simon-Gabriel and Bernhard Schölkopf.
Kernel distribution embeddings: Universal kernels, characteristic
kernels and kernel metrics on distributions.
Journal of Machine Learning Research, 190
(44):0 1–29, 2018.

[Simon-Gabriel et al., 2020]
Carl-Johann Simon-Gabriel, Alessandro Barp, Bernhard Schölkopf, and Lester
Metrizing weak convergence with maximum mean discrepancies.
arXiv:2006.09268, 2020.

[Sohl-Dickstein et al., 2015]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In International Conference on Machine Learning, pp.  2256–2265, 2015.

[Sønderby et al., 2016]
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae
Sønderby, and Ole Winther.
Ladder variational autoencoders.
In Advances in Neural Information Processing Systems, 2016.

[Song & Ermon, 2019]
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
In Advances in Neural Information Processing Systems, 2019.

[Song et al., 2021]
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.
Maximum likelihood training of score-based diffusion models.
In Advances in Neural Information Processing Systems,

[Song et al., 2021]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential
In International Conference on Learning Representations,

[Sorrenson et al., 2023]
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, and Ullrich
Learning distributions on manifolds with free-form flows.
arXiv:2312.09852, 2023.

[Sorrenson et al., 2024]
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea
Zimmerman, and Ullrich Köthe.
Lifting architectural constraints of injective flows.
In International Conference on Learning Representations, 2024.

[Stanczuk et al., 2022]
Jan Stanczuk, Georgios Batzolis, Teo Deveney, and Carola-Bibiane Schönlieb.
Your diffusion model secretly knows the dimension of the data
arXiv:2212.12611, 2022.

[Stein et al., 2023]
George Stein, Jesse C Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross,
Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, J Eric T Taylor, and
Gabriel Loaiza-Ganem.
Exposing flaws of generative model evaluation metrics and their
unfair treatment of diffusion models.
In Advances in Neural Information Processing Systems, 2023.

[Tang & Yang, 2023]
Rong Tang and Yun Yang.
Minimax rate of distribution estimation on unknown submanifolds
under adversarial losses.
The Annals of Statistics, 510 (3):0
1282–1308, 2023.

[Tanielian et al., 2020]
Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary.
Learning disconnected manifolds: A no GAN’s land.
In International Conference on Machine Learning, pp.  9418–9427, 2020.

[Tempczyk et al., 2022]
Piotr Tempczyk, Rafał Michaluk, Lukasz Garncarek, Przemysław Spurek,
Jacek Tabor, and Adam Golinski.
LIDL: Local intrinsic dimension estimation using approximate
In International Conference on Machine Learning, pp.  21205–21231, 2022.

[Tenenbaum et al., 2000]
Joshua B Tenenbaum, Vin de Silva, and John C Langford.
A global geometric framework for nonlinear dimensionality reduction.
Science, 2900 (5500):0 2319–2323, 2000.

[Theis et al., 2016]
Lucas Theis, Aäron van den Oord, and Matthias Bethge.
A note on the evaluation of generative models.
In International Conference on Learning Representations, 2016.

[Tipping & Bishop, 1999]
Michael E Tipping and Christopher M Bishop.
Probabilistic principal component analysis.
Journal of the Royal Statistical Society Series B: Statistical
Methodology, 610 (3):0 611–622, 1999.

[Tishby et al., 2000]
Naftali Tishby, Fernando C Pereira, and William Bialek.
The information bottleneck method.
arXiv:physics/0004057, 2000.

[Tolstikhin et al., 2018]
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf.
Wasserstein auto-encoders.
In International Conference on Learning Representations, 2018.

[Tomczak & Welling, 2018]
Jakub Tomczak and Max Welling.
VAE with a VampPrior.
In International Conference on Artificial Intelligence and
Statistics, pp.  1214–1223, 2018.

[Tong et al., 2024]
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang,
Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio.
Improving and generalizing flow-based generative models with
minibatch optimal transport.
Transactions on Machine Learning Research, 2024.

[Townsend et al., 2019]
James Townsend, Thomas Bird, and David Barber.
Practical lossless compression with latent variables using bits back
In International Conference on Learning Representations, 2019.

[Tran et al., 2023]
Ba-Hien Tran, Giulio Franzese, Pietro Michiardi, and Maurizio Filippone.
One-line-of-code data mollification improves optimization of
likelihood-based generative models.
In Advances in Neural Information Processing Systems, 2023.

[Uria et al., 2013]
Benigno Uria, Iain Murray, and Hugo Larochelle.
RNADE: The real-valued neural autoregressive density-estimator.
In Advances in Neural Information Processing Systems, 2013.

[Vahdat & Kautz, 2020]
Arash Vahdat and Jan Kautz.
NVAE: A deep hierarchical variational autoencoder.
In Advances in Neural Information Processing Systems, 2020.

[Vahdat et al., 2021]
Arash Vahdat, Karsten Kreis, and Jan Kautz.
Score-based generative modeling in latent space.
In Advances in Neural Information Processing Systems, 2021.

[van den Berg et al., 2018]
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling.
Sylvester normalizing flows for variational inference.
In Uncertainty in Artificial Intelligence, pp.  393–402,

[van den Oord et al., 2016]
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex
Graves, and Koray Kavukcuoglu.
Conditional image generation with PixelCNN decoders.
In Advances in Neural Information Processing Systems, 2016.

[van den Oord et al., 2017]
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.
Neural discrete representation learning.
In Advances in Neural Information Processing Systems, 2017.

[van der Maaten & Hinton, 2008]
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-SNE.
Journal of Machine Learning Research, 90
(86):0 2579–2605, 2008.

[Vardanyan et al., 2023]
Elen Vardanyan, Arshak Minasyan, Sona Hunanyan, Tigran Galstyan, and Arnak
Guaranteed optimal generative modeling with maximum deviation from
the empirical distribution.
arXiv:2307.16422, 2023.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In Advances in Neural Information Processing Systems, 2017.

[Villani, 2009]
Cédric Villani.
Optimal Transport: Old and New.
Springer Science & Business Media, 2009.

[Vincent, 2011]
Pascal Vincent.
A connection between score matching and denoising autoencoders.
Neural computation, 230 (7):0 1661–1674,

[Von Rohrscheidt & Rieck, 2023]
Julius Von Rohrscheidt and Bastian Rieck.
Topological singularity detection at multiple scales.
In International Conference on Machine Learning, pp.  35175–35197, 2023.

[Wallace, 1992]
GK Wallace.
The JPEG still picture compression standard.
IEEE Transactions on Consumer Electronics, 380
(1):0 18–34, 1992.

[Wand & Jones, 1994]
Matt P Wand and M Chris Jones.
Kernel Smoothing.
CRC Press, 1994.

[Wang et al., 2021]
Yixin Wang, David Blei, and John P Cunningham.
Posterior collapse and latent variable non-identifiability.
In Advances in Neural Information Processing Systems, 2021.

[Welling & Teh, 2011]
Max Welling and Yee W Teh.
Bayesian learning via stochastic gradient langevin dynamics.
In International Conference on Machine Learning, pp.  681–688, 2011.

[Xiao et al., 2019]
Zhisheng Xiao, Qing Yan, and Yali Amit.
Generative latent flow.
arXiv:1905.10485, 2019.

[Xie et al., 2016]
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu.
A theory of generative ConvNet.
In International Conference on Machine Learning, pp.  2635–2644, 2016.

[Yang et al., 2023]
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao
Zhang, Bin Cui, and Ming-Hsuan Yang.
Diffusion models: A comprehensive survey of methods and applications.
ACM Computing Survey, 2023.

[Yoon et al., 2021]
Sangwoong Yoon, Yung-Kyun Noh, and Frank Park.
Autoencoding under normalization constraints.
In International Conference on Machine Learning, pp.  12087–12097, 2021.

[Yoon et al., 2023]
Sangwoong Yoon, Young-Uk Jin, Yung-Kyun Noh, and Frank Park.
Energy-based models for anomaly detection: A manifold diffusion
recovery approach.
Advances in Neural Information Processing Systems, 2023.

[Zhang et al., 2024]
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao
Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis.
Mixed-type tabular data synthesis with score-based diffusion in
latent space.
In International Conference on Learning Representations, 2024.

[Zhang et al., 2020]
Mingtian Zhang, Peter Hayes, Thomas Bird, Raza Habib, and David Barber.
Spread divergence.
In International Conference on Machine Learning, pp.  11106–11116, 2020a.

[Zhang et al., 2023]
Mingtian Zhang, Yitong Sun, Chen Zhang, and Steven Mcdonagh.
Spread flows for manifold modelling.
In International Conference on Artificial Intelligence and
Statistics, pp.  11435–11456, 2023.

[Zhang et al., 2020]
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull.
Perceptual generative autoencoders.
In International Conference on Machine Learning, pp.  11298–11306, 2020b.

[Zheng et al., 2022]
Yijia Zheng, Tong He, Yixuan Qiu, and David P Wipf.
Learning manifold dimensions with conditional variational
In Advances in Neural Information Processing Systems, 2022.