# Discrete Latent Graph Generative Modeling with Diffusion Bridges

Van Khoa Nguyen  Yoann Boget  Frantzeska Lavda  Alexandros Kalousis

###### Abstract

Learning graph generative models over latent spaces has received less
attention compared to models that operate on the original data space and has
so far demonstrated lacklustre performance. We present GLAD a latent space
graph generative model. Unlike most previous latent space graph generative
models, GLAD operates on a discrete latent space that preserves to a
significant extent the discrete nature of the graph structures making no
unnatural assumptions such as latent space continuity. We learn the prior of
our discrete latent space by adapting diffusion bridges to its structure. By
operating over an appropriately constructed latent space we avoid relying on
decompositions that are often used in models that operate in the original data
space. We present experiments on a series of graph benchmark datasets which
clearly show the superiority of the discrete latent space and obtain state of
the art graph generative performance, making GLAD the first latent space graph
generative model with competitive performance. Our source code is published
at: https://github.com/v18nguye/GLAD

Machine Learning, ICML

##  1 Introduction

Graph generation has posed a longstanding challenge. A versatile range of
solutions have been proposed over time, from simple models (Erdős et al.,
1960; Albert & Barabási, 2002) to more recent deep generative models (Jo et
al., 2022; Vignac et al., 2023; Kong et al., 2023). Within that latter
approach the landscape of existing methods is rich and complex. To navigate
this landscape we will start by a characterisation of the existing methods
along two dimensions: the representation space over which the generative
models are learnt, latent versus observed, and how the graph generation is
done, autoregressively versus one-shot.

When it comes to the graph representation space most state-of-the-art methods
operate over the original data space and learn generative models over node and
edge features (Shi et al., 2020; Luo et al., 2021; Jo et al., 2022; Kong et
al., 2023; Vignac et al., 2023), showcasing a flexible adaptation to learning
over discrete representations. Less attention has been given to methods that
operate over graph latent spaces and the appropriate definition and design of
such latent spaces. Some initial efforts (Simonovsky & Komodakis, 2018; Jin et
al., 2018; Liu et al., 2018; Samanta et al., 2020) propose learning the graph
distribution over the latent space of variational autoecoders (VAEs) (Kingma &
Welling, 2013). Yet, such approaches often suffer from high reconstruction
errors, have to solve challenging graph-matching problems, and/or rely on
heuristic corrections for the generated graphs.

Figure 1: Latent spaces for graphs: (Left) Continuous-graph latent space: We
encode a graph as a distribution (black circle) defined over the pooled
representation (black diamond) of its node embeddings (colored diamonds), and
regularised towards the normal prior $\mathbb{P}_{z}$ (grey disk). (Middle)
Continuous-node latent space: We encode a graph as a product of latent node
distributions (colored circles), each regularized towards the normal prior
$\mathbb{P}_{z}$. (Right) Quantised discrete-node latent space: GLAD encodes a
graph as a set of node embeddings (colored dots) allowed to take values only
over uniformly quantised latent space (black dots). Unlike the two continuous
latent spaces this one preserves the discrete nature of the embdedded objects.

Graph generation can be either auto-regressive or one-shot. Auto-regressive
models generate samples one graph element at the time operating over basic
structural components of the graphs, e.g. nodes, edges or substructures, (You
et al., 2018a; Shi et al., 2020; Luo et al., 2021), and rely on the
availability of an ordering of the graph elements. While these models have
demonstrated promising results and overcome the likelihood intractability, by
the very fact that they require an order they do not provide a principled way
to learn graph distributions in a permutation-invariant manner. Consequently,
they make away with the permutation invariance by resorting to heuristic
orderings (You et al., 2018a; Shi et al., 2020; Luo et al., 2021) or employ
learning approaches to partially address the limitation (Chen et al., 2021).

One-shot graph generative models and in particular diffusion-based models,
have recently emerged as a very compelling approach for modeling graph
distributions (Niu et al., 2020; Jo et al., 2022; Vignac et al., 2023). They
allow for the natural incorporation of the permutation-equivariance property
as an inductive bias within their denoising component. Obviously they are not
without limitations. The initial graph diffusion methods rely on continuous
score functions to model the graph distributions. They learn these score
functions by denoising graph structures that have been noised with real-valued
noise (Niu et al., 2020; Jo et al., 2022). Continuous score functions are a
poor match for structures that are inherently discrete. Note that similar
caveats also appear in VAE-based approaches where the graph latent
representations are constrained to follow the standard normal distribution
(more on that on Section 2.1) There are some diffusion-based works that treat
graphs heads-on as discrete objects (Vignac et al., 2023). In addition,
constructing appropriate score functions over the original graph
representation is challenging. Common approaches factorise these score
functions to node- and adjacency-matrix-specific components which require
their own distinct denoising networks. These partial score functions provide a
fragmented view of the graph and do not reflect well the true score function
making them a poor choice to model graph distributions. Note that such
decompositions are used both in the continuous diffusion-based approaches for
graphs as well as in the discrete ones. Diffusion-based models typically have
a denoising component which operates over the edges. The denoising process
takes place over densely(fully)-connected graphs (Niu et al., 2020; Jo et al.,
2022; Vignac et al., 2023); this poses significant scalability challenges for
larger graphs such as social networks (Wang et al., 2018).

In this work we start from certain desiderata that a graph generative model
should satisfy, which will guide the design of our model. Graph generative
models should should learn graph distributions that are node permutation
invariant. Graphs should be treated in a holisitic manner without relying on
artificial decompositions to their components. Finally graphs should be
treated as discrete objects, as they are. To address these desiderata we
propose Graph Discrete Latent Diffusion (GLAD) a generative model that
operates on a carefully designed permutation-equivariant discrete latent space
that models graphs in an holistic manner, while explicitly accounting for
their discrete nature. We will do generation using a one-shot approach,
relying on Graph Neural Network for encoding and decoding, from and to the
latent space, satisfying also the permutation equivariance desiderata. Our
latent space ensures efficient graph encoding and supports highly accurate
reconstructions. GLAD extends diffusion bridges (Heng et al., 2021;
Peluchetti, 2022), and in particular (Liu et al., 2023) proposed for
constrained data domains, to the domain of graphs by operating over our graph
latent space. We carefully evaluate our method on a set of graph benchmarks
and compare its performance against that of state-of-the-art generative
methods for graphs. GLAD consistently outperforms the baselines validating the
merits of our discrete latent diffussion bridge and its capacity to model
complex dependencies in graph structures. We summarise our key contributions
as follows:

  * •

We present a simple, yet powerful, quantised discrete latent space for graphs
building on finite element quantisation (Mentzer et al., 2023) which delivers
significantly better reconstruction compared to previous methods that operate
on continuous latent spaces.

  * •

We introduce a new generative model for graphs that tailors diffusion bridges
to our discrete latent space; our model does not resort to component specific
decompositions and is permutation-invariant.

  * •

GLAD is the first approach for one-shot diffusion-based graph generative
modelling over a discrete latent space. We demonstrate a robust and
consistently better generative performance against state-of-the-art methods
across a number of graph generation benchmarks.

##  2 Background

We use a tuple $\mathcal{G}=(X,E)$ to denote the spatial structure of a graph,
where $X$ and $E$ are node, edge (adjacency) matrices, respectively. When it
comes to representing graphs for generative modelling there are two main
approaches. In the first, we directly operate in the original graph
representation as it is given by the $\mathcal{G}$ tuple, i.e. over the
original nodes and edges representations and features, and learn the
generative model over it.

Alternatively we can define appropriate latent spaces for graphs and learn the
generative models over them. Concretely, under the latent-space approach
graphs are first encoded by an encoder, $\phi$, to some latent space which is
governed by a distribution $q_{\phi}$. Subsequently a decoder, $\theta$, maps
the latent space encodings back to the original space governed by the
generative distribution $p_{\theta}$. The very nature of graphs makes it that
there are different ways of encoding them to a latent space. A popular choice
is to model a graph as the set of its node embeddings to some latent space. We
will denote that set as $Z=\\{z_{1},\cdots,z_{n}\\}$, where $n$ is the total
number of nodes and $z_{i}\in\mathbb{R}^{d_{Z}}$ is the latent embedding of
the $i$ node. These embeddings are usually produced by graph neural networks
and encode information not only about the node itself but also about its local
neighborhood, essentially providing a description of a subgraph that is
centered at the node $i$. Representing a graph as a set of node embeddings
will be also our starting point towards the design of a latent space that
satisfies the desiderata that we discussed in the introductory section.

###  2.1 Latent Graph Representation

When learning functions over graphs we need to consider whether these need to
be node permutation invariant or equivariant. Informally, a graph function is
permutation invariant if any permutation of its input leaves the output
unchanged and equivariant when for any given permutation of the input the
output is permuted in the same manner. We want to learn graph distributions in
a manner that is node permutation invariant, in the sense that random node
permutations of the training set graphs leave the learnt distribution
unaffected. To that end the components of our model will have to be node
permutation equivariant.

Handling graph invariances when learning latent graph representations for
generative modeling is challenging. Simonovsky & Komodakis (2018) used
Variational Autoencoders (VAEs) to model graph distributions. They establish
the latent space using GNNS to learn the set of node embeddings $Z$, whose
elements they then pool to obtain a global latent graph representation
$z_{\mathcal{G}}=\oplus(Z)$, $\oplus$ denotes the pooling operation. They
constrain the graph latent distribution
$q_{\phi}(z_{\mathcal{G}}|\mathcal{G})$, to follow a standard normal
distribution. We will call such a latent space a continuous-graph latent
space. Pooling operations are not permutation equivariant and one needs to
solve a graph-matching problem on the decoding side to establish the
reconstruction loss, which is computationally expensive and inaccurate. One
way to avoid the need for solving a graph-matching problem is by resorting to
the use of a canonical ordered graph representation such as a sequence- or a
tree-based representation (Jin et al., 2018). Such approaches suffer from the
fact that canonical representations are not unique for a given graph and as a
result learning is not efficient. We summarise this discussion in the
following observation.

######  Observation 2.1.

Global latent graph representations do not capture well the graph
equivariance. The induced decoding distributions are not invariant to node
permutations.

Another way to alleviate the issues that are created by a global graph latent
representation is simply to avoid pooling of the node embeddings altogether
and operate over a latent representation that has a set structure. Such an
approach is followed in different VAE-based methods, (Liu et al., 2018;
Samanta et al., 2020). Since now the latent representation is the set of the
node embeddings in the decoding step there is no ambiguity, we know the node
correspondences, and thus no need to solve the graph matching problem. Unlike
the case of canonical representations such an approach is by construction node
permutation equivariant. These approaches will typically describe the latent
space distribution using a decomposition over the latent node embeddings:
$q_{\phi}(Z|\mathcal{G})=\prod_{i}q_{\phi}(z_{i}|\mathcal{G})$ and regularise
each of the latent node representations towards a standard normal
distribution. We will call such a latent space a continuous-node latent space.
Thus the latent representations of different nodes of a graph are assumed to
have the same prior distribution. We illustrate a cartoon version of this
assumption in the middle graph of Figure 1. This simplistic assumption has a
detrimental effect on accurate graph reconstruction as we will empirically
demonstrate.

######  Observation 2.2.

Constraining the node latent representation to the same prior makes them less
distinguishable form each other hindering the downstream decoding process.

Figure 2:  Overview of GLAD An equivariant GNN (E-GNN) encodes a graph as a
set of continuous latent node embdeddings, colored diamonds. The quantisation
operator maps these embeddings to points on a uniform discrete grid, colored
circles. An E-GNN decoder reconstructs the graph from the set of the discrete
node embeddings. We use diffusion bridges to learn the prior over the discrete
space by diffusing from a fixed point prior, green star, to the conditioned
discrete latent graph structure.

The above works model graph and node embeddings as normal distributions,
making the assumption that the embedded structures lie in a continuous space
with support everywhere; this is a simplistic assumption. A node encoding
contains information about the node itself and the graph neighborood
(subgraph) within which the node lives. These are essentially discrete
structures and while there are exponentially many different possibilities with
the neighborhood size these are countable and inherently discrete. The
appropriate way to model distributions over such objects is by treating the
latent spaces as discrete and use discrete distributions. There is a rather
limited number of works that treat the problem of graph latent generative
modelling as a discrete problem and operate over discrete spaces and discrete
distributions. Notable exceptions to that are: discrete flows for graphs
(GraphDF) (Luo et al., 2021) and Vector-Quantised Graph Auto-Encoder (VQ-GAE)
(Boget et al., 2023). GraphDF is an autoregressive flow-based model, it
traverses the graph elements in breadth-first manner, and operates over
discrete latent node and edge structures. VQ-GAE, similar to GLAD, uses a
permutation-equivariant graph auto-encoder to represent graphs as sets of node
embeddings to a discrete, quantized, space. However unlike GLAD, which models
the latent space distribution in a permutation-invariant manner, VQ-GAE learns
it auto-regressively, by establishing an ordering over the set of node
embeddings.

######  Observation 2.3.

Graph latent spaces should be discrete and modelled using discrete
distributions in order to faithfully represent the discrete nature of graphs.

###  2.2 Diffusion Bridges on Structured Domains

We briefly present diffusion bridges (Heng et al., 2021; Peluchetti, 2022; Liu
et al., 2023), which apply to structured domains. A diffusion bridge is a
conditional stochastic process that ensures that the diffusion process
converges to a given conditioning sample at the terminal state. The
reachability of the conditioning sample is guaranteed by the Doob’s h
transform (Doob & Doob, 1984). Let $\mathbf{Z}=\\{Z_{t}:t\in[0,T]\\}$ be a
non-conditional diffusion process where the dynamics of the state
$Z_{t}\in\mathbb{R}^{d}$ are characterized by the stochastic differential
equation:

$\mathbb{Q}:dZ_{t}=\eta(Z_{t},t)dt+\sigma(Z_{t},t)dW_{t}$ (1)

$W_{t}$ is a Wiener process,
$\sigma:[0,T]\times\mathbb{R}^{d}\mapsto\mathbb{R}$ is a diffusion
coefficient, and $\eta:[0,T]\times\mathbb{R}^{d}\mapsto\mathbb{R}^{d}$ is a
drift function. Following the development in (Liu et al., 2023) we start by
introducing the $x$-bridge: $\mathbb{Q}^{x}(.):=\mathbb{Q}(.|Z_{T}=x)$, where
the conditioning factor that should be reached at the terminal state $Z_{T}$
is the $x$ sample. The $x$-bridge dynamics are given by a stochastic
differential equation that is derived from the h-transform (Oksendal, 2013):

$\mathbb{Q}^{x}:dZ_{t}=\eta^{x}(Z_{t},t)dt+\sigma(Z_{t},t)dW_{t}$ (2)

The $x$-bridge’s drift is defined as
$\eta^{x}(Z_{t},t)=\eta(Z_{t},t)+\sigma^{2}(Z_{t},t)\nabla_{Z_{t}}\log
q_{T|t}(x|Z_{t})$; $q_{T|t}(x|Z_{t})$ is the probability density of obtaining
$x$ at time $T$ when we have $Z_{t}$ at time $t$ and $\nabla_{Z_{t}}\log
q_{T|t}(x|Z_{t})$ acts as a steering force towards $x$; The steering force is
central to guiding $Z_{t}$ towards the specified target $Z_{T}=x$.

Given the $x$-bridge definition we can now construct a bridge process on a
data distribution $\Pi$ that is constrained over a discrete domain $\Omega$.
We call this bridge process a $\Pi$-bridge and denote it by
$\mathbb{Q}^{\Pi}$; $\Pi$-bridges will allow us to model distributions over
constrained domains. $\mathbb{Q}^{\Pi}$ is given by a mixture of $x$-bridges;
their end-points are conditioned on data samples $\\{x^{(i)}\\}_{1}^{N}$ i.i.d
drawn from the data distribution $\Pi$. The dynamics of the $\Pi$-bridge
$\mathbb{Q}^{\Pi}(.):=\mathbb{Q}(.|Z_{T}\in\Omega)$ are governed by the
following stochastic differential equation:

$\mathbb{Q}^{\Pi}:dZ_{t}=\eta^{\Pi}(Z_{t},t)dt+\sigma(Z_{t},t)dW_{t}$ (3)

The $\Pi$-bridge’s drift is
$\eta^{\Pi}(Z_{t},t)=\eta(Z_{t},t)+\sigma^{2}(Z_{t},t)\mathbb{E}_{x\sim
q_{T|t,\Omega}}[\nabla_{Z_{t}}\log q_{T|t}(x|Z_{t})]$, where $x$ is sampled
from a truncated transition probability density given by
$q_{T|t,\Omega}(x|Z_{t}):=q(Z_{T}=x|Z_{t},Z_{T}\in\Omega)$. As we can see, the
structured domain $\Omega$ is taken into consideration in the expectation of
the steering forces. The marginal distribution, $\mathbb{Q}_{T}^{\Pi}$,
induced by the $\Pi$-bridge at $t=T$ will match the data distribution $\Pi$ by
construction, i.e. $\mathbb{Q}_{T}^{\Pi}=\Pi$.

We want to learn a bridge model, $\mathbb{P}^{\psi}$, of the
$\mathbb{Q}^{\Pi}$ bridge. The learned bridge will generalise from the
$\mathbb{Q}^{\Pi}$ bridge and allow us to generate new samples from the $\Pi$
distribution. We construct $\mathbb{P}^{\psi}$ by learning a parametric drift
function with a neural network. The dynamics of $\mathbb{P}^{\psi}$ are given
by:

$\mathbb{P}^{\psi}:dZ_{t}=\eta^{\psi}(Z_{t},t)dt+\sigma(Z_{t},t)dW_{t},\;\;\;Z_{0}\sim\mathbb{P}_{0}$
(4)

where
$\eta^{\psi}(Z_{t},t)=\sigma(Z_{t},t)f_{\psi}(Z_{t},t)+\eta^{\Pi}(Z_{t},t)$,
$f_{\psi}$ is a parametric neural network, and $\mathbb{P}_{0}$ is a bridge
prior. The ${\Pi}$-bridge $\mathbb{Q}^{\Pi}$ imputes noise to data samples at
different time steps which we use to learn the model bridge
$\mathbb{P}^{\psi}$. We train $\mathbb{P}^{\psi}$ by minimizing the Kullback-
Leibler divergence between the two probability path measures
$\min_{\psi}\\{\mathcal{L}(\psi):=\mathcal{KL}(\mathbb{Q}^{\Pi}||\mathbb{P}^{\psi})\\}$;
the Girsanov theorem, (Lejay, 2018), allows us to compute the $\mathcal{KL}$
divergence with the following closed form solution:

$\mathcal{L}(\psi)=\mathbb{E}_{Z_{0}\sim\mathbb{P}_{0},x\sim\Pi,t\sim[0,T],Z_{t}\sim\mathbb{Q}^{x}}\left[\|f_{\psi}(Z_{t},t)-\bar{\eta}(Z_{t},t)\|^{2}\right]$

(5)

with
$\bar{\eta}(Z_{t},t)=\sigma^{-1}(Z_{t},t)(\eta^{x}(Z_{t},t)-\eta^{\Pi}(Z_{t},t))$,
where $\eta^{x}$, $\eta^{\Pi}$ are introduced in Equation 2 and Equation 3,
respectively. Equation 5 will become our training objective. We will use
diffusion-bridges to learn graph generative models over a discrete graph
latent space.

##  3 GLAD: Graph Discrete Latent Diffusion Model

We will now present GLAD our discrete latent diffusion-bridge model for
graphs. GLAD has two components. We first define a discrete latent space for
graphs, Section 3.1, and then show how to learn a prior over this latent space
using diffusion-bridges, Section 3.2, which will allow us to do generation. We
provide a high level view of GLAD in Figure 2.

###  3.1 Graph Equivariant Discrete Latent Space

We drive the design of the latent space by: i) ensuring that we learn
distributions that are invariant to permutations, ii) encoding nodes and their
local structures by ensuring that structural differences are reflected in the
encoding, and iii) preserving the discrete nature of the objects we encode,
nodes and their neighborhoods, and that of the graph itself.

Our graph latent representation will be a set of discrete latent embeddings of
its nodes; unlike VQ-GAE (Boget et al., 2023), which uses learned vector-
quantized representations, we derive these discrete embeddings using finite
scalar quantization, (Mentzer et al., 2023); this produces a high dimensional
quantized latent space in which nodes are embedded. Since we will not
constrain the latent space, other than the quantization operation, it
preserves to a significant extent the structural differences of the nodes and
their neighborhood and at the same time it can encode very diverse local
structures. We empirically verify these statements by demonstrating robust
reconstruction performance across various graph benchmark datasets.

Concretely, we use a GNN encoder, $\phi$, to first map a graph to a set of
continuous node embeddings $Z=\phi(\mathcal{G})=\\{z_{1},\cdots,z_{n}\\}$. The
GNN encoder essentially pushes in the node encoding information about itself
and its local neighborhood. We subsequently apply a quantization operator,
$\mathcal{F}$, on the continuous node embeddings and map them to fixed points
within the same space. The quantization operates independently on each
dimension of the latent space, and the quantization of the $j^{th}$ dimension
of $z_{i}$ embedding is given by:

$z^{q}_{ij}=\mathcal{F}(z_{ij},L_{j})=\mathcal{R}(\frac{L_{j}}{2}\tanh{z_{ij}}),\;1\leqslant
j\leqslant d_{Z}$ (6)

where $\mathcal{R}$ is the rounding operator 111To ensure end to end
differentiability we implement $\mathcal{R}$ with the straight through
estimator. and $L_{j}$ is the number of quantization levels used for the
$j^{th}$ dimension of the node embeddings. The quantisation operator maps any
point in the original continuous latent space to a point from the set
$\bar{Z}=\\{\bar{z}_{1},\cdots,\bar{z}_{K}\\},\;K=\prod_{j}L_{j}$. Each point
from $\bar{Z}$ corresponds to a point on the uniform grid defined by
$[L_{1}]\times\cdots\times[L_{d_{Z}}]$, where $[L_{j}]$ denotes the set of
$L_{j}$ possible quantization values of the $j$ dimension. Points in $\bar{Z}$
encode particular graph substructures. We visualise our discrete latent space,
defined by $\bar{Z}$, as the matrix of black points in Figure 2. The discrete
latent representation of graph $\mathcal{G}$ is the set
$Z_{\mathcal{G}}=\\{z_{1}^{q},\cdots,z_{n}^{q}\\},\;z_{i}^{q}\in\bar{Z}$.

The quantization operator is permutation equivariant and so is the mapping
from the initial graph $\mathcal{G}$ to its discrete representation in the
latent space as long as the $\phi$ graph encoder is permutation equivariant.
Thus for any $P$ permutation matrix the following equivariance relation holds:

$P^{T}Z_{\mathcal{G}}=\mathcal{F}(P^{T}Z)=\mathcal{F}(\phi(P^{T}EP,P^{T}X))$
(7)

Using an equivariant decoder $\theta(Z_{\mathcal{G}})$, which will operate on
the assumption of a fully-connected graph over the node embeddings
$Z_{\mathcal{G}}$, results in reconstructed graphs that are node permutation
equivariant with the original input graphs.

###  3.2 Learning Diffusion Bridges over the Discrete Latent Space of Graphs

We will now show how to modify the diffusion bridge processes to operate over
the discrete latent space we just described. Our latent space describes graphs
as a set of discrete node embeddings222The node embeddings explicitly encode
node neighborhood infromation as a direct result of the GNN encoding. Thus the
adjacency matrix information is encoded in the node embeddings. and thus we
need no decomposition of the score function estimations to node- and
adjacency-specific score components. We use Brownian motion,
$dZ_{t}=\sigma_{t}dW_{t}$, as the non-conditional diffusion process
$\mathbb{Q}$. The dynamic of the $x$-bridge’s drift can be accordingly derived
as:

$\eta^{x=Z_{\mathcal{G}}}(Z_{t},t)=\sigma_{t}^{2}\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}},\;\beta_{t}=\int_{0}^{t}\sigma_{s}^{2}ds$
(8)

where the diffusion coefficient $\sigma_{t}$ now depends only on the time
variable. Since the domain of the discrete latent space is a set we can
decompose it in a node-wise manner as $\Omega=I_{1}\times\cdots\times I_{n}$,
where $I_{i}=\bar{Z}$. As a result, the expectation simplifies to one-
dimensional integrals, and the $\Pi$-bridge’s drift is:

$\displaystyle\eta^{\Pi}(Z_{t},t)$
$\displaystyle=[\eta^{I_{i}}(Z_{t}^{i},t)]_{i=1}^{n}$
$\displaystyle\eta^{I_{i}}(Z_{t}^{i},t)$
$\displaystyle=\sigma_{t}^{2}\nabla_{Z_{t}^{i}}\log\sum_{k=1}^{K}\exp\left(-\frac{\|Z_{t}^{i}-\bar{z}_{k}\|^{2}}{2(\beta_{T}-\beta_{t})}\right)$
(9)

where $\bar{z}_{k}\in\bar{Z}$, K is the number of fixed points of $\bar{Z}$,
$Z_{t}^{i}$ is the feature of $i^{th}$ latent node dimension of $Z_{t}$. By
estimating the drift terms over the discrete set-structured latent space we
eliminate the computational cost of denoising processes that operate over
fully-connected graphs, which is a computational bottleneck present in most
diffusion-based models for graphs.

Moreover, as a result of the steering forces the diffusion processes are
guided towards the conditioning samples for the $x$-bridges and eventually to
the data distributions for the $\Pi$-bridges for arbitrary initialisation.
Consequently, these bridges allow for a flexible design of their prior
distributions $\mathbb{P}_{0}$. We experimentally verify the model performance
with a fixed-point and a standard normal distribution bridge prior. We provide
the proofs and discuss the training algorithms in Appendix A.1.

##  4 Experiments

Table 1: Generation results on the generic graph datasets. We show the mean values of 15 runs for each experiment, where lower results indicate better performance. The baselines are sourced from (Jo et al., 2022; Kong et al., 2023). We use the abbreviations 0.S, A.R, L., D. to stand for one-shot, auto-regressive, latent space, and data space, respectively. Hyphen (-) denotes unreproducible results. The $1^{\text{st}}$ and $2^{\text{nd}}$ best results are bolded and underlined, respectively. We further provide standard deviations and inference time in Appendix. |  |  | Community-small | Ego-small | Enzymes
---|---|---|---|---|---
|  |  | Deg. | Clus. | Orb. | Avg. | Deg. | Clus. | Orb. | Avg. | Deg. | Clus. | Orb. | Avg.
A.R. | D. | GraphAF | 0.180 | 0.200 | 0.020 | 0.133 | 0.030 | 0.110 | 0.001 | 0.047 | 1.669 | 1.283 | 0.266 | 1.073
GraphRNN | 0.080 | 0.120 | 0.040 | 0.080 | 0.090 | 0.220 | 0.003 | 0.104 | 0.017 | 0.062 | 0.046 | 0.042
GraphArm | 0.034 | 0.082 | 0.004 | 0.040 | 0.019 | 0.017 | 0.010 | 0.015 | 0.029 | 0.054 | 0.015 | 0.033
L. | GraphDF | 0.060 | 0.120 | 0.030 | 0.070 | 0.040 | 0.130 | 0.010 | 0.060 | 1.503 | 1.061 | 0.202 | 0.922
| VQ-GAE | 0.032 | 0.062 | 0.005 | 0.033 | 0.021 | 0.041 | 0.007 | 0.023 | - | - | - | -
O.S. | D. | GDSS | 0.045 | 0.086 | 0.007 | 0.046 | 0.021 | 0.024 | 0.007 | 0.017 | 0.026 | 0.061 | 0.009 | 0.032
DiGress | 0.047 | 0.041 | 0.026 | 0.038 | 0.015 | 0.029 | 0.005 | 0.016 | 0.004 | 0.083 | 0.002 | 0.030
L. | GraphVAE | 0.350 | 0.980 | 0.054 | 0.623 | 0.130 | 0.170 | 0.050 | 0.117 | 1.369 | 0.629 | 0.191 | 0.730
GNF | 0.200 | 0.200 | 0.110 | 0.170 | 0.030 | 0.100 | 0.001 | 0.044 | - | - | - | -
GLAD | 0.027 | 0.044 | 0.006 | 0.026 | 0.012 | 0.013 | 0.004 | 0.010 | 0.012 | 0.014 | 0.001 | 0.009

We evaluate GLAD’s reconstruction and generation performance on generic and
molecule graph problems. We compare against a number of baselines, which we
characterise along two dimensions: how they do the generation,
autoregressively (A.R) or one-shot (O.S), and whether they operate on the data
space (D.) or on a latent space (L.).

###  4.1 Generic Graph Generation

We evaluate GLAD’s ability to capture underlying graph structures by
experiments on three generic graph datasets: (a) Ego-small (Sen et al., 2008),
(b) Community-small, and (c) Enzymes (Schomburg et al., 2004). We adopt the
same train/test split as (Jo et al., 2022) for a fair comparison with their
baselines. We evaluate generative performance by measuring the maximum mean
discrepancy (MMD) between the generated set and test set distributions of the
following graph properties: degree (Deg.), clustering coefficient (Clus.), and
orbit occurrences of 4 nodes (Orb.) (You et al., 2018b).

Baselines and Implementation Details: Here under the one-shot approach we
compare to GDSS, (Jo et al., 2022) and DiGress, (Vignac et al., 2023), which
are diffusion based models that operate directly in the original graph data
space, and GraphVAE (Simonovsky & Komodakis, 2018) and GNF (Liu et al., 2019)
which operate over graph latent structures. Under the autoregressive baselines
on the original data space we have GraphRNN, (You et al., 2018a), GraphAF (Shi
et al., 2020), a flow-based model, and GraphArm, (Kong et al., 2023), an auto-
regressive diffusion model, and on a latent space we have GraphDF, (Luo et
al., 2021), an autoregressive flow-based model, and VQ-GAE, (Boget et al.,
2023), an autoregressive model on the discrete latent space. We provide the
additional details on the experiments and the model architectures in Appendix
A.2.

Results: We give the evaluation results for the generic graph generation in
Table 1. GLAD exhibits quite competitive performance when compared to state-
of-the-art baselines on different evaluation metrics; it consistently
maintains the lowest average MMD distance across all three datasets. Moreover,
GLAD significantly outperforms all models that make explicit use of latent
space structures, namely GraphVAE, GNF, GraphDF, and VQ-GAE. These
accomplishments highlight two pivotal strengths of our approach: first, our
discrete latent space is able to encode rich local graph structures, serving
as a cornerstone for any latent-driven graph generative models. Second, our
diffusion bridge model works seamlessly within the proposed discrete latent
space, which is underscored by its capability to capture the set-based latent
representation of graphs. While our model does not operate directly on the
data space of graphs like other diffusion frameworks such as GDSS and DiGress,
it has demonstrated the superior performance to understand the underlying
topologies of graphs in a holistic manner.

###  4.2 Molecule Graph Generation

Table 2: Generation results on the molecule datasets. We show the mean values
of 3 runs for each experiment. The baselines are sourced from (Jo et al.,
2022; Kong et al., 2023). We use the abbreviations 0.S, A.R, L., D. to stand
for one-shot, auto-regressive, latent space, and data space, respectively. The
$1^{\text{st}}$ and $2^{\text{nd}}$ best results are bolded and underlined,
respectively. We further provide standard deviations and inference time in
Appendix.
|  |  | QM9 | ZINC250k
---|---|---|---|---
|  |  |  Val. $\uparrow$ |  Uni. $\uparrow$ |  Nov. $\uparrow$ |  NSPDK $\downarrow$ |  FCD $\downarrow$ |  Val. $\uparrow$ |  Uni. $\uparrow$ |  Nov. $\uparrow$ |  NSPDK $\downarrow$ |  FCD $\downarrow$
A.R. | D. | GraphAF | 74.43 | 88.64 | 86.59 | 0.020 | 5.27 | 68.47 | 98.64 | 100 | 0.044 | 16.02
GraphArm | 90.25 | 95.62 | 70.39 | 0.002 | 1.22 | 88.23 | 99.46 | 100 | 0.055 | 16.26
L. | GraphDF | 93.88 | 98.58 | 98.54 | 0.064 | 10.93 | 90.61 | 99.63 | 100 | 0.177 | 33.55
| VQ-GAE | 88.00 | 97.23 | 78.12 | 0.0023 | 0.8 | 65.7 | 99.96 | 99.97 | 0.008 | 5.4
O.S. | D. | EDP-GNN | 47.52 | 99.25 | 86.58 | 0.005 | 2.68 | 82.97 | 99.79 | 100 | 0.049 | 16.74
GDSS | 95.72 | 98.46 | 86.27 | 0.003 | 2.90 | 97.01 | 99.64 | 100 | 0.019 | 14.66
DiGress | 99.0 | 96.66 | 33.40 | 0.0005 | 0.360 | 91.02 | 81.23 | 100 | 0.082 | 23.06
MoFlow | 91.36 | 98.65 | 94.72 | 0.017 | 4.47 | 63.11 | 99.99 | 100 | 0.046 | 20.93
L. | GLAD | 97.24 | 97.68 | 39.54 | 0.0003 | 0.222 | 81.22 | 100 | 100 | 0.002 | 2.54

Prior | QM9
---|---
Val. $\uparrow$ |  Uni. $\uparrow$ |  Nov. $\uparrow$ |  NSPDK $\downarrow$ |  FCD $\downarrow$
$\mathcal{N}(\mathbf{0},\mathbf{1})$ | 95.38 | 97.38 | 41.54 | 0.0004 | 0.330
$\mathbf{0}$ | 97.24 | 97.68 | 39.54 | 0.0003 | 0.222
$\mathbf{0}^{\dagger}$ | 83.12 | 97.54 | 62.44 | 0.0009 | 1.207
Table 3: The effects of prior $\mathbb{P}_{0}$ and quantization $\mathcal{F}$.
Generation results on QM9 where the prior distribution is initialized as a
standard normal distribution and a Dirac $\Delta$ distribution on the fixed
point $\mathbf{0}$, the latter with and without quantization, denoted
respectively by $\mathbf{0}$ and $\mathbf{0}^{\dagger}$.

Here we want to evaluate the capacity of GLAD to capture the complex
dependencies between atoms and bonds as they appear in different molecules. We
conduct experiments on two standard datasets: QM9 (Ramakrishnan et al., 2014)
and ZINC250k (Irwin et al., 2012). Following (Jo et al., 2022), we remove
hydrogen atoms and kekulize molecules by RDKit Landrum et al. (2016). We
quantitatively evaluate all methods in terms of validity without any post-hoc
corrections (Val.), uniqueness (Uni.), and novelty (Nov.) of 10000 molecules
generated by each of them. We also compute two more salient metrics which
quantify how well the distribution of the generated graphs aligns with the
real data distribution, namely Fréchet ChemNet Distance (FCD) and Neighborhood
Subgraph Pairwise Distance Kernel (NSPDK) MMD. FCD evaluates the distance in
the chemical space between generated and training graphs by using the
activation of the ChemNet’s penultimate layer, while NSPDK measures the MMD
distance, showing the similarity of underlying structures between generated
and test molecules, which typically considers both the atom and bond types in
computation. More details on the experimental setup is provided in Appendix
A.2.

Baseline Details: The one-shot baselines that operate on the data space are:
MoFlow, (Zang & Wang, 2020), a conditional flow-based model; GDSS, DiGress,
and EDP-GNN, (Niu et al., 2020), a continuous score-based model. The auto-
regressive baselines are GraphAF, GraphArm, GraphDF, and VQ-GAE.

Results: We give the evaluation results for the molecule generation in Table
2. GLAD is the first one-shot model that learns to generate molecular
structures from a discrete latent space. Even though our model does not
predict directly atom- and bond-type structures it can still generate
molecules with very good validity scores with no need for any post-hoc
corrections. More importantly, it achieves state of the art performance on the
two most salient metrics NSPDK and FCD that capture how well the generated
distribution matches the training distribution, consistently outperforming by
significant margins all baselines. We provide examples of generated molecules
in Appendix A.3. GLAD can capture well both structural and chemical properties
of molecules. This is achieved through our quantized latent space, which
offers a suitable discrete structure to encode those properties. Furthermore,
our bridge model accurately learns underlying latent molecular structures, and
generates new molecules that are close to training distributions.

Figure 3: Test reconstruction performance of different latent spaces on QM9
dataset. Node (X) and edge (E) reconstruction accuracies. i) continuous-graph
latent representation (C-G). ii) continuous-node latent representation (C-N)
iii) unquantised discrete-node latent space $\text{GLAD}^{\dagger}$, iv)
quantised discrete-node latent space GLAD.

###  4.3 Ablation Studies

We will ablate two dimensions within the overall GLAD method: different design
choices on the latent space, and the sensitivity of the method with different
priors in the diffusion bridge processes.

Latent spaces for graphs.  We claim that the discrete structure of the graph
latent space is central to the success of GLAD. In order to verify that we
will compare the latent space of GLAD to the following three latent spaces and
generative models: i) a continuous-graph latent space where a graph
$\mathcal{G}$ is represented as $z_{\mathcal{G}}\sim
q_{\phi}(z_{\mathcal{G}}|{\mathcal{G}})$, with $z_{\mathcal{G}}=\oplus(Z)$ and
$q_{\phi}(z_{\mathcal{G}}|{\mathcal{G}})$ KL diverged to the standard normal
ii) a continuous-node latent space where the graph representation is
$z_{\mathcal{G}}\sim
q_{\phi}(Z|\mathcal{G})=\prod_{i}q_{\phi}(z_{i}|\mathcal{G})$, with
$q_{\phi}(z_{i}|\mathcal{G})$ KL diverged to the standard normal, both i) and
ii) encode graphs as continuous distributions, with support everywhere in the
embdedding space, and are trained with the ELBO objective. Finally in iii) we
have the same embedding latent space as in ii) but a graph is not anymore a
distribution, but simply $Z$ the set of its node embeddings. The generative
model in iii) is the continuous diffusion bridge; overall iii) is the
unquantised latent space of GLAD, and we denote it as $\text{GLAD}^{\dagger}$.
Even though in iii) we embed on a continuous space, as a result of the graph
nature we are not covering this space, we are rather embedding in a discrete
and countable subspace of it. We evaluate the quality of the underlying graph
latent spaces using their bond-type (X) and atom-type (E) reconstruction
accuracies on QM9, Figure 3.

The two continuous latent spaces in i) and ii) have considerably lower
reconstruction accuracies compared to the two GLAD variants which preserve to
a different extent the original discrete topology. Both GLAD variants converge
very fast to almost 100% reconstruction accuracies. GLAD has a small advantage
when it comes to the bond-type reconstructions for which its training is more
stable than that of its unquantised sibling $\text{GLAD}^{\dagger}$. The
latent spaces of i) and ii) suffer from the so-called node-collapse problem.
They are not able to well describe different atom local structures. As a
result, most reconstructed nodes are assigned to the dominant Carbon atom
which corresponds to roughly the $70\%$ of all atoms in QM9.

While the two GLAD variants have a rather similar reconstruction performance
this is not the case when we compare their generation performance; we train
two variants with a fixed point prior (more details in the following ablation)
and give the generation results in Table 3. There we see that diffusion
bridges can not capture well the graph distribution when they operate in the
unquantised space, with a performance drop for most measures (more importantly
for the two measures capturing distibution differences, NSPDK and FCD),
showing the benefits of the quantised discrete latent space. The latent spaces
of the two GLAD variants are both discrete and are embedded within the same
continuous space. Quantization acts as a spatial regulariser over the non-
quantised discrete space by constructing a uniform grid to which the latent
node embeddings are assigned instead of letting them free.

Latent Diffusion Bridge Priors: Here we evaluate generative performance with
different priors; a fixed point prior $\mathbb{P}_{0}=\mathbf{0}$, which is
typically the central point of our discrete latent space, and a standard
normal distribution prior,
$\mathbb{P}_{0}=\mathcal{N}(\mathbf{0},\mathbf{I})$. We conduct the
experiments on QM9 and give the results in Table 3. As a result of the
steering forces of the bridge processes there are only negligible differences
between the two priors making prior selection less of a nuance. With the fixed
prior, we obtain generated molecules with higher validity scores, closer to
the training distribution in both chemical space (FCD score), structural space
(NSPDK score). Using the normal prior we have better performance in terms of
uniqueness and novel novelty.

##  5 Conclusions

We presented GLAD a graph generative model that operates over a discrete
latent space. We carefully designed that space to satisfy a number of
desiderata that stem from the nature of the graph structures, namely it should
allow for graph invariant distribution modelling, it should have rich
representational power, and it should respect the inherently discrete nature
of graphs. To learn generative models over that space we leveraged diffusion
bridges which we adapted to our discrete latent space by appropriately
adjusting their drift terms. We empirically validate GLAD on a number of
benchmarks and show that it achieves state of the art generative performance,
surpassing other baselines independently of whether they operate on the
original or on latent spaces. We systematically ablated different design
choices for graph latent space representation and show that our discrete
latent space brings clear performance advantages over continuous alternatives
both in terms of reconstruction as well as generation. This paves the way for
a more systematic exploration of graph generative modelling in latent spaces,
something that until now has received rather limited attention.

## Impact

The paper presents a generative model for graphs. Such methods find
applicability in settings such as chemical compound design, protein and drug
design, with a significant potential for positive societal contributions if
successful. At the same time they could also be used by malign actors for the
design of harmful chemical, biological and other types of agents. However, the
actual transfer of the results of such models to the lab and from there to
production, requires significant resources, which creates a fist safeguard
against their malicious use. In addition there are international treaties that
have established strict safeguards when it comes in the manufacturing of
harmful agents.

## References

  * Albert & Barabási (2002) Albert, R. and Barabási, A.-L.  Statistical mechanics of complex networks.  _Reviews of modern physics_ , 74(1):47, 2002.
  * Bengio et al. (2013) Bengio, Y., Léonard, N., and Courville, A.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Boget et al. (2023) Boget, Y., Gregorova, M., and Kalousis, A.  Vector-quantized graph auto-encoder.  _arXiv preprint arXiv:2306.07735_ , 2023.
  * Chen et al. (2021) Chen, X., Han, X., Hu, J., Ruiz, F., and Liu, L.  Order matters: Probabilistic modeling of node sequence for graph generation.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 1630–1639. PMLR, 18–24 Jul 2021.
  * Doob & Doob (1984) Doob, J. L. and Doob, J.  _Classical potential theory and its probabilistic counterpart_ , volume 262.  Springer, 1984.
  * Erdős et al. (1960) Erdős, P., Rényi, A., et al.  On the evolution of random graphs.  _Publ. math. inst. hung. acad. sci_ , 5(1):17–60, 1960.
  * Heng et al. (2021) Heng, J., De Bortoli, V., Doucet, A., and Thornton, J.  Simulating diffusion bridges with score matching.  _arXiv preprint arXiv:2111.07243_ , 2021.
  * Irwin et al. (2012) Irwin, J. J., Sterling, T., Mysinger, M. M., Bolstad, E. S., and Coleman, R. G.  Zinc: a free tool to discover chemistry for biology.  _Journal of chemical information and modeling_ , 52(7):1757–1768, 2012.
  * Jin et al. (2018) Jin, W., Barzilay, R., and Jaakkola, T.  Junction tree variational autoencoder for molecular graph generation.  In _International conference on machine learning_ , pp. 2323–2332. PMLR, 2018.
  * Jo et al. (2022) Jo, J., Lee, S., and Hwang, S. J.  Score-based generative modeling of graphs via the system of stochastic differential equations.  In _International Conference on Machine Learning_ , pp. 10362–10383. PMLR, 2022.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kong et al. (2023) Kong, L., Cui, J., Sun, H., Zhuang, Y., Prakash, B. A., and Zhang, C.  Autoregressive diffusion model for graph generation.  In _International Conference on Machine Learning_ , pp. 17391–17408. PMLR, 2023.
  * Landrum et al. (2016) Landrum, G. et al.  Rdkit: Open-source cheminformatics software, 2016.  _URL http://www. rdkit. org/, https://github. com/rdkit/rdkit_ , 149(150):650, 2016.
  * Lejay (2018) Lejay, A.  The girsanov theorem without (so much) stochastic analysis.  _Séminaire de Probabilités XLIX_ , pp. 329–361, 2018.
  * Liu et al. (2019) Liu, J., Kumar, A., Ba, J., Kiros, J., and Swersky, K.  Graph normalizing flows.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Liu et al. (2018) Liu, Q., Allamanis, M., Brockschmidt, M., and Gaunt, A.  Constrained graph variational autoencoders for molecule design.  _Advances in neural information processing systems_ , 31, 2018.
  * Liu et al. (2023) Liu, X., Wu, L., Ye, M., and qiang liu.  Learning diffusion bridges on constrained domains.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Luo et al. (2021) Luo, Y., Yan, K., and Ji, S.  Graphdf: A discrete flow model for molecular graph generation.  In _International Conference on Machine Learning_ , pp. 7192–7203. PMLR, 2021.
  * Mentzer et al. (2023) Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M.  Finite scalar quantization: Vq-vae made simple.  _arXiv preprint arXiv:2309.15505_ , 2023.
  * Niu et al. (2020) Niu, C., Song, Y., Song, J., Zhao, S., Grover, A., and Ermon, S.  Permutation invariant graph generation via score-based generative modeling.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 4474–4484. PMLR, 2020.
  * Oksendal (2013) Oksendal, B.  _Stochastic differential equations: an introduction with applications_.  Springer Science & Business Media, 2013.
  * Peluchetti (2022) Peluchetti, S.  Non-denoising forward-time diffusions, 2022.  URL https://openreview.net/forum?id=oVfIKuhqfC.
  * Ramakrishnan et al. (2014) Ramakrishnan, R., Dral, P. O., Rupp, M., and Von Lilienfeld, O. A.  Quantum chemistry structures and properties of 134 kilo molecules.  _Scientific data_ , 1(1):1–7, 2014.
  * Samanta et al. (2020) Samanta, B., De, A., Jana, G., Gómez, V., Chattaraj, P., Ganguly, N., and Gomez-Rodriguez, M.  Nevae: A deep generative model for molecular graphs.  _Journal of machine learning research_ , 21(114):1–33, 2020.
  * Schomburg et al. (2004) Schomburg, I., Chang, A., Ebeling, C., Gremse, M., Heldt, C., Huhn, G., and Schomburg, D.  Brenda, the enzyme database: updates and major new developments.  _Nucleic acids research_ , 32(suppl_1):D431–D433, 2004.
  * Sen et al. (2008) Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T.  Collective classification in network data.  _AI magazine_ , 29(3):93–93, 2008.
  * Shi et al. (2020) Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M., and Tang, J.  Graphaf: a flow-based autoregressive model for molecular graph generation.  _arXiv preprint arXiv:2001.09382_ , 2020.
  * Simonovsky & Komodakis (2018) Simonovsky, M. and Komodakis, N.  Graphvae: Towards generation of small graphs using variational autoencoders.  In _Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I 27_ , pp. 412–422. Springer, 2018.
  * Vignac et al. (2023) Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P.  Digress: Discrete denoising diffusion for graph generation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Wang et al. (2018) Wang, H., Wang, J., Wang, J., Zhao, M., Zhang, W., Zhang, F., Xie, X., and Guo, M.  Graphgan: Graph representation learning with generative adversarial nets.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2018.
  * You et al. (2018a) You, J., Ying, R., Ren, X., Hamilton, W., and Leskovec, J.  Graphrnn: Generating realistic graphs with deep auto-regressive models.  In _International conference on machine learning_ , pp. 5708–5717. PMLR, 2018a.
  * You et al. (2018b) You, J., Ying, R., Ren, X., Hamilton, W., and Leskovec, J.  Graphrnn: Generating realistic graphs with deep auto-regressive models.  In _International conference on machine learning_ , pp. 5708–5717. PMLR, 2018b.
  * Zang & Wang (2020) Zang, C. and Wang, F.  Moflow: an invertible flow model for generating molecular graphs.  In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_, pp. 617–626, 2020.

##  Appendix A Appendix.

In the appendix we present additional details of the paper. We give in Section
A.1 the training procedure, in Section A.2 the experimental details, and
provide generated molecule examples in Section A.3.

###  A.1 Training

We train GLAD with a two-stage procedure. We first train a graph autoencoder
and then learn the graph latent diffusion bridge over the quantised discrete
space defined on the graph autoencoder output.

In the first stage, we train the graph-autoencoder, which learns a structural
mapping from the graph data space to the designed latent space and
reconstructs original graphs from their latent representations. As a technical
detail, we note that the rounding operator $\mathcal{R}$ of the quantization
is non-differentiable. To ensure end-to-end training of our discrete graph
latent space mapping, we use the straight-through estimator (STE) (Bengio et
al., 2013), implemented with a stop-gradient operator $\mathcal{S}$ as
$Z_{\mathcal{G}}\mapsto Z+\mathcal{S}(Z_{\mathcal{G}}-Z)$. In addition, we use
mean-squared loss as it offers more stable training experiments. Our graph
autoencoder training is described in Algorithm 1.

Algorithm 1 Graph AutoEncoder Training Algorithm

1: Input: Encoder $\phi$, decoder $\theta$, the number of quantization levels
per dimension $L_{j}$.

2: for # training iterations do

3: Sample a minibatch $\boldsymbol{\mathcal{G}}=({\mathbf{X},\mathbf{E}})$

4: Encode to latent nodes $\mathbf{Z}\leftarrow\phi(\boldsymbol{\mathcal{G}})$

5: Quantise
$\mathbf{Z}_{\boldsymbol{\mathcal{G}}}\leftarrow\mathcal{F}(\mathbf{Z},L_{j})$
{Equation 6}

6: Straight-through estimator
$\mathbf{Z}_{\boldsymbol{\mathcal{G}}}\mapsto\mathbf{Z}+\text{stop-
gradient}(\mathbf{Z}_{\boldsymbol{\mathcal{G}}}-\mathbf{Z})$

7: Decode
${\bar{\boldsymbol{\mathcal{G}}}}=({\mathbf{\bar{X}},\mathbf{\bar{E}}})\leftarrow\theta(\boldsymbol{Z_{\mathcal{G}}})$

8: Compute MSE
$\mathcal{L}\leftarrow\|\mathbf{X}-\bar{\mathbf{X}}\|^{2}+\|\mathbf{E}-\bar{\mathbf{E}}\|^{2}$

9: Update the models $\phi,\theta$ by gradient descent w.r.t. their parameters
$\nabla_{\phi,\theta}\mathcal{L}$

10: end for

In the second stage, we freeze the trained graph-autoencoder parameters, and
train our generative component— namely the drift model. We adjust the
derivations of Liu et al. (2023) to the graph setting. We use Brownian motion
as the non-conditional diffusion process in Equation 1:

$\mathbb{Q}:\;\;dZ_{t}=\sigma_{t}dW_{t},\;\;\;\;\beta_{t}=\int_{0}^{t}\sigma_{s}^{2}ds$
(10)

The transition probability between states follows a Gaussian distribution,
whose density is defined as
$q_{T|t}(Z_{T}|Z_{t})=\mathcal{N}\left(Z_{T};Z_{t},\beta_{T}-\beta_{t}\right)$.
Replacing this density to the transition probability density in Equation 2, we
get the $x$-bridge drift:

$\displaystyle\eta^{x=Z_{\mathcal{G}}}(Z_{t},t)$
$\displaystyle=\eta(Z_{t},t)+\sigma^{2}_{t}\nabla_{Z_{t}}\log
q_{T|t}(Z_{\mathcal{G}}|Z_{t})$
$\displaystyle=0+\sigma^{2}_{t}\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}}$
$\displaystyle=\sigma^{2}_{t}\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}}$
(11)

Thus, the governing law of the $x$-bridge is:

$\displaystyle\mathbb{Q}^{x=Z_{\mathcal{G}}}:\;\;dZ_{t}=\sigma_{t}^{2}\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}}dt+\sigma_{t}dW_{t}$
(12)

We sample from the $t\in[0,T]$ time step of the $x$-bridge as follows:

$\displaystyle
Z_{t}=\frac{\beta_{t}}{\beta_{T}}\left(Z_{\mathcal{G}}+(\beta_{T}-\beta_{t})Z_{0}\right)+\xi\sqrt{\beta_{t}\left(1-\frac{\beta_{t}}{\beta_{T}}\right)},\;\;\;\xi\sim\mathcal{N}(0,I)$
(13)

By plugging the transition probability $q_{T|t}(Z_{T}|Z_{t})$ into Equation 3,
we obtain the $\Pi$-bridge drift:

$\displaystyle\eta^{\Pi}(Z_{t},t)$
$\displaystyle=\eta(Z_{t},t)+\sigma^{2}_{t}\mathbb{E}_{Z_{\mathcal{G}}\sim{q}_{T|t,\Omega}}[\nabla_{Z_{t}}\log
q_{T|t}(Z_{\mathcal{G}}|Z_{t})]$
$\displaystyle=0+\sigma^{2}_{t}\mathbb{E}_{Z_{\mathcal{G}}\sim{q}_{T|t,\Omega}}\left[\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}}\right]$
$\displaystyle=\sigma^{2}_{t}\mathbb{E}_{Z_{\mathcal{G}}\sim{q}_{T|t,\Omega}}\left[\frac{Z_{\mathcal{G}}-Z_{t}}{\beta_{T}-\beta_{t}}\right]$
(14)

where ${q}_{T|t,\Omega}$ is the $\Omega$-truncated transition probability
density function. When $\mathbb{Q}$ is Brownian motion its $\Omega$-truncated
transition probability density is a truncated Gaussian density given by
$q_{T|t,\Omega}(Z_{\mathcal{G}}|Z_{t})\varpropto\mathbb{I}(Z_{\mathcal{G}}\in\Omega)q_{T|t}(Z_{\mathcal{G}}|Z_{t})$.

We observe that the truncated transition density considers the existing
elements $Z_{\mathcal{G}}$ of the discrete domain $\Omega$. However, our graph
latent representation is a set-based structure, the discrete domain $\Omega$
can thus be factorized over latent node dimension, denoted as
$\Omega=I_{1}\times\cdots\times I_{n}$, with $I_{i}=\bar{Z}$ in Section 3.1.
Intuitively, this factorization represents for independent conditional
stochastic processes over each latent node dimension of the given graph latent
structure. Hence, the $\Pi$\- bridge’s drift can be decomposed as:

$\displaystyle\eta^{\Pi}(Z_{t},t)=\left[\eta^{I_{i}}(Z_{t}^{i},t)\right]_{i=1}^{n}$
$\displaystyle\text{where
}\eta^{I_{i}}(Z_{t}^{i},t)=\sigma^{2}_{t}\mathbb{E}_{\bar{z}_{k}\sim
q_{T|t,\bar{Z}}}\left[\frac{\bar{z}_{k}-Z_{t}^{i}}{\beta_{T}-\beta_{t}}\right]$
(15)

The expectation is now taken over the truncated transition probability along
each latent node dimension $I_{i}$, whose density can be represented as
following:

$\displaystyle
q_{T|t,\bar{Z}}(\bar{z}_{k})\varpropto\mathbb{I}(\bar{z}_{k}\in\bar{Z})\frac{q_{T|t}(\bar{z}_{k}|Z_{t}^{i})}{\sum_{j=1}^{K}q_{T|t}(\bar{z}_{j}|Z_{t}^{i})}$
$\displaystyle\text{where the density }q_{T|t}(\bar{z}_{j}|Z_{t}^{i})\text{ is
}\mathcal{N}(\bar{z}_{j};Z_{t}^{i},\beta_{T}-\beta_{t})$ (16)

The $\Pi$-bridge’s drift along each latent node dimension can be written in a
closed-form solution:

$\displaystyle\eta^{I_{i}}(Z_{t}^{i},t)$
$\displaystyle=\sigma^{2}_{t}\mathbb{E}_{\bar{z}_{k}\sim
q_{T|t,\bar{Z}}}\left[\frac{\bar{z}_{k}-Z_{t}^{i}}{\beta_{T}-\beta_{t}}\right]$
$\displaystyle=\sigma^{2}_{t}\sum_{k=1}^{K}\frac{q_{T|t}(\bar{z}_{k}|Z_{t}^{i})}{\sum_{\sum_{j=1}^{K}q_{T|t}(\bar{z}_{j}|Z_{t}^{i})}}\frac{\bar{z}_{k}-Z_{t}^{i}}{\beta_{T}-\beta_{t}}$
$\displaystyle=\sigma^{2}_{t}\sum_{k=1}^{K}\frac{\exp\left(-\frac{\|Z_{t}^{i}-\bar{z}_{k}\|^{2}}{2\left(\beta_{T}-\beta_{t}\right)}\right)}{\sum_{j=1}^{K}\exp\left(-\frac{\|Z_{t}^{i}-\bar{z}_{j}\|^{2}}{2\left(\beta_{T}-\beta_{t}\right)}\right)}\frac{\bar{z}_{k}-Z_{t}^{i}}{\beta_{T}-\beta_{t}}$
$\displaystyle=\sigma_{t}^{2}\nabla_{Z_{t}^{I_{i}}}\log\sum_{k=1}^{K}\exp\left(-\frac{\|Z_{t}^{I_{i}}-\bar{z}_{k}\|^{2}}{2(\beta_{T}-\beta_{t})}\right)$
(17)

For the completeness of the presentation, we show the original proof of the
training objective in a more explicit way to our case. Given that, we need to
minimize the Kullback-Leibler divergence between the two probability path
measures
$\min_{\psi}\\{\mathcal{L}(\psi):=\mathcal{KL}(\mathbb{Q}^{\Pi}||\mathbb{P}^{\psi})\\}$
in Section 2.2, $\Pi$ is our discrete latent distribution of graphs. Following
(Liu et al., 2023), the KL objective is proved to be equivalent to:

$\displaystyle\mathcal{KL}(\mathbb{Q}^{\Pi}||\mathbb{P}^{\psi})=\mathbb{E}_{Z_{\mathcal{G}}\sim\Pi}\left[\mathcal{KL}\left(\mathbb{Q}^{x=Z_{\mathcal{G}}}\|\mathbb{P}^{\psi}\right)\right]+const.$
(18)

By Girsanov theorem (Lejay, 2018), this objective can be further decomposed as
following:

$\displaystyle\mathcal{L}(\psi)=\mathbb{E}_{Z_{\mathcal{G}}\sim\Pi,Z_{t}\sim\mathbb{Q}^{x=Z_{\mathcal{G}}}}\left[-\log
p_{0}(Z_{0})+\frac{1}{2}\int_{0}^{T}\left\|\sigma_{t}^{-1}(\eta^{\psi}(Z_{t},t)-\eta^{x=Z_{\mathcal{G}}})\right\|^{2}\right]+const.$
(19)

In our experiments, the prior distribution $\log p_{0}(Z_{0})$ is a Dirac
$\Delta$ distribution on the fixed point $\mathbf{0}$, the central mass of the
quantized space, thus $\log p_{0}(Z_{0})$ is constant. Applying the definition
of $\eta^{\psi}(Z_{t},t)$ in Section 2.2 to Equation 19, our final training
objective can be obtained with a closed-form solution:

$\displaystyle\mathcal{L}(\psi)=\mathbb{E}_{t\sim[0,T],Z_{\mathcal{G}}\sim\Pi,Z_{t}\sim\mathbb{Q}^{x=Z_{\mathcal{G}}}}\left[\left\|f_{\psi}(Z_{t},t)-\sigma^{-1}_{t}(\eta^{x=Z_{\mathcal{G}}}(Z_{t},t)-\eta^{\Pi}(Z_{t},t))\right\|^{2}\right]+const$
(20)

At the sampling of new data, we apply the Euler–Maruyama discretization scheme
on the model bridge $\mathbb{P}^{\psi}$ that gives:

$\displaystyle
Z_{t+1}=Z_{t}+\left(\sigma_{t}f_{\psi}(Z_{t},t)+\eta^{\Pi}(Z_{t},t)\right)\delta_{t}+\sigma_{t}\sqrt{\delta_{t}}\xi,\;\;\xi\sim\mathcal{N}(0,I),\;\;Z_{0}\sim\mathbb{P}_{0}$
(21)

where $\delta_{t}$ is the discretization step. Our graph generative component
training is shown in Algorithm 2.

Algorithm 2 Graph Latent Diffusion Bridge Training Algorithm.

1: Input: The trained graph autoencoder $\phi$, $\theta$, the number of
quantization levels per dimension $L_{j}$, the drift model $f_{\psi}$, the
number of diffusion steps $T$, the diffusion coefficient scheduler
$\sigma_{s}$, the prior distribution $\mathbb{P}_{0}$.

2: for # training iterations do

3: Sample a minibatch $\boldsymbol{\mathcal{G}}=({\mathbf{X},\mathbf{E}})$, a
time steps vector $\boldsymbol{t}\in[0,T]$, a noise vector
$\boldsymbol{\xi}\sim\mathcal{N}(0,I)$, and $\mathbf{Z_{0}}\sim\mathbb{P}_{0}$

4: Encode to latent nodes $\mathbf{Z}\leftarrow\phi(\boldsymbol{\mathcal{G}})$

5: Quantise
$\mathbf{Z}_{\boldsymbol{\mathcal{G}}}\leftarrow\mathcal{F}(\mathbf{Z},L_{j})$

6: Sample from the $x$-bridge
$\mathbf{Z_{t}}\leftarrow\frac{\boldsymbol{\beta_{t}}}{\boldsymbol{\beta_{T}}}\left(\mathbf{Z}_{\boldsymbol{\mathcal{G}}}+(\boldsymbol{\beta_{T}}-\boldsymbol{\beta_{t}})\mathbf{Z_{0}}\right)+\boldsymbol{\xi}\sqrt{\boldsymbol{\beta_{t}}\left(1-\frac{\boldsymbol{\beta_{t}}}{\boldsymbol{\beta_{T}}}\right)}$

7: Initialize the node-wise objective list
$\boldsymbol{\bar{\eta}}\leftarrow\left[\right]$

8: for each latent node dimension $I_{i}$ in $\Omega$ do

9: Compute the $x$-bridge’s drift
$\boldsymbol{\eta}^{x}\leftarrow\boldsymbol{\sigma_{t}}^{2}\frac{\mathbf{Z}_{\boldsymbol{\mathcal{G}}}^{I_{i}}-\mathbf{Z_{t}}^{I_{i}}}{\boldsymbol{\beta_{T}}-\boldsymbol{\beta_{t}}}$
{Equation 8}

10: Compute the $\Pi$-bridge’s drift
$\boldsymbol{\eta}^{\Pi}\leftarrow\boldsymbol{\sigma_{t}^{2}}\nabla_{\mathbf{Z_{t}}^{I_{i}}}\log\sum_{k=1}^{K}\exp\left(-\frac{\|\mathbf{Z_{t}}^{I_{i}}-\bar{z}_{k}\|^{2}}{2(\boldsymbol{\beta_{T}}-\boldsymbol{\beta_{t}})}\right),\;\bar{z}_{k}\in\bar{Z}$
{Equation 9}

11: Compute the current node objective
$\boldsymbol{\bar{\eta}}^{I_{i}}\leftarrow\boldsymbol{\sigma^{-1}}_{t}(\boldsymbol{\eta}^{x}-\boldsymbol{\eta}^{\Pi})$

12: Append to the objective list
$\boldsymbol{\bar{\eta}}\leftarrow\left[\boldsymbol{\bar{\eta}},\boldsymbol{\bar{\eta}}^{I_{i}}\right]$

13: end for

14: Compute MSE
$\mathcal{L}\leftarrow\|f_{\psi}\left(\mathbf{Z_{t}},t\right)-\boldsymbol{\bar{\eta}}\|^{2}$.
{Equation 5}

15: Update the drift model $f_{\psi}$ by gradient descent w.r.t its parameters
$\nabla_{\psi}\mathcal{L}$

16: end for

###  A.2 Experiments

####  A.2.1 Model Architectures

In our research problem, we process data that lie on both graph- and set-based
structures. We opt to employ a general-purpose architecture such as
transformers. Future works could consider a more specific choice of
architectures especially designed for graphs or sets. We utilize the graph
transformer (Vignac et al., 2023) and further customize the architecture to
adapt to the encoder $\phi$, the decoder $\theta$, and the drift model
$f_{\psi}$. Typically, the graph transformer takes a node feature matrix
$X\in\mathbb{R}^{n\times d_{X}}$, an edge feature matrix
$E\in\mathbb{R}^{n\times n\times d_{E}}$, and a global feature vector
$Y\in\mathbb{R}^{d_{Y}}$ as inputs. For the encoder, we incorporate a second-
order adjacency matrix (Jo et al., 2022) to model long-range dependencies. Due
to the inherent limitations of most GNNs, we compute structural features like
cycle counts and spectral features of the graph-Laplacian decomposition as
hand-crafted input features. The decoder and drift model take a set-based data
structure as input. Moreover, sets are more or less considered as fully-
connected graphs. Hence, we use augmented edge matrices that enable to adapt
the main architecture design. These augmented matrices are simply defined as a
pairwise latent node distance matrix:

$\displaystyle E^{+}=Z\times Z^{T}$ (22)

Here, $Z\in\mathbb{R}^{n\times d_{Z}}$ represents a latent node representation
matrix. Specifically, $Z$ is equal to $Z_{\mathcal{G}}$ for the decoder, and
$Z$ is equal to $Z_{t}$ for the drift model. In addition, we use pooled node
features $Y=\oplus(Z)$ as global features for the decoder and drift model.

####  A.2.2 Quantization

We empirically find that a quantized latent space of dimension $d_{Z}=6$ works
well to encode graphs within low reconstruction errors. In all experiments, we
fix the quantization vector $L=[5,5,5,5,5,5]$ where at each latent node
dimension $L_{j},\;1\leqslant j\leqslant 6$, we quantize to one of five
possible values $[-2,-1,0,1,2]$. In total, we have $K=5^{6}$ quantization
points that are uniformly distributed in the discrete latent space. These
points serve as the referent points to which latent nodes are assigned. For
illustration, given a raw latent node vector $Z^{i}$, its quantized version
$Z^{i}_{q}$ is obtained as following:

$\displaystyle Z^{i}=\left[\begin{array}[]{c}-1.1\\\ -1.7\\\ -0.01\\\ 0.1\\\
3.2\\\
0.6\end{array}\right]\longrightarrow\frac{L_{j}}{2}\tanh{Z^{i}_{j}}\longrightarrow\left[\begin{array}[]{c}-2.00\\\
-2.34\\\ -0.03\\\ 0.25\\\ 2.49\\\
1.34\end{array}\right]\longrightarrow\text{Rounding}\longrightarrow
Z^{i}_{q}=\left[\begin{array}[]{c}-2\\\ -2\\\ 0\\\ 0\\\ 2\\\
1\end{array}\right]$

The same principle is applied to all latent nodes. Moreover, we visualize the
distribution of number latent nodes per quantization point in Figure 6. We
observe that there is a good matching between the train and test
distributions.

####  A.2.3 Experimental Details

We provide the hyperparameter details in the main experiments in Table 8.

Diffusion noise scheduler:  We set to the diffusion coefficient to an
exponential process
$\sigma_{t}^{2}=\sigma_{\min}*\sigma_{\max}\exp(-\sigma_{\max}*t)$ where we
fix $\sigma_{\min}=1$ and conduct an ablation study on
$\sigma_{\max}=\\{1,3,7\\}$.

Table 4: Generic graph statistics on three benchmark datasets: community
small, ego small, and enzymes.
| Number of Graphs | Number of Nodes | Real / Synthetic
---|---|---|---
Community-small | $100$ | $12\leqslant|X|\leqslant 20$ | Synthetic
Ego-small | $200$ | $4\leqslant|X|\leqslant 18$ | Real
Enzymes | $587$ | $10\leqslant|X|\leqslant 125$ | Real

Generic graph generation task: We provide the statistics of three generic
graph datasets in Table 4. We adopt the same train/test split as (Jo et al.,
2022) for a fair comparison with their baselines. In each experiment, we
report the means and standard deviations of 15 runs. We do not include the
enzyme baselines of VQ-GAE and GNF due to the absence open-source codes or
time training constraints. On the generic graphs, there are no explicit
information about nodes available for training. Therefore, we use node degrees
as augmented node features in GLAD. Each graph is represented by a node
feature matrix $X\in\\{0,1\\}^{n\times d_{X}}$ and an edge feature matrix
$E\in\\{0,1\\}^{n\times n}$, where $d_{X}$ corresponds to the number of
possible node degrees. During the generation step, we apply the rounding
operator after the sigmoid function to reconstruct the original edges. We
empirically find that the rounding operator works well for the edge
reconstruction. Similarly, on the diffusion bridges, we round the values of
the terminal state $Z_{T}$ to the nearest integers. We apply an exponential
learning rate scheduler (with a constant learning rate decay of $0.999$) and a
cosine anneal learning scheduler when training the graph autoencoder and the
drift model, respectively. We use the Pytorch automatic differentiation
package to compute the $\Pi$-bridge’s drift derivative in Equation 9. We
conduct an ablation study on architecture sizes with different number of
layers $\\{4,6,8\\}$, hidden node features $\\{64,128,256\\}$, and hidden edge
features $\\{32,64,128\\}$. We choose the best checkpoint that has the lowest
average MMD distance for comparing with the baselines. In each evaluation, we
generate an equal number of graphs to the test set. Follow Jo et al. (2022),
we use the Gaussian Earth Mover’s Distance (EMD) kernel to calculate MMD
thanks to its computational stability. Our detailed results are reported in
Table 6.

Table 5: Molecule graph statistics on QM9 and ZINC250k.
| Number of Graphs | Number of Nodes | Number of Node Types | Number of Edge Types
---|---|---|---|---
QM9 | $133885$ | $1\leqslant|X|\leqslant 9$ | $4$ | $3$
ZINC250k | $249455$ | $6\leqslant|X|\leqslant 38$ | $9$ | $3$

Molecule graph generation task: We provide the statistics of the molecule
graph datasets in Table 7. Following the procedure outlined in (Jo et al.,
2022), we kekulize the molecules by the RDKit library (Landrum et al., 2016)
and remove hydrogen atoms. The input graph is represented by a node feature
matrix $X\in\\{0,1\\}^{n\times d_{X}}$, and a scaled edge feature matrix
$\\{0,1/3,2/3,1\\}^{n\times n}$, where $d_{X}$ is the number of possible atom
types. During the generation step, we un-scale edge predictions back to
$[0,3]$, then round them to the nearest integers. In addition, we apply the
rounding operator to the latent diffusion bridges at the terminal state
$Z_{T}$. Similar to the generic graph experiments, we do an ablation study on
architecture sizes and apply the same learning rate schedulers. We choose the
hyperparameters that exhibit the best score of the product between uniqueness
and validation. We evaluate the generation results on 10000 generated
molecules, measuring Validity w/o correction as the percentage of valid
molecules without valence correction and edge resampling, Uniqueness as the
percentage of valid molecules that is unique, and Novelty as the percentage of
valid molecules not in the training set. Additionally, we compute two most
salient metrics: Fréchet ChemNet Distance (FCD), measuring the distance in the
chemical space between generated and training graphs, Neighborhood Subgraph
Pairwise Distance Kernel (NSPDK) MMD, which measures the similarity of
underlying structures between generated and test molecules. We present the
detailed results in Table 7.

Novelty metric on QM9: Like DiGress (Vignac et al., 2023), our framework
produces a low score on the novelty metric of QM9. However, the dataset
comprises all enumerated small molecules that follow certain constraints.
Hence, the generation of novel graphs beyond this dataset might not accurately
reflect the model’s ability to capture the underlying data distribution.

Bridge prior ablation: We use the same set of hyperparameters as in the main
experiment on QM9. In this study, we only retrain our bridge model with a new
prior $\mathbb{P}_{0}=\mathcal{N}(0,I)$. We provide the results in Table 7,
where we can see that the change of prior has only marginal effect in the
generation performance.

Quantisation ablation: In this experiment, we explore the effect of
quantisation by removing the quantization step. This is still a discrete but
non-quantised latent space which has the same dimensionality as the quantized
latent space in the main experiment ($d_{Z}=6$). We retrain both our graph
autoencoder and the bridge model. However, the domain of graph latent
structures is now non-quantised, denoted as
$\Omega=[L_{\min},L_{\max}]^{(n\times d_{Z})}$, where $L_{\min},L_{\max}$
correspond to the minimum and maximum values of learned latent nodes for the
entire training set. As per (Liu et al., 2023), the $\Pi$-bridge’s drift on a
continuous domain is computed as:

$\displaystyle\eta^{h}=\sigma_{t}^{2}\nabla_{Z_{t}^{h}}\log\left(F(\frac{Z_{t}^{h}-L_{\min}}{\sqrt{\beta_{T}-\beta_{t}}})-F(\frac{Z_{t}^{h}-L_{\max}}{\sqrt{\beta_{T}-\beta_{t}}})\right),\;1\leqslant
h\leqslant(N\times d_{Z})$ (23)

where $Z_{t}\in[L_{\min},L_{\max}]^{(n\times d_{Z})}$, $F$ is the standard
Gaussian Cumulative Distribution Function (CDF). We provide the detailed
results in Table 7. As we can see the removal of quantisation results to a
drop in the generalisation performance as this is captured by NSPDK and FCD.

Table 6: GLAD generation results on the generic graphs. We show the means and
standard deviations of 15 runs for each experiment.
| Deg. | Clus. | Orb.
---|---|---|---
Community-small |  0.027 $\pm$ 0.018 |  0.044 $\pm$ 0.015 |  0.006 $\pm$ 0.004
Ego-small |  0.012 $\pm$ 0.007 |  0.013 $\pm$ 0.006 |  0.004 $\pm$ 0.002
Enzymes |  0.012 $\pm$ 0.004 |  0.014 $\pm$ 0.003 |  0.001 $\pm$ 0.000
Table 7: GLAD generation results on the molecule graphs, bridge prior and
quantisation ablation. Means and standard deviations of 3 runs on each
dataset. ZINC250k and QM9 is standard GLAD, i.e. fixed prior and quantised
discrete latent space. $\text{QM9}^{\star}$ ablates the use of standard normal
distribution for the bridge prior. $\text{QM9}^{\dagger}$ ablates the effect
of non-quantisation in the latent space.
|  Validity $\uparrow$ |  Uniqueness $\uparrow$ |  Novelty $\uparrow$ |  NSPDK $\downarrow$ |  FCD $\downarrow$
---|---|---|---|---|---
ZINC250k |  81.22 $\pm$ 0.00 |  100 $\pm$ 0.00 |  100 $\pm$ 0.00 |  0.002 $\pm$ 0.000 |  2.54 $\pm$ 0.04
QM9 |  97.24 $\pm$ 0.00 |  97.68 $\pm$ 0.00 |  39.54 $\pm$ 0.00 |  0.0003 $\pm$ 0.0000 |  0.222 $\pm$ 0.008
$\text{QM9}^{\star}$ |  95.38 $\pm$ 0.00 |  97.38 $\pm$ 0.00 |  41.54$\pm$ 0.01 |  0.0004 $\pm$ 0.0000 |  0.330 $\pm$ 0.013
$\text{QM9}^{\dagger}$ |  83.12 $\pm$ 0.01 |  97.54 $\pm$ 0.01 |  62.44 $\pm$ 0.53 |  0.0009 $\pm$ 0.0000 |  1.207 $\pm$ 0.003

###  A.3 Visualizations

We provide the visualization of randomly generated molecules from QM9 in
Figure 4, and from ZINC250k in Figure 5.

Figure 4: Visualization of generated samples on QM9 from GLAD.

Figure 5: Visualization of generated samples on ZINC250k from GLAD.

Figure 6: Visualisation of the distribution of number latent nodes per
quantization point. We provide the distribution of latent nodes on the
quantised latent space. For each dataset, we have $K=5^{6}$ quantization
points that are uniformly distributed in the latent space.
| Hyperparameter | Community-small | Ego-small | Enzymes | QM9 | ZINC250k
---|---|---|---|---|---|---
$\phi$ | Number of heads | 8 | 8 | 8 | 8 | 8
Number of layers | 8 | 8 | 8 | 8 | 8
Hidden dimension X | 256 | 256 | 128 | 256 | 256
Hidden dimension E | 128 | 128 | 32 | 128 | 128
Hidden dimension Y | 64 | 64 | 64 | 64 | 64
$\theta$ | Number of heads | 8 | 8 | 8 | 8 | 8
Number of layers | 4 | 4 | 4 | 4 | 4
Hidden dimension X | 256 | 256 | 128 | 256 | 256
Hidden dimension E | 128 | 128 | 32 | 128 | 128
Hidden dimension Y | 64 | 64 | 64 | 64 | 64
$f_{\psi}$ | Number of heads | 8 | 8 | 8 | 8 | 8
Number of layers | 4 | 4 | 8 | 8 | 8
Hidden dimension X | 256 | 256 | 256 | 256 | 256
Hidden dimension E | 128 | 128 | 128 | 128 | 128
Hidden dimension Y | 64 | 64 | 64 | 64 | 64
SDE | $\sigma_{\min}$ | 1.0 | 1.0 | 1.0 | 1.0 | 1.0
$\sigma_{\max}$ | 3.0 | 3.0 | 3.0 | 3.0 | 3.0
Number of diffusion steps | 1000 | 1000 | 1000 | 1000 | 1000
AE | Optimizer | Adam | Adam | Adam | Adam | Adam
Learning rate | $5e^{-4}$ | $5e^{-4}$ | $5e^{-4}$ | $5e^{-4}$ | $5e^{-4}$
Batch size | 32 | 32 | 32 | 5240 | 512
Number of epochs | 3000 | 3000 | 3000 | 1000 | 1000
Model Bridge | Optimizer | Adam | Adam | Adam | Adam | Adam
Learning rate | $[1e^{-4},5e^{-4}]$ | $[1e^{-4},5e^{-4}]$ | $[1e^{-4},5e^{-4}]$ | $[1e^{-4},2e^{-3}]$ | $[3e^{-4},2e^{-3}]$
Batch size | 64 | 64 | 64 | 1280 | 512
Number of epochs | 5000 | 5000 | 5000 | 2000 | 2000
Table 8: GLAD hyperparameters . We show the main training hyperparameters in
the generic and molecule generation tasks. We provide the hyperparameters of
the encoder $\phi$, the decoder $\theta$, the drift model $f_{\psi}$, the
graph autoencoder training (AE), the model bridge training (Model Bridge), and
the latent diffusion process (SDE).