# Feature propagation as self-supervision signals on graphs

Oscar Pina<EMAIL_ADDRESS>Verónica Vilaplana<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning is gaining considerable attention as a solution to
avoid the requirement of extensive annotations in representation learning on
graphs. Current algorithms are based on contrastive learning, which is
computation an memory expensive, and the assumption of invariance under
certain graph augmentations. However, graph transformations such as edge
sampling may modify the semantics of the data so that the iinvariance
assumption may be incorrect. We introduce Regularized Graph Infomax (RGI), a
simple yet effective framework for node level self-supervised learning that
trains a graph neural network encoder by maximizing the mutual information
between output node embeddings and their propagation through the graph, which
encode the nodes’ local and global context, respectively. RGI do not use graph
data augmentations but instead generates self-supervision signals with feature
propagation, is non-contrastive and does not depend on a two branch
architecture. We run RGI on both transductive and inductive settings with
popular graph benchmarks and show that it can achieve state-of-the-art
performance regardless of its simplicity. 111The code is available at
https://github.com/oscar97pina/gssl-rgi

###### keywords:

Graph neural network , self-supervised learning , graph representation
learning , regularization

††journal: Neural Networks

[upc]organization=Universitat Politècnica de Catalunya - BarcelonaTech (UPC),
city=Barcelona, country=Spain

##  1 Introduction

The primary goal of self-supervised learning is to learn meaningful
representations from large amounts of unlabeled data that can be employed to
efficiently fit other downstream tasks with a small amount of labeled samples.
The challenge is to define an appropriate auxiliary task that leads the model
to capture the relevant parts of the input data.

A popular solution within the computer vision field is to force the encoder to
be invariant under certain transformations applied to the input, such as
rotation and cropping, by generating two views of the same image by applying
these random augmentations and maximizing the agreement between their
representations. Owing to their success, these works have been adapted by the
graph learning community in order to train graph neural networks encoders in a
self-supervised manner [1, 2, 3, 4]. Consequently, graph specific data
augmentation techniques are required to generate the graph views. Graph
transformations can focus on both node attributes and the topology of the
graph. Indeed, popular choices are node attribute masking and edge sampling
[5, 6, 7, 8]. However, the idea behind invariance via data augmentations roots
in the fact that it is assumed that these transformations do not change the
semantics of the data and the information contained about the downstream task
is kept. Whereas we can understand image augmentations, it is not clear how
graph transformations modify its semantics nor if they can be applied to all
graph domains [9]. To overcome this issue, graph diffusion has been proposed
to create the alternative views [10], and other works attempt to get rid of
the augmentations by designing strategies that leverage the local neighborhood
of the nodes [11, 12, 13].

In this manuscript, we introduce Regularized Graph Infomax (RGI), a simple yet
effective self-supervised learning framework for graph structured data. RGI
trains the encoder based on embedding propagation as self supervision signals,
which consists of propagating the output embeddings through the graph and
using them as prediction target from the node embeddings, which leads to
context-aware embeddings that do not encode unknown invariances. Additionally,
variance-covariance regularization [14] is used in order to avoid the collapse
of the representations, which is generally more efficient than contrastive
methods. Therefore, the algorithm is augmentation-free, non-contrastive, and
does not require a two-branch architecture nor complex training strategies.

This document is organized as follows, in Section 2 we provide the necessary
knowledge to fully understand the arguments and statements of the method, in
Section 3 we detail our algorithm RGI, motivate and explain the usage of
feature propagation as self-supervision signals (Section 3.2), provide a
local-global context perspective of RGI and feature propagation (Section 3.3),
detail the loss function (Section 3.4) and show that RGI is maximizing the
mutual information between the node embeddings and their propagation through
the graph (Section 3.5). Then, we perform empirical evaluation on popular
graph benchmarks in Section 4 and analyse the influence of different parts of
the algorithm in the ablation study of Section 5. Finally, we discuss RGI and
the results in Section 6 and conclude the work.

##  2 Background and related work

###  2.1 Information theory for representation learning

##### Mutual information

The mutual information (MI) between two random variables $X$ and $Y$, $I(X;Y)$
is a symmetric quantity, $I(X;Y)=I(Y;X)$, that measures how much information
one variable carries about the other. Formally, it is defined as:

$I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$ (1)

where $H(X)$ is the entropy of $X$ and $H(X|Y)$ is the conditional entropy of
$X$ given $Y$. This quantity can be lower-bounded by the expected
reconstruction error, which is usually employed in generative models:

$I(X;Y)=H(X)-H(X|Y)\geq H(X)-\mathcal{R}(X|Y)$ (2)

where $\mathcal{R}(X|Y)$ is the expected reconstruction error of $X$ given
$Y$. In practice, this expected error is approximated with the square loss or
the cross-entropy loss.

##### InfoMax principle

The InfoMax principle [15] states that a neural network can be trained in a
self-supervised manner by maximizing the mutual information between its input
$X$ and its output $Y$, $I(X;Y)$. To do so, [16] suggests that it is enough to
maximize the entropy of $Y$, which, for the example of a one layer n-to-n
network, is achieved by maximizing the logarithm of the jacobian of the
weights. The intuition is that this quantity can be seen as the log of the
volume space of $Y$ onto which the values of $X$ are mapped.

###  2.2 Multi-view representation learning

The InfoMax principle is extended to a multi-view approach, in which rather
than maximizing the MI between the input and output of the network, the
agreement is maximized between the representations of two different views of
the input. This scenario has two main advantages, (i) the loss is computed in
the representation space, which is in general lower dimensional and avoids
focusing on small details of the input and (ii) views can be defined to
capture different aspects of the data [17].

##### Local - global MI

Deep InfoMax [18] trains an encoder maximizing the average mutual information
between local patches and global representations of an image. Deep Graph
InfoMax [19] and InfoGraph [20] extend this work to the graph domain,
targetting the MI between node and graph level embeddings. The graph
representation is obtained with a global pooling layer applied to the local
node embeddings. In DGI, since most datasets consist of one single graph, the
authors create a corrupted version of the graph by shuffling the node features
and contrasting negative and positive pairs.

##### Invariance via data augmentation

SimCLR [2], BYOL [3], Barlow Twins [4] and VICReg [14], among others, create
two augmented views of an image via data augmentation, such as image rotation
and cropping, and train the encoder to be invariant to those augmentations,
not necessarily with MI objectives. These works are also extended for self-
supervised graph representation learning. For instance, GRACE [8] follows a
similar approach to SimCLR based on contrastive learning, BGRL [5] employs the
same asymmetric scheme than BYOL and G-BT, [7] directly extends Barlow Twins
cross-covariance regularization objective.

###  2.3 Avoiding collapse

Maximizing the agreement between views can lead to a total collapse in which
the encoder outputs the same representation independently from the input. In
order to diminish this phenomena, either architectural, regularization or
training tricks can be employed:

##### Contrastive learning

Contrastive methods not only encourage the two views of the input (positive
pairs) to be similar, but they also force views from different inputs
(negative pairs) to be different [2, 8, 9, 10, 19, 18]. They have been
successfully applied to all data domains, nonetheless, their performance is
highly influenced by the number of negative pairs, usually requiring many of
them to work efficiently.

##### Knowledge distillation

Knowledge distillation methods do not need negative pairs [3, 5]. Instead,
they construct a teacher-student asymmetric architecture combined with a stop-
gradient operation. Concretely, the networks are fed with two augmented views
and the student network is trained to predict the output of the teacher model.
Collapse is avoided by not backpropagating gradients through the teacher
network to update its parameters, but setting them to be a moving average of
the student’s parameters.

##### Covariance regularization

Covariance regularization consists of extending the loss function by including
regularization terms on the covariance matrix of the representations, forcing
high variances for every feature and low co-variances [14, 4, 6]. Under the
gaussianity assumption, these loss terms attempt to regularize the entropy of
the latent space to avoid the collapse [21].

Figure 1: Visual illustration of RGI

Given the input graph $\mathcal{G}=(\mathbf{X},\mathbf{A})$, a GNN encoder
$f_{\Theta}$ extracts node features
$\mathbf{U}=f_{\Theta}(\mathbf{X},\mathbf{A})$ and they are propagated through
the graph during $K$ steps to obtain the nodes’ global context
$\mathbf{V}=g_{K}(\mathbf{U},\mathbf{A})$. RGI maximizes the MI between
$\mathbf{U}$ and $\mathbf{V}$ based on the lower bound of the MI. To do so,
two auxiliary neural networks $h_{\phi}$ and $h_{\psi}$ are trained to predict
$V$ from $U$ and $U$ from $V$, respectively. Additionally, the covariance
matrices of the local and global node embeddings are regularized to have large
diagonal elements and off-diagonal elements close to zero, attempting to
regularize the entropy of $\mathbf{U}$ and $\mathbf{V}$. The neural networks
$h_{\phi}$ and $h_{\psi}$, as well as $\mathbf{V}$ are ignored during
inference.

##  3 Regularized graph infomax

In this section, we introduce the concept of feature propagation as self-
supervision signals and detail our algorithm RGI - Regularized Graph Infomax
for self-supervised learning on graphs, which we show that maximizes the
mutual information between the representations output by a graph neural
network encoder and their propagation through the graph. The algorithm is
described in Algorithm 1 and a visual illustration is shown in Figure 1.

Algorithm 1 RGI

Input: node features $\mathbf{X}$; adjacency matrix $\mathbf{A}$; parameters
$\Theta$, $\phi$ and $\psi$; backbone $f_{\Theta}$; reconstruction networks
$h_{\phi}$ and $h_{\psi}$; feature propagation function $g_{K}$.

repeat

// obtain node embeddings

$\mathbf{U}=f_{\Theta}({\mathbf{X}},{\mathbf{A}})$

// propagate node embeddings

$\mathbf{V}=g_{K}({\mathbf{U},\mathbf{A}})$

// reconstruction

$\mathbf{V}^{\prime}=h_{\phi}(\mathbf{U})$

$\mathbf{U}^{\prime}=h_{\psi}(\mathbf{V})$

// covariance matrices

$\mathcal{C}_{u}=\frac{1}{N}\bar{\mathbf{U}}^{T}\bar{\mathbf{U}}$

$\mathcal{C}_{v}=\frac{1}{N}\bar{\mathbf{V}}^{T}\bar{\mathbf{V}}$

// loss function

$\mathcal{L}_{rec}=\parallel\mathbf{U}-\mathbf{U}^{\prime}\parallel^{2}_{F}+\parallel\mathbf{V}-\mathbf{V}^{\prime}\parallel^{2}_{F}$

$\mathcal{L}_{var}=(1-$diag$(\mathcal{C}_{u}))^{2}+(1-$diag$(\mathcal{C}_{v}))^{2}$

$\mathcal{L}_{cov}=($off-diag$(\mathcal{C}_{u}))^{2}+($off-
diag$(\mathcal{C}_{v})^{2}$

$\mathcal{L}=\lambda_{1}\mathcal{L}_{rec}+\lambda_{2}\mathcal{L}_{var}+\lambda_{3}\mathcal{L}_{cov}$

// update parameters

$\Theta,\phi,\psi\leftarrow\bigtriangledown_{\Theta,\phi,\psi}\mathcal{L}$

until convergence

return $\mathbf{U}$

###  3.1 Context and notation

Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a graph of $N$ nodes where
$\mathcal{V}$ is the node set and $\mathcal{E}$ the edge set. Node attributes
$X$ come from a $d$-dimensional distribution and the nodes’ realizations are
arranged in a matrix
$\mathbf{X}=\\{\mathbf{x}_{i}\\}_{i=1}^{N}\in\mathbb{R}^{N\times d}$, where
$\mathbf{x}_{i}\in\mathbb{R}^{d}$ are the attributes of node $i$.
$\mathcal{E}$ comes in the form of (unweighted) adjacency matrix
$\mathbf{A}\in\\{0,1\\}^{N\times N}$, $\mathbf{A}_{ij}=1$ if the edge
$e_{ij}\in\mathcal{E}$, $0$ otherwise. The $K$-hop neighborhood of a node $i$,
$\mathcal{G}_{i}^{(K)}$, is represented by the set of neighbors’ attributes,
$\mathbf{X}_{i}^{(K)}$ and the induced adjacency matrix
$\mathbf{A}_{i}^{(K)}$. The same notation criterion of $X$, $\mathbf{X}$,
$\mathbf{x}_{i}$ and $\mathbf{X}_{i}$ will be employed for other node hidden
representations.

The goal of self-supervised learning on graphs is to fit a graph neural
network encoder $f_{\Theta}:\mathbb{R}^{N\times d}\times\\{0,1\\}^{N\times
N}\xrightarrow{}\mathbb{R}^{N\times D}$,
$\mathbf{U}=f_{\Theta}(\mathcal{G})=f_{\Theta}(\mathbf{X},\mathbf{A})$,
parametrized by $\Theta$, that obtains a $D$-dimensional vector representation
$\mathbf{u}_{i}$ for every node of the input graph without relying on node
annotations.

###  3.2 Feature propagation as supervision signals

RGI train an encoder to maximize the mutual information between the embeddings
output by the encoder, $\mathbf{U}=f_{\Theta}(\mathbf{X},\mathbf{A})$, and
their propagation through the graph,
$\mathbf{V}=g_{K}(\mathbf{U},\mathbf{A})$, named local and global node views
or embeddings, respectively.

As shown in Section 3.3, the propagated embeddings capture the global
information and structure of the graph while being particular to every node.
Leveraging them as supervision signals, the model is trained to output more
informative node embeddings that are aware of both their local and global
context within the graph. The propagation function $g_{K}$ is implemented as a
$K$ order polynomial of the graph adjacency matrix or any of its variants,
such as the symmetrically normalized adjacency matrix, so that RGI does not
rely on a two branch architecture and the computational burden to obtain the
supervision signals is minimal compared to other methods.

We perform feature propagation over the embeddings $\mathbf{U}$ to create the
supervision signals rather than the propagation of the raw input features
$\mathbf{X}$ since real world graph data is usually incomplete and sparse.
Additionally, there may be scenarios without initial node features.

###  3.3 Local and global contexts

We name $\mathbf{U}$ and $\mathbf{V}$ as node local and global node
embeddings, respectively. Note, however, that the global definition is
different from the global pooling proposed in DGI [19], since we define node
level global embeddings rather than graph level. The reason to call
$\mathbf{V}$ global is as follows. For a target node $i$, we obtain its local
embedding with a $L$-layer GNN,
$\mathbf{u}_{i}=f_{\Theta}(\mathbf{X}_{i}^{(L)},\mathbf{A}_{i}^{(L)})$, so
that it depends on its $L$-hop neighborhood. Afterwards, its global context is
computed by propagating the representations through the graph during $K$
steps, $\mathbf{v}_{i}=g_{K}(\mathbf{U}_{i}^{(K)},\mathbf{A}_{i}^{(K)})$.
Therefore, $\mathbf{v}_{i}$ contains information of the $(L+K)$-hop
neighborhood of $i$. It is known that for a small-world network $\mathcal{G}$
of $N$ nodes, the diameter of the graph is $diam(\mathcal{G})=log(N)$.
Consequently, in a small world network, $\mathbf{v}_{i}$ can encode global
information of every node in the graph as long as $L+K\simeq log(N)$.

###  3.4 Loss function

Based on Equation 2, the mean squared error is employed as loss function to
maximize the mutual information. RGI includes an auxiliary network $h_{\phi}$
implemented as a multi-layer perceptron to predict the local embeddings from
the global embeddings. The network is jointly trained with the encoder, but
discarted afterwards.

$\mathcal{L}_{rec}(U|V)=\frac{1}{N}\sum_{i=1}^{N}\parallel\mathbf{u}_{i}-h_{\phi}(\mathbf{v}_{i})\parallel_{2}^{2}$
(3)

However, as the representations $U$ are not fixed, but they depend on the
encoder parameters instead, only addressing the reconstruction error would
lead the encoder to a collapse, in which the input is ignored and it outputs a
constant representation. To alleviate this problem, we incorporate covariance
matrix regularization loss terms to avoid the collapse of the representations
[4, 14, 7, 6]:

$\mathcal{L}_{var}(U)=\frac{1}{D}\sum_{n=1}^{D}\left(1-\mathcal{C}_{nn}\right)^{2}$
(4)

$\mathcal{L}_{cov}(U)=\frac{1}{D}\sum_{n=1}^{D}\sum_{m\neq
n}\mathcal{C}_{nm}^{2}$ (5)

where $\mathcal{C}$ is the sample covariance matrix of $U$. Concretely, the
former guides the diagonal elements (variances) to be close to one whereas the
latter forces the non-diagonal elements (covariances) to be zero. Intuitively,
maximizing the variance avoids the total collapse to a constant
representation. On the other hand, covariance minimization encourages the
encoder to leverage the whole capacity of the representation space rather than
projecting the points to a lower dimensional subspace, also known as
dimensional collapse [22]. The loss function is a weighted combination of
these three terms:

$\mathcal{L}_{u}=\lambda_{1}\mathcal{L}_{rec}(U|V)+\lambda_{2}\mathcal{L}_{var}(U)+\lambda_{3}\mathcal{L}_{cov}(U)$
(6)

Where $\lambda_{1},\lambda_{2},\lambda_{3}\in\mathbb{R}$ are weight
hyperparameters. In practice, we symmetrize this loss by also predicting
$\mathbf{v}_{i}$ from $\mathbf{u}_{i}$ with another auxiliary network
$h_{\psi}$ and applying variance-covariance regularization to $V$.

###  3.5 Mutual information maximization

In this section, we show that, as previously stated in Section 3.2, RGI
maximizes the mutual information between the node embeddings and their
propagation.

######  Assumption 1.

The node embeddings $\mathbf{U}$ follow a Gaussian distribution $U\sim
N(\mu_{u},\Sigma_{u})$.

######  Assumption 2.

The propagated embeddings $\mathbf{V}$ follow a Gaussian distribution $V\sim
N(\mu_{v},\Sigma_{v})$.

######  Proposition 1.

Based on Assumption 1, minimizing the objective $\mathcal{L}_{u}$ of Equation
6 maximizes the mutual information between $U$ and $V$.

###### Proof.

The proof is based on the MI lower bound of Equation 2. In order to maximize
the MI between $U$ and $V$, it is necessary to address both the maximization
of $H(U)$ and the minimization of $\mathcal{R}(U|V)$, as the representations
$U$ are not fixed and only optimizing $\mathcal{R}(U|V)$ would also affect the
value of $H(U)$. The first term of the loss in Equation 6 is the
reconstruction error $\mathcal{R}(U|V)$, since our representations are
continuous-valued, this is achieved with the mean-squared error. Secondly,
under the Gaussianity assumption, the entropy of $U$ is proportional to the
logarithm of the determinant of the covariance matrix, that is, $H(U)\propto
log(\left|\Sigma_{u}\right|)$. Being the logarithm an increasing function, the
entropy maximization can be tackled by maximizing $\left|\Sigma_{u}\right|$.
Although we do not have access to the true covariance matrix, it is
approximated by targetting the sample covariance $\mathcal{C}$. Rather than
directly maximizing the determinant, the proxy consists of maximizing the
diagonal elements of the matrix while also forcing the non-diagonal elements
to be close to 0.

Despite the fact that the mutual information is a symmetric quantity, the
approximation based on the reconstruction error and sample covariance matrix
are not. Hence, the loss is symmetrized given the Assumption 2. ∎

##  4 Evaluation

In this section, we evaluate the quality of the node level representations
output by our method on both transductive and inductive settings.

##### Datasets

For transductive learning we run RGI on 4 popular benchmarks: Amazon
Computers, Amazon Photos, Coauthor CS and Coauthor Physics. Moreover, we
include the large scale ogbn-arxiv dataset. Finally, inductive learning
evaluation is addressed with the challenging PPI dataset. The statistics of
the datasets are shown in Table 1.

##### Linear evaluation

We follow the linear evaluation protocol on graphs to assess the quality of
the representations as proposed in [19]. It consists of first fitting a GNN
encoder in a fully self-supervised manner, freezing the weights of the
encoder, obtaining the node-level representations and fitting a linear
classifier to a downstream task without backpropagating the gradients through
the encoder. For comparability with other methods, the transductive settings
are evaluated in terms of accuracy of the predictions whereas for the PPI
dataset we employ the micro-average F1-score. As usual, since there is no
public split for these datasets, Amazon Computers, Amazon Photos, Coauthor CS
and Coauthor Physics are randomly split into train/validation/test (10% / 10%
/ 80%). The ogbn-arxiv dataset is split with the partition provided by Open
Graph Benchmark [23]. To evaluate on the PPI dataset, we employ the standard
pre-defined split, which has 20 graphs to fit the model, 2 graphs for
validation and another two for testing.

###  4.1 Transductive learning

##### Architecture

As in [19, 5], among others, we fix the encoder $f_{\Theta}$ to be a $L=2$
layer GCN [24] for the Amazon and Co-authorship datasets. We have set the
output dimensionality to be $512$ for all of them and the hidden, $1024$. We
include batch normalization [25] and ReLU activation after the first layer but
none of them is included after the second convolutional layer. We also apply
dropout regularization to the input graph. The architecture for the ogbn-arxiv
dataset is slightly different. Based on [5], we employ a $L=3$ GCN [24]
encoder, with layer normalization and ReLU activation after the first and the
second layer.

To obtain the global node embeddings, we propagate the embeddings with the
normalized adjacency matrix without self-loops for $K=1$ step, that is:

$\mathbf{V}=g_{K}(\mathbf{U},\mathbf{A})=\left({\mathbf{D}}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}\right)\mathbf{U}$
(7)

Finally, the networks $h_{\phi}$ and $h_{\psi}$ are implemented with a two
layer FCNN and ReLU activation in the hidden layer. However, no batch
normalization is employed for these auxiliary networks.

##### Numerical results

Table 2 shows the mean accuracy of the linear evaluation protocol on the
transductive graph settings. Our results are the average of 20 model weight
initializations and data splits. The other results are extracted from previous
reports. We can observe that RGI performs competitively in all datasets
despite its simplicity and achieves state of the art in some of them. In
except of Amazon Computers dataset, RGI is trained for $1,000$ epochs whereas
other methods such as BGRL require $10,000$ epochs [5]. For instance, when
BGRL is trained only for $1,000$ epochs, its performance in all datasets is
dropped [7]. Finally, as for the ogbn-arxiv dataset, while RGI performs
competitively, state-of-the-art performance is achieved by masking and
augmentation-based methods, which suggest that graph data augmentations and
the invariance to them are an appropriate solution for this particular dataset
and task.

Table 1: Transductive and inductive datasets

Name Task Num. Nodes Num. Edges Node features Num. Classes Amazon Computers
Transductive 13,752 245,861 767 10 Amazon Photos Transductive 7,650 119,081
745 8 Coauthor CS Transductive 18,333 81,894 6,805 15 Coauthor Physics
Transductive 34,493 247,962 8,415 5 ogbn-arxiv Transductive 168,343 1,166,243
128 40 PPI (24 graphs) Inductive 56,944 818,716 50 121 (multilabel)

Table 2: Classification accuracies on transductive datasets averaged for 20
weight initializations.

Method Am. Computers Am. Photos Co. CS Co. Physics Ogbn-Arxiv Raw ft. 73.81
$\pm$ 0.00 78.53 $\pm$ 0.00 90.37 $\pm$ 0.00 93.58 $\pm$ 0.00 55.50 $\pm$ 0.23
Random-Init 86.46 $\pm$ 0.38 92.08 $\pm$ 0.48 91.64 $\pm$ 0.29 93.71 $\pm$
0.29 68.94 $\pm$ 0.15 DGI [19] 83.95 $\pm$ 0.47 91.61 $\pm$ 0.22 92.15 $\pm$
0.63 94.51 $\pm$ 0.52 70.34 $\pm$ 0.16 GRACE [8] 89.53 $\pm$ 0.35 92.78 $\pm$
0.45 91.12 $\pm$ 0.20 OOM 71.51 $\pm$ 0.11 G-BT [7] 88.14 $\pm$ 0.33 92.63
$\pm$ 0.44 92.95 $\pm$ 0.17 95.07 $\pm$ 0.17 70.12 $\pm$ 0.18 CCA-SSG [6]
88.74 $\pm$ 0.28 93.14 $\pm$ 0.14 93.31 $\pm$ 0.22 95.38 $\pm$ 0.06 71.24
$\pm$ 0.20 BGRL [5] 90.34 $\pm$ 0.19 93.17 $\pm$ 0.30 93.31 $\pm$ 0.13 95.73
$\pm$ 0.05 71.64 $\pm$ 0.12 RGI (ours) 90.45 $\pm$ 0.08 92.94 $\pm$ 0.09 93.37
$\pm$ 0.07 95.91 $\pm$ 0.09 70.33 $\pm$ 0.25

*OOM refers to out-of-memory error in a 16GB GPU.

###  4.2 Inductive learning

Table 3: Classification micro-average F1 score on PPI dataset averaged for 20
weight initializations.

Method | PPI
---|---
Raw ft. | 42.20
Rdm-Init |  62.60 $\pm$ 0.20
DGI [19] |  63.80 $\pm$ 0.20
GMI [11] |  65.00 $\pm$ 0.02
GRACE [8] |  69.71 $\pm$ 0.17
G-BT [7] |  70.49 $\pm$ 0.19
BGRL [5] |  70.49 $\pm$ 0.05
GraphMAE [26]* |  63.60 $\pm$ 0.29
RGI (ours) | 72.16 $\pm$ 0.11

*Results are replicated with author’s official code but reducing the dimensionality to 512, as it is employed in RGI and others, as well as limiting the linear evaluation method to the one employed in RGI for a fairer comparison.

##### Architecture

Based on previous reports [7, 5], we implement the encoder with a $L=3$ layer
GAT [27] with ELU activation and skip connections. Both hidden and output
dimensionality are set to $D=512$. Although a mean-pooling scheme would be
more appropriate for inductive settings [28], we employ the same propagation
scheme than in transductive setting of Equation 7. In Section 5.1.1 we study
the effect of $g_{K}$. As initial node features are sparse in the PPI dataset,
we apply dropout regularization to the graph before propagation. The
reconstruction networks $h_{\phi}$ and $h_{\psi}$ have the same architecture
as in the transductive setting.

##### Numerical results

Table 3 shows the micro-average F1-score of the linear evaluation averaged for
20 runs. RGI outperforms the current state of the art on this challenging
dataset and only requires $2,000$ epochs to reach this performance.

##  5 Ablation study

In this section, we perform an exhaustive ablation study to evaluate the
influence of the different components of RGI. Generally, results show that RGI
performs robustly for different design configurations, specifically in
transductive settings.

###  5.1 Loss function components

RGI maximizes the mutual information between local and global node embeddings
by tackling the reconstruction error between them as well as entropy
regularization, which is composed by a variance and a covariance term. The
final objective is defined as a weighted sum of reconstruction, variance and
covariance terms by $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$,
respectively. In order to measure the importance of every term, we have
trained RGI by setting their corresponding weight $\lambda$ to zero and
observing the difference in performance. The results are shown in Table 4. We
observe that variance and covariance terms are required to avoid the collapse
of the representations, specifically in the inductive setting, which is more
sensitive to these parameters. Additionally, we observe that the
reconstruction error is also needed to make the algorithm achieve state-of-
the-art performance, validating hence the usage of propagated embeddings as
supervision signals.

Table 4: Effect of the components of the loss function. Linear evaluation
accuracy (micro-average F-Score for PPI dataset) after fitting and encoder
with distinct configurations averaged for 5 weight initializations and data
splits.

$\mathcal{L}$ Am. Computers Am. Photos Co. CS Co. Physics PPI (F-Score)
$\lambda_{1}=0$ 89.78 $\pm$ 0.09 92.25 $\pm$ 0.20 93.53 $\pm$ 0.06 95.96 $\pm$
0.05 68.83 $\pm$ 0.08 $\lambda_{2}=0$ 79.53 $\pm$ 0.96 89.62 $\pm$ 0.39 84.27
$\pm$ 0.39 92.57 $\pm$ 0.22 0.00 $\pm$ 0.00 $\lambda_{3}=0$ 85.70 $\pm$ 0.21
92.32 $\pm$ 0.12 92.51 $\pm$ 0.09 95.48 $\pm$ 0.07 49.43 $\pm$ 0.21
$\lambda_{1},\lambda_{2},\lambda_{3}\neq 0$ 90.45 $\pm$ 0.07 92.88 $\pm$ 0.13
93.37 $\pm$ 0.03 95.93 $\pm$ 0.04 72.25 $\pm$ 0.09

####  5.1.1 Feature propagation function

A propagation function $g_{K}(\mathbf{U},\mathbf{A})$ is used to create the
supervision signals from the node embeddings. In this section, we evaluate
different shift operators, namely the mean-propagation
$\mathbf{D}^{-1}\mathbf{A}$, the normalized adjacency matrix
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$ and the
normalized Laplacian
$\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
as well as different values of $K$ (1, 2 and 5). Note that none of them
include the self-loops as opposed to most GNN propagation schemes. Table 5
shows that RGI performs robustly in all settings as long as the global
propagation scheme captures the low-frequency components of the data, obtained
with the adjacency matrix (either mean pooling or normalized), which is
motivated by the homophily of the downstream task on the graph.

Table 5: Effect of global propagation scheme. Linear evaluation accuracy
(micro-average F-Score for PPI dataset) for different schemes to obtain the
global node embeddings propagating the local ones averaged for 5 weight
initializations and data splits.

$g(\mathbf{A})$ $K$ Am. Computers Am. Photos Co. CS Co. Physics PPI (F-Score)
$\mathbf{D}^{-1}\mathbf{A}$ 1 90.52 $\pm$ 0.09 92.81 $\pm$ 0.14 93.39 $\pm$
0.04 95.89 $\pm$ 0.07 71.29 $\pm$ 0.11 $\mathbf{D}^{-1}\mathbf{A}$ 2 90.43
$\pm$ 0.16 92.84 $\pm$ 0.15 93.37 $\pm$ 0.04 95.89 $\pm$ 0.07 71.28 $\pm$ 0.08
$\mathbf{D}^{-1}\mathbf{A}$ 5 90.41 $\pm$ 0.17 92.84 $\pm$ 0.08 93.26 $\pm$
0.06 95.86 $\pm$ 0.08 70.06 $\pm$ 0.08
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$ 1 90.46
$\pm$ 0.06 92.89 $\pm$ 0.15 93.37 $\pm$ 0.02 95.92 $\pm$ 0.04 72.25 $\pm$ 0.13
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$ 2 90.25
$\pm$ 0.10 92.69 $\pm$ 0.14 93.35 $\pm$ 0.02 95.91 $\pm$ 0.04 71.94 $\pm$ 0.08
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$ 5 90.08
$\pm$ 0.16 92.75 $\pm$ 0.11 93.23 $\pm$ 0.05 95.85 $\pm$ 0.05 70.08 $\pm$ 0.17
$\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
1 84.22 $\pm$ 0.28 90.23 $\pm$ 0.20 93.01 $\pm$ 0.12 95.50 $\pm$ 0.11 61.81
$\pm$ 0.08
$\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
2 84.58 $\pm$ 0.22 90.75 $\pm$ 0.25 92.89 $\pm$ 0.14 95.46 $\pm$ 0.08 61.35
$\pm$ 0.13
$\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
5 88.65 $\pm$ 0.16 92.62 $\pm$ 0.20 91.82 $\pm$ 0.27 95.53 $\pm$ 0.09 64.50
$\pm$ 0.12

##  6 Discussion

We have presented an algorithm that achieves state-of-the-art performance even
though it is much simpler in terms of theoretical interpretation, architecture
and training strategy than other methods. In this section, we detail aspects
of RGI and its advantages and limitations with respect to other methods.

##### Augmentation-free and non-contrastive

Graph contrastive learning is the most popular approch to avoid the collapse
of the representations by relying on negative pair sampling [19, 20, 8, 9,
13]. BGRL [5], CCA-SSG [6] and G-BT [7], instead, do not need negative
samples. The former avoids collapse with an asymmetric architecture and the
other two, regularizing the covariance matrix of the representations. In this
work, we also adopt a regularized solution since it is more interpretable and
naturally avoids the collapse whereas it is still an open problem to
theoretically demonstrate that bootstrapped methods avoid trivial solutions.
However, these methods train an encoder with the invariance via data
augmentation principle. It has been stated that transformations that drop
information may modify the semantics of the graph so the invariance assumption
may be incorrect and not hold for all graph domains. RGI, instead, requires no
transformations as simply employs feature propagation to create the
supervision signals, so that it is much more intuitive and only involves graph
convolution-like operations.

##### Single branch architecture

Current state of the art algorithms usually rely on a joint embedding
architecture that require multiple forward passes at each training step. For
example, BGRL [5] forwards the graph through the encoder four times at each
iteration. On the contrary, RGI is much more efficient and only performs one
single forward pass while achieving similar performance to BGLR and other
methods.

##### Simplicity and effectiveness

Being augmentation free, requiring a single branch architecture and being non-
contrastive, substantially reduce both the time and space complexity of the
algorithm for self-supervised learning on graphs with respect to other
methods. First of all, the complexity of contrastive methods is quadratic with
respect to the number of nodes, that is $\mathcal{O}(N^{2})$. Instead,
covariance regularization makes the complexity be quadratic on the embedding
space dimensionality $\mathcal{O}(D^{2})$, which is generally much lower than
the number of nodes $D<<N$. Additionally, being a single branch architecture
reduces the time and space complexity of computing multiple forward and
backward passes through the encoder during training.

##### Evaluation protocol

In the context of self-supervised learning of graph neural networks, while the
linear evaluation protocol is commonly used to compare different methods, it
is important to consider the impact of other factors such as the differences
in the encoder architecture, as well as the training protocol and evaluation-
related hyperparameters. In our experiments, we observed a drop in performance
on GraphMAE [26], which reports state-of-the-art performance in the original
paper, when we re-ran the method with the same dimensionality and linear
evaluation training than the majority of the other methods of the comparison.
This highlights the importance of a standardized evaluation protocol for self-
supervised GNNs, which can help to ensure fair comparisons between different
methods and provide a more accurate assessment of their performance.
Therefore, we suggest that future work in the field of self-supervised
learning on graphs should consider addressing these concerns.

##### Limitations

RGI has been shown to be an effective algorithm to learn node-level
representations in a self-supervised manner. However, by definition, RGI
operates on homogeneous graphs so that, in general, it could not be directly
applied to heterogeneous scenarios. Although RGI requires a single branch
architecture, which simplifies the training procedure with respect other
methods, it comes at expense of extra depth to obtain the propagated
embeddings, which increase the complexity in settings in which neighbor
sampling is a must. Nonetheless, our experiments show that $K=1$ is generally
the best choice as the number of steps to propagate the information through
the graph, so that this issue is alleviated.

##  7 Conclusion

In this work we have introduced RGI, a simple yet effective framework for
self-supervised learning on graphs based on the propagation of node embeddings
to generate supervision signals. The objective function maximizes the mutual
information between node embeddings and their propagation by addressing the
reconstruction between them and regularizing the covariance matrix of the
representations to avoid the total and dimensional collapse. We have shown
that, in spite of its simplicity and that it does not require training for too
long, it achieves state-of-the-art performance on multiple transductive and
inductive datasets following the linear evaluation protocol on downstream
tasks.

##  8 Acknowledgements

This work has been supported by the Spanish Research Agency (AEI) under
project PID2020-116907RB-I00 of the call MCIN/ AEI /10.13039/501100011033 and
the FI-AGAUR grant funded by Direcció General de Recerca (DGR) of Departament
de Recerca i Universitats (REU) of the Generalitat de Catalunya.

## References

  * [1] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, G. Hinton, Big self-supervised models are strong semi-supervised learners, arXiv preprint arXiv:2006.10029 (2020).
  * [2] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, arXiv preprint arXiv:2002.05709 (2020).
  * [3] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, M. Valko, Bootstrap your own latent: A new approach to self-supervised learning (2020).  arXiv:2006.07733.
  * [4] J. Zbontar, L. Jing, I. Misra, Y. LeCun, S. Deny, Barlow twins: Self-supervised learning via redundancy reduction, arXiv preprint arXiv:2103.03230 (2021).
  * [5] S. Thakoor, C. Tallec, M. G. Azar, M. Azabou, E. L. Dyer, R. Munos, P. Veličković, M. Valko, Large-scale representation learning on graphs via bootstrapping (2021).  arXiv:2102.06514.
  * [6] H. Zhang, Q. Wu, J. Yan, D. Wipf, S. Y. Philip, From canonical correlation analysis to self-supervised graph neural networks, in: Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  * [7] P. Bielak, T. Kajdanowicz, N. V. Chawla, Graph barlow twins: A self-supervised representation learning framework for graphs (2021).  arXiv:2106.02466.
  * [8] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, L. Wang, Deep Graph Contrastive Representation Learning, in: ICML Workshop on Graph Representation Learning and Beyond, 2020\.
URL http://arxiv.org/abs/2006.04131

  * [9] D. Xu, W. Cheng, D. Luo, H. Chen, X. Zhang, Infogcl: Information-aware graph contrastive learning, in: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Vol. 34, Curran Associates, Inc., 2021, pp. 30414–30425.
URL
https://proceedings.neurips.cc/paper/2021/file/ff1e68e74c6b16a1a7b5d958b95e120c-Paper.pdf

  * [10] K. Hassani, A. H. Khasahmadi, Contrastive multi-view representation learning on graphs, in: Proceedings of International Conference on Machine Learning, 2020, pp. 3451–3461.
  * [11] Z. Peng, W. Huang, M. Luo, Q. Zheng, Y. Rong, T. Xu, J. Huang, Graph Representation Learning via Graphical Mutual Information Maximization, in: Proceedings of The Web Conference, 2020.  doi:https://doi.org/10.1145/3366423.3380112.
  * [12] N. Lee, J. Lee, C. Park, Augmentation-free self-supervised learning on graphs (2022).
  * [13] H. Wang, J. Zhang, Q. Zhu, W. Huang, Augmentation-free graph contrastive learning with performance guarantee (2022).  doi:10.48550/ARXIV.2204.04874.
URL https://arxiv.org/abs/2204.04874

  * [14] A. Bardes, J. Ponce, Y. LeCun, Vicreg: Variance-invariance-covariance regularization for self-supervised learning, in: ICLR, 2022.
  * [15] R. Linsker, Self-organization in a perceptual network, Computer 21 (3) (1988) 105–117.  doi:10.1109/2.36.
  * [16] A. J. Bell, T. J. Sejnowski, An information-maximization approach to blind separation and blind deconvolution, Neural Comput 7 (6) (1995) 1129–1159.
  * [17] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, M. Lucic, On mutual information maximization for representation learning, in: International Conference on Learning Representations, 2020.
URL https://openreview.net/forum?id=rkxoh24FPH

  * [18] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, Y. Bengio, Learning deep representations by mutual information estimation and maximization, in: International Conference on Learning Representations, 2019.
URL https://openreview.net/forum?id=Bklr3j0cKX

  * [19] P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, R. D. Hjelm, Deep Graph Infomax, in: International Conference on Learning Representations, 2019.
URL https://openreview.net/forum?id=rklz9iAcKQ

  * [20] F.-Y. Sun, J. Hoffman, V. Verma, J. Tang, Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization, in: International Conference on Learning Representations, 2019.
  * [21] R. Shwartz-Ziv, R. Balestriero, Y. LeCun, What do we maximize in self-supervised learning? (2022).  doi:10.48550/ARXIV.2207.10081.
URL https://arxiv.org/abs/2207.10081

  * [22] T. Hua, W. Wang, Z. Xue, Y. Wang, S. Ren, H. Zhao, On feature decorrelation in self-supervised learning, arXiv e-prints (2021) arXiv–2105.
  * [23] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, J. Leskovec, Open graph benchmark: Datasets for machine learning on graphs (2021).  arXiv:2005.00687.
  * [24] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: International Conference on Learning Representations, 2017.
URL https://openreview.net/forum?id=SJU4ayYgl

  * [25] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, JMLR.org, 2015, p. 448–456.
  * [26] Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, J. Tang, Graphmae: Self-supervised masked graph autoencoders, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 594–604.
  * [27] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: International Conference on Learning Representations, 2018.
URL https://openreview.net/forum?id=rJXMpikCZ

  * [28] W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc., 2017.
  * [29] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch (2017).
  * [30] M. Fey, J. E. Lenssen, Fast graph representation learning with PyTorch Geometric, in: ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  * [31] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [32] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv.org/abs/1412.6980

  * [33] J. McAuley, C. Targett, Q. Shi, A. van den Hengel, Image-based recommendations on styles and substitutes, in: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 43–52.  doi:10.1145/2766462.2767755.
URL https://doi.org/10.1145/2766462.2767755

  * [34] A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. P. Hsu, K. Wang, An overview of microsoft academic service (mas) and applications, in: Proceedings of the 24th International Conference on World Wide Web, WWW ’15 Companion, Association for Computing Machinery, New York, NY, USA, 2015, p. 243–246.  doi:10.1145/2740908.2742839.
URL https://doi.org/10.1145/2740908.2742839

  * [35] M. Zitnik, J. Leskovec, Predicting multicellular function through multi-layer tissue networks, Bioinformatics 33 (14) (2017) 190–198.

##  Appendix A Implementation details

Algorithm 2 Pseudo-code PyTorch implementation of RGI

# obtain local views

u = f(x, adj)

# get symmetric adjacency matrix

deg = adj.sum(dim=-1) # adj dense and symmetric

deg = diagonal(inverse(deg).sqrt())

shift = deg @ adj @ deg

# obtain global views

v = matrix_power(shift,K) @ u

# batch size and dimensionality

N, D = u.size()

# reconstruction between views

v_prime = h_phi(u)

u_prime = h_psi(v)

# mean - centering

u_norm = u - u.mean(dim=0)

v_norm = v - v.mean(dim=0)

# covariance matrices

cov_u = (u_norm.T @ u_norm) / (N-1)

cov_v = (v_norm.T @ v_norm) / (N-1)

# reconstruction loss

rec_loss = mse_loss(u, u_prime) + mse_loss(v, v_prime)

# variance loss

var_loss = diagonal(cov_u).add_(-1).pow_(2).sum() / D + \

diagonal(cov_v).add_(-1).pow_(2).sum() / D

# covariance loss

cov_loss = off_diagonal(cov_u).pow_(2).sum() / D + + \

off_diagonal(cov_v).pow_(2).sum() / D

# total loss

loss = lambd_1 * rec_loss + lambd_2 * var_loss + lambd_3 * cov_loss

# optimizer

loss.backward()

optimizer.step()

##### Implementation

RGI and all neural networks are implemented in PyTorch [29] and PyTorch
Geometric [30]. Algorithm 2 shows a PyTorch-like pseudo-code implementation of
RGI. All our experiments are run in a single 16GB GPU. For linear evaluation,
we have employed Sci-kit Learn library [31] for Amazon Computers, Amazon
Photos, Coauthor CS and Coauthor Physics datasets and PyTorch for the PPI
dataset, taking the implementation from [5].

##### Optimization

All models have been trained with Adam optimizer [32] and a weight decay of
$10^{-5}$. Additionally, we have employed a learning rate scheduler with
linear warmup for $n_{warmup}$ epochs and cosine decay for the remaining
$n_{epochs}-n_{warmup}$, where $n_{epochs}$ is the total number of epochs. The
values of $n_{warmup}$ and $n_{epochs}$ for the different datasets can be
found in Table 6. In transductive settings, we perform full-graph optimization
at each gradient step. Alternatively, for the PPI dataset, which is multi-
graph, we set a batch size of 1 graph for each gradient step due to memory
constraints of the GAT encoder.

##### Node feature normalization

Input node features are normalized row-wise with the L1 norm for transductive
settings whereas no normalization is applied to PPI node features. Before
fitting the linear classifier with the obtained representations, they are row-
wise L2 normalized.

##### Hyperparameters

Some of the hyperparameters have been tuned with a small search whereas others
have been fixed. Table 6 shows the hyper-parameters settings employed to
obtain the results in Table 2 and Table 3. The search space of the optimized
hyper-parameters is the following:

  * 1.

Propagation matrix :
$\\{\mathbf{D}^{-1}\mathbf{A},\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}},\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}\\}$

  * 2.

Number of global propagation steps $(K)$ : $\\{1,2,5\\}$

  * 3.

Number of training epochs $(n_{epochs})$ : $\\{1000,2000,5000\\}$

  * 4.

Learning rate $(lr)$ : $\\{10^{-3},10^{-4},10^{-5}\\}$

  * 5.

Reconstruction loss weight $(\lambda_{1})$: $\\{10,15,20\\}$

  * 6.

Variance loss weight $(\lambda_{2})$: $\\{5,10,15\\}$

  * 7.

Covariance loss weight $(\lambda_{3})$: $\\{1,5,10\\}$

  * 8.

Dropout regularization probability input graph $(p_{input})$ : $\\{0.0,0.5\\}$

  * 9.

Dropout regularization probability before propagation $(p_{local})$ :
$\\{0.0,0.5\\}$

The number of GNN layers $(L)$ and embedding size $(D)$ have been fixed
according to previous reports for better comparability. The number of warmup
epochs is set to be $n_{epochs}/10$.

Table 6: Hyperparameters.

Method Am. Computers Am. Photos Co. CS Co. Physics PPI $L$ 2 2 2 2 3 $K$ 1 1 1
1 1 $D$ 512 512 512 512 512 Propagation
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
$\mathbf{D}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-\frac{1}{2}}$
$n_{epochs}$ 5000 1000 1000 1000 2000 $n_{warmup}$ 500 100 100 100 200 $lr$
$10^{-4}$ $10^{-4}$ $10^{-5}$ $10^{-5}$ $10^{-4}$ $\lambda_{1}$ 10 10 20 20 15
$\lambda_{2}$ 5 5 15 15 10 $\lambda_{3}$ 1 1 1 1 10 $p_{input}$ 0.5 0.5 0.5
0.5 0.0 $p_{local}$ 0.0 0.0 0.0 0.0 0.5

##  Appendix B Dataset details

Amazon Computers, Amazon Photos are extracted from the Amazon co-purchase
graph [33] whose nodes represent products and edges encode whether two
elements are usually purchased together. Node features are a vector
representation of a bag-of-words from product’s reviews and nodes are
classified into 10 (for Computers) and 8 (for Photos) classes, given by the
product category. Since there is no standard split for these datasets, we
randomly split the nodes into (10/10/80%) for train, validation and test,
respectively.

Coauthor Computer Science, Coauthor Physics are from the Microsoft Academic
Graph [34] from the KDD Cup 2016 challenge. Nodes represent authors, which are
connected by an edge if they have co-authored a paper. Node features encode
the keywords of each author’s papers and authors are labeled into 15 (for CS)
and 5 (for Physics) according to their most active field of study. We also use
a random split into (10/10/80%) for train, validation and test, respectively.

PPI is a proteint-protein interaction network [35]. It consists of 24
independent graphs that correspond to different human tissues, whose nodes are
proteins and edges represent interactions between them. Node features are
biological properties and they are labeled according to the protein functions.
We employ 20 graphs for training, 2 for validation and 2 for testing.