# A Bayesian Unification of Self-Supervised Clustering and Energy-Based Models

Emanuele Sansone<EMAIL_ADDRESS>
Department of Electrical Engineering
KU Leuven
Leuven, 3001, Belgium Robin Manhaeve<EMAIL_ADDRESS>
Department of Computer Science
KU Leuven
Leuven, 3001, Belgium

###### Abstract

Self-supervised learning is a popular and powerful method for utilizing large
amounts of unlabeled data, for which a wide variety of training objectives
have been proposed in the literature. In this study, we perform a Bayesian
analysis of state-of-the-art self-supervised learning objectives, elucidating
the underlying probabilistic graphical models in each class and presenting a
standardized methodology for their derivation from first principles. The
analysis also indicates a natural means of integrating self-supervised
learning with likelihood-based generative models. We instantiate this concept
within the realm of cluster-based self-supervised learning and energy models,
introducing a novel lower bound which is proven to reliably penalize the most
important failure modes. Furthermore, this newly proposed lower bound enables
the training of a standard backbone architecture without the necessity for
asymmetric elements such as stop gradients, momentum encoders, or specialized
clustering layers—typically introduced to avoid learning trivial solutions.
Our theoretical findings are substantiated through experiments on synthetic
and real-world data, including SVHN, CIFAR10, and CIFAR100, thus showing that
our objective function allows to outperform existing self-supervised learning
strategies in terms of clustering, generation and out-of-distribution
detection performance by a wide margin. We also demonstrate that GEDI can be
integrated into a neuro-symbolic framework to mitigate the reasoning shortcut
problem and to learn higher quality symbolic representations thanks to the
enhanced classification performance.

Keywords: self-supervised learning, energy-based models, deep learning, neuro-
symbolic integration.

##  1 Introduction

Self-supervised learning (SSL) has achieved impressive results in recent years
due to its ability to learn good representations from a large amount of
unlabeled data. A significant amount of effort has been dedicated to analyzing
existing SSL approaches in order to identify fundamental principles that can
guide the design of new SSL objectives and algorithms. However, a common
framework or methodology capable of encompassing, explaining, or deriving the
multitude of objectives developed in recent years is currently lacking. In
this work, we present a Bayesian interpretation of recent SSL objectives by
uncovering their underlying probabilistic graphical models. This approach
highlights a common methodology that can be employed to derive existing
objectives from foundational principles. This Bayesian perspective naturally
leads to the development of a comprehensive framework enabling to connect SSL
to the realm of likelihood-based generative models. We propose to instantiate
this framework using a novel objective, called the GEnerative DIscriminative
lower bound (GEDI for short) that enables the training of a neural network in
both a generative (akin to an energy-based model) and a discriminative manner
(similar to SSL clustering), thus regarding the generative and discriminative
properties as ”two sides of the same coin”. Theoretical analysis shows that
the proposed objective is guaranteed to avoid the triad of failure modes,
including representation collapse, cluster collapse, and the problem of
permutation invariance and label inconsistency to data augmentations,
ultimately leading to improved performance in terms of clustering, generation,
and out-of-distribution detection compared to state-of-the-art cluster-based
SSL solutions. We substantiate our theoretical findings through experiments
conducted on both toy and real-world datasets. Specifically, our results
demonstrate that GEDI can achieve a significant improvement of 4 and 2
percentage points on SVHN and CIFAR-10, respectively, and 22 percentage points
on CIFAR-100 datasets in terms of clustering performance compared to state-of-
the-art baselines. Additionally, in the context of generation performance,
GEDI can effectively compete with existing energy-based solutions, whereas
traditional SSL approaches fall short. Most importantly, we highlight that the
generative nature of GEDI is a crucial aspect that enhances the model’s
ability to confidently detect out-of-distribution data, surpassing purely
discriminative baselines in this regard. Finally, we show that GEDI can be
easily integrated into a neuro-symbolic framework like DeepProbLog (Manhaeve
et al., 2018), to learn symbolic representations and mitigating the
representational collapse/reasoning shortcuts problem arising in the neuro-
symbolic setting (Manhaeve et al., 2018, 2021; Sansone and Manhaeve, 2023a, b;
Marconato et al., 2023). Overall, this work can be regarded as the integral
version of the paper in (Sansone, 2023), which improves upon the work in
(Sansone and Manhaeve, 2022) by providing a principled objective enabling to
train a backbone network both generatively and discriminatively in a single
joint stage and avoiding to introduce additional computational costs compared
to purely discriminative approaches.

The article is structured as follows: In Section 2, we provide a Bayesian
interpretation of three classes of SSL approaches, namely contrastive,
negative-free and cluster-based methods. In Section 3, we propose a Bayesian
integration of SSL and likelihood-based generative models. In Section 4, we
take a step further and provide an instantiation of the general framework,
thus providing the first principled objective guaranteed to avoid the triad of
failure modes. In Section 5, we review related work on SSL and in Section 6 we
discuss the experimental analysis. Finally, in Section 7, we conclude by
discussing future research directions for SSL.

##  2 A Bayesian Interpretation of SSL

Figure 1: Different function blocks and variables used throughout the analysis
of existing SSL approaches.

(a) Contrastive (CT)

(b) Negative-Free (NF)

(c) Discriminative (DI,GEDI)

Figure 2: Probabilistic graphical models for the different classes of self-
supervised learning approaches. White and grey nodes represent hidden and
observed vectors/variables, respectively. Solid arrows define the generative
process, whereas dashed arrows identify auxiliary posterior
densities/distributions.

Let us introduce the random quantities used throughout this section to analyze
self-supervised learning approaches. We use (i) $x_{i}\in\Omega$, where
$\Omega$ is a compact subset of $\mathbb{R}^{d}$, to identify observed data
drawn independently from an unknown distribution $p(x)$ and (ii)
$x_{i}^{\prime}\in\Omega$ to identify observed data drawn independently from a
stochastic augmentation strategy $\mathcal{T}(x^{\prime}|x)$. We use
$\xi_{i}\in\mathbb{R}^{h}$, and $w\in\mathbb{R}^{l}$, to identify a residual
latent representation and a global latent embedding, respectively (cf. Fig.
1). Importantly, the residual latent representation is obtained by computing
the difference between the encoding of the data and its augmented version
through function $enc:\Omega\rightarrow\mathbb{R}^{h}$, and the global
embedding summarizes the information common to all instances in a dataset (we
are going to see its precise definition later on). Finally, we introduce
$y_{i}\in\\{1,\dots,c\\}$ to identify the latent labels over $c$ categories
associated to input data and their augmented version, respectively. $\ell_{i}$
is a categorical variable defined on $n$ classes (same as the number of
samples) and i is used to index different samples, viz. $i=\\{1,\dots,n\\}$

We distinguish self-supervised learning approaches according to three
different classes: 1) contrastive, 2) negative-free (or non-contrastive) and
3) cluster-based (or discriminative) methods. Fig. 2 shows the corresponding
probabilistic graphical models for each of the self-supervised learning
classes. From Fig. 2, we can also see how the different random quantities, we
have just introduced, are probabilistically related to each other.

We are now ready to give an interpretation of the different SSL classes.

###  2.1 Contrastive SSL

Contrastive self-supervised learning can be interpreted as a probabilistic
instance classification problem by using the graphical model in Fig. 2(a). In
particular, we can define the following conditional density:

$p(\ell_{i}|x_{i};\Theta)=\frac{e^{sim(g(x_{\ell_{i}}),g(x_{i}))/\tau}}{\sum_{j=1}^{n}e^{sim(g(x_{j}),g(x_{i}))/\tau}}$
(1)

where $sim$ is a similarity function, $\tau>0$ is temperature parameter used
to calibrate the uncertainty for $p(\ell_{i}|x_{i};\Theta)$ and
$\Theta=\\{\theta,\\{x_{i}\\}_{i}^{n}\\}$ is the set of parameters , including
the parameters of the embedding function and the observed data. The learning
criterion can be obtained from the expected log-likelihood computed on the
observed random quantities (and using the factorization provided by the
graphical model), namely:

$\displaystyle\mathbb{E}_{p(x_{1:n},\ell_{1:n})}\big{\\{}\log$ $\displaystyle
p(x_{1:n},\ell_{1:n};\Theta)\big{\\}}=$
$\displaystyle=\mathbb{E}_{\prod_{j=1}^{n}p(x_{j})\delta(\ell_{j}-j)}\big{\\{}\log\prod_{i=1}^{n}p(x_{i})p(\ell_{i}|x_{i};\Theta)\big{\\}}$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\big{\\{}\log
p(x_{i})\big{\\}}+\mathbb{E}_{\prod_{j=1}^{n}p(x_{j})\delta(\ell_{j}-j)}\bigg{\\{}\sum_{i=1}^{n}\log
p(\ell_{i}|x_{i};\Theta)\bigg{\\}}$
$\displaystyle=\underbrace{\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\big{\\{}\log
p(x_{i})\big{\\}}}_{\text{Neg. entropy term,
$-H_{p}(x_{1:n})$}}+\underbrace{\mathbb{E}_{\prod_{j=1}^{n}p(x_{j})}\bigg{\\{}\sum_{i=1}^{n}\log
p(\ell_{i}=i|x_{i};\Theta)\bigg{\\}}}_{\text{Conditional log-likelihood term
$\mathcal{L}_{CT}(\Theta)$}}$ (2)

where $\delta$ in the second equality is a Kronecker delta function and we use
for instance $x_{1:n}$ as a compact way to express $x_{1},\dots,x_{n}$. From
Eq. (2), we observe that the expected log-likelihood can be rewritten as the
sum of two quantities, namely a negative entropy and a conditional log-
likelihood terms. However, only the second addend in Eq. (2) is relevant for
maximization purposes over the parameters $\theta$. Importantly, we can now
show that the conditional log-likelikood term in Eq. (2), coupled with the
definition provided in Eq. (1), corresponds to the notorious InfoNCE objective
(den Oord et al., 2018). To see this, let us recall InfoNCE:

$\displaystyle\text{InfoNCE}\propto\mathbb{E}_{\prod_{j=1}^{n}p(x_{j},z_{j})}\bigg{\\{}\sum_{i=1}^{n}\log\frac{e^{f(x_{i},x_{i})}}{\sum_{k=1}^{n}e^{f(x_{k},x_{i})}}\bigg{\\}}$

By choosing $p(x,z)=p(x)\delta(z-g(x))$ and $f(x,z)=sim(g(x),z)/\tau$ we
recover the conditional log-likelihood term in Eq. (2). Importantly, other
contrastive objectives, such as CPC (O. Henaff, 2020), SimCLR (Chen et al.,
2020), ProtoCPC (Lee, 2022), KSCL (Xu et al., 2022) to name a few, can be
obtained once we have a connection to InfoNCE. In the Supplementary Material,
we provide an additional and alternative analysis of contrastive learning
approaches (cf. Section A), we recall the connections with mutual information
objectives (cf. Section B) and we show the derivation of InfoNCE (cf. Section
C) and an example of corresponding surrogate objective based on ProtoCPC (cf.
Section D).

###  2.2 Non-contrastive/Negative-Free SSL

We can provide a probabilistic interpretation also for negative-free SSL using
the graphical model shown in Fig. 2(c). Note that, for the sake of simplicity
in the graph and in the following derivation, the latent variables ($w$ and
all $\xi_{i}$) are considered independent of each other only for the
generation process. In reality, one should consider an alternative but
equivalent model, using a generating process including also the edges
$x_{i}\rightarrow w$, $\xi_{i}\rightarrow x_{i}^{\prime}$ and defining
$p(w|x_{i})=p(w)$ and
$p(x_{i}^{\prime}|x_{i},\xi_{i})=\mathcal{T}(x_{i}^{\prime}|x_{i})$ for all
$i=1\dots,n$. Based on these considerations, we can define the prior and our
auxiliary and inference densities for the model in the following way:

$\displaystyle p(w)$ $\displaystyle=\mathcal{N}(w|0,I)$ $\displaystyle
p(\xi_{i})$ $\displaystyle=\mathcal{N}(\xi_{i}|0,I)$ $\displaystyle
q(w|x_{1:n};\Theta)$ $\displaystyle=\mathcal{N}(w|0,\Sigma)$ $\displaystyle
q(\xi_{i}|x_{i},x_{i}^{\prime};\Theta)$
$\displaystyle=\mathcal{N}(\xi_{i}|enc(x_{i})-enc(x_{i}^{\prime}),I)$ (3)

where $\mathcal{N}(\cdot|\mu,\Sigma)$ refers to a multivariate Gaussian
density with mean $\mu$ and covariance $\Sigma$, $I$ is an identity matrix,
$\Sigma=\sum_{i=1}^{n}(g(x_{i})-\bar{g})(g(x_{i})-\bar{g})^{T}+\beta I$,
$\beta$ is positive scalar used to ensure the positive-definiteness of
$\Sigma$, $\bar{g}=1/n\sum_{i=1}^{n}g(x_{i})$ and $\Theta=\\{\theta\\}$.
Importantly, while $q(w|x_{1:n})$ in Eq. (3) is used to store the global
statistical information of the data in the form of an unnormalized sample
covariance, $q(\xi_{i}|x_{i},x_{i}^{\prime})$ is used to quantify the
difference between a sample and its augmented version in terms of their latent
representation. Similarly to previous SSL classes and by reusing definitions
in Eq. (3), we can devise the learning criterion in the following way:

$\displaystyle\mathbb{E}_{p(x_{1:n})}\\{\log p(x_{1:n};\Theta)\\}$
$\displaystyle\geq-
H_{p}(x_{1:n})-\mathbb{E}_{p(x_{1:n})}\\{KL(q(w|x_{1:n};\Theta)\|p(w))\\}$
$\displaystyle\quad-\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\\{KL(q(\xi_{1:n}|x_{1:n},x_{1:n}^{\prime};\Theta)\|p(\xi_{1:n}))\\}$
$\displaystyle=-H_{p}(x_{1:n})-\mathbb{E}_{p(x_{1:n})}\\{KL(q(w|x_{1:n};\Theta)\|p(w))\\}$
$\displaystyle\quad-\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\\{KL(q(\xi_{i}|x_{i},x_{i}^{\prime};\Theta)\|p(\xi_{i}))\\}$
$\displaystyle\propto\underbrace{\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\big{\\{}\log
p(x_{i})\big{\\}}}_{\text{Negative entropy term,
$-H_{p}(x_{1:n})$}}\underbrace{-\mathbb{E}_{p(x_{1:n})}\left\\{\frac{Tr(\Sigma)}{2}{-}\frac{\log|\Sigma|}{2}\right\\}}_{\text{Negative-
free term, $\mathcal{L}_{NF}(\Theta)$ (first part)}}$
$\displaystyle\qquad\underbrace{-\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\left\\{\frac{dist(x_{i},x_{i}^{\prime})}{2}\right\\}}_{\text{Negative-
free term, $\mathcal{L}_{NF}(\Theta)$ (second part)}}$ (4)

where $dist(x,x^{\prime})=\|enc(x)-enc(x^{\prime})\|^{2}$ and $|\Sigma|$
computes the determinant of matrix $\Sigma$. Importantly, the last row in Eq.
(4) is obtained by leveraging the known result about Kullback Leibler for
multivariate Gaussian distributions. Notably, the maximization of
$\mathcal{L}_{NF}(\Theta)$ promotes both decorrelated embedding features, as
the first two addends in $\mathcal{L}_{NF}(\Theta)$ (obtained from the first
KL term in Eq. (4)) force $\Sigma$ to become an identity matrix, as well as
representations that are invariant to data augmentations, thanks to the third
addend in $\mathcal{L}_{NF}(\Theta)$. It is important to mention that
$\mathcal{L}_{NF}(\Theta)$ in Eq. (4) recovers two recent negative-free
criteria, namely CorInfoMax (Ozsoy et al., 2022) and MEC (Liu et al., 2022).
We can also relate $\mathcal{L}_{NF}(\Theta)$ to other existing negative-free
approaches, including Barlow Twins (Zbontar et al., 2021), VicReg (Bardes et
al., 2022a, b) and W-MSE (Ermolov et al., 2021) (please refer to Section E in
the Supplementary Material for further details).

###  2.3 Discriminative/Cluster-Based SSL

Cluster-based SSL can be interpreted in probabilistic terms using the
graphical model in Fig. 2(b). In particular, we can define the following
conditional density:

$p(y_{i}|x_{i};\Theta)=\frac{e^{U_{:y_{i}}^{T}G_{:i}/\tau}}{\sum_{y}e^{U_{:y}^{T}G_{:i}/\tau}}$
(5)

where $U\in\mathbb{R}^{h\times c}$ is a matrix111We use subscripts to select
rows and columns. For instance, $U_{:y}$ identify $y-$th column of matrix $U$.
of $c$ cluster centers, $G=[g(x_{1}),\dots,g(x_{n})]$ is a matrix of
embeddings of size $h\times n$ and $\Theta=\\{\theta,U\\}$ is the set of
parameters, including the ones for the embedding function and the cluster
centers. The learning criterion can be obtained in a similar way to what we
have done previously for contrastive methods. In particular, we have that

$\displaystyle\mathbb{E}_{p(x_{1:n})}\\{\log$ $\displaystyle
p(x_{1:n};\Theta)\\}=-H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\log\sum_{y_{1:n}}p(y_{1:n}|x_{1:n};\Theta)\right\\}$
(6)
$\displaystyle=-H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\log\sum_{y_{1:n}}\frac{q(y_{1:n}|x_{1:n}^{\prime})}{q(y_{1:n}|x_{1:n}^{\prime})}p(y_{1:n}|x_{1:n};\Theta)\right\\}$
$\displaystyle\geq-
H_{p}(x_{1:n})-\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\\{KL(q(y_{1:n}|x_{1:n}^{\prime})\|p(y_{1:n}|x_{1:n}))\\}$
$\displaystyle=\underbrace{\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\big{\\{}\log
p(x_{i})\big{\\}}}_{\text{Negative entropy term, $-H_{p}(x_{1:n})$}}$
$\displaystyle\quad+\underbrace{\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\left\\{\mathbb{E}_{q(y_{i}|x_{i}^{\prime})}\log
p(y_{i}|x_{i};\Theta)+H_{q}(y_{i}|x_{i}^{\prime})\right\\}}_{\text{Discriminative
term $\mathcal{L}_{DI}(\Theta)$}}$ (7)

where $q(y_{i}|x_{i}^{\prime})$ is an auxiliary distribution. Notably,
maximizing the discriminative term is equivalent to minimize the $KL$
divergence between the two predictive distributions $p(y_{i}|x_{i})$ and
$q(y_{i}|x_{i}^{\prime})$, thus learning to predict similar category for both
sample $x_{i}$ and its augmented version $x_{i}^{\prime}$, obtained through
$\mathcal{T}$. Importantly, we can relate the criterion in Eq. (7) to the
objective used in optimal transport (Cuturi, 2013), by substituting Eq. (5)
into Eq. (7) and adopting a matrix format, as shown in the following equation:

$\displaystyle\mathcal{L}_{DI}(\Theta)$
$\displaystyle=\frac{1}{\tau}\bigg{\\{}\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{i}^{\prime}|x_{i})}\\{Tr(QU^{T}G)\\}+\tau\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{i}^{\prime}|x_{i})}\\{H_{Q}(y_{1:n}|x_{1:n}^{\prime})\\}\bigg{\\}}$
(8)

where $Q=[q(y_{1}|x_{1}^{\prime}),\dots,q(y_{n}|x_{n}^{\prime})]^{T}$ is a
prediction matrix of size $n\times c$ and $Tr(A)$ is the trace of an arbitrary
matrix $A$. Note that a naive maximization of $\mathcal{L}_{DI}(\Theta)$ can
lead to obtain trivial solutions like the one corresponding to uniformative
predictions, namely
$q(y_{i}|x_{i}^{\prime})=p_{\gamma}(y_{i}|x_{i})=\text{Uniform}(\\{1,\dots,c\\})$
for all $i=1,\dots,n$. Fortunately, the problem can be avoided and solved
exactly using the Sinkhorn-Knopp algorithm, which alternates between
maximizing $\mathcal{L}_{DI}(\Theta)$ in Eq. (8) with respect to $Q$ and with
respect to $\Theta$, respectively. This is indeed the procedure used in
several cluster-based SSL approaches, like DeepCluster (Caron et al., 2018),
SeLA (Asano et al., 2020) and SwAV (Caron et al., 2020), to name a few. As we
will see later on, we can obtain a different lower bound for Eq. (6) which
doesn’t require asymmetries, in the form of using a clustering procedure on
the embedding representation, stopping gradients or momentum encoders, while
being guaranteed to avoid important failure modes/trivial solutions.

##  3 A Bayesian Integration of SSL and Likelihood-Based Generative Models

In all three classes of SSL approaches (see Eqs. (2), (4) and (7)), the
expected data log-likelihood can be lower bounded by the sum of two
contribution terms, namely a negative entropy $-H_{p}(x_{1:n})$ and a
conditional log-likelihood term, chosen from
$\mathcal{L}_{CT}(\Theta),\mathcal{L}_{DI}(\Theta)$ and
$\mathcal{L}_{NF}(\Theta)$. A connection to generative models emerges by
additionally lower bounding the negative entropy term, namely:

$\displaystyle-H_{p}(x_{1:n})$ $\displaystyle=\mathbb{E}_{p(x_{1:n})}\\{\log
p(x_{1:n})\\}$ $\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\\{\log
p(x_{i})\\}$ $\displaystyle=\sum_{i=1}^{n}\left[\mathbb{E}_{p(x_{i})}\\{\log
p_{\Psi}(x_{i})\\}+KL(p(x_{i})\|p_{\Psi}(x_{i}))\right]$
$\displaystyle\geq\underbrace{\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\\{\log
p_{\Psi}(x_{i})\\}}_{\text{$-CE(p,p_{\Psi})$}}$ (9)

where $p_{\Psi}(x)$ is a generative model parameterized by $\Psi$. Notably,
the relation in (9) can be substituted in any of the objectives previously
derived for the different SSL classes, thus allowing to integrate generative
and SSL models into a single Bayesian framework. It is important to mention
that much can be gained by synergistically optimizing the resulting hybrid
objectives. For instance, the recent work EBCLR (Kim and Ye, 2022) proposes a
specific instantiation of this general idea, by combining energy-based models
with contrastive SSL approaches and showcasing the improved discriminative
performance. EBCLR can be obtained by observing that the conditional density
in Eq. (1) can be decomposed into joint and marginal densities (similarly to
what is done in (Grathwohl et al., 2020) for a supervised classifier):

$\displaystyle p(\ell,x;\Theta)$
$\displaystyle=\frac{e^{sim(g(x_{\ell}),g(x))/\tau}}{\Gamma(\Theta)}$
$\displaystyle p(x;\Theta)$
$\displaystyle=\frac{\sum_{j=1}^{n}e^{sim(g(x_{\ell_{j}}),g(x))/\tau}}{\Gamma(\Theta)}=\frac{e^{-E(x;\Theta)}}{\Gamma(\Theta)}$
(10)

where $E(x,\Theta)=-\log\sum_{j=1}^{n}e^{sim(g(x_{\ell_{j}}),g(x))/\tau}$
defines the energy score of the marginal density. By choosing
$p_{\Psi}(x)=p(x;\Theta)$ and $sim(z,z^{\prime})=-\|z-z^{\prime}\|^{2}$ in Eq.
(10), one recovers the exact formulation of EBCLR (Kim and Ye, 2022).
Moreover, this integrated framework is not bound to the specific family of
energy-based models, rather one could in principle specify any likelihood-
based model (e.g. VAE, diffusion model, normalizing flow) in Eq. (9),
consequently giving rise to a wide spectrum of possible hybrid solutions.
Indeed, some recent works have started to propose solution aiming towards the
joint integration of VAEs and SSL (Gatopoulos and Tomczak, 2020; Zhue et al.,
2020; Wu et al., 2023; Nakamura et al., 2023).

In the subsequent sections, we are going to focus our analysis on cluster-
based SSL and energy-based models. We provide a unified theory allowing to
learn a backbone classifier network to jointly exploit its generative and
discriminative properties.

##  4 A Bayesian Unification of Self-Supervised Clustering and Energy-Based
Models

Instead of optimizing the ELBO lower bound in Eq. (7), which includes several
trivial solutions and requires to introduce heuristics at the level of the
architecture (Caron et al., 2018, 2020, 2021), we take on a different approach
and devise a new lower bound starting from Eq. (6),viz. the GEnerative and
DIscriminative (GEDI) Lower Bound, that is guaranteed to avoid the main
failure modes and thus avoids architectural asymmetries.

###  4.1 Lower Bound for Cluster-Based SSL

We state the following proposition and leave the proof to Section F in the
Supplementary Material.

###### Proposition 1

Eq. (6) can be lower bounded by the following quantity:

$\displaystyle-H_{p}(x_{1:n})$
$\displaystyle\underbrace{-\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\left\\{CE(p(y_{i}|x_{i}^{\prime};\Theta),p(y_{i}|x_{i};\Theta))\right\\}}_{\text{$\mathcal{L}_{INV}(\Theta)$}}$
$\displaystyle\underbrace{-\sum_{i=1}^{n}CE\left(p(y_{i}),\frac{1}{n}\sum_{j=1}^{n}p(y_{j}=y_{i}|x_{j};\Theta)\right)}_{\text{$\mathcal{L}_{PRIOR}(\Theta)$}}$
(11)

Additionally, the corresponding maximum value for the last two addends in Eq.
(11) is given by the following inequality:222Here, we assume that the
predictive model $p(y|x;\Theta)$ has enough capacity to achieve the optimal
solution.

$\displaystyle\mathcal{L}_{INV}(\Theta)+\mathcal{L}_{PRIOR}(\Theta)\leq$
$\displaystyle-H_{p}(y_{1:n})$ (12)

The above proposition has interesting implications. First of all, by
maximizing the discriminative term $\mathcal{L}_{INV}(\Theta)$ with respect to
$\Theta$, we enforce two properties, namely: (i) label invariance, as we
ensure that the predictive distributions of the discriminative model for a
sample and its augmented version match each other and (ii) confident
predictions, as maximizing the cross-entropy forces also to decrease the
entropy of these distributions.333Indeed, recall that
$CE(p,q)=H_{p}+KL(p\|q)$. Therefore, maximizing $-CE(p,q)$ forces to have both
$KL(p\|q)=0$ and $H_{p}=0$. Secondly, by choosing a uniform prior, viz.
$p(y_{i})=\text{Uniform}(\\{1,\dots,c\\})$, and by maximizing
$\mathcal{L}_{PRIOR}(\Theta)$ with respect to $\Theta$, we ensure to obtain a
balanced cluster assignment. This is also commonly done by approaches based on
optimal transport objectives and corresponding surrogates (Cuturi, 2013; Caron
et al., 2018, 2020; Amrani et al., 2022). Finally, the proposed lower bound
allows for an important key difference over existing cluster-based SSL, as we
don’t need to introduce asymmetries in the discriminative backbones. Indeed,
by observing Eq. (7), we note that cluster-based SSL, specifically SwAV,
assume $p(y|x;\Theta)=\text{Softmax}(U^{T}g(x)/\tau)$ and
$q(y|x^{\prime})=\text{Sinkhorn}(\text{StopGrad}(U^{T}g(x^{\prime})/\tau))$,
where Sinkhorn and StopGrad are two operators performing the Sinkhorn-Knopp
algorithm and stopping the gradients, respectively. However, in our case, we
require that $q(y|x)=p(y|x;\Theta)=\text{Softmax}(f(enc(x))/\tau)$, where
$f:\mathbb{R}^{h}\rightarrow\mathbb{R}^{c}$ is a discriminative network head,
thus simplifying the overall training.

###  4.2 GEDI Lower Bound

For the GEDI instantiation, we consider the graphical model in Fig. 2(d) and
we derive the lower bound on the expected log-likelihood by exploiting the
bound in Eq. (9) and the one in Proposition 1:

$\displaystyle\mathbb{E}_{p(x_{1:n})}\\{\log p(x_{1:n};\Theta)\\}\geq$
$\displaystyle\underbrace{\mathcal{L}_{GEN}(\Psi)}_{\text{Generative term
$-CE(p,p_{\Psi})$}}+\underbrace{\mathcal{L}_{INV}(\Theta)+\mathcal{L}_{PRIOR}(\Theta)}_{\text{Discriminative
terms}}$ (13)

additionally we decompose the discriminative model $p(y|x;\Theta)$ to obtain
$p_{\Psi}$ in a similar manner to what we have already done in Eq. (10),
namely:

$\displaystyle p(y,x;\Theta)$
$\displaystyle=\frac{e^{f_{y}(enc(x))/\tau}}{\Gamma(\Theta)}$ $\displaystyle
p(x;\Psi)$
$\displaystyle\underbrace{=}_{\text{$\Psi=\Theta$}}p(x;\Theta)=\frac{\sum_{y=1}^{c}e^{f_{y}(enc(x))/\tau}}{\Gamma(\Theta)}=\frac{e^{-E(x;\Theta)}}{\Gamma(\Theta)}$
(14)

where $E(x;\Theta)=-\log\sum_{y=1}^{c}e^{f_{y}(enc(x))/\tau}$. We will shortly
analyze the properties of the different objective terms in Eq. (13). For the
moment, we finalize the GEDI instantiation by devising the corresponding
training algorithm.

Learning a GEDI model. We can train the GEDI model by jointly maximizing the
objective in Eq. (13) with respect to the parameters $\Theta$ through
gradient-based strategies. The overall gradient includes the summation of
three terms, viz. $-\nabla_{\Theta}CE(p,p_{\Theta})$,
$\nabla_{\Theta}\mathcal{L}_{INV}(\Theta)$ and
$\nabla_{\Theta}\mathcal{L}_{PRIOR}(\Theta)$. While the last two gradient
terms can be computed easily by leveraging automatic differentiation, the
first one must be computed by exploiting the following identities (obtained by
simply substituting Eq. (14) into the definition of cross-entropy and
expanding $\nabla_{\Theta}\Gamma(\Theta)$):

$\displaystyle-\nabla_{\Theta}CE(p,p_{\Theta})$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\left\\{\nabla_{\Theta}\log\sum_{y=1}^{c}e^{f_{y}(enc(x_{i}))/\tau}\right\\}-n\nabla_{\Theta}\log\Gamma(\Theta)$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\left\\{\nabla_{\Theta}\log\sum_{y=1}^{c}e^{f_{y}(enc(x_{i}))/\tau}\right\\}n\mathbb{E}_{p_{\Theta}(x)}\left\\{\nabla_{\Theta}\log\sum_{y=1}^{c}e^{f_{y}(enc(x))/\tau}\right\\}$
(15)

Importantly, the first and the second expectations in Eq. (15) are estimated
using the training and the generated data, respectively. To generate data from
$p_{\Theta}$, we use a sampler based on Stochastic Gradient Langevin Dynamics
(SGLD), thus following recent best practices to train energy-based models (Xie
et al., 2016; Nijkamp et al., 2019; Du and Mordatch, 2019; Nijkamp et al.,
2020).

Input: $x_{1:n}$, $x_{1:n}^{\prime}$, Iters, SGLD and Adam optimizer
hyperparameters;

Output: Trained model $\Theta$;

For $\text{iter}=1,\dots,\text{Iters}$;

Generate samples from $p_{\Theta}$ using SGLD;

Estimate $\Delta_{1}\Theta=\nabla_{\Theta}\mathcal{L}_{GEN}(\Theta)$ using Eq.
(15);

Compute $\Delta_{2}\Theta=\nabla_{\Theta}\mathcal{L}_{INV}(\Theta)$;

Compute $\Delta_{3}\Theta=\nabla_{\Theta}\mathcal{L}_{PRIOR}(\Theta)$ ;

$\Delta\Theta\leftarrow\sum_{i=1}^{3}\Delta_{i}\Theta$;

$\Theta\leftarrow\text{Adam}$ maximizing using $\Delta\Theta$;

Return $\Theta$;

Algorithm 1 GEDI Training.

The whole learning procedure is summarized in Algorithm 1.

Computational requirements. When comparing our GEDI instantiation with
traditional SSL training, more specifically to SwAV, we observe two main
differences in terms of computation. Firstly, our learning algorithm does not
require to run the Sinkhorn-Knopp algorithm, thus saving computation.
Secondly, our GEDI instantiation requires additional forward and backward
passes to draw samples from the energy-based model $p_{\Theta}$. However, the
number of additional passes through the discriminative model can be limited by
the number of SGLD iterations, necessary to generate data (cf. Experiments).

###  4.3 Analysis of Loss Landscape and the Triad of Failure Modes

Asymmetries have been playing an important role in self-supervised learning in
order to avoid trivial solutions/failure modes (Wang et al., 2022). Here, we
formalize three main failure modes for cluster-based SSL. Then, we study the
GEDI loss landscape and show that these undesired trivial solutions are not
admitted by our objective. This result holds without introducing asymmetries
in the optimization procedure and/or network architecture.

Let’s start by defining the most important failure modes, namely:

######  Definition 2 (Failure Mode 1 - Representational Collapse)

There exists a constant vector $k\in\mathbb{R}^{h}$ such that for all
$x\in\mathbb{R}^{d}$, $enc(x)=k$.

######  Definition 3 (Failure Mode 2 - Cluster Collapse)

There exists a cluster $j\in\\{1,\dots,c\\}$ such that for all
$x\in\mathbb{R}^{d}$, $p(y=j|x;\Theta)=1$.

###### Definition 4

(Failure Mode 3 - Permutation Invariance and Inconsistency of Labels to Data
Augmentation) For all possible permutations
$\pi:\\{1,\dots,c\\}\rightarrow\\{1,\dots,c\\}$, a dataset
$\mathcal{D}=\\{(x_{i},t_{i},t_{i}^{\prime})\\}_{i=1}^{n}$, its permuted
version $\mathcal{D}^{\pi}=\\{(x_{i},t_{\pi(i)},t_{i}^{\prime})\\}_{i=1}^{n}$
and a loss $\mathcal{L}(\Theta;\cdot)$, evaluated at one of the two datasets,
we have that
$\mathcal{L}(\Theta;\mathcal{D})=\mathcal{L}(\Theta;\mathcal{D}^{\pi})$. For
GEDI, $t_{i}\doteq f(enc(x_{i}))$ and $t_{i}^{\prime}\doteq
f(enc(x_{i}^{\prime}))$.

In other words, Definition 1 considers the case where the encoder maps
(collapses) every input to the same output. Definition 2 considers the
situation where the predictive model assigns all samples to the same cluster
with high confidence. And Definition 3 considers the case where a hypothetical
adversary swaps the predictions made by the model on different pair of inputs.
Ideally, we would like to have an objective that does not admit these failure
modes.

Now, we state the properties of the loss landscape of GEDI with the following
theorem (we leave the proof to Section G in the Supplementary Material):

###### Theorem 5

Given definitions 1-3, the following statements tells for a particular loss,
which modes are admitted as optimal solutions:

  * a.

$\mathcal{L}_{GEN}(\Theta)$ admits failure modes 2 and 3.

  * b.

$\mathcal{L}_{INV}(\Theta)$ admits failure modes 1 and 2.

  * c.

$\mathcal{L}_{PRIOR}(\Theta)$ admits failure modes 1 and 3.

Importantly, Theorem 5 tells us that $\mathcal{L}_{GEN}(\Theta)$ can be used
to penalize representational collapse, $\mathcal{L}_{INV}(\Theta)$ can be used
to break the problem of permutation invariance for the cluster assignments,
while $\mathcal{L}_{PRIOR}(\Theta)$ can be used to penalize cluster collapse.
Consequently, by maximizing the objective in Eq. (13), we are guaranteed to
learn solutions which are non-trivial. A table summarizing all these
properties is given below.

Table 1: Summary of loss landscape Does $\downarrow$ penalize $\rightarrow$? | Repr. collapse | Clus. collapse | Perm. Inv.
---|---|---|---
$\mathcal{L}_{GEN}(\Theta)$ | Yes | No | No
$\mathcal{L}_{INV}(\Theta)$ | No | No | Yes
$\mathcal{L}_{PRIOR}(\Theta)$ | No | Yes | No
Eq. (13) | Yes | Yes | Yes

##  5 Related Work

We organize the related work according to different objective categories,
namely contrastive, cluster-based and non-contrastive self-supervised
approaches. Additionally, we discuss recent theoretical results, augmentation
strategies as well as connections to energy-based models. For an exhaustive
overview of self-supervised learning, we invite the interested reader to check
out two recent surveys (Jing and Tian, 2021; Balestriero et al., 2023).

Contrastive objectives and connection to mutual information. Contrastive
learning represents an important family of self-supervised learning
algorithms, which is rooted in the maximization of mutual information between
the data and its latent representation (Linsker, 1988; Becker and Hinton,
1992). Estimating and optimizing mutual information from samples is
notoriously difficult (McAllester and Stratos, 2020), especially when dealing
with high-dimensional data. Most recent approaches focus on devising
variational lower bounds on mutual information (Barber and Agakov, 2004).
Indeed, several popular objectives, like the mutual information neural
estimation (MINE) (Belghazi et al., 2018), deep InfoMAX (Hjelm et al., 2018),
noise contrastive estimation (InfoNCE) (den Oord et al., 2018; O. Henaff,
2020; Chen et al., 2020; Lee, 2022; Xu et al., 2022) to name a few, all belong
to the family of variational lower bounds (Poole et al., 2019). All these
estimators have different properties in terms of bias-variance trade-off
(Tschannen et al., 2019; Song and Ermon, 2020). Our work model contrastive
learning using an equivalent probabilistic graphical model and a corresponding
objective function based on the data log-likelihood, thus providing an
alternative view to the principle of mutual information maximization. This is
similar in spirit to the formulations proposed in the recent works of
(Mitrovic et al., 2021; Tomasev et al., 2022; Scherr et al., 2022; Xu et al.,
2022). Unlike these works, we are able to extend the log-likelihood
perspective to other families of self-supervised approaches and also to
highlight and exploit their connections to energy-based models.

Discriminative/Cluster-based objectives. There are also recent advances in
using clustering techniques in representation learning. For example,
DeepCluster (Caron et al., 2018) uses k-means and the produced cluster
assignments as pseudo-labels to learn the representation. The work in (Huang
et al., 2022) introduces an additional regularizer for deep clustering,
invariant to local perturbations applied to the augmented latent
representations. The work in (Asano et al., 2020) shows that the pseudo-label
assignment can be seen as an instance of the optimal transport problem. SwAV
(Caron et al., 2020) proposes to use the Sinkhorn-Knopp algorithm to optimize
the optimal transport objective (Cuturi, 2013) and to perform a soft cluster
assignment. Finally, contrastive clustering (Li et al., 2021) proposes to
minimize the optimal transport objective in a contrastive setting, leveraging
both positive and negative samples. Our work provides a simple yet concise
formulation of cluster-based self-supervised learning based on the principle
of likelihood maximization. Additionally, thanks to the connection with
energy-based models, we can perform implicit density estimation and leverage
the learnt information to improve the clustering performance.

Non-contrastive objectives. It’s important to mention that new self-supervised
objectives have recently emerged (Zbontar et al., 2021; Grill et al., 2020) as
a way to avoid using negative samples, which are typically required by
variational bounds on the mutual information, namely BYOL (Grill et al.,
2020), SimSiam (Chen et al., 2020), DINO (Caron et al., 2021), Zero-CL (Zhang
et al., 2021), W-MSE (Ermolov et al., 2021), Barlow Twins (Zbontar et al.,
2021), VICReg (Bardes et al., 2022a) and its variants (Bardes et al., 2022b),
MEC (Liu et al., 2022) and CorInfoMax (Ozsoy et al., 2022). DINO proposes to
maximize a cross-entropy objective to match the probabilistic predictions from
two augmented versions of the same image. BYOL, SimSiam, W-MSE consider the
cosine similarity between the embeddings obtained from the augmented pair of
images. Additionally, W-MSE introduces a hard constraint implemented as a
differentiable layer to whiten the embeddings. Similarly, Barlow Twins
proposes a soft whitening by minimizing the Frobenius norm between the cross-
correlation matrix of the embeddings and the identity matrix. Zero-CL pushes
forward the idea of whitening the features by also including an instance
decorrelation term. VICReg and its variant extend over Barlow Twins by
computing the sample covariance matrix, instead of the correlation one (thus
avoiding to use batch normalization), by enforcing an identity covariance and
by introducing an additional regularizer term to minimize the mean squared
error between the embeddings of the two networks and to promote the invariance
of the embeddings. Similarly, the work in (Tomasev et al., 2022) uses an
invariance loss function in conjunction to the contrastive InfoNCE objective.
MEC maximizes the log-determinant of the covariance matrix for the latent
representation, thus promoting maximum entropy under the Gaussian assumption.
Additionally, CorInfoMax extends MEC by introducing a term that enforces the
representation to be invariant under data augmentation. In Section 2.3, we can
cast the non-contrastive problem as a minimization of the Kullback Leibler
divergence between the latent posterior and a standard normal density prior.
In essence, our work allows to compactly represent the family of non-
contrastive methods using a likelihood-based objective.

Additional objectives. Several works have investigated the relation between
different families of self-supervised approaches leading to hybrid objective
functions (Kügelgen et al., 2021; Garrido et al., 2023). In contrast, our work
attempts to provide a unified view from a probabilistic perspective and to
highlight/exploit its connection to energy-based models.

The main idea of generative and discriminative training originally appeared in
the context of Bayesian mixture models (Sansone et al., 2016). Specifically,
the authors proposed to jointly learn a generative model and cluster data in
each class in order to be able to discover subgroups in breast cancer data.
Subsequently, the work in (Liu and Abbeel, 2020) pushed the idea forward and
apply it to a supervised deep learning setting. Instead, our work focuses on
self-supervised learning and generative models, thus avoiding the need for
additional supervision on the categorical variable $y$. Recently, the work in
(Li et al., 2022) uses the maximum coding rate criterion to jointly learn an
embedding and cluster it. However, the training proceeds in a multistage
fashion. In contrast, our work provides a simple formulation enabling to
jointly learn and cluster the embeddings in one shot. Additionally, we can
leverage the generative perspective to further boost the self-supervised
learning performance, as demonstrated by the generation and out-of-
distribution detection experiments.

Theory of self-supervised learning. Several works have theoretically analysed
self-supervised approaches, both for contrastive (Saunshi et al., 2019; Wang
and Isola, 2020; Zimmermann et al., 2021; Tosh et al., 2021; HaoChen et al.,
2021; Saunshi et al., 2022) and non-contrastive methods (Tian et al., 2021;
Kang-Jun et al., 2022; Weng et al., 2022; Wen and Li, 2022), to motivate the
reasons for their successes, identify the main underlying principles and
subsequently provide more principled/simplified solutions. Regarding the
former family of approaches, researchers have (i) identified key properties,
such as representation alignment (i.e. feature for positive pairs need to be
close to each other) and uniformity (to avoid both representational and
dimensional collapse) (Wang and Isola, 2020; Assran, Mahmoud and Caron,
Mathilde and Misra, Ishan and Bojanowski, Piotr and Bordes, Florian and
Vincent, Pascal and Joulin, Armand and Rabbat, Mike and Ballas, Nicolas, 2022;
Assran et al., 2023), (ii) formulated and analyzed the problem using data
augmentation graphs (HaoChen et al., 2021) and (iii) examined generalization
bounds on the downstream supervised performance (Saunshi et al., 2022; Bao et
al., 2022). Regarding the latter family of approaches, the main focus has been
devoted to understanding the reasons on why non-contrastive approaches avoid
trivial solutions. In this regard, asymmetries, in the form of stop-gradient
and diversified predictors, are sufficient to ensure well-behaved training
dynamics (Tian et al., 2021; Weng et al., 2022; Wen and Li, 2022).
Importantly, the asymmetries are shown to implicitly constrain the
optimization during training towards solutions with decorrelated features
(Kang-Jun et al., 2022). Recent works have also looked at identifying
connections between contrastive and non-contrastive methods (Dubois et al.,
2022; Garrido et al., 2023; Balestriero and LeCun, 2022) to unify the two
families. The work in (Dubois et al., 2022) proposes a set of desiderata for
representation learning, including large dimensional representations,
invariance to data augmentations and the use of at least one linear predictor
to ensure good performance on linear probe evaluation tasks. The work in
(Garrido et al., 2023) analyzes the relations between contrastive and non-
contrastive methods, showing their similarities and differences. Both families
of approaches learn in a contrastive manner. However, while contrastive
solutions learn by contrasting between samples, non-contrastive ones focus on
contrasting between the dimensions of the embeddings. The work in (Balestriero
and LeCun, 2022) studies the minima in terms of representations for the
different loss functions proposed in the two families. The authors are able to
show that such minima are equivalent to solutions achieved by spectral
methods. This provides additional evidence on the similarities between the
approaches and the possibility for their integration.

Our work provides a unifying view of the different classes, allowing to derive
several loss functions in a principled manner using variational inference on
the data log-likelihood. Additionally, we provide conditions to learn in a
principled manner, thus avoiding trivial solutions and the use of asymmetries.

Generative models and self-supervised learning. Recently, works have
considered synergies between self-supervised and energy-based models (LeCun,
2022) for the purposes of out-of-distribution detection (Hendrycks et al.,
2019; Winkens et al., 2020; Mohseni et al., 2020). This is a common
characteristics of energy-based models and indeed our work highlight the
explicit connection with self-supervised learning. To the best of our
knowledge, there is only one recent work exploring the integration between
self-supervised learning approaches and energy models (Kim and Ye, 2022). The
authors propose to use an energy-based model to learn a joint distribution
over the two augmented views for the same data. The resulting objective can be
decomposed into a conditional distribution term, leading to a contrastive
learning criterion, and a marginal distribution term, leading to an energy-
based model criterion. Therefore, the work only considers the integration
between contrastive methods and energy-based models. In contrast, our work
goes a step forward by showing a general methodology to integrate generative
and SSL approaches. Moreover, we provide a new lower bound for the class of
cluster-based approaches that guarantees the avoidance of important failure
modes.

There has been also recent interest in integrating VAEs with self-supervised
learning (Gatopoulos and Tomczak, 2020; Zhue et al., 2020; Wu et al., 2023;
Nakamura et al., 2023). Our formulation is general enough to encompass ELBO-
like objectives like the one used in VAEs. IInstead of instantiating the
generative term leveraging an energy-based model, one could proceed to lower
bound the entropy term following traditional ways to derive an ELBO. While
this is a certainly possible direction, we are not interested to pursue it as
we aim at devising an objective function which can exploit a simple classifier
architecture without the need of additional components, such as a decoder
network.

##  6 Experiments

We perform experiments to evaluate the generative/discriminative performance
of GEDI and its competitors, namely an energy-based model JEM (Grathwohl et
al., 2020), which is trained with persistent contrastive divergence Tieleman
(2008) to optimize only the generative term of the GEDI Lower Bound (Eq.
(13)), and 2 self-supervised baselines, viz. a negative-free approach based on
Barlow Twins (Zbontar et al., 2021) and a cluster-based approach based on SwAV
(Caron et al., 2020), which optimize the objectives in Eq. 4 and Eq. 8,
respectively. The whole analysis is divided into four main experimental
settings, the first part provides empirical validation and intuition on the
results of Proposition 1 and Theorem 5, the second part based on two synthetic
datasets, including moons and circles, compares GEDI against the above-
mentioned baselines, the third part extends the comparisons to real-world
data, including SVHN, CIFAR-10 and CIFAR-100, and finally the last part
showcases the utility of the GEDI framework to mitigate the reasoning shortcut
problem arising in neuro-symbolic learning on MNIST data (Sansone and
Manhaeve, 2023a, b; Marconato et al., 2023). We use existing code both as a
basis to build our solution and also to run the experiments for the different
baselines. In particular, we use the code from (Duvenaud et al., 2021) for
training energy-based models and the repository from (da Costa et al., 2022)
for all self-supervised approaches. Our code will be publicly released in its
entirety upon acceptance. Implementation details as well as additional
experiments are reported in the Supplementary Material.

###  6.1 Part 1: Empirical Validation of the Theory

Table 2: Clustering performance in terms of normalized mutual information
(NMI) on test set (moons and circles). Higher values indicate better
clustering performance. Mean and standard deviations are computed from 5
different runs.

Dataset | JEM | GEDI no prior | GEDI no inv | GEDI no gen | GEDI
---|---|---|---|---|---
Moons | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.11$\pm$0.15 | 0.98$\pm$0.00 | 0.94$\pm$0.07
Circles | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.22$\pm$0.13 | 0.83$\pm$0.12 | 1.00$\pm$0.01

(a) GT

(b) JEM

(c) no prior

(d) no inv

(e) no gen

(f) GEDI

(g) GT

(h) JEM

(i) no prior

(j) no inv

(k) no gen

(l) GEDI

Figure 3: Qualitative visualization of the clustering performance for the
different strategies on moons (a-f) and on circles (g-l) datasets. Colors
identify different cluster predictions. Only GEDI and GEDI no gen are able to
perform well on both datasets.

We consider two well-known synthetic datasets, namely moons and circles. We
use a multi-layer perceptron (MLP) with two hidden layers (100 neurons each)
for $enc$ and one with a single hidden layer (4 neurons) for $proj$, we choose
$h=2$ and $\mathcal{T}(x^{\prime}|x)=N(0,\sigma^{2}I)$ with $\sigma=0.03$ as
our data augmentation strategy. We train GEDI for $7k$ iterations using Adam
optimizer with learning rate $1e-3$. Similarly we ablate the contribution of
the different loss terms by training the GEDI Lower Bound using only
$\mathcal{L}_{GEN}$ (equivalent to JEM), without $\mathcal{L}_{INV}$ (called
GEDI no inv) without $\mathcal{L}_{PRIOR}$ (called GEDI no prior). Further
details about the hyperparameters are available in the Supplementary Material
(Section G). We evaluate the clustering performance both quantitatively, by
using the Normalized Mutual Information (NMI) score and qualitatively, by
visualizing the cluster assignments using different colors.

(a) no prior

(b) no inv

(c) GEDI

(d) no prior

(e) no inv

(f) GEDI

Figure 4: Visualization of the main training losses. As suggested by Eq. 12 in
Proposition 1, we can always debug whether the main failure modes in the triad
have occurred by checking the minimum value achieved by the losses and compare
it with its corresponding global value. Indeed, the global minimum value of
each loss is known, that is $\ln(c)$ for $\mathcal{L}_{PRIOR}$ and $0$ for
$\mathcal{L}_{INV}$. In both moons (a-c) and circles (d-f) the values are
$\ln(2)\approx 0.69$ and $0$, respectively.

From the results in Table 2 and Figure 3, we can make the following
observations: (i) GEDI no prior and JEM are subject to cluster collapse on
both datasets. This is expected as failure mode 2 is not penalized during
training due to the omission of $\mathcal{L}_{PRIOR}$; (ii) GEDI no inv is
subject to the problem of permutation invariance to cluster assignments.
Consequently, the obtained cluster labels are not informative and consistent
with the underlying manifold structure of the data distribution. Again, this
confirms the result of Theorem 5, as failure mode 3 could be avoided by the
use of $\mathcal{L}_{INV}$; (iii) GEDI no gen achieves competitive performance
to GEDI despite the absence of $\mathcal{L}_{GEN}$. While in theory the
objective function for this approach admits representation collapse, as
predicted by our Theorem, in practice we never observed such issue. It might
be the case that the learning dynamics of gradient-based optimisation are
enough to avoid the convergence to this trivial solution. Finally (iv) GEDI is
guaranteed to avoid the most important failure modes and therefore solve both
tasks. We will see later the benefits of including also the generative term in
the optimization.

An important consequence of Proposition 1 is that we can use the training
losses to debug whether we achieve the global minimum value and therefore
avoid the triad of failure modes. Figure 4 showcases this property.

###  6.2 Part 2: Generative/Discriminative Comparisons on Synthetic Data

Table 3: Clustering performance in terms of normalized mutual information
(NMI) on test set (moons and circles). Higher values indicate better
clustering performance. Mean and standard deviations are computed from 5
different runs.

Dataset | JEM | Barlow | SwAV | GEDI no gen | GEDI | Gain
---|---|---|---|---|---|---
Moons | 0.00$\pm$0.00 | 0.22$\pm$0.10 | 0.76$\pm$0.36 | 0.98$\pm$0.00 | 0.94$\pm$0.07 | +0.22
Circles | 0.00$\pm$0.00 | 0.13$\pm$0.10 | 0.00$\pm$0.00 | 0.83$\pm$0.12 | 1.00$\pm$0.01 | +0.87

(a) JEM

(b) Barlow

(c) SwAV

(d) no gen

(e) GEDI

(f) JEM

(g) Barlow

(h) SwAV

(i) no gen

(j) GEDI

(k) JEM

(l) Barlow

(m) SwAV

(n) no gen

(o) GEDI

(p) JEM

(q) Barlow

(r) SwAV

(s) no gen

(t) GEDI

Figure 5: Qualitative visualization of the clustering performance for the
different strategies on moons (a-e) and on circles (f-j) datasets. Colors
identify different cluster predictions. Only GEDI and GEDI no gen are able to
perform well on both datasets. Qualitative visualization of the generative
performance for the different strategies on moons (k-o) and on circles (p-t)
datasets. Colors identify different cluster predictions. All GEDI approaches
(except for no gen perform comparably well to the generative solution JEM).

We consider the same experimental setting used in part 1 and train JEM, SwAV
and GEDI for $7k$ iterations using Adam optimizer with learning rate $1e-3$.
Further details about the hyperparameters are available in the Supplementary
Material (Section G). We evaluate the clustering performance both
qualitatively, by visualizing the cluster assignments using different colors,
and quantitatively, by using the Normalized Mutual Information (NMI) score.
Additionally, we qualitatively demonstrate the generative performance of all
approaches.

We report all quantitative performance in Table 3. As expected, Barlow fails
to solve both tasks due to the different nature to cluster-based approaches.
SwAV correctly solves the task for the moons dataset, while it faces
difficulty to solve the one based on circles. This is due to the fact that the
optimized bound in Eq. (8) includes several trivial solutions. The introduced
asymmetries (such as stop gradient and the clustering layers) are not enough
to rule them out. This is confirmed by visually inspecting the latent
representation and observing that the encoder collapses to the identity
function. Therefore, the projector head can solve the task by simply learning
a linear separator. As a consequence, we observe that half of the labels are
permuted across the two manifolds (refer to Figure 5). In contrast, GEDI can
recover the true clusters in both datasets without any additional asymmetry
and have a guarantee to avoid the triad of failure modes. Moreover, GEDI is
able to learn a correct density estimator compared to the SSL baselines (cf.
Figure 5). This is an important property that can come in handy for out-of-
distribution detection, as we will see later in the real-world experiments.

###  6.3 Part 3: Real-world Experiments on SVHN, CIFAR-10, CIFAR-100

Table 4: Generative and discriminative performance on test set (SVHN,
CIFAR-10, CIFAR-100). Normalized mutual information (NMI) and Frechet
Inception Distance (FID) are used as evaluation metrics for the discriminative
and generative tasks, respectively. Higher values of NMI and lower values of
FID indicate better performance. Mean and standard deviations are reported for
5 different initialization seeds.

Task | Method | SVHN | CIFAR-10 | CIFAR-100
---|---|---|---|---
Discriminative | JEM | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
Barlow | 0.20$\pm$0.02 | 0.17$\pm$0.04 | 0.61$\pm$0.05
SwAV | 0.21$\pm$0.01 | 0.44$\pm$0.01 | 0.51$\pm$0.21
GEDI two stage | 0.22$\pm$0.01 | 0.44$\pm$0.01 | 0.71$\pm$0.01
(NMI) | GEDI no gen | 0.27$\pm$0.04 | 0.45$\pm$0.00 | 0.87$\pm$0.00
| GEDI | 0.25$\pm$0.04 | 0.44$\pm$0.01 | 0.87$\pm$0.00
Generative | JEM | 201$\pm$36 | 223$\pm$15 | 271$\pm$85
Barlow | 334$\pm$28 | 382$\pm$22 | 403$\pm$28
SwAV | 480$\pm$67 | 410$\pm$31 | 420$\pm$26
GEDI two stage | 189$\pm$8 | 266$\pm$17 | 243$\pm$28
(FID) | GEDI no gen | 488$\pm$43 | 403$\pm$9 | 435$\pm$11
| GEDI | 193$\pm$10 | 214$\pm$13 | 226$\pm$9

Table 5: OOD detection in terms of AUROC on test set (CIFAR-10, CIFAR-100).
Training is performed on SVHN.

Dataset | JEM | Barlow | SwAV | GEDI two stage | GEDI no gen | GEDI
---|---|---|---|---|---|---
CIFAR-10 | 0.73 | 0.17 | 0.26 | 0.94 | 0.1 | 0.80
CIFAR-100 | 0.72 | 0.24 | 0.32 | 0.93 | 0.15 | 0.80

(a) Barlow

(b) SwAV

(c) GEDI no gen

(d) GEDI

Figure 6: Samples generated by the different discriminative strategies on
CIFAR-10 running Stochastic Langevin Dynamics for 500 iterations.

We consider three well-known computer vision benchmarks, namely SVHN, CIFAR-10
and CIFAR-100. We use a simple 8-layer Resnet network for the backbone encoder
for both SVHN and CIFAR-10 (around 1M parameters) and increase the hidden
layer size for CIFAR-100 (around 4.1M parameters) as from (Duvenaud et al.,
2021). We use a MLP with a single hidden layer for $proj$ (the number of
hidden neurons is double the number of inputs), we choose $h=256$ for
CIFAR-100 and $h=128$ for all other cases. Additionally, we use data
augmentation strategies commonly used in the SSL literature, including color
jitter, and gray scale conversion to name a few. We train JEM, Barlow, SwAV,
GEDI no gen and GEDI using Adam optimizer with learning rate $1e-4$ and batch
size $64$ for $20$, $200$ and $200$ epochs for each respective dataset (SVHN,
CIFAR-10 AND CIFAR-100). Further details about the hyperparameters are
available in the Supplementary Material (Section I). Moreover, we compare
against the two stage approach proposed in (Sansone and Manhaeve, 2022) and
refer to it as GEDI two stage. Similarly to the toy experiments, we evaluate
the clustering performance by using the Normalized Mutual Information (NMI)
score. Additionally, we evaluate the generative performance qualitatively
using the Frechet Inception Distance (Heusel et al., 2017) as well as the OOD
detection capabilities following the methodology in (Grathwohl et al., 2020).

From Table 4, we observe that GEDI is able to outperform all other competitors
by a large margin, thanks to the properties of both generative and self-
supervised models. We observe that the difference gap in clustering
performance increases with CIFAR100. This is due to a larger size of the
backbone used in the CIFAR100 experiments (cf. the size of the latent
representation). In terms of generation performance, GEDI is the only approach
that compares favorably with JEM. We provide a qualitative set of samples
generated by the different discriminative models in Figure 6. Last but not
least, we investigate the OOD detection capabilities of the different methods.
Table 11 provides a quantitative summary of the performance for a subset of
experiments (the complete set is available in Section J). We observe that GEDI
is more robust compared to other discriminative baselines, thanks to its
generative nature. Overall, these experiments provide real-world evidence on
the benefits of the proposed unification and theoretical results.

###  6.4 Part 4: Mitigating the Reasoning Shortcut Problem in Neuro-Symbolic
Learning

For the final task, we consider applying the self-supervised learning approach
to a neuro-symbolic (NeSy) setting. For this, we borrow an experiment from
DeepProbLog (Manhaeve et al., 2018, 2021). In this task, each example consists
of a three MNIST images such that the value of the last one is the sum of the
first two, e.g.
$\vbox{\hbox{\includegraphics[height=9.0pt]{img/MNIST/3.png}}}+\vbox{\hbox{\includegraphics[height=9.0pt]{img/MNIST/5.png}}}=\vbox{\hbox{\includegraphics[height=9.0pt]{img/MNIST/8.png}}}$.
This can thus be considered a minimal neuro-symbolic tasks, as it requires a
minimal reasoning task (a single addition) on top of the image classification
task. This task only contains positive examples. To solve this task, we
optimize $\mathcal{L}_{NESY}$ instead of $\mathcal{L}_{INV}$, as the NeSy loss
also forces a clustering of the digits, but is more informed. To calculate
$\mathcal{L}_{NESY}$, we use GEDI to classify the images to produce a
probability distribution over the classes $0$ to $9$, and we use the inference
mechanism from DeepProbLog to calculate the probability that the sum holds.
For this setting, this coincides with the Semantic Loss function (Xu et al.,
2018).

The NeSy constraint loss is defined as:

$\mathcal{L}_{NESY}(\Theta)=CE(1,p(c|\Theta))$

The probability of the constraint $p(c|\Theta)$ is defined in terms of a set
of rules $\mathcal{R}$ and a set of probabilistic facts
$\mathcal{F}=\\{f_{1}\ldots f_{n}\\}$ with their respective probabilities
$p_{1}\ldots p_{n}$. For this setting, there are 30 probabilistic facts, one
for each possible classifications of each digit. The probability of each
classification $p(y|x;\Theta)$ is thus parameterized by the neural network.
Each subset $F\subset\mathcal{F}$ defines a possible world
$w_{F}=\\{a\leavevmode\nobreak\ |\leavevmode\nobreak\ \mathcal{R}\cup F\models
a\\}$, i.e. the set of all atoms $a$ entailed by the rules and the facts in
$F$. In this setting, this is a total assignment of classes to digits, and the
truth value of the constraint. The probability of the constraint is the sum of
the probability of each possible world (i.e. classifications of the digits) in
which the constraint holds.

$P(c|\Theta))=\sum_{F\leavevmode\nobreak\ |\leavevmode\nobreak\ c\in
w_{F}}P(w_{F}|\Theta))$ (16)

Where the probability of the possible world is defined as the product of the
probability $p$ of each fact that is true in this possible word, and $1-p$ of
each fact that does not hold in the possible world.

$P(w_{F}|\Theta))=\prod_{f_{i}\in
F}p_{i}^{\Theta}\prod_{f_{i}\in\mathcal{F}\setminus
F}\left(1-p_{i}^{\Theta}\right)$ (17)

It is a specifically interesting use case for representation learning, since
when only the constraint probability is optimized, the neural network tends to
collapse onto the trivial solution of classifying each digit as a $0$, as
shown in (Manhaeve et al., 2018, 2021; Sansone and Manhaeve, 2023a, b). This
is a logically correct solution, but an undesirable solution. Optimizing the
uniformity objective should prevent this collapse. A neural network should be
able to correctly classify MNIST digits by using the SSL training objective
and the logical constraint. Here, the focus is on the small data regime, and
see whether the logical constraint is able to provide additional information.
Further details about the hyperparameters and experimental setup are available
in Appendix K. We evaluate the model by measuring the accuracy and NMI of the
ResNet model on the MNIST test dataset for different numbers of training
examples. The results are shown in Table 6. Here, the # examples indicates the
number of addition examples, which each have $3$ MNIST digits. The results
show that, without the NeSy constraint, the mean accuracy is low for all
settings. The NMI is higher, however, and increases as there is more data
available. This is expected, since the model is able to learn how to cluster
from the data, but unable to learn how to classify. By including the
constraint loss, the accuracy improves, as the model now has information on
which cluster belongs to which class. Furthermore, it also has a positive
effect on the NMI, as we have additional information on the clustering which
is used by the model. These results show us that the proposed method is
beneficial to learn to correctly recognize MNIST images using only a weakly-
supervised constraint. Furthermore, we show that the proposed method can
leverage the constraint to further improve NMI and classification accuracy.

Table 6: The median and standard deviation of the accuracy and NMI of GEDI on
the MNIST test set after training on the addition dataset, with and without
the NeSy constraint.

| Without constraint |  | With constraint
---|---|---|---
# Examples | Acc. | NMI |  | Acc. | NMI
100 | $0.08\pm 0.01$ | $0.22\pm 0.03$ |  | $0.42\pm 0.02$ | $0.35\pm 0.01$
1000 | $0.09\pm 0.03$ | $0.29\pm 0.06$ |  | $0.96\pm 0.01$ | $0.91\pm 0.01$
10000 | $0.11\pm 0.05$ | $0.44\pm 0.10$ |  | $0.97\pm 0.00$ | $0.93\pm 0.00$

##  7 Conclusions and Future Research

We have presented a unified perspective on self-supervised clustering and
energy-based models. The corresponding GEDI lower bound is guaranteed to
prevent three main failure modes: representation collapse, cluster collapse,
and permutation invariance to cluster assignments. By shedding new light on
the synergies between self-supervised learning and likelihood-based generative
models, we aim to inspire subsequent studies proposing new implementations of
our general methodology, such as novel approaches to integrating self-
supervised learning and latent variable models. Additionally, we demonstrate
that GEDI can better capture the underlying data manifolds and provide more
precise predictions compared to existing strategies. We believe that further
progress can be achieved by combining self-supervised learning with other
areas of mathematics, such as topology and differential geometry, for instance
to guarantee a notion of connectedness when dealing with the manifold
assumption. Moreover, we show that GEDI can be easily integrated into existing
statistical relational reasoning frameworks, opening the door to new neuro-
symbolic integrations and enabling the handling of low data regimes, which are
currently beyond the reach of existing self-supervised learning solutions.
There are several area of improvements and extension that we will target.
First and foremost, we plan in the near future to apply the existing solution
to larger scale datasets, such as Imagenet. Secondly, the current solution
assumes that information about the number of classes and the class prior is
known in advance. This might limit the applicability of the proposed solution
to real-world cases, such as open-world settings or scenarios with long tailed
distributions. It will be certainly interesting to generalize the framework to
deal with such new settings. Finally, it will be possible to extend the
proposed Bayesian framework towards more object-centric representation
learning, thus going beyond the traditional object classification problem.

##  Appendix A Alternative View of Contrastive SSL

Let us focus the analysis on a different graphical model from the one in
Section 2.1, involving an input vector $x$ and a latent embedding $z$ (cf.
Figure 1). Without loss of generality, we can discard the index $i$ and focus
on a single observation. We will later extend the analysis to the multi-sample
case. Now, consider the conditional distribution of $x$ given $z$ expressed in
the form of an energy-based model
$p(x|z;\Theta)=\frac{e^{f(x,z)}}{\Gamma(z;\Theta)}$, where
$f:\Omega\times\mathcal{S}^{h-1}\rightarrow\mathbb{R}$ is a score function and
$\Gamma(z;\Theta)=\int_{\Omega}e^{f(x,z)}dx$ is the normalizing factor.444We
assume that $f$ is a well-behaved function, such that the integral value
$\Gamma(z;\Theta)$ is finite for all $z\in\mathcal{S}^{h-1}$. Based on this
definition, we can obtain the following lower-bound on the data log-
likelihood:

$\displaystyle\mathbb{E}_{p(x)}\\{\log p(x)\\}$
$\displaystyle=KL(p(x)\|p(x;\Theta))+\mathbb{E}_{p(x)}\\{\log p(x;\Theta)\\}$
$\displaystyle\geq\mathbb{E}_{p(x)}\\{\log p(x;\Theta)\\}$
$\displaystyle\geq\mathbb{E}_{p(x)q(z|x)}\\{\log
p(x|z;\Theta)\\}-\mathbb{E}_{p(x)}\\{KL(q(z|x)\|p(z;\Theta))\\}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{\log
p(x|z;\Theta)\\}+\mathbb{E}_{p(x)q(z|x)}\\{\log p(z;\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{f(x,z)\\}+\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{p(z;\Theta)}{\Gamma(z;\Theta)}\bigg{\\}}$
$\displaystyle\doteq\text{ELBO}_{EBM}$ (18)

where $q(z|x)$ is an auxiliary density induced by a deterministic encoding
function $g:\Omega\rightarrow\mathcal{S}^{h-1}$.555$q(z|x)=\delta(z-g(x))$.
Eq. (18) provides the basic building block to derive variational bounds on
mutual information (Poole et al., 2019) as well as to obtain several popular
contrastive SSL objectives.

$\text{ELBO}_{EBM}$ and variational lower bounds on mutual information. Our
analysis is similar to the one proposed in (Alemi et al., 2018), as relating
$\text{ELBO}_{EBM}$ to the mutual information. However, while the work in
(Alemi et al., 2018) shows that the third line in Eq. (18) can be expressed as
a combination of an upper and a lower bound on the mutual information and it
studies its rate-distortion trade-off, our analysis considers only lower
bounds to mutual information and it makes an explicit connection to them.
Indeed, several contrastive objectives are based on lower bounds on the mutual
information between input and latent vectors (Poole et al., 2019). We can show
that $\text{ELBO}_{EBM}$ is equivalent to these lower bounds under specific
conditions for prior $p(z;\Theta)$ and score function $f$.

Specifically, given a uniform prior $p(z;\Theta)$ and $f(x,z)\doteq\log
p(x)+\tilde{f}(x,z)$ for all admissible pair $x,z$ and for some arbitrary
function $\tilde{f}$, we obtain the following equivalence (see Appendix B for
the derivation):

$\displaystyle\text{ELBO}_{EBM}$ $\displaystyle=\mathbb{E}_{p(x)}\\{\log
p(x)\\}+\mathbb{E}_{p(x)q(z|x)}\\{\tilde{f}(x,z)\\}$
$\displaystyle\quad-\mathbb{E}_{p(x)q(z|x)}\\{\log\mathbb{E}_{p(x^{\prime})}\\{e^{\tilde{f}(x^{\prime},z)}\\}\\}$
$\displaystyle=-H(X)+\mathcal{I}_{UBA}(X,Z)$ (19)

where $H(X)=-\mathbb{E}_{p(x)}\\{\log p(x)\\}$ and $\mathcal{I}_{UBA}(X,Z)$
refers to the popular Unnormalized Barber and Agakov bound on mutual
information. Importantly, other well-known bounds can be derived from
$\mathcal{I}_{UBA}(X,Z)$ (cf. (Poole et al., 2019) and Appendix B for further
details). Consequently, maximizing $\text{ELBO}_{EBM}$ with respect to the
parameters of $f$ (viz. $\theta$) is equivalent to maximize a lower bound on
mutual information.

$\text{ELBO}_{EBM}$ and InfoNCE (den Oord et al., 2018). Now, we are ready to
show the derivation of the popular InfoNCE objective. By specifying a non-
parametric prior666The term non-parametric refers to the fact that the samples
can be regarded as parameters of the prior and therefore their number
increases with the number of samples. Indeed, we have
$\Theta=\\{\theta;\\{x_{i}\\}_{i=1}^{n}\\}$.
$p(z;\Theta)=\frac{\frac{\Gamma(z;\Theta)}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}}}{\Gamma(\Theta)}$,
where
$\Gamma(\Theta)=\int\frac{\Gamma(z;\Theta)}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}}dz$,
we achieve the following equality (see Appendix C for the derivation):

$\displaystyle\text{ELBO}_{EBM}$
$\displaystyle=E_{\prod_{j=1}^{n}p(x_{j},z_{j})}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{f(x_{i},z_{i})}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z_{i})}}\bigg{\\}}-E_{\prod_{j=1}^{n}p(x_{j})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle\doteq
I_{NCE}(X,Z)-E_{\prod_{j=1}^{n}p(x_{j})}\\{\log\Gamma(\Theta)\\}$ (20)

Notably, $\text{ELBO}_{EBM}$ is equivalent to $I_{NCE}(X,Z)$ up to the term
$-\log\Gamma(\Theta)$. Maximizing $\text{ELBO}_{EBM}$ has the effect to
maximize $I_{NCE}(X,Z)$ and additionally to minimize $\Gamma(\Theta)$, thus
ensuring that the prior $p(z;\Theta)$ self-normalizes. However, in practice,
people only maximize the InfoNCE objective, disregarding the normalizing term.

##  Appendix B Connection to Unnormalized Barber Agakov Bound

Firstly, we recall the derivation of the Unnormalized Barber Agakov bound
(Poole et al., 2019) for the mutual information $I_{X,Z}$, adapting it to our
notational convention. Secondly, we derive the equivalence relation in Eq.
(19).

$\displaystyle I_{X,Z}$
$\displaystyle=\mathbb{E}_{p(x,z)}\bigg{\\{}\log\frac{p(x|z)}{p(x)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{p(x|z)q(x|z)}{p(x)q(x|z)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{p(x|z)}{q(x|z)}\bigg{\\}}{+}\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{q(x|z)}{p(x)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{q(z)}KL\\{p(x|z)\|q(x|z)\\}{+}\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{q(x|z)}{p(x)}\bigg{\\}}$
$\displaystyle\geq\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{q(x|z)}{p(x)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{p(x)e^{\tilde{f}(x,z)}}{p(x)Z(z)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{e^{\tilde{f}(x,z)}}{Z(z)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log
Z(z)\\}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{\tilde{f}(x,z)\\}{-}\mathbb{E}_{p(x)q(z|x)}\\{\log\mathbb{E}_{p(x^{\prime})}\\{e^{\tilde{f}(x^{\prime},z)}\\}\\}$
$\displaystyle\doteq I_{UBA}(X,Z)$

where we have introduced both an auxiliary encoder $q(z|x)$ and an auxiliary
decoder defined as $q(x|z)=\frac{p(x)e^{\tilde{f}(x,z)}}{Z(z)}$. Now, we can
use the assumptions of a uniform prior $p(z;\Theta)$ and $f(x,z)=\log
p(x)+\tilde{f}(x,z)$ to achieve the following inequalities:s

$\displaystyle\text{ELBO}_{EBM}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{f(x,z)\\}+\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\log\frac{p(z;\Theta)}{\Gamma(z;\Theta)}\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{f(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log\Gamma(z;\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{\log
p(x)+\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log\mathbb{E}_{p(x^{\prime})}\\{e^{\tilde{f}(x^{\prime},z)}\\}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{\log
p(x)\\}+\mathbb{E}_{p(x)q(z|x)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log\mathbb{E}_{p(x^{\prime})}\\{e^{\tilde{f}(x^{\prime},z)}\\}$
$\displaystyle=-H(X)+\mathbb{E}_{p(x)q(z|x)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log\mathbb{E}_{p(x^{\prime})}\\{e^{\tilde{f}(x^{\prime},z)}\\}$
$\displaystyle=-H(X)+I_{UBA}(X,Z)$

###  B.1 Other Bounds

Notably, $I_{UBA}(X,Z)$ cannot be tractably computed due to the computation of
$Z(z)$. Therefore, several other bounds have been derived (Poole et al., 2019)
to obtain tractable estimators or optimization objectives, namely:

  1. 1.

$I_{TUBA}(X,Z)$,777T in TUBA stands for tractable. which can be used for both
optimization and estimation of mutual information (obtained using the
inequality $\log s\leq\eta^{\prime}s-\log\eta^{\prime}-1$ for all scalar
$s,\eta^{\prime}>0$)

$\displaystyle I_{UBA}(X,Z)$
$\displaystyle=\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log
Z(z)\\}$
$\displaystyle\geq\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\frac{Z(z)}{\eta(z)}+\log\eta(z)-1\bigg{\\}}$
$\displaystyle\doteq I_{TUBA}(X,Z)$

  2. 2.

$I_{NWJ}(X,Z)$, which can be used for both optimization and estimation of
mutual information (obtained from $I_{TUBA}(X,Z)$ by imposing $\eta(z)=e$)

$\displaystyle I_{TUBA}(X,Z)$
$\displaystyle=\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\frac{Z(z)}{\eta(z)}+\log\eta(z)-1\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\bigg{\\{}\frac{Z(z)}{e}+1-1\bigg{\\}}$
$\displaystyle=\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\frac{1}{e}\mathbb{E}_{p(x)q(z|x)}\\{Z(z)\\}$
$\displaystyle=\mathbb{E}_{p(x,z)}\\{\tilde{f}(x,z)\\}-\mathbb{E}_{p(x^{\prime})p(x)q(z|x)}\\{e^{\tilde{f}(x^{\prime},z)-1}\\}$
$\displaystyle\doteq I_{NWJ}(X,Z)$

##  Appendix C Derivation of InfoNCE

Consider a prior
$p(z)=\frac{\frac{\Gamma(z;\Theta)}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}}}{\Gamma(\Theta)}$,
where
$\Gamma(\Theta)=\int\frac{\Gamma(z;\Theta)}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}}dz$,

$\displaystyle\text{ELBO}_{EBM}$
$\displaystyle=\mathbb{E}_{p(x)q(z|x)}\\{f(x,z)\\}-\mathbb{E}_{p(x)q(z|x)}\\{\log\Gamma(z;\Theta)\\}+\mathbb{E}_{p(x)q(z|x)}\\{\log
p(z;\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1})q(z|x_{1})}\\{f(x_{1},z)\\}-\mathbb{E}_{p(x_{1})q(z|x_{1})}\\{\log\Gamma(z;\Theta)\\}+\mathbb{E}_{p(x_{1})q(z|x_{1})}\\{\log
p(z;\Theta)\\}$ $\displaystyle=\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\\{f(x_{1},z)\\}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\\{\log\Gamma(z;\Theta)\\}$
$\displaystyle\qquad+\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\\{\log p(z;\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\\{f(x_{1},z)\\}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\bigg{\\{}\log\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}\bigg{\\}}$
$\displaystyle\qquad-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\\{\log
e^{f(x_{1},z)}\\}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})q(z|x_{1})}\bigg{\\{}\log\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}\bigg{\\}}$
$\displaystyle\qquad-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1},z)p(x_{2})\cdot\cdot\cdot
p(x_{n})}\bigg{\\{}\log\frac{e^{f(x_{1},z)}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z)}}\bigg{\\}}$
$\displaystyle\qquad-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1},z_{1})p(x_{2},z_{2})\cdot\cdot\cdot
p(x_{n},z_{K})}\bigg{\\{}\log\frac{e^{f(x_{1},z_{1})}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z_{1})}}\bigg{\\}}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle=\frac{1}{n}\sum_{i=1}\mathbb{E}_{p(x_{1},z_{1})p(x_{2},z_{2})\cdot\cdot\cdot
p(x_{n},z_{K})}\bigg{\\{}\log\frac{e^{f(x_{i},z_{i})}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z_{i})}}\bigg{\\}}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$
$\displaystyle=\mathbb{E}_{p(x_{1},z_{1})p(x_{2},z_{2})\cdot\cdot\cdot
p(x_{n},z_{K})}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{f(x_{i},z_{i})}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z_{i})}}\bigg{\\}}-\mathbb{E}_{p(x_{1})p(x_{2})\cdot\cdot\cdot
p(x_{n})}\\{\log\Gamma(\Theta)\\}$ $\displaystyle\doteq
I_{NCE}(X,Z)-\mathbb{E}_{\prod_{j=1}^{n}p(x_{j})}\\{\log\Gamma(\Theta)\\}$

##  Appendix D Derivation of ProtoCPC: A Lower Bound of InfoNCE

ProtoCPC (Lee, 2022) can be viewed as a lower bound of the InfoNCE objective.
To save space, we use notation $\mathbb{E}$ to refer to
$\mathbb{E}_{\prod_{j=1}^{n}p(x_{j},z_{j})}$. Therefore, we have that

$\displaystyle I_{NCE}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{f(x_{i},z_{i})}}{\frac{1}{n}\sum_{k=1}^{n}e^{f(x_{k},z_{i})}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
p_{c}^{s}(x_{i})}}{\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
p_{c}^{s}(x_{i})}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\left\\{\frac{1}{n}\sum_{i=1}\log\frac{\begin{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}\end{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}}{\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
p_{c}^{s}(x_{i})}}\right\\}$ (23)
$\displaystyle=\mathbb{E}\left\\{\frac{1}{n}\sum_{i=1}\log\frac{\begin{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}\end{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}}{\begin{multlined}\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}\end{multlined}\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}}\right\\}$
(28)
$\displaystyle=\mathbb{E}\left\\{\frac{1}{n}\sum_{i=1}\log\frac{\begin{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}\end{multlined}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}}{\begin{multlined}\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}\end{multlined}\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}\cdot\\\
e^{-\log\sum_{c^{\prime}=1}^{C}e^{z_{c^{\prime},x_{i}}^{s}/\tau_{s}}}}\right\\}$
(33)
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}}{\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})\log
e^{z_{c,x_{i}}^{s}/\tau_{s}}}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})z_{c,x_{i}}^{s}/\tau_{s}}}{\frac{1}{n}\sum_{k=1}^{n}e^{\sum_{c=1}^{C}p_{c}^{t}(x_{k})z_{c,x_{i}}^{s}/\tau_{s}}}\bigg{\\}}$
$\displaystyle\geq\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})z_{c,x_{i}}^{s}/\tau_{s}}}{\frac{1}{n}\sum_{k=1}^{n}\sum_{c=1}^{C}p_{c}^{t}(x_{k})e^{z_{c,x_{i}}^{s}/\tau_{s}}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})z_{c,x_{i}}^{s}/\tau_{s}}}{\sum_{c=1}^{C}\frac{1}{n}\sum_{k=1}^{n}p_{c}^{t}(x_{k})e^{z_{c,x_{i}}^{s}/\tau_{s}}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{\sum_{c=1}^{C}p_{c}^{t}(x_{i})z_{c,x_{i}}^{s}/\tau_{s}}}{\sum_{c=1}^{C}q_{c}^{t}e^{z_{c,x_{i}}^{s}/\tau_{s}}}\bigg{\\}}$
$\displaystyle=\mathbb{E}\bigg{\\{}\frac{1}{n}\sum_{i=1}\log\frac{e^{p^{t}(x_{i})^{T}z_{x_{i}}^{s}/\tau_{s}}}{\sum_{c=1}^{C}q_{c}^{t}e^{z_{c,x_{i}}^{s}/\tau_{s}}}\bigg{\\}}\doteq
I_{ProtoCPC}$

Specifically, the second equality comes from the fact that
$f(x,z)=\sum_{c=1}^{C}p_{c}^{t}(x)\log p_{c}^{s}(x)$, where $s,t$ stand for
student and teacher networks, respectively, and $p_{c}^{\cdot}(x)$ is the
$c$-th entry of the vector obtained by applying a softmax on the embedding of
the corresponding network. The inequality in the derivation is obtained by
applying Jensen’s inequality to the denominator. Finally,
$q_{c}^{t}=\frac{1}{n}\sum_{k=1}^{n}p_{c}^{t}(x_{k})$ corresponds to the
prototype for class $c$.

##  Appendix E Details about Negative-Free Methods

We can specify different definitions for $\mathcal{L}_{NF}(\Theta)$, namely:

  1. 1.

Barlow Twins. The approach enforces the cross-correlation matrix to be close
to the identity matrix:

$\mathcal{L}_{NF}(\Theta)=-\|CCorr(G,G^{\prime})\odot\Lambda-I\|_{F}^{2}$

where $\|\cdot\|_{F}$ is the Frobenius norm, $\Lambda=J\lambda+(1-\lambda)I$,
$I$ is the identity matrix, $\lambda$ is a positive hyperparameter, $J$ is a
matrix of ones, $CCorr(G,G^{\prime})=H^{T}H^{\prime}$ is the sample cross-
correlation matrix,
$G=[g(x_{1}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times h}$ and
$G^{\prime}=[g(x_{1}^{\prime}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times
h}$ and

$g(x_{i})=\text{BN}(\text{Proj}(enc(x_{i})))$

Note that $BN$ is a batch normalization layer, $Proj$ is a projection head
usually implemented using a multi-layer perceptron and $enc$ is the encoder.

  2. 2.

W-MSE. This approach is similar to Barlow Twins. The main difference lie in
the fact that sample cross-correlation matrix is enforced to be an identity
matrix thanks to a whitening layer:

$\mathcal{L}_{NF}(\Theta)=-Tr(G\otimes G^{\prime})$

where $Tr(G\otimes G^{\prime})$ computes the trace of the outer product for
matrices $G=[g(x_{1}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times h}$
and
$G^{\prime}=[g(x_{1}^{\prime}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times
h}$.

$g(x_{i})=\text{L2-Norm}(\text{Whitening}(\text{Proj}(enc(x_{i}))))$

L2-Norm is a normalization layer based on $L_{2}$ norm.

  3. 3.

VICReg. This approach attempts to simplify the architecture of Barlow Twins by
introducing an invariance regularization term in the score function, thus
avoiding to use a batch normalization layer:

$\displaystyle\mathcal{L}_{NF}(\Theta)$ $\displaystyle=-\lambda
Tr((G-G^{\prime})\otimes(G-G^{\prime}))$
$\displaystyle-\mu[v(G)+v(G^{\prime})]-\nu[w(G)+w(G^{\prime})]$

where $\lambda,\mu,\nu$ are positive hyperparameters and
$G=[g(x_{1}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times h}$ and
$G^{\prime}=[g(x_{1}^{\prime}),\dots,g(x_{n}^{\prime})]^{T}\in\mathbb{R}^{n\times
h}$. The first addend enforces the representation to be invariant to the data
augmentation, whereas the other two addends enforce the sample covariance
matrix to be diagonal. Indeed, the second addend forces the diagonal elements
of the sample covariance matrices to be unitary, namely:

$v(G)=\sum_{j=1}^{h}\max\\{0,1-\sqrt{Var\\{G_{:j}\\}+\epsilon}\\}$

where $\epsilon>0$ is used to avoid numerical issues, $Var$ computes the
variance for each column of matrix $G$. The third addend ensures that the off-
diagonal elements of the sample covariance matrix approach to zero:

$c(G)=\|Cov(G)\odot(J-I)\|_{F}^{2}$

with $Cov(H)=G^{T}G$. Therefore, these last two addends have a similar
behaviour to the score function of Barlow Twins. Finally, the resulting
function is simplified.

$g(x_{i})=\text{Proj}(enc(x_{i}))$

##  Appendix F Proof of Proposition 1

Proof  We recall Eq. (6) (we omit the dependence from $\Theta$ to avoid
clutter), namely:

$\displaystyle\mathbb{E}_{p(x_{1:n})}\\{\log
p(x_{1:n})\\}=-H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\log\sum_{y_{1:n}}p(y_{1:n}|x_{1:n})\right\\}$

and add the zero quantity $\log\sum_{y_{1:n}}p(y_{1:n})$ to the right-hand
side of previous equation, thus obtaining the new equation

$\displaystyle\mathbb{E}_{p(x_{1:n})}\\{\log
p(x_{1:n})\\}=-H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\log\sum_{y_{1:n}}p(y_{1:n}|x_{1:n})\right\\}+\log\sum_{y_{1:n}}p(y_{1:n})$
(34)

We can lower bound the previous equation by exploiting the fact that
$\sum_{z}p(z)\geq\sum_{z}p(z)q(z)$ for any given auxiliary discrete
distribution $q$, viz.:

$\displaystyle\text{Eq.\leavevmode\nobreak\ (\ref{eq:last})}\geq$
$\displaystyle-
H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\log\sum_{y_{1:n}}q(y_{1:n}|x_{1:n}^{\prime})p(y_{1:n}|x_{1:n})\right\\}$
$\displaystyle\qquad+\log\sum_{y_{1:n}}p(y_{1:n})q(y_{1:n})$ (35)

Now, by applying Jensen’s inequality to the last two addends in Eq. (35) and
by defining $q(y_{1:n}|x_{1:n}^{\prime})=p(y_{1:n}|x_{1:n}^{\prime})$ and
$q(y_{1:n})=\frac{1}{n}\sum_{j=1}^{n}p(y_{j}|x_{j})$, we obtain the following
lower bound:

Eq. (35) $\displaystyle\geq-
H_{p}(x_{1:n})+\mathbb{E}_{p(x_{1:n})\mathcal{T}(x_{1:n}^{\prime}|x_{1:n})}\left\\{\sum_{y_{1:n}}p(y_{1:n}|x_{1:n}^{\prime})\log
p(y_{1:n}|x_{1:n})\right\\}$
$\displaystyle\qquad+\sum_{y_{1:n}}p(y_{1:n})\log\left(\frac{1}{n}\sum_{j=1}^{n}p(y_{j}|x_{j})\right)$
(36)

Additionally, by factorizing the distributions according to the probabilistic
graphical model in Fig. 2, namely
$p(y_{1:n}|x_{1:n})=\prod_{i=1}^{n}p(y_{i}|x_{i})$,
$p(y_{1:n}|x_{1:n}^{\prime})=\prod_{i=1}^{n}p(y_{i}|x_{i}^{\prime})$ and
$p(y_{1:n})=\prod_{i=1}^{n}p(y_{i})$, we achieve the following equality:

$\displaystyle\text{Eq.\leavevmode\nobreak\ (\ref{eq:last3})}=$
$\displaystyle-
H_{p}(x_{1:n})+\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\left\\{\sum_{y_{i}}p(y_{i}|x_{i}^{\prime})\log
p(y_{i}|x_{i})\right\\}$
$\displaystyle\qquad+\sum_{i=1}^{n}\sum_{y_{i}}p(y_{i})\log\left(\frac{1}{n}\sum_{j=1}^{n}p(y_{j}=y_{i}|x_{j})\right)$
(37)

And by rewriting the last two addends in Eq. (37) using the definition of
cross-entropy, we obtain our final result.

Now, we can conclude the proof by looking at the maxima for
$\mathcal{L}_{INV}$ and $\mathcal{L}_{PRIOR}$. Indeed, we observe that both
terms compute a negative cross-entropy between two distributions. By
leveraging the fact that $CE(p,q)=H_{p}+KL(p\|q)$ for arbitrary distributions
$p,q$, we can easily see that the maximum of $\mathcal{L}_{INV}$ is attained
when the term is $0$ (corresponding to minimal entropy and minimal KL),
whereas the maximum of $\mathcal{L}_{PRIOR}$ is attained when the term is
equal to $-H_{p}(y_{i})$ (corresponding to minimal KL).

##  Appendix G Proof of Theorem 5

Proof  The overall strategy to prove the statements relies on the evaluation
of the loss terms over the three failure modes and on checking whether these
attain their corresponding maxima.

Let’s start by proving statement a and recalling that
$\mathcal{L}_{GEN}(\Theta,\mathcal{D})=-CE(p,p_{\Theta})$. Firstly, we test
for failure mode 1 (i.e. representational collapse). We observe that for all
$x\in\mathbb{R}^{d}$

$p_{\Theta}(x)=\frac{\sum_{y=1}^{c}e^{f_{y}(k)/\tau}}{\Gamma(\Theta)}$

thus $p_{\Theta}(x)$ assigns constant mass everywhere. Clearly, $p_{\Theta}$
is different from $p$. Therefore, $-CE(p,p_{\Theta})<-CE(p,p)$ and failure
mode 1 is not admissible. Secondly, we test for failure mode 2 (i.e. cluster
collapse). We can equivalently rewrite the definition of cluster collapse by
stating that there exists $j\in\\{1,\dots,c\\}$, such that for all
$x\in\mathbb{R}^{d}$ and for all $y\neq j$,
$f_{j}(enc(x))-f_{y}(enc(x))\rightarrow\infty$. Additionally, we observe that

$\displaystyle p_{\Theta}(x)$
$\displaystyle\underbrace{=}_{\text{$\xi_{x}\doteq
enc(x)$}}\frac{\sum_{y=1}^{c}e^{f_{y}(\xi_{x})/\tau}}{\int\sum_{y=1}^{c}e^{f_{y}(\xi_{x})/\tau}dx}$
$\displaystyle=\frac{e^{f_{j}(\xi_{x})/\tau}\left[1+\sum_{y\neq
j}\cancelto{0}{e^{(f_{y}(\xi_{x})-f_{j}(\xi_{x}))/\tau}}\right]}{\int
e^{f_{j}(\xi_{x})/\tau}\left[1+\sum_{y\neq
j}\cancelto{0}{e^{(f_{y}(\xi_{x})-f_{j}(\xi_{x}))/\tau}}\right]dx}$
$\displaystyle=\frac{e^{f_{j}(\xi_{x})/\tau}}{\int e^{f_{j}(\xi_{x})/\tau}dx}$
(38)

where we have used the failure mode condition to obtain the last equality.
Now, note that Eq. (38) defines a standard energy-based model. Consequently,
given enough capacity for the predictive model, it is trivial to verify that
there exists $\Theta$ such that the condition about failure mode is met and
$p_{\Theta}$ is equal to $p$. Cluster collapse is therefore an admissible
solution. Thirdly, we test for permutation invariance for the cluster
assignments. Indeed, we have that

$\displaystyle\mathcal{L}_{GEN}(\Theta,\mathcal{D})$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\left\\{\log
p_{\Theta}(x_{i})\right\\}$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})}\left\\{\log\frac{\sum_{y=1}^{c}e^{t_{i}(y)/\tau}}{\int\sum_{y=1}^{c}e^{t_{i}(y)/\tau}dx}\right\\}$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}_{p(x_{1:n})}\left\\{\log\frac{\sum_{y=1}^{c}e^{t_{i}(y)/\tau}}{\int\sum_{y=1}^{c}e^{t_{i}(y)/\tau}dx}\right\\}$
(39)

where $t_{i}(y)=f_{y}(enc(x_{i}))$. Similarly, we have that

$\displaystyle\mathcal{L}_{GEN}(\Theta,\mathcal{D}^{\pi})$
$\displaystyle\qquad\underbrace{=}_{\text{from Eq.\leavevmode\nobreak\
(\ref{eq:lgen})}}\sum_{i=1}^{n}\mathbb{E}_{p(x_{1:n})}\left\\{\log\frac{\sum_{y=1}^{c}e^{t_{\pi(i)}(y)/\tau}}{\int\sum_{y=1}^{c}e^{t_{\pi(i)}(y)/\tau}dx}\right\\}$
$\displaystyle\qquad=\sum_{i=1}^{n}\mathbb{E}_{p(x_{\pi(i)})}\left\\{\log\frac{\sum_{y=1}^{c}e^{t_{\pi(i)}(y)/\tau}}{\int\sum_{y=1}^{c}e^{t_{\pi(i)}(y)/\tau}dx}\right\\}$
$\displaystyle\qquad\underbrace{=}_{\text{$l\doteq\pi(i)$}}\sum_{l=1}^{n}\mathbb{E}_{p(x_{l})}\left\\{\log\frac{\sum_{y=1}^{c}e^{t_{l}(y)/\tau}}{\int\sum_{y=1}^{c}e^{t_{l}(y)/\tau}dx}\right\\}$
$\displaystyle\qquad=\mathcal{L}_{GEN}(\Theta,\mathcal{D})$

Hence, failure mode 3 is an admissible solution.

Let’s continue by proving statement b and recalling that

$\displaystyle\mathcal{L}_{INV}(\Theta,\mathcal{D})=-\sum_{i=1}^{n}\mathbb{E}_{p(x_{i})\mathcal{T}(x_{i}^{\prime}|x_{i})}\left\\{CE(p(y_{i}|x_{i}^{\prime};\Theta),p(y_{i}|x_{i};\Theta))\right\\}$
(40)

Firstly, we test for representational collapse. In this case, we have that for
all $i\in\\{1,\dots,n\\}$

$\displaystyle
p(y_{i}|x_{i};\Theta)=p(y_{i}|x_{i}^{\prime};\Theta)=\text{Softmax}(f(k)/\tau)$

Based on this result, we observe that the cross-entropy terms in Eq. (40) can
be made 0 by proper choice of $k$. Therefore, representational collapse is an
admissible solution. Secondly, we test for cluster collapse. Here, it is easy
to see that the cross-entropy terms in Eq. (40) are all 0. Therefore, also
cluster collapse is admissible. Thirdly, we test for permutation invariance
for the cluster assignments. On one hand, we have that the cross-entropy terms
for $\mathcal{L}_{INV}(\Theta,\mathcal{D})$) in Eq. (40) can be rewritten in
the following way:

$\displaystyle CE(p(y_{i}|x_{i}^{\prime};\Theta),p(y_{i}|x_{i};\Theta))$
$\displaystyle\qquad=CE\left(\frac{e^{t_{i}^{\prime}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{i}^{\prime}(y)/\tau}},\frac{e^{t_{i}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{i}(y)/\tau}}\right)$
(41)

and the optimal solution is achieved only when $t_{i}^{\prime}=t_{i}$ for all
$i\in\\{1,\dots,n\\}$. On the other hand, the cross-entropy terms for
$\mathcal{L}_{INV}(\Theta,\mathcal{D}^{\pi})$ are given by the following
equality:

$\displaystyle CE(p(y_{i}|x_{i}^{\prime};\Theta),p(y_{i}|x_{i};\Theta))$
$\displaystyle\qquad=CE\left(\frac{e^{t_{i}^{\prime}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{i}^{\prime}(y)/\tau}},\frac{e^{t_{\pi(i)}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{\pi(i)}(y)/\tau}}\right)$
(42)

However, the optimal solution cannot be achieved in general as
$t_{i}^{\prime}\neq t_{\pi(i)}$ for some $i\in\\{1,\dots,n\\}$.888Indeed, note
that $t_{i}^{\prime}=t_{\pi(i)}$ for all $i$ occurs only when we are in one of
the first two failure modes. Therefore, $\mathcal{L}_{INV}$ is not permutation
invariant to cluster assignments.

Let’s conclude by proving statement c and recalling that

$\displaystyle\mathcal{L}_{PRIOR}(\Theta,\mathcal{D})=-\sum_{i=1}^{n}CE\left(p(y_{i}),\frac{1}{n}\sum_{l=1}^{n}p(y_{l}=y_{i}|x_{l};\Theta)\right)$
(43)

Firstly, we test for representational collapse. One can easily observe that if
$enc(x)=k$ for all $x\in\mathbb{R}^{d}$, $p(y|x;\Theta)$ becomes uniform,
namely $p(y|x;\Theta)=1/c$ for all $y\in\\{1,\dots,c\\}$. Consequently,
$\frac{1}{n}\sum_{l=1}^{n}p(y_{l}=y_{i}|x_{l};\Theta)=1/c$ for all
$i\in\\{1,\dots,n\\}$. Now, since $p(y_{i})=1/c$ for all
$i\in\\{1,\dots,n\\}$, the cross-entropy terms in Eq. (43) reach their maximum
value $-H_{p}(y_{i})$ for all $i\in\\{1,\dots,n\\}$. Therefore,
representational collapse attains the global maximum of $\mathcal{L}_{PRIOR}$
and is an admissible solution. Secondly, we test for cluster collapse. By
using the definition of cluster collapse, we observe that

$\displaystyle\frac{1}{n}\sum_{l=1}^{n}p(y_{l}=y_{i}|x_{l};\Theta)=\left\\{\begin{array}[]{lc}0&y_{i}=j\\\
1&y_{i}\neq j\end{array}\right.$ (46)

Therefore, the resulting distribution is non-uniform, differently from
$p(y_{i})$. The cross-entropy terms in Eq. (43) are not optimized and cluster
collapse is not admissible. Thirdly, we test for permutation invariance of
cluster assignments. We observe that

$\displaystyle\frac{1}{n}\sum_{l=1}^{n}p(y_{l}=y_{i}|x_{l};\Theta)$
$\displaystyle=\frac{1}{n}\sum_{l=1}^{n}\frac{e^{t_{l}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{l}(y)/\tau}}$
$\displaystyle=\frac{1}{n}\sum_{l=1}^{n}\frac{e^{t_{\pi(l)}(y_{i})/\tau}}{\sum_{y=1}^{c}e^{t_{\pi(l)}(y)/\tau}}$
(47)

which is permutation invariant to cluster assignments. Consequently, also
$\mathcal{L}_{PRIOR}(\Theta,\mathcal{D})=\mathcal{L}_{PRIOR}(\Theta,\mathcal{D}^{\pi})$.
This concludes the proof.

##  Appendix H Hyperparameters for Synthetic Data

For the backbone $enc$, we use a MLP with two hidden layers and 100 neurons
per layer, an output layer with 2 neurons and ReLU activation functions. For
the projection head $proj$ ($f$ for GEDI and its variants), we use a MLP with
one hidden layer and 4 neurons and an output layer with 2 neurons (batch
normalization is used in all layers for Barlow and SwAV as required by their
original formulation). All methods use a batch size of 400. Baseline JEM
(following the original paper): Number of iterations $20K$; learning rate
$1e-3$; Adam optimizer($\beta_{1}=0.9$, $\beta_{2}=0.999$); SGLD steps $10$;
buffer size $10000$; reinitialization frequency $0.05$; SGLD step-size
$\frac{0.01^{2}}{2}$; SGLD noise $0.01$. For the self-supervised learning
methods, please refer to Table 7. We also provide an analysis of sensitivity
to hyperparameters for GEDI (Figure 7).

Table 7: Hyperparameters used in the synthetic experiments.

Methods | Barlow | SwAV | GEDI no gen | GEDI
---|---|---|---|---
Iters | $20k$
Learning rate | $1e-3$
Optimizer | Adam $\beta_{1}=0.9,\beta_{2}=0.999$
Data augmentation noise $\sigma$ | $0.03$
SGLD steps $T$ | - | - | 1 | 1
Buffer size $|B|$ | - | - | 10000 | 10000
Reinitialization frequenc | - | - | 0.05 | 0.05
SGLD step size | - | - |  $\frac{0.01^{2}}{2}$ $\frac{0.01^{2}}{2}$ | $\frac{0.01^{2}}{2}$
SGLD noise | - | - | 0.01 | 0.01
Weight for $\mathcal{L}_{GEN}$ | - | - | $1$ | $1$
Weight for $\mathcal{L}_{INV}$ | - | - | $50$ | $50$
Weight for $\mathcal{L}_{PRIOR}$ | - | - | $10$ | $10$

(a) Moons

(b) Circles

Figure 7: Sensitivity analysis on the discriminative performance of GEDI fr
different loss weights (in the range $\\{0,10,20,30,40,50\\}$). Performance
are averaged over 5 different random seeds. Yellow means perfect NMI.

##  Appendix I Hyperparameters for SVHN, CIFAR10, CIFAR100

Table 8: Resnet architecture. Conv2D(A,B,C) applies a 2d convolution to input
with B channels and produces an output with C channels using stride (1, 1),
padding (1, 1) and kernel size (A, A).

Name | Layer | Res. Layer
---|---|---
Block 1 | Conv2D(3,3,F) | AvgPool2D(2)
LeakyRELU(0.2)
Conv2D(3,F,F) | Conv2D(1,3,F) no padding
AvgPool2D(2)
Sum
Block 2 | LeakyRELU(0.2) |
Conv2D(3,F,F) |
LeakyRELU(0.2) |
Conv2D(3,F,F) |
AvgPool2D(2) |
Block 3 | LeakyRELU(0.2) |
Conv2D(3,F,F) |
LeakyRELU(0.2) |
Conv2D(3,F,F) |
Block 4 | LeakyRELU(0.2) |
Conv2D(3,F,F) |
LeakyRELU(0.2) |
Conv2D(3,F,F) |
AvgPool2D(all) |

For the backbone $enc$, we use a ResNet with 8 layers as in (Duvenaud et al.,
2021), where its architecture is shown in Table 8. For the projection head
$proj$ ($f$ for GEDI and its variants), we use a MLP with one hidden layer and
$2*F$ neurons and an output layer with $F$ neurons (batch normalization is
used in all layers for Barlow and SwAV as required by their original
formulation + final $L_{2}$ normalization). $F=128$ for SVHN, CIFAR-10 (1
million parameters) and $F=256$ for CIFAR-100 (4.1 million parameters). For
JEM, we use the same settings of (Duvenaud et al., 2021). All methods use a
batch size of 64. Baseline JEM (following the original paper): number of
epochs $20$, $200$, $200$ for SVHN, CIFAR-10, CIFAR-100, respectively;
learning rate $1e-4$; Adam optimizer; SGLD steps $20$; buffer size $10000$;
reinitialization frequency $0.05$ ; SGLD step-size $1$; SGLD noise $0.01$;
data augmentation (Gaussian noise) $0.03$. For the self-supervised learning
methods, please refer to Table 9.

Table 9: Hyperparameters (in terms of sampling, optimizer, objective and data
augmentation) used in all experiments.

Class | Name param. | SVHN | CIFAR-10 | CIFAR-100 | MNIST | Addition
---|---|---|---|---|---|---
Data augment. | Color jitter prob. | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Gray scale prob. | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Random crop | Yes | Yes | Yes | Yes | Yes
Additive Gauss. noise (std) | 0.03 | 0.03 | 0.03 | 0.2 | 0.2
Random horizontal flip | No | Yes | Yes | No | No
SGLD | SGLD iters | 20 | 20 | 20 | 20 | 20
Buffer size | 10k | 10k | 10k | 10k | 10k
Reinit. frequency | 0.05 | 0.05 | 0.05 | 0.05 | 0.05
SGLD step-size | 1 | 1 | 1 | 1 | 1
SGLD noise | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
Optimizer | Batch size | 64 | 64 | 64 | 60 | 60
Epochs | 20 | 200 | 200 | * | *
Adam $\beta_{1}$ | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
Adam $\beta_{2}$ | 0.999 | 0.999 | 0.999 | 0.999 | 0.999
Learning rate | $1e-4$ | $1e-4$ | $1e-4$ | $1e-4$ | $1e-4$
L2 regularization | $0$ | $0$ | $0$ | $1e-4$ | $1e-4$
Weights for losses | $\mathcal{L}_{GEN}$ | 1 | 1 | 1 | 1 | 1
$\mathcal{L}_{INV}$ | 50 | 50 | 50 | 50 | 0
$\mathcal{L}_{PRIOR}$ | 25 | 25 | 50 | 25 | 400
$\mathcal{L}_{NESY}$ | - | - | - | 25 | 0

*: For the 3 different dataset sizes ($100$, $1000$, $10000$), we trained for $500$, $100$ and $25$ epochs respectively.

##  Appendix J Additional Experiments on SVHN, CIFAR-10, CIFAR-100

We conduct a linear probe evaluation of the representations learnt by the
different models Table 10. These experiments provide insights on the
capabilities of learning representations producing linearly separable classes.
From Table 10, we observe a large difference in results between Barlow and
SwAV. Our approach provides interpolating results between the two approaches.

We also provide additional qualitative analyisis on the generation performance
on SVHN and CIFAR-100. Please, refer to Figure 8 and Figure 9.

Finally, we evaluate the performance in terms of OOD detection, by following
the same methodology used in (Grathwohl et al., 2020). We use the OOD score
criterion proposed in (Grathwohl et al., 2020), namely
$s(x)=-\|\frac{\partial\log p_{\Psi}(x)}{\partial x}\|_{2}$. From Table 5, we
observe that GEDI achieves almost optimal performance. While these results are
exciting, it is important to mention that they are not generally valid.
Indeed, when training on CIFAR-10 and performing OOD evaluation on the other
datasets, we observe that all approaches achieve similar performance both on
CIFAR-100 and SVHN, suggesting that all datasets are considered in-
distribution, see Table 11. A similar observation is obtained when training on
CIFAR-100 and evaluating on CIFAR-10 and SVHN, see Table 12. Importantly, this
is a phenomenon which has been only recently observed by the scientific
community on generative models. Tackling this problem is currently out of the
scope of this work. For further discussion about the issue, we point the
reader to the works in (Nalisnick et al., 2019).

(a) Barlow

(b) SwAV

(c) GEDI no gen

(d) GEDI

Figure 8: Qualitative visualization of the generative performance for the
different discriminative strategies on SVHN. Results are obtained by running
Stochastic Langevin Dynamics for 500 iterations.

(a) Barlow

(b) SwAV

(c) GEDI no gen

(d) GEDI

Figure 9: Qualitative visualization of the generative performance for the
different discriminative strategies on CIFAR-100. Results are obtained by
running Stochastic Langevin Dynamics for 500 iterations. Table 10: Supervised
linear evaluation in terms of accuracy on test set (SVHN, CIFAR-10,
CIFAR-100). The linear classifier is trained for 100 epochs using SGD with
momentum, learning rate $1e-3$ and batch size 100. Mean and standard
deviations are computed over results from 5 different initialization seeds.

Dataset | JEM | Barlow | SwAV | GEDI two stage | GEDI no gen | GEDI
---|---|---|---|---|---|---
SVHN | 0.20$\pm$0.00 | 0.75$\pm$0.01 | 0.44$\pm$0.04 | 0.31$\pm$0.01 | 0.59$\pm$0.02 | 0.54$\pm$0.01
CIFAR-10 | 0.24$\pm$0.00 | 0.65$\pm$0.00 | 0.50$\pm$0.02 | 0.55$\pm$0.01 | 0.63$\pm$0.01 | 0.63$\pm$0.01
CIFAR-100 | 0.03$\pm$0.00 | 0.27$\pm$0.01 | 0.14$\pm$0.01 | 0.16$\pm$0.01 | 0.31$\pm$0.01 | 0.31$\pm$0.01

Table 11: OOD detection in terms of AUROC on test set (SVHN, CIFAR-100).
Training is performed on CIFAR-10.

Dataset | JEM | Barlow | SwAV | GEDI two stage | GEDI no gen | GEDI
---|---|---|---|---|---|---
SVHN | 0.44 | 0.32 | 0.62 | 0.31 | 0.11 | 0.57
CIFAR-100 | 0.53 | 0.56 | 0.51 | 0.55 | 0.51 | 0.61

Table 12: OOD detection in terms of AUROC on test set (SVHN, CIFAR-10).
Training is performed on CIFAR-100.

Dataset | JEM | Barlow | SwAV | GEDI two stage | GEDI no gen | GEDI
---|---|---|---|---|---|---
SVHN | 0.44 | 0.45 | 0.30 | 0.38 | 0.55 | 0.53
CIFAR-10 | 0.49 | 0.43 | 0.47 | 0.50 | 0.46 | 0.48

##  Appendix K Details on the MNIST addition experiment.

The hyperparameters for the experiment without and with the NeSy constraint
are as reported in Table 9. The data was generated by uniformly sampling pairs
$a,b$ such that $0\leq a\leq 9$, $0\leq b\leq 9$ and $0\leq a+b\leq 9$. For
each triplet $(a,b,c)$, we assigned to $a,b,c,$ random MNIST images with
corresponding labels, without replacement. For the experiment with the NeSy
constraint, we use a slightly different uniformity regularization. We maximize
the entropy of the mean output distribution for each batch, cf. Manhaeve et
al. (2021).

###  K.1 Replacing $\mathcal{L}_{INV}$ with $\mathcal{L}_{NESY}$

For these experiments, we replace $\mathcal{L}_{INV}$ with
$\mathcal{L}_{NESY}$ as the NeSy loss also forces a clustering of the digits,
making $\mathcal{L}_{INV}$ redundant. Figure 10 shows that both the NeSy Loss
and the clustering loss progress similarly.

(a) $\mathcal{L}_{NESY}$

(b) $\mathcal{L}_{INV}$

Figure 10: Comparing $\mathcal{L}_{NESY}$ with $\mathcal{L}_{INV}$ on the
Addition experiment, where only the NeSy loss is optimized.

## References

  * Alemi et al. (2018) A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, and K. Murphy.  Fixing a Broken ELBO.  In _ICML_ , 2018.
  * Amrani et al. (2022) E. Amrani, L. Karlinsky, and A. Bronstein.  Self-Supervised Classification Network.  In _ECCV_ , 2022.
  * Asano et al. (2020) Y. M. Asano, C. Rupprecht, and A. Vedaldi.  Self-Labelling via Simultaneous Clustering and Representation Learning.  In _ICLR_ , 2020.
  * Assran et al. (2023) M. Assran, R. Balestriero, Q. Duval, F. Bordes, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, and N. Ballas.  The Hidden Uniform Cluster Prior in Self-Supervised Learning.  In _ICLR 2023_ , 2023.
  * Assran, Mahmoud and Caron, Mathilde and Misra, Ishan and Bojanowski, Piotr and Bordes, Florian and Vincent, Pascal and Joulin, Armand and Rabbat, Mike and Ballas, Nicolas (2022) Assran, Mahmoud and Caron, Mathilde and Misra, Ishan and Bojanowski, Piotr and Bordes, Florian and Vincent, Pascal and Joulin, Armand and Rabbat, Mike and Ballas, Nicolas.  Masked Siamese Networks for Label-Efficient Learning.  In _ECCV_ , 2022.
  * Balestriero and LeCun (2022) R. Balestriero and Y. LeCun.  Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods.  In _NeurIPS_ , 2022.
  * Balestriero et al. (2023) R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, et al.  A Cookbook of Self-Supervised Learning.  _arXiv_ , 2023.
  * Bao et al. (2022) H. Bao, Y. Nagano, and K. Nozawa.  On the Surrogate Gap between Contrastive and Supervised Losses.  In _ICML_ , 2022.
  * Barber and Agakov (2004) D. Barber and F. Agakov.  The IM Algorithm: A Variational Approach to Information Maximization.  In _NeurIPS_ , 2004.
  * Bardes et al. (2022a) A. Bardes, J. Ponce, and Y. Lecun.  VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.  In _ICLR_ , 2022a.
  * Bardes et al. (2022b) A. Bardes, J. Ponce, and Y. LeCun.  VICREGL: Self-Supervised Learning of Local Visual Features.  In _NeurIPS_ , 2022b.
  * Becker and Hinton (1992) S. Becker and G. Hinton.  Self-Organizing Neural Network that Discovers Surfaces in Random-Dot Stereograms.  _Nature_ , 355(6356):161–163, 1992.
  * Belghazi et al. (2018) M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm.  Mutual Information Neural Estimation.  In _ICML_ , 2018.
  * Caron et al. (2018) M. Caron, P. Bojanowski, A. Joulin, and M. Douze.  Deep Clustering for Unsupervised Learning of Visual Features.  In _ECCV_ , 2018.
  * Caron et al. (2020) M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments.  In _NeurIPS_ , 2020.
  * Caron et al. (2021) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging Properties in Self-Supervised Vision Transformers.  In _ICCV_ , 2021.
  * Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A Simple Framework for Contrastive Learning of Visual Representations.  In _ICML_ , 2020.
  * Cuturi (2013) M. Cuturi.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport.  In _NeurIPS_ , 2013.
  * da Costa et al. (2022) V. G. T. da Costa, E. Fini, M. Nabi, N. Sebe, and E. Ricci.  Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning.  _JMLR_ , 23(56):1–6, 2022.
  * den Oord et al. (2018) A. V. den Oord, Y. Li, and O. Vinyals.  Representation Learning with Contrastive Predictive Coding.  In _arXiv_ , 2018.
  * Du and Mordatch (2019) Y. Du and I. Mordatch.  Implicit Generation and Generalization in Energy-Based Models.  _arXiv_ , 2019.
  * Dubois et al. (2022) Y. Dubois, T. Hashimoto, S. Ermon, and P. Liang.  Improving Self-Supervised Learning by Characterizing Idealized Representations.  In _NeurIPS_ , 2022.
  * Duvenaud et al. (2021) D. Duvenaud, J. Kelly, K. Swersky, M. Hashemi, M. Norouzi, and W. Grathwohl.  No MCMC for Me: Amortized Samplers for Fast and Stable Training of Energy-Based Models.  In _ICLR_ , 2021.
  * Ermolov et al. (2021) A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe.  Whitening for Self-Supervised Representation Learning.  In _ICML_ , 2021.
  * Garrido et al. (2023) Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun.  On the Duality Between Contrastive and Non-Contrastive Self-Supervised Learning.  In _ICLR_ , 2023.
  * Gatopoulos and Tomczak (2020) I. Gatopoulos and J. M. Tomczak.  Self-Supervised Variational Auto-Encoders.  In _Symposium on Advances in Approximate Bayesian Inference_ , 2020\.
  * Grathwohl et al. (2020) W. Grathwohl, K.-C. Wang, J.-H. Jacobsen, D. Duvenaud, M. Norouzi, and K. Swersky.  Your Classifier is Secretly an Energy Based Model and You Should Treat It Like One.  In _ICLR_ , 2020.
  * Grill et al. (2020) J. B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning.  In _NeurIPS_ , 2020.
  * HaoChen et al. (2021) J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma.  Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss.  In _NeurIPS_ , 2021.
  * Hendrycks et al. (2019) D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song.  Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.  In _NeurIPS_ , 2019.
  * Heusel et al. (2017) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In _NeurIPS_ , 2017.
  * Hjelm et al. (2018) R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.  Learning Deep Representations by Mutual Information Estimation and Maximization.  In _ICLR_ , 2018.
  * Huang et al. (2022) Z. Huang, J. Chen, J. Zhang, and H. Shan.  Learning Representation for Clustering Via Prototype Scattering and Positive Sampling.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Jing and Tian (2021) L. Jing and Y. Tian.  Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Kang-Jun et al. (2022) L. Kang-Jun, S. Masanori, and O. Takayuki.  Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-Supervised Learning.  In _NeurIPS_ , 2022.
  * Kim and Ye (2022) B. Kim and J. C. Ye.  Energy-Based Contrastive Learning of Visual Representations.  In _NeurIPS_ , 2022.
  * Kügelgen et al. (2021) J. V. Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello.  Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style.  In _NeurIPS_ , 2021.
  * LeCun (2022) Y. LeCun.  A Path Towards Autonomous Machine Intelligence, 2022.
  * Lee (2022) K. Lee.  Prototypical Contrastive Predictive Coding.  In _ICLR_ , 2022.
  * Li et al. (2021) Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng.  Contrastive Clustering.  In _AAAI_ , 2021.
  * Li et al. (2022) Z. Li, Y. Chen, Y. LeCun, and F. T. Sommer.  Neural Manifold Clustering and Embedding.  In _arXiv_ , 2022.
  * Linsker (1988) R. Linsker.  Self-Organization in a Perceptual Network.  _Computer_ , 21(3):105–117, 1988.
  * Liu and Abbeel (2020) H. Liu and P. Abbeel.  Hybrid Discriminative-Generative Training via Contrastive Learning.  In _arXiv_ , 2020.
  * Liu et al. (2022) X. Liu, Z. Wang, Y. Li, and S. Wang.  Self-Supervised Learning via Maximum Entropy Coding.  In _NeurIPS_ , 2022.
  * Manhaeve et al. (2018) R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. D. Raedt.  DeepProbLog: Neural Probabilistic Logic Programming.  In _NeurIPS_ , 2018.
  * Manhaeve et al. (2021) R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. D. Raedt.  Neural probabilistic logic programming in DeepProbLog.  _Artifical Intelligence_ , 298:103504, 2021.
  * Marconato et al. (2023) E. Marconato, S. Teso, and A. Passerini.  Neuro-Symbolic Reasoning Shortcuts: Mitigation Strategies and their Limitations.  In _arXiv_ , 2023.
  * McAllester and Stratos (2020) D. McAllester and K. Stratos.  Formal Limitations on the Measurement of Mutual Information.  In _AISTATS_ , 2020.
  * Mitrovic et al. (2021) J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blundell.  Representation Learning via Invariant Causal Mechanisms.  In _ICLR_ , 2021.
  * Mohseni et al. (2020) S. Mohseni, M. Pitale, J. B. S. Yadawa, and Z. Wang.  Self-Supervised Learning for Generalizable Out-of-Distribution Detection.  In _AAAI_ , 2020.
  * Nakamura et al. (2023) H. Nakamura, M. Okada, and T. Taniguchi.  Representation Uncertainty in Self-Supervised Learning as Variational Inference.  In _ICCV_ , 2023.
  * Nalisnick et al. (2019) E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan.  Do deep generative models know what they don’t know?  In _ICLR_ , 2019.
  * Nijkamp et al. (2019) E. Nijkamp, M. Hill, S. C. Zhu, and Y. N. Wu.  Learning non-convergent non-persistent short-run mcmc toward energy-based model.  In _NeurIPS_ , 2019.
  * Nijkamp et al. (2020) E. Nijkamp, M. Hill, T. Han, S.-C. Zhu, and Y. N. Wu.  On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models.  In _AAAI_ , 2020.
  * O. Henaff (2020) O. O. Henaff.  Data-Efficient Image Recognition with Contrastive Predictive Coding.  In _ICML_ , 2020.
  * Ozsoy et al. (2022) S. Ozsoy, S. Hamdan, S. Arik, D. Yuret, and A. T. Erdogan.  Self-Supervised Learning with an Information Maximization Criterion.  In _NeurIPS_ , 2022.
  * Poole et al. (2019) B. Poole, S. Ozair, A. V. D. Oord, A. Alemi, and G. Tucker.  On Variational Bounds of Mutual Information.  In _ICML_ , 2019.
  * Sansone (2023) E. Sansone.  The Triad of Failure Modes and a Possible Way Out.  In _NeurIPS SSL Theory and Practice Workshop_ , 2023.
  * Sansone and Manhaeve (2022) E. Sansone and R. Manhaeve.  GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning.  _arXiv_ , 2022.
  * Sansone and Manhaeve (2023a) E. Sansone and R. Manhaeve.  Learning Symbolic Representations Through Joint GEnerative and DIscriminative Training.  In _ICLR NeSy-GeMs Workshop_ , 2023a.
  * Sansone and Manhaeve (2023b) E. Sansone and R. Manhaeve.  Learning symbolic representations through joint generative and discriminative training (extended abstract).  In _IJCAI KBCG Workshop_ , 2023b.
  * Sansone et al. (2016) E. Sansone, A. Passerini, and F. G. D. Natale.  Classtering: Joint Classification and Clustering with Mixture of Factor Analysers.  In _ECAI_ , 2016.
  * Saunshi et al. (2019) N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar.  A Theoretical Analysis of Contrastive Unsupervised Representation Learning.  In _ICML_ , 2019.
  * Saunshi et al. (2022) N. Saunshi, J. Ash, S. Goel, D. Misra, C. Zhang, S. Arora, S. Kakade, and A. Krishnamurthy.  Understanding Contrastive Learning Requires Incorporating Inductive Biases.  In _ICML_ , 2022.
  * Scherr et al. (2022) F. Scherr, Q. Guo, and T. Moraitis.  Self-Supervised Learning Through Efference Copies.  In _NeurIPS_ , 2022.
  * Song and Ermon (2020) J. Song and S. Ermon.  Understanding the Limitations of Variational Mutual Information Estimators.  _ICLR_ , 2020.
  * Tian et al. (2021) Y. Tian, X. Chen, and S. Ganguli.  Understanding Self-Supervised Learning Dynamics Without Contrastive Pairs.  In _ICML_ , 2021.
  * Tieleman (2008) T. Tieleman.  Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient.  In _ICML_ , 2008.
  * Tomasev et al. (2022) N. Tomasev, I. Bica, B. McWilliams, L. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic.  Pushing the Limits of Self-Supervised ResNets: Can We Outperform Supervised Learning Without Labels on ImageNet?  In _ICML Pre-Training Workshop_ , 2022.
  * Tosh et al. (2021) C. Tosh, A. Krishnamurthy, and D. Hsu.  Contrastive Estimation Reveals Topic Posterior Information to Linear Models.  _JMLR_ , 2021.
  * Tschannen et al. (2019) M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic.  On Mutual Information Maximization for Representation Learning.  In _ICLR_ , 2019.
  * Wang and Isola (2020) T. Wang and P. Isola.  Understanding Contrastive Representation Learning Through Alignment and Uniformity on the Hypersphere.  In _ICML_ , 2020.
  * Wang et al. (2022) X. Wang, H. Fan, Y. Tian, D. Kihara, and X. Chen.  On the Importance of Asymmetry for Siamese Representation Learning.  In _CVPR_ , 2022.
  * Wen and Li (2022) Z. Wen and Y. Li.  The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning.  In _NeurIPS_ , 2022.
  * Weng et al. (2022) X. Weng, L. Huang, L. Zhao, R. M. Anwer, S. Khan, and F. S. Khan.  An Investigation into Whitening Loss for Self-supervised Learning.  In _NeurIPS_ , 2022.
  * Winkens et al. (2020) J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V. Natarajan, J. R. Ledsam, P. MacWilliams, P. Kohli, A. Karthikesalingam, S. Kohl, et al.  Contrastive Training for Improved Out-of-Distribution Detection.  In _arXiv_ , 2020.
  * Wu et al. (2023) C. Wu, J. Pfrommer, M. Zhou, and J. Beyerer.  Generative-Contrastive Learning for Self-Supervised Latent Representations of 3D Shapes from Multi-Modal Euclidean Input.  _arXiv_ , 2023.
  * Xie et al. (2016) J. Xie, Y. Lu, S. C. Zhu, and Y. N. Wu.  A Theory of Generative Convnet.  In _ICML_ , 2016.
  * Xu et al. (2022) H. Xu, H. Xiong, and G. J. Qi.  K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Xu et al. (2018) J. Xu, Z. Zhang, T. Friedman, Y. Liang, and G. Broeck.  A Semantic Loss Function for Deep Learning with Symbolic Knowledge.  In _ICML_ , 2018.
  * Zbontar et al. (2021) J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction.  In _ICML_ , 2021.
  * Zhang et al. (2021) S. Zhang, F. Zhu, J. Yan, R. Zhao, and X. Yang.  Zero-CL: Instance and Feature Decorrelation for Negative-Free Symmetric Contrastive Learning.  In _ICLR_ , 2021.
  * Zhue et al. (2020) Y. Zhue, R. M. M. Renqiang, A. Kadav, and H. P. Graf.  S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation.  In _CVPR_ , 2020.
  * Zimmermann et al. (2021) R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel.  Contrastive Learning Inverts the Data Generating Process.  In _ICML_ , 2021.