11institutetext:  1 Harvard University 2 Princeton University 3 University of
Chicago 4 RWTH Aachen University 5 Boston University 6 Universidade do Vale do
Rio dos Sinos 7 Argonne National Laboratory 8 Donostia International Physics
Center 9 University of the Basque Country (UPV/EHU) 10 Ikerbasque, Basque
Foundation for Science
11email<EMAIL_ADDRESS>
# AxonEM Dataset: 3D Axon Instance Segmentation of Brain Cortical Regions

Donglai Wei1† Kisuk Lee2† Hanyu Li3 Ran Lu2 J. Alexander Bae2 Zequan Liu4∗
Lifu Zhang5∗ Márcia dos Santos6∗ Zudi Lin1 Thomas Uram7 Xueying Wang1 Ignacio
Arganda-Carreras*8  9  10]  Brian Matejek1 Narayanan Kasthuri*3  7]  Jeff
Lichtman1 Hanspeter Pfister1

###### Abstract

Electron microscopy (EM) enables the reconstruction of neural circuits at the
level of individual synapses, which has been transformative for scientific
discoveries. However, due to the complex morphology, an accurate
reconstruction of cortical axons has become a major challenge. Worse still,
there is no publicly available large-scale EM dataset from the cortex that
provides dense ground truth segmentation for axons, making it difficult to
develop and evaluate large-scale axon reconstruction methods. To address this,
we introduce the AxonEM dataset, which consists of two $30\times 30\times
30~{}\mu$m3 EM image volumes from the human and mouse cortex, respectively. We
thoroughly proofread over 18,000 axon instances to provide dense 3D axon
instance segmentation, enabling large-scale evaluation of axon reconstruction
methods. In addition, we densely annotate nine ground truth subvolumes for
training, per each data volume. With this, we reproduce two published state-
of-the-art methods and provide their evaluation results as a baseline. We
publicly release our code and data at https://connectomics-
bazaar.github.io/proj/AxonEM/index.html to foster the development of advanced
methods.

###### Keywords:

Axon Electron Microscopy 3D Instance Segmentation.

††† Equal contribution.††∗ Works were done during the internship at Harvard
University.

##  1 Introduction

With recent technical advances in high-throughput 3D electron microscopy (EM)
[16], it has become feasible to map out neural circuits at the level of
individual synapses in large brain volumes [10, 29, 30]. Reconstruction of a
wiring diagram of the whole-brain, or a connectome, has seen successes in
invertebrates, such as the complete connectome of the roundworm C. elegans
[35] and the partial connectome of the fruit fly D. melanogaster [9, 29].
Mapping the connectome of a whole mouse brain is currently being considered as
the next transformative challenge in the field of connectomics [3]. Detailed
study of large-scale connectomes may lead to new scientific discoveries with
potential clinical implications [3].

At present, automated reconstruction of neural circuits from the cortex faces
a significant obstacle: the axon instance segmentation. In the cortex, axons
are the most abundant type of neurites by path length [24], with the smallest
diameter of $\sim$50 nm in the case of mouse cortical axons [14]. Given their
small caliber, complex morphology, and densely packed nature (Fig. 1), axons
are prone to reconstruction errors especially when challenged by various image
defects such as imperfect staining and image misalignment [17].

To the best of our knowledge, there is no existing EM reconstruction from any
cortical region that provides densely and fully proofread axon instances.
Previously the largest work was carried out by Motta et al. [24], in which the
team spent $\sim$4,000 human hours in semi-automated proofreading of the
initial automated segmentation. However, their finalized reconstruction for
axons suffered from a relatively high error rate [24],11112.8 errors per 1 mm
of path length, estimated with 10 randomly chosen axons [24]. making their
reconstruction less suitable for developing and evaluating large-scale axon
reconstruction methods.

Figure 1: Example axons in our AxonEM dataset. (a) Due to the unclear cell
boundaries resulted from imperfect staining, an axon is falsely merged with an
abutting dendrite. (b) Axons often form a tight bundle. Due to the similar
shape and small size of their cross-sections, axons in such a tight bundle may
be prone to errors, especially in the presence of image misalignment.

To foster new research to address the challenge, we have created a large-scale
benchmark for 3D axon instance segmentation, AxonEM, which is about
250$\times$ larger than the prior art [2] (Fig. 2). AxonEM consists of two
$30\times 30\times 30~{}\mu$m3 EM image volumes acquired from the mouse and
human cortex, respectively. Both image volumes span, at least partially, layer
2 of the cortex, thus allowing for comparative studies on brain structures
across different mammalian species.

Contributions. First, we thoroughly proofread over 18,000 axon instances to
provide the largest ever ground truth for cortical axons, which enables large-
scale evaluation. Second, we densely annotate a handful of dataset subvolumes,
which can be used to train a broad class of methods for axon reconstruction
and allow for an objective comparison. Third, we publicly release our data and
baseline code to reproduce two published state-of-the-art neuron instance
segmentation methods [15, 18], thereby calling for innovative methods to
tackle the challenge.

###  1.1 Related Works

Axon Segmentation. In non-cortical regions, myelinated axons are thick and
easier to segment. Traditional image processing algorithms have been proposed,
including thresholding and morphological operations [8], axon shape-based
morphological discrimination [23], watershed [33], region growing [39], active
contours without [4] and with discriminant analysis [36]. More recently, deep
learning methods have also been used to segment this type of axons [25, 22,
37].

However, unmyelinated axons in the cortex have been a significant source of
reconstruction errors, because their thin and intricate branches are
vulnerable to various types of image defects [17]. Motta et al. [24] took a
semi-automated reconstruction approach to densely segment axons in a volume of
$\sim$500,000 $\mu$m3 from the mouse cortex. However, their estimate of
remaining axon reconstruction errors was relatively high [24].

Figure 2: Comparison of EM datasets with proofread axon instances. (Left)
Distribution of axon instance length. (Right) 3D rendering of neurons with
somas in AxonEM dataset and the dense segmentation of SNEMI3D dataset [2]. Two
large glial cells (not rendered) in AxonEM-H occupy the space between the blue
and pink neurons, leading to much fewer long axons compared to AxonEM-M.

Neuron Segmentation in EM. There are two mainstream approaches to instance
segmentation from 3D EM image volumes. One approach trains convolutional
neural networks (CNNs) to predict an intermediate representation for neuronal
boundaries [7, 27, 38] or inter-voxel affinities [31, 18, 12], which are then
often conservatively oversegmented with watershed [40]. For further
improvement, adjacent segments are agglomerated by a similarity measure [26,
18, 12], or with graph partitioning [5]. In the other approach, CNNs are
trained to recursively extend a segmentation mask for a single object of
interest [21, 15]. Recently, recurrent neural networks (RNNs) have also been
employed to extend multiple objects simultaneously [20, 13].

##  2 AxonEM Dataset

The proposed AxonEM dataset contains two EM image volumes cropped from the
published cortex datasets, one from mouse [11] (AxonEM-M) and the other from
human [30, 34] (AxonEM-H). We densely annotated axons longer than $5~{}\mu$m
in both volumes (Sec. 2.2), and performed basic analysis (Sec. 2.3).

###  2.1 Dataset Description

Two tissue blocks were imaged with serial section EM from layer 2/3 in the
primary visual cortex of an adult mouse and from layer 2 in the temporal lobe
of an adult human. The image volumes were 2$\times$ in-plane-downsampled to
the voxel resolutions of $7\times 7\times 40$ nm3 and $8\times 8\times 30$
nm3, respectively. We cropped out two $30\times 30\times 30~{}\mu$m3
subvolumes, AxonEM-M and AxonEM-H, intentionally avoiding large blood vessels
in order to contain more axon instances. We refer readers to the original
papers for more dataset details [11, 30].

Figure 3: Visualization of ground truth axon segmentation. We show 3D meshes
of axons with length either 5–6$~{}\mu$m or $>50~{}\mu$m. As expected, the
short axons are fragments near the volume boundaries that briefly enter and
leave the volume.

###  2.2 Dataset Annotation

For training, we annotated nine $512\times 512\times 50$ subvolumes to produce
dense ground truth segmentation, respectively, for each volume. For
evaluation, we annotated all axon segments that are longer than 5 $\mu$m. We
choose this threshold because segments shorter than 5 $\mu$m mostly
concentrate near the borders of the volumes, making it difficult to
confidently classify them into axons due to the lack of context. We took a
semi-automatic approach for annotation. Starting from the initial segmentation
provided by the original datasets [11, 30], proofreaders inspected potential
error locations suggested by automatic detection [19] in Neuroglancer [1], and
manually corrected remaining errors in VAST [6].

Initial Dense Segmentation. For AxonEM-M, we downloaded a publicly available
volume segmentation computed from a variant of 3D U-Net model [18, 11] and
applied the connected components method to obtain the initial segmentation to
start with. For AxonEM-H, we manually created small volumes of dense
segmentation to train the flood-filling network [15] and ran inference on the
entire AxonEM-H volume to generate the initial segmentation to start with.

Axon Identification. We first identified axons from the initial dense
segmentation, along with the skeletonization obtained by the TEASER algorithm
[28]. We computed for each segment the following statistics: volume, path
length, and width (i.e., average radius of all skeleton points). Based on
these stats, we used the K-means method to cluster the segments into $K=10$
groups with similar morphology. Then we visually inspected the 3D meshes of
all segments that are longer than 5 $\mu$m, and classified them into four
categories: glia, dendrite, axon, or uncertain. Instead of going through each
individual segments one by one, we pulled a small-sized batch from each of the
groups with similar morphology. This enabled us to collectively inspect such a
“minibatch” of morphologically similar segments all at once, thus making our
classification more efficient and effective.

Axon Proofreading. After identifying axons, we automatically detected the
potential error locations based on morphological features and biological prior
knowledges [19]. To correct remaining split errors, we measured the direction
of the axon skeletons at every end point, and then selected candidate pairs of
axons whose end points are close in space and pointing to each other in
direction. We examined these candidate pairs, as well as those axon skeletons
that terminate in the middle of the volume without reaching the boundaries,
which may indicate a premature termination caused by a split error. To fix
remaining merge errors, we inspected the junction points in the skeleton
graph, because some of them may exhibit a biologically implausible motif such
as the “X-shaped” junction [21].

Finalization. Four annotation experts with experience in axon reconstruction
were recruited to proofread and cross-check the axon identification and
proofreading results until there remains no disagreement between the
annotators.

###  2.3 Dataset Analysis

The physical size of our EM volumes is about 250$\times$ larger than the
previous SNEMI3D benchmark [2]. AxonEM-H and AxonEM-M have around 11.3K and
6.7K axon instances (longer than 5 $\mu$m), respectively, over 200$\times$
more than that of SNEMI3D. We compare the distribution of axon instance length
in Fig. 2, where our AxonEM dataset contains more and longer axons. Due to the
presence of two glial cells, in AxonEM-H there are fewer long, branching axons
than in AxonEM-M. In Fig. 3, we show 3D meshes of short (5–6 $\mu$m) and long
axons ($>$50 $\mu$m) from both volumes. Again, one can see the vacancy of
long, branching axons in AxonEM-H where the glial cells (not visualized)
occupy the space.

##  3 Methods

For 3D axon instance segmentation, we describe our choice of evaluation metric
(Sec. 3.1) and two state-of-the-art methods adopted for later benchmark (Sec.
4).

###  3.1 Task and Evaluation Metric

Our task is dense segmentation of 3D EM images of the brain, e.g., as in the
SNEMI3D benchmark challenge [2]. We focus, however, our attention only to the
accurate reconstruction of axons, which are challenging due to their long,
thin morphology and complicated branching patterns. To this end, our
evaluation is restricted to axons among all ground truth segments.

To measure the overall accuracy of axon reconstruction, we adopted the
expected run length (ERL, [15]) to estimate the average error-free path length
of the reconstructed axons. The previous ARAND metric used in SNEMI3D [2]
requires dense ground truth segmentation with voxel-level precision, which is
impractical for large-scale datasets. In contrast, the original ERL metric
focuses on the skeleton-level accuracy with a disproportionately larger
penalty on merge errors, in which two or more distinct objects are merged
together erroneously. In practice, due to skeletonization artifacts, the
original ERL metric may assign a good axon segment _zero_ run length if there
exists a merge error with just one outlier skeleton point that is mistakenly
placed near the ground truth segment boundaries. To mitigate this, we extended
the original ERL metric, which is not robust to outlier skeleton nodes, to
have a “tolerance” threshold of 50 skeleton nodes (around 2 $\mu$m in length)
to relax the condition to determine whether the ground truth skeleton
encounters a merge error in a segmentation.

###  3.2 State-of-the-Art Methods

We consider the top two methods for neuron instance segmentation on the
SNEMI3D benchmark [2]: U-Net models predicting the affinity graph (affinity
U-Net222https://github.com/seung-lab/DeepEM [18]) and flood-filling networks
(FFN333https://github.com/google/ffn [15]). For model details, we refer
readers to the original papers. Note that although their implementations are
publicly available, in practice, a substantial amount of engineering effort
may still be required in order to reproduce the reported performance.

##  4 Experiments

###  4.1 Implementation Details

For the FFN pipeline, we used the default configuration in [15] (convolution
module depth=9, FOV size=[33, 33, 17], movement deltas=[8, 8, 4]) and trained
separate models from scratch until accuracy saturates on the training data
(94% for AxonEM-M, 97% for AxonEM-H, and 92% for SNEMI3D). We then performed a
single pass of inference without the over-segmentation consensus and FFN
agglomeration steps [15]. For inference, we ran distributed FFN on $512\times
512\times 256$ voxel subvolumes with $64\times 64\times 32$ voxel overlap and
later reconciliated into one volume [32]. Training and inference were
conducted on a multi-GPU cluster and reconciliation was done on a workstation
[32]. Overall, for AxonEM-M and AxonEM-H, we used 32 NVidia A100 GPUs, spent
20 hours on training and 6.2 and 5.8 hours on inference for each dataset. For
SNEMI3D, we trained the net for 10 hours with 16 NVidia A100 GPUs, then
finished inference in 0.28 hours.

For the affinity U-Net pipeline, we first used the Google Cloud Platform (GCP)
compute engine to predict the affinity, using 12 NVidia T4 GPUs, each paired
with a n1-highmem-8 instance (8 vCPUs and 52 GB memory). Processing the
AxonEM-M and AxonEM-H datasets took 2.6 and 3.5 hours, respectively. From the
predicted affinities, we produced segmentation using the watershed-type
algorithm [40] and mean affinity agglomeration [18]. We used 8 e2-standard-32
instances (32 vCPUs and 128 GB memory). The watershed and agglomeration steps
together took 0.58 hours for AxonEM-M, and 0.75 hours for AxonEM-H.

Table 1: Segmentation results on SNEMI3D dataset. We report the ARAND score for the dense segmentation and ERL score for axon segments among them. The percentage numbers in parentheses represent the ratio of the baseline axon ERL with respect to the upper limit given by the ground truth ERL. Method |  ARAND$\downarrow$ |  ERL$\uparrow$ ($\mu$m)
---|---|---
Ground truth | 0.000 | 8.84 (100%)
PNI’s Aff. U-Net (mean aff. aggl.) [18] | 0.031 | 8.31 (94%)
Google’s FFN [15] | 0.029 | 8.20 (93%)
Our Impl. |  Aff. U-Net (mean aff. aggl.) | 0.038 | 8.22 (93%)
FFN (w/o aggl.) | 0.112 | 5.18 (59%)

###  4.2 Benchmark Results on SNEMI3D Dataset

We first show the results of our reproducing experiments on the SNEMI3D
benchmark in Table 1, where we obtained the adapted Rand (ARAND) error scores
from the challenge evaluation server. Our affinity U-Net error score (0.038)
was comparable to that of [18] (0.031), confirming that we are capable of
faithfully reproducing the affinity U-Net method. However, our FFN error score
(0.112) was severely worse than that of [15] (0.033) by a large margin,
suggesting that one should not consider our FFN baseline as state-of-the-art.

Next, we computed the axon ERL for our reproducing results, as well as for the
ground truth segmentation provided by the challenge organizer and some of the
top submission results [18, 15]. The ERL for the ground truth axons was 8.84
$\mu$m, which is the average path length of the ground truth axons. As
expected from the worst ARAND error score, our FFN result showed the lowest
ERL (5.18 $\mu$m) among others. The axon ERLs for the top submission results
[18, 15] and our affinity U-Net result were all above 8 $\mu$m, approaching
the upper limit given by the ground truth axon ERL (8.84 $\mu$m).
Interestingly, a lower (better) ARAND error score for dense segmentation does
not necessarily result in a higher (better) axon ERL. This is because the
ARAND error score is computed for dense segmentation including all segments
(axon, dendrite, and glia), whereas we computed the ERL selectively for axon.
For example, our affinity U-Net result was better in axon ERL, but worse in
ARAND, than [15] because of more problems in dendrite and glia.

Table 2: Axon segmentation results on AxonEM dataset (ERL). Method | AxonEM-H |  AxonEM-M
---|---|---
Ground truth | 28.5 $\mu$m (100%) | 43.6 $\mu$m (100%)
Our Impl. |  Aff. U-Net (mean aff. aggl.) | 18.5 $\mu$m (65%) | 38.5 $\mu$m (88%)
FFN (w/o aggl.) | 9.6 $\mu$m (34%) | 9.4 $\mu$m (22%)
Figure 4: Qualitative results of our affinity U-Net baseline on AxonEM.
Failure cases are often caused by invaginating structures, thin parts, and
image defects.

###  4.3 Benchmark Results on AxonEM Dataset

We provide axon segmentation baselines for the AxonEM dataset by reproducing
two state-of-the-art methods, affinity U-Net [18] and FFN [15]. One should be
cautious that our FFN baseline does not fully reproduce the state-of-the-art
accuracy of the original method by [15], due to the reasons detailed below.

Quantitative Results. As shown in Table 2, our affinity U-Net baseline
achieved a relatively high axon ERL (88% with respect to the ground truth) on
AxonEM-M, whereas its ERL for AxonEM-H was relatively lower (65%). The axon
ERL of our FFN baseline was consistently and significantly lower (22% on
AxonEM-H and 34% on AxonEM-H) than our affinity U-Net baseline, being
consistent with our SNEMI3D reproducing results in the previous section (Table
1). Due to the lack of open-source code for several inference steps, our FFN
baseline was not a faithful reproduction of the full reconstruction pipeline
proposed by [15]. Our FFN baseline lacks both the multi-scale over-
segmentation consensus and FFN agglomeration steps in [15], which are crucial
in achieving a very low merge error rate while maximizing ERL. A much better
FFN baseline by faithfully reproducing the full reconstruction pipeline
remains to be done in the future.

Qualitative Results. We show both successful and unsuccessful axon
segmentation results of our affinity U-Net baselines (Fig. 4). Due to the thin
parts, the axon reconstruction is more vulnerable to image defects like
missing section, staining artifact, knife mark, and misalignment [17] than
other structures.

##  5 Conclusion

In this paper, we have introduced a large-scale 3D EM dataset for axon
instance segmentation, and provided baseline results by re-implementing two
state-of-the-art methods. We expect that the proposed densely annotated large-
scale axon dataset will not only foster development of better axon
segmentation methods, but also serve as a test bed for various other
applications.

#### Acknowledgment.

KL, RL, and JAB were supported by the Intelligence Advanced Research Projects
Activity (IARPA) via Department of Interior/ Interior Business Center
(DoI/IBC) contract number D16PC0005, NIH/NIMH (U01MH114824, U01MH117072, RF1
MH117815), NIH/NINDS (U19NS104648, R01NS104926), NIH/NEI (R01EY027 036), and
ARO (W911NF-12-1-0594), and are also grateful for assistance from Google,
Amazon, and Intel. DW, ZL, JL, and HP were partially supported by NSF award
IIS-1835231. I. A-C would like to acknowledge the support of the Beca Leonardo
a Investigadores y Creadores Culturales 2020 de la Fundación BBVA. We thank
Viren Jain, Michał Januszewski and their team for generating the initial
segmentation for AxonEM-H, and Daniel Franco-Barranco for setting up the
challenge using AxonEM.

Declaration of Interests. KL, RL, and JAB disclose financial interests in
Zetta AI LLC.

## References

  * [1] Neuroglancer, https://github.com/google/neuroglancer
  * [2] SNEMI3D EM segmentation challenge and dataset, http://brainiac2.mit.edu/SNEMI3D/home
  * [3] Abbott, L.F., Bock, D.D., Callaway, E.M., Denk, W., Dulac, C., Fairhall, A.L., Fiete, I., Harris, K.M., et al.: The mind of a mouse. Cell (2020)
  * [4] Bégin, S., Dupont-Therrien, O., Bélanger, E., Daradich, A., Laffray, S., et al.: Automated method for the segmentation and morphometry of nerve fibers in large-scale cars images of spinal cord tissue. Biomedical optics express (2014)
  * [5] Beier, T., Pape, C., Rahaman, N., Prange, T., Berg, S., Bock, D.D., Cardona, A., Knott, G.W., Plaza, S.M., Scheffer, L.K., et al.: Multicut brings automated neurite segmentation closer to human performance. Nature methods (2017)
  * [6] Berger, D.R., Seung, H.S., Lichtman, J.W.: Vast (volume annotation and segmentation tool): efficient manual and semi-automatic labeling of large 3D image stacks. Frontiers in neural circuits (2018)
  * [7] Ciresan, D., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. In: NeurIPS (2012)
  * [8] Cuisenaire, O., Romero, E., Veraart, C., Macq, B.M.: Automatic segmentation and measurement of axons in microscopic images. In: Medical Imaging (1999)
  * [9] Dorkenwald, S., McKellar, C., et al.: Flywire: Online community for whole-brain connectomics. bioRxiv (2020)
  * [10] Dorkenwald, S., Schubert, P.J., Killinger, M.F., Urban, G., Mikula, S., Svara, F., Kornfeld, J.: Automated synaptic connectivity inference for volume electron microscopy. Nature methods (2017)
  * [11] Dorkenwald, S., Turner, N.L., Macrina, T., Lee, K., Lu, R., Wu, J., Bodor, A.L., Bleckert, A.A., Brittain, D., Kemnitz, N., et al.: Binary and analog variation of synapses between cortical pyramidal neurons. BioRxiv (2019)
  * [12] Funke, J., Tschopp, F., Grisaitis, W., Sheridan, A., Singh, C., Saalfeld, S., Turaga, S.C.: Large scale image segmentation with structured loss based deep learning for connectome reconstruction. TPAMI (2018)
  * [13] Gonda, F., Wei, D., Pfister, H.: Consistent recurrent neural networks for 3d neuron segmentation. In: ISBI (2021)
  * [14] Helmstaedter, M.: Cellular-resolution connectomics: challenges of dense neural circuit reconstruction. Nature Methods (2013)
  * [15] Januszewski, M., Kornfeld, J., Li, P.H., Pope, A., Blakely, T., Lindsey, L., Maitin-Shepard, J., Tyka, M., Denk, W., Jain, V.: High-precision automated reconstruction of neurons with flood-filling networks. Nature Methods (2018)
  * [16] Kornfeld, J., Denk, W.: Progress and remaining challenges in high-throughput volume electron microscopy. Current Opinion in Neurobiology (2018)
  * [17] Lee, K., Turner, N., Macrina, T., Wu, J., Lu, R., Seung, H.S.: Convolutional nets for reconstructing neural circuits from brain images acquired by serial section electron microscopy. Current Opinion in Neurobiology (2019)
  * [18] Lee, K., Zung, J., Li, P., Jain, V., Seung, H.S.: Superhuman accuracy on the snemi3d connectomics challenge. arXiv:1706.00120 (2017)
  * [19] Matejek, B., Haehn, D., Zhu, H., Wei, D., Parag, T., Pfister, H.: Biologically-constrained graphs for global connectomics reconstruction. In: CVPR (2019)
  * [20] Meirovitch, Y., Mi, L., Saribekyan, H., Matveev, A., Rolnick, D., Shavit, N.: Cross-classification clustering: An efficient multi-object tracking technique for 3-d instance segmentation in connectomics. In: CVPR (2019)
  * [21] Meirovitch, Y., et al.: A multi-pass approach to large-scale connectomics. arXiv preprint arXiv:1612.02120 (2016)
  * [22] Mesbah, R., McCane, B., Mills, S.: Deep convolutional encoder-decoder for myelin and axon segmentation. In: IVCNZ (2016)
  * [23] More, H.L., Chen, J., Gibson, E., Donelan, J.M., Beg, M.F.: A semi-automated method for identifying and measuring myelinated nerve fibers in scanning electron microscope images. Journal of neuroscience methods (2011)
  * [24] Motta, A., Berning, M., Boergens, K.M., Staffler, B., Beining, M., Loomba, S., Hennig, P., Wissler, H., Helmstaedter, M.: Dense connectomic reconstruction in layer 4 of the somatosensory cortex. Science (2019)
  * [25] Naito, T., Nagashima, Y., Taira, K., Uchio, N., Tsuji, S., Shimizu, J.: Identification and segmentation of myelinated nerve fibers in a cross-sectional optical microscopic image using a deep learning model. Journal of neuroscience methods (2017)
  * [26] Nunez-Iglesias, J., Kennedy, R., Parag, T., Shi, J., Chklovskii, D.B.: Machine learning of hierarchical clustering to segment 2d and 3d images. PloS one (2013)
  * [27] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [28] Sato, M., Bitter, I., Bender, M., Kaufman, A., Nakajima, M.: TEASAR: tree-structure extraction algorithm for accurate and robust skeletons. In: Pacific Conference on Computer Graphics and Applications
  * [29] Scheffer, L.K., Xu, C.S., Januszewski, M., Lu, Z., Takemura, S.y., Hayworth, K.J., Huang, G.B., Shinomiya, K., Maitlin-Shepard, J., Berg, S., et al.: A connectome and analysis of the adult drosophila central brain. Elife (2020)
  * [30] Shapson-Coe, A., et al.: A connectomic study of a petascale fragment of human cerebral cortex. bioRxiv (2021)
  * [31] Turaga, S.C., Murray, J.F., Jain, V., Roth, F., Helmstaedter, M., Briggman, K., Denk, W., Seung, H.S.: Convolutional networks can learn to generate affinity graphs for image segmentation. Neural computation (2010)
  * [32] Vescovi, R., Li, H., Kinnison, J., Keçeli, M., Salim, M., Kasthuri, N., Uram, T.D., Ferrier, N.: Toward an automated hpc pipeline for processing large scale electron microscopy data. In: XLOOP (2020)
  * [33] Wang, Y.Y., Sun, Y.N., Lin, C.C.K., Ju, M.S.: Segmentation of nerve fibers using multi-level gradient watershed and fuzzy systems. AI in Medicine (2012)
  * [34] Wei, D., Lin, Z., Franco-Barranco, D., et al.: MitoEM dataset: Large-scale 3d mitochondria instance segmentation from em images. In: MICCAI (2020)
  * [35] White, J.G., Southgate, E., Thomson, J.N., Brenner, S.: The structure of the nervous system of the nematode caenorhabditis elegans. Philosophical Transactions of the Royal Society B: Biological Sciences (1986)
  * [36] Zaimi, A., Duval, T., Gasecka, A., Côté, D., Stikov, N., Cohen-Adad, J.: AxonSeg: open source software for axon and myelin segmentation and morphometric analysis. Frontiers in neuroinformatics (2016)
  * [37] Zaimi, A., Wabartha, M., Herman, V., Antonsanti, P.L., Perone, C.S., Cohen-Adad, J.: Axondeepseg: automatic axon and myelin segmentation from microscopy data using convolutional neural networks. Scientific reports (2018)
  * [38] Zeng, T., Wu, B., Ji, S.: Deepem3d: approaching human-level performance on 3d anisotropic em image segmentation. Bioinformatics (2017)
  * [39] Zhao, X., Pan, Z., Wu, J., Zhou, G., Zeng, Y.: Automatic identification and morphometry of optic nerve fibers in electron microscopy images. Computerized Medical Imaging and Graphics (2010)
  * [40] Zlateski, A., Seung, H.S.: Image segmentation by size-dependent single linkage clustering of a watershed basin graph. arXiv:1505.00249 (2015)


Guofeng<EMAIL_ADDRESS>Cristiano
<EMAIL_ADDRESS>Fabio<EMAIL_ADDRESS>Jian
<EMAIL_ADDRESS>Elisa<EMAIL_ADDRESS>Nicu
<EMAIL_ADDRESS>Qiang<EMAIL_ADDRESS>Global Big Data
Technologies Centre
University of Technology Sydney
Sydney, Australia Multimedia and Human Understanding Group
University of Trento
Trento, Italy Technologies of Vision Lab
Fondazione Bruno Kessler
Trento, Italy BMVC Author Guidelines

# Data Augmentation-free Unsupervised Learning for 3D Point Cloud
Understanding

###### Abstract

Unsupervised learning on 3D point clouds has undergone a rapid evolution,
especially thanks to data augmentation-based contrastive methods. However,
data augmentation is not ideal as it requires a careful selection of the type
of augmentations to perform, which in turn can affect the geometric and
semantic information learned by the network during self-training. To overcome
this issue, we propose an augmentation-free unsupervised approach for point
clouds to learn transferable point-level features via soft clustering, named
SoftClu. SoftClu assumes that the points belonging to a cluster should be
close to each other in both geometric and feature spaces. This differs from
typical contrastive learning, which builds similar representations for a whole
point cloud and its augmented versions. We exploit the affiliation of points
to their clusters as a proxy to enable self-training through a pseudo-label
prediction task. Under the constraint that these pseudo-labels induce the
equipartition of the point cloud, we cast SoftClu as an optimal transport
problem. We formulate an unsupervised loss to minimize the standard cross-
entropy between pseudo-labels and predicted labels. Experiments on downstream
applications, such as 3D object classification, part segmentation, and
semantic segmentation, show the effectiveness of our framework in
outperforming state-of-the-art techniques [code].

##  1 Introduction

The rapid progress of 3D capturing devices, such as 3D laser scanners and
depth sensors, led to convenient and effective ways to process 3D data, which,
if combined with RGB images, can further improve the understanding of
environments. Applications like robotic navigation [Biswas and Veloso(2012),
Zhou et al.(2022)Zhou, Wang, Poiesi, Qin, and Wan], autonomous driving [Li et
al.(2020)Li, Ma, Zhong, Liu, Chapman, Cao, and Li] and exploration [Wang and
Bue(2020)], and augmented and virtual reality [Park et al.(2008)Park, Lepetit,
and Woo] are among the major reasons for a higher attention toward 3D data
understanding.

Learning discriminative and transferable point cloud features is a crucial
problem in the area of 3D shape understanding [Lin et al.(2021)Lin, Chen, and
Jia, Poiesi and Boscaini(2022)], as it allows efficient training of downstream
tasks, such as object detection [Shi et al.(2020a)Shi, Guo, Jiang, Wang, Shi,
Wang, and Li] and tracking [Yin et al.(2021)Yin, Zhou, and Krahenbuhl],
segmentation [Xu et al.(2020)Xu, Zhou, and Qiao], reconstruction [Yuan et
al.(2018)Yuan, Khot, Held, Mertz, and Hebert], classification [Qi et
al.(2017)Qi, Su, et al.], and registration [Huang et al.(2020)Huang, Mei, and
Zhang, Mei(2021), Mei et al.(2022)Mei, Huang, Zhang, and Wu]. Therefore,
learning from unlabeled or partially labeled data to alleviate human labeling
efforts is an emerging research topic in point cloud understanding. Along this
line, unsupervised representation learning is an attractive, yet potent,
alternative approach to learning features without human intervention [Eckart
et al.(2021)Eckart, Yuan, Liu, and Kautz].

Unsupervised learning approaches can be broadly categorized as generative or
discriminative [Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond,
Buchatskaya, Doersch, Pires, Guo, Azar, et al.]. The former includes self-
reconstruction or auto-encoding [Yang et al.(2018)Yang, Feng, Shen, and Tian],
generative adversarial network [Sarmad et al.(2019)Sarmad, Lee, et al.], and
auto-regressive [Sun et al.(2020)Sun, Wang, Liu, Siegel, and Sarma] methods.
These methods can map an input point cloud into a global latent representation
[Rao et al.(2020)Rao, Lu, and Zhou, Shi et al.(2020b)Shi, Xu, Yuan, and Fang],
or a latent distribution in the variational case [Hassani and Haley(2019), Han
et al.(2019)Han, Wang, et al.] through an encoder and then attempt to
reconstruct the input by a decoder. Generative methods can be effective to
model high-level and structural properties of the input point clouds. However,
because they are sensitive to Euclidean transformations, they typically assume
that all 3D objects have the same pose in a given category [Sanghi(2020)].

Unlike generative methods, discriminative methods learn to predict or
discriminate augmented versions of the input. These methods can yield rich
latent representations for downstream tasks [Wang et al.(2020)Wang, Liu, Yue,
Lasenby, and Kusner]. Examples of these include contrastive methods [Du et
al.(2021)Du, Gao, Hu, and Li, Rao et al.(2020)Rao, Lu, and Zhou,
Sanghi(2020)], which have shown remarkable results for unsupervised
representation learning. Contrastive methods also promote learning of
rotation-invariant representations via data augmentation [Poursaeed et
al.(2020)Poursaeed, Jiang, Qiao, Xu, and Kim]. Typically, these algorithms
require several negative samples and heavily depend on the selection criteria
to mine negatives [Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond,
Buchatskaya, Doersch, Pires, Guo, Azar, et al., Chen and He(2021)]. Often,
they require large batch sizes, memory banks, or customized strategies to
retrieve informative pairs [Grill et al.(2020)Grill, Strub, Altché, Tallec,
Richemond, Buchatskaya, Doersch, Pires, Guo, Azar, et al.]. Moreover, it is
somewhat unclear what constitutes an effective semantics-preserving data
augmentation strategy given that a point cloud is typically defined as a set
of 3D coordinates. Any disturbance of the original geometry, such as cropping
or view-based occlusions, could potentially degrade its semantics [Eckart et
al.(2021)Eckart, Yuan, Liu, and Kautz]; for example, random crops of a point
cloud may correspond to different objects and introduce inconsistent learning
signals. For this reason, contrastive approaches need humans to carefully
design combinations of data augmentations to learn informative
representations. On the other hand, training on whole object instances can
lead to the learning of global representations, which in turn can produce
fewer discriminant representations as local geometric differences may be
disregarded [Rao et al.(2020)Rao, Lu, and Zhou, Han et al.(2019)Han, Wang, et
al., Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany]. Therefore, our
first motivation is to design a data augmentation-free unsupervised learning
approach to avoid the inconvenience of building chains of ad-hoc combinations
of data augmentations. Second, we develop an unsupervised method that is not
based on global features but instead can optimize local features, which
facilitates the network to learn 3D spatial geometric information of point
clouds.

In this paper, we propose an unsupervised method to learn informative point-
level representations of 3D point clouds without using data augmentation. Our
framework learns cluster affiliation scores to softly group the 3D points of
each point cloud into a given number of geometric partitions, i.e. through
soft clustering. We learn point-level feature representations by minimizing
the standard cross-entropy of a single equation, which is the result of an EM-
like algorithm [Moon(1996)]. The Expectation step employs an optimal transport
[Peyré et al.(2019)Peyré, Cuturi, et al.] based clustering algorithm to
generate point-level pseudo-labels, i.e. focusing on local geometric
information. In particular, we softly label points based on their distance
from the centroids in both feature and geometric spaces, with the constraint
that labels partition data in equally-sized subsets (uniform distribution).
Optimal transport serves as a potent means for comparing probability
distributions with each other, as well as for producing optimal mappings to
minimize distances [Peyré et al.(2019)Peyré, Cuturi, et al.]. The Maximization
step adapts the E-step for a point-to-cluster loss to optimize the metric
learning network. Our approach learns the partitioning network itself and
softly assigns points into geometrically coherent overlapping clusters,
overcoming the weakness of conventional GMM and K-means that involve expensive
iterative procedures. In doing so, we avoid data augmentation that might
degrade its geometric coherency and thus its semantic information. Our
approach is inspired by DeepCluster [Caron et al.(2018)Caron, Bojanowski,
Joulin, and Douze], SeLa [Asano et al.(2020)Asano, Rupprecht, and Vedaldi] and
SwAV [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin]
but it differs from them, as they implement clustering in the feature space at
the instance level, they use data-augmentation, and they may degrade the
geometric information when used with 3D data. We show that pre-training on
datasets using SoftClu can improve the performance of a range of downstream
tasks and outperforms the recent state-of-the-art methods without any data
augmentation and also across different domains.

To summarize, our contributions are:

  * •

We propose a data augmentation-free unsupervised method, which does not rely
on data augmentations, negative pair sampling, and large batches, to learn
transferable point-level features on a 3D point cloud;

  * •

We extend the pseudo-label prediction to an optimal transport problem, which
can be efficiently solved by using an efficient variant of the Sinkhorn-Knopp
[Cuturi(2013)] algorithm;

  * •

We conduct thorough experiments, and SoftClu achieves state-of-the-art
performance without having to use data augmentation.

##  2 Related Work

In this section, we briefly review existing works related to unsupervised
learning on point clouds, which can be classified into two categories:
generative and discriminative methods.

Generative methods. Generative methods learn features via self-reconstruction
[Han et al.(2019)Han, Wang, et al.]. For instance, FoldingNet [Yang et
al.(2018)Yang, Feng, Shen, and Tian] leverages a graph-based encoder and a
folding-based decoder to deform a canonical 2D grid onto the surface of a
point cloud. L2g [Liu et al.(2019)Liu, Han, et al.] uses a local-to-global
auto-encoder to simultaneously learn the local and global structure of point
clouds. In [Chen et al.(2019)Chen, Duan, Yang, Li, Feng, and Tian], a graph-
based decoder with a learnable graph topology is used to push the codeword to
preserve representative features. In [Achlioptas et al.(2018)Achlioptas,
Diamanti, Mitliagkas, and Guibas] a combination of hierarchical Bayesian and
generative models is trained to generate plausible point clouds. GraphTER [Gao
et al.(2020)Gao, Hu, and Qi] self-trains a feature encoder by reconstructing
node-wise transformations from the representations of the original and
transformed graphs. However, generative models are sensitive to
transformations, weakening the learning of robust point cloud representations
for different downstream tasks. Moreover, it is not always feasible to
reconstruct back the shape from pose-invariant features [Sanghi(2020)].

Discriminative methods. Discriminative methods are based on auxiliary
handcrafted prediction tasks to learn point cloud representations. Jigsaw3D
[Sauder and Sievers(2019)] uses a 3D jigsaw puzzle approach as the self-
supervised learning task. Recently, contrastive approaches [Rao et
al.(2020)Rao, Lu, and Zhou, Sanghi(2020), Chen and He(2021), Xie et
al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany], which are robust to
transformation, achieved state-of-the-art performance. Info3D [Sanghi(2020)]
maximizes the mutual information between the 3D shape and a geometric
transformed version of the 3D shape. PointContrast [Xie et al.(2020)Xie, Gu,
Guo, Qi, Guibas, and Litany] is the first to research a unified framework of
the contrastive paradigm for 3D representation learning. We argue that the
success of contrastive methods relies on the correct design of negative mining
strategies and on the correct choice of data augmentations that should not
affect the semantics of the input. To mitigate these issues, we propose an
unsupervised learning method, SoftClu, which is formulated by implementing an
EM-like algorithm and provides point-level supervision to extract
discriminative point-wise features. SoftClu needs no data augmentation
procedures to learn informative representations.

##  3 Proposed method

We formulate the problem of representation learning as a soft-clustering
problem. Figure 1 illustrates our framework that consists of three steps:
prototype computation, soft-label assignment, and optimization. Given a 3D
point cloud as an unordered set $\bm{\mathcal{P}}=\\{\bm{p}_{i}\\}_{i=1}^{N}$
of $N$ points where each point $\bm{p}_{i}\in\mathbb{R}^{3}$ is represented by
a 3D coordinate $\bm{p}_{i}=\\{x,y,z\\}$, our goal is to train, in an
unsupervised way, a feature encoder $f_{\varphi}$ with parameters $\varphi$
($e.g.$, PointNet) that extracts informative point-wise features
$\bm{\mathcal{F}}=\\{f_{\varphi}(\bm{p}_{i})\\}_{i=1}^{N}$ from
$\bm{\mathcal{P}}$. To this end, we apply a segmentation head $\phi_{\alpha}$
that takes as input $\bm{\mathcal{F}}$ and outputs joint log probabilities,
and a softmax operator that acts on log probabilities to generate a
classification score matrix $\bm{S}$. The prototype computation block
estimates the $J$ cluster centroids (prototypes) $\bm{C}^{E}$ and $\bm{C}^{F}$
to represent each partitions. Next, soft-labels
$\bm{\gamma}_{ij}\in\bm{\gamma}$ of each input point $\bm{p}_{i}$ are based on
these prototypes and we use the Sinkhorn-Knopp [Cuturi(2013)] algorithm to
perform the soft-label assignment, i.e., $\bm{\gamma}$ softly groups
$\bm{\mathcal{P}}$ into partitions. $\bm{\gamma}_{ij}\in[0,1]$ is a soft-label
score that point $\bm{p}_{i}$ belongs to cluster $j$. The final optimization
step is to minimize the average cross-entropy loss $\mathcal{L}_{tot}$ between
the soft-label $\bm{\gamma}$ and the predicted class probability $\bm{S}$.

Figure 1: The architecture of our SoftClu. It consists of three steps:
prototype computation (blue line), soft-label $\bm{\gamma}$ assignment (green
line), and optimization (red line).

Prototype computation. We begin by computing a prototype for each cluster
(partition) as the most representative feature for a set of points.
Specifically, we use the point-wise features
$\mathcal{F}=\\{\bm{f}_{i}\\}_{i=1}^{N}$, where
$\bm{f}_{i}=f_{\varphi}(\bm{p}_{i})$, to compute a set of classification
scores $\bm{S}$ as $\bm{S}=\\{\sigma(\phi_{\alpha}(\bm{f}_{i})\\}_{i=1}^{N}$.
$\sigma$ is the softmax operation, and $\phi_{\alpha}$ is the segmentation
layer with parameters $\alpha$. For each feature $\bm{f}_{i}$, $\phi_{\alpha}$
produces a probability score $s_{ij}$ indicating the likelihood that
$\bm{p}_{i}$ belongs to partition $j$. We use two types of prototypes as the
representatives for each category, one in the feature space and another in the
geometric space. Specifically, according to the type, we compute $J$
prototypes as the weighted average of the features $\bm{\mathcal{F}}$ or 3D
coordinates $\bm{\mathcal{P}}$ based on their scores $\bm{S}$. Let
$\bm{C}^{E}=\\{\bm{c}^{E}_{j}\\}_{j=1}^{J}$ and
$\bm{C}^{F}=\\{\bm{c}^{F}_{j}\\}_{j=1}^{J}$ be the set of prototypes in the
geometric and the features space, respectively, which are defined as

$\vspace{-.2cm}\bm{c}^{E}_{j}=\frac{1}{\sum_{i=1}^{N}s_{ij}}\sum_{i=1}^{N}s_{ij}\bm{p}_{i},\quad\bm{c}^{F}_{j}=\frac{1}{\sum_{i=1}^{N}s_{ij}}\sum_{i=1}^{N}s_{ij}\bm{f}_{p_{i}}.$
(1)

Soft-label assignment. We introduce the soft-label assignment step that labels
points based on their distance to the prototypes estimated by Eq. (1). If we
only use features for soft-label assignment, this would highly likely produce
disconnected and scattered clusters. Hence, we concatenate point coordinates
with the features so that the label is more localized. Specifically, we encode
the pseudo-labels $\bm{\gamma}=\\{\gamma_{ij}\in[0,1]\\}_{i,j}^{N,J}$ as
posterior distributions, i.e. soft-labels, satisfying
$\sum_{j=1}^{J}\gamma_{ij}=1$. $\gamma_{ij}$ is the posterior probability that
$\bm{p}_{i}$ belongs to partition $j$. We base the assignment of soft-labels
to the respective points on the prototypes $\bm{C}^{E}$ and $\bm{C}^{F}$, and
by following two assumptions:

  1. i)

Cluster cohesion: If a point $\bm{p}_{i}$ belongs to partition $j$, point
$\bm{p}_{i}$ and prototype $\bm{c}^{E}_{j}$ should have the shortest distance
among the distances of $\bm{p}_{i}$ with other prototypes in $\bm{C}^{E}$. The
same holds true in the feature space.

  2. ii)

Uniform distribution: Each point cloud is assumed to be segmented into
equally-sized partitions of $\left\lfloor\frac{N}{J}\right\rfloor$ elements,
where $\left\lfloor\cdot\right\rfloor$ indicates the greatest integer less
than or equal to its argument.

Assumption i) inspires us to label points based on their distance from the
centroids. It can be formalized as an expression, i.e., if $\bm{p}_{i}$
belongs to cluster $j$, then
$\|\bm{p}_{i}-\bm{c}^{E}_{j}\|_{2}\leq\|\bm{p}_{i}-\bm{c}^{E}_{k}\|_{2},\|\bm{f}_{i}-\bm{c}^{F}_{j}\|_{2}\leq\|\bm{f}_{i}-\bm{c}^{F}_{k}\|_{2}$
and $\gamma_{ij}\geq\gamma_{ik},k\neq j,k=1,\cdots,J$. $\|\cdot\|_{2}$ is the
L2 norm. This can be ensured by minimizing the following objective,

$\vspace{-.2cm}\min_{\bm{\gamma}}\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{J}\left(\lambda\|\bm{p}_{i}-\bm{c}^{E}_{j}\|^{2}_{2}+\left(1-\lambda)\right\|\bm{f}_{i}-\bm{c}^{F}_{j}\|^{2}_{2}\right)\gamma_{ij},$
(2)

where $\lambda\in[0,1]$ is a learned parameter. For convenience, we define the
following matrix form
$\bm{D}=\lambda\bm{D}^{E}+\left(1-\lambda\right)\bm{D}^{F}$, where
$\bm{D}^{F}=\\{\|\bm{f}_{i}-\bm{c}^{F}_{j}\|^{2}_{2}\\}_{i,j}^{N,J}$ and
$\bm{D}^{E}=\\{\|\bm{p}_{i}-\bm{c}^{E}_{j}\|^{2}_{2}\\}_{i,j}^{N,J}$ are
matrices of size equal to $N\times J$. Then, Eq. (2) can be rewritten as:

$\vspace{-.2cm}\min_{\bm{\gamma}}\left<\frac{\bm{\gamma}}{N},\bm{D}\right>,$
(3)

where $\left<\cdot,\cdot\right>$ is the Frobenius matrix dot product.

Assumption ii) is formulated in a constraint condition as
$\sum_{i=1}^{N}\gamma_{ij}=\frac{N}{J}$, which can mitigate the problem that
all data points are assigned to a single (arbitrary) label. Therefore, based
on $\sum_{i=1}^{N}\gamma_{ij}=\frac{N}{J}$ and the property of the posterior
probability $\sum_{j=1}^{J}\gamma_{ij}=1$, $\bm{\gamma}$ satisfies the
following constraints

$\vspace{-.2cm}\frac{1}{N}\bm{\gamma}^{\top}\bm{1}_{N}=\frac{1}{J}\bm{1}_{J},\frac{1}{N}\bm{\gamma}\bm{1}_{J}=\frac{1}{N}\bm{1}_{N},$
(4)

where $\bm{1}_{k}(k=N,J)$ denotes the vector of ones in dimension $k$.

Let $\bm{\Gamma}=\frac{\bm{\gamma}}{N}$ with elements defined as
$\Gamma_{ij}=\frac{\gamma_{ij}}{N}$. $\bm{\Gamma}$ satisfies
$\sum_{ij}^{NJ}\bm{\Gamma}_{ij}=1$. By replacing the variable $\bm{\gamma}$
with $\bm{\Gamma}$ in Eq. (3) and Eq. (4), the joint objective of assumptions
i) and ii) can be formulated as an optimal transport (OT) problem [Peyré et
al.(2019)Peyré, Cuturi, et al.] as

$\min_{\bm{\Gamma}}\left<\bm{\Gamma},\bm{D}\right>,~{}\mbox{s.t.}~{}\bm{\Gamma}^{\top}\bm{1}_{N}=\frac{1}{J}\bm{1}_{J},\bm{\Gamma}\bm{1}_{J}=\frac{1}{N}\bm{1}_{N}.$
(5)

The minimization of Eq. (5) can be solved in polynomial time as a linear
program. However, the linear program involves millions of data points and
thousands of classes and traditional algorithms hardly scale to large problems
[Cuturi(2013)]. We address this issue by adopting an efficient version of the
Sinkhorn-Knopp algorithm [Cuturi(2013)]. Our implementation of the Sinkhorn-
Knopp algorithm is described in Alg. 2 (Appendix). This requires the following
regularization term

$\min_{\bm{\Gamma}}\left<\bm{\Gamma},\bm{D}\right>-\epsilon
H\left(\bm{\Gamma}\right),\quad\mbox{s.t.}~{}\bm{\Gamma}^{\top}\bm{1}_{N}=\frac{1}{J}\bm{1}_{J},\,\,\,\bm{\Gamma}\bm{1}_{J}=\frac{1}{N}\bm{1}_{N},$
(6)

where $H\left(\bm{\Gamma}\right)=\left<\bm{\Gamma},\log\bm{\Gamma}-1\right>$
denotes the entropy of $\bm{\Gamma}$ and $\epsilon>0$ is a regularization
parameter. For very small $\epsilon$, optimizing Eq. (6) is equivalent to
optimizing Eq. (5), but even for moderate values of $\epsilon$, the objective
tends to have approximately the same optimizer [Cuturi(2013)]. The larger the
$\epsilon$, the faster the convergence, please refer to [Cuturi(2013)] for
details. In our case, using a fixed $\epsilon=1e-3$ is appropriate as we are
interested in the final clustering and representation learning results, rather
than in solving the transport problem exactly. The solution to Eq. (6) takes
the form of the following normalized exponential matrix [Cuturi(2013)],

$\vspace{-.2cm}\bm{\Gamma}=\mbox{diag}\left(\bm{\mu}\right)\exp\left(\bm{D}\big{/}\epsilon\right)\mbox{diag}\left(\bm{\nu}\right),$
(7)

where $\bm{\mu}=(\mu_{1},\mu_{2},\cdots,\mu_{N})$ and
$\bm{\nu}=(\nu_{1},\nu_{2},\cdots,\nu_{J})$ are renormalization vectors in
$\mathbb{R}^{N}$ and $\mathbb{R}^{J}$. The vectors $\bm{\mu}$ and $\bm{\nu}$
can be obtained by iterating the updates via
$\bm{\mu}_{i}=\left[\exp\left(\bm{D}\big{/}\epsilon\right)\bm{\nu}\right]^{-1}_{i}$
and
$\bm{\nu}_{j}=\left[\exp\left(\bm{D}\big{/}\epsilon\right)^{\top}\bm{\mu}\right]^{-1}_{j}$
with initial values $\bm{\mu}=\frac{1}{N}\bm{1}_{N}$ and
$\bm{\nu}=\frac{1}{J}\bm{1}_{J}$, respectively. The initialization of
$\bm{\mu}$ and $\bm{\nu}$ can be any distribution, and choosing the
constraints as initial values allow a faster convergence [Cuturi(2013)].
$\left[\cdot\right]^{-1}_{j}$ defines as the inverse value of the $j^{th}$
element of its argument. In all our experiments, we use 20 iterations as we
found it works well in practice. After solving Eq. (7), we can infer the soft-
label matrix as $\bm{\gamma}=N\cdot\bm{\Gamma}$.

Optimization. The optimization step follows an EM-like scheme where the
Expectation step E optimizes prototypes and soft labels, while the
Maximization step M optimizes the trained parameters for representation
learning. Each step can be detailed as follows:

  * •

E: Given the current encoder and segmentation layer, we compute prototypes
$\bm{C^{E}}$ and $\bm{C}^{F}$ following Eq. (1), and obtain soft-labels
$\bm{\gamma}$ through $\bm{\gamma}=N\cdot\bm{\Gamma}$.

  * •

M: Given the current soft-labels $\bm{\gamma}$ from step E, we optimize the
encoder $f_{\varphi}$ and segmentation layer $\phi_{\alpha}$ parameters.

During E, we solve the OT problem with the Sinkhorn-Knopp algorithm. During M,
we minimize the segmentation loss based on the resulting soft labels, such as

$\vspace{-.2cm}\mathcal{L}_{soft}(\bm{\gamma},\bm{S})=-\frac{1}{N}\left<\bm{\gamma},\log\bm{S}\right>=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{J}\bm{\gamma}_{ij}\log
s_{ij},$ (8)

which corresponds to the minimization of the standard cross-entropy loss
between soft-labels $\bm{\gamma}$ and predictions $\bm{S}$. However, the
optimization of Eq. (8) does not ensure encoder $f_{\varphi}$ from predicting
the same features for all the points, i.e. all centroids collapse into the
same vector. We further promote centroids separation by minimizing the
orthogonal regularization loss
$\mathcal{L}_{orth}(\bm{C})=\|{\bm{C}^{E}}_{*}^{\top}\bm{C}^{E}_{*}-\bm{I}\|_{Fr}+\|{\bm{C}^{F}}_{*}^{\top}\bm{C}^{F}_{*}-\bm{I}\|_{Fr}$,
where
$\bm{C}^{k}_{*}=[\frac{\bm{c}^{k}_{1}}{\|\bm{c}^{k}_{1}\|_{2}},\frac{\bm{c}^{k}_{2}}{\|\bm{c}^{k}_{2}\|_{2}},\cdots,\frac{\bm{c}^{k}_{J}}{\|\bm{c}^{k}_{J}\|_{2}}]$
with $k=E,F$, and $\|\cdot\|_{Fr}$ is Frobenius norm. We define the final
objective loss for the M step as

$\vspace{-.2cm}\mathcal{L}_{tot}=\mathcal{L}_{soft}+\eta\mathcal{L}_{orth},$
(9)

where $\eta=0.01$ is a weighting parameter. We set the value of $\eta$
empirically and found that $\eta\leq 0.01$ can slightly improve the
performance. The minimization of this loss leads to the maximization of the
expected number of points correctly classified, associating the correct
neighbor prototypes. This facilitates the encoder to learn more local
geometric information. Our implementation of SoftClu is described in Alg. 1 in
the supplementary material.

##  4 Experiments

In this section, we present the implementation details, the setup of pre-
training and the downstream fine-tuning. We invite the reader to check in the
supplementary material for additional results on few-shot learning, ablation
studies, Transformer, and additional visualizations.

###  4.1 Pre-training setup

We explore pre-training strategies on single objects (ShapeNet [Chang et
al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva,
Song, Su, Xiao, Yi, and Yu]) and complex scenes with multiple objects (ScanNet
[Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner]) to
evaluate the effectiveness of SoftClu. We implemented SoftClu in PyTorch and
executed our experiments on two Tesla V100-PCI-E-32G GPUs. During pre-
training, we set $J=64$ and $\epsilon=1e-3$.

ShapeNet [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li,
Savarese, Savva, Song, Su, Xiao, Yi, and Yu] is a collection of single-object
CAD models and contains $57448$ synthetic objects from 55 object categories.
We follow the experimental setup presented in [Sauder and Sievers(2019), Huang
et al.(2021)Huang, Xie, Zhu, and Zhu], we use PointNet [Qi et al.(2017)Qi, Su,
et al.] and DGCNN [Wang et al.(2019)Wang, Sun, Liu, Sarma, Bronstein, and
Solomon] as encoder networks. The latent dimension of both encoders is $1024$.
Following [Huang et al.(2021)Huang, Xie, Zhu, and Zhu], each point cloud is
randomly downsampled to $2048$ points. We use the official training split of
ShapeNet for pre-training. Our pre-training involves $250$ epochs by using the
AdamW [Loshchilov and Hutter(2018)] optimizer, the batch size is equal to
$32$, and initial learning is equal to $0.001$ that decays by $0.7$ every $20$
epochs.

ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner]
is a dataset of indoor scenes with multiple objects and consists of 1513
reconstructed meshes for 707 unique scenes. Following [Xie et al.(2020)Xie,
Gu, Guo, Qi, Guibas, and Litany], we choose SR-UNet provided in PointContrast
[Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] as backbone. For pre-
training, we use an SGD optimizer with a learning rate of 0.1 and a batch size
of 32. The learning rate is decreased by a factor of 0.99 every 1K iteration.
The model is trained for 30K iterations.

###  4.2 Downstream fine-tuning

We evaluate SoftClu on three downstream tasks: classification, part
segmentation, and semantic segmentation. We compare SoftClu to state-of-the-
art discriminative approaches (Jigsaw3D [Sauder and Sievers(2019)], STRL
[Huang et al.(2021)Huang, Xie, Zhu, and Zhu], CrossPoint [Afham et
al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and
Rodrigo], SimCLR [Chen et al.(2020)Chen, Kornblith, et al.], STRL [Huang et
al.(2021)Huang, Xie, Zhu, and Zhu], PointContrast [Xie et al.(2020)Xie, Gu,
Guo, Qi, Guibas, and Litany], and ContrastiveScene [Hou et al.(2021)Hou,
Graham, Nießner, and Xie]) and a generative approach (OcCo [Wang et
al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] and ParAE [Eckart et
al.(2021)Eckart, Yuan, Liu, and Kautz]). The details of setups for downstream
tasks can be found in supplementary for three encoders.

Object classification. We use linear SVM classification on ModelNet40 [Sharma
et al.(2016)Sharma, Grau, and Fritz] and ModelNet10 [Sharma et
al.(2016)Sharma, Grau, and Fritz] datasets to evaluate the quality of their
pre-trained versions on ShapeNet. Table 1 reports the classification accuracy
of SoftClu, compared to the other approaches. Results show that the SoftClu is
more effective than the alternative pre-training methods on both datasets.
Specifically, on ModelNet40, SoftClu with PointNet backbone achieves the same
classification accuracy ($90.3\%$) as ParAE [Eckart et al.(2021)Eckart, Yuan,
Liu, and Kautz] while outperforming the contrastive approach STRL [Huang et
al.(2021)Huang, Xie, Zhu, and Zhu] ($88.3\%$). The linear SVM classification
performance of our method even surpasses the performance of the fully
supervised PointNet, which achieves an 89.2% test accuracy. With the DGCNN
encoder, our method achieves a $91.9\%$ test accuracy, outperforming ParAE
($91.6\%$) by $0.3\%$, and generating model OcCo [Wang et al.(2020)Wang, Liu,
Yue, Lasenby, and Kusner] by $2.7\%$. On ModelNet10, SoftClu outperforms OcCo
[Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] and Jigsaw3D [Sauder
and Sievers(2019)] with both the encoding networks. Compared to jigsaw tasks
that coarsely segment a point cloud into disjoint partitions, SoftClu learns
the partitioning function itself to softly assign point clouds into coherent
clusters. Moreover, we also report results (SoftClu⋆) of SoftClu pre-trained
on ModelNet40, since some methods are pretrained on ModelNet40. SoftClu also
achieves competitive results. We further use visualization to explore pre-
trained features before fine-tuning with the DGCNN encoder. Figure 3 shows the
features visualized with T-SNE [Van der Maaten and Hinton(2008)] of OcCo and
SoftClu on ModelNet10. Our method yields a better separation of the features
than OcCo, which indicates a better ability of SoftClu in clustering objects
in the feature space. In Figure 3, we color the points based on the PCA
projections of the network features and shows how the pre-trained encoder can
effectively embed geometric information.

Table 1: Linear classification comparisons on ModelNet40 and ModelNet10.
$\star$ indicates that models are pre-trained on ModelNet40, otherwise, models
are pre-trained on ShapeNet.

Method | Year | PointNet | DGCNN
---|---|---|---
ModelNet40 | ModelNet10 | ModelNet40 | ModelNet10
DeepCluster [Caron et al.(2018)Caron, Bojanowski, Joulin, and Douze] | 2018 | 86.3 | 91.6 | 90.4 | 94.1
Jigsaw3D⋆ [Sauder and Sievers(2019)] | 2019 | 87.5 | 91.3 | 87.8 | 92.6
Jigsaw3D [Sauder and Sievers(2019)] | 2019 | 87.3 | 91.6 | 90.6 | 94.5
Rotation3D [Poursaeed et al.(2020)Poursaeed, Jiang, Qiao, Xu, and Kim] | 2020 | 88.6 | - | 90.8 | -
SwAV [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin] | 2020 | 85.4 | 92.1 | 90.3 | 93.5
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | 2021 | 88.7 | 91.4 | 89.2 | 92.7
SimCLR [Chen et al.(2020)Chen, Kornblith, et al.] | 2021 | 88.4 | 91.4 | 90.1 | 92.1
STRL [Huang et al.(2021)Huang, Xie, Zhu, and Zhu] | 2021 | 88.3 | - | 90.9 | -
ParAE [Eckart et al.(2021)Eckart, Yuan, Liu, and Kautz] | 2021 | 90.3 | - | 91.6 | -
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 2022 | 89.1 | - | 91.2 | -
SoftClu⋆ (Ours) | - | 88.4 | 93.0 | 91.4 | 94.5
SoftClu (Ours) | - | 90.3 | 93.5 | 91.9 | 94.8

|
---|---
Figure 2: T-SNE-processed network features obtained with (left) OcCo [Wang et
al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] and (right) SoftClu on
ModelNet10.

|
---|---
Figure 3: Color-coded points based on PCA projections of the learned features:
(left) ModelNet40, (right) ShapeNet.

Part segmentation. We follow [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and
Kusner] and use the ShapeNetPart [Yi et al.(2016)Yi, Kim, Ceylan, Shen, Yan,
Su, Lu, Huang, Sheffer, and Guibas] benchmark dataset. Table 3 reports the
part segmentation results of SoftClu in comparison with the alternative
approaches on ShapeNetPart [Yi et al.(2016)Yi, Kim, Ceylan, Shen, Yan, Su, Lu,
Huang, Sheffer, and Guibas]. SoftClu outperforms all the other approaches with
both PointNet and DGCNN encoders in terms of both OA and mIoU. With the
PointNet encoder, SoftClu achieves $93.9\%$ OA and $83.8\%$ mIoU, improving
over state-of-the-art CrossPoint ($93.2\%$ OA, $82.7\%$ mIoU) by $0.7\%$ OA
and $1.1\%$ mIoU. With the DGCNN encoder, we achieve $94.6\%$ OA and $85.7\%$
mIoU, outperforming CrossPoint ($94.4$ OA, $85.3$% mIoU) of about $0.2\%$ OA
and $0.4\%$ mIoU. Figure 4 in the supplementary material shows examples of
qualitative part segmentation results.

Table 2: Part segmentation results.

Encoder | Method | Metrics
---|---|---
OA ($\%$) |  mIoU ($\%$)
PointNet | Random | 92.8 | 82.2
Jigsaw3D [Sauder and Sievers(2019)] | 93.1 | 82.2
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | 93.4 | 83.4
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 93.2 | 82.7
SoftClu (Ours) | 93.9 | 83.8
DGCNN | Random | 92.2 | 84.4
Jigsaw3D [Sauder and Sievers(2019)] | 92.7 | 84.3
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | 94.4 | 85.0
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 94.4 | 85.3
SoftClu (Ours) | 94.6 | 85.7

Table 3: Semantic segmentation results.

Encoder | Method | Metrics
---|---|---
OA ($\%$) |  mIoU ($\%$)
PointNet | Random | 78.9 | 47.0
Jigsaw3D [Sauder and Sievers(2019)] | 80.1 | 52.6
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | 82.0 | 54.9
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 81.8 | 54.5
SoftClu (Ours) | 82.9 | 55.3
DGCNN | Random | 83.7 | 54.9
Jigsaw3D [Sauder and Sievers(2019)] | 84.1 | 55.6
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | 84.6 | 58.0
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 84.7 | 58.4
SoftClu (Ours) | 85.4 | 59.2

Semantic segmentation. We first evaluate SoftClu features on semantic
segmentation by using the S3DIS [Armeni et al.(2016)Armeni, Sener, Zamir,
Jiang, Brilakis, Fischer, and Savarese] benchmark dataset, where features are
pre-trained on ShapeNet. Table 3 reports the segmentation results of SoftClu
and that of the other baselines on S3DIS [Armeni et al.(2016)Armeni, Sener,
Zamir, Jiang, Brilakis, Fischer, and Savarese]. SoftClu outperforms all the
other approaches with both PointNet and DGCNN encoders. With the PointNet
encoder, SoftClu achieves $82.9\%$ OA and $55.3\%$ mIoU, outperforming both
the state-of-the-art OcCo ($82.0\%$ OA, $55.3\%$ mIoU) and Jigsaw3D ($80.1\%$
OA, $52.6\%$ mIoU). With the DGCNN encoder, SoftClu achieves $85.4\%$ OA and
$59.2\%$ mIoU, outperforming CrossPoint [Afham et al.(2022)Afham, Dissanayake,
Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] (84.7% OA and 58.4%
mIoU), OcCo ($84.6\%$ OA, $58.0\%$ mIoU) and Jigsaw3D ($84.1\%$ OA, $55.6\%$
mIoU). We also compare SoftClu with point-level methods such as PointContrast
[Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] and ContrastiveScene
[Hou et al.(2021)Hou, Graham, Nießner, and Xie] when features are pre-trained
on ScanNet with SR-UNet backbone. Table 4 shows that our pre-training method
achieves 73.4% mIoU and 79.1% mAcc, outperforming the pre-training results of
PointContrast [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] and
ContrastiveScene [Hou et al.(2021)Hou, Graham, Nießner, and Xie].

Table 4: Results of semantic segmentation with SR-UNet backbone [Armeni et
al.(2016)Armeni, Sener, Zamir, Jiang, Brilakis, Fischer, and Savarese].

Method | Scratch |  PointContrast [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] |  ContrastiveScene [Hou et al.(2021)Hou, Graham, Nießner, and Xie] | SoftClu
---|---|---|---|---
mIoU | 68.2 | 70.3 | 72.2 | 73.4
mAcc | 75.5 | 76.9 | - | 79.1

###  4.3 Ablation study and analysis

Feature and geometric prototypes. We study the contribution of feature and
geometric prototypes of SoftClu (Eq. 1). We perform pre-training by using (i)
only feature prototypes, (ii) only geometric prototypes, and (iii) both
prototypes. We perform this study on ModelNet40 and ModelNet10. In Table 5 we
can observe that when only feature prototypes are used, SoftClu achieves the
lowest performance. This is due to wrong cluster assignments of those points
sharing similar features but belonging to different geometric regions (e.g.
wings of an airplane). Using both feature and geometric prototypes, SoftClu
can achieve the best performance on ModelNet40 and ModelNet10, with both
PointNet and DGCNN.

Table 5: Ablation study of SoftClu by using different prototypes.

Encoder | Geometry | Feature | Accuracy
---|---|---|---
ModelNet40 | ModelNet10
PointNet | $\checkmark$ |  | 88.7 | 92.9
| $\checkmark$ | 86.5 | 92.7
$\checkmark$ | $\checkmark$ | 90.3 | 93.5
DGCNN | $\checkmark$ |  | 91.4 | 94.5
| $\checkmark$ | 90.5 | 93.3
$\checkmark$ | $\checkmark$ | 91.9 | 94.8

Number of clusters. We assess the effect of selecting different numbers of
cluster partitions $J$ by using ModelNet40. We pre-train SoftClu with
different values of $J$, i.e. from $16$ to $128$, and report the results in
Table 6. SoftClu achieves the best results with $J=64$ for both PointNet and
DGCNN. We observed stability with the results throughout different values of
$J$. Additional results of the batch size and soft-labels please refer to the
supplementary material.

Table 6: Ablation study results of SoftClu with different number of clusters
$J$.

Method | 16 | 32 | 48 | 64 | 72 | 96 | 112 | 128
---|---|---|---|---|---|---|---|---
PointNet | 92.4 | 93.0 | 93.1 | 93.5 | 93.4 | 93.3 | 93.2 | 93.1
DGCNN | 94.2 | 94.8 | 94.6 | 94.8 | 94.7 | 94.6 | 94.6 | 94.5

Running times. Our method is used only for pre-training, where each iteration
consists of two parts: a backbone forward pass and SoftClu optimization. We
run our method on one Tesla V100 GPU (32G) and two Intel(R) 6226 CPUs and
measured the iteration time over several iterations. SoftClu adds an average
overhead of 0.014ms for each iteration on ShapeNet with a DGCNN backbone.
Notice that, the inference time of each backbone used remains the original
one.

##  5 Conclusions

We presented SoftClu, a data augmentation-free unsupervised representation
learning scheme for 3D point cloud understanding. SoftClu implicitly
alternates between clustering the point-level features to produce point-wise
pseudo-labels and utilizing these soft-labels to train the representations.
Our approach showed promising results in transferring the pre-trained
representations to different downstream 3D understanding tasks, such as
classification, part segmentation, and semantic segmentation. Our SoftClu is
independent from specific deep network architectures, enabling us to use our
method as a generic method for feature extraction from raw point cloud data to
improve other 3D models performances.

Acknowledgments. This work was supported by the EU H2020 AI4Media No. 951911
project and the EUREGIO project OLIVER.

## References

  * [Achlioptas et al.(2018)Achlioptas, Diamanti, Mitliagkas, and Guibas] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In _ICML_ , 2018.
  * [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo.  Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.  In _CVPR_ , June 2022.
  * [Armeni et al.(2016)Armeni, Sener, Zamir, Jiang, Brilakis, Fischer, and Savarese] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese.  3d semantic parsing of large-scale indoor spaces.  In _CVPR_ , 2016.
  * [Asano et al.(2020)Asano, Rupprecht, and Vedaldi] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  In _ICLR_ , 2020.
  * [Biswas and Veloso(2012)] Joydeep Biswas and Manuela Veloso.  Depth camera based indoor mobile robot localization and navigation.  In _ICRA_ , 2012.
  * [Caron et al.(2018)Caron, Bojanowski, Joulin, and Douze] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In _ECCV_ , 2018.
  * [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _NeurIPS_ , 33:9912–9924, 2020.
  * [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, Xiao, Yi, and Yu] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  In _arXiv_ , 2015.
  * [Chen et al.(2019)Chen, Duan, Yang, Li, Feng, and Tian] Siheng Chen, Chaojing Duan, Yaoqing Yang, Duanshun Li, Chen Feng, and Dong Tian.  Deep unsupervised learning of 3d point clouds via graph topology inference and filtering.  _TIP_ , 29:3183–3198, 2019.
  * [Chen et al.(2020)Chen, Kornblith, et al.] Ting Chen, Simon Kornblith, et al.  A simple framework for contrastive learning of visual representations.  _ICML_ , 2020.
  * [Chen and He(2021)] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _CVPR_ , 2021.
  * [Cuturi(2013)] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  _NeurIPS_ , 2013.
  * [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , pages 5828–5839, 2017.
  * [Du et al.(2021)Du, Gao, Hu, and Li] Bi’an Du, Xiang Gao, Wei Hu, and Xin Li.  Self-contrastive learning with hard negative sampling for self-supervised point cloud learning.  In _ACM MM_ , pages 3133–3142, 2021.
  * [Eckart et al.(2021)Eckart, Yuan, Liu, and Kautz] Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz.  Self-supervised learning on 3d point clouds by learning discrete generative models.  In _CVPR_ , 2021.
  * [Gao et al.(2020)Gao, Hu, and Qi] Xiang Gao, Wei Hu, and Guo-Jun Qi.  Graphter: Unsupervised learning of graph transformation equivariant representations via auto-encoding node-wise transformations.  In _CVPR_ , 2020.
  * [Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Pires, Guo, Azar, et al.] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In _NeurIPS_ , 2020.
  * [Han et al.(2019)Han, Wang, et al.] Zhizhong Han, Xiyang Wang, et al.  Multi-angle point cloud-vae: unsupervised feature learning for 3d point clouds from multiple angles by joint self-reconstruction and half-to-half prediction.  In _ICCV_ , pages 10441–10450, 2019.
  * [Hassani and Haley(2019)] Kaveh Hassani and Mike Haley.  Unsupervised multi-task feature learning on point clouds.  In _ICCV_ , 2019.
  * [Hou et al.(2021)Hou, Graham, Nießner, and Xie] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie.  Exploring data-efficient 3d scene understanding with contrastive scene contexts.  In _CVPR_ , pages 15587–15597, 2021.
  * [Huang et al.(2021)Huang, Xie, Zhu, and Zhu] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu.  Spatio-temporal self-supervised representation learning for 3d point clouds.  In _ICCV_ , 2021.
  * [Huang et al.(2020)Huang, Mei, and Zhang] Xiaoshui Huang, Guofeng Mei, and Jian Zhang.  Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences.  In _CVPR_ , pages 11366–11374, 2020.
  * [Li et al.(2020)Li, Ma, Zhong, Liu, Chapman, Cao, and Li] Ying Li, Lingfei Ma, Zilong Zhong, Fei Liu, Michael A Chapman, Dongpu Cao, and Jonathan Li.  Deep learning for lidar point clouds in autonomous driving: a review.  _TNNLS_ , 2020.
  * [Lin et al.(2021)Lin, Chen, and Jia] Xuanxiang Lin, Ke Chen, and Kui Jia.  Object point cloud classification via poly-convolutional architecture search.  In _ACM MM_ , pages 807–815, 2021.
  * [Liu et al.(2022)Liu, Cai, and Lee] Haotian Liu, Mu Cai, and Yong Jae Lee.  Masked discrimination for self-supervised learning on point clouds.  _arXiv preprint arXiv:2203.11183_ , 2022.
  * [Liu et al.(2019)Liu, Han, et al.] Xinhai Liu, Zhizhong Han, et al.  L2g auto-encoder: Understanding point clouds by local-to-global reconstruction with hierarchical self-attention.  In _ACM MM_ , pages 989–997, 2019.
  * [Loshchilov and Hutter(2018)] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2018.
  * [Mei(2021)] Guofeng Mei.  Point cloud registration with self-supervised feature learning and beam search.  In _DICTA_ , pages 01–08, 2021.
  * [Mei et al.(2022)Mei, Huang, Zhang, and Wu] Guofeng Mei, Xiaoshui Huang, Jian Zhang, and Qiang Wu.  Overlap-guided coarse-to-fine correspondence prediction for point cloud registration.  In _ICME_ , pages 1–6. IEEE, 2022.
  * [Moon(1996)] Todd K Moon.  The expectation-maximization algorithm.  _IEEE Signal processing magazine_ , 1996.
  * [Park et al.(2008)Park, Lepetit, and Woo] Youngmin Park, Vincent Lepetit, and Woontack Woo.  Multiple 3d object tracking for augmented reality.  In _ISMAR_ , 2008.
  * [Peyré et al.(2019)Peyré, Cuturi, et al.] Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 2019\.
  * [Poiesi and Boscaini(2022)] F. Poiesi and D. Boscaini.  Learning general and distinctive 3D local deep descriptors for point cloud registration.  _TPAMI_ , 2022.
  * [Poursaeed et al.(2020)Poursaeed, Jiang, Qiao, Xu, and Kim] Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim.  Self-supervised learning of point clouds via orientation estimation.  In _3DV_ , 2020.
  * [Qi et al.(2017)Qi, Su, et al.] Charles R Qi, Hao Su, et al.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _CVPR_ , pages 652–660, 2017.
  * [Rao et al.(2020)Rao, Lu, and Zhou] Yongming Rao, Jiwen Lu, and Jie Zhou.  Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds.  In _CVPR_ , 2020.
  * [Sanghi(2020)] Aditya Sanghi.  Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning.  In _ECCV_ , 2020.
  * [Sarmad et al.(2019)Sarmad, Lee, et al.] Muhammad Sarmad, Hyunjoo Jenny Lee, et al.  Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion.  In _CVPR_ , pages 5898–5907, 2019.
  * [Sauder and Sievers(2019)] Jonathan Sauder and Bjarne Sievers.  Self-supervised deep learning on point clouds by reconstructing space.  In _NeurIPS_ , pages 12942–12952, 2019.
  * [Sharma et al.(2016)Sharma, Grau, and Fritz] Abhishek Sharma, Oliver Grau, and Mario Fritz.  Vconv-dae: Deep volumetric shape learning without object labels.  In _ECCV_ , pages 236–250, 2016.
  * [Sharma and Kaul(2020)] Charu Sharma and Manohar Kaul.  Self-supervised few-shot learning on point clouds.  _NeurIPS_ , 33:7212–7221, 2020.
  * [Shi et al.(2020a)Shi, Guo, Jiang, Wang, Shi, Wang, and Li] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In _CVPR_ , 2020a.
  * [Shi et al.(2020b)Shi, Xu, Yuan, and Fang] Yi Shi, Mengchen Xu, Shuaihang Yuan, and Yi Fang.  Unsupervised deep shape descriptor with point distribution learning.  In _CVPR_ , pages 9353–9362, 2020b.
  * [Sun et al.(2020)Sun, Wang, Liu, Siegel, and Sarma] Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma.  Pointgrow: Autoregressively learned point cloud generation with self-attention.  In _WACV_ , pages 61–70, 2020.
  * [Van der Maaten and Hinton(2008)] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _JMLR_ , 2008.
  * [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matthew J Kusner.  Unsupervised point cloud pre-training via view-point occlusion, completion.  In _ICCV_ , 2020.
  * [Wang and Bue(2020)] Y. Wang and A. Del Bue.  Where to Explore Next? ExHistCNN for History-aware Autonomous 3D Exploration.  In _ECCV_ , 2020.
  * [Wang et al.(2019)Wang, Sun, Liu, Sarma, Bronstein, and Solomon] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  _ACM TOG_ , 38(5):1–12, 2019.
  * [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In _ECCV_ , 2020.
  * [Xu et al.(2020)Xu, Zhou, and Qiao] Mingye Xu, Zhipeng Zhou, and Yu Qiao.  Geometry sharing network for 3d point cloud classification and segmentation.  In _AAAI_ , pages 12500–12507, 2020.
  * [Yang et al.(2018)Yang, Feng, Shen, and Tian] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In _CVPR_ , pages 206–215, 2018.
  * [Yi et al.(2016)Yi, Kim, Ceylan, Shen, Yan, Su, Lu, Huang, Sheffer, and Guibas] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas.  A scalable active framework for region annotation in 3d shape collections.  _ACM TOG_ , 2016.
  * [Yin et al.(2021)Yin, Zhou, and Krahenbuhl] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In _CVPR_ , 2021.
  * [Yu et al.(2022)Yu, Tang, Rao, Huang, Zhou, and Lu] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In _CVPR_ , pages 19313–19322, 2022.
  * [Yuan et al.(2018)Yuan, Khot, Held, Mertz, and Hebert] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert.  Pcn: Point completion network.  In _3DV_ , 2018.
  * [Zhou et al.(2022)Zhou, Wang, Poiesi, Qin, and Wan] Y. Zhou, Y. Wang, F. Poiesi, Q. Qin, and Y. Wan.  Loop closure detection using local 3D deep descriptors.  _IEEE RAL_ , 2022.

Supplementary Materials for Data Augmentation-free Unsupervised Learning for
3D Point Cloud Understanding

##  Appendix A Algorithm

Here, we provide a pseudo-code for SoftClu training loop in Algorithm 1.

Algorithm 1 Soft clustering (pseudocode).

Input: $\\{\mathcal{P}\\}$ a set of 3D point clouds and each point cloud has
$N$ points; $K$ number of optimization steps.
Output: the backbone $f_{\varphi}$ pretrained by using our algorithm.

1:for $i$ in range(0, K) do

2: $\mathcal{L}_{tot}=0$

3: for $\mathcal{P}\in\left\\{\mathcal{P}\right\\}$ do

4: # compute class scores

5:
$\bm{S}=\mbox{softmax}\left(\phi_{\alpha}\left(f_{\varphi}\left(\mathcal{P}\right)\right)\right)$

6: # compute prototypes

7:
$\bm{C}^{E}=\left\\{\frac{1}{\sum_{i=1}^{N}s_{ij}}\sum_{i=1}^{N}s_{ij}\bm{p}_{i}\right\\}_{j=1}^{N}$

8:
$\bm{C}^{F}=\left\\{\frac{1}{\sum_{i=1}^{N}s_{ij}}\sum_{i=1}^{N}s_{ij}\bm{f}_{i}\right\\}_{j=1}^{N}$

9: # compute $\bm{D}$

10:
$\bm{D}=\left\\{\lambda\|\bm{p}_{i}-\bm{c}^{E}_{j}\|^{2}_{2}+\left(1-\lambda)\right\|\bm{f}_{i}-\bm{c}^{F}_{j}\|^{2}_{2}\right\\}_{i,j}^{N,J}$

11: # compute $\gamma$

12:
$\bm{\Gamma}=\mbox{SINKHORN}\left(\mbox{stopgrad}\left(\bm{D}\right),1e-3,20\right)$

13: $\bm{\gamma}=N\cdot\bm{\Gamma}$

14: # compute loss

15: $\mathcal{L}_{tot}\mathrel{{+}{=}}\mathcal{L}_{soft}$ \+
$\eta\mathcal{L}_{orth}$

16: end for

17: # update backbone and segmentation head

18:
$f_{\varphi},\phi_{\alpha}\leftarrow\mbox{optimize}\left(\frac{\mathcal{L}_{tot}}{N}\right)$

19:end for

20:return $f_{\varphi}$

For the Sinkhorn-Knopp algorithm, we provide a detailed pseudo-code in
Algorithm 2.

Algorithm 2 Sinkhorn-Knopp algorithm (pseudocode).

Input: $D$ distance matrix, $\epsilon=1e-3$ and $niters$ iterations.

1:function sinkhorn($\bm{D}$, $\epsilon$, $niters$)

2: $\bm{\Gamma}=\exp(\bm{D}/\epsilon)$

3: $\bm{\Gamma}/=\mbox{sum}(\bm{\Gamma})$

4: $N,J=\bm{\Gamma}$.shape

5:
$\bm{u},\bm{\mu},\bm{\nu}=\mbox{zeros}(N),\mbox{ones}(N)/N,\mbox{ones}(J)/J$

6: for _ in range(0, niters) do

7: $\bm{u}=$ sum($\bm{\Gamma}$, dim=1)

8: $\bm{\Gamma}*=(\bm{\mu}/\bm{u})$.unsqueeze(1)

9: $\bm{\Gamma}*=(\bm{\nu}/\mbox{sum}(\bm{\Gamma},\mbox{dim=0}))$.unsqueeze(0)

10: end for

11:return $\bm{\Gamma}$

12:end function

##  Appendix B Downstream Tasks Setups

Classification. We use ModelNet40 [Sharma et al.(2016)Sharma, Grau, and Fritz]
and ModelNet10 [Sharma et al.(2016)Sharma, Grau, and Fritz] benchmark
classification datasets. ModelNet40 is composed of $12331$ meshed models from
$40$ object categories, split into $9843$ training meshes and $2468$ testing
meshes, where the points are sampled from CAD models. ModelNet10 dataset
contains $4899$ pre-aligned shapes from $10$ categories with $3991$ ($80\%$)
shapes for training and $908$ ($20\%$) shapes for testing. For SVM training,
we randomly sample $1024$ points for each shape as in [Sauder and
Sievers(2019)].

Part segmentation. We follow [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and
Kusner] and use the ShapeNetPart [Yi et al.(2016)Yi, Kim, Ceylan, Shen, Yan,
Su, Lu, Huang, Sheffer, and Guibas] benchmark dataset that contains $16881$
objects of $2048$ points from $16$ categories with $50$ parts in total. We
train the linear fully connected layer for $100$ epochs by using the AdamW
[Loshchilov and Hutter(2018)] optimizer with batch size of $24$, initial
learning rate of $0.001$, learning rate decay of $0.5$ every $20$ epoch. We
report the overall accuracy (OA) and the mean class intersection over union
(mIoU) to evaluate segmentation quality as in [Wang et al.(2020)Wang, Liu,
Yue, Lasenby, and Kusner].

Semantic segmentation. We evaluate SoftClu features on semantic segmentation
by using the S3DIS [Armeni et al.(2016)Armeni, Sener, Zamir, Jiang, Brilakis,
Fischer, and Savarese] benchmark dataset. S3DIS consists of 3D scans collected
with the Matterport scanner in six indoor areas, featuring 271 rooms and 13
semantic classes. Following the pre-processing, post-processing and training
settings as in [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner], we
split each room into $1m\times 1m$ blocks and use 4,096 points as the model
input. For PointNet and DGCNN, we finetune the pre-trained model on areas
1,2,3,4,6 and test them on area 5. As in part segmentation, we report OA and
mIoU to quantify the segmentation quality.

For SR-UNet backbone, we finetune the pre-trained model on areas 1-4 and 6 and
test them on area 5. The fine-tuning experiments are trained with a batch size
of 48 for a total of 10K iterations. The initial learning rate is 0.1, with
polynomial decay with a power of 0.9. We set voxel size as 0.05 (5cm) and
weight decay as 0.0001. We report mIoU and mAcc to evaluate segmentation
quality as in [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany].

##  Appendix C More Results

#### Part segmentation visualizations.

Figure 4 shows examples of qualitative part segmentation results obtained with
SoftClu after the fine-tuning on the downstream task compared to ground-truth
annotations (GT). We can observe that our method provides consistent
predictions throughout shapes, also in the case of complex shapes (chair and
motorcycle).

#### Few-shot learning.

Few-shot learning (FSL) aims to train a model that generalizes with limited
data. We conduct FSL ($N$-way $K$-shot learning) for the classification task
on ModelNet40 [Sharma et al.(2016)Sharma, Grau, and Fritz] and ModelNet10
[Sharma et al.(2016)Sharma, Grau, and Fritz] benchmark datasets, where the
model is evaluated on $N$ classes, and each class contains $N$ samples. We use
the same setting and train/test split as OcCo [Wang et al.(2020)Wang, Liu,
Yue, Lasenby, and Kusner] and CrossPoint [Afham et al.(2022)Afham,
Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] and report
the mean and standard deviation across 10 runs. Table 7 shows the FSL results
on ModelNet40, where SoftClu outperforms prior works in all the FSL settings
in the DGCNN backbone. Our method with PointNet backbone performs slightly
poorly in 5-way 10-short and 10-way 20-short settings compared to results of
CrossPoint with PointNet.

Table 7: Few-shot object classification results on ModelNet40. We report mean and standard error over 10 runs. Top results of each backbone is bold. Encoder | Method | 5-way | 10-way
---|---|---|---
10-shot | 20-shot | 10-shot | 20-shot
PointNet | Rand |  52.0 $\pm$ 3.8 |  57.8 $\pm$ 4.9 |  46.6 $\pm$ 4.3 |  35.2 $\pm$ 4.8
Jigsaw [Sauder and Sievers(2019)] |  66.5 $\pm$ 2.5 |  69.2 $\pm$ 2.4 |  56.9 $\pm$ 2.5 |  66.5 $\pm$ 1.4
cTree [Sharma and Kaul(2020)] |  63.2 $\pm$ 3.4 |  68.9 $\pm$ 3.0 |  49.2 $\pm$ 1.9 |  50.1 $\pm$ 1.6
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] |  89.7 $\pm$ 1.9 |  92.4 $\pm$ 1.6 |  83.9 $\pm$ 1.8 |  89.7 $\pm$ 1.5
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] | 90.9 $\pm$ 4.8 |  93.5 $\pm$ 4.4 |  84.6 $\pm$ 4.7 | 90.2 $\pm$ 2.2
SoftClu |  90.6 $\pm$ 4.0 | 93.8 $\pm$ 3.2 | 84.7 $\pm$ 3.6 |  90.1 $\pm$ 4.5
DGCNN | Rand |  31.6 $\pm$ 2.8 |  40.8 $\pm$ 4.6 |  19.9 $\pm$ 2.1 |  16.9 $\pm$ 1.5
Jigsaw [Sauder and Sievers(2019)] |  34.3 $\pm$ 1.3 |  42.2 $\pm$ 3.5 |  26.0 $\pm$ 2.4 |  29.9 $\pm$ 2.6
cTree [Sharma and Kaul(2020)] |  68.4 $\pm$ 3.4 |  71.6 $\pm$ 2.9 |  42.4 $\pm$ 2.7 |  43.0 $\pm$ 3.0
OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] |  90.6 $\pm$ 2.8 |  92.5 $\pm$ 1.9 |  82.9 $\pm$ 1.3 |  86.5 $\pm$ 2.2
CrossPoint [Afham et al.(2022)Afham, Dissanayake, Dissanayake, Dharmasiri, Thilakarathna, and Rodrigo] |  92.5 $\pm$ 3.0 |  94.9 $\pm$ 2.1 |  83.6 $\pm$ 5.3 |  87.9 $\pm$ 4.2
SoftClu | 93.6 $\pm$ 3.3 | 97.3 $\pm$ 2.0 | 89.1 $\pm$ 1.4 | 93.2 $\pm$ 3.4

Ours

GT

Figure 4: Part segmentation results on ShapeNetPart [Yi et al.(2016)Yi, Kim,
Ceylan, Shen, Yan, Su, Lu, Huang, Sheffer, and Guibas] of SoftClu using the
DGCNN encoder (top row) compared to the ground-truth annotations (bottom row).

#### Batch size.

Contrastive methods mine negative examples from the mini-batch can suffer from
performance drops when their batch size is not sufficiently large [Grill et
al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Pires,
Guo, Azar, et al.]. Because SoftClu does not rely on negative examples, we
expect it to be more robust to smaller batch sizes when compared to the
contrastive approaches. We empirically show the effect of different batch
sizes by comparing the performance of SoftClu with SimCLR [Chen et
al.(2020)Chen, Kornblith, et al.]. We variate batch sizes from 8 to 48 during
pre-training. Table 8 shows that SimCLR experiences degradation of performance
when the batch size is 8, likely due to the small number of available negative
samples. By contrast, SoftClu maintains a fairly stable performance throughout
different batch size configurations.

Table 8: Ablation study results of SoftClu by using DGCNN on ModelNet10 with different batch sizes during pre-training. Encoder | Method | 8 | 16 | 24 | 32 | 40 | 48
---|---|---|---|---|---|---|---
PointNet | SimCLR | 87.5 | 88.0 | 88.2 | 88.1 | 88.5 | 88.4
SoftClu | 89.9 | 89.8 | 90.2 | 90.3 | 90.1 | 89.9
DGCNN | SimCLR | 88.6 | 89.3 | 89.4 | 89.7 | 89.7 | 90.1
SoftClu | 91.6 | 91.8 | 91.7 | 91.9 | 91.9 | 91.8

#### Computation of soft-labels.

We assess our strategy for soft-label assignment based on optimal transport
(OT) by comparing it with a typical L2 distance-based approach on ModelNet40
and ModelNet10. Therefore, we assess SoftClu by using $\Gamma$ computed with
Eq. (7) and by using the L2 approach in [Caron et al.(2018)Caron, Bojanowski,
Joulin, and Douze]. Table 9 shows that OT achieves the best performance on all
the datasets with both PointNet and DGCNN encoders. This is due to the equal
partition constraint in Alg. 2 which prevents solutions from being assigned to
the same cluster and affecting the performance.

Table 9: Ablation study of SoftClu on ModelNet40 and ModelNet10 with soft-labels computed with our approach (OT) and with a typical distance-based assignment (L2). Dataset | Encoder | Accuracy
---|---|---
L2 | OT
ModelNet10 | PointNet | 91.5 | 93.4
DGCNN | 94.1 | 94.8
ModelNet40 | PointNet | 86.5 | 90.3
DGCNN | 90.4 | 91.9

#### SoftClu with Transformer backbone.

Following [Liu et al.(2022)Liu, Cai, and Lee] setups, we also provide the
results with the recent Transformer backbone provided by [Liu et al.(2022)Liu,
Cai, and Lee]. As shown in Tab. 10, SoftClu also achieves competitive result.

Table 10: Classification results with a Transformer backbone on ModelNet40. Encoder | SoftClu | PointViT-OcCo [Wang et al.(2020)Wang, Liu, Yue, Lasenby, and Kusner] | Point-BERT [Yu et al.(2022)Yu, Tang, Rao, Huang, Zhou, and Lu] | MaskPoint [Liu et al.(2022)Liu, Cai, and Lee]
---|---|---|---|---
SoftClu | 93.8 | 92.1 | 93.2 | 93.8