# COLA: COarse-LAbel multi-source LiDAR semantic segmentation for autonomous
driving

Jules Sanchez1, Jean-Emmanuel Deschaud1, François Goulette1,2 1Centre of
Robotics, Mines Paris - PSL, PSL University, Paris, France.
<EMAIL_ADDRESS>ENSTA Paris, Institut Polytechnique
de Paris, Palaiseau, France<EMAIL_ADDRESS>
###### Abstract

LiDAR semantic segmentation for autonomous driving has been a growing field of
interest in the past few years. Datasets and methods have appeared and
expanded very quickly, but methods have not been updated to exploit this new
availability of data and continue to rely on the same classical datasets.

Different ways of performing LIDAR semantic segmentation training and
inference can be divided into several subfields, which include the following:
domain generalization, the ability to segment data coming from unseen domains
; source-to-source segmentation, the ability to segment data coming from the
training domain; and pre-training, the ability to create re-usable geometric
primitives.

In this work, we aim to improve results in all of these subfields with the
novel approach of multi-source training. Multi-source training relies on the
availability of various datasets at training time and uses them together
rather than relying on only one dataset.

To overcome the common obstacles found for multi-source training, we introduce
the coarse labels and call the newly created multi-source dataset COLA. We
propose three applications of this new dataset that display systematic
improvement over single-source strategies: COLA-DG for domain generalization
(up to +10%), COLA-S2S for source-to-source segmentation (up to +5.3%), and
COLA-PT for pre-training (up to +12%).

###### Index Terms:

Semantic Scene Understanding, Computer Vision for Transportation, Deep
Learning in Robotics and Automation, 3D Computer Vision

##  I Introduction

Transferability and robustness have become the center of attention for novel
works in LiDAR semantic segmentation. Specifically, the topics of domain
adaptation, domain generalization, and pre-training are all of interest to re-
usable models and training methods to improve performance and robustness.

Current works study the impact of multi-task training [1], the use of the
underlying geometry [2], data augmentation strategies [3] and the availability
of a large dataset for unsupervised training [4]. However, these works have
only looked at single-source strategies, in which only one dataset is used at
training time, and have left out multi-source strategies.

Multi-source training relies on the availability of a variety of datasets to
be used at training time. Implicitly, it is expected that these datasets
represent a variety of domains, which means that some domain shifts exist
between them. In the case of LiDAR scene understanding for autonomous driving,
domain shifts can manifest themselves as shift in the following: the
acquisition hardware, such as sensor resolution; geographical location of the
acquired scenes; or scene type, such as urban centers or suburban cities. This
variety is expected to improve the performances of the trained models.
Following [2] we will divide the domain shifts between sensor shift, scene
shift, and appearance shift.

Multi-source strategies have been under-exploited in 3D scene understanding
with very few works looking into them [5, 6, 7]. As pointed out in these
works, this lack of interest stems from the difficulty of leveraging several
datasets at once due to disparities in their label sets. Some of the
approaches in 2D scene understanding overcame this by re-annotating data [8],
which results in a significant human cost.

In our previous work [7], which this paper is based on, we proposed COLA, a
relabelling strategy that allowed us to use several datasets at once. This
relabelling strategy relies on remapping existing labels to a common coarser
set, which is a process that is almost cost-free. We employed this strategy
for pre-training purposes only. In [7], COLA leveraged multi-source semantic
segmentation as a pre-training task for outdoor LiDAR semantic segmentation.
To achieve this, it presented a novel label set: the coarse labels
specifically designed for LiDAR semantic segmentation for autonomous driving.

Figure 1: COLA and its applications: domain generalization (COLA-DG), source-
to-source segmentation (COLA-S2S), pre-training (COLA-PT).

In this work, we expand the use of COLA to several other cases, namely domain
generalization and source-to-source semantic segmentation. Furthermore, we
update and enrich experiments conducted in [7] for pre-training. The
difference between these three tasks can be found in Figure 1. Contrary to
[7], all experiments here are done on at least two largely different
architectures to ensure that the conclusions are meaningful and that they are
done on a wider variety of data for pre-training and for evaluation.

Overall, the goal of this work is to highlight the effectiveness of multi-
source strategies, and we can summarize its contributions as follows:

  * •

We re-introduce COLA, a relabelling strategy that allows for the easy
implementation of multi-source training for LiDAR semantic segmentation for
autonomous driving.

  * •

We propose the first multi-source domain generalization method, called COLA-
DG, for LiDAR semantic segmentation for autonomous driving.

  * •

We propose a multi-source semantic segmentation strategy to improve source-to-
source performances, called COLA-S2S.

  * •

We update the study of multi-source pre-training, called COLA-PT, in
comparison with more recent works and a wider variety of models.

##  II Related work

###  II-A LiDAR semantic segmentation

LiDAR semantic segmentation (LSS) for autonomous driving requires high speed
performance of at least 10 frames processed per seconds. As such, older 3D
deep learning relying on redefining convolutions, such as TangentConv [9],
SpiderCNN [10], and KPConv [11], are not used despite reaching satisfying
performances. Methods that use structured representations are favored, and
they include the following: projecting the point clouds in 2D with a range-
based image [12, 13], bird’s-eye-view image [14] and leveraging sparse
convolution-based architectures as introduced by MinkowskiNet [15] under the
name Sparse Residual U-Net (SRU-Net). SRU-Nets have been used as the backbone
for more refined methods, such as point-voxel methods [16], cylindrical voxel
convolutions as done by Cylinder3D [17], and mixed-input representation
methods [18]. These methods are the best performing.

###  II-B Domain generalization for LSS

As mentioned in the introduction, the focus has shifted from source-to-source
performances, in which the training and evaluation sets belong to the same
domain, to robustness evaluation. Domain generalization addresses the
transferability of a model or an algorithm from one source to a target domain,
without the availability of the target domain prior to the inference. [19, 20]
propose a survey of domain generalization and a classification of the various
strategies adopted. Among these methods, we can cite meta-learning [21, 22],
multi-task training [23, 24], data augmentation [25], neural network design
[26, 27, 28], and domain alignment [29, 30].

These methods and paradigms were designed for 2D computer vision; more
recently, researchers have created methods specificaly designed for 3D. Among
them, we can cite DGLSS [31], LIDOG [1], 3D-VField [3] and 3DLabelProp [2].

3D-VField [3] is an adversarial data augmentation method that relies on
adversially attacking objects of a scene to robustify predictions. DGLSS [31]
is a domain alignment method which uses a degraded version of the considered
scan at training time to reduce the dependency to the resolution of the
sensors, 3DLabelProp [2] is a domain alignment method as well which uses
pseudo-dense point clouds to increase the geometric similarity between point
clouds acquired with various sensors. Finally, LIDOG [1] is a multi-task
method that segments the point clouds both in the voxel space and in the
bird’s-eye-view space.

All these methods are single-source, that is, they use only one dataset at
training time despite the availability of a growing number of datasets.

###  II-C Multi-source computer vision

Multi-source training has been been used to improve the robustness of computer
vision deep learning models. When all the datasets have the same labels, it is
easy to use, as simply concatenating datasets together is already a multi-
source strategy and improves robustness. More elaborated strategy exists as
well, such as meta-learning. Both [21, 22] leverage the availability of
several datasets to ensure good domain generalization performance.

In most cases, available datasets do not have the same label set, and
intermediate steps need to be taken to be able to carry out multi-source
training. The most naive approaches propose a new label set from existing ones
by taking the intersection [32, 33], union [34], or sum with multi-head (MH)
training [35, 36], which has one classificaton head per dataset in the
training set. This is illustrated in Figure 2.

Figure 2: Illustration of naive multi-source methods.

These methods make a strong assumption about the inference label set. In the
case of domain generalization, the target label set cannot be guaranteed to
lie in the union of the training set, making the union and MH methods
unreliable. The intersection methods can result in many interesting labels
being discarded and is very dependant on annotation choices.

Other works observed that despite having different label sets, datasets
designed for similar tasks share ontological connections between their label
sets. Some works have tried to explicitly model these relationships [37, 38,
39].

In 3D, [6] noticed the variety of 3D datasets and the complexity of using them
together. To overcome this, they used a pre-trained language model to encode
label proximity between label sets. Through this, they were able to make
predictions in a continuous label space rather than in a discrete one. [6]
applied this work only to indoor scene understanding.

###  II-D Pre-training for LSS

There are several strategies for pre-training such as: perform auxiliary tasks
on the same data as the finetuning one, or perform a simple task on a large
quantity of data beforehand to learn general primitives. As there are no very
large annotated data for 3D, unsupervised methods are largely favored,
especially contrastive training.

Amongst these methods PointContrast [40], CSC [41], and DepthContrast [42] are
the standards. The difference between these methods stem from the modality of
the input data, point clouds or RGBD images, or the tweaks on the loss
function. PC-FractalDB [43] proposes an alternative to typical contrastive
approaches by learning fractal geometry as a pre-training. These methods are
particularly efficient for indoor scene understanding as there is more
available data for this task [44, 45, 46].

Two methods were specfically designed for LSS pre-training: SegContrast [4]
and TARL [47]. SegContrast is inspired by PointContrast but generates positive
pairs by data augmentation rather than registration. TARL leverages a vehicle
model to create object pairs based on the acquisition trajectory and tries to
minimize the distance between their representation in two different point
clouds. In both cases, it is unsupervised training, which they perform on
SemanticKITTI [48].

Our work is at the intersection of these different fields. We propose a novel
method of performing multi-source learning for LSS that can be used either to
improve source-to-source performances or domain generalization performances or
even be used for training by enabling large annotated datasets to be
leveraged.

##  III COLA

###  III-A Datasets

To perform multi-source training, several datasets need to be available. For
this work, we use seven different datasets that we split into two categories:
training sets (SemanticKITTI [48], KITTI-360 [49], nuScenes [50], and Waymo
[51]) and evaluation sets (SemanticPOSS [52], PandaSet [53], ParisLuco3D
[54]).

PandaSet is divided into two subsets called Panda64 and PandaFF. This
partition is made by seperating the scans depending on their acquisition
sensors. Panda64 is made of scans acquired by the 64-fiber LiDAR and PandaFF
by the solid-state LiDAR. Through this, both datasets present the same scene
but with very different acquisition sensors.

Training sets are selected due to their size, all having much more than 20,000
scans dedicated for training. Furthermore, they represent three different
acquisition sensors and three different acquisition settings, providing a
variety of domains for training, and thus can be safely considered a multi-
source dataset. Details about the datasets can be found in Table I. Among
them, SemanticKITTI and nuScenes are considered the standard datasets for LSS
and will be used for comparisons.

Evaluation sets are selected due to their complexity and difference from the
training sets. In order to verify the generalization capabilities of LSS
methods, a significant domain gap needs to be found between training and
evaluation sets. PandaFF, with its solid-state LiDAR, provides a strong sensor
shift, whereas SemanticPOSS and ParisLuco3D provide strong scene shifts by
being acquired in densely populated scenes. The three of them are small
datasets, but they are on-par with sequence 08 from SemanticKITTI, which is
widely used for method comparison. Details about the datasets can be found in
Table I.

Name | # scans | # sequences | # labels | Scene | Country | Sensor
---|---|---|---|---|---|---
SemanticKITTI [48] | 23000 | 11 | 19 | Suburban | Germany | HDL-64E
KITTI-360 [49] | 67626 | 9 | 19 | Suburban | Germany | HDL-64E
nuScenes [50] | 35000 | 850 | 16 | Urban |  | Singapore
---
USA
HDL-32E
Waymo [51] | 30000 | 1150 | 23 |  | Suburban
---
Urban
USA |  | N/C
---
64 fibers
SemanticPOSS [52] | 3000 | 6 | 13 | Campus | China | Pandora
PandaSet [53] | 6000 | 76 | 37 |  | Suburban
---
Urban
USA |  | Pandar64
---
PandarGT
ParisLuco3D [54] | 7500 | 1 | 44 |  | Urban
---
France |  | HDL-32E
---
TABLE I: Summary of the various datasets properties. Number of scans and
number of sequences reflect the available annotated data.

###  III-B Current Limitations

As highlighted by the previous table, every single LSS dataset has its own set
of labels, and no one-to-one correspondence exists between any datasets.
Similar to the issue highlighted by [6], LSS datasets cannot be simply
concatenated to create a multi-source training set. Besides being an issue for
training, it is also an issue for evaluation.

As mentioned in the related work section, applying union or intersection to
create a new label set require strong constraints on the evaluation set, which
are not fulfilled here. Inspecting the label sets clearly reveals how the
union would fail, as some evaluation sets display very fine annotation, namely
PandaSet and ParisLuco3D. Furthermore, the union strategy amplifies the label
unbalance already observed in the autonomous driving datasets.

Regarding the intersection, we observe that even only for the training set, it
raises the issue of large amounts of information discarded, which is
highlighted in Figure 3, as label sets do not even agree on a common
definition of buildings.

Figure 3: Resulting intersection of the training label sets.

Other 3D domain generalization methods provided label mapping , which combined
label sets and grouping labels under a common coarser label to manage cross-
domain evaluation. [31] found a common set for SemanticKITTI, Waymo, nuScenes,
and SemanticPOSS, but it has flaws as some labels are basically discarded in a
background label, and SemanticPOSS cannot be easily mapped to this label set.
[2] needs to define a novel label set for each pair of datasets, making it
especially tedious to use.

###  III-C Coarse labels and their use

Inspired by label tree methods, we propose to leverage a label tree for the
semantic segmentation for autonomous driving. The higher levels of the tree
represent coarser labels, which are common in datasets annotated for similar
tasks. Given sufficiently coarse labels, they can be applied to any autonomous
driving label set for almost no human cost by extracting this tree-like
structure and merging labels together to make these coarse labels.

We identify seven unambiguous labels that can be used for any autonomous
driving dataset: Vehicle, Driveable Ground, Other Ground, Person, Object,
Structure, and Vegetation. The process of how to remap to them is shown in
Figure 4, in which SemanticKITTI’s label set is remapped to the coarse labels.
The mapping from each dataset used in this paper, alongside all the pre-
trained models, will be uploaded to github.

Figure 4: SemanticKITTI’s label set and its mapping to the coarse labels.

The various labels have self-explanatory names except for Structure and
Object. The difference between both stems from nuScenes’ object detection
dataset, which distinguishes them. Objects correspond to countable and
separable objects, such as poles and signs, whereas structures are continuous
background elements, such as buildings and barriers.

We illustrate the coarse label Vehicle and which fine labels it corresponds to
in the training sets in Figure 5. This label set is slightly different from
the one used in [7], as the difference between dynamic and static objects was
deemed ambiguous.

Figure 5: Coarse label vehicle and the training labels that constitute it.

This new label set can be applied to every existing autonomous driving
datasets, allowing us to use all the datasets at once to perform multi-source
training. Furthermore, as this label set can also be applied to evaluation
sets, domain generalization performances can be done on this label set.

##  IV Multi-source domain generalization

###  IV-A Introduction

We defined multi-source domain generalization in the introduction as the
strategy that revolves around using several different datasets at training
time in order to increase robustness. Furthermore, in Section II, we provided
several examples of existing strategies to perform it in 2D, and the lack of
3D methods. Finally, in Section III, we describe a means of leveraging any
autonomous driving datasets simultaneously.

In this section, we apply COLA to propose a multi-source domain generalization
strategy called COLA-DG for 3D semantic segmentation in autonomous driving. We
expect COLA-DG to improve performances compared to single-source learning, as
the COLA set we propose displays a very varied datasets in scenes and
acquisition sensors. Furthermore, this dataset is much bigger than any
existing single-source dataset. As a result, COLA-DG, unlike single-source
domain generalization, can be divided into two components: an increase in the
size of the training set and an increase in the diversity of the training set.
An illustration of this method can be found in Figure 6

Figure 6: Illustration of COLA-DG. It takes as input several datasets
extracted from various domains as input and remaps them to single label set
through COLA. This new dataset is used to learn a domain-robust semantic
segmentation model.

We are interested in understanding the effect of both of these components
separately. Furthermore, we want to understand how diversity improves
generalization performance. More specifically, the manner through which the
various domain shifts are introduced inside the training set is helpful. For
this, this section will be divided into three parts: a preliminary study
proving the usefulness of multi-source training, a study of COLA-DG for
several state-of-the-art models, and an ablation study to understand how
multi-source training works.

###  IV-B Preliminary study

In the preliminary study, we interest ourselves in the impact of introducing
different sensor resolutions inside the training set. Similar to [31, 2], we
use new data by lowering the resolution of SemanticKITTI; specifically, we use
SemanticKITTI-32 (SK32) and SemanticKITTI-16 (SK16) that are emulating 32- and
16-fiber LiDAR. These datasets allow us to evaluate the effect of introducing
sensor shift in the training set without incorporating scene and appearance
shift, as we use the exact same sequences.

Following related works [31, 4, 2], we use SRU-Net as our baseline model. We
investigate its generalization performance depending on the training set,
i.e., trained with only SemanticKITTI and trained with SemanticKITTI and its
subsampled versions.

The domain generalization results from this preliminary experiment can be
found in Table II.

Training sets | SemanticKITTI | SemanticKITTI-16 | SemanticKITTI-32 | SemanticPOSS | ParisLuco3D | Panda64 | PandaFF
---|---|---|---|---|---|---|---
SK | 81.9 | 66.2 | 77.7 | 35.3 | 44.3 | 42.8 | 42.8
SK+SK16 | 79.5 | 73.9 | 77.8 | 33.5 | 52.3 | 46.6 | 41.7
| SK+SK16
---
+SK32
82.5 | 77.5 | 80.8 | 36.9 | 55.2 | 50.7 | 50.7
TABLE II: Impact of the introduction of resolution variety on domain generalization performances, results computed with SRU-Net Method | Training sets | SemanticKITTI | SemanticPOSS | ParisLuco3D | Panda64 | PandaFF
---|---|---|---|---|---|---
KPConv [11] | SK | 78.6 | 27.3 | 39.7 | 25.4 | 16.5
KPConv [11] | COLA-DG | 73.8 | 52.3 | 50.0 | 56.0 | 53.4
Cylinder3D [17] | SK | 81.8 | 32.0 | 41.0 | 29.1 | 7.6
Cylinder3D [17] | COLA-DG | 82.5 | 56.6 | 67.0 | 59.8 | 28.3
SRU-Net [15] | SK | 81.9 | 35.3 | 44.3 | 42.8 | 42.8
SRU-Net [15] | COLA-DG | 75.1 | 57.7 | 59.6 | 62.5 | 57.8
TABLE III: COLA-DG performances compared to single-source training.

The first observation is that the introduction of sensor shift in the training
set (SK+SK16) improves domain generalization results, specifically toward
lower resolution datasets (SK+SK16$\rightarrow$PL3D) even if the resolution
did not appear in the training sensors. Nonetheless, a slight decrease in the
high resolution sensor is observed (SK+SK16$\rightarrow$SK and
SK+SK16$\rightarrow$PFF).

When increasing the amount of sensor shifts in the training set
(SK+SK16+SK32), every case improved. The improvement is especially large for
datasets sharing sensor topology with the newly introduced training sets
(PL3D) but was also seen in unrelated evaluation sets (SK, SP).

Overall, introducing sensor diversity improves domain generalization
performance, even for unseen sensors. This preliminary experiment predicts
good generalization capabilities for multi-source approaches.

###  IV-C COLA for COLA-DG

For the following experiments, we used three different semantic segmentation
models: KPConv [11], Cylinder3D [17], and SRU-Net [15]. These are the models
currently used by current domain generalization approaches [31, 1, 2, 3]. In
[2, 3], it is shown that the naive utilisation of the reflectivity channel is
detrimental for domain generalization. Based on these observations, we decided
to discard the reflectivity channel for all domain generalization experiments.

To assess the usefulness of COLA-DG for domain generalization, for each model
we performed two experiments. We trained each model with only SemanticKITTI
(SK) and then with the COLA dataset, which is the concatenation of Waymo,
SemanticKITTI, KITTI-360, and nuScenes. In both cases, we used the coarse
labels for the training and for evaluation.

Each experiment was done with the same number of epochs (5), as every models
was observed to have reached its peak validation performance on the training
data by this point. The models were trained by following the parameters found
in their respective official repositories as follows: Cylinder3D
https://github.com/xinge008/Cylinder3D, KPConv
https://github.com/HuguesTHOMAS/KPConv-PyTorch, SRU-Net
https://github.com/mit-han-lab/spvnas.

The quantitative results can be found in Table III.

The first global observation is the systematic improvement for domain
generalization when models were trained with COLA, regardless of the model. As
mentioned before, it stems from two effects: the size and the domain variety
inside the COLA training set.

We want to demonstrate that it is indeed domain variety that is the cause of
domain generalization improvement. As dataset size cannot be left out of our
analysis, we will dissect both of these effects in Section IV.D. For the
remainder of this discussion, we will assume that indeed, domain variety
improves domain generalization performance.

Performance improvements are significant for each domain shift: the
significant improvement when evaluating ParisLuco3D and PandaFF proves
resilience to sensor shift, and the significant improvement when evaluating on
SemanticPOSS and Panda64 proves resilience to scene and appearance shifts.

KPConv uses the z-coordinate as an input vector for semantic segmentation. As
such, when trained on only one dataset, it is very sensitive to sensor
location and displays mediocre domain generalization results. When it is
trained on a variety of data, this overfitting pattern is partially erased,
which results in drastic improvement (from +11.3% to +46.9%). Nonetheless,
source-to-source results are slightly reduced.

Cylinder3D results are particularly remarkable. In [2], it is shown that
Cylinder3D is prone to overfitting and is not a very good generalization
model. Similar results were obtained in our single-source experiment. However,
when Cylinder3D is trained with COLA, it becomes a very strong generalization
model, displaying the best source-to-source results alongside the best results
on ParisLuco3D.

These newfound performances can be explained the same way we can explain the
overfitting in the single-source case: the pre-processing PointNet. For
Cylinder3D, voxelwise features are extracted with a PointNet before the
segmentation architecture. While this PointNet overfits when fed only one
dataset, it generalizes very well when fed a variety of domains. It is a known
results in the registration field [55, 56].

The very low performance on PandaFF stems from the cylindrical voxelization,
which is not parameterized for long-range sensors.

SRU-Net, as found in [2], is the single-source generalization model. Despite
the initially decent performances, multi-source training improves them
significantly. Similarly to KPConv, a decrease in source-to-source performance
is observed.

Overall, the considered label set is very simple, but we can already observe
patterns in the capacity of COLA-DG as a way to tackle domain generalization.

###  IV-D Impact of the data variety for COLA-DG

As mentioned beforehand, we are interested in understanding the effect of
introducing new datasets on for domain generalization performance. In Table
IV, we incrementally add datasets to understand how variety and size affect
generalization.

Here, KITTI-360 plays the role of an expanded SemanticKITTI. Both datasets are
acquired in the same location with a similar acquisition setup, but KITTI-360
is much larger. This way we can assume that there is almost no domain shift
between them, and KITTI-360 serves to gauge the effect of dataset size when
compared with SemanticKITTI.

Using KITTI-360 improves overall generalization results, showing that dataset
size has an impact on the performances. Improvement is nonetheless smaller
than when using more diverse training sets.

Results are quite expected and confirm implicit hypotheses. Adding nuScenes to
the training set improves results on ParisLuco3D, which employs the same
acquisition sensor as using KITTI-360 and SemanticKITTI. Similarly,
introducing Waymo in the training set improves results over Panda64.

Using all the datasets together results in a larger improvement than the best
improvement when using datasets one at a time. It shows that variety and size
interact together and that COLA is better than the sum of its part.

As COLA-based training could be considered overly simple, we compare it with a
method referenced in a related work, the multi-head (MH) strategy. For domain
generalization, each head is trained with the fine label set of the associated
training set. In our case, there are four heads. At inference time, the score
of the heads are remapped to the coarse labels. Then, the scores for each head
are added together, and a coarse label can be predicted.

This inference strategy has two effects: when an inference set is similar to a
training set, its associated head will be very confident and therefore provide
the final classification, otherwise it will be an average of every head. This
results in very good results for sensors very close to only one of the
training sets, in the case of ParisLuco3D. In the other cases, it performs
much worse than COLA.

The generalization performances of COLA-DG compared to MH demonstrates why
this method is interesting and efficient.

Training sets | SK | SP | PL3D | P64 | PFF
---|---|---|---|---|---
SK | 81.9 | 44.3 | 35.3 | 42.8 | 42.8
K360 | 61.8 | 42.5 | 50.1 | 44.2 | 42.7
K360 + SK | 70.7 | 43.2 | 48.5 | 49.9 | 47.4
K360 + NS | 61.0 | 46.3 | 53.5 | 48.7 | 39.4
K360 + W | 63.1 | 54.3 | 52.4 | 59.5 | 52.2
K360 + SK + W + NS | 75.1 | 57.7 | 59.6 | 62.5 | 57.8
K360 + SK + W + NS (MH) | 59.9 | 54.2 | 62.6 | 49.4 | 30.3
TABLE IV: Impact of data variety on domain generalization performance; results
computed with SRU-Net.

###  IV-E Benchmark multi-source domain generalization

In addition to the analysis presented here, we expand the ParisLuco3D [54]
online benchmark to include a track with the coarse labels. This new track
will allow people to compare multi-source domain generalization methods on a
unified label set. This new track will be available soon.

##  V Multi-source LiDAR semantic segmentation

###  V-A Intuition

In the previous section, we restricted the discussion of one result in Table
III. When trained with COLA, Cylinder3D improves its source-to-source
segmentation performance. This result suggests that multi-source training
could be used to perform LSS. We call this task multi-source LiDAR semantic
segmentation.

In this task, we expect the evaluation set to be available, at least
partially, during training as it would be for typical LSS. Based on this
hypothesis, we design a pipeline that can allow us to leverage multi-source
information for source-to-source segmentation, called COLA-S2S. COLA-S2S is
illustrated in Figure 7.

The difference between COLA-DG and COLA-S2S is that COLA-S2S requires the
inference labels to be the same as the fine label set of the target, which
means COLA cannot be applied throughout the pipeline. We propose a two-step
training process. First, following the COLA-DG training process, the COLA-S2S
model is trained with a concatenation of datasets, including the target set,
supervised by the coarse labels. This step allows the model to learn useful
deep geometric primitives and disambiguates the main road components. Then,
the COLA-S2S model is finetuned with the target set only, supervised by its
fine labels.

Figure 7: Illustration of COLA-S2S. In the first step, it takes several
datasets extracted from various domains as input and remaps them to a single
label set through COLA. This new dataset, which includes the target set, is
used to learn a domain-robust semantic segmentation model. This model is then
refined with the target dataset to be able to infer on the right label set.

The remainder of this section describes the experimental protocols more
precisely and presents the results.

###  V-B Experiments

We setup two experiments. As COLA-S2S is expected to leverage large amounts of
data, we use nuScenes (NS) and SemanticKITTI (SK) as target sets, to ensure
that standard LSS methods can still learn and that we can evaluate the
usefulness of COLA-S2S. All experiments were performed with SRU-Net and
Cylinder3D. SRU-Net is the baseline for LSS, and Cylinder3D is reaching state-
of-the-art performances.

Inspired by related works [4, 47], we emulate to cases: one where the target
set is fully finely annotated and one where the target set is partially finely
annotated and fully coarsely annotated. For the second case, the coarse labels
are known for all the data but only a small part has the target label
annotations.

Following the TARL [47] protocol, found in https://github.com/PRBonn/TARL, we
trained our SRU-Net models with the same number of epochs and the same
learning rate as the one the TARL authors propose for finetuning. For the
first training step, we re-employ the model trained for COLA-DG, which do not
use reflectivity. For the second step, we add reflectivity to our data to
ensure the best performances.

In the case of low fine annotation availability, we propose to add a baseline
that uses only coarsely annotated SemanticKITTI as the pre-training set. This
way, we can validate the usefulness of coarse-level annotation for single-
source training, and we can assess the improvement done by using COLA-S2S.

We compare our approach with SegContrast [4] and TARL [47]. These methods are
designed for pre-training and not COLA-S2S, but they are involved in similar
tasks as us: leveraging information available in the dataset to improve LSS
performances. They also involve the same use cases, namely low and full
availability of fine annotations. One difference is that for pre-training they
use only the dataset used for finetuning. Due to availability, we performed
comparison only on SRU-Net.

###  V-C Results

Results for SRU-Net can be found in Table V and Table VI, and results for
Cylinder3D in Table VII and Table VIII.

COLA-S2S results for SRU-Net on full data availability are particularly good,
showing a significant improvement for both SemanticKITTI (+5.3%) and nuScenes
(+4.0%) over the standard LSS method. Furthermore, COLA-S2S is a more
meaningful strategy than geometry-based exploitation of the data as it
outperforms SegContrast and TARL. This result highlights the importance of
data diversity for LSS.

Regarding the low availability experiment, COLA-S2S proves to be useful, being
on par with the state-of-art method, TARL. Nonetheless, we again demonstrate
the benefit of using a multi-source dataset rather than a single-source
dataset, as COLA-S2S systematically outperforms using only coarsely annotated
SemanticKITTI for training.

Method | SK | NS
---|---|---
No pre-training | 59.6 | 66.0
SegContrast [4] | 60.5 | 67.7
TARL [47] | 61.5 | 68.3
COLA-S2S | 64.9 | 70.0
TABLE V: COLA-S2S results on nuScenes and SemanticKITTI; computed on SRU-Net. Method | 0.1% | 1 % | 10% | 50% | 100%
---|---|---|---|---|---
No pre-training | 25.6 | 41.7 | 53.9 | 58.3 | 59.6
SK | 36.3 | 50.2 | 59.5 | 62.8 | 62.5
SegContrast [4] | 34.8 | 47.4 | 55.2 | 58.3 | 60.5
TARL [47] | 38.6 | 51.4 | 60.3 | 61.4 | 61.5
COLA-S2S | 38.3 | 51.8 | 58.0 | 60.8 | 64.9
TABLE VI: COLA-S2S results on SemanticKITTI, depending on the amount of fine
labels available; computed on SRU-Net.

COLA-S2S results for Cylinder3D are mixed. On full data availability, COLA-S2S
provides minor improvement for nuScenes (+0.4%) and none for SemanticKITTI. In
the case of low availability, COLA-S2S improves results over using only
SemanticKITTI for training, similar to the SRU-Net case.

Overall, COLA-S2S is an interesting strategy that can improve results over
class LSS training methods, for a very small cost, as it leverages coarse
annotations. It is specifically useful for SRU-Net, allowing it to reach
better performances than Cylinder3D thanks to COLA-S2S despite having 4% less
mIoU (mean Intersection-over-Union) with a standard approach.

Method | SK | NS
---|---|---
No pre-training | 63.7 | 74.8
COLA-S2S | 63.0 | 75.2
TABLE VII: COLA-S2S results on nuScenes and SemanticKITTI; computed on Cylinder3D. Method | 0.1% | 1 % | 10% | 50% | 100%
---|---|---|---|---|---
No pre-training | 34.4 | 48.5 | 58.2 | 61.1 | 63.7
SK | 43.8 | 51.3 | 58.0 | 61.4 | 61.3
COLA-S2S | 46.5 | 53.7 | 60.5 | 62.2 | 63.0
TABLE VIII: COLA-S2S results on SemanticKITTI, depending on the amount of fine
labels available; computed on Cylinder3D.

##  VI Pre-training for LiDAR semantic segmentation

###  VI-A Intuition

Based on great COLA-DG results and encouraging COLA-S2S results, the next step
would be to consider multi-source pre-training. As a reminder, the objective
of pre-training neural networks is to learn re-usable geometric primitives
that can be easily leveraged during finetuning to enhance final performance or
to require less finetuning data. Here, the pre-training and finetuning tasks
are the same, namely semantic segmentation. Based on COLA-DG results, we have
a guarantee that learned primitives are useful in new unseen data. Our
proposed multi-source approach to pre-training is called COLA-PT and is
illustrated in Figure 8

Contrary to COLA-S2S, COLA-PT expects finetuning data to be unavailable during
pre-training. The training pipeline is as follows: the COLA-PT model is pre-
trained with COLA, then the COLA-PT model is finetuned with the target data
supervised by its fine label set.

Figure 8: Illustration of COLA-PT. In The first step, it takes several
datasets extracted from various domains as input and remaps them to single
label set through COLA. This new dataset is used to learn a domain-robust
semantic segmentation model. This model is then refined with the target
dataset to be able to infer on the right label set.

In this section, we introduce our experiments, propose a qualitative analysis
displaying why COLA can serve as a useful pre-training method, and finally
present the quantitative results.

###  VI-B Experiments

We follow a very similar protocol to the one for COLA-S2S. The main difference
is the target set. Here it is SemanticPOSS. Despite the use of SemanticKITTI
as the target in the literature, we do not use it here, as the resulting pre-
training set was deemed too small. Indeed, for fairness of evaluation, when
SemanticKITTI is the target set, KITTI-360 would need to be removed from the
pre-training set, resulting in a much smaller dataset. When using SemanticPOSS
as the target set, we are not subject to this issue.

We study the finetuning results on the full SemanticPOSS dataset and on
various subsets of annotated frames. We introduce a baseline supervised pre-
training method that uses only SemanticKITTI for pre-training. This way, we
can gauge the impact of COLA-PT relative to simple supervised pre-training. We
perform these experiments with SRU-Net and Cylinder3D. Furthermore, unlike in
previous sections, we perform these experiments with and without the
reflectivity channel. While it made sense to not use it for generalization,
and to use it for source-to-source segmentation, here, for pre-training, it
was not clear which one was better. Indeed, following COLA-DG, the pre-trained
model is trained without reflectivity but, contrary to COLA-S2S, the
finetuning dataset is small, and it is unsure whether the model will
transition smoothly from data without reflectivity for pre-training and with
reflectivity for finetuning.

We compare our results with SegContrast [4], which is the only method that
performed this experiment. Following the SegContrast protocol, found in
https://github.com/PRBonn/segcontrast, we trained our SRU-Net models with the
same number of epochs and the same learning rate as the one the SegContrast
authors propose for finetuning.

In their paper, the SegContrast authors concluded that geometric unsupervised
pre-training results in better performance than supervised pre-training. Their
results back their claims. Our work differs from theirs in that, we leverage a
larger amount of available annotated data.

(a) SemanticKITTI

(b) COLA-PT

(c) SegContrast

Figure 9: t-SNE analysis of the feature extracted by an SRU-Net depending on
the pre-training strategy. From left to right: SemanticKITTI, COLA-PT,
SegContrast.

###  VI-C Qualitative results

We first perform a qualitative analysis to highlight the generalization power
of our pre-training and to show its meaningfulness compared to geometric pre-
training. For this, we do a t-SNE analysis of extracted features from the
target set before finetuning. This way, we can see if some information is
already in the embedded space, even though the target set was never seen by
the pre-trained model. We then color each point by its associated label. These
graphs can be seen in Figure 9.

We expect supervised pre-training to semantically pre-cluster points in the
t-SNE space, as they have already learned coarse semantic information, whereas
SegContrast was only exposed to geometric information.

We observed that SegContrast is confused between classes of similar geometry
or with similar neighbors, such as Traffic Sign and Pole. Inversely,
supervised pre-training can manage to disambiguate classes based on their
semantic such as Car, Building, and Fence. Nonetheless, the impact of the
supervised pre-training label set can be observed as there is confusion
between Rider and Pedestrian, which belong to the same coarse label.

Classes that are very distinctive both geometrically and semantically, such as
ground, are well recognized by all types of pre-training.

Qualitatively, we see that classes seem to be more understood by semantic pre-
training, proving its usefulness.

###  VI-D Results

Even though the qualitative analysis makes it appear that the understanding of
SemanticPOSS is way better at the end of the supervised pre-training, the
quantitative results are not as clear cut.

For SRU-Net (Table IX), we can see that COLA-PT outperforms systematically the
single-source supervised pre-training.

Then, compared to SegContrast, COLA-PT is more efficient for very low amounts
of available data (0.01%) and is on par in other cases, which means that it
achieves state-of-the-art performances. We observed that despite the pre-
trained model not using reflectivity it, is more valuable to use it for
finetuning.

Method | 0.1% | 1 % | 10% | 50% | 100%
---|---|---|---|---|---
No pre-training | 33.1 | 43.1 | 57.3 | 63.3 | 64.2
SK | 42.9 | 54.4 | 60.2 | 64.3 | 64.5
SegContrast [4] | 43.7 | 55.2 | 60.3 | 64.6 | 64.9
COLA-PT w/o r | 44.5 | 53.1 | 58.8 | 62.0 | 63.3
COLA-PT | 45.1 | 54.5 | 60.6 | 63.6 | 64.9
TABLE IX: Finetuning results on SemanticPOSS, depending on the amount of
labels available, computed on SRU-Net.

On Cylinder3D (Table X), results are slightly different from those of SRU-Net.
While COLA-PT results in the best outcome in most cases, it is unclear if it
is useful to use reflectivity or not as an input channel. For a very small
amount of data (0.1%) COLA-PT does not bring an improvement, which is
surprising.

Method | 0.1% | 1 % | 10% | 50% | 100%
---|---|---|---|---|---
No pre-training | 39.6 | 52.1 | 59.0 | 64.3 | 65.0
SK | 38.4 | 52.4 | 60.3 | 64.6 | 65.6
COLA-PT w/o r | 35.7 | 52.9 | 58.3 | 64.6 | 65.9
COLA-PT | 38.7 | 51.1 | 60.1 | 65.2 | 65.5
TABLE X: Finetuning results on SemanticPOSS, depending on the amount of labels
available, computed on Cylinder3D.

Overall COLA-PT, is a strong choice of semantic segmentation pre-training, and
it reaches state-of-the-art performances in several cases.

###  VI-E Influence of the COLA-PT label set

To conclude on COLA-PT, we would like to discuss the effect that the pre-
training label set has on finetuning performances. Here, because the pre-
trained model is not concerned with evaluation, methods such as interesection-
and union-based training can be used.

As such, we study six different approaches for labeling of the pre-training
set in two different cases: 0.1% available label, which is the hardest
learning task, and 100% available label.

These six different strategies are as follows:

  * •

COLA: our coarse labels method (seven labels),

  * •

COLA-5: a reduction to five labels of COLA (Ground, Vehicle, Manmade,
Pedestrian, Vegetation),

  * •

COLA-9: an extension to nine labels of COLA (Driveable Ground, Structure,
4-Wheeled, Nature, Pedestrian, Object, Other Ground, Pole&Sign, 2-Wheeled),

  * •

Intersection: Keep the nine labels identified in Figure 3,

  * •

Union: use the 32 different labels that compose the union of four training
label sets,

  * •

MH: the multi-head method.

The results are shown in Table XI. The finer the pre-training label set is,
the better it performs for very low amounts of data available. Conversely, the
coarser the label, the better it performs at high amounts of available data.

It must be noted that COLA strategies are independent of the original label
sets, whereas Union, Intersection, and MH are not.

Method | # labels | 0.1% | 100%
---|---|---|---
No pre-training | N/A | 33.1 | 64.2
COLA | 7 | 44.5 | 64.9
COLA-5 | 5 | 43.7 | 65.0
COLA-9 | 9 | 45.2 | 64.8
Intersection | 9 | 43.3 | 64.6
Union | 32 | 46.6 | 63.6
MH | 59 | 45.3 | 64.6
TABLE XI: Label set choice influence on SemanticPOSS finetuning results;
computed on SRU-Net.

##  VII Conclusion

We have introduced a novel relabelling strategy called COLA that can be used
to perform multi-source training for almost no human cost. This way, we
performed, to the best of our knowledge, the first multi-source experiments
for LiDAR semantic segmentation in autonomous driving.

We explored the usefulness of multi-source training to improve domain
generalization, source-to-source segmentation, and pre-training performances.
We achieved systematic improvements and even demonstrated that these results
were robust relative to the label set chosen.

We demonstrated the importance and impact of multi-source training and
presented it as an alternative to single-source strategies in the
aforementioned fields.

## References

  * [1] C. Saltori, A. Ošep, E. Ricci, and L. Leal-Taixé, “Walking your lidog: A journey through multiple domains for lidar semantic segmentation,” 2023.
  * [2] J. Sanchez, J.-E. Deschaud, and F. Goulette, “Domain generalization of 3d semantic segmentation in autonomous driving,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 18 077–18 087.
  * [3] A. Lehner, S. Gasperini, A. Marcos-Ramiro, M. Schmidt, N. Navab, B. Busam, and F. Tombari, “3d adversarial augmentations for robust out-of-domain predictions,” _arXiv preprint arXiv:2308.15479_ , 2023.
  * [4] L. Nunes, R. Marcuzzi, X. Chen, J. Behley, and C. Stachniss, “Segcontrast: 3d point cloud feature representation learning through self-supervised segment discrimination,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 2116–2123, 2022.
  * [5] L. Soum-Fontez, J.-E. Deschaud, and F. Goulette, “MDT3D: Multi-Dataset Training for LiDAR 3D Object Detection Generalization,” _arXiv e-prints_ , p. arXiv:2308.01000, Aug. 2023.
  * [6] Y. Liang, H. He, S. Xiao, H. Lu, and Y. Chen, “Label name is mantra: Unifying point cloud segmentation across heterogeneous datasets,” _CoRR_ , vol. abs/2303.10585, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.10585
  * [7] J. Sanchez, J.-E. Deschaud, and F. Goulette, “Cola: Coarse label pre-training for 3d semantic segmentation of sparse lidar datasets,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 11 343–11 350.
  * [8] J. Lambert, Z. Liu, O. Sener, J. Hays, and V. Koltun, “Mseg: A composite dataset for multi-domain semantic segmentation,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 2876–2885.
  * [9] M. Tatarchenko, J. Park, V. Koltun, and Q.-Y. Zhou, “Tangent convolutions for dense prediction in 3d,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3887–3896.
  * [10] Y. Xu, T. Fan, M. Xu, L. Zeng, and Y. Qiao, “Spidercnn: Deep learning on point sets with parameterized convolutional filters,” in _Computer Vision – ECCV 2018_ , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 90–105.
  * [11] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. Guibas, “Kpconv: Flexible and deformable convolution for point clouds,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6410–6419.
  * [12] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “Rangenet ++: Fast and accurate lidar semantic segmentation,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 4213–4220.
  * [13] C. Xu, B. Wu, Z. Wang, W. Zhan, P. Vajda, K. Keutzer, and M. Tomizuka, “Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 1–19.
  * [14] Y. Zhang, Z. Zhou, P. David, X. Yue, Z. Xi, B. Gong, and H. Foroosh, “Polarnet: An improved grid representation for online lidar point clouds semantic segmentation,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9598–9607.
  * [15] C. Choy, J. Gwak, and S. Savarese, “4d spatio-temporal convnets: Minkowski convolutional neural networks,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3070–3079.
  * [16] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han, “Searching efficient 3d architectures with sparse point-voxel convolution,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 685–702.
  * [17] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 9934–9943.
  * [18] J. Xu, R. Zhang, J. Dou, Y. Zhu, J. Sun, and S. Pu, “Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 16 024–16 033.
  * [19] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, “Generalizing to unseen domains: A survey on domain generalization,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–1, 2022.
  * [20] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–20, 2022.
  * [21] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [22] D. Li, J. Zhang, Y. Yang, C. Liu, Y.-Z. Song, and T. Hospedales, “Episodic training for domain generalization,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 1446–1455.
  * [23] M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi, “Domain generalization for object recognition with multi-task autoencoders,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2551–2559.
  * [24] F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2229–2238.
  * [25] R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese, “Generalizing to unseen domains via adversarial data augmentation,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [26] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1501–1510.
  * [27] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Style normalization and restitution for domain generalization and adaptation,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 3636–3651, 2022.
  * [28] X. Pan, P. Luo, J. Shi, and X. Tang, “Two at once: Enhancing learning and generalization capacities via ibn-net,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 464–479.
  * [29] T. Matsuura and T. Harada, “Domain generalization using a mixture of multiple latent domains,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 2020, pp. 11 749–11 756.
  * [30] I. Albuquerque, J. Monteiro, M. Darvishi, T. H. Falk, and I. Mitliagkas, “Generalizing to unseen domains via distribution matching,” _arXiv preprint arXiv:1911.00804_ , 2019.
  * [31] H. Kim, Y. Kang, C. Oh, and K.-J. Yoon, “Single domain generalization for lidar semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 17 587–17 598.
  * [32] G. Ros, S. Stent, P. F. Alcantarilla, and T. Watanabe, “Training constrained deconvolutional networks for road scene semantic segmentation,” _ArXiv_ , vol. abs/1604.01545, 2016.
  * [33] P. Bevandić, M. Oršić, I. Grubišić, J. Šarić, and S. Šegvić, “Multi-domain semantic segmentation with overlapping labels,” in _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2022, pp. 2422–2431.
  * [34] X. Zhao, S. Schulter, G. Sharma, Y.-H. Tsai, M. Chandraker, and Y. Wu, “Object detection with a unified label space from multiple datasets,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 178–193.
  * [35] M. Leonardi, D. Mazzini, and R. Schettini, “Training efficient semantic segmentation cnns on multiple datasets,” in _Image Analysis and Processing – ICIAP 2019_ , E. Ricci, S. Rota Bulò, C. Snoek, O. Lanz, S. Messelodi, and N. Sebe, Eds. Cham: Springer International Publishing, 2019, pp. 303–314.
  * [36] X. Zhou, V. Koltun, and P. Krähenbühl, “Simple multi-dataset detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7571–7580.
  * [37] G. Varma, A. Subramanian, A. Namboodiri, M. Chandraker, and C. Jawahar, “Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2019, pp. 1743–1751.
  * [38] P. Meletis and G. Dubbelman, “Training of convolutional networks on multiple heterogeneous datasets for street scene semantic segmentation,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 1045–1050.
  * [39] X. Liang, E. Xing, and H. Zhou, “Dynamic-structured semantic propagation network,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 752–761.
  * [40] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany, “Pointcontrast: Unsupervised pre-training for 3d point cloud understanding,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 574–591.
  * [41] J. Hou, B. Graham, M. Nießner, and S. Xie, “Exploring data-efficient 3d scene understanding with contrastive scene contexts,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 15 582–15 592.
  * [42] A. Thabet, H. Alwassel, and B. Ghanem, “Self-supervised learning of local features in 3d point clouds,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 4048–4052.
  * [43] R. Yamada, H. Kataoka, N. Chiba, Y. Domae, and T. Ogata, “Point cloud pre-training with natural 3d structures,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 21 283–21 293.
  * [44] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2432–2443.
  * [45] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “ShapeNet: An Information-Rich 3D Model Repository,” Stanford University — Princeton University — Toyota Technological Institute at Chicago, Tech. Rep. arXiv:1512.03012 [cs.GR], 2015.
  * [46] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1912–1920.
  * [47] L. Nunes, L. Wiesmann, R. Marcuzzi, X. Chen, J. Behley, and C. Stachniss, “Temporal consistent 3d lidar representation learning for semantic perception in autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5217–5228.
  * [48] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “Semantickitti: A dataset for semantic scene understanding of lidar sequences,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 9296–9306.
  * [49] Y. Liao, J. Xie, and A. Geiger, “Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 3, pp. 3292–3310, 2023.
  * [50] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 618–11 628.
  * [51] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, “Scalability in perception for autonomous driving: Waymo open dataset,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 2443–2451.
  * [52] Y. Pan, B. Gao, J. Mei, S. Geng, C. Li, and H. Zhao, “Semanticposs: A point cloud dataset with large quantity of dynamic instances,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020, pp. 687–693.
  * [53] P. Xiao, Z. Shao, S. Hao, Z. Zhang, X. Chai, J. Jiao, Z. Li, J. Wu, K. Sun, K. Jiang _et al._ , “Pandaset: Advanced sensor suite dataset for autonomous driving,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2021, pp. 3095–3101.
  * [54] J. Sanchez, L. Soum-Fontez, J.-E. Deschaud, and F. Goulette, “Parisluco3d: A high-quality target dataset for domain generalization of lidar perception,” 2023.
  * [55] S. Horache, J. Deschaud, and F. Goulette, “3d point cloud registration with multi-scale architecture and unsupervised transfer learning,” in _2021 International Conference on 3D Vision (3DV)_. Los Alamitos, CA, USA: IEEE Computer Society, dec 2021, pp. 1351–1361. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/3DV53792.2021.00142
  * [56] F. Poiesi and D. Boscaini, “Learning general and distinctive 3d local deep descriptors for point cloud registration,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 3, pp. 3979–3985, 2022.

Jules Sanchez is a PhD canditate in computer science and robotics at Mines
Paris - PSL University and plan on finishing at the end of 2023. His research
topics mainly focus on LiDAR semantic segmentation and domain generalization
for autonomous vehicles, with publications in international conferences on
Computer Vision and Robotics.
---
Jean-Emmanuel Deschaud received his PhD degree in computer science and
robotics in 2010. He is currently an Associate Professor at the Centre for
Robotics at Mines Paris - PSL University. His research topics center on LiDAR
SLAM, neural networks for point clouds, and 3D perception for autonomous
vehicles, with publications in international conferences and journals on
Computer Vision and Robotics.
---
François Goulette graduated from Mines Paris with an engineering degree in
1992 and a PhD in computer science and robotics in 1997. He obtained a
Habilitation to Conduct Research from Sorbonne University in 2009. He worked
for a few years as a research engineer at the French electrical company EDF
and then as an assistant, associate and full Professor at Mines Paris - PSL
University. Since 2022, he has served as a full professor and deputy director
of the ”U2IS” Lab at ENSTA Paris - Institut Polytechnique de Paris. His
research interests are 3D perception, mobile mapping, photogrammetry, 3D data
processing for robotics, autonomous vehicles, and other applications.
---