# Depth Based Semantic Scene Completion with Position Importance Aware Loss

Yu Liu${}^{1,3^{*}}$, Jie Li${}^{2^{*}}$, Xia Yuan2, Chunxia Zhao2, Roland
Siegwart3, Ian Reid1 and Cesar Cadena3 Yu Liu and Jie Li contributed equally
to this work, part of the work was done when Yu Liu doing internship at ETH
ASL. (Corresponding author: Cesar Cadena.)1Y. Liu and I. Reid are with School
of Computer Science, The University of Adelaide, 5005, North Terrace, SA
<EMAIL_ADDRESS>Li, X. Yuan and C. Zhao are with School of
Computer Science and Engineering, Nanjing University of Science and
Technology, Nanjing, 210094, China<EMAIL_ADDRESS>Siegwart and C. Cadena
are with Autonomous Systems Lab, ETH Zurich, Leonhardstrasse 21, 8092, Zurich
<EMAIL_ADDRESS>
###### Abstract

Semantic Scene Completion (SSC) refers to the task of inferring the 3D
semantic segmentation of a scene while simultaneously completing the 3D
shapes. We propose PALNet, a novel hybrid network for SSC based on single
depth. PALNet utilizes a two-stream network to extract both 2D and 3D features
from multi-stages using fine-grained depth information to efficiently captures
the context, as well as the geometric cues of the scene. Current methods for
SSC treat all parts of the scene equally causing unnecessary attention to the
interior of objects. To address this problem, we propose Position Aware
Loss(PA-Loss) which is position importance aware while training the network.
Specifically, PA-Loss considers Local Geometric Anisotropy to determine the
importance of different positions within the scene. It is beneficial for
recovering key details like the boundaries of objects and the corners of the
scene. Comprehensive experiments on two benchmark datasets demonstrate the
effectiveness of the proposed method and its superior performance. Code and
demo111Video demo can be found here:  https://youtu.be/j-LAMcMh0yg are
avaliable at: https://github.com/UniLauX/PALNet.

###### Index Terms:

Semantic Scene Understanding, Deep Learning in Robotics and Automation, RGB-D
Perception

##  I INTRODUCTION

Semantic scene completion (SSC) [1, 2, 3] is composed of scene completion and
semantic labeling, which targets at inferring the completed 3D shape and the
semantic categories of the occupied voxels (grid units) in the scene
simultaneously. As illustrated in Fig. 1, given a single-view depth, SSC
assigns a class label to every voxel including the surface of the scene and
the occluded parts in the frustum. SSC is useful and widely applied in
numerous real-world applications such as virtual reality [4, 5], robot
grasping [6], automatic navigation [7], etc.

Figure 1: Semantic Scene Completion: Based on input depth or its corresponding
3D voxel volume, the goal of SSC is to simultaneously complete the partial 3D
shapes and predict the dense semantic labels of both observed and unobserved
parts in the view frustum.

3D convolutional neural network (3D-CNN) is frequently used in the task of 3D
scene prediction. Since 3D-CNN requires a regular grid as input, voxels are
naturally chosen to represent the 3D scene in these 3D-CNN based methods, and
different loss functions are employed for training them [3, 8, 9].

A limitation of previous methods is that they ignore the position importance
of the voxels. Usually, all of the voxels are treated equally regardless of
their positions, appearance, and shape. However, from the human perception
system [10, 11], when we recognize objects, the two most prominent features
that make the target object distinguish from other surrounding objects are the
corners and uneven surfaces. Furthermore, there are a large amount of voxels
located within the objects which are likely to share the similar texture and
are redundant for classification task. A relatively small portion of the
voxels (at the edges of a scene or the corners of the objects) are the
critical clues for identifying the classes. Based on these observations, we
propose Local Geometric Anisotropy (LGA) to quantitatively determine the
importance of different positions, and further introduce Position Aware Loss
(PA-Loss) which gives different importance to voxels based on their LGA. With
PA-Loss, the network is forced to pay more attention to the surfaces, edges,
and corners which leads to slightly faster convergence and a boost in
performance of the SSC task.

Another limitation of the previous methods is that they do not use the
complete depth information. During voxelization, several pixels in depth can
be projected into one voxel in the 3D grid volume and due to the memory
limitation, the input resolution used in 3D-CNN is unlikely to be very large,
which leads to a coarse voxel level representation. In particular, a common
voxelized input for 3D-CNN models is $240\times 144\times 240$ [3, 12, 13],
while the original input depth has resolution of 640 $\times$ 480 or much
higher. In this paper, we take both the advantages of the full resolution
depth and the encoded voxels by designing a hybrid network structure with two
input branches. In the other branch, we feed the network directly with the
original depth to avoid the loss of detailed information. In another branch,
we adopt the 3D voxels as the input to take advantage of the Truncated Signed
Distance Field (TSDF) encoding [14]. Through the use of differentiable 2D-3D
projection layer, the 2D CNN and 3D CNN blocks are effectively combined within
the network.

The main contributions of this paper are two-fold.

  * •

Firstly, we propose a novel PA-Loss for SSC task, which emphasizing the rare
voxels that are located on the surface or corners of the scene, while diluting
the voxels which carry redundant information within the objects. Our
experiments indicate that PA-Loss has the benefit of slightly faster
convergence for training and can achieve better performance than previous
works.

  * •

Secondly, based on single-view depth, we propose a hybrid network, which takes
full advantage of both fine-grained depth and TSDF. The detail information
extracted from the full resolution depth is beneficial for semantic labeling
as well as scene completion, which also distinguish our approach from the
existing mainstream approaches that only use the voxelized TSDF as the sole
input.

##  II Related work

###  II-A Semantic Scene Completion

Recently several methods have been proposed for SSC using deep learning
techniques [3, 12, 13, 8]. Among them, the most representative work is the
SSCNet [3] which conducts the semantic labeling and scene completion
simultaneously and also proves that these two tasks can benefit from each
other. SSCNet takes advantage of TSDF and uses voxels to represent the 3D
space. Although better results have been achieved compared with the previous
methods, SSCNet ignores the fine-grained information of depth. Zhang et al.
[13] introduces spatial group convolution (SGC) to reduce the computation
costs but with poor performance than SSCNet [3]. SEGCloud [9] employs fine-
grained 3D point as input but the computing and memory costs are incredibly
high. VVNet [8] combines 2D-CNN and 3D-CNN by replacing some 3D volume layers
with the corresponding 2D view network layers which leads to a much accurate
and efficient network compared with SSCNet. However, it discards TSDF, which
can provide the prior knowledge about the space encoding and is vital to
distinguish between the empty and occupied parts of the scene.

Two representives of generative models for SSC are 3D-GAN [15] and ASSC [16].
3D-GAN [15] uses volumetric convolutional networks to generate 3D objects from
a probabilistic space. ASSC [16] applies auto-encoder to encode the latent
context of the single-view depth and uses a 3D generator to rebuild the 3D
complete scene.

To enrich the input information and boost the accuracy of SSC, TS3D [12] and
DDR-SSC [17] proposed to add a RGB branch in addition to the voxel branch,
which introduce extra network or parameters, and are less accurate than our
method.

Different from the previous methods, our proposed Position Aware Loss Network
(PALNet) takes advantages of both the fine grained depth and the TSDF encoded
3D volume. Specifically, we formulate the depth detail as a vital ingredient
in the proposed network and make full use of the high-resolution depth by a 2D
CNN. In the 2D CNN, a differentiable projection layer is employed to
accurately project features in 2D space to the corresponding locations in 3D
volume. Moreover, the encoded TSDF provides the prior geometric knowledge of
scene which contributes to a much accurate model for SSC.

Figure 2: Local Geometric Anisotropy: voxels at different positions contain
different local geometric information. (a)-(d) respectively indicate voxels
inside the object, voxels on the surface, voxels on the edge, and a voxel on
the vertex. For a strip-shaped object, (e),(f) indicate voxels on the edge and
a voxel on the vertex. (g) indicates an isolated voxel.  Figure 3:  The
percentage of voxels with different LGA values in dataset NYU [18]. More than
84% of the voxels have $LGA=0$, indicating that most of the voxels are inside
the objects, and less than 16% of the voxels are outside of the objects.
Voxels in free space are not taken into account.

###  II-B Loss Function for 3D Dense Prediction

Compared to 2D image segmentation, 3D dense prediction has three
characteristics. Firstly, the number of voxels in 3D dense prediction is much
larger than the amount of pixels in the 2D image segmentation. Secondly, the
number of voxels ranges a lot among objects with different sizes. Finally, the
number of voxels outside the objects is far beyond that of inside the objects.
And we further observed that in 3D semantic scene completion, voxels at
different positions make different contributions as well as deliver various
training difficulties to the scene understanding as mentioned before. Based on
the above observations, it is thus vital to choose a suitable loss function to
train the 3D network effectively with the consideration of voxel-wise data-
balance. There are many classic loss functions available to train 3D networks.

####  II-B1 Cross-entropy Loss

Extended from 2D vision tasks, cross-entropy loss can be used in 3D dense
prediction. In essence, it treats all the predicted targets equally and is
proved less efficient in 3D tasks [19, 3].

####  II-B2 Weighted Cross-entropy Loss

Weighting factor $w_{c}\in[0,1]$ is introduced based on cross-entropy loss to
handle the imbalance problem. Weighted cross-entropy loss can emphasize the
importance of classes with rare samples, while it relies on manually set
weight parameters. A compromise approach instead of manual selection of
weights is to set $w_{c}$ as the inverse frequency for the corresponding
class. And it can only handle category-level data imbalance, but not voxel-
wise imbalance.

####  II-B3 Focal Loss

Focal loss aims to address the data imbalance in object detection especially
when the dataset contains too many easy negatives that contribute no
meaningful learning signals. However, one limitation of using focal loss is
that it underestimates the importance of well classified samples [20, 21].
Also, the training process becomes sensitive to incorrectly labeled samples
[22].

####  II-B4 Dice Loss

Dice loss [19] is proposed to address the data imbalance in volumetric medical
image segmentation. It is a good choice to address the imbalance between the
foreground and background in binary segmentation, but does not generalize well
to multi-category segmentation. Besides, it is not as easy to optimize as
cross-entropy loss, as its gradient may blow up to some enormous value when
both the value of the target label and the prediction are small.

In summary, none of those loss functions can take into account the importance
of different positions. In this paper, we propose Local Geometric Anisotropy
(LGA) to determine the importance of geometric information contained in
different voxels, and LGA is then used to calculate the weight factor for
cross-entropy loss to form the proposed PA-Loss, which fully considers the
impact of each element, leads to the better performance compared to other loss
functions.

Figure 4: The hybrid network structure of PALNet for semantic scene
completion. In depth-stream, 2D CNN is used to process the full resolution
depth, the feature maps are projected to 3D space with the 2D-3D projection
layer, and followed by 3D convolutions. In voxel-stream, TSDF encoded voxels
are sent as input, all convolution operations employed are 3D convolutions.
After the two streams of information are aggregated, they are sent to the
subsequent network with large receptive field to capture 3D context to predict
the complete semantic scene. We represent the 3D operation in the form of a
cuboid in this figure. Res($c,k,d,s$) is the dilated residual blocks with
bottleneck. $c$ is the output channel, $k$ is the kernel size, $d$ is the
dilation rate, and $s$ is stride in the convolution. Conv($c,k,d,s$) is the 3D
convolutional layer.

##  III Methodology

We propose a framework for semantic scene completion in which: (1) Voxel-wise
data imbalance is handled by the proposed PA-Loss. (2) Both depth details as
well as geometry prior can be fully utilized by the novel PALNet as
illustrated in Fig. 4. The details of the PA-Loss and PALNet are presented in
the following subsections.

###  III-A Position Importance Aware Constraint

####  III-A1  Local Geometric Anisotropy (LGA)

The geometric information contained in voxels at different positions has high
variability. In particular, the voxels inside the same object are more
homogeneous and likely belong to the same semantic category as their
surrounding voxels. Meanwhile, the voxels at the surfaces, edges, and vertices
have richer geometric information (have different semantic labels with their
surroundings) than those that are inside the objects. We refer to the semantic
difference between the current voxel and its surrounding neighbors as Local
Geometric Anisotropy.

To measure the LGA of a voxel with a specific semantic label, we focus on
voxels that belong to the current semantic category and treat all other voxels
with different semantic labels as another category. Specifically, the voxels
which are sharing the same category with the current one are denoted
‘occupied’ voxels and the voxels that belong to other categories are denoted
‘unoccupied’ voxels, as shown in Fig 2. Given a voxel $p$, its LGA is
calculated based on the 6-neighbor voxels, which is expressed as Eq 1:

$M_{LGA}=\sum_{i=1}^{K}{\left({c}_{p}\oplus{c}_{q_{i}}\right)}$ (1)

where $c_{p}$ is the semantic label of current voxel $p$, ${q}_{i}$ is one of
its $K$ neighbours and $i\in\\{1,2,\cdots,K\\}$, ${c}_{q_{i}}$ is the semantic
label of ${q}_{i}$. And $\oplus$ is the exclusive or (XOR) operation. If voxel
$p$ and its neighbour ${q}_{i}$ have the same semantic label, then
${c}_{p}\oplus{c}_{q_{i}}=0$, otherwise ${c}_{p}\oplus{c}_{q_{i}}=1$.

As can be seen in Fig 2, voxels that differ significantly from surroundings
get a higher $M_{LGA}$ value than voxels consistent with surrounding voxels.
For instance, in Fig 2(a), voxels which are inside an object are consistent
with the surrounding voxels and have LGA value equal to $0$. In Fig 2(b),
voxels which are at the surface of an object are consistent with the interior
voxels but different from other voxels regarding to the semantic categories
and have LGA value equal to $1$. We calculate the LGA value of all the voxels
excluding those within the empty space.

The voxels inside the object account for a very high proportion of the overall
voxels. However, the dominant homogeneous voxels with consistent semantic
properties are easier to be trained and be classified correctly at a very
early stage, and contribute little to the gradient update for the whole
training process, which leads to slow convergence of the network. A histogram
of LGA value is shown in Fig 3, where 84.4% of the voxels with LGA equal to
$0$ are in the interior of the object. All the remaining voxels with LGA
values equal to $1$ to $6$ add up to only 15.6%.

####  III-A2 PA-Loss

LGA is used to measure the spatial importance of different voxels and the
position importance aware factor to construct the PA-Loss. Through this way,
the proposed loss function has a strong response to the voxels with rich
detail.

Based on LGA values, we set a base value $\lambda$ and a constant $\alpha$ to
control LGA importance factor $I$ which is denoted as follows:

${I}=\lambda+\alpha{M}_{LGA}$ (2)

PA-Loss is defined as follows:

${L}_{PA}=-\frac{1}{N}\sum_{n=1}^{N}{\sum_{c=1}^{C}{{I}_{n}{y}_{nc}\log{{\hat{y}}_{nc}}}}$
(3)

where $N$ is the number of total voxels used to calculate the loss, and $C$ is
the number of classes. $\mathbf{{y}_{nc}}$, $\mathbf{{\hat{y}}_{nc}}$ are the
one-hot vector of the ground truth labels and the corresponding predictions in
class $c$, ${I}_{n}$ is the LGA importance factor of the voxel $n$.

The PA-Loss is differentiable, since the gradient of the PA-Loss can be easily
computed from Eq. 1, 2 and Eq. 3 which are all differentiable.

###  III-B 2D and 3D Hybrid Network Architecture

The proposed PALNet is composed of four parts as illustrated in Fig 4. The
depth stream (III-B1) takes a full resolution depth as input with a 2D-3D
projection layer (III-B2) to transfer the feature tensors in 2D space into 3D
space. The voxel stream (III-B3) employs 3D CNN and takes the 3D grid as
input. Then, both features extracted from the depth stream and voxel stream
are fed into the multi-level feature aggregation module (III-B4). After that,
the reconstruction part (III-B5) predicts the dense volume to perform the
semantic scene completion.

TABLE I: Results of various methods on NYU dataset [18]. Bold numbers represent the best score. | scene completion | semantic scene completion
---|---|---
Method | prec. | recall | IoU | ceil. | floor | wall | win. | chair | bed | sofa | table | tvs | furn. | objs. | mIoU
Lin et al. [23] | 58.5 | 49.9 | 36.4 | 0.0 | 11.7 | 13.3 | 14.1 | 9.4 | 29.0 | 24.0 | 6.0 | 7.0 | 16.2 | 1.1 | 12.0
Geiger et al. [24] | 65.7 | 58.0 | 44.4 | 10.2 | 62.5 | 19.1 | 5.8 | 8.5 | 40.6 | 27.7 | 7.0 | 6.0 | 22.6 | 5.9 | 19.6
SSCnet [3] | 59.3 | 92.9 | 56.6 | 15.1 | 94.6 | 24.7 | 10.8 | 17.3 | 53.2 | 45.9 | 15.9 | 13.9 | 31.1 | 12.6 | 30.5
TS3D [12] | 64.9 | 88.8 | 60.2 | 8.2 | 94.1 | 26.4 | 19.2 | 17.2 | 55.5 | 48.4 | 16.4 | 22.0 | 34.0 | 17.1 | 32.6
EsscNet [13] | 71.9 | 71.9 | 56.2 | 17.5 | 75.4 | 25.8 | 6.7 | 15.3 | 53.8 | 42.4 | 11.2 | 0 | 33.4 | 11.8 | 26.7
DDR-SSC [17] | 71.5 | 80.8 | 61.0 | 21.1 | 92.2 | 33.5 | 6.8 | 14.8 | 48.3 | 42.3 | 13.2 | 13.9 | 35.3 | 13.2 | 30.4
VVNet [8] | 69.8 | 83.1 | 61.1 | 19.3 | 94.8 | 28.0 | 12.2 | 19.6 | 57.0 | 50.5 | 17.6 | 11.9 | 35.6 | 15.3 | 32.9
PALNet(ours) | 68.7 | 85.0 | 61.3 | 23.5 | 92.0 | 33.0 | 11.6 | 20.1 | 53.9 | 48.1 | 16.2 | 24.2 | 37.8 | 14.7 | 34.1
TABLE II:  Results of various methods on NYUCAD dataset [25]. Bold numbers
represent the best score.

| scene completion | semantic scene completion
---|---|---
Method | prec. | recall | IoU | ceil. | floor | wall | win. | chair | bed | sofa | table | tvs | furn. | objs. | mIoU
Zheng et al. [26] | 60.1 | 46.7 | 34.6 | - | - | - | - | - | - | - | - | - | - | - | -
Firman et al. [25] | 66.5 | 69.7 | 50.8 | - | - | - | - | - | - | - | - | - | - | - | -
SSCnet [3] | 75.4 | 96.3 | 73.2 | 32.5 | 92.6 | 49.2 | 8.9 | 33.9 | 57.0 | 59.5 | 28.3 | 8.1 | 44.8 | 25.1 | 40.0
TS3D [12] | 80.2 | 94.4 | 76.5 | 34.4 | 93.6 | 47.7 | 31.8 | 32.2 | 65.2 | 54.2 | 30.7 | 32.5 | 50.1 | 30.7 | 45.7
DDR-SSC [17] | 88.7 | 88.5 | 79.4 | 54.1 | 91.5 | 56.4 | 14.9 | 37.0 | 55.7 | 51.0 | 28.8 | 9.2 | 44.1 | 27.8 | 42.8
VVNet [8] | 86.4 | 92.0 | 80.3 | - | - | - | - | - | - | - | - | - | - | - | -
PALNet | 87.2 | 91.7 | 80.8 | 54.8 | 92.8 | 60.3 | 15.3 | 43.1 | 60.7 | 59.9 | 37.6 | 8.1 | 48.6 | 31.7 | 46.6

####  III-B1 Depth-Stream

As depicted in the upper left of Fig. 4, the depth-stream contains both 2D CNN
and 3D CNN modules. The full resolution depth is first passed a 2D CNN module
to extract the 2D features. The 2D CNN module consists two 2D dilated residual
blocks with bottle-neck structure (Resblock in Fig. 4). The 3D CNN module
consists of a 3D convolution layer, two 3D Resblocks and one 3D pooling layer.
Through the 2D-3D projection layer, 2D features extracted from depth can be
mapped into corresponding 3D space, which helps the network to preserve
detailed information. Within the depth stream, each convolutional layer
employs the Residual blocks [27]. We adopt the dilated convolution [28] to
increase the receptive field of the network. We also take the advantage of its
bottleneck version to increase the capacity of the network and reduce its
parameters. Compared to the pure 3D CNN based approaches, a part of our hybrid
network uses 2D CNN, which introduces a small number of parameters but can
effectively improve the network depth and capacity.

####  III-B2 2D-3D Projection

Each pixel in depth can be projected to a voxel in 3D space. The mapping index
${\mathcal{T}}_{u,v}$ of a pixel in depth at $(u,v)$ can be computed using the
depth value $I_{u,v}$ and the camera parameters which are provided along with
each image. The 2D feature map can be mapped to the 3D space according to the
correspondence between the depth and 3D voxels.

####  III-B3 Voxel-Stream

In the voxel-stream, flipped-TSDF (f-TSDF) [3, 12] is adopted as the network
input. As depicted in the bottom left of Fig. 4, the 3D CNN module contains a
3D convolution, two 3D Resblocks, and one 3D pooling layer. Meanwhile, we
voxelize the full 3D scene with object labels as the ground truth and the rest
voxels in the scene are labeled as empty space resulting in a fully labeled
voxel grid representation of the scene.

####  III-B4 Multi-level Feature Aggregation Module

The multi-level feature aggregation module combines the two stream features
through element-wise addition. Then, the combined features are handled by a
series of 3D dilated residual blocks to increase the receptive field and
different dilation rates are employed within each block to reduce the gridding
problem [29].

####  III-B5 Dense Grid Reconstruction

Finally, in the dense grid reconstruction, there are three standard 3D
convolutional layers piled up to give the outputs. PALNet predicts the
probability distribution of voxel occupancy and object categories for all
voxels inside the camera view frustum.

Figure 5: For a scene shown in (a), Semantic Scene Completion (SSC) takes (b)
depth image as input and output dense 3D semantic volume. (c) gives the ground
truth. (d) and (e) are the predictions of the proposed PALNet and SSCNet [3].

##  IV Experimental results

###  IV-A Datasets

We evaluate the proposed method on the NYU [18] and NYUCAD [25] datasets. The
NYU dataset includes 1449 depth maps (795 for training, 654 for testing)
captured by the Kinect depth sensor. The ground truth for completion and
semantic segmentation are obtained from [30] by voxelizing the 3D mesh
annotations. In some cases, the manually labeled volumes and their
corresponding depth are not well aligned in the NYU dataset. To address the
misalignment, Firman et al. [25] provide the NYUCAD dataset, in which the
depth is rendered from the labeled volume.

###  IV-B Implementation Details

We implement the PALNet with PyTorch and train it from scratch. The dimensions
of the 3D space $\left(H\times D\times V\right)$ are $4.8\times 2.88\times
4.8$ m. We encode the 3D scene into a flipped TSDF (f-TSDF) with grid size of
$0.02$ m and a truncation value of $0.24$ m resulting in a volume of
resolution $240\times 144\times 240$ as the input of the 3D branch. For the
importance factor $I$, the base value $\lambda$, and the constant $\alpha$ are
set to $1.0$ and $0.5$ respectively.

The details of the network architecture are shown in Fig. 4. During training,
we minimize the PA-Loss using SGD with a momentum 0.9 and a weight decay
0.0001. The model is trained with 4 GTX 1080Ti GPUs for 40 epochs with batch
size 4. The initial learning rate is set to 0.01, and it is reduced by a
factor of 0.1 every 10 epochs.

###  IV-C Evaluation Metric

For semantic scene completion (SSC), we measure the intersection over union
(IoU) between the ground truth and the predicted volume, excluding voxels
outside the view or the room. The IoU score is calculated for each category
and then it is averaged over all the classes to obtain mean IoU (mIoU) score.
For the scene completion (SC) task, we treat all non-empty object classes as
one category and evaluate precision, recall, and IoU of the binary predictions
on occupied voxels.

###  IV-D Comparisons with the State-of-the-arts

We set the new state-of-the-art performance for both scene completion (SC) and
semantic scene completion (SSC) on both NYU [18] and NYUCAD [25] datasets.

We compare PALNet with previous state-of-the-art methods as represented in
Table I on NYU dataset and PALNet achieves the best performance in both SC and
SSC tasks. SSCNet and VVNet are pre-trained on the synthetic dataset SUNCG
[3]. Compared to SSCNet, the IoUs of PALNet for SC and SSC tasks increased by
4.7% and 3.6%, respectively. Compared to VVNet [8], PALNet leads the SC task
by 0.2% and the SSC task by 1.2%. DDR-SSC [17] and TS3D [12] use additional
color information. Our method achieves 3.7% and 1.5% higher IoU in SSC than
these methods, respectively. Table I also lists the IoU for each object class,
and our method is relatively more accurate than most of the state-of-the-art
methods in each category.

We also compare our method with Zheng et al. [26] and Firman et al. [25] on
the NYUCAD dataset. The results are shown in Table II, and our PALNet achieves
the best performance among all the methods as well.

###  IV-E Qualitative Results

Fig. 5 illustrates the results of SSC task on NYUCAD dataset generated by
different methods. In Fig. 5, (a) is the color image, (b) is the input depth,
(c) is the ground truth, (d) and (e) are the results of PALNet and SSCNet [3].
As can be seen, PALNet gives more accurate predictions than SSCNet [3] such as
the floor and the chairs in the first row. Although the windows in the second
row are hard to distinguish, our method still achieves better semantic
segmentation results. The paintings in the third row also demonstrate the
superiority of our approach. In the fourth row, the ground-truth furniture
circled by the red dashed rectangle is very similar to the object in terms of
shape and neither SSCNet nor PALNet can correctly identify the furniture
beside the chairs.

TABLE III:  Results of variant PALNets and SSCNet [3].

| scene completion | semantic scene completion
---|---|---
Method | prec. | recall | IoU | ceil. | floor | wall | win. | chair | bed | sofa | table | tvs | furn. | objs. | avg.
SSCnet [3] | 75.4 | 96.3 | 73.2 | 32.5 | 92.6 | 49.2 | 8.9 | 33.9 | 57.0 | 59.5 | 28.3 | 8.1 | 44.8 | 25.1 | 40.0
PALNet-3D | 83.3 | 94.5 | 79.1 | 49.1 | 92.3 | 56.2 | 6.8 | 36.6 | 62.5 | 57.6 | 34.6 | 5.9 | 45.6 | 30.7 | 43.5
PALNet-2D | 83.5 | 93.4 | 78.8 | 52.2 | 92.0 | 56.6 | 20.1 | 40.2 | 61.1 | 58.8 | 33.9 | 17.1 | 45.3 | 30.0 | 46.1
PALNet-hybrid | 87.2 | 91.7 | 80.8 | 54.8 | 92.8 | 60.3 | 15.3 | 43.1 | 60.7 | 59.9 | 37.6 | 8.1 | 48.6 | 31.7 | 46.6

##  V Ablation study

We validate the proposed PALNet with a set of ablation tests on NYUCAD [25]
dataset. Specifically, we examine three things as listed below: 1)
effectiveness of PA-Loss; 2) hybrid structure i.e.combined 2D depth and 3D
grid.

Figure 6:  Training curves of network with different loss functions for scene
completion (left) and semantic scene completion (right).

###  V-A PA-Loss

In order to prove the effectiveness of the proposed Position Aware Loss(PA-
Loss), we design two sets of comparative experiments. Firstly, we train PALNet
with different loss functions (PA-Loss and other loss functions) to compare
their training results. Then, we apply PA-Loss to SSCNet to demonstrate that
PA-Loss can improve the accuracy of SSCNet. The details of these ablation
studies are presented as follows.

####  V-A1 Comparison with Different Loss Functions

The three comparison methods are weighted cross-entropy (WCE) loss, focal loss
and dice loss.

Weighted Cross-entropy Loss. The weighted cross-entropy loss is defined as
follows:

${L}_{WCE}=-\frac{1}{N}\sum_{n=1}^{N}{\sum_{c=1}^{C}{{w}_{c}{y}_{nc}\log{{\hat{y}}_{nc}}}}$
(4)

where $N$ is the number of total voxels, $C$ is the number of classes. Voxels
belong to each class have different weights ${w}_{c}$. Specifically, the ratio
of each category is counted including empty voxels on the training set and use
the reciprocal of the ratio is used as the weight of each category.
${\hat{y}}_{nc}$ and ${y}_{nc}$ denote the prediction and ground truth of
voxel $n$ belong to class $c$.

Focal Loss. Focal loss adds a factor $(1-p_{t})^{\gamma}$ to the standard
cross-entropy criterion, as follows:

${L}_{FL}=-\frac{1}{N}\sum_{n=1}^{N}{\sum_{c=1}^{C}{{\left(1-{\hat{y}}_{nc}\right)}^{\gamma}{y}_{nc}\log{{\hat{y}}_{nc}}}}$
(5)

In the experiment, we set $\gamma=2$ as it is suggested to reduce the relative
weights for easy classified examples.

Multi-class Dice Loss. We generalize the standard dice loss for multi-class
segmentation by applying binary version on each class iteratively. The multi-
class dice loss is defined as:

$L_{MDL}=\sum_{c=1}^{C}{1-\frac{2\sum_{n=1}^{N}y_{nc}\hat{y}_{nc}}{\sum_{n=1}^{N}y_{nc}^{2}+\sum_{n=1}^{N}\hat{y}_{nc}^{2}}}$
(6)

TABLE IV: Results of PALNet trained with various loss functions.  Loss function | Precision | Recall | IoU | mIoU
---|---|---|---|---
WCE Loss | 79.4 | 95.9 | 79.9 | 46.3
Dice Loss | 88.2 | 83.3 | 78.64 | 45.3
Focal Loss | 85.6 | 91.8 | 79.3 | 46.1
PA-Loss | 87.2 | 91.7 | 80.8 | 46.6

We use these three loss functions to train the PALNet from scratch and
evaluate the accuracy of the network after each epoch. As shown in Table. IV,
the network trained with PA-Loss provides more accurate results than that
trained with other loss functions on both SC and SSC tasks. In the task of
semantic scene completion, PALNet trained with PA-Loss achieves 0.3%, 1.3% and
0.5% higher mIoU than that trained with weighted cross-entropy loss, dice loss
and focal loss.

In addition, we find that training convergence speed of using PA-Loss is
slightly faster than that of other loss functions. We plot the testing
accuracy of each epoch as a curve, as shown in Fig. 6 for scene completion
(left) and semantic scene completion (right).

TABLE V: Results of SSCNet trained with PA-Loss and Weighted Cross-Entropy
Loss (WCE Loss). With SSCNet* denotes our implementation of SSCNet [3].

Methods | Loss function | Precision | Recall | IoU | mIoU
---|---|---|---|---|---
SSCNet | WCE Loss | 75.4 | 96.3 | 73.2 | 40.0
SSCNet* | WCE Loss | 76.5 | 95.7 | 74.8 | 42.1
SSCNet* | PA-Loss | 81.6 | 91.6 | 76.0 | 43.4

####  V-A2 Apply PA-Loss to SSCNet

To further verify the effectiveness and generalization ability of PA-Loss, we
apply PA-Loss to SSCNet [3] in SSC task. We reimplemented SSCNet in PyTorch
and denote our implementation as SSCNet* in the following discussions. We
train SSCNet* with WCE Loss and PA-Loss separately and evaluate the
performance of SSCNet* on NYUCAD dataset. The experimental results are shown
in Table. V. The results reported in SSCNet [3] are also listed for reference.
Our reimplemented SSCNet* gets comparable (slightly better) accuracy than
SSCNet [3]. Compared to SSCNet* trained with WCE Loss, the one trained with
our PA-Loss gets better accuracy on both SC and SSC tasks.

###  V-B Hybrid Structure Combined 2D Depth and 3D Voxels

We explore the contribution of the 2D and 3D parts of the hybrid structure. To
investigate the effect of each stream, we develop two variants of the PALNet
where each variant contains only one of the two input branches. Note that the
element-wise add operation is no longer needed in the single-stream network
and the feature maps are fed into the multi-level feature aggregation stage
immediately. Since the 3D branch is extended from SSCNet , therefore, to
conclusively exemplify our method we select SSCNet for comparison in this
section.

TABLE VI:  The number of parameters, inference speed and GPU memory usage of
SSCNet [3] and the proposed PALNet.

Method |  | Params
---
(k)
| FLOPs
---
(G)
| Speed
---
(FPS)
| Memory
---
(M)
| Network
---
Depth
| SSC
---
IoU(%)
SSCNet | 930.0 | 163.8 | 0.7 | 5305 | 14 | 40.0
PALNet | 223.0 | 78.8 | 1.2 | 3717 | 25 | 46.6

As shown in Table III, both variants with one stream as input achieve better
performance than SSCNet [3] on the SC and SSC tasks. The combination of 2D and
3D streams increases the accuracy of PALNet in SC to 80.8% and SSC to 46.6%.

The variant network with 2D depth as the input achieves 46.1% IoU on the SSC
task and is 2.6% higher than that of the other variant with 3D voxels as the
input. The significant advantages of the 2D network over 3D network strongly
suggest that the details in depth are very useful for semantic scene
completion. The 3D branch with manually designed f-TSDF gets better result
(0.3%) than the 2D branch on the task of scene completion. This is reasonable
due to the carefully designed f-TSDF encodes the geometry information well.
The symbolic information of f-TSDF can explicitly tell the network where the
occluded space is, and help the network to pay attention to these parts for
corresponding shape completion.

In Table VI, the parameters, Flops, inference speed, memory and accuracy of
the networks among SSCNet [3] and PALNet are listed out. By taking advantage
of the bottle-neck residual network, PALNet has much deeper structure than
SSCNet [3] and achieve better accuracy.

##  VI Conclusions

We introduce PALNet which takes both the depth and TSDF as inputs for semantic
scene completion. The feature from 2D-stream are projected and concatenated
with the feature in 3D-stream and trained in an end-to-end manner. We also
propose a position importance aware loss, PA-Loss, which leads to slightly
faster training convergence and better performance. The experiments on both
synthetic and real datasets validate the superior performance of the proposed
method. The two interesting topics that are worth exploring in our future work
include: (1) a better trade-off between 2D CNNs and 3D CNNs to employ less 3D
convolutional layers without sacrificing the performance, and (2) to extend
our PALNet framework with RGB-D as input.

## Acknowledgement

We gratefully acknowledge the support of the Australian Research Council
through the Centre of Excellence for Robotic Vision CE140100016 and Laureate
Fellowship FL130100102 to IR. This work is also supported by the National
Science Foundation of China under Grants 61773210 and 61603184.

## References

  * [1] W. Choi, Y.-W. Chao, C. Pantofaru, and S. Savarese, “Indoor scene understanding with geometric and semantic contexts,” _Int. J. Computer Vision_ , vol. 112, no. 2, pp. 204–220, 2015.
  * [2] S. Gupta, P. Arbeláez, R. Girshick, and J. Malik, “Indoor scene understanding with rgb-d images: Bottom-up segmentation, object detection and semantic segmentation,” _Int. J. Computer Vision_ , vol. 112, no. 2, pp. 133–149, Apr 2015.
  * [3] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “Semantic scene completion from a single depth image,” in _Proc. IEEE Conf. CVPR_ , 2017, pp. 190–198.
  * [4] L. Chen, K. Francis, and W. Tang, “Semantic augmented reality environment with material-aware physical interactions,” in _Proc. IEEE ISMAR_. IEEE, 2017, pp. 135–136.
  * [5] D. Van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” _Int. J. Virtual Reality_ , vol. 9, no. 2, p. 1, 2010.
  * [6] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen, “Shape completion enabled robotic grasping,” in _Int. Conf. IROS_ , 2017, pp. 2442–2447.
  * [7] V. Vineet, O. Miksik, M. Lidegaard, M. Nießner, S. Golodetz, V. A. Prisacariu, O. Kähler, D. W. Murray, S. Izadi, P. Pérez, _et al._ , “Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction,” in _Proc. ICRA_ , 2015, pp. 75–82.
  * [8] Y. Guo and X. Tong, “View-volume network for semantic scene completion from a single depth image,” in _Proc. IJCAI_ , 7 2018, pp. 726–732.
  * [9] L. Tchapmi, C. Choy, I. Armeni, J. Gwak, and S. Savarese, “Segcloud: Semantic segmentation of 3d point clouds,” in _Proc. IEEE Conf. 3D Vision_ , 2017, pp. 537–547.
  * [10] P. H. Lindsay and D. A. Norman, _Human information processing: An introduction to psychology_. Academic press, 2013.
  * [11] K. Desingh, K. M. Krishna, D. Rajan, and C. Jawahar, “Depth really matters: Improving visual salient region detection with depth.” in _Proc. BMVC_ , 2013, pp. 98.1–98.11.
  * [12] M. Garbade, Y.-T. Chen, J. Sawatzky, and J. Gall, “Two stream 3d semantic scene completion,” in _Proc. IEEE Conf. CVPR Workshops_ , 2019, pp. 0–0.
  * [13] J. Zhang, H. Zhao, A. YaoE, Y. Chen, L. Zhang, and H. LiaoE, “Efficient semantic scene completion network with spatial group convolution,” in _Proc. ECCV_ , 2018, pp. 733–749.
  * [14] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, _et al._ , “Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera,” in _Proc. ACM Symp. UIST_. ACM, 2011, pp. 559–568.
  * [15] J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum, “Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling,” in _Advances in NeurIPS_ , 2016, pp. 82–90.
  * [16] Y. Wang, D. J. Tan, N. Navab, and F. Tombari, “Adversarial semantic scene completion from a single depth image,” in _Proc. IEEE Conf. 3D Vision_. IEEE, 2018, pp. 426–434.
  * [17] J. Li, Y. Liu, D. Gong, Q. Shi, X. Yuan, C. Zhao, and I. Reid, “Rgbd based dimensional decomposition residual network for 3d semantic scene completion,” in _Proc. IEEE Conf. CVPR_ , 2019, pp. 7693–7702.
  * [18] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _Proc. ECCV_ , 2012, pp. 746–760.
  * [19] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _Proc. IEEE Conf. 3D Vision_. IEEE, 2016, pp. 565–571.
  * [20] T. Nguyen, T. Ozaslan, I. D. Miller, J. Keller, G. Loianno, C. J. Taylor, D. D. Lee, V. Kumar, J. H. Harwood, and J. Wozencraft, “U-net for mav-based penstock inspection: an investigation of focal loss in multi-class segmentation for corrosion identification,” _arXiv preprint arXiv:1809.06576_ , 2018.
  * [21] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [22] J. Guo, P. Ren, A. Gu, J. Xu, and W. Wu, “Locally adaptive learning loss for semantic image segmentation,” _arXiv preprint arXiv:1802.08290_ , 2018.
  * [23] D. Lin, S. Fidler, and R. Urtasun, “Holistic scene understanding for 3d object detection with rgbd cameras,” in _Proc. ICCV_ , 2013, pp. 1417–1424.
  * [24] A. Geiger and C. Wang, “Joint 3d object and layout inference from a single rgb-d image,” in _Proc. GCPR_ , 2015, pp. 183–195.
  * [25] M. Firman, O. Mac Aodha, S. Julier, and G. J. Brostow, “Structured prediction of unobserved voxels from a single depth image,” in _Proc. IEEE Conf. CVPR_ , 2016, pp. 5431–5440.
  * [26] B. Zheng, Y. Zhao, J. C. Yu, K. Ikeuchi, and S.-C. Zhu, “Beyond point clouds: Scene understanding by reasoning geometry and physics,” in _Proc. IEEE Conf. CVPR_ , 2013, pp. 3127–3134.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. CVPR_ , 2016, pp. 770–778.
  * [28] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” 2016.
  * [29] P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell, “Understanding convolution for semantic segmentation,” in _Proc. IEEE WACV_ , 2018, pp. 1451–1460.
  * [30] R. Guo, C. Zou, and D. Hoiem, “Predicting complete 3d models of indoor scenes,” _arXiv preprint arXiv:1504.02437_ , 2015.