11affiliationtext: Hunter College, CUNY
New York City, USA22affiliationtext: The Graduate Center, CUNY
New York City, USA affiliationtext:
<EMAIL_ADDRESS>affiliationtext:
<EMAIL_ADDRESS>
# You Only Need One Detector: Unified Object Detector for Different Modalities
based on Vision Transformers

Xiaoke Shen  Zhujun Li  Jaime Canizales  Ioannis Stamos

###### Abstract

Most systems use different models for different modalities, such as one model
for processing RGB images and one for depth images. Meanwhile, some recent
works discovered that an identical model for one modality can be used for
another modality with the help of cross modality transfer learning. In this
article, we further find out that by using a vision transformer together with
cross/inter modality transfer learning, a unified detector can achieve better
performances when using different modalities as inputs. The unified model is
useful as we don’t need to maintain separate models or weights for robotics,
hence, it is more efficient. One application scenario of our unified system
for robotics can be: without any model architecture and model weights
updating, robotics can switch smoothly on using RGB camera or both RGB and
Depth Sensor during the day time and Depth sensor during the night time .
Experiments on SUN RGB-D dataset show: Our unified model is not only
efficient, but also has a similar or better performance in terms of mAP50
based on SUNRGBD16 category: compare with the RGB only one, ours is slightly
worse (52.3 $\to$ 51.9). compare with the point cloud only one, we have
similar performance (52.7 $\to$ 52.8); When using the novel inter modality
mixing method proposed in this work, our model can achieve a significantly
better performance with 3.1 (52.7 $\to$ 55.8) absolute improvement comparing
with the previous best result. Code (including training/inference logs and
model checkpoints) is available: https://github.com/liketheflower/YONOD.git

##  1 Introduction

With the development of computer vision and artificial intelligence, more and
more robotics are showing up to enrich people’s lives: autonomous vehicles can
drive us to our target location without any human driver/operator; Autonomous
Mobile Robots working in warehouses can help us prepare orders. Although most
of those robotics require more than one sensor such as cameras and 3D sensors
(LiDAR or depth), not all the scenarios all of those sensors are available.
For example, the camera sensor can not work well during the night time if no
extra lighting is provided or in some cases when people’s privacy are being
protected and hence that 2D cameras are not allowed to be deployed on
robotics[1]. Meanwhile, Environmentally Friendly Robots are important to our
planet as people and robotics are not the only residents. Reducing the light
pollution used for robotics during the night time can be something meaningful
to our planet. Hence, developing robotic vision systems which only use the
natural light during the day time and no extra light during the night time is
always one of our research goals.

Figure 1: Model A: only process RGB images. The visualization is generated
from the model trained only based on RGB image from this work; Model B: only
process pseudo images converted from point clouds. The visualization is
generated from the simCrossTrans[1] work which trains on the pseudo images
converted from point cloud. Model C can process any of the following images:
RGB images, pseudo images converted from point clouds or inter-modality mixing
of RGB image and pseudo images converted from point clouds. This visualization
is based on YONOD with Swin-T [2] as the backbone network.

Thanks to the powerful ConvNets[3] based and Transformer[4] based feature
extractors, the camera based vision system can achieve excellent 2D detection
performance. At the same time, the simCrossTrans[1] work shows by converting
the 3D sensor data to pseudo images, together with cross-modality transfer

Figure 2: 2D detection results based on the SUN RGB-D validation data split.
It has the 2D detection visualization of four examples based on RGB image
(left), pseudo images converted from point clouds (middle) and inter-modality
mixing of RGB image and pseudo images converted from point clouds (right).
This is based on the Swin-S backbone network. More results can be found in
https://youtu.be/PuRQCLLSUDI

learning, the 3D data based 2D object detection system’s can achieve good
performance by using an identical network used for the RGB images. This is
promising, and a very natural question will be: can we do better if we train a
unified network by using both the RGB and 3D data with a unified network (with
the same architecture and same weights )? The unified network can process any
of the following sensor data: 1) RGB images; 2) pseudo images converted from
3D sensor; 3) both the RGB images and pseudo images converted from 3D sensor.
If we can achieve similar or even better performance compared with separate
networks with each network only performing well for a specified modality, then
the environment friendly system (using natural light during the day time and
no extra light during the night time) mentioned in the previous paragraph will
be possible. In this article, we are mainly exploring the possibilities of
this system. As in the simCrossTrans[1] work, the Vision Transformer based
performs better than the ConvNets based network, in this article, we focus on
using the vision transformer network only. In summary, in this article, we
want to find answers for the following questions:

  1. 1.

Whether a unified model can have a similar performance or even better
performance on both RGB images and pseudo images converted from point cloud?

  2. 2.

If a unified model of processing both the RGB and pseudo images converted from
point cloud is feasible, whether we can further fuse the RGB images and pseudo
images? By doing this we can enhance the unified model further to support
processing both the RGB and point cloud data.

Through experiments, we indeed have interesting observations and achieve SOTA
results on 2D object detection. Our unified model named You Only Need One
Detector (YONOD) can process all of the following images: RGB images, pseudo
images converted from point clouds or inter modality mixing of RGB image and
pseudo images converted from point clouds. The difference of our model
compared with other works can be found in Figure 1. Comparison of the
performance for different methods can be found in Table 3. YONOD outputs are
visualized in Figure 2.

Here are our contributions:

  1. 1.

By using vision transformers, we achieve a new SOTA on 2D object detection
based on RGB images on SUN RGB-D dataset.

  2. 2.

We propose two inter-modality mixing methods which can mixup the data from
different modalities to further feed to our unified model.

  3. 3.

We propose a unified model which can process any of the following images: RGB
images, pseudo images converted from point clouds or inter-modality mixing of
RGB image and pseudo images converted from point clouds. This unified model
achieves similar performance to RGB only model and point cloud only model.
Meanwhile, by using the inter-modality mixing data as input, our model can
achieve a significantly better 2D detection performance.

  4. 4.

We open source our code, training/testing logs and model checkpoints to
benefit the vision community.

##  2 Related Work

Projecting 3D sensor data to 2D Pseudo Images:  There are different ways to
project 3D data to 2D features. HHA was proposed in [5] where the depth image
is encoded with three channels: Horizontal disparity, Height above ground, and
the Angle of each pixel’s local surface normal with gravity direction. The
signed angle feature described in[6] measures the elevation of the vector
formed by two consecutive points and indicates the convexity or concavity of
three consecutive points. Input features converted from depth images of
normalized depth(D), normalized relative height(H), angle with up-axis(A),
signed angle(S), and missing mask(M) were used in [7]. DHS images are used in
[8, 9].
Object Detection Based on RGB images or Pseudo images from point cloud by
Vision Transformers:  Object detection approaches can be summarized as two-
stage frameworks (proposal and detection stages) and one-stage frameworks
(proposal and detection in parallel). Generally speaking, two-stage methods
such as R-CNN [10], Fast RCNN [11], Faster RCNN [12], FPN [13] and mask R-CNN
[14] can achieve a better detection performance while one-stage systems such
as YOLO[15], YOLO9000[16], RetinaNet [17] are faster at the cost of reduced
accuracy. For deep learning based systems, as the size of the network is
increased, larger datasets are required. Labeled datasets such as PASCAL VOC
dataset [18] and COCO (Common Objects in Context) [19] have played important
roles in the continuous improvement of 2D detection systems. Most systems
introduced here are based on ConvNets. Nice reviews of 2D detection systems
can be found in [20]. When replacing the backbone network from ConvNets to
Vision Transformers, the systems will be adopted to Vision Transformers
backbone based object detection systems. The most successful systems are Swin-
transformer[2] and Swin-transformer v2 [21]. simCrossTrans[1] explored the
cross modality transfer learning by using both the ConvNets and Vision
Transfomers based on SUN RGB-D dataset based on the mask R-CNN [14] approach.
Inter modality mixing:  [22] learns a dynamical and local linear interpolation
between the different regions of cross-modality images in data-dependent
fashion to mix up the RGB and infrared (IR) images. We explored both the
static and dynamic mixing methods and found the static has a better
performance. [23] uses an interpolation between the RGB and thermal images at
pixel level. As we are training a unified model supporting both the single
modality image and multiple modality images as input, we do not apply the
interpolation to keep the original signal of each modality. We leverage the
transformer architecture itself to automatically build up the gap between
different modalities.
Transfer learning with same modality or cross modality: Transfer learning is
widely used in computer vision (CV), natural language processing (NLP) and
biochemistry. Most transfer learning systems are based on the same modality
(e.g. RGB image in CV and text in NLP). For the CV, common transfer learning
is based on supervised way such as works in R-CNN [10], Fast RCNN [11], Faster
RCNN [12], FPN [13], mask R-CNN [14], YOLO[15], YOLO9000[16], RetinaNet [17]
use a pretrained backbone network model based on ImageNet classification task
and the model is further trained based on the following task datasets such as
COCO to achieve object detection or/and instance segmentation tasks. In the
NLP, the transfer learning such as BERT [24], GPT[25], GPT-2[26], GPT-3[27]
are mainly based on self-supervised way and achieve great success. Inspired by
the success of the self-supervised way transfer learning, the CV community is
also exploring the self-supervised way to explore new possibilities, one
recent work which is similar to the BERT in NLP is MAE [28]. The MolGNN [29,
30] in bioinformatics use a self-supervised way based on Graph Neural network
(GNN) in the pretraining stage and achieve good performance in a few shot
learning framework for the following subtasks. For this work, we explore the
cross modality transfer learning from a pretrained model under the supervised
learning approach. Recently, Frustum-Voxnet [8] used pretrained weights from
the RGB images to fine tune the point cloud converted pseudo image based on
ConvNets[3]. simCrossTrans[1] further explored the cross modality transfer
learning by using both the ConvNets[3] and Vision Transfomers[31, 2] and
showed significant improvement.

##  3 Approach

###  3.1 Convert point clouds to pseudo 2D image

Figure 3: An example of converted pseudo three channel image from the point
cloud. This example is one image selected from the validation set of the SUN
RGB-D dataset. The corresponding RGB one can be found in Figure 1

In order to use pretrained models based on RGB images, we convert point clouds
to pseudo 2D images with 3 channels. The point clouds can be converted to HHA
or any three channels from DHASM introduced in [32].

For this work, we follow the same approaches in Frustum VoxNet[8] and
simCrossTrans [1] by using DHS to project 3D depth image to 2D due to: 1) [32]
shows DHS can provide a solid result; 2) have a fair comparison with the
Frustum VoxNet[8] and simCrossTrans [1] works.

Here is a summary of the DHS encoding method[1]: Similar to[5, 32], we adopt
Depth from the sensor and Height along the sensor-up (vertical) direction as
two reliable measures. Signed angle was introduced in [33]: denote
$X_{i,k}=[x_{ik},y_{ik},z_{ik}]$ the vector of 3D coordinates of the $k$-th
point in the $i$-th scanline. Knowledge of the vertical direction (axis
$\mathbf{z}$) is provided by many laser scanners, or even can be computed from
the data in indoor or outdoor scenarios (based on line/plane detection or
segmentation results from machine learning models) and is thus assumed known.
Define $D_{i,k}=X_{i,k+1}-X_{i,k}$ (difference of two successive measurements
in a given scanline $i$), and $A_{ik}$: the angle of the vector ${D_{i,k}}$
with the pre-determined $\mathbf{z}$ axis (0 to 180 degrees). The Signed angle
$S_{ik}=sgn(\mathbf{D_{i,k}}\cdot\mathbf{D_{i,k-1}})*A_{ik}$: the sign of the
dot product between the vectors $D_{i,k}$ and $D_{i,k-1}$, multiplied by
$V_{ik}$. This sign is positive when the two vectors have the same orientation
and negative otherwise. Those three channel pseudo images are normalized to 0
to 1 for each channel. Some samples DHS images can be seen in Figure 3 and 4.

###  3.2 Inter modality mixing

Figure 4: Inter modality mixing: left: the upper one is the original RGB image
and the bottom one is the original DHS image; Middle: the Chessboard Per Patch
Mixing image, the upper one has the patch size of 15 by 15 pixels and the
bottom one is 1 by 1 pixel. Right: the Stochastic Flood Fill Mixing image. The
upper one has the edge connection probability of 0.5, 0.5 for RGB and DHS
separately and the bottom one has the probability of 0.1 and 0.1 for RGB and
DHS separately.

In order to support using multiple modality inputs based on our unified model,
we propose an inter modality mixing approach. For the inter modality mixing,
we mix the images from different modality together to generate a mixed three
channel image which can be consumed by our model. By doing this, we can
further extend our model’s capability without updating the model’s
architecture. Meanwhile, by training a model based on RGB, DHS images and
inter modality Mixing image from RGB and DHS images, we can achieve a unified
detector which can support different modality as input.
Different methods can be designed to fuse the images from different
modalities, we proposed two ways:

  * •

Per Patch Mixing (PPM): divide the whole image into different patches with
equal patch size. Randomly or alternatively select one image source for each
patch.

  * •

Stochastic Flood Fill Mixing (SFFM): Using a stochastic way to mix the images
from different modalities.

The Per Patch Mixing is relatively simple to implement. In our experiment, we
alternatively select a modality image for each patch. Also, we chose square
patches. By doing this, the modality image selection mask looks like a
chessboard, so we call our implementation as Chessboard Per Patch Mixing
(CPPM). Examples of the CPPM are shown in the middle of Figure 4.

Figure 5: Python style of the Stochastic Flood Fill Mixing pseudocode: the
mixture$\\_$mask defines which image’s pixel value we should use to generate
the mixing image.

For Stochastic Flood Fill Mixing, it is adjusted from the flood fill algorithm
[34]. For this mixing, we generate a mask of using which modality image based
on the flood fill results. For the stochastic flood fill, the edge between two
neighbor pixels is connected with a probability $p$. The RGB and DHS have
separate edge connection probabilities. The Python style pseudocode of this
algorithm is shown in Figure 5.

We can use both four neighbors (top, right, bottom, left) to build the graph
or eight neighbors (adding four closest pixels along diagonal offsets besides
the four neighbors) to build the graph. In our experiment, we use four
neighbors. Examples of the SFFM are shown on the right of Figure 4.

###  3.3 2D detection framework

For 2D detection and instance segmentation, we use the classical object
detection framework: Mask R-CNN [14] in mmdetection [35]. It is a two-stage
detection and segmentation framework [20]: region proposal and
detection/segmentation. We disable the training of the mask branch when fine-
tuning the model on SUN RGB-D dataset. By using the default weights from the
pretrained model, the mask prediction branch can still generate reasonable
mask predictions as shown in Figure 1. This is consistent with the observation
from the simCrossTrans [1] work.

###  3.4 2D detection backbone networks

For the backbone network, we use Swin Transformers[2], specifically we
explored Swin-Tiny and Swin-Small’s performance. The complexity of Swin-T and
Swin-S are similar to those of ResNet-50 and ResNet-101, respectively. The
window size is set to $M=7$ by default. The query dimension of each head is
$d=32$, and the expansion layer of each MLP is $\alpha=4$. The architecture
hyper-parameters of these two models are:

  * •

Swin-T: C = 96, layer numbers = $\\{2,2,6,2\\}$

  * •

Swin-S: C = 96, layer numbers = $\\{2,2,18,2\\}$

where C is the channel number of the hidden layers in the first stage. Detail
of the model architecture, please check the Swin Transformers[2] paper.

###  3.5 SUN RGB-D dataset used in this work

SUN RGB-D[36] dataset is an indoor dataset which provides both the point cloud
and RGB images. In this work, since we are building a 3D only object detection
system, we only use the point clouds for fine tuning. The RGB images are not
used during the fine tuning process. For the point clouds, they are collected
based on 4 types of sensors: Intel RealSense, Asus Xtion, Kinect v1 and Kinect
v2. The first three sensors are using an IR light pattern. The Kinect v2 is
based on time-of-flight. The longest distance captured by the sensors are
around 3.5 to 4.5 meters.

SUN RGB-D dataset splits the data into a training set which contains 5285
images and a testing set which contains 5050 images. For the training set, it
further splits into a training only, which contains 2666 images and a
validation set, which contains 2619 images. Similar to [37, 38, 8, 9] , we are
fine-tuning our model based on the training only set and evaluate our system
based on the validation set.

###  3.6 Pre-train

Both the Swin-T and Swin-S based networks111The pretrained weights are loaded
from mmdetection [35]. are firstly pre-trained on ImageNet [39] and then pre-
trained on the COCO dataset [19].
Data augmentation When pre-training on COCO dataset, the images augmentation
are applied during the training stage by: randomly horizontally flip the image
with probability of 0.5; randomly resize the image with width of 1333 and
height of several values from 480 to 800 (details see the configure file from
the github repository); randomly crop the original image with size of 384
(height) by 600 (width) and resize the cropped image to width of 1333 and
height of several values from 480 to 800.

###  3.7 Fine-tuning

Data augmentation: We follow the same augmentation with the pre-train stage.
The raw input images have the width of 730 and height of 530. Those raw images
are randomly resized and cropped during the training. During testing, the
images are resized to width of 1120 and height of 800 which can be divided by
32.
Hardware: For the fine-tuning, we use a standard single NVIDIA Titan-X GPU,
which has 12 GB memory. We fine-tune the network for 133K iterations for 100
epochs. It took about 29 hours for Swin-T based network with batch size of 2
(for 133K iterations) for the RGB only model. For the YONOD without the inter
modality mixing, it took about 2 days to train the model. For with the inter
modality mixing, the speed depends on the number of inter modality mixing
images added to the training data.
Fine-tuning subtasks: We focus on the 2D object detection performance, so we
fine-tune the model based on the 2D detection related labels. Similar to
simCrossTrans [1], we kept the mask branch without training to further verify
whether reasonable mask detection can be created by using the weights from the
pre-train stage.

##  4 Results on SUN RGB-D dataset

###  4.1 Experiments

Our experiments mainly focus on using different inputs data to train the model
and compare the performance difference. For the YONOD without the inter
modality mixing, we use both the RGB and DHS images to train a unified model.
For the YONOD with the inter modality mixing, besides the RGB and DHS images,
we also add the inter modality mixing images to the training data.

###  4.2 Evaluation metrics

Following the previous works mentioned in Table 3, we firstly use the AP50:
Average Precision at IoU = 0.5 as evaluation metric. We also use the COCO
object detection metric which is AP75: Average Precision at IoU = 0.75 and a
more strict one: AP at IoU = .50:.05:.95 to evaluate the 2D detection
performance.

###  4.3 Evaluation subgroups

We use the same subgroups as simCrossTrans[1] to evaluate the performance. The
subgroups are SUNRGBD10, SUNRGBD16, SUNRGBD66 and SUNRGBD79, which have 10,
16, 66 and 79 categories. Detail list of those sub groups can be found in
simCrossTrans[1].

###  4.4 The performance of YONOD without the inter modality mixing

We first evaluate the performance of YONOD without inter modality mixing. For
this one, the model is trained based on both the RGB and DHS images. Our model
architecture is the same as the simCrossTrans [1] work, which is using only
the DHS image to train the model. We train a RGB images only model based on
the same network to compare with the YONOD one’s performance.

Figure 6: Comparison of the unified model and separate models performance with
the training epochs. The RGB only one is our new trained model based on RGB
images. The DHS only model is from simCrossTrans [1]. The YONOD is trained
based on both the RGB and DHS images. The backbone for all those experiments
are based on the Swin-T model.

The performance based on the mAP50 on the SUNRGBD79 can be found in Figure 6.
From the result, we can see the YONOD can achieve excellent performance when
testing on both RGB and DHS images. We can also observe that YONOD on DHS
images’ performance is significantly improved compared to the DHS only model.
The DHS images’ performance boosting should be benefited from the inter
modality transfer learning from the RGB images. The cost of the DHS
performance boosting is slightly performance reduce on the RGB images.
Overall, the YONOD’s performance is promising as only one model can work for
different modalities. It is more efficient than maintain two architectures or
one architectures with two different weights. This is especially meaningful
for robotics and edge devices. We can build a smoothly perception system when
change from day time to night time for robotics if our system is deployed.
More results can be found in Table 1 for our YONOD and single modality only
models. Similar conclusion can be observed from the result.

model | test on | backbone | sunrgbd10 | sunrgbd16 | sunrgbd66 | sunrgbd79
---|---|---|---|---|---|---
RGB only (ours) | RGB | Swin-T | 54.2 | 52.3 | 29.3 | 25.2
YONOD (ours) | RGB | Swin-T | 53.9 | 52.5 | 28.7 | 24.7
DHS only (simCrossTrans [1]) | DHS | Swin-T | 55.8 | 52.7 | 26.1 | 22.1
YONOD (ours) | DHS | Swin-T | 56.6 | 53.4 | 27.7 | 23.5

Table 1: Result comparison based on mAP50 for different subgroups of YONOD and
single modality only models.

###  4.5 The performance of YONOD with the inter modality mixing

model | test on | backbone | sunrgbd10 | sunrgbd16 | sunrgbd66 | sunrgbd79
---|---|---|---|---|---|---
YONOD | RGB | Swin-T | 53.9 | 52.5 | 28.7 | 24.7
YONOD + SFFM | RGB | Swin-T | 24.6 | 17.5 | 19.2 | 20.1
YONOD + CPPM | RGB | Swin-T | 54.2 | 51.9 | 27.7 | 23.7
YONOD + CPPM | RGB | Swin-S | 54.6 | 52.7 | 27.5 | 23.6
YONOD | DHS | Swin-T | 56.6 | 53.4 | 27.7 | 23.5
YONOD + SFFM | DHS | Swin-T | 25.6 | 18.7 | 20.0 | 21.3
YONOD + CPPM | DHS | Swin-T | 55.8 | 52.8 | 26.3 | 22.4
YONOD + CPPM | DHS | Swin-S | 57.4 | 52.5 | 24.8 | 21.1
YONOD + CPPM | CPPM | Swin-T | 58.1 | 55.8 | 29.5 | 25.2
YONOD + CPPM | CPPM | Swin-S | 58.4 | 56.1 | 28.4 | 24.5

Table 2: Result comparison based on mAP50 for different subgroups of YONOD and
single modality only models.

For the inter-modality mixing, we tried both the SFFM and CPPM. For the SFFM,
we have six SFFM images for each RGB and DHS image pair. The connection
probabilities for RGB and DHS are randomly selected from 0.1 to 0.9. The first
pixel’s RGB and DHS mask are randomly set up with equal probability. For CPPM,
we use patch size of 1 by 1 and each RGB and DHS image pair has one CPPM
image.

The result is shown in Table 2. From the result, we can see that the simple
YONOD with CPPM has a better performance than the YONOD with SFFM. The reason
should be that the SFFM generates too many random images which makes the
unified network fail to perform well on the RGB and DHS images. However, for
using the simple CPPM one, it has similar performance compared with the plain
YONOD model’s results. When using the CPPM image from RGB and DHS, the 2D
detection can achieve the best performance. As YONOD with CPPM can support
three types of images: RGB, DHS and CPPM images from RGB and DHS, we propose
to use the YONOD with CPPM as a more powerful unified model.

###  4.6 Influence of different backbone networks

Table 2 also shows the results of using Swin-T and Swin-S as the different
backbone networks. Swin-S is a more powerful network, however, the performance
gain is limited. Hence, we propose to use the lightweight Swin-T as the
backbone network to achieve a faster inference speed (see Table 4).

###  4.7 Detail per category results compared with other works

Image Source | Methods | Backbone | bed | toilet |  night stand | bathtub | chair | dresser | sofa | table | desk | bookshelf |  sofa chair |  kitchen counter |  kitchen cabinet |  garbage bin | microwave | sink |  SUNRGBD10 $mAP_{50}$ |  SUNRGBD16 $mAP_{50}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
RGB | 2D-driven[38] | VGG-16 | 74.5 | 86.2 | 49.5 | 45.5 | 53.0 | 29.4 | 49.0 | 42.3 | 22.3 | 45.7 | N/A | N/A | N/A | N/A | N/A | N/A | 49.7 | N/A
Frustum PointNets[40] | adjusted VGG from SSD | 56.7 | 43.5 | 37.2 | 81.3 | 64.1 | 33.3 | 57.4 | 49.9 | 77.8 | 67.2 | N/A | N/A | N/A | N/A | N/A | N/A | 56.8 | N/A
F-VoxNet[8] | ResNet 101 | 81.0 | 89.5 | 35.1 | 50.0 | 52.4 | 21.9 | 53.1 | 37.7 | 18.3 | 40.4 | 47.8 | 22.0 | 29.8 | 52.8 | 39.7 | 31.0 | 47.9 | N/A
rgb only model (ours) | Swin-T | 83.2 | 93.9 | 51.8 | 54.2 | 60.4 | 23.7 | 51.3 | 46.3 | 22.5 | 54.4 | 60.4 | 32.7 | 39.8 | 67.0 | 48.1 | 47.3 | 54.2 | 52.3
| YONOD (ours) | Swin-T | 83.6 | 87.1 | 53.3 | 58.8 | 62.5 | 22.6 | 54.2 | 46.8 | 22.0 | 48.0 | 63.7 | 28.9 | 38.4 | 67.1 | 57.4 | 46.2 | 53.9 | 52.5
| YONOD + CPPM (ours) | Swin-T | 83.6 | 88.6 | 53.0 | 59.1 | 60.8 | 26.5 | 50.7 | 46.1 | 22.0 | 52.0 | 60.6 | 31.2 | 37.8 | 64.7 | 54.3 | 40.1 | 54.2 | 51.9
Point Cloud only | F-VoxNet[8] | ResNet 101 | 78.7 | 77.6 | 34.2 | 51.9 | 51.8 | 16.5 | 48.5 | 34.9 | 14.2 | 19.2 | 48.7 | 19.1 | 18.5 | 30.3 | 22.2 | 30.1 | 42.8 | 37.3
simCrossTrans[1] | Swin-T | 87.2 | 87.7 | 51.6 | 69.5 | 69.0 | 27.0 | 60.5 | 48.1 | 19.3 | 38.3 | 68.1 | 30.7 | 35.5 | 61.2 | 41.9 | 47.7 | 55.8 | 52.7
YONOD (ours) | Swin-T | 88.1 | 87.6 | 53.8 | 66.8 | 69.5 | 28.7 | 62.2 | 47.2 | 19.7 | 41.9 | 68.5 | 28.5 | 35.8 | 62.8 | 41.9 | 51.5 | 56.6 | 53.4
YONOD + CPPM (ours) | Swin-T | 88.0 | 85.6 | 51.8 | 68.3 | 68.6 | 26.9 | 61.6 | 45.5 | 20.2 | 41.7 | 67.6 | 29.2 | 33.0 | 61.6 | 47.4 | 47.5 | 55.8 | 52.8
RGB $\&$ Depth/Point Cloud | RGB-D RCNN[5] | VGG | 76.0 | 69.8 | 37.1 | 49.6 | 41.2 | 31.3 | 42.2 | 43.0 | 16.6 | 34.9 | N/A | N/A | N/A | 46.8 | N/A | 41.9 | 44.2 | N/A
YONOD \+ CPPM (ours) | Swin-T | 86.5 | 91.0 | 54.4 | 70.2 | 67.2 | 30.3 | 57.5 | 48.7 | 22.8 | 52.7 | 66.6 | 29.2 | 41.2 | 68.6 | 57.9 | 48.0 | 58.1 | 55.8

Table 3: 2D detection results based on SUN RGB-D validation set. Evaluation
metric is average precision with 2D IoU threshold of 0.5.

We provide a comprehensive per category results comparison with previous works
as shown in Table 3. Compare the YONOD with CPPM to our RGB only model, the
YONOD one has a slightly worse (52.3 $\to$ 51.9) performance. compare with the
point cloud only one, we have similar performance (52.7 $\to$ 52.8); When
using the novel inter modality mixing method proposed in this work, our model
can achieve a significantly better performance with 3.1 (52.7 $\to$ 55.8)
absolute improvement comparing with the previous best result from the
simCrossTrans [1]. For the most commonly used subgroup of SUNRGBD10, we also
achieve a new SOTA of 58.1 ($56.8\to 58.1$) when using YONOD with CPPM.

###  4.8 More results based on extra evaluation metrics

More results based on mAP/mAP75 can be found in the appendix.

###  4.9 Number of parameters and inference time

Method | Backbone Network | # Parameters (M) | GFLOPs | Inference Time (ms) | FPS
---|---|---|---|---|---
F-VoxNet [8] | ResNet-101 | 64 | - | 110 | 9.1
simCrossTrans[1] | ResNet-50 | 44 | 472.1 | 70 | 14.3
simCrossTrans [1] | Swin-T | 48 | 476.5 | 105 | 9.5
YONOD (ours) | Swin-T | 48 | 476.5 | 105 | 9.5
YONOD (ours) | Swin-S | 69 | 419.7 | 148 | 6.8

Table 4: Number of parameters and inference time comparison. All speed testing
are based on a standard single NVIDIA Titan-X GPU.

Number of parameters and inference time is shown in Table 4. As we use the
same network and hardware, we use the inference time from the simCrossTrans
[1] for our Swin-T based network. For the larger Swin-S based network, the
inference time is slower as expected.

##  5 Conclusion

We introduce a unified model which can process different modality data: RGB
from camera, DHS from depth sensor, inter modality mixing image from RGB and
DHS. This unified system is powerful with excellent performance. By using this
system, it can work smoothly with different scenarios related to the
availability of the sensors: such as only an RGB camera is available during
the day time and only depth sensor is available during the night time. Hence,
it is more robust and environment friendly. By using RGB camera during the day
time and depth sensor during the night time, this unified model can be more
environment friendly due to low and even no extra light needed. Meanwhile,
with the novel inter-modality mixing method, we can achieve a significantly
better performance with the unified model when both RGB camera and depth
sensor are available.

## References

  * [1] Shen, X., Stamos, I.:  simcrosstrans: A simple cross-modality transfer learning for object detection with convnets or vision transformers (2022)
  * [2] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.:  Swin transformer: Hierarchical vision transformer using shifted windows.  International Conference on Computer Vision (ICCV) (2021)
  * [3] LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.:  Backpropagation applied to handwritten zip code recognition.  Neural Comput. 1 (1989) 541–551
  * [4] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.:  Attention is all you need.  In Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., eds.: Advances in Neural Information Processing Systems. Volume 30., Curran Associates, Inc. (2017)
  * [5] Gupta, S., Girshick, R.B., Arbeláez, P.A., Malik, J.:  Learning rich features from rgb-d images for object detection and segmentation.  In Fleet, D.J., Pajdla, T., Schiele, B., Tuytelaars, T., eds.: Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII. Volume 8695 of Lecture Notes in Computer Science., Springer (2014) 345–360
  * [6] Stamos, I., Hadjiliadis, O., Zhang, H., Flynn, T.:  Online algorithms for classification of urban objects in 3d point clouds.  In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization Transmission. (2012) 332–339
  * [7] Zelener, A., Stamos, I.:  Cnn-based object segmentation in urban lidar with missing points.  In: 2016 Fourth International Conference on 3D Vision (3DV). (2016) 417–425
  * [8] Shen, X., Stamos, I.:  Frustum voxnet for 3d object detection from rgb-d or depth images.  In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). (2020)
  * [9] Shen, X., Stamos, I.:  3d object detection and instance segmentation from 3d range and 2d color images.  Sensors 21 (2021)
  * [10] Girshick, R.B., Donahue, J., Darrell, T., Malik, J.:  Rich feature hierarchies for accurate object detection and semantic segmentation.  CoRR abs/1311.2524 (2013)
  * [11] Girshick, R.B.:  Fast R-CNN.  In: 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, IEEE Computer Society (2015) 1440–1448
  * [12] Ren, S., He, K., Girshick, R.B., Sun, J.:  Faster R-CNN: towards real-time object detection with region proposal networks.  In Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R., eds.: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. (2015) 91–99
  * [13] Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.:  Feature pyramid networks for object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2017)
  * [14] He, K., Gkioxari, G., Dollár, P., Girshick, R.:  Mask r-cnn.  In: 2017 IEEE International Conference on Computer Vision (ICCV). (2017) 2980–2988
  * [15] Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.:  You only look once: Unified, real-time object detection.  CoRR abs/1506.02640 (2015)
  * [16] Redmon, J., Farhadi, A.:  YOLO9000: better, faster, stronger.  CoRR abs/1612.08242 (2016)
  * [17] Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.:  Focal loss for dense object detection.  CoRR abs/1708.02002 (2017)
  * [18] Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.:  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html (2012)
  * [19] Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.:  Microsoft COCO: common objects in context.  CoRR abs/1405.0312 (2014)
  * [20] Shen, X.:  A survey of object classification and detection based on 2d/3d data.  CoRR abs/1905.12683 (2019)
  * [21] Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., Guo, B.:  Swin transformer V2: scaling up capacity and resolution.  CoRR abs/2111.09883 (2021)
  * [22] Huang, Z., Liu, J., Li, L., Zheng, K., Zha, Z.:  Modality-adaptive mixup and invariant decomposition for rgb-infrared person re-identification.  CoRR abs/2203.01735 (2022)
  * [23] Ling, Y., Zhong, Z., Luo, Z., Rota, P., Li, S., Sebe, N.  In: Class-Aware Modality Mix and Center-Guided Metric Learning for Visible-Thermal Person Re-Identification. Association for Computing Machinery, New York, NY, USA (2020) 889–897
  * [24] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.:  BERT: Pre-training of deep bidirectional transformers for language understanding.  In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, Association for Computational Linguistics (2019) 4171–4186
  * [25] Radford, A., Narasimhan, K.:  Improving language understanding by generative pre-training.  (2018)
  * [26] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.:  Language models are unsupervised multitask learners.  (2018)
  * [27] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.:  Language models are few-shot learners.  CoRR abs/2005.14165 (2020)
  * [28] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.:  Masked autoencoders are scalable vision learners.  CoRR abs/2111.06377 (2021)
  * [29] Shen, X., Liu, Y., Wu, Y., Xie, L.:  Molgnn: Self-supervised motif learning graph neural network for drug discovery.  Machine Learning for Molecules Workshop at NeurIPS 2020 (2020)
  * [30] Liu, Y., Wu, Y., Shen, X., Xie, L.:  Covid-19 multi-targeted drug repurposing using few-shot learning.  Frontiers in Bioinformatics 1 (2021) 18
  * [31] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.:  An image is worth 16x16 words: Transformers for image recognition at scale.  In: International Conference on Learning Representations. (2021)
  * [32] Zelener, A., Stamos, I.:  Cnn-based object segmentation in urban lidar with missing points.  International Conference on 3D Vision (2016)
  * [33] Stamos, I., Hadjiliadis, O., Zhang, H., Flynn, T.:  Online algorithms for classification of urban objects in 3d point clouds.  In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization Transmission. (2012) 332–339
  * [34] Wikipedia contributors:  Flood fill — Wikipedia, the free encyclopedia.  https://en.wikipedia.org/w/index.php?title=Flood_fill&oldid=1087894346 (2022) [Online; accessed 26-June-2022].
  * [35] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C.C., Lin, D.:  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155 (2019)
  * [36] Song, S., Lichtenberg, S.P., Xiao, J.:  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2015)
  * [37] Song, S., Xiao, J.:  Deep sliding shapes for amodal 3d object detection in RGB-D images.  CoRR abs/1511.02300 (2015)
  * [38] Lahoud, J., Ghanem, B.:  2d-driven 3d object detection in rgb-d images.  In: The IEEE International Conference on Computer Vision (ICCV). (2017)
  * [39] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.:  Imagenet: A large-scale hierarchical image database.  In: 2009 IEEE conference on computer vision and pattern recognition, Ieee (2009) 248–255
  * [40] Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.:  Frustum pointnets for 3d object detection from rgb-d data.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018)