# HODINet: High-Order Discrepant Interaction Network for RGB-D Salient Object
Detection

Kang Yi, Jing Xu, Xiao Jin, Fu Guo and Yan-Feng Wu Manuscript received 14 June
2022. This work was supported in part by Tianjin Natural Science Foundation,
China (22JCQNJC01650, 19JCQNJC00300), and Fundamental Research Funds for the
Central Universities of Nankai University (63201192, 63211116). (Kang Yi and
Jing Xu contributed equally to this work.) (Corresponding author: Xiao Jin.)
Yi Kang, Jing Xu, Xiao Jin and Yan-Feng Wu are with the College of Artificial
Intelligence, Nankai University, Tianjin 300350, China (e-mail:
jinxiao@nankai.edu.cn). Guo Fu is with Tandon School of Engineering, The New
York University, New York 11201, USA.

###### Abstract

RGB-D salient object detection (SOD) aims to detect the prominent regions by
jointly modeling RGB and depth information. Most RGB-D SOD methods apply the
same type of backbones and fusion modules to identically learn the
multimodality and multistage features. However, these features contribute
differently to the final saliency results, which raises two issues: 1) how to
model discrepant characteristics of RGB images and depth maps; 2) how to fuse
these cross-modality features in different stages. In this paper, we propose a
high-order discrepant interaction network (HODINet) for RGB-D SOD. Concretely,
we first employ transformer-based and CNN-based architectures as backbones to
encode RGB and depth features, respectively. Then, the high-order
representations are delicately extracted and embedded into spatial and channel
attentions for cross-modality feature fusion in different stages.
Specifically, we design a high-order spatial fusion (HOSF) module and a high-
order channel fusion (HOCF) module to fuse features of the first two and the
last two stages, respectively. Besides, a cascaded pyramid reconstruction
network is adopted to progressively decode the fused features in a top-down
pathway. Extensive experiments are conducted on seven widely used datasets to
demonstrate the effectiveness of the proposed approach. We achieve competitive
performance against 24 state-of-the-art methods under four evaluation metrics.

###### Index Terms:

RGB-D SOD, cross-modality fusion, high-order attention mechanism, transformer,
discrepant interaction.

##  I Introduction

SALIENT object detection (SOD) refers to locating the most attractive parts of
a given scene [1, 2]. It plays an important role in the field of computer
vision and has been applied to various scenarios. Although the SOD methods
have made notable progress and improvement, the performance often degrades
when facing complex and indistinguishable cases. With the rapid development of
3D image sensors, people can obtain high-quality depth information easily,
which has attracted growing interest among researchers in RGB-D SOD methods.
Numerous studies have shown that utilizing RGB-D image pairs for saliency
detection can achieve superior performance in some challenging scenes.

###  I-A Motivation

The motivation of this work mainly derives from two aspects: On one hand, even
though RGB images and depth maps contain completely different information,
most of the existing RGB-D SOD methods still employ the same type of backbones
to identically extract features from different modalities [3]. On the other
hand, different stages of backbones focus on extracting distinct spatial and
channel features, most of the existing RGB-D SOD methods only apply the same
kind of fusion modules [4, 5]. Concretely, there still remains a few problems
that demand further discussions and improvements in RGB-D SOD.

Figure 1: The results of our HODINet and some other first-order fusion
methods, including DCF [5], DMRA [3] and TriTransNet [6]. Note that
TriTransNet [6] adopts transformer-based backbones, we still obtain
competitive results.

####  I-A1 how to model discrepant characteristics of RGB images and depth
maps

The RGB images contain local color and texture contents, while the depth maps
provide shape and position clues in a global view [7, 8]. The RGB and depth
features with mismatched information will generate suboptimal fusion outcomes,
and result in a negative effect on the performance. Most previous methods
employ the same type of backbones or siamese networks to extract features
identically, which is lack of emphasis on the complementary information
between different modalities. One possible solution to this problem is to
adopt more specific backbones based on the characteristics of the modalities.
For example, local continuity and structure information should be emphasized
when encoding RGB images with rich texture features. Since the practical
receptive fields are much smaller than the theoretical ones [9], the backbone
for the depth branch needs to take a stack of convolutional layers to include
global position relationship rather than local details.

####  I-A2 how to fuse features from two modalities in different stages

Spatial and channel attention plays a crucial role in fusion modules and
brings significant performance improvements. Nevertheless, due to the
discrepancy in resolutions and channels of features maps in different stages,
utilizing the same fusion module in all stages cannot guarantee satisfactory
results. The neural networks tend to obtain some specific features in the
shallow layers, while some abstracted semantic features are learned in the
deep layers. Different types of features should be discrepantly integrated and
fused to achieve better experimental results. Therefore, it is significant to
design suitable fusion modules that take into account the diversity of
different stages.

###  I-B Contribution

To tackle the above issues, we propose a high-order discrepant interaction
network (HODINet) for RGB-D SOD. It is a dual-stream asymmetric framework
where transformer-based and CNN-based architectures are used as backbones to
extract features from RGB images and depth maps. Moreover, high-order
statistical information has been proved to be able to distinguish subtle
differences between images in fine-grained image classification [10, 11].
Inspired by this, to exploit complementary information between different
modalities, high-order representations are carefully embedded into spatial and
channel attention to fuse the features from RGB images and depth maps. Since
low-level features contain spatial information and high-level features
concentrate on channel dimensions, a high-order spatial fusion (HOSF) module
and a high-order channel fusion (HOCF) module are introduced to provide
detailed and accurate guidance for feature fusion. Besides, an effective
cascaded pyramid reconstruction network (CPRN) is adopted for cross-stage
decoding. Extensive experiments are conducted on seven benchmark datasets, and
the results indicate that the proposed HODINet achieves competitive
performance against 24 RGB-D SOD methods. Fig. 1 illustrates a comparison of
detection results of our HODINet and other conventional fusion methods.

In summary, the main contributions in this paper can be described as follows.

  * •

We present a high-order discrepant interaction network (HODINet) for RGB-D
SOD, which is a dual-stream asymmetric framework using transformer-based and
CNN-based architectures to extract features from RGB images and depth inputs.

  * •

To distinguish complementary information between two modalities, we design a
high-order spatial fusion (HOSF) module and a high-order channel fusion (HOCF)
module, which learn high-order representations in spatial and channel
attention.

  * •

We also develop a cascaded pyramid reconstruction network (CPRN) to
progressively decode the multi-scale fused features in a top-down pathway.

  * •

The proposed method is evaluated on seven publicly available datasets under
four widely used metrics. Compared with 24 state-of-the-art approaches,
HODINet shows comparable performance.

The rest of this paper is organized as follows. In Section II, the related
work is presented. Then, the detailed methodology is described in Section III.
The experimental settings and result discussion are shown in Section IV.
Finally, Section V concludes this paper.

##  II Related Work

###  II-A RGB-D Salient Object Detection

Early RGB-D SOD methods usually design handcrafted features and various fusion
strategies to integrate RGB and depth inputs. Following this direction,
numerous models have been proposed to detect salient objects [12, 13, 14, 15].
However, they merely regard the depth stream as auxiliary information, which
results in unsatisfactory performance.

With the rapid development of deep learning, CNN-based methods have achieved
remarkable progress [16, 17, 18, 19]. Huang et al. [20] explore cross-modality
interactions between RGB and depth features using linear and bilinear fusion
strategies. Zhang et al. [21] estimate depth maps by the corresponding RGB
images and integrate depth features to enhance the saliency detection
performance. Huang et al. [22] capture cross-level and cross-modal
complementary information by multi-branch feature fusion and selection. Jiang
et al. [23] incorporate intra-view and correlation information in RGB-D SOD by
a novel cross-modality saliency generative adversarial network. Liu et al.
[24] build an attentive cross-modal fusion network based on residual attention
for RGB-D SOD. Zhou et al. [25] present a cross-level and cross-scale adaptive
fusion network to detect RGB-D saliency. Mao et al. [26] propose a novel
cross-modality fusion and progressive integration network to address saliency
prediction on stereoscopic 3D images.

With the recent success in other computer vision tasks [27], the transformer-
based models have opened up new directions in RGB-D SOD. Liu et al. [28]
design a pure transformer framework, which utilizes the T2T-ViT to divide
images into patches and the RT2T transformation to decode patch tokens to
saliency maps. Liu et al. [4] introduce a triplet transformer scheme to model
the long-range relationship among different stages, which serves as
supplementary encoding strategy for feature fusion. Liu et al. [29] propose
SwimNet for RGB-D SOD. They fully explore the advantages of CNN and
transformer in modeling local and global features. Due to the inherent
discrepancy of the two modalities, using the exact same type of backbones to
extract features indiscriminately may not obtain satisfactory results. Thus,
the proposed HODINet is designed as a dual-stream asymmetric framework using
transformer-based and CNN-based architectures to extract features from RGB
images and depth maps. Section III-B presents this part in details.

###  II-B High-Order Attention Mechanism

In some visual tasks, high-order relationship has been proved to be important
for performance improvements. For example, fine-grained image categorization
is a task need to distinguish tiny differences between images. High-order
relationship has demonstrated its effectiveness in this task. Lin et al. [30]
propose bilinear models that use outer product to represent local pairwise
feature interactions. Chen et al. [31] design a high-order attention module
that utilizes the high-order polynomial predictor to capture subtle
differences among objects in the image. Zhao et al. [32] introduce a graph-
based relation discovery approach to learn positional and semantic features.
When applied to RGB-D SOD, the high-order representation helps the encoder to
distinguish tiny differences between two modalities. Complementary information
in this visual task can be further investigated.

Besides, various forms of attention mechanisms are exploited to focus on the
pivotal information in the image, and have significantly improved the
performance of computer vision tasks. Hou et al. [33] present a mixed pooling
module that captures long-range dependencies and also focuses on local details
simultaneously. Fu et al. [34] design a position attention module and a
channel attention module to model the semantic dependencies in spatial and
channel dimensions, respectively. Yang et al. [35] observe the mechanism of
human attention and propose a 3D attention module, which uses an energy
function to calculate the attention weights. Hou et al. [36] decompose the
channel attention into two 1D feature coding processes. Inspired by these
above attention mechanisms, we design the HOSF and HOCF modules to promote the
interactions in cross-modality features, which will be elaborated in Section
III-C and Section III-D.

##  III Proposed Method

In this section, we describe the proposed HODINet in detail.

###  III-A Overview

The whole framework is shown in Fig. 2. Given an RGB image and a corresponding
depth map, we use transformer-based and CNN-based networks as backbones to
extract features from RGB and depth inputs, respectively. Then, we divide the
backbones into four stages. Inspired by the recent success of high-order
representation [10, 11], output features from the first two stages are fed
into high-order spatial fusion (HOSF) modules to enhance the spatial
characterization, while output features from the last two stages are put into
high-order channel fusion (HOCF) modules to obtain the channel information.
Subsequently, the fused features are input to the cascaded pyramid
reconstruction network (CPRN). The CPRN progressively upsamples the resolution
of feature maps to integrate the multistage information. In addition, we use
deep supervision [37] at each stage of the decoder to enhance the expression
of different resolution features. The binary cross-entropy (BCE) loss is
replaced by the hybrid loss [38] for accurate boundary prediction and higher
performance. Table I lists some operations and notations used in this paper.

TABLE I: Descriptions of notations and operations used in this paper. Notations | Descriptions
---|---
${\rm Conv}_{k\times k}(\cdot)$ |  $k$$\times$$k$ convolutional layer
${\rm MO}(\cdot)$ | moment normalization
${\rm Norm}(\cdot)$ |  $l$-2 normalization
${\rm GMP}(\cdot)$ | global max pooling
${\rm GMP}^{c\times c}(\cdot)$ | global max pooling with stride of $c$$\times$$c$
${\rm GAP}(\cdot)$ | global average pooling
${\rm Cat}[\cdot;\cdot]$ | concatenation
${\rm FC}(\cdot)$ | full connected layer
$\sigma(\cdot)$ | sigmoid activation function
${\rm BN}(\cdot)$ | batch normalization (BN)
${\rm ReLU}(\cdot)$ | rectified linear unit (ReLU) activation function
${\rm NFE}(\cdot)$ | 1$\times$1 convolutional layer with BN and ReLU
${\rm Up}_{\times s}(\cdot)$ | upsampling $s$ times with bilinear interpolations
${\rm Up}_{ori}(\cdot)$ | upsampling to the original size of the input data
$\otimes$ | element-wise multiplication
$\oplus$ | element-wise addition
Figure 2: Detailed framework of the proposed HODINet. We adopt transformer-
based and CNN-based models as backbones to extract features of RGB and depth
inputs, respectively. Then, the high-order spatial fusion (HOSF) module and
the high-order channel fusion (HOCF) module are proposed to fuse feature maps
extracted from the first two stages and the last two stages, respectively.
Finally, the fused feature maps of all stages are decoded by the cascaded
pyramid reconstruction network (CPRN).

###  III-B Encoder

When encoding the input data of RGB stream and depth stream, we need to fully
consider the differences between these two modalities. As discussed before,
the RGB images contain abundant local structure information, while the depth
maps intuitively describe object position relationships in the whole image.
Therefore, we enhance the local continuity in the RGB branch by a transformer-
based backbone equipped with overlapping patch embedding. For the depth
stream, we take a stack of convolutional layers to cover the position
information in a global view.

RGB Stream. We employ PVTv2-B3 [39] as the backbone of the RGB stream, because
it emphasize local continuity by overlapping patch embedding. PVTv2 can be
divided into four stages when generating features of different scales. At each
stage, the input image is decomposed into patches with position embedding.
Then, these patches pass through a transformer-based encoder with several
layers. Afterwards, the output is reshaped into feature maps for multi-stage
prediction tasks. Moreover, another advantage of PVTv2 is that the resolution
of output feature maps of each stage is always the same as ResNet models [40],
making it easy to incorporate with the CNN-based networks.

Depth Stream. To fully investigate the position information in a global view,
we stack more convolutional layers in the backbone of the depth branch. Since
the actual receptive field is often much smaller than the theoretical ones
[9], a deeper backbone network helps to obtain more sufficient global
information. Some recent work also use deep network [41] or large
convolutional kernel [7] to preserve large receptive field in the depth
stream. In this paper, we adopt ResNet-101 [40] as the backbone of the depth
branch. To apply it to RGB-D SOD task, we remove the last global average
pooling layer and 1000-way fully-connected layer in the original ResNet-101
network. As a result, all of the remaining blocks will be divided into four
stages, expect for the first 7$\times$7 convolution and global max pooling
layer. To be more specific, we set the feature maps of 11-th, 23-th, 92-th and
101-th layers as the outputs of four stages.

###  III-C High-Order Spatial Fusion (HOSF) Module

Figure 3: The illustration of the proposed HOSF module.

Complementary information between RGB images and depth maps plays a crucial
role in RGB-D SOD [42, 8]. In some visual tasks, high-order representation has
shown its effectiveness to distinguish tiny differences between images.
Inspired by this, we propose two high-order attention modules to further
improve the performance of RGB-D SOD.

The feature maps extracted from shallower layers have a large resolution,
which often contain more spatial details. Based on this, we design a High-
Order Spatial Fusion (HOSF) module to integrate the RGB modality with the
depth features effectively. The HOSF module utilizes the spatial high-order
representation to enhance complementary information. The complete architecture
of HOSF is illustrated in Fig. 3.

Specifically, we obtain the RGB features $\mathcal{F}^{RGB}_{i}$ and the depth
features $\mathcal{F}^{Depth}_{i}$ from their corresponding backbones, where
$i\in\left\\{1,2\right\\}$ indicates the stage indexes of the encoder. Since
outputs of different backbones are inconsistent in channel dimension, we first
apply a 1$\times$1 convolutional layer to align the number of channels,

$\displaystyle f^{RGB}_{i}=$ $\displaystyle{\rm ReLU}({\rm BN}({\rm
Conv}_{1\times 1}(\mathcal{F}^{RGB}_{i}))),$ (1) $\displaystyle
f^{Depth}_{i}=$ $\displaystyle{\rm ReLU}({\rm BN}({\rm Conv}_{1\times
1}(\mathcal{F}^{Depth}_{i}))),$ (2)

where ${\rm Conv}_{1\times 1}(\cdot)$ denotes a 1$\times$1 convolutional
layer, ${\rm BN}(\cdot)$ represents a batch normalization (BN) layer [43] and
${\rm ReLU}(\cdot)$ symbolizes a rectified linear unit (ReLU) activation
function.

Then, we extract high-order representations [44] to jointly learn the spatial
relationship between $f^{RGB}_{i}$ and $f^{Depth}_{i}$,

${\mathcal{A}^{sp}_{i}={\rm Norm}({\rm MO}((f^{RGB}_{i})^{T}\otimes
f^{Depth}_{i}))\otimes(f^{RGB}_{i})^{T},}$ (3)

where $T$ stands for the transposition operation, $\otimes$ indicates the
element-wise multiplication operation, ${\rm MO}(x)=sign(x)\cdot x^{-1/2}$
denotes the moment, and ${\rm Norm}(x)=x/\left\|x\right\|_{2}^{2}$ represents
the $l$-2 normalization.

Depth data is generally obtained by depth sensors. The foreground and
background have different depth values because of their distances to the
sensor, which allows backbone networks to learn complementary information
apart from RGB data. However, not all depth information can bring beneficial
improvements to the RGB-D SOD task. The low-quality depth maps will introduce
additional noise to the predicted saliency maps [18, 45]. To fuse the depth
maps effectively, we integrate the depth information selectively by using a
global max pooling operation followed by two 3$\times$3 convolutional layers,
which can be expressed as,

$f^{DW}_{i}={\rm\sigma}({\rm Conv}_{3\times 3}({\rm Conv}_{3\times 3}({\rm
GMP}(f^{Depth}_{i})))),$ (4)

where ${\rm GMP}=(\cdot)$ denotes the global max pooling operation,
$\sigma(\cdot)$ represents the sigmoid activation function. Then, we
supplement the depth clues to the fused features by element-wise
multiplication operation. Finally, a residual connection is used to retain the
global information in RGB stream. The final fused feature maps is formulated
as:

$\begin{split}\mathcal{F}^{fus}_{i}=(\mathcal{A}^{sp}_{i}\otimes
f^{DW}_{i})\oplus f^{RGB}_{i},\\\ i\in\\{1,2\\},\end{split}$ (5)

where $\oplus$ represents the element-wise addition operation.

The HOSF module is proposed to interact the feature maps in shallow layers of
the encoders. In the next subsection, we will introduce anther high-order
attention module for deep layers in the backbones.

###  III-D High-Order Channel Fusion (HOCF) Module

Figure 4: The illustration of the proposed HOCF module.

In encoder networks, the features extracted from deep layers have more
channels than the features in shallow layers. Each channel represents a type
of semantic information. By modeling the channel interactions in a high-order
manner, we can emphasize complementary feature representation for specific
semantics. Thus, we introduce the High-Order Channel Fusion (HOCF) module to
exploit the channel relationships between two modalities. The illustration of
HOCF is shown in Fig. 4.

In the HOCF module, we first adopt a 1$\times$1 convolutional layer to align
the number of channels,

$\displaystyle f^{RGB}_{i}=$ $\displaystyle{\rm ReLU}({\rm BN}({\rm
Conv}_{1\times 1}(\mathcal{F}^{RGB}_{i}))),$ (6) $\displaystyle
f^{Depth}_{i}=$ $\displaystyle{\rm ReLU}({\rm BN}({\rm Conv}_{1\times
1}(\mathcal{F}^{Depth}_{i}))),$ (7)

where $i\in\left\\{3,4\right\\}$ is the indexes of stage in encoder backbones.
Then, the global average pooling operation is added to generate the
corresponding channel attention mask. We employ the element-wise
multiplication operation to calculate the interactive channel attention map
$\mathcal{A}^{ch}_{i}\in\mathbb{R}^{C\times C}$,

$\displaystyle\mathcal{A}^{RGB}_{i}=$ $\displaystyle{\rm GAP}(f^{RGB}_{i}),$
(8) $\displaystyle\mathcal{A}^{Depth}_{i}=$ $\displaystyle{\rm
GAP}(f^{Depth}_{i}),$ (9) $\displaystyle\mathcal{A}^{ch}_{i}=$
$\displaystyle\mathcal{A}^{RGB}_{i}\otimes\mathcal{A}^{Depth}_{i}.$ (10)

After these operations, we obtain the maximum value of each row and each
column in the interactive channel attention map, respectively. Then, we apply
a series of operations to further integrate the most prominent features,

$\displaystyle\tilde{\mathcal{A}}^{ch}_{i}=$ $\displaystyle\sigma({\rm
FC}({\rm Cat}[{\rm GMP}^{C\times 1}(\mathcal{A}^{ch}_{i});{\rm GMP}^{1\times
C}(\mathcal{A}^{ch}_{i})]),$ (11)

where ${\rm Cat}[\cdot,\cdot]$ denotes a concatenation operation, ${\rm
FC}(\cdot)$ stands for a fully connected (FC) layer, $\sigma(\cdot)$ is a
sigmoid activation function. Here, ${\rm GMP}^{C\times
1}(\mathcal{A}^{ch}_{i})$ calculates how each channel of RGB feature interacts
with the depth modality, and ${\rm GMP}^{1\times C}(\mathcal{A}^{ch}_{i})$
obtains how each channel of depth feature correlates with the RGB modality.
Finally, we use element-wise multiplication operation and residual connection
to obtain fused feature maps,

$\begin{split}\mathcal{F}^{fus}_{i}=(\tilde{\mathcal{A}}^{ch}_{i}\otimes
f^{Depth}_{i})\oplus f^{RGB}_{i},\\\ i\in\\{3,4\\}.\end{split}$ (12)

These feature maps are then fed into the decoder to recover the resolutions.

###  III-E Decoder

In the decoder, if we upsample the feature maps directly to the raw size of
the inputs, it will produce excessive noise and imprecise results [4]. This is
unacceptable for pixel-level prediction tasks. As mentioned in [46], combining
upsample operations with convolutional layers is crucial for restoring the
resolution in decoding process. Inspired by the principle of PVT [47] and FPN
[48], we employ a cascaded pyramid reconstruction network (CPRN) to gradually
integrate multi-level features via a top-down pathway. Given the output
$\mathcal{F}^{fus}_{i}$, $i\in\\{1,2,3,4\\}$, from four high-order fusion
modules, we first apply two 1$\times$1 convolutional layers to reduce the
number of channels. Then, the feature maps are upsampled two times with
bilinear interpolations to make the resolution compatible to next stage. After
that, we concatenate these feature maps. The above operations are continued
until the feature map is restored to the size of original inputs.

For brevity, we first denote a nonlinear feature enhancement (NFE) unit before
introducing the cascaded pyramid reconstruction network,

${\rm NFE}(\cdot)={\rm ReLU}({\rm BN}({\rm Conv}_{1\times 1}(\cdot))).$ (13)

Thus, we use the following formulas to present the process of cascaded pyramid
decoder,

$\displaystyle\mathcal{F}_{4}^{out}=$ $\displaystyle{\rm NFE}({\rm
NFE}(\mathcal{F}^{fus}_{4})),$ (14) $\displaystyle\mathcal{F}_{3}^{out}=$
$\displaystyle{\rm NFE}({\rm NFE}({\rm Cat}(\mathcal{F}^{fus}_{3};{\rm
Up}_{\times 2}(\mathcal{F}_{4}^{out})))),$ (15)
$\displaystyle\mathcal{F}_{2}^{out}=$ $\displaystyle{\rm NFE}({\rm NFE}({\rm
Cat}(\mathcal{F}^{fus}_{2};{\rm Up}_{\times 2}(\mathcal{F}_{3}^{out})))),$
(16) $\displaystyle\mathcal{F}_{1}^{out}=$ $\displaystyle{\rm NFE}({\rm
NFE}({\rm Cat}(\mathcal{F}^{fus}_{1};{\rm Up}_{\times
2}(\mathcal{F}_{2}^{out})))).$ (17)

Meanwhile, we add deep supervisions [49] to the outputs
$\mathcal{F}^{out}_{i}$, $i\in\\{1,2,3,4\\}$, of the cascaded pyramid decoder
to speed up convergence. The predicted saliency maps can be formulated as,

$\begin{split}P_{i}={\rm Up}_{ori}({\rm Conv}_{1\times 1}({\rm ReLU}({\rm
BN}({\rm Conv}_{3\times 3}(\mathcal{F}^{out}_{i}))))),\\\
i\in\\{1,2,3,4\\},\end{split}$ (18)

where $P_{i}$ represents the predictions in the $i$-th stage, ${\rm
Up}_{ori}(\cdot)$ denotes upsampling the feature maps to the original
resolution of the input image. Note that only $P_{1}$ is used as the final
saliency map, while other three predictions are omitted in the test phase.

###  III-F Loss Function

For each of the decoder stage, we adopt the hybrid loss function [38] as the
loss function, which is a combination of binary cross-entropy (BCE) loss [50],
SSIM loss [51] and IoU loss [52]. It can be defined as below,

$\begin{split}\mathcal{L}_{i}^{hybrid}=\mathcal{L}_{i}^{BCE}+\mathcal{L}_{i}^{SSIM}+\mathcal{L}_{i}^{IoU},\\\
i\in\left\\{{1,2,3,4}\right\\}.\end{split}$ (19)

BCE loss measures the pixel-level difference between the predicted saliency
map $P\in\left[0,1\right]^{W\times H}$ and ground truth
$G\in\left\\{0,1\right\\}^{W\times H}$, which can be denoted as,

$\begin{split}&\mathcal{L}^{BCE}=\\\
&-\sum\limits_{w,h=1}^{W,H}{[{G_{wh}}\log({P_{wh}})+(1-{G_{wh}})\log(1-{P_{wh}})]}.\end{split}$
(20)

SSIM loss evaluates the structural similarity, which is defined as,

$\mathcal{L}^{SSIM}=1-\frac{{(2{\mu_{P}}{\mu_{G}}+{C_{1}})(2{\sigma_{PG}}+{C_{2}})}}{{(\mu_{P}^{2}+\mu_{G}^{2}+{C_{1}})(\sigma_{P}^{2}+\sigma_{G}^{2}+{C_{2}})}},$
(21)

where $\mu_{P},\mu_{G}$ and $\sigma_{P},\sigma_{G}$ are the means and the
standard deviations of the two patches from the predicted saliency map $P$ and
the ground truth $G$, respectively. $\sigma_{PG}$ is their covariance.
Additionally, $C_{1}=0.01^{2}$ and $C_{2}=0.03^{2}$ are set to avoid the
numerator being divided by zero.

IoU loss is adopted to capture fine structures in image level, which is
computed as,

$\mathcal{L}^{IoU}=1-{\frac{\sum\limits_{w,h=1}^{W,H}{P_{wh}}{G_{wh}}}{\sum\limits_{w,h=1}^{W,H}({P_{wh}}+{G_{wh}}-{P_{wh}}{G_{wh}})}}.$
(22)

As a result, the total loss can be represented as,

$\begin{split}\mathcal{L}_{total}=\sum\limits_{i=1}^{4}{{\mathcal{L}_{i}^{hybrid}}({P_{i}},G)},\\\
i\in\\{1,2,3,4\\}.\end{split}$ (23)

All the predicted saliency maps and the ground truths have the same resolution
as original input images.

##  IV Discussion

###  IV-A The novelty of the proposed HOSF and HOCF

The key of RGB-D SOD lies in the complementary information between the two
modalities, and high-order information has been proven to distinguish subtle
differences between images in some other computer vision tasks. Although high-
order attention mechanism has been proposed for many years, it is often only
adopted for single modality feature enhancement. The cross-modality higher-
order attention obtains relatively less research interests. To the best of our
knowledge, we are the first to use higher-order statistical information for
this cross-modality vision task, namely RGB-D SOD. As we can see in the
ablation experiment in Section V-D, HOSF and HOCF actually improve the
performance.

###  IV-B The choices of backbones in RGB branch and depth branch

In this paper, we employ PVTv2 and ResNet-101 as the backbones for the RGB
branch and the depth branch, respectively. In the preliminary experiment, we
found that this implementation achieved the best results. The reason may be
that RGB images contain more valuable information. If both streams use PVTv2
as encoder identically, there will be a slight decrease in performance.
Compared with other transformer-based SOD methods [6], we still achieved
competitive results. In other fields of computer vision, more advanced
backbone networks are naturally used to improve the performance [53].

##  V Experiments

In this section, we first introduce the experimental setup and implementation
details. Then, we compare the proposed HODINet with other state-of-the-art
RGB-D SOD methods on seven benchmark datasets and analyse the experimental
results. The ablation studies are also conducted to verify the effectiveness
of each component. Finally, some failure cases are discussed.

TABLE II: Quantitative results compared with state-of-the-art RGB-D SOD
methods. “-” means that the results are unavailable since the authors did not
release them. $\uparrow$ ($\downarrow$) indicates the larger (smaller), the
better. The best and the second best results are highlighted in bold and
underline, respectively.

| NLPR [54] | DUT-RGBD [55] | NJUD [56] | STEREO [57] | SSD [58] | LFSD [59] | SIP [18]
---|---|---|---|---|---|---|---
Model |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$
Ours | 0.934 | 0.018 | 0.942 | 0.022 | 0.925 | 0.030 | 0.917 | 0.033 | 0.885 | 0.038 | 0.886 | 0.054 | 0.900 | 0.039
SMAC(21TPAMI)[60] | 0.922 | 0.027 | 0.926 | 0.033 | 0.903 | 0.044 | 0.905 | 0.042 |  0.884 |  0.044 | 0.875 |  0.063 | 0.883 |  0.049
CDINet(21MM)[7] | 0.927 | 0.024 | 0.926 | 0.030 | 0.918 | 0.036 | 0.905 | 0.041 | 0.852 | 0.057 | 0.870 | 0.064 | 0.875 | 0.055
DCF(21CVPR)[5] | 0.921 | 0.023 | 0.924 | 0.031 | 0.903 | 0.039 | 0.905 |  0.037 | 0.851 | 0.054 | 0.855 | 0.071 | 0.873 | 0.052
DSA2F(21CVPR)[61] | 0.918 | 0.024 | 0.921 | 0.030 | 0.903 | 0.039 | 0.904 | 0.038 | 0.876 | 0.045 |  0.882 | 0.054 | 0.860 | 0.057
DQSD(21TIP)[62] | 0.915 | 0.030 | 0.844 | 0.073 | 0.898 | 0.051 | 0.891 | 0.052 | 0.868 | 0.053 | 0.850 | 0.085 | 0.863 | 0.065
DRLF(21TIP)[63] | 0.902 | 0.032 | 0.825 | 0.080 | 0.886 | 0.055 | 0.888 | 0.050 | 0.834 | 0.066 | 0.834 | 0.090 | 0.850 | 0.071
HAINet(21TIP)[64] | 0.821 | 0.025 | 0.910 | 0.038 | 0.909 | 0.038 | 0.909 | 0.038 | 0.857 | 0.053 | 0.853 | 0.080 |  0.886 | 0.048
CDNet(21TIP)[65] | 0.927 | 0.025 | 0.927 | 0.031 | 0.918 | 0.036 | 0.906 | 0.040 | 0.876 | 0.045 | 0.858 | 0.073 | 0.874 | 0.054
RD3D(21AAAI)[66] |  0.930 | 0.022 |  0.932 | 0.031 | 0.916 | 0.036 | 0.911 |  0.037 | - | - | 0.857 | 0.074 | 0.885 | 0.048
DRSD(21TMM)[21] | 0.906 | 0.038 | 0.864 | 0.072 | 0.886 | 0.050 | 0.899 | 0.046 | 0.861 | 0.049 | 0.827 | 0.092 | - | -
S3Net(21TMM)[67] | 0.927 |  0.021 | 0.912 | 0.035 | 0.913 |  0.034 |  0.913 | 0.038 | - | - | 0.874 | 0.066 | 0.875 | 0.051
ATSA(20ECCV)[68] | 0.909 | 0.027 | 0.916 | 0.033 | 0.885 | 0.047 | 0.896 | 0.038 | 0.820 | 0.052 | 0.845 | 0.078 | 0.849 | 0.063
CoNet(20ECCV)[69] | 0.912 | 0.027 | 0.923 |  0.029 | 0.896 | 0.046 | 0.905 |  0.037 | 0.851 | 0.056 | 0.848 | 0.076 | 0.860 | 0.058
DANet(20ECCV)[70] | 0.920 | 0.027 | 0.899 | 0.042 | 0.899 | 0.046 | 0.901 | 0.044 | 0.864 | 0.050 | 0.841 | 0.087 | 0.875 | 0.055
BBSNet(20ECCV)[71] | 0.931 | 0.023 | 0.882 | 0.058 |  0.921 | 0.035 | 0.908 | 0.041 | 0.863 | 0.052 | 0.835 | 0.092 | 0.879 | 0.055
A2dele(20CVPR)[19] | 0.899 | 0.029 | 0.885 | 0.043 | 0.871 | 0.051 | 0.879 | 0.045 | 0.803 | 0.070 | 0.825 | 0.084 | 0.829 | 0.070
SSF(20CVPR)[72] | 0.915 | 0.027 | 0.915 | 0.033 | 0.899 | 0.043 | 0.837 | 0.065 | 0.790 | 0.084 | 0.851 | 0.074 | 0.799 | 0.091
UCNet(20CVPR)[73] | 0.920 | 0.025 | 0.871 | 0.059 | 0.897 | 0.043 | 0.903 | 0.039 | 0.865 | 0.049 | 0.856 | 0.074 | 0.875 | 0.051
S2MA(20CVPR)[74] | 0.915 | 0.030 | 0.903 | 0.043 | 0.894 | 0.053 | 0.890 | 0.051 | 0.868 | 0.052 | 0.837 | 0.094 | 0.872 | 0.058
JLDCF(20CVPR)[41] | 0.925 | 0.022 | 0.906 | 0.042 | 0.902 | 0.041 | 0.903 | 0.040 | 0.860 | 0.053 | 0.853 | 0.077 | 0.880 | 0.049
D3Net(20TNNLS)[18] | 0.912 | 0.030 | 0.850 | 0.071 | 0.900 | 0.046 | 0.899 | 0.046 | 0.857 | 0.058 | 0.825 | 0.095 | 0.860 | 0.063
DMRA(19ICCV)[3] | 0.899 | 0.031 | 0.889 | 0.048 | 0.886 | 0.051 | 0.886 | 0.047 | 0.857 | 0.058 | 0.847 | 0.075 | 0.806 | 0.085
CPFP(19CVPR)[17] | 0.888 | 0.036 | 0.818 | 0.076 | 0.878 | 0.053 | 0.879 | 0.051 | 0.807 | 0.082 | 0.828 | 0.088 | 0.850 | 0.064
TANet(19TIP)[75] | 0.886 | 0.041 | 0.808 | 0.093 | 0.878 | 0.060 | 0.871 | 0.060 | 0.839 | 0.063 | 0.801 | 0.111 | 0.835 | 0.075
| NLPR [54] | DUT-RGBD [55] | NJUD [56] | STEREO [57] | SSD [58] | LFSD [59] | SIP [18]
Model |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$
Ours | 0.966 | 0.933 | 0.973 | 0.958 | 0.952 | 0.935 | 0.953 | 0.924 |  0.926 | 0.891 | 0.925 |  0.899 | 0.942 | 0.928
SMAC(21TPAMI)[60] | 0.953 | 0.904 | 0.956 | 0.928 | 0.937 | 0.896 | 0.941 | 0.897 | 0.928 | 0.869 | 0.911 | 0.870 | 0.925 | 0.886
CDINet(21MM)[7] | 0.953 | 0.923 | 0.956 | 0.944 | 0.944 | 0.827 | - | 0.903 | 0.906 | 0.867 |  0.914 | 0.889 | 0.911 |  0.903
DCF(21CVPR)[5] | 0.956 | 0.917 | 0.956 | 0.940 | 0.940 | 0.917 | 0.943 | 0.914 | 0.905 | 0.857 | 0.902 | 0.878 | 0.920 | 0.899
DSA2F(21CVPR)[61] | 0.950 | 0.916 | 0.950 | 0.939 | 0.923 | 0.918 | 0.933 | 0.911 | 0.904 |  0.877 | 0.923 | 0.903 | 0.911 | 0.891
DQSD(21TIP)[62] | 0.934 | 0.909 | 0.889 | 0.859 | 0.912 | 0.910 | 0.911 | 0.900 | 0.890 |  0.877 | 0.883 | 0.875 | 0.900 | 0.890
DRLF(21TIP)[63] | 0.935 | 0.904 | 0.870 | 0.851 | 0.901 | 0.883 | 0.915 | 0.878 | 0.879 | 0.859 | 0.872 | 0.857 | 0.891 | 0.868
HAINet(21TIP)[64] | 0.954 | 0.908 | 0.944 | 0.920 | 0.941 | 0.909 |  0.947 | 0.909 | 0.906 | 0.859 | 0.892 | 0.876 | 0.927 |  0.903
CDNet(21TIP)[65] | 0.955 |  0.926 | 0.956 |  0.945 |  0.950 | 0.919 | 0.942 | 0.898 | 0.924 | 0.871 | 0.896 | 0.861 | 0.917 | 0.902
RD3D(21AAAI)[66] |  0.965 | 0.919 |  0.960 | 0.939 | 0.947 | 0.914 |  0.947 | 0.906 | - | - | 0.897 | 0.878 | 0.924 | 0.889
DRSD(21TMM)[21] | 0.936 | 0.882 | 0.902 | 0.853 | 0.927 | 0.876 | 0.933 | 0.887 | 0.917 | 0.832 | 0.866 | 0.813 | - | -
S3Net(21TMM)[67] | 0.962 | 0.923 | 0.939 | 0.922 | 0.944 |  0.928 | 0.945 |  0.918 | - | - | 0.902 | 0.892 |  0.933 | 0.891
ATSA(20ECCV)[68] | 0.951 | 0.898 | 0.953 | 0.928 | 0.930 | 0.893 | 0.942 | 0.901 | 0.920 | 0.853 | 0.893 | 0.859 | 0.901 | 0.861
CoNet(20ECCV)[69] | 0.948 | 0.893 | 0.959 | 0.932 | 0.937 | 0.893 |  0.947 | 0.901 | 0.917 | 0.837 | 0.895 | 0.852 | 0.917 | 0.873
DANet(20ECCV)[70] | 0.955 | 0.909 | 0.939 | 0.904 | 0.935 | 0.898 | 0.937 | 0.892 | 0.914 | 0.843 | 0.874 | 0.840 | 0.918 | 0.876
BBSNet(20ECCV)[71] | 0.961 | 0.918 | 0.912 | 0.870 | 0.949 | 0.919 | 0.942 | 0.903 | 0.914 | 0.843 | 0.870 | 0.828 | 0.922 | 0.884
A2dele(20CVPR)[19] | 0.944 | 0.882 | 0.928 | 0.891 | 0.916 | 0.874 | 0.928 | 0.880 | 0.862 | 0.777 | 0.866 | 0.828 | 0.890 | 0.834
SSF(20CVPR)[72] | 0.953 | 0.896 | 0.950 | 0.923 | 0.935 | 0.896 | 0.912 | 0.840 | 0.867 | 0.765 | 0.892 | 0.863 | 0.870 | 0.786
UCNet(20CVPR)[73] | 0.956 | 0.903 | 0.908 | 0.864 | 0.936 | 0.895 | 0.944 | 0.899 | 0.907 | 0.855 | 0.898 | 0.860 | 0.919 | 0.879
S2MA(20CVPR)[74] | 0.953 | 0.902 | 0.937 | 0.901 | 0.930 | 0.889 | 0.932 | 0.882 | 0.909 | 0.848 | 0.873 | 0.835 | 0.919 | 0.877
JLDCF(20CVPR)[41] | 0.963 | 0.918 | 0.941 | 0.910 | 0.944 | 0.904 |  0.947 | 0.904 | 0.902 | 0.833 | 0.894 | 0.863 | 0.925 | 0.889
D3Net(20TNNLS)[18] | 0.953 | 0.897 | 0.889 | 0.842 | 0.939 | 0.900 | 0.938 | 0.891 | 0.910 | 0.834 | 0.862 | 0.810 | 0.909 | 0.861
DMRA(19ICCV)[3] | 0.947 | 0.879 | 0.933 | 0.898 | 0.927 | 0.886 | 0.938 | 0.886 | 0.906 | 0.844 | 0.900 | 0.856 | 0.875 | 0.821
CPFP(19CVPR)[17] | 0.932 | 0.863 | 0.859 | 0.795 | 0.923 | 0.877 | 0.925 | 0.874 | 0.852 | 0.766 | 0.872 | 0.826 | 0.903 | 0.851
TANet(19TIP)[75] | 0.941 | 0.863 | 0.861 | 0.790 | 0.925 | 0.874 | 0.923 | 0.861 | 0.897 | 0.810 | 0.847 | 0.796 | 0.895 | 0.830

Figure 5: Visual comparisons of the proposed HODINet and other state-of-the-
art RGB-D SOD methods. Our approach obtain competitive performance in a
variety of complex scenarios.

###  V-A Datasets and Evaluation Metrics

Datasets. To demonstrate the generalization capability of the proposed model,
we conduct experiments on seven publicly available datasets. NLPR [54]
includes 1,000 stereo images from 11 types of indoor and outdoor scenes. DUT-
RGBD [55] contains 1,200 images captured by Lytro Illum camera. NJUD [56] is
composed of 1,985 image pairs from 3D movies and photographs. STEREO [57]
collects 1,000 pairs of binocular images from the Internet. SSD [58] and LFSD
[59] are two small-scale datasets with only 80 and 100 image pairs,
respectively. SIP [18] involves 929 images with high quality depth maps. It
covers many challenging scenes of salient objects and persons. Following the
same settings as previous work [72, 5, 7], we use a combined set of three
datasets to train the proposed model, including 700 images from NLPR, 800
pairs from DUT-RGBD and 1,485 samples from NJUD. The remaining images with
corresponding depth maps in these datasets are used for test.

Evaluation metrics. We adopt four widely used metrics to comprehensively
compare our HODINet with other methods, including mean absolute error (MAE)
[76], S-measure ($S_{\alpha}$) [77], max F-measure ($F_{\beta}$) [78] and max
E-measure ($E_{\xi}$) [79]. MAE calculates the mean absolute difference in
pixel-level between the predicted saliency map and the ground truth. The
S-measure is designed to evaluate the structural similarities in object-level
and region-level. Both F-measure and E-measure first obtain the binary
saliency maps by varying the thresholds. The max F-measure computes the
harmonic mean of average precision and average recall in multiple thresholds,
while E-measure utilizes global image-level and local pixel-level information
to measure the converted binary maps.

###  V-B Implementation Details

We conduct all experiments with an NVIDIA GeForce RTX 2080Ti GPU on the
Pytorch framework. The transformer-based and CNN-based backbones are
pretrained on ImageNet [80]. The other layers are initialized to the default
settings in Pytorch. During the training and test phase, we simply copy the
input depth maps to three channels and resize all images to 256$\times$256\.
Besides, multiple data augmentations, such as random cropping, flipping,
rotating and color enhancement, are used in the training phase to avoid
overfitting. We use the Adam optimizer [81] to train the proposed network in
an end-to-end manner with a batch size of 4 for 100 epochs. The initial
learning rate is 1e-4 and then decreases by a factor of 0.9 every epoch. The
whole training procedure takes about 5 hours for convergence. The inference
speed is around 35 FPS when testing an image with a size of 256$\times$256 on
a single NVIDIA RTX 2080Ti.

###  V-C Comparison With State-of-the-art Methods

In this subsection, we compare the proposed HODINet with 24 state-of-the-art
RGB-D SOD methods, including SMAC [60], CDINet [7], DCF [5], DSA2F [61], DQSD
[62], DRLF [63], HAINet [64], CDNet [65], RD3D [66], DRSD [21], S3Net [67],
ATSA [68], CoNet [69], DANet [70], BBSNet [71], A2dele [19], SSF [72], UCNet
[73], S2MA [74], JLDCF [41], D3Net [18], DMRA [3], CPFP [17], and TANet [75].
We directly report the quantitative results in the published papers if
accessible. For the other baselines, we calculate the evaluation metrics by
the saliency maps provided by the authors. All the metrics are calculated by
the official evaluation tools.

####  V-C1 Quantitative Analysis

Table. II illustrates the quantitative results of our method. We calculate
four evaluation metrics on seven datasets for comprehensive comparisons.
Higher values of $S_{\alpha}$, $F_{\beta}$ and $E_{\xi}$ indicate better
performance. On the contrary, ${\rm MAE}$ is the opposite. We place other
baselines in reverse chronological order from top to bottom. Compared with
other state-of-the-art methods, the proposed HODINet obtains comparable
performance. Concretely, the results of HODINet are the best under all the
evaluation metrics, except that $F_{\beta}$ on LFSD and $E_{\xi}$ on SSD are
the second best. Especially on the SIP dataset, the performance gain of our
method achieves 2.8% of $F_{\beta}$, 1.6% of $S_{\alpha}$, 1.0% of $E_{\xi}$
and 20.4% of ${\rm MAE}$ score, when comparing to the second best approach.
Besides, we also improves 1.4% of $F_{\beta}$, 1.1% of $S_{\alpha}$, 1.4% of
$E_{\xi}$ and 24.1% of ${\rm MAE}$ score on the DUT-RGBD dataset. As a
consequence, all the results demonstrate the effectiveness of our model.

####  V-C2 Qualitative Analysis

For qualitative comparison, we provide some representative saliency maps
predicted by HODINet and other SOTA methods in Fig. 5. These samples include a
variety of challenging scenarios, such as fine structures (1st row), fine-
grained details (2nd row), large objects (3rd-4th rows), small objects (5th
row), poor-quality depth maps (6th-7th rows), multiple objects (8th row), low
contrast (9th row) and complex scenes (10th-11th rows). For example, we
achieve structural integrity and internal consistency of the salient objects
in the 2nd, 3rd and 7th images. On the contrary, other methods either miss
detailed parts or mistakenly recognize the background as the foreground.

###  V-D Ablation Study

To further investigate the effectiveness of each main component in the
proposed HODINet, we conduct a series of ablation studies. These mainly
include four aspects: 1) why bridging transformer-based and CNN-based networks
to extract features of RGB and depth modalities; 2) how to effectively utilize
the high-order attention mechanisms to discrepantly interact with different
stages of feature maps; 3) whether or not can we replace the cascaded pyramid
reconstruction network (CPRN) with convolutions and upsamplings to decode
feature maps directly; 4) how much does the hybrid loss contribute to the
performance. Note that we only modify one component at a time and retrain the
networks with the same settings in Section V-B. Since the trends of
experimental results found on these seven datasets are similar, we only
reported ablation results on NLPR, DUT-RGBD, and LFSD in this subsection.

TABLE III: Ablation studies for the backbones on three datasets. $\uparrow$
($\downarrow$) indicates the larger (smaller), the better.

| Model | NLPR | DUT-RGBD | LFSD
---|---|---|---|---
|  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$
| Ours | 0.934 | 0.018 | 0.966 | 0.933 | 0.942 | 0.022 | 0.973 | 0.958 | 0.886 | 0.054 | 0.925 | 0.899
Backbone | CNN+Transformer | 0.915 | 0.026 | 0.949 | 0.909 | 0.912 | 0.039 | 0.948 | 0.931 | 0.854 | 0.070 | 0.899 | 0.880
CNN+CNN | 0.922 | 0.023 | 0.954 | 0.915 | 0.912 | 0.040 | 0.950 | 0.935 | 0.866 | 0.068 | 0.906 | 0.882
Transformer+Transformer | 0.933 | 0.020 | 0.962 | 0.930 | 0.941 | 0.023 | 0.972 | 0.957 | 0.881 | 0.057 | 0.913 | 0.894

TABLE IV: Ablation studies for the high-order fusion modules on three
datasets. $\uparrow$ ($\downarrow$) indicates the larger (smaller), the
better.

| Model | NLPR | DUT-RGBD | LFSD
---|---|---|---|---
|  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$
| Ours | 0.934 | 0.018 | 0.966 | 0.933 | 0.942 | 0.022 | 0.973 | 0.958 | 0.886 | 0.054 | 0.925 | 0.899
Fusion | HOCF-HOSF | 0.923 | 0.025 | 0.953 | 0.922 | 0.931 | 0.030 | 0.960 | 0.948 | 0.869 | 0.064 | 0.908 | 0.879
w/o HOCF | 0.913 | 0.035 | 0.940 | 0.909 | 0.923 | 0.032 | 0.953 | 0.940 | 0.861 | 0.062 | 0.897 | 0.873
w/o HOSF | 0.912 | 0.034 | 0.942 | 0.910 | 0.923 | 0.037 | 0.943 | 0.938 | 0.858 | 0.071 | 0.885 | 0.873

TABLE V: Ablation studies for the proposed decoder and the loss function on
three datasets. $\uparrow$ ($\downarrow$) indicates the larger (smaller), the
better.

| Model | NLPR | DUT-RGBD | LFSD
---|---|---|---|---
|  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$ |  $S_{\alpha}$ $\uparrow$ | MAE $\downarrow$ |  $E_{\xi}$ $\uparrow$ |  $F_{\beta}$ $\uparrow$
| Ours | 0.934 | 0.018 | 0.966 | 0.933 | 0.942 | 0.022 | 0.973 | 0.958 | 0.886 | 0.054 | 0.925 | 0.899
Decoder | w/o CPRN | 0.921 | 0.025 | 0.952 | 0.912 | 0.927 | 0.033 | 0.956 | 0.942 | 0.872 | 0.066 | 0.903 | 0.872
Loss | only L-BCE | 0.931 | 0.023 | 0.955 | 0.927 | 0.937 | 0.029 | 0.965 | 0.952 | 0.884 | 0.059 | 0.915 | 0.893

####  V-D1 the necessity of using CNN-based and transformer-based networks as
backbones

Figure 6: Visualization results of different backbone choices.

The backbones for RGB and depth streams focus on different aspects in
extracting features. In this subsection, we conduct experiments to verify the
impact of different backbone choices. For example, if we employ CNN-based
model to encode both RGB images and depth maps, the performance will decrease.
We denote this attempt as “CNN-CNN” in Table. III. The reason can be explained
as that PVTv2 enhances the local continuity in RGB images by overlapping patch
embedding [39]. However, compared with other CNN-based methods in Table. II,
we still obtain comparable results, when only using CNN-based backbones for
encoding both RGB and depth streams. If we apply transformer-based backbone to
the depth stream, the performance is slightly lower than that of CNN-based
model. Fig. 6 shows the comparison of different backbone choices.

####  V-D2 the significance of the HOSF and HOCF modules

Complementary information between RGB images and depth maps is critical in
RGB-D SOD. High-order relationship has shown its effectiveness in modeling
tiny differences between images [10, 11]. In our framework, HOSF and HOCF play
an important role in fusing cross-modality features. We directly substitute
them with two convolutions and a concatenation operation, which are denoted as
“w/o HOSF” and “w/o HOCF” in Table. IV. On top of that, we also exchange the
sequence of HOSF and HOCF modules to investigate whether they have the same
effect on different stages. Under different fusion strategies, we observe
varying degrees of performance degradation in the above three datasets. This
study fully indicates that the proposed high-order attention mechanisms
capture critical information well in both spatial and channel dimensions. The
representative examples are visualized in Fig. 7.

####  V-D3 the effectiveness of the CPRN

To demonstrate the importance of the proposed CPRN, we remove all cross-stage
connections and directly predict the final feature maps. Specifically, we
employ convolutions to reduce the number of channels to one. Then, the
predictions are upsampled to the original input size. The results are shown as
“w/o CPRN” in Table. V. Compared to the complete HODINet, we find that the
performance of this attempt degrades severely on all three datasets. The
reason is that the top-town integration pathway decodes the adjacent feature
maps gradually, preserving the consistent saliency information. In the
penultimate column of Fig. 8, we show some qualitative examples in this
circumstance.

####  V-D4 the usefulness of the hybrid loss

Several loss functions have been used in the RGB-D SOD task, achieving
remarkable results [5, 82, 17]. The hybrid loss [38] improves the integrity of
salient objects with sharp and clear boundaries. In this ablation experiment,
we replace the hybrid loss with the binary cross-entropy (BCE) loss. As shown
in the last row “only $\mathcal{L}^{BCE}$” in Table. V, we can see significant
improvements due to the hybrid loss, especially on the $E_{\xi}$ metric and
MAE score. Fig. 8 shows some qualitative comparisons of BCE loss and hybrid
loss.

Figure 7: Visualization of saliency detection results for different variations
of fusion modules. Figure 8: Ablation study results of the proposed CPRN and
the hybrid loss.

###  V-E Failure Cases and Analyses

It is also worthwhile to analyze the reasons for failure cases. Fig. 9 shows a
few samples where our model fails to detect. First, the performance of the
proposed method will decrease when the object is composed of fine structures.
Since these contiguous pixels are easily adjacent to backgrounds, we can not
obtain ideal boundaries, e.g., the first row of Fig. 9. In addition, as shown
in the second and the third rows in Fig. 9, the blur depth maps provide little
useful information, while the salient objects do not have explicit semantic
information in the RGB images. In this circumstance, our method can not
produce satisfactory results. Finally, in some scenes with multiple different
objects, our model fails to detect the salient objects due to contradictory
depth cues. The fourth row of Fig. 9 is an example.

Figure 9: Failure cases of the proposed HODINet.

##  VI Conclusion

In this paper, we propose a novel high-order discrepant interaction network
(HODINet) for RGB-D SOD. In the proposed framework, we adopt transformer-based
and CNN-based architectures to discrepantly extract features from different
modalities. Inspired by the recent success of high-order representations in
other visual tasks, we design high-order spatial fusion (HOSF) and high-order
channel fusion (HOCF) modules to select the most effective information. A
cascaded pyramid reconstruction network (CPRN) is also employed to gradually
decode the feature maps in a top-down pathway. The experimental results also
prove the effectiveness of the proposed model. Compared with 24 state-of-the-
art methods, HODINet achieves competitive performance on seven popular
datasets.

## References

  * [1] W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang, “Salient Object Detection in the Deep Learning Era: An In-Depth Survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3239–3259, 2022.
  * [2] R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang, “Review of visual saliency detection with comprehensive information,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 2941–2959, 2019.
  * [3] Y. Piao, W. Ji, J. Li, M. Zhang, and H. Lu, “Depth-induced multi-scale recurrent attention network for saliency detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7254–7263.
  * [4] Z. Liu, Y. Wang, Z. Tu, Y. Xiao, and B. Tang, “TriTransNet: RGB-D salient object detection with a triplet transformer embedding network,” in _Proceedings of the ACM International Conference on Multimedia_ , 2021, pp. 4481–4490.
  * [5] W. Ji, J. Li, S. Yu, M. Zhang, Y. Piao, S. Yao, Q. Bi, K. Ma, Y. Zheng, H. Lu, and L. Cheng, “Calibrated RGB-D Salient Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9466–9476.
  * [6] Z. Liu, Y. Wang, Z. Tu, Y. Xiao, and B. Tang, “TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network,” in _Proceedings of the ACM International Conference on Multimedia_ , 2021, pp. 4481–4490.
  * [7] C. Zhang, R. Cong, Q. Lin, L. Ma, F. Li, Y. Zhao, and S. Kwong, “Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection,” in _Proceedings of the ACM International Conference on Multimedia_ , 2021, pp. 2094–2102.
  * [8] G. Li, Z. Liu, L. Ye, Y. Wang, and H. Ling, “Cross-Modal Weighting Network for RGB-D Salient Object Detection,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 665–681.
  * [9] W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” vol. 29, 2016.
  * [10] H. Zheng, J. Fu, Z.-J. Zha, and J. Luo, “Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5012–5021.
  * [11] S. Kong and C. Fowlkes, “Low-rank bilinear pooling for fine-grained classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 365–374.
  * [12] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “RGBD salient object detection: A benchmark and algorithms,” in _Proceedings of the European Conference on Computer Vision_ , 2014, pp. 92–109.
  * [13] H. Song, Z. Liu, H. Du, G. Sun, O. Le Meur, and T. Ren, “Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4204–4216, 2017.
  * [14] D. Feng, N. Barnes, S. You, and C. McCarthy, “Local background enclosure for RGB-D salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2343–2350.
  * [15] J. Ren, X. Gong, L. Yu, W. Zhou, and M. Ying Yang, “Exploiting global priors for RGB-D saliency detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2015, pp. 25–32.
  * [16] H. Chen, Y. Li, and D. Su, “Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection,” _Pattern Recognition_ , vol. 86, pp. 376–385, 2019.
  * [17] J.-X. Zhao, Y. Cao, D.-P. Fan, M.-M. Cheng, X.-Y. Li, and L. Zhang, “Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3922–3931.
  * [18] D.-P. Fan, Z. Lin, Z. Zhang, M. Zhu, and M.-M. Cheng, “Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 5, pp. 2075–2089, 2021.
  * [19] Y. Piao, Z. Rong, M. Zhang, W. Ren, and H. Lu, “A2dele: Adaptive and attentive depth distiller for efficient RGB-D salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9060–9069.
  * [20] N. Huang, Y. Yang, D. Zhang, Q. Zhang, and J. Han, “Employing Bilinear Fusion and Saliency Prior Information for RGB-D Salient Object Detection,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 1651–1664, 2022.
  * [21] Y.-f. Zhang, J. Zheng, W. Jia, W. Huang, L. Li, N. Liu, F. Li, and X. He, “Deep RGB-D Saliency Detection Without Depth,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 755–767, 2022.
  * [22] N. Huang, Y. Liu, Q. Zhang, and J. Han, “Joint Cross-Modal and Unimodal Features for RGB-D Salient Object Detection,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 2428–2441, 2021.
  * [23] B. Jiang, Z. Zhou, X. Wang, J. Tang, and B. Luo, “cmSalGAN: RGB-D Salient Object Detection With Cross-View Generative Adversarial Networks,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 1343–1353, 2021.
  * [24] D. Liu, K. Zhang, and Z. Chen, “Attentive Cross-Modal Fusion Network for RGB-D Saliency Detection,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 967–981, 2021.
  * [25] W. Zhou, Y. Zhu, J. Lei, J. Wan, and L. Yu, “CCAFNet: Crossflow and Cross-Scale Adaptive Fusion Network for Detecting Salient Objects in RGB-D Images,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 2192–2204, 2022\.
  * [26] Y. Mao, Q. Jiang, R. Cong, W. Gao, F. Shao, and S. Kwong, “Cross-Modality Fusion and Progressive Integration Network for Saliency Prediction on Stereoscopic 3D Images,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 2435–2448, 2022.
  * [27] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _Proceedings of the International Conference on Learning Representations_ , 2021.
  * [28] N. Liu, N. Zhang, K. Wan, L. Shao, and J. Han, “Visual saliency transformer,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 4722–4732.
  * [29] Z. Liu, Y. Tan, Q. He, and Y. Xiao, “SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4486–4497, 2021.
  * [30] T.-Y. Lin, A. RoyChowdhury, and S. Maji, “Bilinear CNN Models for Fine-Grained Visual Recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1449–1457.
  * [31] B. Chen, W. Deng, and J. Hu, “Mixed High-Order Attention Network for Person Re-Identification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 371–381.
  * [32] Y. Zhao, K. Yan, F. Huang, and J. Li, “Graph-based high-order relation discovery for fine-grained recognition,” in _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 074–15 083.
  * [33] Q. Hou, L. Zhang, M.-M. Cheng, and J. Feng, “Strip pooling: Rethinking spatial pooling for scene parsing,” in _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4002–4011.
  * [34] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3141–3149.
  * [35] L. Yang, R.-Y. Zhang, L. Li, and X. Xie, “SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks,” in _Proceedings of the International Conference on Machine Learning_ , 2021, pp. 11 863–11 874.
  * [36] Q. Hou, D. Zhou, and J. Feng, “Coordinate attention for efficient mobile network design,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 13 708–13 717.
  * [37] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-Supervised Nets,” in _Proceedings of the International Conference on Artificial Intelligence and Statistics_ , 2015, pp. 562–570.
  * [38] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “BASNet: Boundary-Aware Salient Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7471–7481.
  * [39] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “PVT v2: Improved baselines with Pyramid Vision Transformer,” _Computational Visual Media_ , vol. 8, pp. 415–424, 2022.
  * [40] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [41] K. Fu, D.-P. Fan, G.-P. Ji, and Q. Zhao, “JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3052–3062.
  * [42] Y. Pang, L. Zhang, X. Zhao, and H. Lu, “Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 235–252.
  * [43] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of the International Conference on Machine Learning_ , 2015, pp. 448–456.
  * [44] Y. Zhao, K. Yan, F. Huang, and J. Li, “Graph-Based High-Order Relation Discovery for Fine-Grained Recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 079–15 088.
  * [45] Y. Yang, Q. Qin, Y. Luo, Y. Liu, Q. Zhang, and J. Han, “Bi-directional Progressive Guidance Network for RGB-D Salient Object Detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 8, pp. 5346–5360, 2022.
  * [46] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1874–1883.
  * [47] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 568–578.
  * [48] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2117–2125.
  * [49] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-Supervised Nets,” in _Proceedings of the International Conference on Artificial Intelligence and Statistics_ , vol. 38, 2015, pp. 562–570.
  * [50] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” _Annals of Operations Research_ , vol. 134, no. 1, pp. 19–67, 2005.
  * [51] Z. Wang, E. Simoncelli, and A. Bovik, “Multiscale structural similarity for image quality assessment,” in _Proceedings of the Asilomar Conference on Signals, Systems Computers_ , vol. 2, 2003, pp. 1398–1402.
  * [52] G. Máttyus, W. Luo, and R. Urtasun, “Deeproadmapper: Extracting road topology from aerial images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3438–3446.
  * [53] C.-L. Guo, Q. Yan, S. Anwar, R. Cong, W. Ren, and C. Li, “Image Dehazing Transformer with Transmission-Aware 3D Position Embedding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5812–5820.
  * [54] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “RGBD salient object detection: a benchmark and algorithms,” in _Proceedings of the European Conference on Computer Vision_ , 2014, pp. 92–109.
  * [55] Y. Piao, W. Ji, J. Li, M. Zhang, and H. Lu, “Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7253–7262.
  * [56] R. Ju, L. Ge, W. Geng, T. Ren, and G. Wu, “Depth saliency based on anisotropic center-surround difference,” in _Proceedings of the IEEE International Conference on Image Processing_ , 2014, pp. 1115–1119.
  * [57] Y. Niu, Y. Geng, X. Li, and F. Liu, “Leveraging stereopsis for saliency analysis,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 454–461.
  * [58] C. Zhu and G. Li, “A Three-Pathway Psychobiological Framework of Salient Object Detection Using Stereoscopic Technology,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 3008–3014.
  * [59] N. Li, J. Ye, Y. Ji, H. Ling, and J. Yu, “Saliency detection on light field,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 2806–2813.
  * [60] N. Liu, N. Zhang, L. Shao, and J. Han, “Learning Selective Mutual Attention and Contrast for RGB-D Saliency Detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 12, pp. 9026–9042, 2022\.
  * [61] P. Sun, W. Zhang, H. Wang, S. Li, and X. Li, “Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1407–1417.
  * [62] C. Chen, J. Wei, C. Peng, and H. Qin, “Depth-quality-aware salient object detection,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2350–2363, 2021.
  * [63] X. Wang, S. Li, C. Chen, Y. Fang, A. Hao, and H. Qin, “Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 458–471, 2021.
  * [64] G. Li, Z. Liu, M. Chen, Z. Bai, W. Lin, and H. Ling, “Hierarchical Alternate Interaction Network for RGB-D Salient Object Detection,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3528–3542, 2021.
  * [65] W.-D. Jin, J. Xu, Q. Han, Y. Zhang, and M.-M. Cheng, “Cdnet: Complementary depth network for rgb-d salient object detection,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3376–3390, 2021.
  * [66] Q. Chen, Z. Liu, Y. Zhang, K. Fu, Q. Zhao, and H. Du, “RGB-D Salient Object Detection via 3D Convolutional Neural Networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021, pp. 1063–1071.
  * [67] L. Zhu, X. Wang, P. Li, X. Yang, Q. Zhang, W. Wang, C.-B. Schonlieb, and C. P. Chen, “S3Net: Self-supervised Self-ensembling Network for Semi-supervised RGB-D Salient Object Detection,” _IEEE Transactions on Multimedia_ , vol. 25, pp. 676–689, 2023.
  * [68] M. Zhang, S. X. Fei, J. Liu, S. Xu, Y. Piao, and H. Lu, “Asymmetric two-stream architecture for accurate RGB-D saliency detection,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 374–390.
  * [69] W. Ji, J. Li, M. Zhang, Y. Piao, and H. Lu, “Accurate RGB-D salient object detection via collaborative learning,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 52–69.
  * [70] X. Zhao, L. Zhang, Y. Pang, H. Lu, and L. Zhang, “A single stream network for robust and real-time RGB-D salient object detection,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 646–662.
  * [71] D.-P. Fan, Y. Zhai, A. Borji, J. Yang, and L. Shao, “BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 275–292.
  * [72] M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu, “Select, supplement and focus for RGB-D saliency detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3472–3481.
  * [73] J. Zhang, D.-P. Fan, Y. Dai, S. Anwar, F. S. Saleh, T. Zhang, and N. Barnes, “UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8582–8591.
  * [74] N. Liu, N. Zhang, and J. Han, “Learning Selective Self-Mutual Attention for RGB-D Saliency Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 756–13 765.
  * [75] H. Chen and Y. Li, “Three-Stream Attention-Aware Network for RGB-D Salient Object Detection,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 2825–2835, 2019.
  * [76] A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A benchmark,” _IEEE Transactions on Image Processing_ , vol. 24, no. 12, pp. 5706–5722, 2015.
  * [77] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A new way to evaluate foreground maps,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4558–4567.
  * [78] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned salient region detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 1597–1604.
  * [79] D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2018, pp. 698–704.
  * [80] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [81] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014.
  * [82] Y.-H. Wu, Y. Liu, J. Xu, J.-W. Bian, Y.-C. Gu, and M.-M. Cheng, “MobileSal: Extremely Efficient RGB-D Salient Object Detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 12, pp. 10 261–10 269, 2021.

CRediT authorship contribution statement

Kang Yi: Software, Data Curation, Writing - Original Draft.

Jing Xu: Supervision, Resources.

Xiao Jin: Conceptualization, Methodology, Software, Writing - Original Draft,
Writing - Review & Editing.

Fu Guo: Validation.

Yan-Feng Wu: Validation.

In the final published version, Xiao Jin has renounced the authorship of this
manuscript.