# Disentangled High Quality Salient Object Detection

Lv Tang Bo Li1 Shouhong Ding1 Mofei Song2,3
1Youtu Lab Corresponding author and equal contribution to first author. This
work was supported by National Natural Science Foundation of China 61906036
and the Fundamental Research Funds for the Central Universities
(2242021k30056). Tencent  Shanghai  China
2The School of Computer Science and Engineering
3The Key Lab of Computer Network and Information Integration (Ministry of
Education)
Southeast University  Nanjing  China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Aiming at discovering and locating most distinctive objects from visual
scenes, salient object detection (SOD) plays an essential role in various
computer vision systems. Coming to the era of high resolution, SOD methods are
facing new challenges. The major limitation of previous methods is that they
try to identify the salient regions and estimate the accurate objects
boundaries simultaneously with a single regression task at low-resolution.
This practice ignores the inherent difference between the two difficult
problems, resulting in poor detection quality. In this paper, we propose a
novel deep learning framework for high-resolution SOD task, which disentangles
the task into a low-resolution saliency classification network (LRSCN) and a
high-resolution refinement network (HRRN). As a pixel-wise classification
task, LRSCN is designed to capture sufficient semantics at low-resolution to
identify the definite salient, background and uncertain image regions. HRRN is
a regression task, which aims at accurately refining the saliency value of
pixels in the uncertain region to preserve a clear object boundary at high-
resolution with limited GPU memory. It is worth noting that by introducing
uncertainty into the training process, our HRRN can well address the high-
resolution refinement task without using any high-resolution training data.
Extensive experiments on high-resolution saliency datasets as well as some
widely used saliency benchmarks show that the proposed method achieves
superior performance compared to the state-of-the-art methods.

##  1 Introduction

Salient object detection (SOD) is derived with the goal of accurately
detecting and segmenting the most distinctive objects from visual scenes. As a
preliminary step, it plays an essential role in various visual systems, such
as video object segmentation [49], light field image segmentation [45], image-
sentence matching [21], person re-identiﬁcation [27] and instance segmentation
[70].

Recently, the rapid development of the commodity imaging and display device,
has resulted in higher requirements for the producing and editing of high-
resolution (e.g., 720p, 1080p and 4K) images. Salient object detection as well
as many state-of-the-art computer vision tasks are facing various challenges
when encountering high-resolution scenarios. A good high-resolution salient
object detection method should not only accurately detect the whole salient
object but also predict the precise boundaries of salient objects. Despite the
conventional Deep Neural Networks (DNNS) based SOD models have achieved
remarkable performance at low-resolution (e.g., typical size $224\times 224$,
$384\times 384$), they often fail to generate high quality detection results
for high-resolution images. The major reason for this drawback is that the
most previous methods try to identify the salient regions and estimate the
accurate objects boundaries simultaneously in one step, which are two
difficult and inherently different problems for high-resolution salient object
detection. To address the first problem, a network is required to capture
sufficient semantics by maintaining a larger receptive field. However, since
the memory usage increases dramatically along with the image resolution, it is
impractical for these models to directly learn sufficient semantics for high-
resolution images. One plausible way is introducing downsample operations, but
the structure details are inevitably lost during the downsampling, which
however is precisely the key to solving the second problem.

Figure 1: Comparison with state-of-the-art method in high-resolution SOD. Best
viewed by zooming in.

Unfortunately, most of the existing low-resolution SOD methods [28, 63, 5, 53]
try to address the aforementioned two problems with a single regression
framework, which ignore the inherent difference between the two problems and
result in blurry boundaries. As shown in Fig.1, if we take a deeper look at
the saliency map generated by the representative existing methods LDF [53] and
GateNet [68], we can observe that pixels can be divided into three different
sets: (1) most pixels inside the salient object have the highest saliency
value, and we call these pixels as definite salient pixels; (2) most pixels in
the background regions have the lowest salient value, which belong to definite
background pixels; (3) saliency values of the pixels at blurry object
boundaries fluctuate between 0 and 1, so we call these pixels as uncertain
pixels. An ideal SOD method should effectively identify the definite salient
and background regions in the image and accurately calculate the saliency
value of pixels in the uncertain region to preserve a clear object boundary.
From this perspective, there are essentially two tasks in SOD which demand
quite different abilities to address the aforementioned two problems. The
former task can be viewed as a classic classification task, while the later
one is a typical regression task.

Despite the demand for effective high-resolution SOD methods, this line of
work is rarely studied. In this paper, motivated by the new observation that
SOD should be disentangled into two tasks, we propose a novel deep learning
framework for high-resolution salient object detection. Specifically, we
decouple the high-resolution salient object detection into a low-resolution
saliency classification network (LRSCN) and a high-resolution refinement
network (HRRN). LRSCN is designed to capture sufficient semantics at low-
resolution and classify the pixels into three different sets for later
process. HRRN aims at accurately refining the saliency value of pixels in the
uncertain region to preserve a clear object boundary at high-resolution with
limited GPU memory. As discussed above, HRRN requires structure details in
high-resolution image. However, widely used low-resolution saliency datasets
generally have some problems in annotation quality [58], making it almost
impossible to directly obtain enough object boundary details from these
defective datasets to train the high-resolution network. In the very recent
work, Zeng et al. [58] proposed to train their SOD network by using high-
resolution images with accurate annotation. However, such high-quality image
annotation requires heavy labor costs. In our paper, we argue that it is
unnecessary to use such accurately annotated high-resolution images in network
training. By introducing uncertainty [22] in the training process, our HRRN
can well address the high-resolution refinement task only using the low-
resolution training datasets with poor annotation.

Our major contributions can be summarized as:

  * •

We provide a new perspective that high-resolution salient object detection
should be disentangled into two tasks, and demonstrate that the
disentanglement of the two tasks is essential for improving the performance of
DNN based SOD models.

  * •

Motivated by the principle of disentanglement, we propose a novel framework
for high-resolution salient object detection, which uses LRSCN to capture
sufficient semantics at low-resolution and HRRN for accurate boundary
refinement at high-resolution.

  * •

We make the earliest efforts to introduce the uncertainty into SOD network
training, which empowers HRRN to well address the high-resolution refinement
task without any high-resolution training datasets.

  * •

We perform extensive experiments to demonstrate the proposed method refreshes
the SOTA performance on high-resolution saliency datasets as well as some
widely used saliency benchmarks by a large margin.

Figure 2: The framework of the proposed disentangled high quality salient
object detection method.

##  2 Related Work

Over the past decades, a large amount of SOD algorithms have been developed.
Traditional models [20, 7, 42, 46, 23]detect salient objects by utilizing
various heuristic saliency priors with hand-crafted features. More details
about the traditional methods can be found in the survey [1]. Recently, with
the development of deep learning, the performance of saliency detection has
archived great improvment [28, 10, 30, 39, 71, 44, 65]. Here we mainly focus
on deep learning based saliency detection models.

Recently, some DNN-based models use various feature enhancement strategies to
improve the ability of localization and awareness of salient objects [18, 54,
41, 6, 52, 32, 68, 13], or take advantage of edge features to restore the
structural details of salient objects [50, 66, 55, 69]. For example, Pang et
al. [32] applied the transformation-interaction-fusion strategy on multi-level
and multi-scale features to learn discriminant feature representation. Zhao et
al. [68] designed a gated dual branch structure to build the cooperation among
different levels of features and improve the discriminability of the whole
network. In [66], edge features from edge detection branch was fused with
salient features as complementary information to enhance the structural
details for accurate saliency detection. Zhou et al. [69] used two individual
branches for representing saliency and contour stream respectively, and a
novel feature fusion module for their correlation combination.

Different from the above methods, some methods consider leveraging predict-
refine architecture to generate fine salient objects. For example, Wang et al.
[47] proposed to localize salient objects globally and then refine them by a
local boundary refinement module. Qin et al. [35] was composed of an Encoder-
Decoder network and a residual refinement module, which were respectively in
charge of saliency prediction and saliency map refinement.

However, all these methods cannot handle high-resolution salient object
detection problem well since such simple regression framework cannot identify
the salient regions and estimate the accurate objects boundaries
simultaneously and their architectures are not optimized for high-resolution
SOD. Zeng et al. [58] tried to alleviate this problem by leveraging both
global semantic information and local high-resolution details to accurately
detect salient objects in high-resolution images. However, Zeng et al. [58]
relies on high-resolution training images with accurate annotation, which
requires heavy labor costs. Different from the above methods, we disentangle
high-resolution SOD into two tasks at different resolutions: identifying the
salient regions at low-resolution and estimating the accurate objects
boundaries at high-resolution. Moreover, unlike Zeng et al. [58], we introduce
novel uncertainty loss, which empowers our HRRN to well address the high-
resolution refinement task without using any high-resolution training
datasets. Recently, Wei et al. [53] and Zhang et al. [61] also leverage
disentanglement in their SOD methods. However, they still try to address the
SOD task under a single regression framework but with decoupled supervisions.
Unlike our proposed methods, their disentanglement frameworks barely touch the
very nature of the SOD, which essentially contains two different tasks. For
more information about the DNN-based methods, please refer to survey [48, 16].

##  3 Proposed Method

In this section, we first describe the overall architecture of the proposed
disentangled high quality salient object detection network, then elaborate our
main contributions, which are corresponding to LRSCN and HRRN.

###  3.1 Network Overview

The architecture of the proposed approach is illustrated in Fig.2. As can be
seen, the disentanglement includes two decoupled tasks at two different
resolutions. LRSCN aims at capturing sufficient semantics at low-resolution
and classifying the pixels into three different sets, which also can save the
memory usage. While estimating the accurate objects boundaries needs more
local details at high-resolution. So, we design HRRN to regress the saliency
value of pixels and preserve a clear object boundary at high-resolution.

LRSCN has a simple U-Net like Encoder-Decoder architecture [36]. VGG-16 [38]
is used as backbone. Following [18, 66], we connect another side path to the
last pooling layer in VGG-16. Hence, we obtain six side features Conv1-2,
Conv2-2, Conv3-3, Conv4-3, Conv5-3 and Conv6-3 from backbone network. Because
Conv1-2 and Conv2-2 are too close to the input and their receptive fields are
too small, following [66, 54], we only use the last four levels features for
the following process. Conv6-3 is denoted as $\\{F_{h}|h=6\\}$, the other
three levels features are denoted as $\\{F_{l}|l=3,4,5\\}$. Multi-scale
feature extraction and Cross-level feature fusion (MECF) module is added
between encoder and decoder to help improve the discriminability of feature
representations. Decoder fuses the output features from MECF and the upsampled
features from the previous stage in a bottom-up manner. The output of each
decoder is deﬁned as $\\{D_{i}|i=3,4,5,6\\}$. Finally, SGA module is built
upon $D_{3}$ for accurate trimap $T$ generation.

As described, LRSCN is classification task and aims at capturing sufficient
semantics at low-resolution. To regress a clear object boundary value, the
input of HRRN is a high-resolution image under guidance of the trimap provided
by LRSCN. HRRN has a basic Encoder-Decoder architecture and with the help of
uncertainty loss, the network can be more robust to noisy data and predict a
high-resolution saliency map with clear boundary.

###  3.2 Architecture of LRSCN

To capture sufficient semantics at low-resolution, learning discriminant
feature representations is essential. The network should not only consider
scale and location variations of different salient objects, but also
distinguish the appearance difference between the salient object and the non-
salient regions. To achieve the first goal, we develop a multi-scale feature
extraction module (ME) based on Global Convolutional Network (GCN) [33] to
enlarge the feature receptive field and obtain multi-scale information. To
achieve the second goal, we utilize cross-level feature fusion module (CF) to
leverage the advantages of features at different levels.Moreover, in designing
the network architecture, inspired by [56], we use split-transform-merge
strategy to further enlarge feature receptive fields and hence results in more
discriminative feature representations. Specifically, we uniformly split the
input $F$ into two portions $\\{F^{1},F^{2}\\}$ by channel dimension, then
$F^{1}$ is sent into multi-scale feature extraction pathway and $F^{2}$ is
sent into cross-level feature fusion pathway. The outputs of these two
pathways are concatenated together as the final output. we call this bridge
module as MECF module, which is shown in Fig.3. More details about MECF module
can be found in section 6 of supplementary materials.

Figure 3: Architecture of MECF and SGA Modules.

SGA Module. As illustrated in Fig.2, each decoder fuses features from MECF
module and previous decoder stage, then uses $3\times 3$ convolutional layer
for final prediction. To maintain consistency between trimap and saliency map
and ensure the uncertain regions of the trimap can accurately cover the
boundary of saliency map, we design a saliency guide attention module (SGA) on
$D_{3}$. Specifically, we first use a $3\times 3$ convolution and sigmoid
function to compute a saliency map. Then, the saliency map is treated as
spatial weight map which can help refine feature and generate an accurate
trimap. Finally, the output trimap $T$ is 3-channel classiﬁcation logits. The
whole SGA module guarantees the alignment of trimap and saliecny map.

###  3.3 Architecture of HRRN

Following the principle of disentanglement, HRRN aims at accurately refining
the saliency value of pixels in the uncertain region to preserve a clear
object boundary at high-resolution under the guidance of the trimap provided
by LRSCN. The architecture of HRRN is shown in Fig.2. HRRN has a simple U-NET
like architecture. For better prediction at high-resolution, we then do some
non-trivial modifications. First, lower-level features contain rich spatial
and detail information which play a crucial role in restoring a clear object
boundary, so decoder combines encoder features before each upsampling block
instead of after each upsampling block. Moreover, we use a two layers short
cut block to align channels of encoder features for feature fusion. Second, to
let network pay more attention to detail information, we directly feed the
original input to the last convolutional layer through a short cut block to
generate better results. Finally, learning from image generation tasks [3,
59], we use the spectral normalization [31] to each convolutional layer to add
a constraint on Lipschitz constant of the network and stable the training.

###  3.4 Loss Function of LRSCN

Figure 4: Examples of trimap. Column 3 shows the trimaps generated from GT.
Column 4 shows the trimaps predicted by LRSCN.

To supervise LRSCN, inspired by some typical matting mehtods [4, 40], we
generate trimap groundtruth $T^{gt}$, which can represent the definite
salient, definite background and uncertain regions. As described, uncertain
regions exist mainly at the boundaries of the objects. So we erase and dilate
binary groundtruth maps at the object boundaries with a random pixel number
(5,7,9,11,13) to generate the GT uncertain regions. The remaining foreground
and background regions represent definite salient and background regions.
$T^{gt}$ is defined as:

$T^{gt}(x,y)=\begin{cases}2,\ &{T^{gt}(x,y)\in definite\ salient}\\\ 0,\
&{T^{gt}(x,y)\in definite\ background}\\\ 1,\ &{T^{gt}(x,y)\in uncertain\
region}\end{cases}$ (1)

where $(x,y)$ stands for each pixel location on the image. Some examples can
be seen in Fig.4.

For trimap supervision, we use Softmax cross-entropy loss, which is defined
as:

$\displaystyle
L_{trimap}=\frac{1}{N}\sum_{i}-log(\frac{e^{T_{i}}}{\sum_{j}e^{T_{j}}}).$ (2)

To guarante the arruracy of trimap, we add extra saliency supervision
$L_{saliency}$ as the supplement of trimap supervision. Similar to BASNet
[35], we use pixel-level, region-level and object-level supervision strategy
on multi-levels to better keep the uniformity and wholeness of the salient
objects. Specifically, binary cross-entropy (BCE) [8], SSIM [51] and F-measure
loss [67] are denoted as pixel-level, region-level and object-level loss. Note
that all parts of LRSCN are trained jointly, so the overall loss function is
given as:

$L_{LRSCN}=L_{saliency}+L_{trimap}.$ (3)

We do not use uncertainty loss because the main goal of LRSCN is to capture
sufficient semantics, not accurate boundary. More details about $L_{saliency}$
can be found in section 5 of supplementary materials.

###  3.5 Loss Function of HRRN

We perform a $L_{1}$ loss and novel uncertainty loss to restore the fine
structures and boundaries of salient objects. For an input high-resolution
image $I$, let $G^{H}$ denote its groundtruth, and predicted saliency map is
$S^{H}$.

We leverage the $L_{1}$ loss to compare an absolute difference between
predicted saliency map and groundtruth over the definite salient and
background regions:

$L_{1}=\frac{1}{E}\sum_{i\in E}|S^{H}_{i}-G^{H}_{i}|,$ (4)

where $E$ indicates the number of pixels which are labeled as definite salient
or background in the trimap, $S^{H}_{i}$ and $G^{H}_{i}$ denote the predicted
and groundtruth value at position $i$.

We cannot directly compute $L_{1}$ loss between predicted saliency map and
groundtruth over the uncertain regions because widely used saliency training
datasets have some problems in annotation quality [58]. We show these low
quality annotations in section 4 of supplementary materials. It is almost
impossible to directly obtain enough object boundary details from these
defective datasets to train the high-resolution network. To address this
problem, we design uncertainty loss, which empowers our HRRN to well address
the high-resolution refinement task only using these defective low-resolution
training datasets. It is worth noting that there are some previous works [60,
64] involving “uncertainty” in their titles, which seem relevant to our
method. However, in [60], “uncertainty” means the human perceptual uncertainty
modeled by CVAE. While in [64], “uncertainty” indicates the saliency
prediction system uncertainty modeled by R-dropout. Obviously, their usages of
uncertainty are different from ours.

Inspired by [22], Gaussian likelihood is used to model the uncertainty. Let
$x$ and $f(x)$ be the input and output of HRRN, and Gaussian likelihood is
defined as:

$p(y|f(x))=\mathcal{N}(f(x),\sigma^{2}),$ (5)

where $\sigma$ measures uncertainty of the estimation, $y$ is the label of
output. In maximum likelihood inference, we maximize the log likelihood of the
model, which is written as:

$logp(y|f(x))\propto-\frac{||y-f(x)||^{2}}{2\sigma^{2}}-\frac{1}{2}log\sigma^{2},$
(6)

so the proposed uncertainty loss is defined as:

$L_{uncertainty}=\frac{||y-f(x)||^{2}}{2\sigma^{2}}+\frac{1}{2}log\sigma^{2}.$
(7)

We only care about the pixels in uncertain regions, so $L_{uncertainty}$ is
written as:

$L_{uncertainty}=\frac{1}{U}\sum_{i\in
U}\frac{||S^{H}_{i}-G^{H}_{i}||^{2}}{2\sigma_{i}^{2}}+\frac{1}{2}log\sigma_{i}^{2},$
(8)

where $U$ is the total number of pixels in uncertain region, $\sigma_{i}$ is
the uncertainty of each pixel and is generated from HRRN. Different from
directly learning from noisy data, uncertainty loss can allow the network to
learn how to attenuate the effect from erroneous labels. Specifically, pixels
for which the network learned to predict high uncertainty will have a smaller
value of the first term of Eq.8, so have little effect on the loss. Meanwhile,
large uncertainty increases the contribution of the second term of Eq.8, and
in turn penalizes the model and lets the model make a better prediction that
has low uncertainty. Note that all parts of HRRN are trained jointly, so the
over all loss function is given as:

$L_{HRRN}=L_{uncertainty}+L_{1}.$ (9)

Figure 5: The impact of the losses. Best viewed by zooming in.

To show how proposed uncertainty loss makes the network attenuate the effect
from erroneous labels during training, we visualize the impact of $L_{1}$ loss
and uncertainty loss at the same training iteration in Fig.5. We show the
impact of both these losses in the same image. The images shown in Fig.5 are
these which have problems in annotation quality. Compared with column.4 and
column.5, if we only use $L_{1}$ loss, the weight of the loss in the
uncertainty region will be large, which leads the network hard to converge.
While uncertainty loss will make the weight of the loss in the uncertainty
region be small and let the network ignore effects from noisy data as much as
possible. Column.6 shows the uncertainty value of pixels in uncertain regions.
It can be seen that pixels in uncertain regions usually have higher
uncertainty value. In general, compared to $L_{1}$ loss, uncertainty loss
reduces the weight of the loss in the uncertainty region, thus mitigating the
impact of noisy data on the network. But due to the uncertainty value, it will
allow the network to learn how to predict a better prediction with a low
certainty value, instead of ignoring the learning of the uncertainty region
completely. These visual comparisons show how uncertainty loss makes network
more robust to noisy data.

Table 1: Quantitative comparison with SOTA on two high-resolution and three
low-resolution datasets. The best three results are in  red ,  green and  blue
fonts. ”$\dagger$” means the results are post-processed by dense conditional
random field(CRF) [24]. ”*” means using ResNeXt-101 [56] backbone. ”$\star$”
means using ResNet-101 backbone. ”${\ddagger}$” means using Res2Net50 [14]
backbone. MK: MSRA10K [7], DUTS: DUTS-TR [43], MB: MSRA-B [29], HR: HRSOD-
Training [58], HR-L: HRSOD-Training resized in low-resolution. Smaller MAE,
BDE and $B_{\mu}$, larger $F_{\beta}^{max}$, $F_{\beta}$ and $S_{m}$
correspond to better performance.

|  | HRSOD-TE | DAVIS-S | DUT-OMRON | DUTS-TE | HKU-IS
---|---|---|---|---|---|---
Models |  Training datasets | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE
VGG-16 backbone
Amulet(ICCV2017) | MK | 0.799 | 0.717 | 0.829 | 0.075 | 139.889 | 0.947 | 0.802 | 0.755 | 0.848 | 0.042 | 64.827 | 0.856 | 0.743 | 0.647 | 0.781 | 0.098 | 0.778 | 0.678 | 0.804 | 0.085 | 0.897 | 0.841 | 0.886 | 0.051
DGRL(CVPR2018) | DUTS | 0.821 | 0.789 | 0.847 | 0.055 | 95.034 | 0.889 | 0.803 | 0.772 | 0.859 | 0.038 | 50.323 | 0.826 | 0.774 | 0.709 | 0.810 | 0.063 | 0.828 | 0.794 | 0.842 | 0.050 | 0.910 | 0.881 | 0.896 | 0.037
DSS$\dagger$(TPAMI2019) | MB | 0.826 | 0.756 | 0.840 | 0.060 | 145.403 | 0.952 | 0.830 | 0.728 | 0.865 | 0.041 | 94.069 | 0.890 | 0.781 | 0.740 | 0.790 | 0.062 | 0.825 | 0.808 | 0.820 | 0.057 | 0.916 | 0.902 | 0.878 | 0.040
CPD(CVPR2019) | DUTS | 0.876 | 0.829 | 0.887 | 0.039 | 72.686 | 0.824 | 0.878 | 0.822 | 0.903 | 0.025 | 36.649 | 0.703 | 0.794 | 0.745 | 0.818 | 0.057 | 0.864 | 0.813 | 0.867 | 0.043 | 0.924 | 0.896 | 0.904 | 0.033
EGNet(ICCV2019) | DUTS | 0.883 | 0.814 | 0.888 | 0.044 | 73.500 | 0.896 | 0.886 | 0.794 | 0.897 | 0.030 | 37.369 | 0.799 | 0.803 | 0.744 | 0.813 | 0.057 | 0.877 | 0.800 | 0.866 | 0.044 | 0.927 | 0.893 | 0.910 | 0.035
MINet(CVPR2020) | DUTS | 0.902 | 0.851 | 0.903 | 0.032 | 76.291 | 0.849 | 0.915 | 0.864 | 0.926 | 0.019 | 32.304 | 0.742 | 0.794 | 0.741 | 0.822 | 0.057 | 0.877 | 0.823 | 0.875 | 0.039 | 0.930 | 0.904 | 0.912 | 0.031
ITSD(CVPR2020) | DUTS | 0.824 | 0.715 | 0.834 | 0.071 | 139.943 | 0.924 | 0.806 | 0.687 | 0.843 | 0.055 | 92.864 | 0.861 | 0.802 | 0.745 | 0.828 | 0.063 | 0.876 | 0.798 | 0.877 | 0.042 | 0.927 | 0.890 | 0.906 | 0.035
GateNet(ECCV2020) | DUTS | 0.905 | 0.825 | 0.906 | 0.035 | 79.468 | 0.886 | 0.914 | 0.825 | 0.923 | 0.023 | 44.827 | 0.778 | 0.794 | 0.723 | 0.821 | 0.061 | 0.870 | 0.783 | 0.870 | 0.045 | 0.929 | 0.889 | 0.910 | 0.036
HRNet(ICCV2019) | DUTS+HR | 0.905 | 0.888 | 0.897 | 0.030 | 88.017 | 0.888 | 0.899 | 0.888 | 0.876 | 0.026 | 44.359 | 0.801 | 0.743 | 0.690 | 0.762 | 0.065 | 0.835 | 0.788 | 0.824 | 0.050 | 0.910 | 0.886 | 0.877 | 0.042
Ours | DUTS | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536 | 0.804 | 0.769 | 0.829 | 0.053 | 0.882 | 0.855 | 0.879 | 0.036 | 0.935 | 0.918 | 0.913 | 0.029
Ours-DH | DUTS+HR-L | 0.921 | 0.907 | 0.917 | 0.024 | 45.462 | 0.706 | 0.938 | 0.926 | 0.936 | 0.014 | 14.412 | 0.531 | 0.795 | 0.764 | 0.820 | 0.052 | 0.894 | 0.865 | 0.879 | 0.035 | 0.933 | 0.914 | 0.904 | 0.031
ResNet-50/ResNet-101/ResNeXt-101/Res2Net50 backbone
R3Net*(IJCAI2018) | MK | 0.798 | 0.744 | 0.812 | 0.081 | 108.910 | 0.931 | 0.806 | 0.753 | 0.835 | 0.041 | 47.373 | 0.868 | 0.785 | 0.690 | 0.819 | 0.073 | 0.778 | 0.716 | 0.837 | 0.067 | 0.915 | 0.853 | 0.894 | 0.047
BASNet(CVPR2019) | DUTS | 0.878 | 0.831 | 0.890 | 0.038 | 67.643 | 0.823 | 0.857 | 0.806 | 0.881 | 0.039 | 46.283 | 0.705 | 0.805 | 0.766 | 0.838 | 0.056 | 0.859 | 0.791 | 0.866 | 0.048 | 0.928 | 0.895 | 0.909 | 0.032
PFPN$\star$(AAAI2020) | DUTS | 0.889 | 0.825 | 0.897 | 0.042 | 65.048 | 0.896 | 0.886 | 0.822 | 0.912 | 0.025 | 30.488 | 0.848 | 0.818 | 0.748 | 0.841 | 0.057 | 0.885 | 0.805 | 0.887 | 0.041 | 0.937 | 0.896 | 0.919 | 0.033
GCPA(AAAI2020) | DUTS | 0.889 | 0.827 | 0.894 | 0.039 | 70.320 | 0.873 | 0.912 | 0.833 | 0.924 | 0.021 | 24.132 | 0.759 | 0.812 | 0.748 | 0.838 | 0.056 | 0.888 | 0.817 | 0.891 | 0.038 | 0.938 | 0.898 | 0.920 | 0.031
F3N(AAAI2020) | DUTS | 0.900 | 0.853 | 0.897 | 0.035 | 65.901 | 0.817 | 0.915 | 0.845 | 0.913 | 0.020 | 45.106 | 0.719 | 0.813 | 0.766 | 0.838 | 0.053 | 0.891 | 0.840 | 0.888 | 0.035 | 0.937 | 0.910 | 0.917 | 0.028
LDF(CVPR2020) | DUTS | 0.905 | 0.866 | 0.905 | 0.032 | 58.655 | 0.812 | 0.911 | 0.864 | 0.922 | 0.019 | 35.496 | 0.713 | 0.817 | 0.773 | 0.839 | 0.052 | 0.894 | 0.855 | 0.890 | 0.034 | 0.939 | 0.914 | 0.919 | 0.028
CSF${\ddagger}$(ECCV2020) | DUTS | 0.894 | 0.832 | 0.900 | 0.038 | 71.293 | 0.922 | 0.899 | 0.822 | 0.912 | 0.025 | 30.488 | 0.848 | 0.815 | 0.750 | 0.838 | 0.055 | 0.894 | 0.823 | 0.890 | 0.038 | 0.935 | 0.902 | 0.921 | 0.030
Ours | DUTS | 0.915 | 0.902 | 0.919 | 0.024 | 47.804 | 0.750 | 0.935 | 0.923 | 0.937 | 0.013 | 14.396 | 0.576 | 0.818 | 0.785 | 0.842 | 0.051 | 0.895 | 0.870 | 0.892 | 0.033 | 0.943 | 0.928 | 0.923 | 0.025
Ours-DH | DUTS | 0.922 | 0.909 | 0.922 | 0.022 | 46.495 | 0.746 | 0.938 | 0.926 | 0.939 | 0.012 | 14.266 | 0.571 | 0.820 | 0.791 | 0.843 | 0.048 | 0.900 | 0.876 | 0.892 | 0.031 | 0.944 | 0.929 | 0.922 | 0.026

Figure 6: Comparison of PR curves across two high-resolution and three low-
resolution datasets. Figure 7: Visual comparison between our method and other
SOTA methods. Each sample occupies two rows. Best viewed by zooming in. It can
be clearly observed that our method achieves impressive performance in all
these cases.

##  4 Experiment

###  4.1 Experimental Settings

Implementation Details. Following the works [35, 66, 32, 53], we train our
proposed network on DUTS-TR. We use Pytorch111https://pytorch.org/ to
implement our model. A GTX 1080Ti GPU is used for acceleration. VGG-16 [38] is
used as the backbone network of LRSCN, and the whole network is trained end-
to-end by stochastic gradient descent (SGD). For a more comprehensive
demonstration, we also trained our network with ResNet-50 [17] backbone.
Maximum learning rate is set to 0.001 for backbone and 0.01 for other parts.
Warm-up and linear decay strategies are used to adjust the learning rate.
Momentum and weight decay are set to 0.9 and 0.0005 respectively. Batchsize is
set to 32 and maximum epoch is set to 100. Horizontal flip and multi-scale
input images are utilized for data augmentation as done in [35, 66, 52].
During testing, the input of LRSCN is about $352\times 352$ resolution.

The learning rate of HRRN is initialized to 0.0005. Warmup and cosine decay
are applied to the learning rate. The network HRRN is trained for 10000
iterations with a batch size of 20. During training, the resolution of input
image and trimap is $512\times 512$. During testing, we first resize the image
and trimap to $1024\times 1024$, then we split the image and trimap into four
sub-images and sub-trimaps with $512\times 512$ resolution, as shown in Fig.2.
Finally, we send each sub-image and sub-trimap together to HRRN to generate
sub-prediction result, and use 4 sub-predictions stitched together to make one
high-resolution saliency result.

Evaluation Datasets. Following work [58], we evaluate our method on two high-
resolution saliency detection datasets, including HRSOD-TE and DAVIS-S, which
contain 400 and 92 images. DAVIS-S dataset is collected from DAVIS [34].
Images in these two datasets are precisely annotated and have very high
resolutions (i.e.,$1920\times 1080$). We also evaluate our method on three
low-resolution datasets, including DUT-OMRON [57], DUTS-TE [43] and HKU-IS
[25], which contain 5168, 5019 and 4447 images. Our results are available at
https://github.com/luckybird1994/HQSOD.

Evaluation Metrics. Six metrics are used to evaluate the performance of our
method. The first is Mean Absolute Error (MAE), which characterize the average
$1$-norm distance between ground truth maps and predictions. The second is
F-measure ($F_{\beta}$ and $F_{\beta}^{max}$), a weighted mean of average
precision and average recall, calculated by
$F_{\beta}=\frac{(1+\beta^{2})\times Precision\times Recall}{\beta^{2}\times
Precision+Recall}$. We set $\beta^{2}$ to be 0.3 as suggested in [2]. The
third is Structure Measure ($S_{m}$), a metric to evaluate the spatial
structure similarities of saliency maps based on both region-aware structural
similarity $S_{r}$ and object-aware structural similarity $S_{o}$, defined as
$S_{\alpha}=\alpha\ast S_{r}+(1-\alpha)\ast S_{o}$, where $\alpha=0.5$ [11].
In addition, precision-recall (PR) curve is used to show the whole
performance. To further evaluate the boundary quality, Following [58] and
[62], we use Boundary Displacement Error (BDE) [12] and $B_{\mu}$ metrics.
More details about BDE and $B_{\mu}$ can be found in section 7 of
supplementary materials. The last two metrics are only used in two high-
resolution datasets, because their boundaries annotation is accurate and
evaluating results are reliable.

###  4.2 Comparisons with the State-of-the-Arts

We compare our approach with 16 SOTA methods, including Amulet [63], R3Net
[9], DGRL [47], DSS [18], BASNet [35], CPD [54], EGNet [66], PFPN [41], GCPA
[6], F3N [52], MINet [32], ITSD [69], LDF [53], GateNet [68], CSF [13] and
HRNet [58]. For a fair comparison, we use either the implementations with
recommended parameter settings or the saliency maps provided by the authors.
The evaluation toolbox used in this paper is same as F3N [52].

Quantitative Evaluation. From Table.1, when we train our network only using
DUTS (Ours), our method can already improve the $F_{\beta}^{max}$,$F_{\beta}$,
$S_{m}$ and MAE achieved by the best-performing existing algorithms,
especially two high-resolution test datasets. It is worth noting that for
boundary accuracy, our method is far better than other methods on two high-
resolution. These results demonstrate the efficiency of the proposed
disentangled SOD framework in both identifying the salient regions and
estimating the accurate objects boundaries. Other than numerical results, we
also show the PR curves on two high-resolution datasets and three low-
resolution datasets in Fig.6. As can be seen, the PR curves by our method (red
ones) are especially outstanding compared to all other previous methods.
Besides, shorter PR curves imply that our saliency maps are usually more
assertive with sharper boundaries than the results of other methods. An
interesting observation is that when we add HRSOD-training datasets (resized
to low-resolution like $352\times 352$) in LRSCN, the performance in two high-
resolution datasets HRSOD-TE and DAVIS-S can be further improved. However,
this practice seems to be of little help in improving the performance in other
three low-resolution datasets. A similar phenomenon can also be found in the
performance of HRNet [58]. We think there could be some image selection or
data annotation biases between the high-resolution datasets and the low-
resolution datasets, which cause this phenomenon.

Qualitative Evaluation. To exhibit the superiority of the proposed approach,
Fig.7 shows representative examples of saliency maps generated by our approach
and other state-of-the-art algorithms. As can be seen, with the help of LRSCN,
our method can not only keep the wholeness of the salient object (row 3), but
also accurately locate salient objects and suppress non-salient regions (row
5), compared to other methods. HRRN can help the model to restore accurate and
complete boundaries of salient objects, which are more consistent with the GT
boundaries. It can be clearly observed that our method achieves impressive
performance in all these cases, which indicates the eﬀectiveness of
disentangled framework and uncertainty loss. More comparison experiments can
be found in section 2 of supplementary materials.

Table 2: Ablation Studies of LRSCN.

|  | HRSOD-TE |  | DAVIS-S
---|---|---|---|---
Conﬁgurations | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
Baseline+HRRN | 0.900 | 0.880 | 0.896 | 0.034 | 65.732 | 0.842 | 0.899 | 0.887 | 0.919 | 0.021 | 31.201 | 0.678
Baseline+ME+HRRN | 0.910 | 0.894 | 0.902 | 0.031 | 60.040 | 0.801 | 0.920 | 0.904 | 0.924 | 0.019 | 24.022 | 0.612
Baseline+CF+HRRN | 0.909 | 0.892 | 0.900 | 0.030 | 59.028 | 0.804 | 0.918 | 0.905 | 0.923 | 0.018 | 23.988 | 0.605
Baseline+MECF+HRRN | 0.913 | 0.898 | 0.909 | 0.029 | 54.377 | 0.766 | 0.928 | 0.915 | 0.929 | 0.016 | 20.010 | 0.578
Baseline+MECF+SGA+HRRN | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536

Table 3: Ablation Studies of HRRN.

|  | HRSOD-TE | DAVIS-S
---|---|---|---
Ablation | Configurations | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
| Ours($L1+L_{uncertainy}$) | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536
| Ours($L1$) | 0.907 | 0.896 | 0.908 | 0.029 | 53.891 | 0.780 | 0.921 | 0.909 | 0.927 | 0.016 | 18.014 | 0.622
Loss | Ours-DH($L1+L_{uncertainy}$) | 0.918 | 0.901 | 0.911 | 0.027 | 48.878 | 0.712 | 0.933 | 0.920 | 0.934 | 0.015 | 17.670 | 0.540
| Ours(LRSCN) | 0.898 | 0.885 | 0.899 | 0.034 | 64.805 | 0.822 | 0.909 | 0.898 | 0.920 | 0.022 | 28.798 | 0.684
| Ours(LRSCN+HRRN) | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536
| Ours(LRSCN+CRF) | 0.905 | 0.896 | 0.897 | 0.029 | 60.521 | 0.797 | 0.920 | 0.907 | 0.918 | 0.018 | 24.455 | 0.665
| EGNet | 0.883 | 0.814 | 0.888 | 0.044 | 73.500 | 0.896 | 0.886 | 0.794 | 0.897 | 0.030 | 37.369 | 0.799
| EGNet(+HRRN) | 0.900 | 0.862 | 0.889 | 0.039 | 72.982 | 0.753 | 0.904 | 0.858 | 0.898 | 0.024 | 34.860 | 0.603
| EGNet(+CRF) | 0.895 | 0.858 | 0.882 | 0.039 | 73.348 | 0.796 | 0.902 | 0.846 | 0.892 | 0.025 | 34.721 | 0.679
| CPD | 0.876 | 0.829 | 0.887 | 0.039 | 72.686 | 0.824 | 0.878 | 0.822 | 0.903 | 0.025 | 36.649 | 0.703
| CPD(+HRRN) | 0.891 | 0.855 | 0.888 | 0.036 | 72.796 | 0.734 | 0.893 | 0.859 | 0.907 | 0.022 | 35.294 | 0.564
| CPD(+CRF) | 0.885 | 0.851 | 0.884 | 0.037 | 76.440 | 0.772 | 0.884 | 0.852 | 0.903 | 0.023 | 36.915 | 0.633
| BASNet(En-De) | 0.873 | 0.827 | 0.888 | 0.039 | 70.944 | 0.824 | 0.852 | 0.802 | 0.880 | 0.039 | 48.309 | 0.703
| BASNet(En-De+HRRN) | 0.895 | 0.858 | 0.892 | 0.036 | 67.191 | 0.719 | 0.875 | 0.832 | 0.882 | 0.036 | 45.922 | 0.566
| BASNet(En-De+RRM) | 0.878 | 0.831 | 0.890 | 0.038 | 67.643 | 0.823 | 0.857 | 0.806 | 0.881 | 0.039 | 46.283 | 0.705
Architecture | BASNet(En-De+CRF) | 0.888 | 0.850 | 0.886 | 0.036 | 77.235 | 0.758 | 0.867 | 0.825 | 0.878 | 0.037 | 46.423 | 0.629

Figure 8: Training the network with noisy data.

###  4.3 Ablation Studies

To validate the effectiveness of the proposed components of our method, we
conduct a series of experiments on two high-resolution datasets with different
settings under VGG-16 backbone. Specifically, we first verify the
effectiveness of MECF and SGA in LRSCN. Then we validate the effectiveness of
uncertainty loss in HRRN and the superiority of the proposed disentangled
architecture.

Ablation Studies of LRSCN. To prove the effectiveness of MECF and SGA module,
we report the quantitative comparison results of LRSCN with different
architectures in Table.2. Baseline denotes that we conduct a experiment over
on $L_{LRSCN}$ with a pure U-Net architecture. We can see that only using ME
or CF can already heavily improve the performance. A better performance has
been achieved through the combination of these two architectures. Finally,
performace can be further improved by SGA module, especially BDE and
$B_{\mu}$, which means that SGA can help generate accurate trimap. While
$L_{saliency}$ is not our core innovation, more ablation studies about
$L_{saliency}$ can be found in section 5 of supplementary materials.

Ablation Studies of HRRN. In HRRN, uncertainty loss play a key role to
estimate the accurate objects boundaries, so we first investigate the
eﬀectiveness of our proposed uncertainty loss. From Table.3, we can see that
without uncertainty loss (Ours($L_{1}$)), the performance decreased a lot.
Besides, when we add high-resolution HRSOD-Training datasets in HRRN (Ours-
DH($L_{1}+L_{uncertainty}$)), the performance has no obvious improvement,
which demonstrates that our network is not reliant on accurately annotated
high-resolution images during training. To further demonstrate the
effectiveness of our disentangled framework, we compare our HRRN with CRF
[24], a widely used post-processing for saliency detection. Results in Table.3
show that our proposed method (Ours (LRSCN + HRRN)) outperforms CRF (Ours
(LRSCN + CRF)) by a large margin. The same phenomenon can be found in the
refinement of EGNet, CPD and BASNet (Their trimaps are generated with the
corresponding saliency maps following Eq.1). Moreover, compared to RRM module
proposed in BASNet, our HRRN can better improve performance. This ablation
study demonstrates the superiority of HRRN within our novel disentangled
framework. More analyses of the proposed disentangled framework can be found
in section 3 of supplementary materials.

Analysis of Uncertainty Loss To further demonstrate that uncertainty loss can
make network more robust to noisy data, we erase and dilate binary groundtruth
maps of DUT-TR at the object boundaries with a random pixel number
(3,4,5,6,7,9,11,13) to generate noisy training data. Then we train the network
on these noisy data, BDE and $B_{\mu}$ results on HRSOD-TE are reported in
Fig.8(a) and Fig.8(b). When the erosion or dilation kernel ranges from 3 to 7,
the network trained with uncertainty loss has a rather stable performance.
With the increase in erosion or dilation kernels, even though the performance
is dropped, training with uncertainty loss still yields a better model than
training without uncertainty loss. This experiment further validates the
effectiveness of uncertainty loss.

##  5 Conclusions

In this paper, we argue that there are two difficult and inherently different
problems in high-resolution SOD. From this perspective, we propose a novel
deep learning framework to disentangle the high-resolution SOD into two tasks:
LRSCN and HRRN. LRSCN can identify the definite salient, background and
uncertain regions at low-resolution with sufficient semantics. While HRRN can
accurately refining the saliency value of pixels in the uncertain region to
preserve a clear object boundary at high-resolution with limited GPU memory.
We also make the earliest efforts to introduce the uncertainty into SOD
network training, which empower HRRN to learn rich details without using any
high-resolution training datasets. Extensive evaluations on high-resolution
datasets and popular benchmark datasets not only verify the superiority of our
method but also demonstrate the importance of disentanglement for SOD. We
believe our novel disentanglement view in this work can contribute to other
high-resolution computer vision tasks in the future.

##  Appendix A More Quantitative and Qualitative Results

###  A.1 Quantitative Comparison on more datasets

We compare our method with other SOTA methods on another two conventional low-
resolution datasets ECSSD [37] and PASCAL-S [26], which have 1000 and 850
images respectively. The results are reported in Table.4. It can be seen that
our method consistently outperforms other methods across these two
conventional datasets. We also show their PR curves in Fig.9. It should be
noted that $F_{max}$ represents $F^{max}_{\beta}$. We apologize for this
writing error of Table.2 in the main text.

F-measure curves of different methods are displayed in Fig.10, for overall
comparisons. One can observe that our approach noticeably outperforms all the
other state-of-the-art methods. These observations demonstrate the efficiency
and robustness of our proposed method across various challenging datasets.

SOC [10] is a new challenging dataset with nine attributes. In Table.5, we
evaluate the mean F-measure score of our method as well as 11 state-of-the-art
methods. We can see the proposed model achieves the competitive results among
most of attributes and the overall score is best.

Model size and running time comparisons among different methods are also
reported in Table.6. It can be seen that with the high-resolution input, our
method is more efficient than HRNet. For fair, the running time analysis of
our method is also conducted with the low-resolution input ($352\times 352$),
and our method runs at a competitive efficiency.

Table 4: Quantitative comparison with SOTA methods on another two conventional
datasets.

|  | ECSSD | PASCAL-S
---|---|---|---
Models |  Training datasets | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE
VGG-16 backbone
Amulet(ICCV2017) | MK | 0.915 | 0.868 | 0.894 | 0.059 | 0.828 | 0.757 | 0.818 | 0.100
DGRL(CVPR2018) | DUTS | 0.922 | 0.903 | 0.906 | 0.043 | 0.849 | 0.807 | 0.834 | 0.074
DSS(TPAMI2019) | MB | 0.921 | 0.904 | 0.882 | 0.052 | 0.831 | 0.802 | 0.798 | 0.094
CPD(CVPR2019) | DUTS | 0.936 | 0.917 | 0.917 | 0.037 | 0.861 | 0.824 | 0.842 | 0.072
EGNET(ICCV2019) | DUTS | 0.943 | 0.913 | 0.913 | 0.041 | 0.858 | 0.809 | 0.848 | 0.077
MINet(CVPR2020) | DUTS | 0.943 | 0.922 | 0.917 | 0.036 | 0.865 | 0.829 | 0.854 | 0.064
ITSD(CVPR2020) | DUTS | 0.939 | 0.875 | 0.914 | 0.040 | 0.869 | 0.773 | 0.853 | 0.068
GateNet(ECCV2020) | DUTS | 0.941 | 0.896 | 0.917 | 0.041 | 0.870 | 0.797 | 0.853 | 0.068
HRNet(ICCV2019) | DUTS+HR | 0.925 | 0.905 | 0.888 | 0.052 | 0.846 | 0.804 | 0.817 | 0.079
Ours | DUTS | 0.948 | 0.931 | 0.918 | 0.034 | 0.874 | 0.845 | 0.854 | 0.063
Ours-DH | DUTS+HR-L | 0.938 | 0.918 | 0.904 | 0.040 | 0.871 | 0.845 | 0.851 | 0.061
ResNet-50/ResNet-101/ResNeXt-101/Res2Net50 backbone
R3Net(IJCAI2018) | MK | 0.934 | 0.883 | 0.910 | 0.051 | 0.834 | 0.775 | 0.809 | 0.101
BasNet(CVPR2019) | DUTS | 0.942 | 0.880 | 0.916 | 0.037 | 0.854 | 0.775 | 0.832 | 0.076
PFPN(AAAI2020) | DUTS | 0.947 | 0.917 | 0.927 | 0.035 | 0.870 | 0.824 | 0.851 | 0.065
GCPA(AAAI2020) | DUTS | 0.948 | 0.919 | 0.927 | 0.035 | 0.869 | 0.827 | 0.860 | 0.062
F3N(AAAI2020) | DUTS | 0.945 | 0.925 | 0.924 | 0.036 | 0.872 | 0.840 | 0.855 | 0.062
LDF(CVPR2020) | DUTS | 0.950 | 0.930 | 0.924 | 0.034 | 0.874 | 0.843 | 0.859 | 0.061
CSF(ECCV2020) | DUTS | 0.950 | 0.925 | 0.927 | 0.033 | 0.874 | 0.823 | 0.858 | 0.069
Ours | DUTS | 0.952 | 0.941 | 0.928 | 0.029 | 0.880 | 0.852 | 0.861 | 0.059
Ours-DH | DUTS+HR-L | 0.953 | 0.941 | 0.926 | 0.030 | 0.878 | 0.852 | 0.859 | 0.060

Figure 9: Comparison of PR curves across another two conventional low-
resolution datasets. Figure 10: Comparison of the F-measure curves across on
two high-resolution and five low-resolution datasets. Table 5: Performance on
SOC of different attributes. The last row shows the whole performance on the
SOC dataset.

Attr | BASNet | CPD | EGNet | F3N | GCPA | PFPN | ITSD | LDF | MINet | CSF | GateNet | Ours | Ours-DH
---|---|---|---|---|---|---|---|---|---|---|---|---|---
AC | 0.723 | 0.750 | 0.756 | 0.784 | 0.780 | 0.772 | 0.611 | 0.796 | 0.790 | 0.730 | 0.748 | 0.793 | 0.788
BO | 0.511 | 0.794 | 0.702 | 0.791 | 0.882 | 0.837 | 0.499 | 0.807 | 0.814 | 0.825 | 0.737 | 0.858 | 0.848
CL | 0.682 | 0.771 | 0.726 | 0.757 | 0.765 | 0.765 | 0.610 | 0.763 | 0.770 | 0.751 | 0.754 | 0.789 | 0.789
HO | 0.772 | 0.777 | 0.756 | 0.790 | 0.780 | 0.777 | 0.685 | 0.797 | 0.792 | 0.779 | 0.788 | 0.817 | 0.817
MB | 0.687 | 0.715 | 0.687 | 0.761 | 0.691 | 0.705 | 0.589 | 0.758 | 0.708 | 0.702 | 0.725 | 0.764 | 0.768
OC | 0.686 | 0.719 | 0.702 | 0.724 | 0.720 | 0.729 | 0.629 | 0.739 | 0.729 | 0.703 | 0.728 | 0.771 | 0.771
OV | 0.720 | 0.764 | 0.764 | 0.793 | 0.802 | 0.806 | 0.639 | 0.805 | 0.788 | 0.772 | 0.787 | 0.798 | 0.802
SC | 0.708 | 0.723 | 0.683 | 0.747 | 0.707 | 0.697 | 0.592 | 0.746 | 0.726 | 0.690 | 0.715 | 0.785 | 0.782
SO | 0.632 | 0.643 | 0.614 | 0.668 | 0.640 | 0.636 | 0.523 | 0.691 | 0.652 | 0.621 | 0.641 | 0.713 | 0.713
Avg | 0.680 | 0.740 | 0.710 | 0.757 | 0.752 | 0.747 | 0.597 | 0.767 | 0.753 | 0.730 | 0.736 | 0.788 | 0.787

Table 6: Model size and running time comparisons between our approach and SOTA
methods.

| Ours | Ours | DGRL | DSS | BASNet | EGNet | GCPA | PFPN | R3Net
---|---|---|---|---|---|---|---|---|---
Model Size(MB) | 309.6 | 309.6 | 648 | 447.3 | 412.2 | 332.1 | 255.8 | 243.0 | 214.2
Time(s) | 0.21 | 0.05 | 0.52 | 5.12 | 0.04 | 0.15 | 0.02 | 0.05 | 0.27
Size | $1024\times 1024$ | $352\times 352$ | $384\times 384$ | $224\times 224$ | $256\times 256$ | $400\times 300$ | $320\times 320$ | $256\times 256$ | $256\times 256$
| HRNet | MINet | CSF | Amulet | CPD | F3N | LDF | ITSD | GateNet
Model Size(MB) | 129.6 | 181.4 | 139.3 | 132.6 | 111.5 | 97.4 | 95.9 | 63.7 | -
Time(s) | 0.39 | 0.01 | 0.01 | 0.05 | 0.02 | 0.03 | 0.02 | 0.02 | 0.03
Size | $1024\times 1024$ | $320\times 320$ | $224\times 224$ | $256\times 256$ | $352\times 352$ | $352\times 352$ | $352\times 352$ | $288\times 288$ | $384\times 384$

Table 7: Quantitative comparison with SOTA methods where the inputs are
resized to high-resolution.

| HRSOD-TE | DAVIS-S
---|---|---
Models | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
CPD(High-Resolution) | 0.868 | 0.735 | 0.809 | 0.073 | 181.770 | 0.819 | 0.720 | 0.679 | 0.799 | 0.062 | 126.281 | 0.748
CPD(Low-Resolution) | 0.876 | 0.829 | 0.887 | 0.039 | 72.686 | 0.824 | 0.878 | 0.822 | 0.903 | 0.025 | 36.649 | 0.703
EGNet(High-Resolution) | 0.745 | 0.693 | 0.791 | 0.082 | 213.333 | 0.867 | 0.692 | 0.644 | 0.801 | 0.069 | 149.537 | 0.821
EGNet(Low-Resolution) | 0.883 | 0.814 | 0.888 | 0.044 | 73.500 | 0.896 | 0.886 | 0.794 | 0.897 | 0.030 | 37.369 | 0.799
F3N(High-Resolution) | 0.834 | 0.757 | 0.825 | 0.066 | 187.942 | 0.798 | 0.698 | 0.712 | 0.826 | 0.054 | 130.603 | 0.716
F3N(Low-Resolution) | 0.900 | 0.853 | 0.897 | 0.035 | 65.901 | 0.817 | 0.915 | 0.845 | 0.913 | 0.020 | 45.106 | 0.719
GCPA(High-Resolution) | 0.810 | 0.771 | 0.830 | 0.066 | 164.142 | 0.793 | 0.750 | 0.714 | 0.829 | 0.057 | 122.068 | 0.708
GCPA(Low-Resolution) | 0.889 | 0.827 | 0.894 | 0.039 | 70.320 | 0.873 | 0.912 | 0.833 | 0.924 | 0.021 | 24.132 | 0.759
MINet(High-Resolution) | 0.687 | 0.629 | 0.742 | 0.111 | 250.149 | 0.913 | 0.580 | 0.508 | 0.681 | 0.129 | 176.671 | 0.888
MINet(Low-Resolution) | 0.902 | 0.851 | 0.903 | 0.032 | 76.291 | 0.849 | 0.915 | 0.864 | 0.926 | 0.019 | 32.304 | 0.742
LDF(High-Resolution) | 0.650 | 0.586 | 0.673 | 0.133 | 208.545 | 0.898 | 0.590 | 0.553 | 0.696 | 0.101 | 150.540 | 0.844
LDF(Low-Resolution) | 0.905 | 0.866 | 0.905 | 0.032 | 58.655 | 0.812 | 0.911 | 0.864 | 0.922 | 0.019 | 35.496 | 0.713
CSF(High-Resolution) | 0.802 | 0.756 | 0.843 | 0.063 | 181.705 | 0.873 | 0.700 | 0.685 | 0.824 | 0.058 | 137.592 | 0.816
CSF(Low-Resolution) | 0.894 | 0.832 | 0.900 | 0.038 | 71.293 | 0.922 | 0.899 | 0.822 | 0.912 | 0.025 | 30.488 | 0.848
Ours | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536

Table 8: Quantitative comparison with SOTA methods which are finetuned on
HRSOD-Training dataset.

| HRSOD-TE | DAVIS-S
---|---|---
Models | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
BASNet(finetune) | 0.885 | 0.836 | 0.904 | 0.035 | 64.475 | 0.813 | 0.866 | 0.838 | 0.911 | 0.023 | 25.924 | 0.659
BASNet(original) | 0.878 | 0.831 | 0.890 | 0.038 | 67.643 | 0.823 | 0.857 | 0.806 | 0.881 | 0.039 | 46.283 | 0.705
CPD(finetune) | 0.890 | 0.846 | 0.899 | 0.035 | 80.857 | 0.783 | 0.890 | 0.871 | 0.925 | 0.020 | 29.376 | 0.671
CPD(original) | 0.876 | 0.829 | 0.887 | 0.039 | 72.686 | 0.824 | 0.878 | 0.822 | 0.903 | 0.025 | 36.649 | 0.703
EGNet(finetune) | 0.890 | 0.857 | 0.911 | 0.031 | 69.084 | 0.797 | 0.899 | 0.881 | 0.926 | 0.021 | 30.674 | 0.686
EGNet(original) | 0.883 | 0.814 | 0.888 | 0.044 | 73.500 | 0.896 | 0.886 | 0.794 | 0.897 | 0.030 | 37.369 | 0.799
GCPA(finetune) | 0.895 | 0.837 | 0.912 | 0.032 | 64.656 | 0.846 | 0.918 | 0.857 | 0.927 | 0.019 | 22.312 | 0.746
GCPA(original) | 0.889 | 0.827 | 0.894 | 0.039 | 70.320 | 0.873 | 0.912 | 0.833 | 0.924 | 0.021 | 24.132 | 0.759
F3N(finetune) | 0.905 | 0.865 | 0.909 | 0.033 | 60.803 | 0.787 | 0.920 | 0.860 | 0.921 | 0.019 | 29.106 | 0.661
F3N(original) | 0.900 | 0.853 | 0.897 | 0.035 | 65.901 | 0.817 | 0.915 | 0.845 | 0.913 | 0.020 | 45.106 | 0.719
PFPN(finetune) | 0.896 | 0.840 | 0.904 | 0.038 | 55.027 | 0.786 | 0.901 | 0.845 | 0.920 | 0.022 | 21.388 | 0.728
PFPN(original) | 0.889 | 0.825 | 0.897 | 0.042 | 65.048 | 0.897 | 0.886 | 0.822 | 0.912 | 0.025 | 30.488 | 0.848
ITSD(finetune) | 0.834 | 0.774 | 0.863 | 0.052 | 117.554 | 0.906 | 0.820 | 0.754 | 0.873 | 0.041 | 75.461 | 0.830
ITSD(original) | 0.824 | 0.715 | 0.834 | 0.071 | 139.943 | 0.924 | 0.806 | 0.687 | 0.843 | 0.055 | 92.864 | 0.861
MINet(finetune) | 0.908 | 0.871 | 0.908 | 0.029 | 66.089 | 0.749 | 0.923 | 0.879 | 0.928 | 0.017 | 25.408 | 0.692
MINet(original) | 0.902 | 0.851 | 0.903 | 0.032 | 76.291 | 0.849 | 0.915 | 0.864 | 0.926 | 0.019 | 32.304 | 0.742
LDF(finetune) | 0.910 | 0.862 | 0.910 | 0.031 | 77.098 | 0.812 | 0.920 | 0.867 | 0.922 | 0.018 | 42.226 | 0.727
LDF(original) | 0.905 | 0.866 | 0.905 | 0.032 | 58.655 | 0.812 | 0.911 | 0.864 | 0.922 | 0.019 | 35.496 | 0.713
GateNet(finetune) | 0.910 | 0.856 | 0.909 | 0.029 | 76.434 | 0.821 | 0.923 | 0.872 | 0.930 | 0.019 | 36.984 | 0.706
GateNet(original) | 0.905 | 0.825 | 0.906 | 0.035 | 79.468 | 0.886 | 0.914 | 0.825 | 0.923 | 0.023 | 44.827 | 0.778
CSF(finetune) | 0.902 | 0.859 | 0.909 | 0.029 | 56.425 | 0.884 | 0.910 | 0.870 | 0.931 | 0.017 | 24.669 | 0.791
CSF(original) | 0.894 | 0.832 | 0.900 | 0.038 | 71.293 | 0.922 | 0.899 | 0.822 | 0.912 | 0.025 | 30.488 | 0.848
Ours | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536

###  A.2 Quantitative Comparison with different settings

Although the effectiveness of our method has been confirmed by existing
quantitative comparison experiments, to further illustrate the superiority of
our method in handling high-resolution SOD task, we modify the setting of
existing methods to allow for a more comprehensive comparison.

First, we change the input for the current SOTA methods from low-resolution
(e.g., typical size $320\times 320$, $352\times 352$) to high-resolution
($1024\times 1024$). The results are reported in Table.7. It can be found that
all the compared SOTA methods perform better at low-resolution on most
evaluation metrics. Therefore, we only compare our methods to these SOTA
methods’ low-resolution results in our main paper. In particular, it is worth
pointing out that due to GPU memory limitations, we cannot run BASNet, PFPN
and ITSD at high-resolution. So we don’t report their results in Table.7.

Then, we fine-tune 11 SOTA methods on high-resolution datasets (HRSOD-
Training) which have high quality annotations, the results are reported on
Table.8. As can be seen, high annotation quality can improve their original
performance. However, even fine-tuned on HRSOD-Training datasets, our method
(only trained on DUTS) still outperforms all of them by a large margin.

Figure 11: Visual comparison between our method and other SOTA methods. Each
sample occupies two rows. Best viewed by zooming in. It can be clearly
observed that our method achieves impressive performance in all these cases.

###  A.3 Qualitative Comparison

As shown in Fig.11, we provide a comprehensive qualitative comparison of our
method with other 12 methods on challenging cases. These visual examples can
further demonstrate that our method is able to restore accurate and complete
boundaries of salient objects.

Figure 12: Examples of coarse saliency maps, trimaps and refined saliency map.

##  Appendix B More analyses of the proposed disentangled framework

Table 9: Ablation Studies of disentangled framework.

|  | HRSOD-TE |  |  |  | DAIVS-S |  |
---|---|---|---|---|---|---|---|---
Conﬁgurations | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
Regression-Regression | 0.912 | 0.894 | 0.899 | 0.031 | 56.251 | 0.814 | 0.923 | 0.909 | 0.918 | 0.019 | 22.737 | 0.649
Classification-Classification | 0.913 | 0.895 | 0.898 | 0.030 | 54.143 | 0.809 | 0.921 | 0.907 | 0.921 | 0.020 | 23.892 | 0.662
Ours | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536

As described, high-resolution salient object detection task should be
disentangled into two tasks. One can be viewed as a classic classification
task, while the other one is a typical regression task. To further illustrate
the validity of our theory, we conduct additional experiments. Specifically,
we consider these two tasks as regression or classification tasks
simultaneously. The results are reported in Table.9. Compared with our
proposed method, if we take the disentangled framework as the combination of
the two regression or classification tasks, the performance will be degraded.
Because the purpose of the proposed disentangled framework is to capture
sufficient semantics at low-resolution (LRSCN Stage) and refine accurate
boundary at high-resolution (HRRN Stage), which should be viewed as a classic
classification task and a typical regression task. Fig.12 shows some examples
that our proposed HRRN can further refine accurate boundary, guided by
trimaps. Specifically, column.3 and column.4 show the saliency maps and
trimaps generated by LRSCN, and column.5 shows the results refined by HRRN.
From Fig.12, guided by trimaps, our proposed HRRN can further refine the
pixels value in uncertain regions to get more clear saliency results.

Aforementioned work LDF [53] has also introduced concepts related to
decoupling. However, they still try to address the SOD task under a single
regression framework. Their approach is essentially an expansion of additional
boundary supervision, which barely touches the very nature of the SOD. As
illustrated in our experiments, it is more natural to disentangle the SOD into
two different tasks.

##  Appendix C Annotation Problems

Figure 13: Examples that have annotation quality problem. Best viewed by
zooming in. Table 10: Ablation Studies of $L_{saliency}$.

|  | HRSOD-TE |  |  |  | DAIVS-S |  |
---|---|---|---|---|---|---|---|---
Conﬁgurations | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$ | $F_{\beta}^{max}$ | $F_{\beta}$ | $S_{m}$ | MAE | BDE | $B_{\mu}$
LRSCN($L_{trimap}$)+HRRN | 0.895 | 0.870 | 0.883 | 0.035 | 75.732 | 0.879 | 0.900 | 0.880 | 0.890 | 0.026 | 41.221 | 0.733
LRSCN($L_{P}+L_{trimap}$)+HRRN | 0.912 | 0.898 | 0.908 | 0.029 | 53.040 | 0.764 | 0.925 | 0.910 | 0.926 | 0.018 | 19.022 | 0.569
LRSCN($L_{P}+L_{R}+L_{trimap}$)+HRRN | 0.917 | 0.900 | 0.910 | 0.029 | 52.048 | 0.743 | 0.932 | 0.914 | 0.930 | 0.017 | 17.688 | 0.552
LRSCN($L_{P}+L_{R}+L_{O}+L_{trimap}$)+HRRN | 0.918 | 0.902 | 0.912 | 0.027 | 48.468 | 0.711 | 0.933 | 0.919 | 0.933 | 0.015 | 15.676 | 0.536

As described in [58], widely used saliency datasets have some problems in
annotation quality. So, to quantify the annotation quality problem, we
randomly select 100 images from DUT-TR, and 10 of them have easily spotted
annotation errors. We manually relabel the 10 images. The $B_{\mu}$ between
the two different annotations is 0.49 and 42% of the boundary pixel
annotations are inaccurate. Fig.13 shows some examples which have annotation
problems, including wrong semantic annotation (row 1 and row 2), boundary
annotation shifting (row 3) and low contour accuracy (row4, row5 and row 6).
In conclusion, the DUTS-TR training dataset does have annotation problems
[58], and we relabeled some examples to demonstrate these problems in the
supplemental material. Since correcting annotations for the whole DUT-TR is a
time-consuming task, we will provide an accurate GT of DUT-TR in the future
for statistical analysis

##  Appendix D Details of $L_{saliency}$

As described, to guarantee the arruracy of trimap, we add extra saliency
supervision $L_{saliency}$ as the supplement of trimap supervision. Here we
give more details about $L_{saliency}$.

After LRSCN, the prediction saliency map is $S$, and the binary groundtruth is
$G$. In SOD, binary cross entropy (BCE) is the most widely used loss function,
and it is a pixel-wise loss which is defined as:

$\displaystyle L_{Pixel}=-(Glog(S)+(1-G)log(1-S)).$ (10)

To learn the structural information of the salient objects, following the
setting of [51, 11], we use the sliding window fashion to model region-level
similarity between groundtruth and saliency map. The corresponding regions are
denoted as $S_{i}=\\{S_{i}:i=1,...M\\}$ and $G_{i}=\\{G_{i}:i=1,...M\\}$,
where $M$ is the total number of region. Then we use SSIM to evaluate the
similarity between $S_{i}$ and $G_{i}$, which is defined as:

$SSD_{i}=\frac{(2\mu_{s}\mu_{g}+C_{1})(2\sigma_{sg}+C_{2})}{(\mu_{s}^{2}+\mu_{g}^{2}+C_{1})(\sigma_{s}^{2}+\sigma_{g}^{2}+C_{2})}$
(11)

where local statistics $\mu_{s}$, $\sigma_{s}$ is mean and std vector of
$S_{i}$, $\mu_{g}$, $\sigma_{g}$ is mean and std vector of $G_{i}$. The
overall loss function is defined as:

$L_{Region}=1-\frac{1}{M}\sum_{i=1}^{M}SSD_{i}.$ (12)

Finally, inspired by [67], we directly optimize the F-measure to learn the
global information from groundtruth. For easy remembering, we denote F-measure
as $F_{\beta}$ in the following. $F_{\beta}$ is defined as:

$precision=\frac{\sum{S\cdot G}}{\sum{S}+\epsilon},\ \
recall=\frac{\sum{S\cdot G}}{\sum{G}+\epsilon},$ (13)
$F_{\beta}=\frac{(1+\beta^{2})\cdot precision\cdot recall}{\beta^{2}\cdot
precision+recall},$ (14)

where $\cdot$ means pixel-wise multiplication, $\epsilon=1e^{-7}$ is a
regularization constant to avoid division of zero. $L_{Object}$ loss function
is defined as:

$L_{Object}=1-F_{\beta}.$ (15)

The whole loss is defined as:

$L=L_{Object}+L_{Region}+L_{Pixel}.$ (16)

Besides, following [35, 52], we used multi-levels saliency supervision to
facilitate sufficient training, so the whole saliency loss is defined as:

$L_{saliency}=\sum_{i=1}^{4}\frac{1}{2^{i-1}}L_{i},$ (17)

where $i$ means the i-th level.

To further validate the role of $L_{saliency}$, we train the LRSCN with
different loss functions and the results are reported on Table.10. As can be
can, without $L_{saliency}$, the performance is dropped lot. Because the
trimap groundtruth is randomly generated from binary groundtruth, so only
using $L_{trimap}$ cannot maintain consistency between trimap and saliency
map. When we only add $L_{P}$ on multi-levels, the model can already achieve
the largest performance boost. A better performance has been achieved through
the combination of $L_{P}$, $L_{R}$ and $L_{O}$.

Figure 14: Architecture of MECF Module.

##  Appendix E Details of MECF Module

As described, we develop a multi-scale feature extraction module (ME) and
cross-level feature fusion module (CF) to help LRSCN capture sufficient
semantics at low-resolution. Here we give more details about MECF module. The
architecture of MECF Module is shown in Fig.14.

Multi-scale feature extraction module can allow each spatial location to view
the local context at small scale spaces and capture multi-scale contextual
information, which can enlarge the feature $F^{1}_{l}$ receptive field.
Specifically, we first use an average pooling and a $3\times 3$ convolutional
layer to downsample $F^{1}_{l}$. Then upsampled feature from small scale is
added with $F^{1}_{l}$. Finally, Global Convolutional Network (GCN) [33] is
used to further enlarge the feature receptive field. Because $F_{3}^{1}$ and
$F_{4}^{1}$ are close to the input and receptive field is relatively small, we
use GCNs with $k=7,11,15$ to fully enlarge receptive field. Receptive fields
of $F_{5}^{1}$ and $F_{6}^{1}$ are relatively bigger, we only use GCNs with
$k=7,11$ and $k=7$.

Low-level features have rich details but full of background noises, so we
design cross-level feature fusion module, which can leverage the rich
semantics of high-level feature $F^{2}_{h}$ and help restrain the non-salient
regions in low-level features. Specifically, we first use a $1\times 1$
convolutional layer to compress the channels of $F^{2}_{l}$, then use two
$3\times 3$ convolutional layer to transfer the feature for SOD task. Finally,
the transferred feature is fused with high-level feature $F^{2}_{h}$ as the
output of this module. Each of these convolution layers is followed by a batch
normalization [19] and a ReLU activation [15].

##  Appendix F Formulas of Evaluation Metrics

Following [58] and [62], we use Boundary Displacement Error(BDE) [12] and
$B_{\mu}$ metrics to evaluate the boundary quality.

BDE measures the average displacement error of boundary pixels between two
predictions, which can be formulated as:

$BDE(X,Y)=\frac{\sum_{x}inf_{y\in Y}d(x,y)}{2N_{X}}+\frac{\sum_{y}inf_{x\in
X}d(x,y)}{2N_{Y}},$ (18)

where $X$ and $Y$ are two boundary pixel sets which represent saliency
prediction and their corresponding groundtruth, and $x$, $y$ are pixels in
them. $N_{x}$ and $N_{y}$ denote the number pixels in $X$ and $Y$. $inf$
represents for the infimum and $d(\cdot)$ denotes Euclidean distance.

$B_{\mu}$ evaluates the structure alignment between saliency map and their
groundtruth, it can be expressed as:

$B_{\mu}=1-\frac{2\sum(g_{s}g_{y})}{\sum(g^{2}_{s}+g^{2}_{y})},$ (19)

where $g_{s}$ and $g_{y}$ represent the binarized edge maps of predicted
saliency map and groundtruth. Following [62], we use Canny edge detector to
compute edge maps. $B_{\mu}$ reﬂect the sharpness of predictions which is
consistent with human perception.

## References

  * [1] Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and Jia Li.  Salient object detection: A survey.  Computational Visual Media, 5(2):117–150, 2019.
  * [2] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li.  Salient object detection: A benchmark.  IEEE TIP, 24(12):5706–5722, 2015.
  * [3] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In ICLR, 2019.
  * [4] Shaofan Cai, Xiaoshuai Zhang, Haoqiang Fan, Haibin Huang, Jiangyu Liu, Jiaming Liu, Jiaying Liu, Jue Wang, and Jian Sun.  Disentangled image matting.  In ICCV, pages 8818–8827. IEEE, 2019.
  * [5] Shuhan Chen, Xiuli Tan, Ben Wang, and Xuelong Hu.  Reverse attention for salient object detection.  In ECCV(2018), volume 11213 of Lecture Notes in Computer Science, pages 236–252. Springer, 2018.
  * [6] Zuyao Chen, Qianqian Xu, Runmin Cong, and Qingming Huang.  Global context-aware progressive aggregation network for salient object detection.  In AAAI, pages 10599–10606. AAAI Press, 2020.
  * [7] Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu.  Global contrast based salient region detection.  IEEE TPAMI, 37(3):569–582, 2015.
  * [8] Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein.  A tutorial on the cross-entropy method.  Ann. Oper. Res., 134(1):19–67, 2005.
  * [9] Zijun Deng, Xiaowei Hu, Lei Zhu, Xuemiao Xu, Jing Qin, Guoqiang Han, and Pheng-Ann Heng.  R3net: Recurrent residual refinement network for saliency detection.  In IJCAI, pages 684–690. ijcai.org, 2018.
  * [10] Deng-Ping Fan, Ming-Ming Cheng, Jiangjiang Liu, Shanghua Gao, Qibin Hou, and Ali Borji.  Salient objects in clutter: Bringing salient object detection to the foreground.  In ECCV (15), volume 11219 of Lecture Notes in Computer Science, pages 196–212. Springer, 2018.
  * [11] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji.  Structure-measure: A new way to evaluate foreground maps.  In ICCV, pages 4558–4567(2017). IEEE, 2017.
  * [12] Jordi Freixenet, Xavier Muñoz, David Raba, Joan Martí, and Xavier Cufí.  Yet another survey on image segmentation: Region and boundary information integration.  In ECCV (3), volume 2352 of Lecture Notes in Computer Science, pages 408–422. Springer, 2002.
  * [13] Shanghua Gao, Yong-Qiang Tan, Ming-Ming Cheng, Chengze Lu, Yunpeng Chen, and Shuicheng Yan.  Highly efficient salient object detection with 100k parameters.  CoRR, abs/2003.05643, 2020.
  * [14] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr.  Res2net: A new multi-scale backbone architecture.  IEEE TPAMI, 2020.
  * [15] Richard H. R. Hahnloser and H. Sebastian Seung.  Permitted and forbidden sets in symmetric threshold-linear networks.  In NIPS, pages 217–223. MIT Press, 2000.
  * [16] Junwei Han, Dingwen Zhang, Gong Cheng, Nian Liu, and Dong Xu.  Advanced deep-learning techniques for salient and category-specific object detection: A survey.  IEEE Signal Process. Mag., 35(1):84–100, 2018.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778. IEEE, 2016.
  * [18] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip H. S. Torr.  Deeply supervised salient object detection with short connections.  IEEE TPAMI, 41(4):815–828, 2019.
  * [19] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML(2015), volume 37, pages 448–456. JMLR.org, 2015.
  * [20] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  IEEE TPAMI, 20(11):1254–1259, 1998.
  * [21] Zhong Ji, Haoran Wang, Jungong Han, and Yanwei Pang.  Saliency-guided attention network for image-sentence matching.  In ICCV, pages 5753–5762. IEEE, 2019.
  * [22] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In NIPS, pages 5574–5584, 2017.
  * [23] Dominik A. Klein and Simone Frintrop.  Center-surround divergence of feature statistics for salient object detection.  In ICCV, pages 2214–2219(2011). IEEE, 2011.
  * [24] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  In NIPS, pages 109–117, 2011.
  * [25] Guanbin Li and Yizhou Yu.  Visual saliency based on multiscale deep features.  In CVPR, pages 5455–5463. IEEE Computer Society, 2015.
  * [26] Yin Li, Xiaodi Hou, Christof Koch, James M. Rehg, and Alan L. Yuille.  The secrets of salient object segmentation.  In CVPR, pages 280–287(2014), 2014.
  * [27] Jiawei Liu, Zheng-Jun Zha, Xierong Zhu, and Na Jiang.  Co-saliency spatio-temporal interaction network for person re-identification in videos.  In IJCAI, pages 1012–1018. ijcai.org, 2020.
  * [28] Nian Liu and Junwei Han.  Dhsnet: Deep hierarchical saliency network for salient object detection.  In CVPR, pages 678–686. IEEE, 2016.
  * [29] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum.  Learning to detect a salient object.  IEEE TPAMI, 33(2):353–367, 2011.
  * [30] Yi Liu, Qiang Zhang, Dingwen Zhang, and Jungong Han.  Employing deep part-object relationships for salient object detection.  In ICCV, pages 1232–1241. IEEE, 2019.
  * [31] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In ICLR, 2018.
  * [32] Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu.  Multi-scale interactive network for salient object detection.  In CVPR, pages 9410–9419. IEEE, 2020.
  * [33] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun.  Large kernel matters - improve semantic segmentation by global convolutional network.  In CVPR, 2017.
  * [34] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus H. Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, pages 724–732. IEEE, 2016.
  * [35] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jägersand.  Basnet: Boundary-aware salient object detection.  In CVPR, pages 7479–7489. Computer Vision Foundation / IEEE, 2019.
  * [36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI (3), pages 234–241. Springer, 2015.
  * [37] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia.  Hierarchical image saliency detection on extended CSSD.  Trans. Pattern Anal. Mach. Intell., 38(4):717–729, 2016.
  * [38] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [39] Shaoyue Song, Hongkai Yu, Zhenjiang Miao, Jianwu Fang, Kang Zheng, Cong Ma, and Song Wang.  Multi-spectral salient object detection by adversarial domain adaptation.  In AAAI, pages 12023–12030. AAAI Press, 2020.
  * [40] Jingwei Tang, Yagiz Aksoy, Cengiz Öztireli, Markus H. Gross, and Tunç Ozan Aydin.  Learning-based sampling for natural image matting.  In CVPR, pages 3055–3063. Computer Vision Foundation / IEEE, 2019.
  * [41] Bo Wang, Quan Chen, Min Zhou, Zhiqiang Zhang, Xiaogang Jin, and Kun Gai.  Progressive feature polishing network for salient object detection.  In AAAI, pages 12128–12135. AAAI Press, 2020.
  * [42] Jingdong Wang, Huaizu Jiang, Zejian Yuan, Ming-Ming Cheng, Xiaowei Hu, and Nanning Zheng.  Salient object detection: A discriminative regional feature integration approach.  IJCV, 123(2):251–268, 2017.
  * [43] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan.  Learning to detect salient objects with image-level supervision.  In CVPR, pages 3796–3805(2017). IEEE, 2017.
  * [44] T. Wang, A. Borji, L. Zhang, P. Zhang, and H. Lu.  A stagewise refinement model for detecting salient objects in images.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 4039–4048, 2017.
  * [45] Tiantian Wang, Yongri Piao, Huchuan Lu, Xiao Li, and Lihe Zhang.  Deep learning for light field saliency detection.  In ICCV, pages 8837–8847. IEEE, 2019.
  * [46] Tiantian Wang, Lihe Zhang, Huchuan Lu, Chong Sun, and Jinqing Qi.  Kernelized subspace ranking for saliency detection.  In ECCV(2016), volume 9912 of Lecture Notes in Computer Science, pages 450–466. Springer, 2016.
  * [47] Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang Yang, Xiang Ruan, and Ali Borji.  Detect globally, refine locally: A novel approach to saliency detection.  In CVPR, pages 3127–3135. IEEE, 2018.
  * [48] Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, and Haibin Ling.  Salient object detection in the deep learning era: An in-depth survey.  CoRR, abs/1904.09146, 2019.
  * [49] Wenguan Wang, Jianbing Shen, Ruigang Yang, and Fatih Porikli.  Saliency-aware video object segmentation.  IEEE TPAMI, 40(1):20–33, 2018.
  * [50] Wenguan Wang, Shuyang Zhao, Jianbing Shen, Steven C. H. Hoi, and Ali Borji.  Salient object detection with pyramid attention and salient edges.  In CVPR, pages 1448–1457(2019), 2019.
  * [51] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE TIP, 13(4):600–612, 2004.
  * [52] Jun Wei, Shuhui Wang, and Qingming Huang.  F3net: Fusion, feedback and focus for salient object detection.  CoRR, abs/1911.11445, 2019.
  * [53] Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, and Qi Tian.  Label decoupling framework for salient object detection.  In CVPR, pages 13022–13031. IEEE, 2020.
  * [54] Zhe Wu, Li Su, and Qingming Huang.  Cascaded partial decoder for fast and accurate salient object detection.  In CVPR, pages 3907–3916. Computer Vision Foundation / IEEE, 2019.
  * [55] Zhe Wu, Li Su, and Qingming Huang.  Stacked cross refinement network for edge-aware salient object detection.  In ICCV, pages 7263–7272. IEEE, 2019.
  * [56] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, pages 5987–5995. IEEE, 2017.
  * [57] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang.  Saliency detection via graph-based manifold ranking.  In CVPR, pages 3166–3173. IEEE Computer Society, 2013.
  * [58] Yi Zeng, Pingping Zhang, Zhe L. Lin, Jianming Zhang, and Huchuan Lu.  Towards high-resolution salient object detection.  In ICCV, pages 7233–7242. IEEE, 2019.
  * [59] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 7354–7363. PMLR, 2019.
  * [60] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Sadat Saleh, Tong Zhang, and Nick Barnes.  Uc-net: Uncertainty inspired RGB-D saliency detection via conditional variational autoencoders.  In CVPR, pages 8579–8588. IEEE, 2020.
  * [61] Jing Zhang, Jianwen Xie, and Nick Barnes.  Learning noise-aware encoder-decoder from noisy labels by alternating back-propagation for saliency detection.  CoRR, abs/2007.12211, 2020.
  * [62] Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, and Yuchao Dai.  Weakly-supervised salient object detection via scribble annotations.  In CVPR, pages 12543–12552. IEEE, 2020.
  * [63] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan.  Amulet: Aggregating multi-level convolutional features for salient object detection.  In ICCV, pages 202–211, 2017.
  * [64] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Baocai Yin.  Learning uncertain convolutional features for accurate saliency detection.  In ICCV, pages 212–221. IEEE, 2017.
  * [65] Xiaoning Zhang, Tiantian Wang, Jinqing Qi, Huchuan Lu, and Gang Wang.  Progressive attention guided recurrent network for salient object detection.  In CVPR, pages 714–722. IEEE Computer Society, 2018.
  * [66] Jiaxing Zhao, Jiangjiang Liu, Deng-Ping Fan, Yang Cao, Jufeng Yang, and Ming-Ming Cheng.  Egnet: Edge guidance network for salient object detection.  In ICCV, pages 8778–8787. IEEE, 2019.
  * [67] Kai Zhao, Shanghua Gao, Wenguan Wang, and Ming-Ming Cheng.  Optimizing the f-measure for threshold-free salient object detection.  In ICCV, pages 8848–8856. IEEE, 2019.
  * [68] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang.  Suppress and balance: A simple gated network for salient object detection.  CoRR, abs/2007.08074, 2020.
  * [69] Huajun Zhou, Xiaohua Xie, Jian-Huang Lai, Zixuan Chen, and Lingxiao Yang.  Interactive two-stream decoder for accurate and fast saliency detection.  In CVPR, pages 9138–9147. IEEE, 2020.
  * [70] Yanzhao Zhou, Xin Wang, Jianbin Jiao, Trevor Darrell, and Fisher Yu.  Learning saliency propagation for semi-supervised instance segmentation.  In CVPR, pages 10304–10313. IEEE, 2020.
  * [71] Ziqi Zhou, Zheng Wang, Huchuan Lu, Song Wang, and Meijun Sun.  Multi-type self-attention guided degraded saliency detection.  In AAAI, pages 13082–13089. AAAI Press, 2020.