# A Unified Structure for Efficient RGB and RGB-D Salient Object Detection

Peng Peng, and Yong-Jie Li P. Peng, Yong-Jie. Li are with the MOE Key
Laboratory for Neuroinformation, the School of Life Science and Tecnology,
University of Electronic Science and Technology of China, Chengdu 610054,
China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>author: Yong-jie Li.Manuscript received April 19, 20XX; revised August 26,
20XX.

###### Abstract

Salient object detection (SOD) has been well studied in recent years,
especially using deep neural networks. However, SOD with RGB and RGB-D images
is usually treated as two different tasks with different network structures
that need to be designed specifically. In this paper, we proposed a unified
and efficient structure with a cross-attention context extraction (CRACE)
module to address both tasks of SOD efficiently. The proposed CRACE module
receives and appropriately fuses two (for RGB SOD) or three (for RGB-D SOD)
inputs. The simple unified feature pyramid network (FPN)-like structure with
CRACE modules conveys and refines the results under the multi-level
supervisions of saliency and boundaries. The proposed structure is simple yet
effective; the rich context information of RGB and depth can be appropriately
extracted and fused by the proposed structure efficiently. Experimental
results show that our method outperforms other state-of-the-art methods in
both RGB and RGB-D SOD tasks on various datasets and in terms of most metrics.

###### Index Terms:

Salient object detection, unified structure, RGB SOD, RGB-D SOD, cross
attention.

Human visual system can focus on a target object with little movement in a
scene. This ability helps humans search and find targets efficiently. Inspired
by this fact, researchers proposed a family of methods [1, 2, 3, 4, 5] using
handcrafted features to accomplish the task of salient object detection (SOD).
SOD is believed to facilitate image segmentation [6], tracking [7], retrieval
[8], compression [9] and other computer vision tasks.

Deep learning has dominated the field of SOD since the use of the
convolutional neural network (CNN) in pioneering works [10, 11]. The fully
convolutional network (FCN) has addressed the problem of pixel-level
classification of images with the utilization of an upsampling layer in the
decoder stage. The FCN has shown advantages in many research fields,
especially image segmentation [12]. Therefore, SOD has also been remarkably
facilitated after the introduction of the FCN [13].

However, suffering from pooling–upsampling layers, many image details are
lost, which means that the results are usually fuzzy and incomplete. The
feature pyramid network (FPN) [14] structure was proposed to solve this
problem by adding connections between the encoder and decoder, and has become
popular in image segmentation tasks. Many methods [15, 16] for SOD have
adopted FPN-like structures and achieved good performance. However, due to the
overly simple structure of the decoder and the connections, segmentation
problems still exist, and obtaining clear and accurate results remains
challenging.

As aforementioned, an attention mechanism is a natural component of the human
visual system. Interestingly, artificial neural networks also seem to have
such a mechanism when processing natural language and images [17]. The
utilization of attention in the DNN has led to breakthroughs in many research
fields, including improved SOD performance. However, simply applying spatial
attention or channel attention is not enough to find the salient object;
meanwhile, the non-local (attention) [18] costs too much in both time and
space.

SOD in RGB-D images is usually treated as a task different from RGB image SOD,
and most of the existing methods are specifically designed for RGB-D SOD [19,
20]. However, RGB-D SOD is just an extension of RGB SOD with an additional
modal, and thus there should be a structure or framework to unify these two
tasks.

In this work, to overcome the aforementioned problems, we proposed a cross-
attention context extraction (CRACE) module to serve as the decoder of an FPN-
like network, as shown in Fig.1. This structure is designed as a unified
framework to accomplish both tasks of RGB image SOD and RGB-D image SOD. When
dealing with RGB-D features, depth information is simply extracted with a
backbone network (usually VGG [21] or ResNet [22]) and then fed into the CRACE
module.

In this CRACE module, we first design a cross-attention block to extract the
attention map from high-level global features and low-level local features,
and then optimize the features so that we can use the attention map. This
block is also designed to extract and fuse cross-modality features. Except for
cross-level RGB features, depth features can also be extracted and fused in
this block. It may also be possible to extract and fuse other modalities, such
as optical flow, but this is beyond the scope of this work.

After feature optimization, a channel attention block, a multi-scale
operation, and attentive fusion are implemented to extract rich context
information. The overall structure of the CRACE module is shown in Fig. 2.
These context features are conveyed and fused from global-level deep layers to
local-level shallow layers in the proposed network like a kind of information
flow. The source code and results will be available on our
website111http://www.neuro.uestc.edu.cn/vccl/publications.html. In summary,
our main contributions are as follows:

1) We proposed a cross-attention context extraction module that appropriately
extracts and fuses the coarser global feature and the finer local features,
including cross modality features such as depth information.

2) Based on this module, we proposed a simple unified FPN-like structure for
RGB and RGB-D SOD; this structure conveys and fuses the context information
from global to local, or cross-modality features under the supervision of
ground-truth and object boundary.

3) Quantitative experiments on five popular RGB image SOD datasets and seven
RGB-D image SOD datasets (and using seven metrics) demonstrate that our model
achieves state-of-the-art (SOTA) performance, thereby validating the
effectiveness of our model on both tasks.

Figure 1: Overview of the proposed structure for RGB image (with the blue
arrows) and RGB-D image SOD (with all blue and orange arrows). The proposed
cross-attention context extraction (CRACE) module can extract and fuse cross-
level features and even cross-modal features.

##  I Related work

Since the pioneering work of [23], there has been tremendous progress in SOD.
There are many methods based on some cognitive hypothesis to process extracted
handcrafted features for SOD [1, 2, 3]. Some researchers adopted informatics
or mathematics while others used machine learning to segment salient objects
[5, 24]. Overviews of the use of these handcrafted methods in RGB image SOD
can be found in [25, 19], and the handcrafted methods employed in RGB-D image
SOD can be found in [26, 20]. Since there is a great number of methods based
on deep learning in this field, we only review some of the deep learning based
methods in terms of RGB and RGB-D image SOD.

#### RGB Salient Object Detection

[11] is one of the earliest works treating saliency detection as a high-level
task and exploiting a pre-trained deep model to capture the local and global
contexts of images. Hou et al. [27] adopted short connections to integrate
high-level location knowledge and low-level structure information. Liu et
al.[28] proposed a pixel-wise contextual attention and then integrated it into
a U-Net to detect saliency. Meanwhile, the recurrent fully convolution network
was leveraged in [29] to automatically learn and refine the results, and
reversed saliency was adopted to refine the finer structure of salient objects
in [30].

Information including fixation, semantics, and edges has also been adopted in
many methods. Wang et al. [13] jointly trained their model for fixation and
SOD in a convolutional LSTM network and obtained good performance. Semantic
prior [31] and captioning [32] were leveraged to detect salient objects from
cluttered scenes. Boundaries are frequently used in many SOD methods [33, 34,
15] and usually help produce better results. More reviews of RGB image SOD in
the deep learning era can be found in [35, 19, 36].

Figure 2: Cross-attention context extraction (CRACE) module. The feature
blocks of $f_{i}$ ($f_{l}$) and $F_{i-1}$ ($f_{g}$) (and depth feature
$d_{i}$) are first fused in the cross-attention block and then optimized in
the channel attention block; more information is extracted in the multi-scale
block and attentive fusion block. The output of the CRACE module ($F_{i}$)
then serves as one of the inputs of the next-level CRACE module.

#### RGB-D Salient Object Detection

Depth information is important in many tasks and plays a role in RGB-D SOD.
[37] fed their deep model handcrafted features and produced more accurate
results than traditional methods. Piao et al. [38] proposed a depth-induced
multi-scale network to fully extract the complementary cues from RGB and depth
information. Meanwhile, Zhao et al. [39] introduced the contrast prior into a
deep model to enhance the depth information, and then integrated it with RGB
information. Piao et al. [40] proposed a depth distiller to transfer the depth
knowledge to RGB features. Furthermore, Fan et al. [41] proposed an efficient
three-stream structure and depth depurator to learn cross-modal features.
Selective self-mutual attention was learned and fused in a two-stream CNN
model in [42]. Inspired by the data labeling process, Zhao et al. [43]
proposed a conditional variational auto-encoder to model annotation
uncertainty and produce saliency maps. More reviews of RGB-D image SOD can be
found in [19, 20]

All of the methods mentioned above are designed for a single task of either
RGB or RGB-D image SOD. There are very few deep learning-based methods
designed for both tasks using the same structure. Zhao et al. [44] proposed a
gated dual branch structure to transfer valuable information from the encoder
to the decoder, and produced accurate RGB and RGB-D saliency maps. Zhao et al.
[45] proposed a model where the depth maps were utilized in the training
process but not in the test process, which allowed the model to compute both
RGB and RGB-D saliency maps.

However, these methods are either too complicated in regards to the structure
and training process or achieve suboptimal accuracy. Our method adopts a basic
FPN structure with three proposed CRACE modules and achieves competitive
results with SOTA methods.

##  II The Proposed Network

In this section, we proposed a unified network based on a cross-attention
context extraction (CRACE) module for RGB and RGB-D image SOD.

###  II-A Cross-attention Context Extraction Module

To extract the context information from the features derived by the backbone
network, we proposed a CRACE module. The CRACE module consists of three main
blocks and one fusion operation, i.e., a cross-attention block, a channel
attention block, a multi-scale block, and attentive fusion.

Figure 3: The architecture of the proposed unified structure dealing with the
task of RGB image SOD (the part indicated by blue arrows in Fig.1). Multi-
level saliency and boundary supervisions are also adopted in the training
stage. For the task of RGB-D image SOD, the depth map is processed by the
backbone network and then the multi-level features are fed into the CRACE
modules following the orange arrows in Fig.1. Please see the text for more
details.

#### Cross-attention Block

For a regular neural network the features extracted in the deep layers contain
more global-level semantic information of images while the shallow layers
contain more local-level structure information (such boundary information)
[46, 14]. The widely used spatial attention block extracts the attention map
in a single layer, which usually results in either a loss of structural
information in the deep layers or a loss of semantic information in the
shallow layers.

The proposed cross-attention block addresses this problem by extracting and
fusing two different layers. In this block, two feature blocks (usually one
shallower $f_{l}$ and one deeper $f_{g}$ layer) are first converted to the
same channel by some basic conventional blocks. Then, we add them up and
reduce the number of channels to 1 to produce a cross-attention map $A_{lg}$.
This cross-attention map contains attention information for both feature
layers: more fine structural information for the deeper layer and more
semantic information for the shallower layer. This mechanism can be formulated
as

$A_{lg}=\sigma(C^{n\rightarrow 1}_{1\times 1}[C_{l}(f_{l})+C_{g}(up(f_{g}))])$
(1) $f^{\prime}_{l}=C_{l}(f_{l})+A_{lg}\cdot C_{l}(f_{l})$ (2)
$f^{\prime}_{g}=C_{g}(up(f_{g}))+A_{lg}\cdot C_{g}(up(f_{g}))$ (3)
$F_{lg}=concate(f^{\prime}_{l},f^{\prime}_{g})$ (4)

where these $C(\cdot)$ are conventional blocks combined with batch
normalization and ReLU, $\sigma$ is the Sigmoid function, and $C^{n\rightarrow
1}_{1\times 1}$ is the convolution operation with a filter size of
1$\times$1\. We reduce the number of channels from $n$ to 1, $up(\cdot)$ is
the up-sampling operation, and $n$ is set to 64 in our model (taking into
consideration accuracy and efficiency).

Our CRACE module is different from the cross feature module (CFM) proposed in
[47]. We extract both global-level semantic and local-level structural
information to create a cross-attention mechanism to guide the optimization of
different-level features; in contrast, CFM simply multiplies and adds up the
features. Furthermore, our CRACE module is designed to expand so that it can
process multiple modalities when additional input is added, such as depth
information.

#### Channel Attention Block

Channel attention is widely used in many methods as it is simple but effective
[41, 44] in selecting features by their weightings. Different from CBAM [48],
we combine a channel attention module after the cross-attention block to
select the optimal output of features. Unlike the conventional channel
attention block, we adopt a residual form to preserve more information and
compute the feature maps $F_{ch}$, which can be formulated as

$F_{ch}=C^{2n\rightarrow n}_{1\times 1}(concate[GAP(F_{lg})\cdot
F_{lg},F_{lg}])$ (5)

where $GAP(\cdot)$ is the channel-wise global average pooling, and
$C^{2n\rightarrow n}_{1\times 1}$ is the operation to reduce the number of
channels to $n$. This block selects and assigns the weights of each channel
for the features, which facilitates context information extraction.

#### Multi-scale Block

The usefulness of the multi-scale strategy has been validated in many methods
[30, 44]; it can extract more information from multi-scale features. To
exploit this ability, we apply a multi-scale operation before the attentive
fusion block. The structure of this block is shown in Fig. 2 and formulated as

$F_{ms}=\sum_{j=1}^{4}Branch_{j}(F_{ch})$ (6)

where the $Branch_{j},j=\\{1,2,3,4\\}$ are the operations to extract multi-
scale features; these operations are down-sampling, convolution, up-sampling,
and finally their combination. In the down- and up-sampling operations, the
sampling rates are set to $\\{1,2,4,8\\}$, and the convolution filter sizes
are 3 with dilation rates $\\{1,4,6\\}$ to strike a balance between accuracy
and efficiency.

#### Attentive Fusion

Finally, the outputs of the multi-scale block are attentively fused using the
global layer feature $f_{g}$. The fused output $F$ is designed as

$F=C_{1\times 1}^{2n\rightarrow n}(concate(A_{g}\cdot f_{g}+f_{g},F_{ms}))$
(7)

where $C_{1\times 1}^{2n\rightarrow n}$ is a convolution operation with a
filter size of $1\times 1$ to reduce the number of channels, and $A_{g}$ is
the spatial attention map of $f_{g}$, which is calculated as

$A_{g}=\sigma(C^{n\rightarrow 1}_{1\times 1}(f_{g}))$ (8)

$\sigma$ and $C^{n\rightarrow 1}_{1\times 1}(\cdot)$ are the same as in Eqn.1.
This fusion operation leverages higher global information and fuses this
information into the local features.

###  II-B Network Architecture

We adopt the widely used backbone network ResNet50 [22] as the feature
extractor. We pre-trained the backbone on ImageNet [49], as in most of the
literature.

#### Context Flow

Our network is compact, with a simple connection of the three CRACE modules in
an FPN-like structure. This is illustrated in Fig.3. Only the output of the
last four encoders are processed, as in many other methods [30, 47], since the
first block is computationally expensive. Therefore, the output features
($f_{2}\sim f_{5}$) of each encoder are the inputs of the proposed CRACE
modules.

The features of $f_{4}$ and $f_{5}$ are first input into the CRACE module to
extract the fused feature $F_{4}$ that contains the global context information
and local structural features. Then $F_{4}$ serves as the global input feature
of the next CRACE module. Consequently, context information flows from deep to
shallow. If we treat the operation of the CRACE module as a function
$CRACE(\cdot)$, it can be formulated as follows:

$F_{i-1}=CRACE(f_{i-1},F_{i})$ (9)

where $i\in\\{3,4,5\\}$, and $F_{5}$ is the result of $f_{5}$ after a
convolution operation, which occurs inside the CRACE block.

###  II-C Loss Function

Jointly training boundaries and multi-level supervision is common in SOD
methods, and this usually allows us to obtain better performance compared to
when we employ single-supervision methods [47, 34].

#### Multi-level supervision

Following the idea of multi-level supervision, we train the proposed network
by adopting multi-level supervision with ground-truth $S$ using

$L_{S}=\sum_{i=2}^{5}l_{s}(P_{i},S)$ (10)

where the multi-level saliency predictions $P_{i}$ of the model are calculated
from the features as demonstrated in Eqn.9 using a basic convolution block:

$P_{i}=Conv(F_{i})$ (11)

The loss function $l_{s}$ in Eqn.10 is the combination of conventional binary
cross entropy (BCE) loss and IoU loss [47, 30, 45]. Let
$P,S\in\mathbb{R}^{W\times H\times 1}$ be the saliency prediction from any
level of the CRACE module and ground-truth maps, and the BCE loss and IoU loss
are defined as

$L_{bce}=-\sum_{j=1}^{H}\sum_{k=1}^{W}P_{jk}log(S_{jk})$ (12)
$L_{iou}=1-\frac{\sum_{j=1}^{H}\sum_{k=1}^{W}P_{jk}\cdot
S_{jk}+1}{\sum_{j=1}^{H}\sum_{k=1}^{W}(P_{jk}+S_{jk}-P_{jk}\cdot S_{jk})+1}$
(13)

We also jointly train the proposed network with the boundaries $E$ generated
from the ground-truth maps by

$E=S-erode(S)$ (14)

where $erode(\cdot)$ is the image erosion operation [50]. The supervision of
boundaries can be formulated as

$L_{E}=\sum_{i=2}^{5}l_{e}(E_{i},E)$ (15)

where $E_{i}$ is the boundary features of $F_{i}$ after a basic convolution
operation. The loss function of $l_{e}$ is similar to Eqn. 12 where the
saliency prediction and ground-truth are replaced with edge prediction and
ground-truth.

The whole training loss $L$ of our network is the sum of $L_{S}$ in Eqn.10 and
$L_{E}$ in Eqn.15:

$L=(L_{S}+L_{E})/2$ (16)

Figure 4: Visual comparisons of the proposed method and state-of-the-art
methods. Our method obtains more accurate results than others in both tasks of
RGB and RGBD image SOD. Figure 5: Comparison of precision–recall curves. Our
method obtains the best performance on four datasets and the second best
performance on DUT-OMRON. TABLE I: Quantitative comparisons of RGB image SOD
methods on five datasets in terms of six metrics. The best results are adopted
from the original work, if provided: for example, CPD-r [51] and PiCA-r [28]
are the results of ResNet versions, and PAGE_crf [52] is the result after
post-processing. Values in red and green indicate the best and second best
results.

Publish Date | Ours | AAAI’20 | TIP’20 | CVPR’20 | ECCV’20 | ICCV’20 | CVPR’19 | CVPR’18
---|---|---|---|---|---|---|---|---
Dataset | Metric | F3Net | RASv2 | ITSD | GateNet | EGNet | PoolNet | BASNet | CPD-r | PAGE_crf | AFNet | PiCA-r | PAGRN
ECSSD | maxF $\uparrow$ | 0.951 | 0.945 | 0.948 | 0.947 | 0.945 | 0.947 | 0.949 | 0.942 | 0.939 | 0.932 | 0.935 | 0.935 | 0.927
mF $\uparrow$ | 0.924 | 0.925 | 0.923 | 0.895 | 0.916 | 0.92 | 0.919 | 0.88 | 0.917 | 0.915 | 0.908 | 0.886 | 0.894
MAE $\downarrow$ | 0.032 | 0.033 | 0.034 | 0.034 | 0.04 | 0.037 | 0.035 | 0.037 | 0.037 | 0.04 | 0.042 | 0.046 | 0.061
wF $\uparrow$ | 0.918 | 0.912 | 0.913 | 0.91 | 0.894 | 0.903 | 0.904 | 0.904 | 0.898 | 0.895 | 0.886 | 0.867 | 0.833
Sm $\uparrow$ | 0.93 | 0.924 | 0.925 | 0.925 | 0.92 | 0.925 | 0.926 | 0.916 | 0.918 | 0.911 | 0.913 | 0.917 | 0.889
Em $\uparrow$ | 0.922 | 0.927 | 0.928 | 0.927 | 0.924 | 0.927 | 0.925 | 0.921 | 0.925 | 0.921 | 0.918 | 0.913 | 0.914
PASCAL
_S | maxF | 0.889 | 0.882 | 0.881 | 0.882 | 0.881 | 0.876 | 0.892 | 0.86 | 0.87 | 0.856 | 0.871 | 0.868 | 0.855
mF | 0.849 | 0.844 | 0.836 | 0.797 | 0.83 | 0.829 | 0.838 | 0.777 | 0.829 | 0.823 | 0.828 | 0.802 | 0.805
MAE | 0.064 | 0.064 | 0.067 | 0.066 | 0.071 | 0.076 | 0.067 | 0.079 | 0.074 | 0.077 | 0.071 | 0.078 | 0.095
wF | 0.83 | 0.823 | 0.82 | 0.823 | 0.804 | 0.804 | 0.819 | 0.797 | 0.8 | 0.796 | 0.804 | 0.779 | 0.734
Sm | 0.863 | 0.857 | 0.855 | 0.859 | 0.854 | 0.85 | 0.864 | 0.834 | 0.844 | 0.834 | 0.85 | 0.852 | 0.814
Em | 0.856 | 0.863 | 0.855 | 0.859 | 0.856 | 0.854 | 0.859 | 0.85 | 0.853 | 0.848 | 0.853 | 0.837 | 0.848
DUTS
-TEST | maxF | 0.896 | 0.891 | 0.886 | 0.883 | 0.888 | 0.889 | 0.889 | 0.859 | 0.865 | 0.839 | 0.863 | 0.86 | 0.854
mF | 0.857 | 0.84 | 0.831 | 0.804 | 0.807 | 0.815 | 0.819 | 0.791 | 0.805 | 0.796 | 0.792 | 0.759 | 0.784
MAE | 0.036 | 0.035 | 0.037 | 0.041 | 0.04 | 0.039 | 0.037 | 0.048 | 0.043 | 0.05 | 0.046 | 0.051 | 0.056
wF | 0.844 | 0.835 | 0.827 | 0.823 | 0.809 | 0.815 | 0.817 | 0.803 | 0.795 | 0.779 | 0.785 | 0.754 | 0.724
Sm | 0.892 | 0.888 | 0.884 | 0.859 | 0.885 | 0.887 | 0.886 | 0.866 | 0.869 | 0.851 | 0.867 | 0.869 | 0.838
Em | 0.908 | 0.902 | 0.901 | 0.895 | 0.889 | 0.891 | 0.896 | 0.884 | 0.886 | 0.873 | 0.879 | 0.862 | 0.88
HKU-IS | maxF | 0.939 | 0.937 | 0.933 | 0.934 | 0.933 | 0.935 | 0.936 | 0.928 | 0.925 | 0.919 | 0.923 | 0.918 | 0.918
mF | 0.918 | 0.91 | 0.906 | 0.899 | 0.899 | 0.901 | 0.903 | 0.895 | 0.891 | 0.895 | 0.888 | 0.87 | 0.886
MAE | 0.027 | 0.028 | 0.03 | 0.031 | 0.033 | 0.031 | 0.03 | 0.032 | 0.034 | 0.034 | 0.036 | 0.043 | 0.048
wF | 0.907 | 0.9 | 0.894 | 0.894 | 0.88 | 0.887 | 0.888 | 0.889 | 0.875 | 0.877 | 0.869 | 0.84 | 0.82
Sm | 0.922 | 0.917 | 0.915 | 0.917 | 0.915 | 0.918 | 0.919 | 0.909 | 0.905 | 0.902 | 0.905 | 0.904 | 0.887
Em | 0.955 | 0.953 | 0.952 | 0.952 | 0.949 | 0.95 | 0.953 | 0.946 | 0.944 | 0.941 | 0.942 | 0.936 | 0.939
DUT-OMRON | maxF | 0.819 | 0.813 | 0.815 | 0.821 | 0.818 | 0.815 | 0.805 | 0.805 | 0.797 | 0.792 | 0.797 | 0.803 | 0.771
mF | 0.776 | 0.766 | 0.763 | 0.756 | 0.746 | 0.755 | 0.752 | 0.756 | 0.747 | 0.747 | 0.738 | 0.717 | 0.711
MAE | 0.058 | 0.053 | 0.055 | 0.061 | 0.055 | 0.053 | 0.054 | 0.056 | 0.056 | 0.061 | 0.057 | 0.065 | 0.071
wF | 0.752 | 0.747 | 0.743 | 0.75 | 0.729 | 0.738 | 0.725 | 0.751 | 0.719 | 0.729 | 0.717 | 0.695 | 0.622
Sm | 0.841 | 0.838 | 0.836 | 0.84 | 0.838 | 0.841 | 0.831 | 0.836 | 0.825 | 0.822 | 0.826 | 0.832 | 0.775
Em | 0.873 | 0.87 | 0.872 | 0.863 | 0.862 | 0.867 | 0.868 | 0.869 | 0.866 | 0.851 | 0.853 | 0.841 | 0.842

TABLE II: Quantitative comparisons of RGB-D image SOD methods on seven
datasets in terms of six metrics. The best results are compared if authors
provided results or they could be generated from released code with
recommended configurations. Values in red and green indicate the best and
second best results.

Publish Date | Ours | ECCV’20 | CVPR’20 | TNNLS’20 | TIP’20 | ICCV’19 | CVPR’19
---|---|---|---|---|---|---|---
Dataset | Metric | ATSA | PGAR | CoNet | JL-DCF | A2dele | S2MA | UCNet | SSF | D3Net | ICNet | DMRA | CPFP
DUT-RGBD | maxF $\uparrow$ | 0.946 | 0.943 | 0.938 | 0.936 | 0.923 | 0.907 | 0.91 | 0.883 | 0.934 | 0.771 | 0.874 | 0.909 | 0.571
mF $\uparrow$ | 0.933 | 0.919 | 0.913 | 0.909 | 0.883 | 0.891 | 0.886 | 0.857 | 0.915 | 0.739 | 0.83 | 0.884 | 0.514
MAE $\downarrow$ | 0.027 | 0.031 | 0.035 | 0.033 | 0.043 | 0.041 | 0.043 | 0.056 | 0.033 | 0.099 | 0.072 | 0.048 | 0.148
wF $\uparrow$ | 0.92 | 0.908 | 0.893 | 0.896 | 0.868 | 0.87 | 0.868 | 0.828 | 0.9 | 0.668 | 0.79 | 0.858 | 0.412
Sm $\uparrow$ | 0.927 | 0.921 | 0.919 | 0.919 | 0.906 | 0.88 | 0.903 | 0.863 | 0.915 | 0.775 | 0.852 | 0.888 | 0.605
Em $\uparrow$ | 0.953 | 0.946 | 0.944 | 0.948 | 0.931 | 0.921 | 0.921 | 0.903 | 0.946 | 0.85 | 0.897 | 0.927 | 0.685
LFSD | maxF | 0.894 | 0.884 | 0.874 | 0.877 | 0.887 | 0.857 | 0.862 | 0.878 | 0.893 | 0.849 | 0.891 | 0.872 | 0.849
mF | 0.874 | 0.862 | 0.852 | 0.848 | 0.854 | 0.835 | 0.803 | 0.859 | 0.867 | 0.792 | 0.861 | 0.849 | 0.813
MAE | 0.06 | 0.064 | 0.074 | 0.071 | 0.07 | 0.074 | 0.095 | 0.066 | 0.066 | 0.099 | 0.071 | 0.075 | 0.088
wF | 0.847 | 0.841 | 0.803 | 0.819 | 0.825 | 0.81 | 0.774 | 0.834 | 0.831 | 0.759 | 0.825 | 0.814 | 0.778
Sm | 0.866 | 0.865 | 0.853 | 0.862 | 0.862 | 0.837 | 0.837 | 0.865 | 0.859 | 0.832 | 0.868 | 0.847 | 0.828
Em | 0.908 | 0.905 | 0.889 | 0.896 | 0.882 | 0.87 | 0.863 | 0.897 | 0.895 | 0.833 | 0.891 | 0.899 | 0.867
NJUD | maxF | 0.923 | 0.917 | 0.917 | 0.902 | 0.911 | 0.889 | 0.899 | 0.907 | 0.911 | 0.831 | 0.901 | 0.897 | 0.89
mF | 0.905 | 0.893 | 0.892 | 0.873 | 0.883 | 0.872 | 0.836 | 0.888 | 0.886 | 0.784 | 0.867 | 0.872 | 0.836
MAE | 0.037 | 0.04 | 0.042 | 0.046 | 0.042 | 0.049 | 0.054 | 0.043 | 0.043 | 0.076 | 0.052 | 0.051 | 0.053
wF | 0.892 | 0.883 | 0.876 | 0.856 | 0.874 | 0.851 | 0.846 | 0.872 | 0.871 | 0.747 | 0.849 | 0.853 | 0.834
Sm | 0.907 | 0.901 | 0.909 | 0.895 | 0.903 | 0.867 | 0.894 | 0.896 | 0.899 | 0.827 | 0.894 | 0.886 | 0.878
Em | 0.926 | 0.921 | 0.916 | 0.912 | 0.912 | 0.896 | 0.896 | 0.903 | 0.913 | 0.873 | 0.904 | 0.908 | 0.895
NLPR | maxF | 0.919 | 0.908 | 0.925 | 0.898 | 0.926 | 0.896 | 0.91 | 0.915 | 0.912 | 0.905 | 0.919 | 0.888 | 0.888
mF | 0.898 | 0.869 | 0.883 | 0.848 | 0.875 | 0.878 | 0.847 | 0.886 | 0.875 | 0.833 | 0.867 | 0.854 | 0.821
MAE | 0.023 | 0.028 | 0.025 | 0.031 | 0.022 | 0.028 | 0.03 | 0.025 | 0.026 | 0.034 | 0.028 | 0.031 | 0.036
wF | 0.894 | 0.863 | 0.886 | 0.85 | 0.887 | 0.867 | 0.857 | 0.883 | 0.874 | 0.833 | 0.87 | 0.845 | 0.82
Sm | 0.921 | 0.905 | 0.93 | 0.908 | 0.925 | 0.896 | 0.915 | 0.92 | 0.914 | 0.906 | 0.922 | 0.899 | 0.888
Em | 0.956 | 0.942 | 0.954 | 0.934 | 0.952 | 0.945 | 0.937 | 0.952 | 0.949 | 0.931 | 0.943 | 0.941 | 0.924
RGBD135 | maxF | 0.938 | 0.937 | 0.929 | 0.916 | 0.936 | 0.89 | 0.946 | 0.937 | 0.912 | 0.916 | 0.928 | 0.906 | 0.883
mF | 0.902 | 0.902 | 0.869 | 0.862 | 0.886 | 0.864 | 0.893 | 0.906 | 0.876 | 0.862 | 0.877 | 0.857 | 0.818
MAE | 0.02 | 0.018 | 0.025 | 0.027 | 0.02 | 0.027 | 0.02 | 0.018 | 0.025 | 0.03 | 0.026 | 0.029 | 0.037
wF | 0.893 | 0.908 | 0.862 | 0.856 | 0.897 | 0.845 | 0.897 | 0.913 | 0.86 | 0.837 | 0.875 | 0.849 | 0.794
Sm | 0.92 | 0.928 | 0.916 | 0.911 | 0.931 | 0.878 | 0.943 | 0.935 | 0.905 | 0.906 | 0.923 | 0.901 | 0.874
Em | 0.953 | 0.969 | 0.94 | 0.945 | 0.966 | 0.919 | 0.971 | 0.973 | 0.948 | 0.953 | 0.958 | 0.945 | 0.927
SSD | maxF | 0.874 | 0.873 | 0.85 | 0.851 | 0.799 | 0.815 | 0.878 | 0.723 | 0.838 | 0.872 | 0.855 | 0.858 | 0.801
mF | 0.844 | 0.827 | 0.799 | 0.806 | 0.768 | 0.79 | 0.818 | 0.693 | 0.799 | 0.793 | 0.798 | 0.821 | 0.725
MAE | 0.048 | 0.05 | 0.066 | 0.059 | 0.071 | 0.068 | 0.053 | 0.1 | 0.061 | 0.058 | 0.064 | 0.058 | 0.082
wF | 0.822 | 0.814 | 0.769 | 0.792 | 0.731 | 0.753 | 0.794 | 0.636 | 0.772 | 0.792 | 0.781 | 0.797 | 0.718
Sm | 0.863 | 0.86 | 0.844 | 0.853 | 0.814 | 0.802 | 0.868 | 0.741 | 0.837 | 0.867 | 0.848 | 0.857 | 0.807
Em | 0.91 | 0.901 | 0.872 | 0.896 | 0.886 | 0.858 | 0.891 | 0.844 | 0.887 | 0.887 | 0.878 | 0.892 | 0.832
STEREO | maxF | 0.915 | 0.909 | 0.914 | 0.912 | 0.89 | 0.896 | 0.835 | 0.831 | 0.902 | 0.862 | - | 0.895 | 0.737
mF | 0.899 | 0.884 | 0.885 | 0.885 | 0.871 | 0.883 | 0.896 | 0.811 | 0.88 | 0.82 | - | 0.868 | 0.696
MAE | 0.038 | 0.039 | 0.044 | 0.04 | 0.051 | 0.042 | 0.078 | 0.073 | 0.044 | 0.062 | - | 0.047 | 0.097
wF | 0.885 | 0.874 | 0.86 | 0.871 | 0.837 | 0.867 | 0.758 | 0.774 | 0.862 | 0.785 | - | 0.85 | 0.629
Sm | 0.902 | 0.897 | 0.901 | 0.908 | 0.885 | 0.883 | 0.838 | 0.829 | 0.893 | 0.857 | - | 0.886 | 0.741
Em | 0.932 | 0.921 | 0.922 | 0.924 | 0.917 | 0.915 | 0.868 | 0.881 | 0.923 | 0.898 | - | 0.92 | 0.809

###  II-D RGB-D Salient Object Detection

With a slight adaption, our structure can be directly used for RGB-D SOD.
Compared to RGB image SOD, there is one more input (depth) in RGB-D image SOD.
Therefore, we add depth feature inputs via a backbone network, as shown in
Fig.1 .

#### CRACE with Depth Flow

Since the depth features are added into the network, the CRACE module
formulated by Eqn.9 is slightly adjusted for the task of RGB-D image SOD:

$F_{i-1}=CRACE(f_{i-1},d_{i-1},F_{i})$ (17)

where $d_{i}$ represents the depth feature generated from the backbone
network.

#### Supervision and Loss

Following the design of multi-level supervision in the previous section, we
add 4-level saliency supervision for the depth context. For simplicity, we do
not apply an edge supervision to the depth.

$L_{rgbd}=(L_{S}+L_{D}+L_{E})/3$ (18)

where $L_{D}$ is the same loss function as in Eqn.10 but the prediction map is
generated by depth.

##  III Experiments

To thoroughly evaluate our model, we conducted a series of experiments for
both RGB and RGB-D image SOD tasks on various popular datasets in terms of
seven widely used metrics. Ablation experiments and analyses are also
conducted to exhaustively validate the effectiveness of the proposed module
and framework. Some failure cases are also listed to provide direction for
further improvement.

###  III-A Implementation Details

Our network is trained on the DUTS-TR dataset, like many other methods [53];
this dataset contains 10553 images. We use the ResNet50 [22] pre-trained on
ImageNet [49] as a backbone network. The input image is first resized to
352$\times$352 pixels. The training strategy is the same as for RASv2 [30] and
F3Net [47]. In this stage, we adopted the stochastic gradient descent (SGD)
with a momentum of 0.9 and a weight decay set to 0.0005. The initial learning
rate for the backbone is 0.005 and that for the head is 0.05. Warm-up
processing is applied to adjust the learning rate during training. The
horizontal flip, random crop, and multi-scale pre-processing are adopted for
data augmentation. Our model is written in Pytorch [54] and runs on an NVIDIA
1080ti GPU. The whole training process takes about 7.5 hours with a batch size
of 32. The inference speed for an image is about 37 fps.

For training for the RGB-D task, we follow [40] and adopt their training set,
which contains 800 samples from the DUT-RGBD dataset, 1485 samples from the
NJUD dataset, and 700 samples from the NLPR dataset. The learning rate of the
depth branch is the same as that of the RGB decoder in our network. All other
settings are the same as those used in RGB image SOD.

###  III-B Metrics

Both tasks can be evaluated in terms of the same metrics. We use the
precision–recall (PR) curve, maximum F-measure ($maxF$), mean F-measure
($mF$), weighted F-score ($wF$), mean absolute error ($MAE$), structure
measure ($S_{m}$), and enhanced-alignment measure ($E_{m}$). Precision and
recall are computed by comparing the threshed saliency maps with the ground-
truth maps. Usually, precision and recall values are not suitable for the
evaluation because precision is more important than recall [55]. Consequently,
$F_{\beta}$ is proposed to solve this problem. The F-measure is defined as

$F_{\beta}=\frac{(1+\beta^{2})\cdot Precision\cdot Recall}{\beta^{2}\cdot
Precision+Recall}$ (19)

where $\beta^{2}$ is set to 0.3, as suggested by [1, 2]. Metrics $maxF$ and
$mF$ are the maximum and mean values of the F-measure. $MAE$ is the mean
absolute error between saliency maps and ground-truth maps [56]. The weighted
F-measure ($wF$) [57] is utilized since it is believed to be more reliable by
eliminating the interpolation, dependency, and equal importance flaws of the
previously described measures. Recently, new metrics have been proposed: the
structure measure ($S_{m}$) [58] and enhanced-alignment measure ($E_{m}$)[59].
These are believed to be better for measuring the structural similarity and
image-pixel level matching information between saliency maps and ground-truth
maps.

###  III-C RGB image SOD

#### Datasets

Five popular SOD datasets are used: ECSSD [3] with 1000 structurally complex
images, PASCAL-S [60] with 850 images with cluttered backgrounds, DUT-OMRON
[61] with 5168 multi-object images, HKU-IS [10] with 4447 challenging images,
and DUTS-TE [53] with 5019 images for testing.

#### Comparison to state-of-the-art methods

We compare our method with the 15 most recent state-of-the-art models,
including RASv2 [30], F3Net [47], ITSD [50], GateNet [44], PAGE [52], AFNet
[62], PoolNet [34], BASNet [15], CPD [51], EGNet [63], PAGRN [64], and PiCANet
[28]. For fair comparison, all saliency maps are provided by the authors of
these works or computed by their released code with recommended
configurations.

Fig. 4 shows visual comparisons of the proposed method and other SOTA methods,
demonstrating that our method can achieve more accurate results. The
precision–recall curve (PR) in Fig. 5 shows that our method outperforms other
methods on most datasets while only performing slightly worse than ITSD [50]
on the DUT-OMRON dataset. Results in Table I show that the proposed method
obtains the best results on all datasets in terms of all metrics. This fact
demonstrates the superiority of our method in the task of RGB image SOD.

###  III-D RGB-D image SOD

#### Datasets

Seven popular datasets are used here: LFSD [65] (100 indoor/outdoor images
collected by a Lytro IIlum camera), RGBD135 [66] (135 indoor images collected
by Microsoft Kinect), STEREO [67] (797 stereoscopic images obtained from the
Internet), SSD [68] (80 multi-object images from movies), DUT-RGBD [38] (test
set of 400 multi-object images), NJUD [69] (test set of 500 images) and NLPR
[26] (test set of 300 images collected by Microsoft Kinect).

#### Comparisons to state-of-the-art methods

We compared the performance of the proposed method with 12 recent state-of-
the-art methods, including DMRA [38], CPFP [39], A2dele [40], CoNet [70], JL-
DCF [71], PGAR [72], S2MA [42], UCNet [43], SSF [73], D3Net [41], ATSA [74],
and ICNet [75]. For fair comparison, all saliency maps are provided by the
authors of these works or computed by their released code with recommended
configurations.

Visual comparisons are shown in Fig. 4, where the proposed method is compared
with some of the SOTA. This figure demonstrated our model can obtain more
accurate results than other methods. The quantitative results listed in Table
I demonstrate that our method outperforms the other methods on most datasets
in terms of most metrics, except that the performance of our method on the
RGBD135 dataset is not as good as on other datasets. We will discuss the
reason for this in Sec.III-G (Failure Cases).

TABLE III: RGB-Ablation. CA denotes the cross-attention block, ChA denotes the
channel attention block, MS denotes the multi-scale block, and AF denotes the
attentive fusion block. ‘w/o’ means ‘without’ this loss function or operation.

Model | Component | HKI-IS
---|---|---
CA | ChA | MS | AF | maxF $\uparrow$ | mF $\uparrow$ | wF $\uparrow$ | Sm $\uparrow$
baseline |  |  |  |  | 0.916 | 0.874 | 0.867 | 0.9
+CA | $\checkmark$ |  |  |  | 0.934 | 0.901 | 0.894 | 0.914
+CA+ChA | $\checkmark$ | $\checkmark$ |  |  | 0.936 | 0.904 | 0.899 | 0.918
+CA+ChA+MS | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.939 | 0.911 | 0.904 | 0.921
w/o Edge | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.938 | 0.914 | 0.903 | 0.92
w/o BCE | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.937 | 0.917 | 0.909 | 0.917
w/o IoU | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.938 | 0.899 | 0.887 | 0.92
w/o MLS | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.938 | 0.912 | 0.904 | 0.921
Full | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.939 | 0.918 | 0.907 | 0.922

###  III-E Ablation Studies

To evaluate the effectiveness of each component, we conducted an additional
series of experiments. Table III shows the ablation experiment results of our
method on RGB image SOD in terms of the maximum, mean, and weighted
F-measures, and the structure measure. As the table shows, the sub-modules of
the proposed CRACE module can improve performance. Moreover, supervision of
the boundary, the BCE loss, and the IoU loss also benefit the task, which
further validates the proposed structure. Fig. 6 shows that the results of the
saliency outputs and the boundary outputs are gradually better when following
the information flow.

Similar conclusions can be drawn from the ablation experiments for the RGB-D
image SOD task. As Table IV shows, all sub-modules of the CRACE module
facilitated task performances; the edge supervision, and BCE and IoU loss
functions also aided performance. When the depth information is not used, the
performance is degraded ($\sim 3\%$) in terms of most metrics. This fact
proves that our CRACE module can appropriately extract and make good use of
depth information.

TABLE IV: RGB-D-Ablation. CA denotes the cross-attention block, ChA denotes
the channel attention block, MS denotes the multi-scale block, and AF denotes
the attentive fusion block. ‘w/o’ means ‘without’ this loss function or
operation.

Model | Component | NJUD
---|---|---
CA | ChA | MS | AF | maxF $\uparrow$ | mF $\uparrow$ | wF $\uparrow$ | Sm $\uparrow$
baseline |  |  |  |  | 0.91 | 0.889 | 0.875 | 0.899
+CA | $\checkmark$ |  |  |  | 0.915 | 0.902 | 0.886 | 0.904
+CA+ChA | $\checkmark$ | $\checkmark$ |  |  | 0.917 | 0.899 | 0.886 | 0.905
+CA+ChA+MS | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.919 | 0.903 | 0.891 | 0.907
w/o Depth | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.901 | 0.869 | 0.863 | 0.889
w/o Edge | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.922 | 0.903 | 0.892 | 0.907
w/o BCE | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.92 | 0.903 | 0.891 | 0.905
w/o IoU | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.92 | 0.879 | 0.883 | 0.912
w/o MLS | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.923 | 0.904 | 0.890 | 0.906
Full | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.923 | 0.905 | 0.892 | 0.907

###  III-F Time Efficiency

Time efficiency is an important indicator in SOD. Since the proposed method
has a simple structure with only three CRACE modules, our method can
efficiently accomplish both tasks. Table V compares our method and other
representative methods in terms of frame per second (fps). As the table shows,
our method is fast in the inference stage when processing images in real-time
($\geq 25fps$). Other leading methods can also operate in real time in the
inference stage, but they have relatively poor accuracy.

Similar observations are also found for the task of RGB-D SOD. Our method with
Resnet50 can run as fast as 23 fps (near real time), which is as fast as SSF
[73] and faster than PGAR [72] and S2MA [42], which all have vgg16 as the
backbone. Overall, considering the performances shown in Tables I and II, our
method has promising performance both in efficiency and accuracy.

###  III-G Failure Cases

Fig. 7 shows two failure cases on the RGBD135 dataset since our model seems
weaker than others on this dataset. The top row shows a complex example where
the depth map highlights the flowers and vase, while only the vase is
considered as a salient object by our model. The bottom row shows that our
method cannot segment the chair as accurate as the ground-truth, but the
washbasin is successfully found. This is probably because our method pays more
attention to the semantics that RGB information provided in these scenes,
where the vase and washbasin are segmented. Unfortunately, there are 22 images
that contain both the vase and washbasin in this dataset (for a total of 135
images), which significantly degrades the performance of our model. In future
works, we will try to improve the model by adaptively balance the
contributions of RGB and depth information so that it can achieve good
performance on such images.

##  IV Conclusion

In this work, we first proposed a new CRACE module. Based on this module, we
proposed a simple unified but effective network to accomplish SOD for both RGB
and RGB-D images. The proposed CRACE module makes good use of global
information from higher layers and local information from lower layers by
extracting and applying a cross-attention mechanism. Cross-modal features are
also able to be processed within this module. Experimental results show that
the proposed unified structure achieves the best overall performance compared
with existing state-of-the-art methods on both RGB and RGB-D image SOD tasks.

Figure 6: Context gain after the cross-attention block. From left to right,
(a) shows the images, depth, and ground-truth maps. (b) shows the boundary
results ($E_{4},E_{3},E_{2}$ as in Fig.3) with the CRACE module computed from
the deep to shallow layer. (c) shows the saliency prediction map
($P_{4},P_{3},P_{2}$ as in Fig.3) with the CRACE module. Figure 7: Failure
cases on two images selected from the RGBD135 dataset. TABLE V: Time
Efficiency.

RGB-Methods | Ours | RASv2 | EGNet | PoolNet | F3Net
---|---|---|---|---|---
Backbone | ResNet50 | ResNet50 | ResNet50 | ResNet50 | ResNet50
Speed (fps) | 37 | 40 | 8 | 25 | 34
RGBD-Methods | Ours | SSF | D3Net | PGAR | S2MA
Backbone | ResNet50 | VGG16 | VGG16 | VGG16 | VGG16
Speed (fps) | 23 | 23 | 40 | 18 | 16

## Acknowledgment

This work was supported by Guangdong Key R&D Project (#2018B030338001) and
Natural Science Foundations of China (#61806041, #61703075). We also thank the
LetPub for its linguistic assistance during the preparation of this
manuscript.

## References

  * [1] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency tuned salient region detection,” in _Computer Vision and Pattern Recognition_ , 2009, pp. 1597–1604.
  * [2] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S.-M. Hu, “Global contrast based salient region detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 3, pp. 569–582, 2015.
  * [3] J. Shi, Q. Yan, L. Xu, and J. Jia, “Hierarchical image saliency detection on extended cssd,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 4, pp. 717–729, 2016.
  * [4] K.-F. Yang, H. Li, C.-Y. Li, and Y.-J. Li, “A unified framework for salient structure detection by contour-guided visual search,” _IEEE Transactions on Image Processing_ , vol. 25, no. 8, pp. 3475–3488, 2016.
  * [5] H. Peng, B. Li, H. Ling, W. Hu, W. Xiong, and S. J. Maybank, “Salient object detection via structured matrix decomposition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 4, pp. 818–832, 2016\.
  * [6] E. Rahtu, J. Kannala, M. Salo, and J. Heikkilä, “Segmenting salient objects from images and videos,” _European Conference on Computer Vision_ , pp. 366–379, 2010.
  * [7] C. Ma, Z. Miao, X.-P. Zhang, and M. Li, “A saliency prior context model for real-time object tracking,” _IEEE Transactions on Multimedia_ , vol. 19, no. 11, pp. 2415–2424, 2017.
  * [8] Y. Gao, M. Shi, D. Tao, and C. Xu, “Database saliency for fast image retrieval,” _IEEE Transactions on Multimedia_ , vol. 17, no. 3, pp. 359–369, 2015.
  * [9] Q. Ji, Z. Fang, Z. Xie, and Z. Lu, “Video abstraction based on the visual attention model and online clustering,” _Signal Processing-image Communication_ , vol. 28, no. 3, pp. 241–253, 2013.
  * [10] G. Li and Y. Yu, “Visual saliency based on multiscale deep features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 5455–5463.
  * [11] R. Zhao, W. Ouyang, H. Li, and X. Wang, “Saliency detection by multi-context deep learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1265–1274.
  * [12] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3431–3440.
  * [13] L. Wang, L. Wang, H. Lu, P. Zhang, and X. Ruan, “Saliency detection with recurrent fully convolutional networks,” in _European Conference on Computer Vision_. Springer, 2016, pp. 825–841.
  * [14] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2117–2125.
  * [15] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “Basnet: Boundary-aware salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7479–7489.
  * [16] W. Wang, J. Shen, X. Dong, A. Borji, and R. Yang, “Inferring salient objects from human fixations,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [18] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7794–7803.
  * [19] R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang, “Review of visual saliency detection with comprehensive information,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 2941–2959, 2019.
  * [20] T. Zhou, D.-P. Fan, M.-M. Cheng, J. Shen, and L. Shao, “Rgb-d salient object detection: A survey,” _arXiv preprint arXiv:2008.00230_ , 2020.
  * [21] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [23] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , no. 11, pp. 1254–1259, 1998.
  * [24] Y. Yuan, C. Li, J. Kim, W. Cai, and F. DD, “Reversion correction and regularized random walk ranking for saliency detection,” _IEEE Transactions on Image Processing_ , vol. 27, no. 3, pp. 1–1, 2017.
  * [25] A. Borji, M.-M. Cheng, Q. Hou, H. Jiang, and J. Li, “Salient object detection: A survey,” _Computational Visual Media_ , pp. 1–34, 2014.
  * [26] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji, “Rgbd salient object detection: A benchmark and algorithms,” in _European Conference on Computer Vision_. Springer, 2014, pp. 92–109.
  * [27] Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. H. Torr, “Deeply supervised salient object detection with short connections,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3203–3212.
  * [28] N. Liu, J. Han, and M.-H. Yang, “Picanet: Learning pixel-wise contextual attention for saliency detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3089–3098.
  * [29] L. Wang, L. Wang, H. Lu, P. Zhang, and X. Ruan, “Salient object detection with recurrent fully convolutional networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 7, pp. 1734–1746, 2018.
  * [30] S. Chen, X. Tan, B. Wang, H. Lu, X. Hu, and Y. Fu, “Reverse attention-based residual network for salient object detection,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3763–3776, 2020.
  * [31] T. V. Nguyen, K. Nguyen, and T.-T. Do, “Semantic prior analysis for salient object detection,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 3130–3141, 2019.
  * [32] L. Zhang, J. Zhang, Z. Lin, H. Lu, and Y. He, “Capsal: Leveraging captioning to boost semantics for salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6024–6033.
  * [33] X. Li, F. Yang, H. Cheng, W. Liu, and D. Shen, “Contour knowledge transfer for salient object detection,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 355–370.
  * [34] J.-J. Liu, Q. Hou, M.-M. Cheng, J. Feng, and J. Jiang, “A simple pooling-based design for real-time salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3917–3926.
  * [35] A. Borji, “Saliency prediction in the deep learning era: Successes and limitations,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [36] W. Wang, Q. Lai, H. Fu, J. Shen, H. Ling, and R. Yang, “Salient object detection in the deep learning era: An in-depth survey,” _arXiv preprint arXiv:1904.09146_ , 2019.
  * [37] L. Qu, S. He, J. Zhang, J. Tian, Y. Tang, and Q. Yang, “Rgbd salient object detection via deep fusion,” _IEEE Transactions on Image Processing_ , vol. 26, no. 5, pp. 2274–2285, 2017.
  * [38] Y. Piao, W. Ji, J. Li, M. Zhang, and H. Lu, “Depth-induced multi-scale recurrent attention network for saliency detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7254–7263.
  * [39] J.-X. Zhao, Y. Cao, D.-P. Fan, M.-M. Cheng, X.-Y. Li, and L. Zhang, “Contrast prior and fluid pyramid integration for rgbd salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3927–3936.
  * [40] Y. Piao, Z. Rong, M. Zhang, W. Ren, and H. Lu, “A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection,” _2020 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9057–9066, 2020\.
  * [41] D.-P. Fan, Z. Lin, J. Zhao, Y. Liu, Z. Zhang, Q. Hou, M. Zhu, and M.-M. Cheng, “Rethinking rgb-d salient object detection: Models, datasets, and large-scale benchmarks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [42] N. Liu, N. Zhang, and J. Han, “Learning selective self-mutual attention for rgb-d saliency detection,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 13 753–13 762, 2020.
  * [43] J. Zhang, D.-P. Fan, Y. Dai, S. Anwar, F. Saleh, T. Zhang, and N. Barnes, “Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 8579–8588, 2020.
  * [44] X.-Q. Zhao, Y. Pang, L. Zhang, H. Lu, and L. Zhang, “Suppress and balance: A simple gated network for salient object detection,” in _European Conference on Computer Vision_ , ser. Lecture Notes in Computer Science, vol. 12347\. Springer, 2020, pp. 35–51.
  * [45] J. Zhao, Y. Zhao, J. Li, and X. Chen, “Is depth really necessary for salient object detection?” _Proceedings of the 28th ACM International Conference on Multimedia_ , pp. 1745–1754, 2020.
  * [46] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_ , vol. 9351. Springer, 2015, pp. 234–241.
  * [47] J. Wei, S. Wang, and Q. Huang, “F3net: Fusion, feedback and focus for salient object detection,” _arXiv preprint arXiv:1911.11445_ , 2019.
  * [48] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 3–19.
  * [49] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. Ieee, 2009, pp. 248–255.
  * [50] H. Zhou, X. Xie, J.-H. Lai, Z. Chen, and L. Yang, “Interactive two-stream decoder for accurate and fast saliency detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9141–9150.
  * [51] Z. Wu, L. Su, and Q. Huang, “Cascaded partial decoder for fast and accurate salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3907–3916.
  * [52] W. Wang, S. Zhao, J. Shen, S. C. Hoi, and A. Borji, “Salient object detection with pyramid attention and salient edges,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1448–1457.
  * [53] L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan, “Learning to detect salient objects with image-level supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 136–145.
  * [54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 8026–8037.
  * [55] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H.-Y. Shum, “Learning to detect a salient object,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 2, pp. 353–367, 2010.
  * [56] A. Hornung, Y. Pritch, P. Krahenbuhl, and F. Perazzi, “Saliency filters: Contrast based filtering for salient region detection,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 733–740.
  * [57] R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 248–255.
  * [58] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A new way to evaluate foreground maps,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4548–4557.
  * [59] D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , 2018, pp. 698–704.
  * [60] Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille, “The secrets of salient object segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 280–287.
  * [61] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, “Saliency detection via graph-based manifold ranking,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 3166–3173.
  * [62] M. Feng, H. Lu, and E. Ding, “Attentive feedback network for boundary-aware salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1623–1632.
  * [63] J.-X. Zhao, J.-J. Liu, D.-P. Fan, Y. Cao, J. Yang, and M.-M. Cheng, “Egnet: Edge guidance network for salient object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8779–8788.
  * [64] X. Zhang, T. Wang, J. Qi, H. Lu, and G. Wang, “Progressive attention guided recurrent network for salient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 714–722.
  * [65] N. Li, J. Ye, Y. Ji, H. Ling, and J. Yu, “Saliency detection on light field,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 2806–2813.
  * [66] Y. Cheng, H. Fu, X. Wei, J. Xiao, and X. Cao, “Depth enhanced saliency detection method,” in _Proceedings of International Conference on Internet Multimedia Computing and Service_ , 2014, pp. 23–27.
  * [67] Y. Niu, Y. Geng, X. Li, and F. Liu, “Leveraging stereopsis for saliency analysis,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 454–461.
  * [68] C. Zhu and G. Li, “A three-pathway psychobiological framework of salient object detection using stereoscopic technology,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 3008–3014.
  * [69] R. Ju, L. Ge, W. Geng, T. Ren, and G. Wu, “Depth saliency based on anisotropic center-surround difference,” in _2014 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2014, pp. 1115–1119.
  * [70] W. Ji, J. Li, M. Zhang, Y. Piao, and H. Lu, “Accurate rgb-d salient object detection via collaborative learning,” _arXiv preprint arXiv:2007.11782_ , 2020.
  * [71] K. Fu, D.-P. Fan, G.-P. Ji, and Q. Zhao, “Jl-dcf: Joint learning and densely-cooperative fusion framework for rgb-d salient object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3052–3062.
  * [72] S. Chen and Y. Fu, “Progressively guided alternate refinement network for rgb-d salient object detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 520–538.
  * [73] M. Zhang, W. Ren, Y. Piao, Z. Rong, and H. Lu, “Select, supplement and focus for rgb-d saliency detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3472–3481.
  * [74] M. Zhang, S. X. Fei, J. Liu, S. Xu, Y. Piao, and H. Lu, “Asymmetric two-stream architecture for accurate rgb-d saliency detection,” in _Proceedings of the European Conference on Computer Vision_ , 2020.
  * [75] G. Li, Z. Liu, and H. Ling, “Icnet: Information conversion network for rgb-d based salient object detection,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4873–4884, 2020.

|  Peng Peng received the BS degree in biomedical engineering from the
University of Electronic Science and Technology of China (UESTC), in 2015. He
is now pursuing his Ph.D. degree in UESTC. His research interests include the
visual perceptual organization in early vision, brain-inspired computer
vision, saliency detection, image segmentation and machine learning.
---|---
|  Yong-Jie Li (M’14-SM’17) received the Ph.D. degree in biomedical
engineering from the University of Electronic Science and Technology of China
(UESTC) in 2004. He is currently a Professor with the MOE Key Laboratory for
Neuroinformation, School of Life Science and Technology, UESTC, China. His
research interests include visual mechanism modeling, and the applications in
image processing and computer vision.
---|---


#  Quantum dynamics in one and two dimensions via recursion method

Filipp Uskov  Skolkovo Institute of Science and Technology
Bolshoy Boulevard 30, bld. 1, Moscow 121205, Russia  Gubkin Russian State
University of Oil and Gas
65 Leninsky Prospekt, Moscow 119991, Russia  Oleg Lychkovskiy
<EMAIL_ADDRESS>Skolkovo Institute of Science and Technology
Bolshoy Boulevard 30, bld. 1, Moscow 121205, Russia

###### Abstract

We report an implementation of the recursion method that addresses quantum
many-body dynamics in the nonperturbative regime. The implementation has two
key ingredients: a computer-algebraic routine for symbolic calculation of
nested commutators and a procedure to extrapolate the sequence of Lanczos
coefficients according to the universal operator growth hypothesis. We apply
the method to calculate infinite-temperature correlation functions for
spin-$1/2$ systems on one- and two-dimensional lattices. In two dimensions the
accessible timescale is large enough to essentially embrace the relaxation to
equilibrium. The method allows one to accurately calculate transport
coefficients. As an illustration, we compute the diffusion constant for the
transverse-field Ising model on a square lattice.

Introduction. Quantum dynamics is one of the central topics in condensed
matter physics. While for one-dimensional (1D) systems various numerical
approaches typically deliver highly satisfactory results, addressing higher
dimensions turns out to be much more challenging. Diverse techniques are being
developed to tackle quantum dynamics in two and three dimensions, including
determinant quantum Monte-Carlo Huang  _et al._ (2019), MPS computations on
infinite cylinders James and Konik (2015); Gohlke  _et al._ (2017); Hashizume
_et al._ (2022); De Nicola  _et al._ (2022), methods based on projected
entangled pair states Dziarmaga (2021); Kaneko and Danshita (2023), functional
renormalization group Krieg and Kopietz (2019); Tarasevych and Kopietz (2021),
classical approximations Schubert  _et al._ (2021), hybrid quantum-classical
methods Starkov and Fine (2018, 2020), unfolding of two-dimensional (2D) to
nonlocally coupled 1D systems Thomson and Eisert (2023) etc.

Here we report an implementation of the recursion method capable of addressing
high-temperature dynamics of 1D and 2D lattice systems. The recursion method
has a long history Viswanath and Müller (2008), however instances of its
application to many-body systems are relatively scarce Jensen and Hansen
(1973); Engelsberg and Lowe (1975); Viswanath  _et al._ (1994); Böhm  _et al._
(1994); Lindner and Auerbach (2010); Khait  _et al._ (2016); Auerbach (2018);
Parker  _et al._ (2019); de Souza  _et al._ (2020); Yates  _et al._ (2020a,
b); Yuan  _et al._ (2021). The basic object of the recursion method is a
sequence of Lanczos coefficients $b_{n}$, $n=0,1,2,\dots$, that are to be
obtained from the nested commutators of the system Hamiltonian with the
observable in question. This sequence becomes infinite in the thermodynamic
limit. At the same time, the complexity of calculating the Lanczos
coefficients grows factorially with $n$. This has been hindering the
application of the method for decades.

We alleviate the above difficulty by two complementary remedies. First, we
develop a computer algebra routine to calculate a record number of nested
commutators. The computation is performed directly in the thermodynamic limit
and keeps the Hamiltonian parameters symbolic. Second, we extrapolate the
remaining part of the Lanczos sequence according to the universal operator
growth hypothesis (UOGH) Parker  _et al._ (2019) and other recent insights in
the asymptotic behaviour of this sequence Yates  _et al._ (2020a, b);
Bhattacharjee  _et al._ (2022); Avdoshkin  _et al._ (2022). Remarkably, the
extrapolation works better the further the system is from integrable points.
This makes our approach inherently nonperturbative.

The paper is organized as follows. We start from introducing basic concepts
and definitions, in particular, the autocorrelation function. Then we discuss
the truncated Taylor expansion of the autocorrelation function. After that we
outline the recursion method, the UOGH, and a procedure to obtain transport
coefficients from the Lanczos sequence. Next we describe our implementation of
the recursion method. Then the method is applied to one 1D model and two 2D
models. Discussion and outlook conclude the paper.

Autocorrelation function. We consider a quantum system with a Hamiltonian $H$
and focus on some observable given by a self-ajoint Shrödinger operator $A$.
The same observable in the Heisenberg representation reads
$A(t)=e^{itH}A\,e^{-itH}$. It is convenient to introduce the commutation
superoperator ${\mathcal{L}}\equiv[H,\bullet]$. Then the Heisenberg equation
of motion reads $\partial_{t}A(t)=i\,{\mathcal{L}}A(t)$, and its formal
solutionis is given by $A(t)=e^{i\,t\,{\mathcal{L}}}A$.

Throughout the paper we focus on the normalized infinite-temperature
autocorrelation function

$C(t)\equiv\operatorname{\mbox{tr}}\big{(}A(t)A\big{)}/\operatorname{\mbox{tr}}A^{2}.$
(1)

It has the properties $C(0)=1$ and $C(-t)=C(t)$. We remark that strong long-
lived quantum correlations can well exist at infinite temperature Kanász-Nagy
_et al._ (2017).

It is convenient to introduce a scalar product in the space of operators
according to

$\big{(}A|B\big{)}\equiv\operatorname{\mbox{tr}}\big{(}A^{\dagger}B\big{)}/d,$
(2)

where $d$ is the Hilbert space dimension (which is assumed to be finite). The
scalar product entails the norm $\|A\|=\sqrt{(A|A)}$. In this notations, the
autocorrelation function can be written as $C(t)=(A(t)|A)/\|A\|^{2}$. The
superoperator ${\mathcal{L}}$ is self-adjoint with respect to this scalar
product.

Truncated Taylor expansion. Expanding $A(t)$ in powers of $t$, one obtains the
Taylor expansion of the autocorrelation function,

$C(t)\equiv\sum_{m=0}^{\infty}(-1)^{m}\,\frac{\mu_{2m}}{(2m)!}\,t^{2m},$ (3)

with even moments given by

$\mu_{2m}\equiv({\mathcal{L}}^{2m}A|A)/\|A\|^{2}=({\mathcal{L}}^{m}A|{\mathcal{L}}^{m}A)/\|A\|^{2}$
(4)

and odd moments being zero, ensuring that the autocorrelation function is
even. Note that $\mu_{0}=1$ by definition.

The Taylor expansion (3) is known to have an infinite convergence radius for
1D systems with short-range interactions Araki (1969) and a finite convergence
radius in higher dimensions Parker  _et al._ (2019).

Truncating the Taylor expansion (3) at the order $2n$, one obtains a
polynomial $P_{2n}(t)$. Remarkably, these polynomials constitute rigorous
upper and lower bounds on the autocorrelation function Platz and Gordon
(1973); Bohm and Leschke (1992),

$P_{4l+2}(t)\leq C(t)\leq P_{4l}(t),\quad l=1,2,\dots$ (5)

These two-sided bounds are extremely tight up to a certain time, allowing one
to precisely benchmark more sophisticated approximations to $C(t)$, see Fig.
1.

Recursion method. We employ the Heisenberg-picture version of the recursion
method Viswanath and Müller (2008). It is essentially about solving coupled
Heisenberg equations in the orthogonal Lanczos basis
$\\{A_{n}\\},\,n=0,1,2,\dots$ defined iteratively as follows:
$|A_{0})=\|A\|^{-1}\,|A)$, $|A_{1})={\mathcal{L}}|A_{0})$,

$\displaystyle b_{n}$ $\displaystyle=\|A_{n}\|,\quad n=0,1,2,\dots,$
$\displaystyle|A_{n})$
$\displaystyle=b_{n-1}^{-1}\,{\mathcal{L}}|A_{n-1})-b_{n-1}\,b_{n-2}^{-1}|A_{n-2})\quad
n=2,3,\dots$ (6)

The superoperator ${\mathcal{L}}$ acquires a tridiagonal form in this basis,
with the zero main diagonal and the sequence of Lanczos coefficients $b_{n}$
in the sub/supra-diagonals. As a result, the autocorrelation function (1)
enters a set of coupled equations

$\displaystyle\partial_{t}\varphi_{n}(t)$
$\displaystyle=-b_{n+1}\,\varphi_{n+1}(t)+b_{n}\,\varphi_{n-1}(t),\quad
n=0,1,2,\dots$ $\displaystyle C(t)$ $\displaystyle=\varphi_{0}(t),$ (7)

where $\varphi_{-1}(t)\equiv 0$ and $\varphi_{n}(0)=\delta_{0n}$.

This way the autocorrelation function becomes implicitly determined by the
sequence of Lanczos coefficients $b_{n}$. These coefficients can be obtained
recurrently according to (Quantum dynamics in one and two dimensions via
recursion method) or, alternatively, from the moments (4) Sanchez-Dehesa
(1978).

UOGH and extrapolation of Lanczos coefficients. In practice, only a finite
number of $b_{n}$ can be computed. Other coefficients are to be extrapolated.
The UOGH put forward in Parker  _et al._ (2019) states that for generic
systems the leading asymptotics of $b_{n}$ is linear (with a logarithmic
correction in one dimension). It has been further revealed that certain
subleading terms of the asymptotics can be equally important for the dynamics
Viswanath  _et al._ (1994); Parker  _et al._ (2019); Yates  _et al._ (2020a,
b); Dymarsky and Smolkin (2021); Bhattacharjee  _et al._ (2022); Avdoshkin
_et al._ (2022); Camargo  _et al._ (2023). Guided by these insights, we employ
the following extrapolation formulae for $n\gg 1$:

$\displaystyle b_{n}$ $\displaystyle\simeq\alpha n/\log
n+\gamma+(-1)^{n}\gamma_{*}$ $\displaystyle{\rm for}~{}~{}1{\rm D},$ (8)
$\displaystyle b_{n}$ $\displaystyle\simeq\alpha n+\gamma+(-1)^{n}\gamma_{*}$
$\displaystyle{\rm for}~{}~{}2{\rm D},$ (9)

Here $\alpha$, $\gamma$ and $\gamma_{*}$ are the fitting parameters. In
particular, $\gamma_{*}$ parameterizes odd-even alterations in the Lanczos
sequence that emerge whenever
$\overline{C}\equiv\lim_{t\rightarrow\infty}C(t)$ is nonzero (cf. Viswanath
_et al._ (1994); Yates  _et al._ (2020a, b)).

Transport coefficients. Whenever $A=J$ is the current of some conserved
quantity, the autocorrelation function of $J$ determines the corresponding
transport coefficient Kubo  _et al._ (1957). In particular, when the conserved
quantity in question is energy, one can calculate the energy diffusion
constant $D$ as Kubo  _et al._ (1957); Steinigeweg and Gemmer (2009)

$D=\frac{\|J\|^{2}}{\|H\|^{2}}\,\c{C},\qquad\c{C}\equiv\int_{0}^{\infty}dt\,C(t),$
(10)

where $\operatorname{\mbox{tr}}H=0$ is assumed.

It has been shown recently that, employing the UOGH, one can obtain a precise
approximation $\c{C}_{r}$ to $\c{C}$ from a moderate number $r$ of known
Lanczos coefficients Wang  _et al._ (2023). The approximation reads Wang  _et
al._ (2023) (see also Joslin and Gray (1986))

$\c{C}_{r}=\frac{1}{b_{r}}\prod_{m=1}^{[r/2]}\frac{b_{2m}^{2}}{b_{2m-1}^{2}}\times\begin{cases}1/p_{r}&\text{for
even $r$,}\\\ p_{r}&\text{for odd $r$,}\end{cases}$ (11)

where $[r/2]$ is the integer value of $(r/2)$ and

$p_{r}=\Gamma\left(\frac{r}{2}+\frac{\gamma}{2\alpha}\right)\Gamma\left(\frac{r}{2}+\frac{\gamma}{2\alpha}+1\right)/\left(\Gamma\left(\frac{r}{2}+\frac{\gamma}{2\alpha}+\frac{1}{2}\right)\right)^{2}.$
(12)

$\c{C}_{r}$ usually converges to $\c{C}$ rapidly upon increasing $r$ Wang  _et
al._ (2023), which is confirmed by our calculations, see Fig. 2(b). We note
that one can also substitute a truncated Taylor expansion in eq. (10), however
the transport coefficients obtained this way are less accurate Morita (1972);
Labrujere  _et al._ (1982); Cowan  _et al._ (1989); Tarasevych and Kopietz
(2021).

Figure 1: Upper row: Lanczos coefficients for three models considered in the
text. Dashed lines indicate extrapolating functions (8),(9). Insets highlight
the subleading contribution $\Delta b_{n}$, where $\Delta
b_{n}=b_{n}-(\alpha\,n/\log\,n+\gamma)$ in 1D and $\Delta
b_{n}=b_{n}-(\alpha\,n+\gamma)$ in 2D. Lower row: correlation functions for
the same models (solid lines) plotted up to $t=t_{\max}$. Dashed lines – upper
and lower polynomial bounds (5). Horizontal dash-dotted lines - long-time
averages $\overline{C}$. The result for the 1D Ising model is benchmarked by
the exact diagonalization (dots).

Symbolic implementation. We consider one-dimensional chains and two-
dimensional square lattices of spins $1/2$ with nearest-neighbour
interactions. Both the Hamiltonian $H$ and the observable $A$ are considered
to be translation-invariant.

The core routine of our method is a symbolic computation of nested commutators
${\mathcal{L}}^{n}A$. Importantly, the Hamiltonian parameters are also kept
symbolic. As compared to computation with numerical parameters, this requires
essentially no overhead in terms of computational time and a moderate overhead
in terms of memory. The major advantage of a fully symbolic calculation is
that it covers the whole parameter space in a single run.

The computation is performed in the thermodynamic limit from the outset. The
support of ${\mathcal{L}}^{n}A$ grows linearly with $n$, while the number of
terms grows factorially. Since ${\mathcal{L}}$ is linear, the computation is
straightforwardly parallelizable Brenes  _et al._ (2019). Computation of
${\mathcal{L}}^{n}A$ is the most resource-consuming routine of our code.

At the next step the moments (4) are computed. They have the form of
polynomials with respect to Hamiltonian parameters. For each model considered
below, we list several first moments in the text. The complete list of
computed moments is available as a Supplementary Material sup .

Next we use the relation between Lanczos coefficients and moments Sanchez-
Dehesa (1978) to compute $b_{n}$. At this step numerical values of the
Hamiltonian parameters are plugged in. To avoid numerical instabilities, the
rational arithmetics is used. As a result, a sequence of numerical Lanczos
coefficients $b_{n}$, $n=0,1,\dots,n_{\max}$ is obtained.

Finally, the Lanczos coefficients $b_{n}$ are extrapolated beyond $n_{\max}$
according to eqs. (8), (9), and the autocorrelation function is calculated by
numerically solving equations (Quantum dynamics in one and two dimensions via
recursion method). To estimate the maximal time $t_{\max}$ until which our
results are reliable, we reiterate this final step with the extrapolation
based on $(n_{\max}-1)$ Lanczos coefficients, and require that the discrepancy
between the two approximations to $C(t)$ remains below some small $\epsilon$
($\epsilon=10^{-3}$ for plots in Fig. 1).

1D Ising model. The Hamiltonian of the model reads

$H=\sum\nolimits_{j}\sigma^{x}_{j}\sigma^{x}_{j+1}+h_{z}\sum\nolimits_{j}\sigma^{z}_{j}+h_{x}\sum\nolimits_{j}\sigma^{x}_{j},$
(13)

where $\sigma^{x,y,z}_{j}$ are Pauli matrices at the $j$’th site and $h_{x}$,
$h_{z}$ are two parameters of the Hamiltonian. This model is integrable when
$h_{x}=0$ or $h_{z}=0$, and nonintegrable otherwise. The observable we
consider is the magnetization in the $z$-direction,

$A=\sum\nolimits_{j}\sigma^{z}_{j}.$ (14)

We are able to calculate $n_{\max}=45$ nested commutators and corresponding
moments symbolically (the previous record result was $n_{\max}=38$ moments
calculated numerically Noh (2021)). For example,

$\displaystyle\mu_{2}$ $\displaystyle=8+4h_{x}^{2},$ $\displaystyle\mu_{4}$
$\displaystyle=128+192\,h_{x}^{2}+128\,h_{z}^{2}+16\,h_{x}^{4}+16\,h_{x}^{2}h_{z}^{2}.$
(15)

The corresponding Lanczos coefficients for $h_{x}=h_{y}=1$ are shown in Fig.
1(a). They are consistent with the UOGH and feature pronounced odd-even
alterations on top of the leading asymptote. The corresponding autocorrelation
function is presented in Fig. 1(d). We benchmark our result by a numerically
exact computation for a finite spin chain large enough to neglect finite size
effects on the considered timescale. The long-time average $\overline{C}$ is
nonzero, consistent with odd-even alterations of the Lanczos coefficients.

Note that, as evident from Fig. 1(d), the relaxation is far from being
complete up to the maximal time available. This can be attributed to an
unusually long relaxation timescale of the model (13) Birnkammer  _et al._
(2022) (see also Wang  _et al._ (2023) for a related observation).

2D $XX$-$YY$ model. This is a spin-$1/2$ model on a square lattice with the
Hamiltonian

$H=\sum_{\langle{\mathbf{i}}{\mathbf{j}}\rangle_{-}}\sigma^{x}_{{\mathbf{i}}}\sigma^{x}_{{\mathbf{j}}}+v\sum_{\langle{\mathbf{i}}{\mathbf{j}}\rangle_{|}}\sigma^{y}_{{\mathbf{i}}}\sigma^{y}_{{\mathbf{j}}}.$
(16)

Here ${\mathbf{i}}$ and ${\mathbf{j}}$ enumerate sites of the lattice, and the
first (the second) sum runs over nearest neighbour sites connected by
horizontal (vertical) bonds, each bond being counted once. We choose the first
term of the above Hamiltonian as the observable $A$.

We manage to calculate $n_{\max}=17$ moments, with the first few given by

$\displaystyle\mu_{2}$ $\displaystyle=16\,v^{2},$ $\displaystyle\mu_{4}$
$\displaystyle=640\,v^{2}(1+v^{2}),$ $\displaystyle\mu_{6}$
$\displaystyle=2048\,v^{2}(17+39v^{2}+17v^{4}).$ (17)

In Fig 1(b),(e) we plot the Lanczos coefficients and the autocorrelation
function for $v=1$. At this specific value of $v$ the long-time average of the
autocorrelation function is fixed by symmetry to be $\overline{C}=1/2$. One
can see that, in contrast to the previous case, $C(t)$ relaxes close to this
value within the timescale accessible by our method. $C(t)$ is additionally
benchmarked by the polynomial bounds (5).

2D Ising model. The Hamiltonian is defined on a square lattice and reads

$H=\sum_{\langle{\mathbf{i}}{\mathbf{j}}\rangle}\sigma^{x}_{{\mathbf{i}}}\sigma^{x}_{{\mathbf{j}}}+h_{z}\sum_{{\mathbf{j}}}\sigma^{z}_{{\mathbf{j}}},$
(18)

where the first sum runs over pairs of neighbouring sites.

With an eye on computing the diffusion constant, we choose the energy current
along the horizontal direction as the observable:

$A=J=h_{z}\sum_{\begin{subarray}{c}~{}\,\langle{\mathbf{i}}{\mathbf{j}}\rangle_{-}\\\
{\mathbf{i}}\prec\,{\mathbf{j}}\end{subarray}}(\sigma^{x}_{{\mathbf{i}}}\sigma^{y}_{{\mathbf{j}}}-\sigma^{x}_{{\mathbf{j}}}\sigma^{y}_{{\mathbf{i}}}).$
(19)

Here the sum runs over horizontal bonds, the site ${\mathbf{i}}$ being always
to the left of the site ${\mathbf{j}}$.

We are able to calculate $n_{\max}=21$ nested commutators and corresponding
moments symbolically (previously 13 moments were calculated for a different
observable Heveling  _et al._ (2022)). First three moments read

$\displaystyle\mu_{2}$ $\displaystyle=8,$ $\displaystyle\mu_{4}$
$\displaystyle=64\,(2+h_{z}^{2}),$ $\displaystyle\mu_{6}$
$\displaystyle=1024\,(2+5h_{z}^{2}+h_{z}^{4}).$ (20)

The Lanczos coefficients and the autocorrelation function are shown in Fig.
1(c). In contrast to previous cases, the irregularities of the Lanczos
coefficients do not follow the odd-even alteration pattern. This is consistent
with the fact that the autocorrelation function of the current relaxes to
zero. We therefore do not include the alteration term in the extrapolation.
One can see that again the autocorrelation function essentially relaxes to
equilibrium within the accessible timescale.

We further compute the diffusion constant for a range of magnetic fields
$h_{z}$, see Fig. 2. The convergence of the approximation (11) appears to be
quite good away from the integrable points $h_{z}=0$ and
$h_{z}\rightarrow\infty$, as illustrated in Fig. 2(b). We conservatively
estimate the uncertainty of our calculation as a maximal discrepancy between
ten approximations obtained for $r$ from $(n_{\max}-9)$ to $n_{\max}$. This
uncertainty is indicated in Fig. 2(a). It is below 1% for fields $h_{z}\sim 1$
but grows rapidly when $h_{z}$ or $h_{z}^{-1}$ get close to zero.

Figure 2:  (a) Diffusion constant for the 2D Ising model (18) as a function of
the transverse field $h_{z}$. The width of the line indicates the estimated
uncertainty. (b) Convergence of the diffusion constant with the approximation
order $r$. Shown are data for fields $h_{z}=0.5,1,1.5,2,2.5,3$ (from bottom to
top).

Discussion and outlook. In summary, we have advanced the recursion method to
the point it can handle the dynamics of two-dimensional lattice systems over
the whole relaxation timescale. We have illustrated the power of the method by
computing infinite-temperature autocorrelation functions and the diffusion
constant for spins $1/2$ on a square lattice.

The most resource-consuming part of our computations is performed
symbolically, which means that the whole parameter space of the Hamiltonian is
covered in a single run. The accuracy of the method, however, differs across
the parameter space. Remarkably, the method works best deep in the
nonperturbative regime, where the sequence of the Lanczos coefficients
converges to its asymptotic form most rapidly Parker  _et al._ (2019).

An important ingredient of the method is the extrapolation of Lanczos
coefficients beyond those explicitly computed. The extrapolation is based on
the conjectured asymptotic form of the coefficients. Our study confirms the
conjecture about leading terms of the asymptotics Parker  _et al._ (2019) and
strengthens previous observations about the importance of subleading terms
Viswanath  _et al._ (1994); Parker  _et al._ (2019); Yates  _et al._ (2020a,
b); Dymarsky and Smolkin (2021); Bhattacharjee  _et al._ (2022); Avdoshkin
_et al._ (2022); Camargo  _et al._ (2023). The method will benefit from better
theoretical understanding of the subleading terms.

The generalization of the method to different lattice geometries, higher
spins, lattice fermions or bosons is conceptually straightforward. Finite but
high temperatures can be handled by using the recursion method in conjunction
with the high-temperature expansion. Addressing lower temperatures can be more
challenging, most likely necessitating a considerable amendment of the method.
In particular, employing more complex scalar products Parker  _et al._ (2019);
Dymarsky and Gorsky (2020); Tan  _et al._ (2024) beyond the simplest one (2)
may be required.

Finally, we note that recent approaches Rakovszky  _et al._ (2022); Schuster
and Yao (2023); White (2023); Igor Ermakov (2024) to effectively constraint
the Heisenberg evolution within smaller subspaces of the operator space can
potentially greatly reduce the computational cost of the method. Another very
recent promising move in the same direction is a stochastic sampling of
operator growth De  _et al._ (2024).

###### Acknowledgements.

Acknowledgments. OL thanks Anatoly Dymarsky and Alexander Avdoshkin for a
useful discussion at the initial stage of this study. This work was supported
by the Russian Science Foundation under grant № 24-22-00331,
https://rscf.ru/en/project/24-22-00331/

## References

  * Huang  _et al._ (2019) Edwin W. Huang, Ryan Sheppard, Brian Moritz, and Thomas P. Devereaux, “Strange metallicity in the doped hubbard model,” Science 366, 987–990 (2019).
  * James and Konik (2015) A. J. A. James and R. M. Konik, “Quantum quenches in two spatial dimensions using chain array matrix product states,” Phys. Rev. B 92, 161111 (2015).
  * Gohlke  _et al._ (2017) Matthias Gohlke, Ruben Verresen, Roderich Moessner, and Frank Pollmann, “Dynamics of the kitaev-heisenberg model,” Phys. Rev. Lett. 119, 157203 (2017).
  * Hashizume  _et al._ (2022) Tomohiro Hashizume, Ian P. McCulloch, and Jad C. Halimeh, “Dynamical phase transitions in the two-dimensional transverse-field ising model,” Phys. Rev. Res. 4, 013250 (2022).
  * De Nicola  _et al._ (2022) Stefano De Nicola, Alexios A. Michailidis, and Maksym Serbyn, “Entanglement and precession in two-dimensional dynamical quantum phase transitions,” Phys. Rev. B 105, 165149 (2022).
  * Dziarmaga (2021) Jacek Dziarmaga, “Time evolution of an infinite projected entangled pair state: Neighborhood tensor update,” Phys. Rev. B 104, 094411 (2021).
  * Kaneko and Danshita (2023) Ryui Kaneko and Ippei Danshita, “Dynamics of correlation spreading in low-dimensional transverse-field ising models,” arXiv preprint arXiv:2301.01407 (2023).
  * Krieg and Kopietz (2019) Jan Krieg and Peter Kopietz, “Exact renormalization group for quantum spin systems,” Phys. Rev. B 99, 060403 (2019).
  * Tarasevych and Kopietz (2021) Dmytro Tarasevych and Peter Kopietz, “Dissipative spin dynamics in hot quantum paramagnets,” Phys. Rev. B 104, 024423 (2021).
  * Schubert  _et al._ (2021) Dennis Schubert, Jonas Richter, Fengping Jin, Kristel Michielsen, Hans De Raedt, and Robin Steinigeweg, “Quantum versus classical dynamics in spin models: Chains, ladders, and square lattices,” Phys. Rev. B 104, 054415 (2021).
  * Starkov and Fine (2018) Grigory A. Starkov and Boris V. Fine, “Hybrid quantum-classical method for simulating high-temperature dynamics of nuclear spins in solids,” Phys. Rev. B 98, 214421 (2018).
  * Starkov and Fine (2020) Grigory A. Starkov and Boris V. Fine, “Free induction decays in nuclear spin-$\frac{1}{2}$ lattices with a small number of interacting neighbors: The cases of silicon and fluorapatite,” Phys. Rev. B 101, 024428 (2020).
  * Thomson and Eisert (2023) SJ Thomson and J Eisert, “Unraveling long-time quantum dynamics using flow equations,” arXiv preprint arXiv:2308.13005 (2023).
  * Viswanath and Müller (2008) VS Viswanath and Gerhard Müller,  _The Recursion Method: Application to Many-Body Dynamics_ , Vol. 23 (Springer Science & Business Media, 2008).
  * Jensen and Hansen (1973) S. J. Knak Jensen and E. Kjaersgaard Hansen, “Sixth and eighth moments of the magnetic-resonance lines of a dipolar-coupled rigid lattice,” Phys. Rev. B 7, 2910–2917 (1973).
  * Engelsberg and Lowe (1975) M. Engelsberg and I. J. Lowe, “Approximants to the nuclear-spin autocorrelation function. application to ca${\mathrm{f}}_{2}$,” Phys. Rev. B 12, 3547–3552 (1975).
  * Viswanath  _et al._ (1994) V. S. Viswanath, Shu Zhang, Joachim Stolze, and Gerhard Müller, “Ordering and fluctuations in the ground state of the one-dimensional and two-dimensional s=1/2 xxz antiferromagnets: A study of dynamical properties based on the recursion method,” Phys. Rev. B 49, 9702–9715 (1994).
  * Böhm  _et al._ (1994) Markus Böhm, V. S. Viswanath, Joachim Stolze, and Gerhard Müller, “Spin diffusion in the one-dimensional s=1/2 xxz model at infinite temperature,” Phys. Rev. B 49, 15669–15681 (1994).
  * Lindner and Auerbach (2010) Netanel H. Lindner and Assa Auerbach, “Conductivity of hard core bosons: A paradigm of a bad metal,” Phys. Rev. B 81, 054512 (2010).
  * Khait  _et al._ (2016) Ilia Khait, Snir Gazit, Norman Y. Yao, and Assa Auerbach, “Spin transport of weakly disordered heisenberg chain at infinite temperature,” Phys. Rev. B 93, 224205 (2016).
  * Auerbach (2018) Assa Auerbach, “Hall number of strongly correlated metals,” Phys. Rev. Lett. 121, 066601 (2018).
  * Parker  _et al._ (2019) Daniel E. Parker, Xiangyu Cao, Alexander Avdoshkin, Thomas Scaffidi, and Ehud Altman, “A universal operator growth hypothesis,” Phys. Rev. X 9, 041017 (2019).
  * de Souza  _et al._ (2020) Wescley Luiz de Souza, Érica de Mello Silva, and Paulo H. L. Martins, “Dynamics of the spin-1/2 ising two-leg ladder with four-spin plaquette interaction and transverse field,” Phys. Rev. E 101, 042104 (2020).
  * Yates  _et al._ (2020a) Daniel J. Yates, Alexander G. Abanov, and Aditi Mitra, “Lifetime of almost strong edge-mode operators in one-dimensional, interacting, symmetry protected topological phases,” Phys. Rev. Lett. 124, 206803 (2020a).
  * Yates  _et al._ (2020b) Daniel J. Yates, Alexander G. Abanov, and Aditi Mitra, “Dynamics of almost strong edge modes in spin chains away from integrability,” Phys. Rev. B 102, 195419 (2020b).
  * Yuan  _et al._ (2021) Xiao-Juan Yuan, Jing-Fen Zhao, Hui Wang, Hong-Xia Bu, Hui-Min Yuan, Bang-Yu Zhao, and Xiang-Mu Kong, “Spin dynamics of an ising chain with bond impurity in a tilt magnetic field,” Physica A: Statistical Mechanics and its Applications 583, 126279 (2021).
  * Bhattacharjee  _et al._ (2022) Budhaditya Bhattacharjee, Xiangyu Cao, Pratik Nandy, and Tanay Pathak, “Krylov complexity in saddle-dominated scrambling,” Journal of High Energy Physics 2022, 1–27 (2022).
  * Avdoshkin  _et al._ (2022) Alexander Avdoshkin, Anatoly Dymarsky, and Michael Smolkin, “Krylov complexity in quantum field theory, and beyond,” arXiv preprint arXiv:2212.14429 (2022).
  * Kanász-Nagy  _et al._ (2017) Márton Kanász-Nagy, Izabella Lovas, Fabian Grusdt, Daniel Greif, Markus Greiner, and Eugene A. Demler, “Quantum correlations at infinite temperature: The dynamical nagaoka effect,” Phys. Rev. B 96, 014303 (2017).
  * Araki (1969) Huzihiro Araki, “Gibbs states of a one dimensional quantum lattice,” Communications in Mathematical Physics 14, 120–157 (1969).
  * Platz and Gordon (1973) Ole Platz and Roy G. Gordon, “Rigorous bounds for time-dependent correlation functions,” Phys. Rev. Lett. 30, 264–267 (1973).
  * Bohm and Leschke (1992) M Bohm and H Leschke, “Dynamic spin-pair correlations in a heisenberg chain at infinite temperature based on an extended short-time expansion,” Journal of Physics A: Mathematical and General 25, 1043 (1992).
  * Sanchez-Dehesa (1978) Jesus Sanchez-Dehesa, “The spectrum of jacobi matrices in terms of its associated weight function,” Journal of Computational and Applied Mathematics 4, 275–283 (1978).
  * Dymarsky and Smolkin (2021) Anatoly Dymarsky and Michael Smolkin, “Krylov complexity in conformal field theory,” Phys. Rev. D 104, L081702 (2021).
  * Camargo  _et al._ (2023) Hugo A Camargo, Viktor Jahnke, Keun-Young Kim, and Mitsuhiro Nishida, “Krylov complexity in free and interacting scalar field theories with bounded power spectrum,” Journal of High Energy Physics 2023, 1–48 (2023).
  * Kubo  _et al._ (1957) Ryogo Kubo, Mario Yokota, and Sadao Nakajima, “Statistical-mechanical theory of irreversible processes. ii. response to thermal disturbance,” Journal of the Physical Society of Japan 12, 1203–1211 (1957).
  * Steinigeweg and Gemmer (2009) Robin Steinigeweg and Jochen Gemmer, “Density dynamics in translationally invariant spin-$\frac{1}{2}$ chains at high temperatures: A current-autocorrelation approach to finite time and length scales,” Phys. Rev. B 80, 184402 (2009).
  * Wang  _et al._ (2023) Jiaozi Wang, Mats H Lamann, Robin Steinigeweg, and Jochen Gemmer, “Diffusion constants from the recursion method,” arXiv preprint arXiv:2312.02656 (2023).
  * Joslin and Gray (1986) C.G. Joslin and C.G. Gray, “Calculation of transport coefficients using a modified mori formalism,” Molecular Physics 58, 789–797 (1986).
  * Morita (1972) Tohru Morita, “Spin diffusion in the heisenberg magnets at infinite temperature,” Phys. Rev. B 6, 3385–3393 (1972).
  * Labrujere  _et al._ (1982) J Labrujere, T O Klaassen, and N J Poulis, “Spin dynamics in a 3d heisenberg ferromagnet in the paramagnetic state ii,” Journal of Physics C: Solid State Physics 15, 999 (1982).
  * Cowan  _et al._ (1989) B Cowan, WJ Mullin, and E Nelson, “Spin diffusion in 2d and 3d quantum solids,” Journal of Low Temperature Physics 77, 181–193 (1989).
  * Brenes  _et al._ (2019) Marlon Brenes, Vipin Kerala Varma, Antonello Scardicchio, and Ivan Girotto, “Massively parallel implementation and approaches to simulate quantum dynamics using krylov subspace techniques,” Computer Physics Communications 235, 477–488 (2019).
  * (44) The Supplementary material contains Mathematica© files with symbolic moments for models described in the text.
  * Noh (2021) Jae Dong Noh, “Operator growth in the transverse-field ising spin chain with integrability-breaking longitudinal field,” Phys. Rev. E 104, 034112 (2021).
  * Birnkammer  _et al._ (2022) Stefan Birnkammer, Alvise Bastianello, and Michael Knap, “Prethermalization in one-dimensional quantum many-body systems with confinement,” Nature Communications 13, 7663 (2022).
  * Heveling  _et al._ (2022) Robin Heveling, Jiaozi Wang, and Jochen Gemmer, “Numerically probing the universal operator growth hypothesis,” Phys. Rev. E 106, 014152 (2022).
  * Dymarsky and Gorsky (2020) Anatoly Dymarsky and Alexander Gorsky, “Quantum chaos as delocalization in krylov space,” Phys. Rev. B 102, 085137 (2020).
  * Tan  _et al._ (2024) Chengming Tan, Zhiyang Wei, and Ren Zhang, “Scaling relations of spectrum form factor and krylov complexity at finite temperature,” arXiv preprint arXiv:2401.10499 (2024).
  * Rakovszky  _et al._ (2022) Tibor Rakovszky, C. W. von Keyserlingk, and Frank Pollmann, “Dissipation-assisted operator evolution method for capturing hydrodynamic transport,” Phys. Rev. B 105, 075131 (2022).
  * Schuster and Yao (2023) Thomas Schuster and Norman Y. Yao, “Operator growth in open quantum systems,” Phys. Rev. Lett. 131, 160402 (2023).
  * White (2023) Christopher David White, “Effective dissipation rate in a liouvillian-graph picture of high-temperature quantum hydrodynamics,” Phys. Rev. B 107, 094311 (2023).
  * Igor Ermakov (2024) Tim Byrnes Igor Ermakov, Oleg Lychkovskiy, “Unified framework for efficiently computable quantum circuits,” arXiv preprint arXiv:2401.08187 (2024).
  * De  _et al._ (2024) Ayush De, Umberto Borla, Xiangyu Cao, and Snir Gazit, “Stochastic sampling of operator growth,” arXiv preprint arXiv:2401.06215 (2024).