# Exploring Progress in Multivariate Time Series Forecasting: Comprehensive
Benchmarking and Heterogeneity Analysis

Zezhi Shao1,2, Fei Wang1,2,†, Yongjun Xu1,2, Wei Wei3, Chengqing Yu1,2, Zhao
Zhang1, Di Yao1, Guangyin Jin4, Xin Cao5, Gao Cong6, Christian S. Jensen7,
Xueqi Cheng1,2,† $\dagger$ Corresponding author. 1Institute of Computing
Technology, Chinese Academy of Sciences, Beijing, China
2University of Chinese Academy of Sciences, Beijing, China
3School of Computer Science and Technology, Huazhong University of Science and
Technology, Wuhan, China
4Tsinghua University, Beijing, China
5School of Computer Science and Engineering, The University of New South
Wales, Australia
6School of Computer Science and Engineering, Nanyang Technological University,
Singapore
7Department of Computer Science, Aalborg University, Denmark
{shaozezhi19b, wangfei, yuchengqing22b, zhaozhang2021, yaodi, xyj,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Multivariate Time Series (MTS) widely exists in real-word complex systems,
such as traffic and energy systems, making their forecasting crucial for
understanding and influencing these systems. Recently, deep learning-based
approaches have gained much popularity for effectively modeling temporal and
spatial dependencies in MTS, specifically in Long-term Time Series Forecasting
(LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking
issue and the choice of technical approaches have been hotly debated in
related work. Such controversies significantly hinder our understanding of
progress in this field. Thus, this paper aims to address these controversies
to present insights into advancements achieved. To resolve benchmarking
issues, we introduce BasicTS, a benchmark designed for fair comparisons in MTS
forecasting. BasicTS establishes a unified training pipeline and reasonable
evaluation settings, enabling an unbiased evaluation of over 30 popular MTS
forecasting models on more than 18 datasets. Furthermore, we highlight the
heterogeneity among MTS datasets and classify them based on temporal and
spatial characteristics. We further prove that neglecting heterogeneity is the
primary reason for generating controversies in technical approaches. Moreover,
based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct
an exhaustive and reproducible performance and efficiency comparison of
popular models, providing insights for researchers in selecting and designing
MTS forecasting models.

benchmarking, multivariate time series forecasting

##  1\. Introduction

Multivariate Time Series (MTS) involves multiple time series, each
representing key indicators of real-world complex systems. For instance, the
PEMS (Chen et al., 2001) dataset captures traffic data at critical locations
within a specific area’s transportation system, while the Electricity (Lai et
al., 2018) dataset records the electricity consumption for key clients in the
power system. Consequently, forecasting MTS has become a fundamental technique
for understanding and influencing real-world complex systems, ranging from
traffic management (Shao et al., 2022c) and emergency management (Bi et al.,
2023) to resource optimization (Sun et al., 2023).

Understanding MTS data holistically requires analyzing its temporal and
spatial characteristics (Li et al., 2018; Zhou et al., 2021). In the temporal
aspect, MTS data originating from real-world systems often exhibit complex
dynamics, posing challenges such as non-stationarity, periodicity, and
randomness. Additionally, interdependencies among these time series, known as
spatial dependencies (Li et al., 2018) or cross-dimension dependencies (Zhang
and Yan, 2023), may significantly influence prediction accuracy. As a result,
effectively modeling these complex patterns from temporal and spatial
perspectives stands as a key challenge, and has been widely studied in
existing literature.

Recently, deep learning-based approaches have been the most popular and
promising research topics (Zhou et al., 2021; Wu et al., 2021; Zhou et al.,
2022b; Li et al., 2018; Wu et al., 2019; Yu et al., 2018). Among them, two
most prominent topics are Long-term Time Series Forecasting (LTSF) and
Spatial-Temporal Forecasting (STF), which primarily focus on modeling the
temporal and spatial patterns of MTS, respectively. LTSF is concerned with
accurate long-term forecasting, often employing Transformer-based (Vaswani et
al., 2017) architectures to modeling the long-term temporal dependencies.
Notable examples include efficient Transformers (Kitaev et al., 2020; Li et
al., 2019; Zhou et al., 2021; Liu et al., 2022b), series-level correlations
(Wu et al., 2021), frequency-based methods (Zhou et al., 2022b, a), and
Transformers utilizing patchified time series (Nie et al., 2023; Zhang and
Yan, 2023). In contrast, STF aims to make accurate predictions by effectively
modeling spatial correlations. The prevalent approach is to combine Graph
Convolution Networks (GCN) (Defferrard et al., [n. d.]; Kipf and Welling,
2017) with different sequence models (Cho et al., 2014; Yu and Koltun, 2016;
Li et al., 2019) to form Spatial-Temporal Graph Neural Networks (STGNN).
Noteworthy examples include combining GCN with RNN (Li et al., 2018; Bai et
al., 2020), CNN (Wu et al., 2019, 2020), and Transformer (Zheng et al., 2020;
Guo et al., 2022), etc.

Despite various models claiming significant achievement, some key
controversies in related work remain underexplored, hindering our
understanding of actual progress. In this paper, we identify these
controversies concerning benchmarking and technical approaches, and discuss
them in detail to outline our motivations.

One critical issue is the lack of fair and reasonable benchmarking tools,
leading to (i) inconsistent performance issues and (ii) unreasonable
evaluation approaches. On the one hand, the performance of the same model
reported across different papers is often significantly inconsistent. For
instance, key baselines DCRNN (Li et al., 2018) and GWNet (Wu et al., 2019)
are frequently underestimated in recent studies (Song et al., 2020; Cao et
al., 2020; Chen et al., 2021; Fang et al., 2021), with a performance decrease
of up to 33%. On the other hand, LTSF models (Zhou et al., 2021; Wu et al.,
2021; Zhou et al., 2022b; Zhang and Yan, 2023) usually evaluate models solely
using metrics like MAE and MSE based on normalized time series, making the
prediction error seems to be very low. A more practical approach should
evaluate on re-normalized data and calculate more metrics like MAPE and WAPE,
which are not affected by the data magnitude. More detailed discussions are
presented in Section 4. Overall, these issues prevent researchers from judging
the strengths and weaknesses of different models and assess the satisfaction
of current predictions.

Another controversy lies in the choice of technical routes in both LTSF and
STF. In the temporal aspect, (i) the effectiveness of Transformers has been
debated (Zeng et al., 2023; Nie et al., 2023). The LTSF-Linear (Zeng et al.,
2023), which employs a simple linear layer, significantly outperforms all the
Transformer-based models (Zhou et al., 2021; Wu et al., 2021; Zhou et al.,
2022b; Liu et al., 2022b), and concludes that Transformer architecture are not
as effective as previous works claimed. Although subsequent Transformer-based
models (Nie et al., 2023) outperform LTSF-Linear, the huge gap in their model
size makes it difficult to determine the effectiveness of Transformers. In the
spatial aspect, (ii) the necessity of GCN has been questioned (Liu et al.,
2023b; Shao et al., 2022b). While STGNNs brought significant improvements,
many recent studies have highlighted its inefficiency and have explored
alternative techniques to model the dependencies among time series, e.g.,
normalization (Deng et al., 2021), spatial-temporal identity (Shao et al.,
2022b), and contrastive learning (Liu et al., 2022a). The success of these
non-GCN methods indicates the need for a deeper understanding of spatial
dependencies and insights into when these alternative methods are effective.

To address the above urgent issues and offer insights into the advancements
achieved, in this paper, we comprehensively analyze and compare both MTS
forecasting datasets and models.

First, to address the issue of lacking benchmarking tools, we introduce
BasicTS, a benchmark designed for fair comparisons. BasicTS establishes a
unified training pipeline and reasonable evaluation settings. The former
resolves inconsistent performance issues caused by unique data and
experimental setups in previous works, while the latter provides a more
intuitive and sufficient evaluation of prediction errors. Currently, BasicTS
supports over 30 popular MTS forecasting models, including both STF and LTSF,
and comprehensively evaluates them on more than 18 datasets. Overall, BasicTS
enables a fair and unbiased evaluation of existing models.

Second, we address controversies in technical approaches by highlighting the
heterogeneity among MTS datasets. Heterogeneity refers to completely different
patterns observed in different MTS datasets. In the temporal dimension, we
classify datasets into those with stable patterns, significant distribution
drifts, and unclear patterns. In the spatial dimension, we reveal that the
spatial sample indistinguishability is one of the essences of spatial
dependencies, and divide datasets into those with and without significant
indistinguishability. Experimental results shows that previous conclusions are
valid only for certain types of data. For example, LTSF-Linear only
outperforms Transformers on datasets without stable temporal patterns, and
approaches for modeling spatial dependencies, such as GCN, are only effective
on datasets with significant indistinguishability. We suggest that blindly
adopting conclusions in previous works can lead researchers to draw biased
inferences.

Additionally, leveraging the proposed BasicTS benchmark and rich heterogeneous
datasets, we conduct an exhaustive and reproducible performance and efficiency
comparison of popular models. We firmly believe that these experimental
results will accelerate researchers’ progress and help to draw more reliable
conclusions. Furthermore, based on the insights from our study, we provide a
road map for selecting or designing MTS forecasting models, while also
highlighting directions that deserve more attention.

In summary, our main contributions are the following:

  * •

We introduce BasicTS, the first benchmark specifically designed for fair
comparisons in MTS forecasting, especially STF and LTSF. Currently, BasicTS
facilitates a fair and sufficient evaluation of over 30 popular models and 18
datasets.

  * •

We highlight heterogeneity among MTS datasets, and classify them based on
different temporal and spatial characteristics. We reveal that neglecting
heterogeneity is the cause of controversies in technical approaches, and
previous conclusions are applicable only to certain types of data.

  * •

We conduct an extensive performance and efficiency comparison of popular
models using the BasicTS and heterogeneous datasets. These experimental
results offer valuable insights into the progress we have made, and help
researchers to select or design MTS forecasting models.

This paper is organized as follows. Section 2 provides discussions of related
work on LTSF, STF, and MTS benchmarking. Section 3 covers preliminaries and
essential definitions. In Section 4, we introduce the BasicTS benchmark to
address the issue related to inconsistent performance and unreasonable
evaluation. Section 5 thoroughly delves into the heterogeneity among MTS
datasets, and provides hypotheses for existing controversies about technical
approaches. Section 6 gives experimental setups and presents detailed
experimental results to address controversies, and gives more discussions.
Section 7 concludes the paper.

##  2\. Related Work

In this section, we introduce works related to LTSF and STF, which are the two
most prominent topics in MTS forecasting. We present their goals, techniques,
and existing controversies. Furthermore, we provide a brief discussion on
existing MTS benchmarking works.

###  2.1. Long-term Time Series Forecasting

To achieve accurate long-term time series forecasting, researchers concentrate
on capturing the temporal patterns of MTS data, and have introduced numerous
methods to efficiently and effectively incorporate longer-term historical
information.

Early studies usually fall into traditional statistical methods (e.g., ARIMA
(Ariyo et al., 2014) and ETS (Gardner Jr, 1985)) and machine learning
techniques (e.g., GBRT (Friedman, 2001) and SVR (Drucker et al., 1996)). These
methods often struggle to handle high non-linearity of time series well, and
they typically rely heavily on stationarity-related assumptions (Shao et al.,
2022c). With the advent of deep learning, researchers have turned to more
powerful and complex neural networks for time series modeling, such as TCN (Yu
and Koltun, 2016), LSTM (Sutskever et al., 2014a), and Transformer (Vaswani et
al., 2017). Among these, Transformer-based models have garnered increasing
attention. Informer (Zhou et al., 2021) proposes a ProbSparse self-attention
mechanism and distilling operation to address the quadratic complexity of the
Transformer, leading to significant performance improvements and being
recognized as a milestone work in LTSF (AAAI 2021 Best Paper). Subsequently,
Autoformer (Wu et al., 2021) designs an efficient auto-correlation mechanism
to discover and aggregate information at the series level, while FEDformer
(Zhou et al., 2022b) proposes a attention mechanism with low-rank
approximation in frequency and a mixture of experts decomposition to control
the distribution shifting. Additionally, Pyraformer designs pyramidal
attention to effectively describe short and long temporal dependencies with
low time and space complexity. Overall, the Transformer architecture is widely
regarded as one of the most effective and promising approaches for MTS
forecasting.

However, a recent work, LTSF-Linear (Zeng et al., 2023), has debated the
effectiveness of Transformer architectures for time series forecasting. LTSF-
Linear employs a simple linear layer and remarkably outperforms all the
aforementioned models. It carefully examines every key components of
Transformers and concludes that they are ineffective for time series
forecasting. However, this conclusion has been further challenged by the
latest work (Nie et al., 2023), which employs Transformer architecture to
surpass the performance of LTSF-Linear. Nevertheless, considering the
substantial gap in model size and the small difference in predictive
performance, determining the effectiveness of Transformer models remains
challenging. Furthermore, more exploration is required to understand why the
simple linear model achieves state-of-the-art performance.

###  2.2. Spatial-Temporal Forecasting

In contrast to LTSF, spatial-temporal forecasting tasks encompass not only
temporal dynamics but also significant dependencies among time series. As a
result, numerous studies have been dedicated to exploring methods that can
effectively capture both spatial and temporal patterns simultaneously.

Early deep learning approaches often employed CNN to process spatial
information and combined CNN and RNN (Shi et al., 2015; Yao et al., 2018; Lai
et al., 2018). However, as the relationships among time series are usually
non-Euclidean, grid-based CNN may not be optimal for handling spatial
dependencies. With the development of GCN (Defferrard et al., [n. d.]; Kipf
and Welling, 2017), Spatial-Temporal Graph Neural Networks (STGNNs) (Li et
al., 2018; Yu et al., 2018) have gained increased attention. STGNNs utilize
GCNs to model spatial dependencies based on a pre-defined prior graph, and
further combine them with sequential models (Yu and Koltun, 2016; Cho et al.,
2014; Vaswani et al., 2017). For example, models like DCRNN (Li et al., 2018),
ST-MetaNet (Pan et al., 2019), DGCRN (Li et al., 2021) incorporate GCN with
RNN (Cho et al., 2014) and their variants, and predict step by step following
the seq2seq (Sutskever et al., 2014b) architecture. Graph WaveNet (Wu et al.,
2019), STGCN (Yu et al., 2018), Auto-DSTSGN (Jin et al., 2022) integrate GCNs
with gated TCN and their variants to facilitate parallel computation.
Additionally, attention mechanisms (Vaswani et al., 2017) are widely used in
STGNNs, such as GMAN (Zheng et al., 2020) and ASTGNN (Guo et al., 2022), and
so on (Wang et al., 2020). However, many recent works argue that the pre-
defined prior graph might be biased, incorrect, or even unavailable in many
cases. Thus, they propose to jointly learn the graph structure (i.e., latent
graph) and optimize STGNNs, e.g., AGCRN (Bai et al., 2020), MTGNN (Wu et al.,
2020), StemGNN (Cao et al., 2020), GTS (Shang et al., 2021), and STEP (Shao et
al., 2022a).

However, both prior graph-based STGNNs and latent graph-based STGNNs are
usually associated with a complexity ranging from $O(N^{2}L)$ to
$O(N^{2}L^{2})$ due to the graph convolution operation, where $N$ is the
number of time series and $L$ is the length of input time series.
Consequently, many recent works (Deng et al., 2023; Liu et al., 2023a) have
raised questions about the necessity of STGNN (Deng et al., 2021; Shao et al.,
2022b; Liu et al., 2023c; Wang et al., 2023) and have explored alternative
techniques (Shao et al., 2022b; Deng et al., 2021; Liu et al., 2022a). For
instance, STNorm (Deng et al., 2021) introduced spatial-temporal
normalization, and STID (Shao et al., 2022b) implemented a simple yet
effective spatial-temporal identity attaching approach. They achieved similar
performance to STGNNs but with significantly higher efficiency. The success of
these non-GCN methods highlights the need for a deeper understanding of
spatial dependencies and when and how these methods are effective.

###  2.3. MTS Forecasting Benchmarking

There are several benchmarking studies in MTS forecasting and its downstream
tasks. For instance, DGCRN (Li et al., 2021), LibCity (Wang et al., 2021), and
DL-Traff (Jiang et al., 2021) have developed benchmarks for downstream tasks
in STF, such as urban spatial-temporal forecasting (Jin et al., 2023).
Similarly, LTSF-Linear (Zeng et al., 2023) and TimesNet (Wu et al., 2023)
propose benchmarks for LTSF. However, these benchmarks have several
limitations. Firstly, they only cover a portion of the research in either STF
or LTSF and do not comprehensively cover the discussion of the temporal and
spatial dimensions of MTS. Secondly, many of them lack a unified pipeline, and
each baseline is trained individually with a unique pipeline. Lastly, these
benchmarks do not adequately discuss and address the controversies in
technical approaches that arise in the temporal and spatial dimensions of MTS
forecasting.

##  3\. Preliminaries

We first define the notions of multivariate time series, and then define the
forecasting task. We additionally define the heterogeneity among multivariate
time series forecasting datasets.

###### Definition 0.

Multivariate Time Series. A multivariate time series has multiple time-
dependent variable. It can be denoted as a matrix
$\mathbf{X}\in\mathbb{R}^{T\times N}$, where $T$ is the number of time steps,
$N$ is the number of variables. We additionally denote the data of time series
$i$ ranging from $t_{1}$ to $t_{2}$ as
$\mathbf{X}^{i}_{t_{1}:t_{2}}\in\mathbb{R}^{|t_{2}-t_{1}|}$.

###### Definition 0.

Multivariate Time Series Forecasting. Given historical data
$\mathbf{X}\in\mathbb{R}^{T_{h}\times N}$ from the past $T_{h}$ time steps,
multivariate time series forecasting aims to predict the values
$\mathbf{Y}\in\mathbb{R}^{T_{f}\times N}$ of the $T_{f}$ nearest future time
steps.

###### Definition 0.

Heterogeneity among Multivariate Time Series Forecasting Datasets. Time series
from different datasets often exhibit significantly different patterns. For
example, different time series may evolve in very different ways, and some
datasets may not have significant spatial dependencies.

##  4\. Benchmark Building

A fair and reasonable benchmark are the foundational cornerstone of all
subsequent analyses. Existing evaluation of baseline models faces significant
challenges, including issues of inconsistent performance and unreasonable
evaluation methods. In fact, the lack of objective benchmarks has led to
situations where progress has been exaggerated. For example, many state-of-
the-art works actually fail to outperform some earlier baselines. Furthermore,
seemingly low forecast errors are not necessarily satisfactory in reality. To
address these issues, in this section, we introduce BasicTS, a benchmark
specifically designed for fair comparisons in MTS forecasting.

Table 1. Inconsistent performance of GWNet and DCRNN in many highly cited
papers. The pink background marks the worst performance reported in related
work, while the green background marks the performance produced by BasicTS.

| Source | PEMS04 | PEMS08
---|---|---|---
MAE | RMSE | MAPE | MAE | RMSE | MAPE
GWNet | (Song et al., 2020; Li and Zhu, 2021; Cao et al., 2020; Liu et al., 2021, 2022a) | 25.45 | 39.70 | 17.29% | 19.13 | 31.05 | 12.68%
(Fang et al., 2021; Choi et al., 2022) | 24.89 | 39.66 | 17.29% | 18.28 | 30.04 | 12.15%
(Guo et al., 2022) | 19.36 | 31.72 | 13.31% | 15.07 | 23.85 | 9.51%
(Chen et al., 2021) | 28.15 | 39.88 | 18.52% | 20.30 | 30.82 | 13.84%
BasicTS | 18.80 | 30.14 | 13.19% | 14.67 | 23.55 | 9.46%
| Gap | 33.21%$\uparrow$ | 24.42%$\uparrow$ | 28.78%$\uparrow$ | 27.73%$\uparrow$ | 23.59%$\uparrow$ | 31.64%$\uparrow$
DCRNN | (Song et al., 2020; Li and Zhu, 2021; Cao et al., 2020; Liu et al., 2021, 2022a) | 24.70 | 38.12 | 17.12% | 17.86 | 27.83 | 11.45%
(Fang et al., 2021) | 24.63 | 37.65 | 17.01% | 17.46 | 27.83 | 11.39%
(Guo et al., 2022) | 23.65 | 37.12 | 16.05% | 18.22 | 28.29 | 11.56%
(Chen et al., 2021; Bai et al., 2020; Choi et al., 2022) | 21.22 | 33.44 | 14.17% | 16.82 | 26.36 | 10.92%
BasicTS | 19.66 | 31.18 | 13.45% | 15.23 | 24.17 | 10.21%
Gap | 20.40%$\uparrow$ | 12.20%$\uparrow$ | 21.43%$\uparrow$ | 16.41%$\uparrow$ | 14.56%$\uparrow$ | 11.67%$\uparrow$

###  4.1. Unified Training Pipeline

In this subsection, we delve into the root causes of performance
inconsistencies and propose a unified training pipeline to effectively address
it, thereby enabling a fair comparison of forecasting models.

####  4.1.1. The inconsistent performance issue.

The inconsistency implies that the performance of a given baseline exhibits
notable variations across different papers, even when applied to the same
dataset and experimental settings. To illustrate this intuitively, we collate
the performance of two key baselines: DCRNN (Li et al., 2018) and Graph
WaveNet (Wu et al., 2019), on PEMS04 and PEMS08 datasets from a range of
papers. The results are summarized in Table 1. Notably, all referenced papers
employ an identical experimental setup, i.e., utilizing the last 12 time steps
to predict the subsequent 12, and reporting MAE, RMSE, and MAPE for the
prediction. Each row in Table 1 corresponds to the performance of Graph
WaveNet (GWNet in short) or DCRNN from distinct sources.

We can see a significant performance divergence for these two baselines across
different papers. It’s worth mentioning that GWNet and DCRNN provide public
model implementations. As such, this variance is likely attributed to varying
training pipelines employed in different studies. Furthermore, our approach
yields markedly improved performance compared to the results reported in these
papers, with a maximum gap up to 33% (MAE of GWNet on PEMS04 dataset). Such
substantial performance gaps significantly impact the efficacy of many new
models. Intriguingly, GWNet and DCRNN can still outperform many recent works,
as detailed in Section 6.4.

To address the aforementioned issues, we conduct a comprehensive analysis of
existing codebases, and identify three primary sources of unfairness: data
processing, training configurations, and evaluation implementation. These
details are often overlooked, but have a great influence on the evaluation
results.

  * •

Data Processing: A crucial step in the learning or inference process involves
normalizing raw time series data. Common approaches encompass min-max
normalization and z-score normalization, each exerting varying effects on
prediction performance. For example, a few studies (Guo et al., 2022) employ
min-max normalization, whereas most studies usually adopt z-score
normalization.

  * •

Training Configurations: Training configurations include optimization
strategies and various training tricks. Different setups have substantial
impact on the optimization. For example, most of studies (Li et al., 2018; Wu
et al., 2019; Shao et al., 2022a, c) employ masked MAE for model training,
which excludes abnormal values that could adversely affect predictions for
normal values. In contrast, some studies (Fang et al., 2021; Chen et al.,
2021) adopt naive MAE as their optimization function, which tends to yield
inferior results. Moreover, the incorporation of training tricks, such as
gradient clipping and curriculum learning, may also significantly influence
performance (Shao et al., 2022c).

  * •

Evaluation Implementation: While metrics possess precise mathematical
definitions, the implementation details can diverge across different studies,
including aspects such as handling outliers, and mini-batch computations
(Shang et al., 2021). These difference causes significant deviations from test
results and actual performance.

Figure 1. Architecture of the proposed benchmark BasicTS.

####  4.1.2. Implementation of BasicTS

To facilitate fair and comprehensive comparisons of the provided baselines,
BasicTS introduces a unified training pipeline, as depicted in Figure 1. This
unified training pipeline mainly incorporates unified dataloader, runner, and
evaluation components to address the aforementioned sources of unfairness. The
unified dataloader is equipped with z-score normalization as the default
choice, which generally yields superior performance. Additionally, it adds
external temporal features to the raw data such as time of day and day of the
week attributes. The unified runner controls the entire training, validation,
and testing procedure. By default, we employ masked MAE as the loss function,
which typically outperforms alternatives like naive MAE or MSE. Moreover, the
unified runner integrates commonly-used training tricks like curriculum
learning and gradient clipping. Lastly, the unified evaluation provides
standard metric implementations including MAE, RMSE, MAPE, WAPE, MSE, and
their masked versions. These three components form the foundation of BasicTS,
enabling the fair comparison. Given a network following to the standard model
interface, BasicTS can yield results for that model. Furthermore, BasicTS also
supports many extensible features, such as the logging system, customizable
loss and metrics, and compatibility with diverse devices (cpu or gpu, single
card or multi-card).

Additionally, BasicTS is very user-friendly. It provides users with easy-to-
use and extensible interfaces to facilitate the quick design and evaluation of
new models. With its built-in training pipeline, users are only required to
provide their network architecture, while other details can be configured
through a form-like configuration file. For example, users can enable
curriculum learning by simply setting the option in configuration file to
true.

Table 2. Evaluation on normalized and re-normalized data.

Data | Methods | normalized | re-normalized
---|---|---|---
MAE | MSE | MAE | MAPE | WAPE
ETTh1 | Autoformer | 0.483 | 0.510 | 1.74 | 69.96% | 37.61%
FEDformer | 0.460 | 0.467 | 1.71 | 68.92% | 36.89%
Crossformer | 0.456 | 0.461 | 1.83 | 64.96% | 39.44%
PatchTST | 0.426 | 0.432 | 1.60 | 64.38% | 34.49%
ETTh2 | Autoformer | 0.448 | 0.433 | 3.40 | 59.17% | 22.67%
FEDformer | 0.431 | 0.418 | 3.35 | 56.14% | 22.33%
Crossformer | 0.453 | 0.447 | 3.72 | 66.76% | 24.76%
PatchTST | 0.395 | 0.390 | 2.97 | 55.22% | 19.78%

###  4.2. Evaluation Settings

An reasonable evaluation have to present results in a clear and intuitive
manner. In the field of LTSF, many existing studies adopt metrics such as MAE
and MSE, and calculate performance based on normalized data (z-score
normalized). However, MAE and MSE represent absolute errors that can be
significantly influenced by data magnitude, rendering them less intuitive for
interpretation. Additionally, evaluating prediction performance on normalized
data can yield seemingly very-low prediction errors, potentially misleading
readers unfamiliar with these details. These unreasonable evaluation
approaches make it difficult for readers to judge whether the prediction of
the model is satisfactory.

In this study, we suggest a more practical approach: evaluating on re-
normalized data and incorporating additional metrics such as MAPE and WAPE.
The performance of these two different settings of important LTSF models on
ETTh1 and ETTh2 datasets are summarized in Table 2. Obviously, we can see that
prediction performance is less satisfactory when considering the high
MAPE111The values that close to zeros have been masked. and WAPE values, in
contrast to the seemingly low MAE and MSE on normalized data. To enhance
understanding, we also provide prediction visualizations for these models
using a randomly selected sample in Figure 2, offering a more intuitive
demonstration of their prediction capabilities.

Figure 2. Prediction visualizations on the ETTh1 dataset.

In summary, our evaluation is conducted on re-normalized data, employing
metrics such as MAE, MSE, RMSE, MAPE, and WAPE. Formally, assuming $\Omega$
represents the indices of all observed samples, $y_{i}$ denotes the $i$-th
actual sample, and $\hat{y_{i}}$ is the corresponding prediction, these
metrics are formulated as following:

(1)
$\displaystyle\text{MAE}(y,\hat{y})=\frac{1}{|\Omega|}\sum_{i\in\Omega}|y_{i}-\hat{y}_{i}|$
$\displaystyle\text{MSE}(y,\hat{y})=\frac{1}{|\Omega|}\sum_{i\in\Omega}(y_{i}-\hat{y}_{i})^{2}$
$\displaystyle\text{RMSE}(y,\hat{y})=\sqrt{\frac{1}{|\Omega|}\sum_{i\in\Omega}(y_{i}-\hat{y}_{i})^{2}}$
$\displaystyle\text{MAPE}(y,\hat{y})=\frac{1}{|\Omega|}\sum_{i\in\Omega}|\frac{y_{i}-\hat{y}_{i}}{y_{i}}|$
$\displaystyle\text{WAPE}(y,\hat{y})=\frac{\sum_{i\in\Omega}|y_{i}-\hat{y}_{i}|}{\sum_{i\in\Omega}|y_{i}|}$

where MAE, MSE, and RMSE metrics reflect prediction accuracy, while MAPE and
WAPE can eliminate the influence of data units.

In summary, BasicTS emerges as a fair and extensible benchmark for MTS
forecasting. On one front, it employs a unified training pipeline to offer
fair and reasonable comparisons among popular deep learning-based models.
BasicTS underpins all subsequent experiments and conclusions, ensuring their
validity and credibility. Simultaneously, BasicTS is easy-to-use and
extensible, which greatly facilitates researchers to design and evaluate
models.

##  5\. Heterogeneity among MTS Datasets

This section aims to clarify the inherent heterogeneity among MTS datasets and
delve into the reasons for the controversies related to the choice of
technical approaches. Unlike datasets in computer vision or natural language
processing, which often share common patterns, MTS datasets usually exhibit
distinct patterns derived from diverse underlying source systems. In this
section, we thoroughly investigate this heterogeneity and categorize the
dataset based on characteristics on the temporal and spatial dimensions. We
argue that varying patterns entail unique challenges, implying that specific
technical approaches are applicable only to corresponding types of data.
Neglecting the heterogeneity emerges as a pivotal factor contributing to
controversies in technical approaches.

###  5.1. Temporal Dimension

Figure 3. Distinct temporal patterns in multiple MTS datasets.

According to the different patterns on the temporal dimension, we categorize
MTS datasets into three types: datasets with clear and stable patterns,
datasets with significant distribution drift, and datasets with unclear
patterns. We argue that they are sequentially less predictable. However,
quantifying the predictability (Xu et al., 2019; Garland et al., 2014) of time
series remains an unsolved challenge. Thus, in this study, we intuitively
analyze various datasets through visualizations. We select three typical
datasets: PEMS03, ETTh2, and ExchangeRate as examples, and visualizations are
shown in Figure 3.

We can see significantly distinct patterns among these datasets. Firstly,
PEMS03, which records traffic flow across urban locations, exhibits clear and
stable patterns, e.g., the periodicity with a fixed period size. This pattern
conform to the overall periodicity and stability of urban traffic systems.
Secondly, ETT records the data from transformer sensors. Although it presents
evident cycling patterns, the periods are not fixed and the mean values are
shifting, indicating distribution drift. This is because it is significantly
affected by external unobserved factors, such as weather, sensor quality, etc.
Lastly, ExchangeRate records the exchange rate of several countries and
displays minimal discernible patterns. This outcome stems from the fact that
exchange rates are primarily governed by unpredictable factors, such as
economic policies. Thus, the historical data offers limited values,
particularly for long-term predictions.

Expanding upon these insights, we argue that the inherent heterogeneity
greatly fuels the controversies when comparing Transformers (Zhou et al.,
2021; Wu et al., 2021; Zhou et al., 2022b; Nie et al., 2023) and Linear models
(Zeng et al., 2023). Transformer variants have strong fitting capabilities,
coupled with a strong inductive bias which means that their architectures
imply strong assumptions about data distribution. Conversely, Linear models,
due to their simplicity, struggle to capture complex patterns, but their
inductive bias is relatively weak. Considering the different model
capabilities and heterogeneous temporal patterns, we argue that when learned
on datasets with stable and clear patterns, Transformer models should be able
to capture complex patterns such as periodicity, while Linear models remain
under-fitted due to their limited capacities. On the contrary, in datasets
with significant distribution drift or unclear patterns, Transformer models
are more likely to capture the false features that exist only in the training
dataset, leading to over-fitting problems. Based on the above discussion, we
summarize the following hypotheses:

Hypothesis 1: 1.1 Transformer models outperform Linear models on datasets with
clear and stable temporal patterns, while Linear models suffer from the under-
fitting problem.

1.2 Linear models outperform most of the Transformer models on the datasets
with significant distribution drift and datasets with unclear patterns, while
Transformer models suffer from the over-fitting problem.

LTSF-Linear (Zeng et al., 2023) ignores the heterogeneity among datasets, and
only conducts experiments on datasets without clear and stable patterns,
leading to a biased conclusion that the Transformer architecture is
ineffective for MTS forecasting. We will verify these hypotheses through
extensive experiments in Section 6.2.

###  5.2. Spatial Dimension

Unlike easy-to-understand temporal patterns, the meaning of spatial
dependencies is ambiguous, let alone finding clear metrics that distinguish
heterogeneous MTS datasets. Most studies loosely interpret spatial patterns as
interactions between time series and model them with GCNs, without discussing
in depth how to understand and quantify it. Fortunately, two recent studies,
ST-Norm (Deng et al., 2021) and STID (Shao et al., 2022b), point out that the
indistinguishability of samples in the spatial dimension (spatial
indistinguishability for short) is one of the essences of spatial
dependencies. In this subsection, we adopt this idea and, for the first time,
design quantitative metrics to distinguish heterogeneous datasets.
Specifically, we divide MTS datasets into two types: with and without
significant spatial indistinguishability, and discuss when and how to model
spatial dependencies.

The samples of MTS forecasting are generated by a sliding window with a size
of $T_{p}+T_{f}$, where $T_{p}$ and $T_{f}$ denote the length of historical
data and future data, respectively. The spatial indistinguishability means
that, for a given time $t$, we can expect to generate many samples with
similar historical data but different future data. Simple regression models
(e.g., MLPs, RNNs) cannot predict their different future data based on the
similar historical data, that is, they can not distinguish these samples (Shao
et al., 2022b). Based on such a definition, we propose the following
quantitative metrics:

(2) $\displaystyle r_{1}=\frac{\text{sum}(\mathbf{A}^{P}>e_{u}\ \&\
\mathbf{A}^{F}<e_{l})}{T*N*N}$ $\displaystyle
r_{2}=\frac{\text{sum}(\mathbf{A}^{P}>e_{u}\ \&\
\mathbf{A}^{F}<e_{l})}{\text{sum}(\mathbf{A}^{P}>e_{u})}$
$\displaystyle\mathbf{A}^{P}_{t,i,j}=\text{sim}$
$\displaystyle(\mathbf{X}_{t-T_{p}:t}^{i},\mathbf{X}_{t-T_{p}:t}^{j})=\frac{\mathbf{X}_{t-T_{p}:t}^{i}\cdot\mathbf{X}_{t-T_{p}:t}^{j}}{\|\mathbf{X}_{t-T_{p}:t}^{i}\|\|\mathbf{X}_{t-T_{p}:t}^{j}\|}$
$\displaystyle\mathbf{A}^{F}_{t,i,j}=\text{sim}$
$\displaystyle(\mathbf{X}_{t:t+T_{f}}^{i},\mathbf{X}_{t:t+T_{f}}^{j})=\frac{\mathbf{X}_{t:t+T_{f}}^{i}\cdot\mathbf{X}_{t:t+T_{f}}^{j}}{\|\mathbf{X}_{t:t+T_{f}}^{i}\|\|\mathbf{X}_{t:t+T_{f}}^{j}\|}$

where $\mathbf{A}^{P}_{t,i,j}$ is the similarity between the historical data
of time series $i$ and $j$ at time slice $t$, and
$\mathbf{X}_{t-T_{p}:t}^{i}\in\mathbb{R}^{T_{p}}$ is the historical data of
time series $i$. Similarly, $\mathbf{A}^{F}_{t,i,j}$ is the similarity between
the future data of time series $i$ and $j$ at time slice $t$, and
$\mathbf{X}_{t:t+T_{f}}^{i}\in\mathbb{R}^{T_{f}}$ is the future data of time
series $i$. We adopt the cosine similarity. Considering $N$ time series and
$T$ time slices in total, we define $r_{1}$ and $r_{2}$ to quantify spatial
indistinguishability, where $r_{1}$ represents the ratio of indistinguishable
samples to all samples, and $r_{2}$ represents the ratio of indistinguishable
samples to samples with similar historical data. $e_{u}$ and $e_{l}$ are
similar and dissimilar thresholds. We set $e_{u}=0.9$ and $e_{l}=0.4$.
Further, we calculate these two metrics for 11 common datasets. The results
are shown in Figure 4.

Figure 4. Spatial indistinguishability in different datasets.

From Figure 4, we can clearly see that the $r_{1}$ and $r_{2}$ of ETT,
Electricity (ELC), ExchangeRate (ER), Weather datasets are very low, while the
$r_{1}$ and $r_{2}$ of METR-LA (LA), PEMS-BAY (BAY), PEMS04 (04), and PEMS08
(08) datasets are significantly higher. Interestingly, although these datasets
have exactly the same data format, they are rarely used in combination in
related work. ETT, ELC, ER, and Weather datasets are often used in LTSF, which
cares less about spatial dependencies. LA, BAY, 04, and 08 datasets are used
in STF, which mainly focuses on spatial dependencies.

Based on the above insights, we discuss when and how to model spatial
dependencies. First, there is no urgent need to model spatial dependencies on
datasets without significant spatial indistinguishability, and forcibly
modeling spatial dependencies may even lead to performance degradation.
Second, on datasets with significant spatial indistinguishability, modeling
spatial dependencies by addressing spatial indistinguishability can improve
performance. Below we discuss how STGNNs (Wu et al., 2019; Li et al., 2018),
ST-Norm (Deng et al., 2021), and STID (Shao et al., 2022b) work. (i) GCNs in
STGNNs (Wu et al., 2019; Li et al., 2018) usually rely on graph structures
that conform to the homophily assumption (Zhu et al., 2021, 2020), where
connected nodes often share similar labels222In regression, the label is a
real-value response corresponding to the instance (Hyndman and Athanasopoulos,
2018).. Therefore, nodes (i.e., time series) with similar historical data but
different future data (i.e., labels) are often tends to be disconnected. Based
on such graph structures, GCNs perform message aggregation to make historical
data distinguishable. (ii) ST-Norm (Deng et al., 2021) normalizes data on the
spatial dimension by separately refine the high-frequency component and the
local component underlying the raw data, making the historical data
distinguishable as well. (iii) STID (Shao et al., 2022b) proposes a simple yet
effective idea of attaching a trainable spatial identity to each time series
to distinguish similar historical data. Based on the above discussion, we
summarize the following hypothesises:

Hypothesis 2: 2.1 On datasets with significant spatial indistinguishability,
modeling spatial dependencies by addressing spatial indistinguishability can
improve performance.

2.2 On datasets without significant spatial indistinguishability, forcing the
modeling of spatial dependencies may cause performance degradation.

We will verify these hypotheses through extensive experiments in Section 6.3.

##  6\. Experiments

In this section, we conduct extensive experiments to confirm our hypotheses
and address controversies in technical approaches. In addition, we provide
fair and reproducible comparisons of performance and efficiency of popular MTS
forecasting models, and discuss how to select or design of such models.
Specifically, Section 6.1 illuminates datasets, baselines, and implementation
details employed in this study. In Section 6.2, we discuss the effectiveness
of Transformer-based and Linear-based models, thereby confirming the
hypotheses presented in Section 5.1. In Section 6.3, we delve into when and
how to model spatial dependencies, confirming the hypotheses in Section 5.2.
Section 6.4 presents the performance and efficiency comparisons, and discusses
how much progress we have made. Furthermore, Section 6.5 provides a roadmap
for selecting or designing MTS forecasting models. All codes, datasets,
experimental scripts, and results can be accessed through the public
repository at https://github.com/zezhishao/BasicTS.

###  6.1. Experimental Setup

####  6.1.1. Datasets

Following previous LTSF and STF studies (Lai et al., 2018; Zhou et al., 2021;
Wu et al., 2021; Guo et al., 2019; Li et al., 2018), we use the following MTS
datasets:

  * •

METR-LA is a traffic speed dataset collected from loop-detectors located on
the LA County road network. It contains data of 207 sensors over a period of 4
months from Mar 2012 to Jun 2012. The traffic information is recorded at the
rate of every 5 minutes. METR-LA also includes a sensor graph to indicate
dependencies between sensors (Li et al., 2018).

  * •

PEMS-BAY is a traffic speed dataset collected from California Transportation
Agencies (CalTrans) Performance Measurement System (PeMS). It contains data of
325 sensors in the Bay Area over a period of 6 months from Jan 2017 to June
2017. The traffic information is recorded at the rate of every 5 minutes.
PEMS-BAY also includes a sensor graph to indicate dependencies between sensors
(Li et al., 2018).

  * •

PEMS03, PEMS04, PEMS07, and PEMS08 are a series of traffic flow dataset. The
number represents the code of district where the data is collected. The
traffic information is recorded at the rate of every 5 minutes. Similar to
METR-LA and PEMS-BAY, they also include sensor graphs to indicate dependencies
between sensors (Guo et al., 2019).

  * •

ETTh1, ETTh2, ETTm1, and ETTm2 record electricity transformer temperatures of
different transformers, which is a crucial indicator in the electric power
long-term deployment. The ”1” and ”2” represent different transformers, while
”h” and ”m” represent different sampling frequencies (every hour and every 15
minutes) (Zhou et al., 2021).They contain 7 variables.

  * •

Electricity records the electricity consumption in kWh every 1 hour from 2012
to 2014. It contains 321 clients (Lai et al., 2018).

  * •

Weather is recorded every 10 minutes for the whole year 2020, which contains
21 meteorological indicators, such as air temperature, humidity, etc (Wu et
al., 2021).

  * •

ExchangeRate collects the daily exchange rates of eight countries including
Australia, British, Canada, Switzerland, China, Japan, New Zealand, and
Singapore. (Lai et al., 2018)

The statistical information is summarized in Table 3. Kindly note that not all
the datasets in BasicTS are included in the paper due to space limitations.
Interest readers can find more datasets in our repository, e.g., large-scale
MTS datasets (Liu et al., 2023d).

Table 3. Statistics of datasets.

Dataset | Samples | Variates | Frequency | Time Span | Graph
---|---|---|---|---|---
METR-LA | 34272 | 207 | 5 mins | 4 months | Yes
PEMS-BAY | 52116 | 325 | 5 mins | 6 months | Yes
PEMS03 | 26208 | 358 | 5 mins | 3 months | Yes
PEMS04 | 16992 | 307 | 5 mins | 2 months | Yes
PEMS07 | 28224 | 883 | 5 mins | 4 months | Yes
PEMS08 | 17856 | 170 | 5 mins | 2 months | Yes
ETTh1 | 14400 | 7 | 1 hour | 20 months | No
ETTh2 | 14400 | 7 | 1 hour | 20 months | No
ETTm1 | 57600 | 7 | 15 mins | 20 months | No
ETTm2 | 57600 | 7 | 15 mins | 20 months | No
Electricity | 26304 | 321 | 1 hour | 2 years | No
Weather | 52696 | 21 | 10 mins | 1 year | No
Exchange Rate | 7588 | 8 | 1 day | 27 Years | No

####  6.1.2. Baselines

We select a wealth of popular baselines that have official public code,
including LTSF models and STF models. For conciseness, we omit their detailed
descriptions and briefly categorize them based on their distinct technical
approaches.

For STF models, we embrace a selection of influential baselines that have high
citations or state-of-the-art performance. First, STGCN (Yu et al., 2018),
DCRNN (Li et al., 2018), GWNet (Wu et al., 2019), DGCRN (Li et al., 2021), and
D2STGNN (Shao et al., 2022c) are prior-graph-based methods, which require pre-
defined graphs to indicate spatial dependencies among time series. Second,
AGCRN (Bai et al., 2020), MTGNN (Wu et al., 2020), StemGNN (Cao et al., 2020),
GTS (Shang et al., 2021), and STEP (Shao et al., 2022a) are latent-graph-based
methods, which propose to jointly learn graph structures and optimize STGNNs.
Lastly, we also adopt two non-graph based methods, ST-Norm (Deng et al., 2021)
and STID (Shao et al., 2022b), which seek alternative approaches to model
spatial dependencies. For LTSF models, we collect both Transformer-based and
Linear-based models. Informer (Zhou et al., 2021), Autoformer (Wu et al.,
2021), FEDformer (Zhou et al., 2022b), Triformer (Cirstea et al., 2022),
Pyraformer (Liu et al., 2022b), Crossformer (Zhang and Yan, 2023), PatchTST
(Nie et al., 2023) are Transformer-based models, which utilize Transformer
architectures or their variants to capture long-term historical information.
On the contrary, Linear, DLinear, and NLinear are a series Linear-based models
proposed in LTSF-Linear (Zeng et al., 2023), which try to outperform
Transformers with a simple linear layer. Table 4 gives a brief summary.

Due to the space limitations, we can not include all baselines in BasicTS in
the paper, interested readers can find more baselines in our repository, e.g.,
STGODE (Fang et al., 2021), PDFormer (Jiang et al., 2023), etc (Liu et al.,
2023a). Furthermore, we additionally note that there is no clear boundary
between STF models and LTSF models. For example, efficient STF models, such as
ST-Norm and STID, can also be used for LTSF.

Table 4. Overview of baselines.

Methods | Fied | Category | Notes
---|---|---|---
STGCN | STF |  Prior- Graph- based | GCN + TCN
DCRNN | GCN + RNN
GWNet | GCN + TCN
DGCRN | GCN + RNN
D2STGNN | Mixed
AGCRN |  Latent- Graph- based | GCN + RNN
MTGNN | GCN + TCN
StemGNN | GCN + TCN + Spectral
GTS | GCN + RNN
STEP | Pretraining-Enhanced
ST-Norm |  Non-Graph- based | Normalization
STID | Identity Embedding
Informer | LTSF |  Transformer- based | Efficient Self-Attention
Autoformer | Auto-Correlation
FEDformer | Frequency-Enhanced
Pyraformer | Pyramidal Attention
Triformer | Triangular Attention
Crossformer | Cross-Dimension Attention
PatchTST |  | Channel Independent + Patchify
Linear |  Linear- based | Vanilla Linear Layer
NLinear | Linear + Normalization
DLinear |  | Linear + Decomposition

Table 5. Performance of Transformer-based models and Linear-based models on
heterogeneous MTS datasets.

Methods | PEMS04 | PEMS08 | ETTh2 | ETTm2
---|---|---|---|---
MAE | RMSE | WAPE | MAE | RMSE | WAPE | MAE | RMSE | WAPE | MAE | RMSE | WAPE
Informer | 27.94 | 44.74 | 12.84% | 26.92 | 43.79 | 11.63% | 7.12 | 6.87 | 47.44% | 5.84 | 7.90 | 38.97%
Autoformer | 34.72 | 50.33 | 14.81% | 33.75 | 51.23 | 14.13% | 3.33 | 4.91 | 22.17% | 2.74 | 4.58 | 18.27%
FEDformer | 26.89 | 41.46 | 12.39% | 25.14 | 39.17 | 10.87% | 3.27 | 4.93 | 21.78% | 2.70 | 4.54 | 17.99%
Linear | 37.42 | 62.14 | 17.22% | 34.04 | 57.07 | 14.71% | 3.18 | 5.04 | 21.19% | 2.52 | 4.24 | 16.80%
DLinear | 37.51 | 62.21 | 17.26% | 34.15 | 57.18 | 14.76% | 3.13 | 5.00 | 20.85% | 2.49 | 4.23 | 16.63%
NLinear | 37.62 | 62.38 | 17.31% | 34.11 | 57.26 | 14.74% | 3.16 | 5.06 | 21.09% | 2.49 | 4.21 | 16.60%
Gap | 39.49%$\downarrow$ | 49.87%$\downarrow$ | 39.30%$\downarrow$ | 35.40%$\downarrow$ | 45.69%$\downarrow$ | 35.32%$\downarrow$ | 4.28%$\uparrow$ | 1.83%$\uparrow$ | 4.26%$\uparrow$ | 7.78%$\uparrow$ | 7.27%$\uparrow$ | 7.72%$\uparrow$

####  6.1.3. Implementation Details

For dataset partitioning, we adopt settings consistent with previous work for
each dataset. For simplicity, we omit their details and interested readers can
refer to our repository. We set the length of the historical data and future
data of the STF task to 12. For the LTSF task, we set the length of future
data to be 336. We test historical data of length {96, 192, 336, 720} and
report the best performance. For error calculations, we only report the
average error between the forecast time series and the true future time series
due to space limitation. Interested readers can find results at each horizon
in our repository. For the STF task, we adopt the MAE, RMSE, MAPE and WAPE
metrics. For the LTSF task, we remove the MAPE metric, considering that there
are many zero values in commonly used LTSF datasets. For efficiency studies in
Section 6.4, we report the average training time per epoch (unit: seconds) and
model parameter number (unit: million). We set the batch size to 64. If an OOM
situation occurs, we will reduce it by half (minimum to 8). For model
implementation and hyperparameter tuning, we adopt their public model
implementation and model hyperparameters. Furthermore, we try our best to tune
the hyperparameters of each model on each dataset to ensure that its
performance is at least as good as that reported in the paper (if available).
All experiments are conducted using a NVIDIA 3090 GPU and 128 GB memory on
Ubuntu 20.04.

Figure 5. Changes of MAE with epoch increasing.

###  6.2. Transformers v.s. Linear Models

In this subsection, we discuss the effectiveness of Transformers and Linears,
and verify the hypotheses in Section 5.1. Specifically, we select four
datasets, where PEMS04 and PEMS08 have clear and stable patterns and ETTh2 and
ETTm2 have significant distribution drift or unclear patterns. We choose six
baseline models adopt in LTSF-Linear (Zeng et al., 2023), where Informer,
Autoformer, and FEDformer are three Transformer-based models, and Linear,
DLinear, and NLinear are three Linear-based models. They all follow the same
LTSF setup mentioned in Section 6.1.3. We report MAE and WAPE. Furthermore, we
also calculate the performance gap between the best Transformer-based method
and Linear-based method. The results are shown in Table 5.

Firstly, Transformer-based models generally outperform Linear-based models by
a very large margin (green background) on datasets with clear and stable
patterns. On the contrary, Linear-based models consistently outperform
Transformer-based models on datasets with distribution drifts or unclear
patterns. Such a huge and inconsistent performance gap is puzzling. To
intuitively understand why, we visualize the changes of MAE with epoch
increasing for FEDformer and DLinear on PEMS08 and ETTh2 datasets. The results
are shown in Figure 5. On the PEMS08 dataset, the train, validation, and test
MAEs of FEDformer start from similar values and keep decreasing. However,
DLinear’s MAEs, even the train MAE, do not decrease with epoch increasing,
which indicates that DLinear suffers from an under-fitting problem. On the
other hand, on the ETTh2 dataset, the train MAE of FEDformer keeps decreasing,
while its validation and test MAE start to increase from very early epochs
(e.g., two epochs), which indicates that FEDformer suffers from a serious
over-fitting problem. These results are consistent with the hypotheses
proposed in Section 5.1.

Table 6. Performance of STID, AGCRN, and their variants on datasets with
varying degrees of spatial indistinguishability.

Data | Metrics | STID | AGCRN |  STID∗ |  AGCRN∗ | Gap
---|---|---|---|---|---|---
LA | MAE | 3.12 | 3.16 | 3.58 | 3.36 | 12.85%$\uparrow$
RMSE | 6.49 | 6.44 | 7.24 | 6.90 | 10.35%$\uparrow$
MAPE | 9.13% | 8.88% | 10.32% | 9.66% | 11.53%$\uparrow$
BAY | MAE | 1.56 | 1.60 | 1.80 | 1.70 | 13.33%$\uparrow$
RMSE | 3.59 | 3.67 | 4.21 | 3.96 | 14.72%$\uparrow$
MAPE | 3.53% | 3.65% | 4.12% | 3.92% | 14.32%$\uparrow$
ER | MAE | 0.0325 | 0.0455 | 0.0312 | 0.0421 | 8.07%$\downarrow$
WAPE | 4.28% | 5.98% | 4.11% | 5.51% | 8.52%$\downarrow$
MAPE | 7.21% | 12.03% | 6.89% | 9.60% | 25.31%$\downarrow$
ETT${}_{\text{m1}}$ | MAE | 1.63 | 2.29 | 1.41 | 1.89 | 21.16%$\downarrow$
WAPE | 35.24% | 49.42% | 30.64% | 40.89% | 20.86%$\downarrow$
MAPE | 64.86% | 75.64% | 55.13% | 68.39% | 17.65%$\downarrow$

Table 7. STF on METR-LA, PEMS-BAY, PEMS03, PEMS04, PEMS07, and PEMS08
datasets.

Data | Metrics | STGCN | DCRNN | GWNet | DGCRN | D2STGNN | AGCRN | MTGNN | StemGNN | GTS | STEP | STNorm | STID
---|---|---|---|---|---|---|---|---|---|---|---|---|---
METR-LA | MAE | 3.11 | 3.03 | 3.03 | 2.94 | 2.88 | 3.16 | 3.05 | 3.72 | 3.13 | 2.93 | 3.14 | 3.12
RMSE | 6.31 | 6.23 | 6.12 | 6.04 | 5.91 | 6.44 | 6.16 | 7.33 | 6.32 | 5.96 | 6.49 | 6.49
MAPE | 8.37% | 8.31% | 8.17% | 7.79% | 7.83% | 8.88% | 8.16% | 10.43% | 8.62% | 8.00% | 8.84% | 9.13%
WAPE | 5.38% | 5.24% | 5.24% | 5.10% | 4.99% | 5.48% | 5.28% | 6.45% | 5.42% | 5.07% | 5.44% | 5.40%
Param | 0.25 | 0.37 | 0.31 | 0.20 | 0.39 | 0.75 | 0.41 | 1.20 | 38.49 | 40.48 | 0.22 | 0.12
Speed | 21.01 | 94.87 | 27.70 | 128.84 | 152.33 | 28.22 | 24.37 | 16.19 | 52.23 | 497.26 | 25.26 | 7.50
PEMS-BAY | MAE | 1.63 | 1.59 | 1.59 | 1.58 | 1.52 | 1.60 | 1.60 | 1.99 | 1.68 | 1.48 | 1.58 | 1.56
RMSE | 3.72 | 3.69 | 3.68 | 3.65 | 3.53 | 3.67 | 3.71 | 4.49 | 3.79 | 3.42 | 3.65 | 3.59
MAPE | 3.69% | 3.58% | 3.60% | 3.52% | 3.44% | 3.65% | 3.59% | 4.61% | 3.78% | 3.31% | 3.52% | 3.53%
WAPE | 2.61% | 2.55% | 2.55% | 2.53% | 2.43% | 2.56% | 2.57% | 3.19% | 2.68% | 2.37% | 2.52% | 2.50%
Param | 0.31 | 0.37 | 0.31 | 0.21 | 0.40 | 0.75 | 0.57 | 1.39 | 58.67 | 60.47 | 0.28 | 0.12
Speed | 51.91 | 223.35 | 70.24 | 401.66 | 374.32 | 62.07 | 46.47 | 30.30 | 179.98 | 1406.90 | 63.90 | 13.28
PEMS03 | MAE | 15.83 | 15.54 | 14.59 | 14.60 | 14.63 | 15.24 | 14.85 | 16.95 | 15.41 | N/A | 15.32 | 15.33
RMSE | 27.51 | 27.18 | 25.24 | 26.20 | 26.31 | 26.65 | 25.23 | 28.52 | 26.15 | N/A | 25.93 | 27.40
MAPE | 16.13% | 15.62% | 15.52% | 14.87% | 15.32% | 15.89% | 14.55% | 19.61% | 15.39% | N/A | 14.37% | 16.40%
WAPE | 9.11% | 8.94% | 8.39% | 8.40% | 8.42% | 8.77% | 8.54% | 9.75% | 8.86% | N/A | 8.81% | 8.82%
Param | 0.32 | 0.37 | 0.31 | 0.21 | 0.40 | 0.75 | 0.62 | 1.46 | 25.27 | N/A | 0.30 | 0.12
Speed | 25.55 | 102.77 | 30.82 | 191.13 | 187.48 | 28.92 | 21.66 | 13.50 | 59.16 | N/A | 29.58 | 5.78
PEMS04 | MAE | 19.76 | 19.66 | 18.80 | 18.84 | 18.32 | 19.28 | 19.13 | 22.98 | 21.32 | 18.32 | 19.21 | 18.35
RMSE | 31.51 | 31.18 | 30.14 | 30.48 | 29.89 | 31.02 | 31.03 | 36.00 | 33.55 | 29.91 | 32.30 | 29.86
MAPE | 13.48% | 13.45% | 13.19% | 12.92% | 12.51% | 13.18% | 13.22% | 16.56% | 14.85% | 12.60% | 13.05% | 12.50%
WAPE | 8.98% | 8.94% | 8.55% | 8.57% | 8.33% | 8.77% | 8.70% | 10.45% | 9.70% | 8.33% | 8.73% | 8.34%
Param | 0.30 | 0.37 | 0.31 | 0.21 | 0.40 | 0.75 | 0.55 | 1.35 | 16.42 | 25.47 | 0.27 | 0.12
Speed | 14.21 | 57.67 | 16.88 | 117.09 | 124.82 | 16.04 | 12.01 | 7.51 | 29.38 | 745.62 | 16.87 | 3.54
PEMS07 | MAE | 22.25 | 21.16 | 20.44 | 20.04 | 19.49 | 20.68 | 21.01 | 22.50 | 22.47 | N/A | 20.59 | 19.61
RMSE | 35.83 | 34.15 | 33.38 | 32.86 | 32.59 | 34.45 | 34.14 | 36.41 | 35.42 | N/A | 34.86 | 32.69
MAPE | 9.47% | 9.02% | 8.71% | 8.63% | 8.09% | 8.77% | 8.92% | 9.57% | 9.56% | N/A | 8.61% | 8.31%
WAPE | 7.11% | 6.77% | 6.54% | 6.41% | 6.23% | 6.61% | 6.72% | 7.19% | 7.18% | N/A | 6.58% | 6.27%
Param | 0.59 | 0.37 | 0.32 | 0.25 | 0.41 | 0.75 | 1.37 | 3.44 | 27.18 | N/A | 0.57 | 0.14
Speed | 67.14 | 324.75 | 94.51 | 634.47 | 748.51 | 93.79 | 60.59 | 50.93 | 512.84 | N/A | 73.45 | 13.63
PEMS08 | MAE | 16.19 | 15.23 | 14.67 | 14.77 | 14.10 | 15.78 | 15.25 | 16.90 | 16.92 | 14.00 | 15.39 | 14.21
RMSE | 25.51 | 24.17 | 23.55 | 23.81 | 23.36 | 24.76 | 24.22 | 26.30 | 26.68 | 23.41 | 24.80 | 23.35
MAPE | 10.82% | 10.21% | 9.46% | 9.77% | 9.33% | 10.42% | 10.66% | 11.89% | 10.88% | 9.50% | 9.91% | 9.32%
WAPE | 6.98% | 6.56% | 6.32% | 6.36% | 6.07% | 6.80% | 6.57% | 7.28% | 7.29% | 6.03% | 6.63% | 6.12%
Param | 0.23 | 0.37 | 0.31 | 0.20 | 0.39 | 0.15 | 0.35 | 1.15 | 17.25 | 26.56 | 0.20 | 0.12
Speed | 7.46 | 35.36 | 10.34 | 138.65 | 109.42 | 11.33 | 9.57 | 6.64 | 17.49 | 420.86 | 10.98 | 3.38

Table 8. LTSF on PEMS04, PEMS08, ETTh1, ETTh2, ETTm1, ETTm2, Electricity,
Weather, and ExchangeRate (ER) datasets.

Data | Metrics | Informer | Autoformer | Pyraformer | FEDformer | Triformer | Crossformer | PatchTST | Linear | DLinear | NLinear
---|---|---|---|---|---|---|---|---|---|---|---
PEMS04 | MAE | 27.94 | 34.72 | 25.49 | 26.89 | 23.81 | 26.75 | 25.72 | 37.42 | 37.52 | 37.62
RMSE | 44.74 | 50.33 | 41.74 | 41.46 | 39.42 | 45.24 | 40.13 | 62.14 | 62.21 | 62.38
WAPE | 12.84% | 14.81% | 11.72% | 12.39% | 10.95% | 12.31% | 13.35% | 17.22% | 17.26% | 11.35%
Param | 12.40 | 12.07 | 218.42 | 18.36 | 1.69 | 13.51 | 2.34 | 0.11 | 0.23 | 0.11
Speed | 28.62 | 137.61 | 25.22 | 147.75 | 313.52 | 338.94 | 147.42 | 21.13 | 21.24 | 21.27
PEMS08 | MAE | 26.92 | 33.75 | 22.03 | 25.14 | 18.74 | 21.75 | 19.86 | 34.04 | 34.15 | 34.11
RMSE | 43.79 | 51.09 | 38.39 | 39.17 | 31.03 | 36.86 | 33.44 | 57.07 | 57.18 | 57.26
WAPE | 11.63% | 15.37% | 9.52% | 10.87% | 8.13% | 9.40% | 8.34% | 14.71% | 14.76% | 14.74%
Param | 11.91 | 11.37 | 123.93 | 17.66 | 1.64 | 12.53 | 2.34 | 0.11 | 0.23 | 0.11
Speed | 23.34 | 81.54 | 20.94 | 147.89 | 200.50 | 224.90 | 86.81 | 12.60 | 12.78 | 13.04
ETTh1 | MAE | 2.92 | 1.74 | 2.68 | 1.71 | 1.80 | 1.83 | 1.60 | 1.60 | 1.58 | 1.59
RMSE | 4.60 | 3.12 | 4.26 | 3.15 | 3.31 | 3.19 | 3.08 | 3.08 | 3.06 | 3.09
WAPE | 62.87% | 37.61% | 57.75% | 36.89% | 38.72% | 39.44% | 34.49% | 34.47% | 34.06% | 34.29%
Param | 11.33 | 10.54 | 11.59 | 16.30 | 1.59 | 11.36 | 0.24 | 0.11 | 0.23 | 0.11
Speed | 14.18 | 39.37 | 11.98 | 43.44 | 26.24 | 11.91 | 3.90 | 1.25 | 1.35 | 1.28
ETTh2 | MAE | 7.12 | 3.33 | 7.17 | 3.27 | 3.65 | 3.72 | 2.97 | 3.18 | 3.13 | 3.16
RMSE | 8.79 | 4.91 | 9.00 | 4.93 | 5.19 | 5.46 | 4.76 | 5.04 | 5.00 | 5.06
WAPE | 47.44% | 22.17% | 47.76% | 21.78% | 24.29% | 24.76% | 19.78% | 21.19% | 20.85% | 21.09%
Param | 11.33 | 10.54 | 11.59 | 16.83 | 1.59 | 11.36 | 0.24 | 0.11 | 0.23 | 0.11
Speed | 24.04 | 39.29 | 20.85 | 71.80 | 29.71 | 11.89 | 3.76 | 1.25 | 1.36 | 1.28
ETTm1 | MAE | 2.37 | 1.93 | 2.45 | 1.53 | 1.57 | 1.73 | 1.37 | 1.39 | 1.38 | 1.38
RMSE | 4.20 | 3.67 | 3.92 | 2.89 | 2.94 | 3.01 | 2.78 | 2.80 | 2.80 | 2.81
WAPE | 51.16% | 41.69% | 52.79% | 33.00% | 33.83% | 37.52% | 29.60% | 30.02% | 29.80% | 29.86%
Param | 11.33 | 10.54 | 11.59 | 16.83 | 1.59 | 11.79 | 0.24 | 0.11 | 0.23 | 0.11
Speed | 57.32 | 121.91 | 46.57 | 235.17 | 108.11 | 98.13 | 14.77 | 4.35 | 4.79 | 4.40
ETTm2 | MAE | 5.84 | 2.74 | 6.19 | 2.70 | 3.03 | 2.99 | 2.43 | 2.52 | 2.49 | 2.49
RMSE | 7.90 | 4.58 | 7.54 | 4.54 | 4.44 | 4.42 | 4.09 | 4.24 | 4.23 | 4.21
WAPE | 38.97% | 18.27% | 41.28% | 17.99% | 20.26% | 19.95% | 16.21% | 16.80% | 16.63% | 16.60%
Param | 11.33 | 10.54 | 11.59 | 16.30 | 1.59 | 11.60 | 0.24 | 0.11 | 0.23 | 0.11
Speed | 57.07 | 93.04 | 85.45 | 175.20 | 107.44 | 79.77 | 14.67 | 4.35 | 4.66 | 4.48
Electricity | MAE | 325.53 | 295.98 | 335.23 | 317.20 | 275.42 | 283.86 | 253.57 | 256.60 | 250.08 | 251.80
RMSE | 2938.56 | 2933.97 | 2761.75 | 2935.53 | 2968.97 | 2732.70 | 2881.53 | 2896.04 | 2883.63 | 2892.13
WAPE | 12.22% | 11.11% | 12.58% | 11.91% | 10.33% | 10.56% | 9.51% | 9.62% | 9.38% | 9.44%
Param | 12.45 | 12.14 | 228.20 | 17.91 | 1.69 | 1.20 | 0.24 | 0.11 | 0.23 | 0.11
Speed | 51.00 | 69.04 | 70.86 | 113.94 | 484.90 | 198.54 | 95.92 | 40.92 | 39.21 | 40.71
Weather | MAE | 12.88 | 29.37 | 38.94 | 15.61 | 11.29 | 11.36 | 10.85 | 12.25 | 12.08 | 12.02
RMSE | 41.57 | 84.75 | 142.50 | 44.74 | 40.74 | 41.27 | 41.70 | 43.22 | 43.35 | 43.32
WAPE | 7.70% | 17.56% | 23.28% | 9.33% | 6.75% | 6.79% | 6.49% | 7.32% | 7.22% | 7.19%
Param | 11.38 | 10.61 | 21.30 | 16.90 | 1.59 | 11.46 | 2.21 | 0.11 | 0.23 | 0.11
Speed | 78.30 | 99.51 | 50.52 | 503.17 | 120.21 | 107.23 | 42.72 | 8.19 | 8.73 | 8.38
ER | MAE | 0.0611 | 0.0366 | 0.0632 | 0.0376 | 0.0367 | 0.0504 | 0.0332 | 0.0352 | 0.0350 | 0.0322
RMSE | 0.0803 | 0.0568 | 0.0870 | 0.0578 | 0.0533 | 0.0732 | 0.0525 | 0.0550 | 0.0547 | 0.0508
WAPE | 8.0355% | 4.8152% | 8.3192% | 4.9405% | 4.8431% | 6.6406% | 4.3804% | 4.6343% | 4.6086% | 4.2450%
Param | 11.3300 | 10.5400 | 12.2600 | 16.3100 | 1.5900 | 0.7800 | 0.2400 | 0.1100 | 0.2300 | 0.1100
Speed | 8.6000 | 13.7200 | 13.0300 | 26.2000 | 15.9800 | 6.6500 | 2.3800 | 0.9300 | 0.9500 | 0.9100

We summarize our findings as follows. On the one hand, benefiting from its
strong capacities, Transformer-based architectures are far more powerful than
Linear-based models when data has clear and stable patterns. On the other
hand, models with less inductive bias (Dosovitskiy et al., 2021) (e.g., models
based on vanilla Transformer (Nie et al., 2023) or MLPs) usually perform
better when there is no explicit pattern. Moreover, many recent studies have
claimed themselves as general MTS forecasting models, but we believe that
effective general models should first perform well on data with stable and
clear patterns, and then explore modeling time series with less clear
patterns.

###  6.3. Delving into Spatial Dependencies

In this part, we discuss when and how to model the spatial dependencies, and
verify the hypotheses in Section 5.2. We select four datasets, where METR-LA
(LA for short) and PEMS-BAY (BAY for short) are with high spatial
indistinguishability (i.e., $r_{1}$ and $r_{2}$ in Section 5.2), while
ExchangeRate (ER for short) and ETTm1 are with very low spatial
indistinguishability. We choose two baseline models that adopt different
approaches to model spatial dependencies, STID (Shao et al., 2022b) and AGCRN
(Bai et al., 2020). STID designs trainable spatial identity embeddings, while
AGCRN adopts GCN with a graph structure learning module. Additionally, we
remove the spatial modeling components for each of them, forming two variants
STID∗ and AGCRN∗, where STID∗ removes the spatial identities and AGCRN∗ forces
the adjacency matrix to be the identity matrix.

The results are shown in Table 6. On datasets with significant spatial
indistinguishability, both trainable spatial identity embeddings and GCNs can
lead to significant performance gains. Conversely, on datasets with low
spatial indistinguishability, adding these spatial modeling components even
degrades the performance, suggesting that modeling spatial dependencies (or
named cross-dimension dependencies) on these datasets may be not necessary.

Based on the above discussion, we conclude that spatial indistinguishability
is one of the essences of spatial dependencies and that we do not always need
to model spatial dependencies. When there is significant spatial
indistinguishability in the data, it is necessary to adopt spatial modeling
approaches, e.g., GCN, normalization (Deng et al., 2021), and spatial identity
(Shao et al., 2022b), to improve performance. On the contrary, on datasets
without significant spatial indistinguishability, designing spatial modeling
modules needs to be done with extreme care, as this may cause performance
degradation.

###  6.4. How Much Progress Have We Made?

In the preceding sections, we thoroughly examined the impact of heterogeneity
among datasets on various methodologies. In this section, we conduct a
comprehensive analysis of the performance and efficiency of baseline models
using rich datasets, aiming to see how much progress we have made.

First, we compare LTSF methods on datasets with and without clear and stable
patterns. The experimental results are shown in Table 8. When dealing with
datasets without clear and stable patterns, the state-of-the-art Transformer-
based method and the Linear-based method exhibit comparable performance.
Considering the simplicity of Linear models, we believe that designing new
training strategies or engaging in feature engineering to address distribution
drift or ambiguous patterns poses a more noteworthy challenge than designing
more complex time series forecasting models. Moreover, on datasets with clear
and stable patterns, we are surprised that some recent works struggle to
outperform the earliest baseline, Informer (Zhou et al., 2021). Considering
that making predictions on such datasets should be a more straightforward
task, this raises concerns that the architectures of existing LTSF models
might have been over-fitted to LTSF commonly-used datasets like ETT,
Electricity, Weather, and ExchangeRate.

Second, we compare rich STF methods on datasets with significant spatial
indistinguishability (i.e., spatial dependency). Table 7 exhibits the results
of our experiments. Benefiting from the incorporation of prior knowledge,
prior-graph-based methods generally perform better than latent-graph-based or
non-graph-based methods. Furthermore, it is apparent that learning the graph
structure is a very challenging task. Among various methods, only MTGNN (Wu et
al., 2020) and STEP (Shao et al., 2022a) have learned effective graph
structures, which does not significantly degrade the prediction performance.
Overall, it’s obvious that more intricate network structures yield very
limited improvement.  For example, although D2STGNN (Shao et al., 2022c)
published in 2022 is SOTA for STF predictions, its MAE on the METR-LA dataset
is only 6% higher than the Graph WaveNet (Wu et al., 2019) published in 2019.
In addition, it is even more surprising that Graph WaveNet (Wu et al., 2019)
and its variant MTGNN are still able to significantly outperform many newer
works, such as StemGNN (Cao et al., 2020), GTS (Shang et al., 2021), and so on
(Fang et al., 2021; Chen et al., 2021; Choi et al., 2022). Therefore, we
propose that compared to improving prediction accuracy through complex models,
it is more valuable to focus on other important and challenging issues, such
as efficiency, graph structure learning, etc. For example, STNorm and STID are
highly efficient and have achieved satisfactory results on most datasets.

###  6.5. Road Map and Future Work

Based on the aforementioned discussion, we present a straightforward road map
for selecting or designing MTS prediction models, along with noteworthy
research challenges. The road map is depicted in Figure 6. For a given MTS
dataset, patterns in the temporal dimension should be examined first. For data
exhibiting significant distribution drift or lacking clear patterns, unbiased
or weakly biased models should be chosen, e.g., linear layers, MLPs, and the
vanilla Transformer. If the data displays clear and stable patterns, powerful
sequence models are a more reasonable option at this point, e.g., TCN, RNN,
and advanced Transformer architectures. Additionally, based on such data, we
need to investigate whether there is significant spatial indistinguishability
on the spatial dimension. If exists, a spatial dependency modeling module is
required. Alternative approaches include graph convolution, normalization, and
spatial identity information. Moreover, we recommend LTSF-Linear (Zeng et al.,
2023) and STID (Shao et al., 2022b) as baselines. Given their simplicity and
efficiency, we believe that more complex LTSF or STF models are only effective
if they significantly outperform them.

Overall, current efforts are mainly focused on designing complex sequence
models or spatial-temporal models. These endeavors are effective only when the
data exhibits strong patterns.  However, unlike image data and natural
language whose patterns are usually consistent and stable, time series data
are greatly affected by noise and external factors, resulting in distribution
drift or unpredictable changes widely existing in most cases. Therefore, we
believe that more attention should be paid to more valuable directions such as
distribution drift modeling, feature engineering, uncertainty estimation, etc.
For instance, adopting online learning, transfer learning, and similar
techniques to address distribution drift issues, and studying the
predictability of time series are urgent topics.

Figure 6. Road map for selecting or designing MTS models.

##  7\. Conclusion

In this study, we address the controversies surrounding benchmarking and
technical approaches in the MTS forecasting field, shedding light on the
advancements achieved. Firstly, we introduce a novel benchmark BasicTS, which
is designed to enable fair and reasonable comparisons in MTS forecasting. By
adopting a unified training pipeline, BasicTS addresses the inconsistent
performance issue, and provides more reasonable evaluation methods. Secondly,
we delve into the heterogeneity among MTS datasets. On the temporal dimension,
we categorize datasets into those exhibiting clear and stable patterns,
significant distribution drift, and unclear patterns. On the spatial
dimension, we devise metrics to quantify spatial dependencies, and divide
datasets into those with and without significant spatial indistinguishability.
We emphasize that many conclusions drawn in prior research hold validity only
within certain types of data, and blindly adopting these conclusions can lead
researchers to draw biased inferences. Additionally, leveraging the BasicTS
and extensive MTS datasets, we conduct a fair and comprehensive performance
and efficiency comparison to figure out how much progress we have made.
Furthermore, we offer a straightforward road map for selecting or designing
MTS forecasting models, along with noteworthy research challenges.

## References

  * (1)
  * Ariyo et al. (2014) Adebiyi Ariyo Ariyo, Aderemi Oluyinka Adewumi, and Charles K. Ayo. 2014.  Stock Price Prediction Using the ARIMA Model. In _UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, UKSim 2014, Cambridge, United Kingdom, March 26-28, 2014_ , David Al-Dabass, Alessandra Orsoni, Richard J. Cant, Jasmy Yunus, Zuwairie Ibrahim, and Ismail Saad (Eds.). IEEE, 106–112.  https://doi.org/10.1109/UKSim.2014.67
  * Bai et al. (2020) Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. 2020.  Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Bi et al. (2023) Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023.  Accurate medium-range global weather forecasting with 3D neural networks.  _Nature_ 619, 7970 (01 Jul 2023), 533–538.  https://doi.org/10.1038/s41586-023-06185-3
  * Cao et al. (2020) Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020.  Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2001) Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. 2001.  Freeway performance measurement system: mining loop detector data.  _Transportation Research Record_ (2001).
  * Chen et al. (2021) Yuzhou Chen, Ignacio Segovia-Dominguez, and Yulia R. Gel. 2021.  Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecasting. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 1684–1694.  http://proceedings.mlr.press/v139/chen21o.html
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In _Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014_. Association for Computational Linguistics, 103–111.
  * Choi et al. (2022) Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and Noseong Park. 2022.  Graph Neural Controlled Differential Equations for Traffic Forecasting. In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_. AAAI Press, 6367–6374.  https://ojs.aaai.org/index.php/AAAI/article/view/20587
  * Cirstea et al. (2022) Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. 2022.  Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_. ijcai.org, 1994–2001.
  * Defferrard et al. ([n. d.]) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. [n. d.].  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (Eds.). 3837–3845.
  * Deng et al. (2021) Jinliang Deng, Xiusi Chen, Renhe Jiang, Xuan Song, and Ivor W. Tsang. 2021.  ST-Norm: Spatial and Temporal Normalization for Multi-variate Time Series Forecasting. In _KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021_. ACM, 269–278.
  * Deng et al. (2023) Jinliang Deng, Xiusi Chen, Renhe Jiang, Du Yin, Yi Yang, Xuan Song, and Ivor W. Tsang. 2023.  Learning Structured Components: Towards Modular and Interpretable Multivariate Time Series Forecasting.  _CoRR_ abs/2305.13036 (2023).  https://doi.org/10.48550/arXiv.2305.13036 arXiv:2305.13036
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Drucker et al. (1996) Harris Drucker, Christopher J. C. Burges, Linda Kaufman, Alexander J. Smola, and Vladimir Vapnik. 1996.  Support Vector Regression Machines. In _Advances in Neural Information Processing Systems 9, NIPS, Denver, CO, USA, December 2-5, 1996_ , Michael Mozer, Michael I. Jordan, and Thomas Petsche (Eds.). MIT Press, 155–161.  http://papers.nips.cc/paper/1238-support-vector-regression-machines
  * Fang et al. (2021) Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. 2021.  Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting. In _KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021_ , Feida Zhu, Beng Chin Ooi, and Chunyan Miao (Eds.). ACM, 364–373.  https://doi.org/10.1145/3447548.3467430
  * Friedman (2001) Jerome H Friedman. 2001.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ (2001), 1189–1232.
  * Gardner Jr (1985) Everette S Gardner Jr. 1985.  Exponential smoothing: The state of the art.  _Journal of forecasting_ 4, 1 (1985), 1–28.
  * Garland et al. (2014) Joshua Garland, Ryan James, and Elizabeth Bradley. 2014.  Model-free quantification of time-series predictability.  _Physical Review E_ 90, 5 (2014), 052910.
  * Guo et al. (2019) Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019.  Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. In _AAAI_.
  * Guo et al. (2022) Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. 2022.  Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting.  _IEEE Trans. Knowl. Data Eng._ 34, 11 (2022), 5415–5428.
  * Hyndman and Athanasopoulos (2018) Rob J Hyndman and George Athanasopoulos. 2018.  _Forecasting: principles and practice_.  OTexts.
  * Jiang et al. (2023) Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, and Jingyuan Wang. 2023.  PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_ , Brian Williams, Yiling Chen, and Jennifer Neville (Eds.). AAAI Press, 4365–4373.  https://ojs.aaai.org/index.php/AAAI/article/view/25556
  * Jiang et al. (2021) Renhe Jiang, Du Yin, Zhaonan Wang, Yizhuo Wang, Jiewen Deng, Hangchen Liu, Zekun Cai, Jinliang Deng, Xuan Song, and Ryosuke Shibasaki. 2021.  DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_ , Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 4515–4525.  https://doi.org/10.1145/3459637.3482000
  * Jin et al. (2022) Guangyin Jin, Fuxian Li, Jinlei Zhang, Mudan Wang, and Jincai Huang. 2022.  Automated dilated spatio-temporal synchronous graph modeling for traffic prediction.  _IEEE Transactions on Intelligent Transportation Systems_ (2022).
  * Jin et al. (2023) Guangyin Jin, Yuxuan Liang, Yuchen Fang, Jincai Huang, Junbo Zhang, and Yu Zheng. 2023.  Spatio-temporal graph neural networks for predictive learning in urban computing: A survey.  _arXiv preprint arXiv:2303.14483_ (2023).
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The Efficient Transformer. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lai et al. (2018) Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018.  Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 95–104.  https://doi.org/10.1145/3209978.3210006
  * Li et al. (2021) Fuxian Li, Jie Feng, Huan Yan, Guangyin Jin, Fan Yang, Funing Sun, Depeng Jin, and Yong Li. 2021.  Dynamic graph convolutional recurrent network for traffic prediction: Benchmark and solution.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ (2021).
  * Li and Zhu (2021) Mengzhang Li and Zhanxing Zhu. 2021.  Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecasting. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4189–4196.
  * Li et al. (2019) Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_. 5244–5254.
  * Li et al. (2018) Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Liu et al. (2022a) Dachuan Liu, Jin Wang, Shuo Shang, and Peng Han. 2022a.  MSDR: Multi-Step Dependency Relation Networks for Spatial Temporal Forecasting. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_ , Aidong Zhang and Huzefa Rangwala (Eds.). ACM, 1042–1050.  https://doi.org/10.1145/3534678.3539397
  * Liu et al. (2023a) Hangchen Liu, Zheng Dong, Renhe Jiang, Jiewen Deng, Jinliang Deng, Quanjun Chen, and Xuan Song. 2023a.  Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting. In _Proceedings of the 32st ACM International Conference on Information & Knowledge Management_. ACM.
  * Liu et al. (2021) Minhao Liu, Ailing Zeng, Qiuxia Lai, and Qiang Xu. 2021.  Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction.  _CoRR_ abs/2106.09305 (2021).  arXiv:2106.09305 https://arxiv.org/abs/2106.09305
  * Liu et al. (2022b) Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. 2022b.  Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Liu et al. (2023b) Xu Liu, Yuxuan Liang, Chao Huang, Hengchang Hu, Yushi Cao, Bryan Hooi, and Roger Zimmermann. 2023b.  Do We Really Need Graph Neural Networks for Traffic Forecasting?  _CoRR_ abs/2301.12603 (2023).  https://doi.org/10.48550/arXiv.2301.12603 arXiv:2301.12603
  * Liu et al. (2023c) Xu Liu, Yuxuan Liang, Chao Huang, Hengchang Hu, Yushi Cao, Bryan Hooi, and Roger Zimmermann. 2023c.  Do We Really Need Graph Neural Networks for Traffic Forecasting?  _CoRR_ abs/2301.12603 (2023).  https://doi.org/10.48550/arXiv.2301.12603 arXiv:2301.12603
  * Liu et al. (2023d) Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. 2023d.  LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting.  _CoRR_ abs/2306.08259 (2023).  https://doi.org/10.48550/arXiv.2306.08259 arXiv:2306.08259
  * Nie et al. (2023) Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023.  A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Pan et al. (2019) Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. 2019.  Urban Traffic Prediction from Spatio-Temporal Data Using Deep Meta Learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_. ACM, 1720–1730.
  * Shang et al. (2021) Chao Shang, Jie Chen, and Jinbo Bi. 2021.  Discrete Graph Structure Learning for Forecasting Multiple Time Series. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Shao et al. (2022b) Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. 2022b.  Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022_. ACM, 4454–4458.
  * Shao et al. (2022a) Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. 2022a.  Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_. ACM, 1567–1577.
  * Shao et al. (2022c) Zezhi Shao, Zhao Zhang, Wei Wei, Fei Wang, Yongjun Xu, Xin Cao, and Christian S. Jensen. 2022c.  Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting.  _Proc. VLDB Endow._ 15, 11 (2022), 2733–2746.
  * Shi et al. (2015) Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 802–810.
  * Song et al. (2020) Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. 2020.  Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In _Proceedings of the AAAI_. 914–921.
  * Sun et al. (2023) Luming Sun, Shijin Gong, Tieying Zhang, Fuxin Jiang, Zhibing Zhao, Jianjun Chen, and Xinyu Zhang. 2023.  SUFS: A Generic Storage Usage Forecasting Service Through Adaptive Ensemble Learning. In _39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023_. IEEE, 3168–3181.  https://doi.org/10.1109/ICDE55515.2023.00243
  * Sutskever et al. (2014a) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014a.  Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_. 3104–3112.
  * Sutskever et al. (2014b) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014b.  Sequence to Sequence Learning with Neural Networks. In _NeurIPS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_. 5998–6008.
  * Wang et al. (2021) Jingyuan Wang, Jiawei Jiang, Wenjun Jiang, Chao Li, and Wayne Xin Zhao. 2021.  LibCity: An Open Library for Traffic Prediction. In _SIGSPATIAL ’21: 29th International Conference on Advances in Geographic Information Systems, Virtual Event / Beijing, China, November 2-5, 2021_ , Xiaofeng Meng, Fusheng Wang, Chang-Tien Lu, Yan Huang, Shashi Shekhar, and Xing Xie (Eds.). ACM, 145–148.  https://doi.org/10.1145/3474717.3483923
  * Wang et al. (2023) Xu Wang, Pengfei Gu, Pengkun Wang, Binwu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. 2023.  Graph-Free Learning in Graph-Structured Data: A More Efficient and Accurate Spatiotemporal Learning Perspective.  _CoRR_ abs/2301.11742 (2023).  https://doi.org/10.48550/arXiv.2301.11742 arXiv:2301.11742
  * Wang et al. (2020) Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. 2020.  Traffic Flow Prediction via Spatial Temporal Graph Neural Network. In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_. ACM / IW3C2, 1082–1092.
  * Wu et al. (2023) Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023.  TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=ju_Uqw384Oq
  * Wu et al. (2021) Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021.  Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_. 22419–22430.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020.  Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_. ACM, 753–763.
  * Wu et al. (2019) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019.  Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_. ijcai.org, 1907–1913.
  * Xu et al. (2019) Paiheng Xu, Likang Yin, Zhongtao Yue, and Tao Zhou. 2019.  On predictability of time series.  _Physica A: Statistical Mechanics and its Applications_ 523 (2019), 345–351.
  * Yao et al. (2018) Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. 2018.  Deep Multi-View Spatial-Temporal Network for Taxi Demand Prediction. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, 2588–2595.  https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16069
  * Yu et al. (2018) Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018.  Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_ , Jérôme Lang (Ed.). ijcai.org, 3634–3640.
  * Yu and Koltun (2016) Fisher Yu and Vladlen Koltun. 2016.  Multi-Scale Context Aggregation by Dilated Convolutions. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Zeng et al. (2023) Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023.  Are Transformers Effective for Time Series Forecasting?  _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zhang and Yan (2023) Yunhao Zhang and Junchi Yan. 2023.  Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Zheng et al. (2020) Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. 2020.  GMAN: A Graph Multi-Attention Network for Traffic Prediction. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_. AAAI Press, 1234–1241.
  * Zhou et al. (2021) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021.  Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 11106–11115.
  * Zhou et al. (2022a) Tian Zhou, Ziqing Ma, Xue Wang, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. 2022a.  FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting. In _NeurIPS_.
  * Zhou et al. (2022b) Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022b.  FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ _(Proceedings of Machine Learning Research, Vol. 162)_. PMLR, 27268–27286.
  * Zhu et al. (2021) Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, and Danai Koutra. 2021.  Graph Neural Networks with Heterophily. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 11168–11176.  https://ojs.aaai.org/index.php/AAAI/article/view/17332
  * Zhu et al. (2020) Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).