# PDMLP: Patch-based Decomposed MLP for Long-Term Time Series Forecasting

Peiwang Tang
IFLYTEK
Hefei, Anhui 230088
<EMAIL_ADDRESS>
&Weitai Zhang
University of Science and Technology of China
Hefei, Anhui 230026
<EMAIL_ADDRESS>

###### Abstract

Recent studies have attempted to refine the Transformer architecture to
demonstrate its effectiveness in Long-Term Time Series Forecasting (LTSF)
tasks. Despite surpassing many linear forecasting models with ever-improving
performance, we remain skeptical of Transformers as a solution for LTSF. We
attribute the effectiveness of these models largely to the adopted Patch
mechanism, which enhances sequence locality to an extent yet fails to fully
address the loss of temporal information inherent to the permutation-invariant
self-attention mechanism. Further investigation suggests that simple linear
layers augmented with the Patch mechanism may outperform complex Transformer-
based LTSF models. Moreover, diverging from models that use channel
independence, our research underscores the importance of cross-variable
interactions in enhancing the performance of multivariate time series
forecasting. The interaction information between variables is highly valuable
but has been misapplied in past studies, leading to suboptimal cross-variable
models. Based on these insights, we propose a novel and simple Patch-based
Decomposed MLP (PDMLP) for LTSF tasks. Specifically, we employ simple moving
averages to extract smooth components and noise-containing residuals from time
series data, engaging in semantic information interchange through channel
mixing and specializing in random noise with channel independence processing.
The PDMLP model consistently achieves state-of-the-art results on several
real-world datasets. We hope this surprising finding will spur new research
directions in the LTSF field and pave the way for more efficient and concise
solutions.

##  1 Introduction

Figure 1: The self-attention scores of a 2-layer Transformer with different
Patch size trained on ETTh1. We follow the setup of PathcTST Nie et al.
(2022), retaining only the Encoder while replacing the Decoder with a simple
MLP, and using a channel independent approach. A patch size of 1 is equivalent
to the original Transformer, indicating that time series data often exhibits a
trend of being segmented into patch Zhang & Yan (2022); Tang & Zhang (2023),
and an increase in patch size can mitigate this to some extent.

Long-term Time Series Forecasting (LTSF) is a critical area of research in the
fields of statistics and machine learning, aimed at using historical data to
predict the future of one or more variables over a certain period Oreshkin et
al. (2019); Gong et al. (2023); Lin et al. (2023). Time series data are
organized in chronological order and reveal underlying dynamic patterns, be
they cyclical or non-cyclical Cryer (1986). Such forecasts play a vital role
in various sectors, including biomedicine Liu et al. (2018), economics and
finance Patton (2013), electricity Zhou et al. (2021), and transportation Yin
& Shang (2016).

Multivariate Time Series (MTS) consist of multiple variables recorded at the
same time point, where each dimension may represent an individual univariate
time series or be considered as a signal with multiple channels. With the
advancement of deep learning, a number of models have been developed to
enhance the performance of MTS forecasting Borovykh et al. (2017); Bai et al.
(2018); Liu et al. (2020). In particular, recent models based on the
Transformer Vaswani et al. (2017) architecture have demonstrated significant
potential in capturing long-term dependencies Radford et al. (2018); Devlin et
al. (2018). In recent years, Transformers have emerged as a leading
architecture in time series forecasting Lim et al. (2021); Liu et al. (2022);
Shen et al. (2023), initially applied in the field of Natural Language
Processing (NLP) Radford et al. (2019); Brown et al. (2020) and subsequently
expanded to other domains such as Computer Vision (CV) Liu et al. (2021b);
Feichtenhofer et al. (2022) through the adaptation of the Patch method,
becoming a versatile framework.

Originally, these models utilized a channel mixing approach Li et al. (2023b),
which involves projecting vectors from different channels recorded at the same
time point into an embedding space and blending the information Zhou et al.
(2021); Tang & Zhang (2023) ,and many models adopt the idea of time series
decomposition Wu et al. (2021); Zhou et al. (2022); Wang et al. (2023).
Nonetheless, recent studies have shown that a channel independence model might
be more effective Nie et al. (2022); Han et al. (2023). Channel independence
means that each input token contains information from only one channel, and
intuitively, the models treat the MTS as separate univariate series for
individual processing. These studies suggest that for LTSF tasks, an emphasis
on channel independence may be more effective than channel mixing strategies
Gong et al. (2023).

Moreover, the effectiveness of Transformers in LTSF tasks has been challenged
by a reconsideration of the Multi-Layer Perceptron (MLP) Zeng et al. (2023);
Ekambaram et al. (2023); Chen et al. (2023), whose surprisingly simple
architecture has outperformed all previous Transformer models. This raises a
compelling question: Are Transformers effective for LTSF tasks? In response to
this skepticism, recent Transformer-based models have adopted patch-based
representations and achieved noteworthy performance in LTSF Nie et al. (2022);
Chen et al. (2024).

This study raises three critical inquiries:

  * •

Is the method of channel mixing truly ineffective in MTS forecasting?

  * •

Can simply decomposing the original time series truly better predict the trend
and seasonal components?

  * •

Does the impressive performance of the Patch-based Transformer derive from the
inherent strength of the Transformer architecture, or is it merely due to the
use of patch as the input representation?

(a)

(b)

(c)

Figure 2: Experimental results of Patch Transformer on the ETTh2 dataset. (a)
Maintain all other parameters constant, and present the MSE outcomes for four
forecast lengths with only the input length altered. (b) Keeping all other
parameters constant and only altering the size of the patch, the MSE results
for a forecast length of 720 with five different input lengths. (c)
Maintaining all other parameters unchanged and only varying the patch size,
the MSE results for five different $d_{model}$ values with both input and
forecast lengths set to 720.

In this paper, we specifically discuss the three issues mentioned above and
introduce the PDMLP, a novel and concise Patch-based decomposed MLP model
tailored for LTSF task. PDMLP is entirely based on fully connected networks
and incorporates the concept of patch. Specifically, we employ a patch
methodology to embed the original sequences into representational spaces, then
extract the smooth components and noisy residuals of the series using a simple
moving average technique for separate processing. We independently process
stochastic noise across channels and facilitate semantic information
interchange between variables through channel mixing. Upon evaluation on
numerous real-world datasets, the PDMLP model demonstrated state-of-the-art
(SOTA) performance. To be precise, our contributions can be summarized in
three aspects:

  * •

We analyze the effectiveness of patch in time series forecasting and propose a
Multi-Scale Patch Embedding (MPE) approach. Unlike previous embedding methods
that used a single linear layer, MPE is capable of capturing multiscale
relationships between input sequences.

  * •

We introduce a novel entirely MLP-based model, named PDMLP. By utilizing
moving averages, it performs the decomposition of latent vector and adopts a
different approach to channel mixing for semantic information exchange across
variables.

  * •

Our experiments across a wide range of datasets in various fields show that
PDMLP consistently achieves SOTA performance across multiple forecasting
benchmarks. Moreover, we conduct an extensive analysis of Patch-based methods
with the aim of charting new directions for future research in time series
forecasting.

##  2 Patch for Long-Term Time Series Forecasting

Why is Patch effective in time series forecasting? Lee et al. (2023); Zhong et
al. (2023) We have studied a patch-based Transformer model adopting channel
independent mechanisms, and we set the decoder layer as a simple single-layer
MLP for time series forecastine Nie et al. (2022). As shown in the Figure
2(b), we can clearly see that for the same input length, as the size of the
patch increases, the overall Mean Squared Error (MSE) of the model exhibits a
trend of decreasing and then increasing, or decreasing and then stabilizing.
As the input length increases, the patch size required for the model to
achieve optimal performance also gradually increases. In the original input
data, the effect of Attention is so poor that it cannot even outperform a
simple single-layer MLP, which raises the question of whether Attention might
not be the optimal choice for time series modeling to some extent, as it
cannot handle long-term time series well.

Compared to textual data in the NLP field, original time series data contains
many redundant features due to high-frequency sampling, and is easily
influenced by noise to some extent during the sampling process Tang & Zhang
(2022). We believe that for data with excessive noise in the original data,
the original Transformer’s Attention mechanism is not the optimal choice as it
cannot effectively eliminate the noise. Sparse Attention performs better
because this sparse mechanism can effectively mitigate the impact of noise to
some extent Li et al. (2019); Wu et al. (2020); Liu et al. (2021a), but it may
also attenuate the impact of original features.

Patch, on the other hand, compresses the data, reduces the dimensionality of
the input data, and decreases redundant features. Additionally, patch provides
a certain degree of smoothing, which can reduce the influence of outliers to
some extent, and help filter out fluctuations and random noise, retaining more
stable and representative information. In the CV field, patch also works well
in ViT Dosovitskiy et al. (2020), MAE He et al. (2022). Time series data
usually contains patterns at different scales, and patch provides a modeling
of short-term time series, enhancing local information in the sequence,
allowing the model to better learn and capture local features. Therefore, we
believe that the effectiveness of Transformer is not due to the effect of
Attention, but rather due to the presence of patch.

Figure 3: Overall structure of PDMLP. (a) Raw time series of different
variates are independently embedded as tokens. (b)The Feature Decomposition
utilizing moving averages to decompose the tokens into the smooth components
and noisy residuals. (c) The MLP processed sequences in two ways: intra-
variable and inter-variable. (d) A projection maps the latent vectors back to
future sequences.

So, is a larger patch always better? In Figure 2(b), a larger patch does not
seem to achieve better results, which may be partly influenced by the size of
the hidden layer ($d_{model}$). With a larger patch size, more time points are
projected into a fixed-size $d_{model}$, which may lead to excessive
compression. As shown in Figure 2(c), as $d_{model}$ increases, the
performance of the model gradually improves, indicating that larger patch
perform better. In the extreme case, a patch that treats the entire input
sequence as one patch and projects it into a vector is similar to iTransformer
Liu et al. (2023). However, a larger $d_{model}$ indicates that there are more
parameters for the model to learn, which can easily lead to underfitting and
result in decreased model performance.

##  3 PDMLP

The problem of multivariate time series forecasting is to input the historical
observations
$\mathcal{X}=\left\\{x_{1},\cdots,x_{L}|x_{i}\in\mathbb{R}^{M}\right\\}$, and
the output is to predict corresponding future sequence
$\mathcal{X}=\left\\{x_{L+1},\cdots,x_{L+T}|x_{i}\in\mathbb{R}^{M}\right\\}$,
where $L$ and $T$ are the lengths of input and output sequences respectively,
and $M$ is the dimension of variates.

As shown in Figure 3, PDMLP consists of four network components: Multi-Scale
Patch Embedding layer, Feature Decomposition layer, Multi-Layer Perceptron
(MLP) layer, Projection layer. The Multi-Scale Patch Embedding layer embeds
the multivariate time series into latent space. The Feature Decomposition
layer decomposes the latent vector into the smooth components and noisy
residuals, then operate separately through MLP layer. Finally, the latent
vector are mapped back to the feature space through the Projection layer to
obtain the future sequences $\hat{\mathcal{X}}$. Next, we will introduce the
above modules separately.

###  3.1 Multi-Scale Patch Embedding

Time series analysis relies on the accurate identification of local
information within the sequence and optimization of the model. The use of
patch can provide the model with a local view of the time series in the short
term, thereby enhancing the representation of local information within the
sequence and enabling the model to learn and capture these local features more
precisely. However, traditional methods typically employ patch of a single-
scale to embed the raw time series, which makes the model inclined to learn
single-scale temporal relationships while neglecting the multi-scale nature
and complexity of time series data.

In practical applications, the single-scale patch strategy may lead to the
model capturing inaccurate or incomplete local features. This is because the
data characteristics of different time series often have variation, and a
patch of the same scale cannot universally adapt to all types of time series.
For example, a time series containing multiple cyclical patterns, a single-
scale patch cannot effectively identify and learn the cyclical features
present at different frequencies.

To capture local information in more detail and to fully understand the
temporal relationships within the time series, we adopt multi-scale patche.
This includes shorter patch to capture local high-frequency patterns, as well
as longer patch to unearth long-term seasonal, trends and periodic
fluctuations. Through the introduction of multi-scale patch, the model can
flexibly learn representative features over different lengths of time spans,
thus enhancing predictive accuracy and model generalization capabilities.

To decompose the multivariate time series $\mathcal{X}$ into a univariate
series $x$, we have developed a suite of patch $\mathcal{P}$ across various
scales to process $x$. For a particular scale $p\in\mathcal{P}$, $x$ is first
divided into non-overlapping patch where the length of patch corresponds to
$p$, thus the patching process yields the sequence of patch
$x_{p}\in\mathbb{R}^{N\times p}$, where $N$ is the number of patches.
Subsequently, we employ a single-layer linear layer to embed the patch
$x_{p}$, resulting in latent vectors $x_{e}\in\mathbb{R}^{N\times d}$, $d$ is
the embedding dimension, the $d$ corresponding to different scales patch can
be different. These latent vectors are then unfolded and concatenated with
latent vectors obtained from patch at alternative scales, thus assembling the
ultimate embedding vector $X\in\mathbb{R}^{1\times d_{model}}$, $d_{model}$ is
the final embedding dimension input into the model. This multi-scale patch
strategy permits the model to capture and learn the intrinsic dynamics of time
series at different levels, thereby providing more precise insights when
forecasting future trends and patterns.

###  3.2 Feature Decomposition

Time series often contain complex temporal patterns, including seasonal
fluctuations, trends, and other irregular influencing factors. Many studies
attempt to decompose time series into these fundamental components through
decomposition methods, to achieve better understanding and forecasting.
Although decomposition is a powerful tool, traditional methods may encounter
some difficulties when dealing with the raw sequences, especially when there
are complex and mixed patterns in the sequences. These methods often struggle
to accurately separate clear trends and seasonal components, particularly when
the data contains substantial noise or nonlinear components.

In response to this issue, we propose a different idea, instead of directly
decomposing the original sequence, we decompose the sequence’s embedding
vector. Embedding vector are representations formed by mapping time series to
a high-dimensional space, and these representations often capture the core
information of the original data. By decomposing embedding vector, we can
distinguish smoother components and noise-containing residuals, which helps
eliminate the interference of random fluctuations on analysis and forecasting.
In practice, we use the operation of Average Pooling ($AvgPool$) to smooth the
time series, aiming to reduce random fluctuations and noise in the data.
Moreover, in order to maintain the length of the time series unchanged while
smoothing, we applied a padding operation.

$\begin{split}X_{s}&=AvgPool(X)\\\ X_{r}&=X-X_{s}\end{split}$ (1)

$X_{s}$ and $X_{r}$ respectively represent the extracted smooth components and
residual components. By operating on embedding vector rather than the original
sequence, the model can extract and identify the fundamental components of
time series more precisely, enabling a better understanding of the intrinsic
structure of time series, and ultimately, improving the accuracy of time
series forecasting.

Figure 4: Overall structure of MLP layer. (a) The embedded vectors first
interact with the temporal information within the variable through the Intra-
Variable MLP. (b) Then interact with the feature domain information between
variables through the Intra-Variable MLP. (c) Subsequently, they are
multiplied by the input of the Inter-Variable MLP using a dot-product
approach. (d) Finally, they are added to the initial input of the MLP Layer
using skip connections.

###  3.3 MLP layer

In the context of MTS forecasting, the MLP layer alternately applies MLPs
within Intra-Variable (time domain) and Inter-Variable(feature domain) to
enhance predictive performance. The specific architecture, as depicted in
Figure 4, can be described in detail as follows:

Intra-Variable MLP: This component is focused on the identification of time-
correlated patterns within time series. Specifically, it utilizes a network
architecture composed of fully connected layers, nonlinear activation
functions, and dropout. Parameters are applied in the temporal domain and are
shared across inter-variable. Simple linear models are easy to understand and
implement, and have been proven to learn complex time patterns and capture
more complex time dependencies.

Inter-Variable MLP: Complementary to the intra-variable MLP, the inter-
variable MLP aims to model the mutual influences among MTS variables. This
component also consists of fully connected layers, activation functions, and
dropout, applying fully connected layers in the feature domain to achieve
parameter sharing within the intra-variable. To enhance the cross-variable
interactivity, we utilize a dot product mechanism, integrating the dot product
results between the MLP outputs and inputs, which enhances the model’s
nonlinear representational capability.

Residual Connections: Residual connections He et al. (2016) are applied after
each MLP, enabling the model to learn deeper architectures more efficiently.
These connections not only help to mitigate the problems of vanishing
gradients in deep networks but also provide the model with a shortcut path
that ensures key temporal or feature information is not overlooked.

While the minimalist architecture of PDMLP may not appear as eye-catching as
some of the recently proposed, more complex Transformer models, empirical
results indicate that PDMLP exhibits competitive performance, both in terms of
training efficiency and accuracy on standard benchmark tests, relative to SOTA
models.

Table 1: Full results of the Long-Term Time Series Forecasting task. For all
baselines, we adhere to the setting of the iTransformer with an input sequence
length of 96. We conducted comparisons among an array of competitive models
during various forecast horizons, and the outcomes were obtained from the
iTransformer. Avg denotes the average result across all four prediction
lengths. We denote the optimal results with boldface, and the second-best with
an underline for clarity.

Methods | PDMLP | iTransformer | RLinear | PatchTST | Crossformer | TiDE | TimesNet | DLinear | SCINet | FEDformer
---|---|---|---|---|---|---|---|---|---|---
Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
ETTm1 | 96 | 0.316 | 0.357 | 0.334 | 0.368 | 0.355 | 0.376 | 0.329 | 0.367 | 0.404 | 0.426 | 0.364 | 0.387 | 0.338 | 0.375 | 0.345 | 0.372 | 0.418 | 0.438 | 0.379 | 0.419
192 | 0.357 | 0.378 | 0.377 | 0.391 | 0.391 | 0.392 | 0.367 | 0.385 | 0.450 | 0.451 | 0.398 | 0.404 | 0.374 | 0.387 | 0.380 | 0.389 | 0.439 | 0.450 | 0.426 | 0.441
336 | 0.396 | 0.402 | 0.426 | 0.420 | 0.424 | 0.415 | 0.399 | 0.410 | 0.532 | 0.515 | 0.428 | 0.425 | 0.410 | 0.411 | 0.413 | 0.413 | 0.490 | 0.485 | 0.445 | 0.459
720 | 0.462 | 0.439 | 0.491 | 0.459 | 0.487 | 0.450 | 0.454 | 0.439 | 0.666 | 0.589 | 0.487 | 0.461 | 0.478 | 0.450 | 0.474 | 0.453 | 0.595 | 0.550 | 0.543 | 0.490
Avg | 0.382 | 0.394 | 0.407 | 0.410 | 0.414 | 0.407 | 0.387 | 0.400 | 0.513 | 0.496 | 0.419 | 0.419 | 0.400 | 0.406 | 0.403 | 0.407 | 0.485 | 0.481 | 0.448 | 0.452
ETTm2 | 96 | 0.175 | 0.258 | 0.180 | 0.264 | 0.182 | 0.265 | 0.175 | 0.259 | 0.287 | 0.366 | 0.207 | 0.305 | 0.187 | 0.267 | 0.193 | 0.292 | 0.286 | 0.377 | 0.203 | 0.287
192 | 0.240 | 0.302 | 0.250 | 0.309 | 0.246 | 0.304 | 0.241 | 0.302 | 0.414 | 0.492 | 0.290 | 0.364 | 0.249 | 0.309 | 0.284 | 0.362 | 0.399 | 0.445 | 0.269 | 0.328
336 | 0.306 | 0.344 | 0.311 | 0.348 | 0.307 | 0.342 | 0.305 | 0.343 | 0.597 | 0.542 | 0.377 | 0.422 | 0.321 | 0.351 | 0.369 | 0.427 | 0.637 | 0.591 | 0.325 | 0.366
720 | 0.404 | 0.403 | 0.412 | 0.407 | 0.407 | 0.398 | 0.402 | 0.400 | 1.730 | 1.042 | 0.558 | 0.524 | 0.408 | 0.403 | 0.554 | 0.522 | 0.960 | 0.735 | 0.421 | 0.415
Avg | 0.281 | 0.326 | 0.288 | 0.332 | 0.286 | 0.327 | 0.281 | 0.326 | 0.757 | 0.610 | 0.358 | 0.404 | 0.291 | 0.333 | 0.350 | 0.401 | 0.571 | 0.537 | 0.305 | 0.349
ETTh1 | 96 | 0.382 | 0.399 | 0.386 | 0.405 | 0.386 | 0.395 | 0.414 | 0.419 | 0.423 | 0.448 | 0.479 | 0.464 | 0.384 | 0.402 | 0.386 | 0.400 | 0.654 | 0.599 | 0.376 | 0.419
192 | 0.437 | 0.431 | 0.441 | 0.436 | 0.437 | 0.424 | 0.460 | 0.445 | 0.471 | 0.474 | 0.525 | 0.492 | 0.436 | 0.429 | 0.437 | 0.432 | 0.719 | 0.631 | 0.420 | 0.448
336 | 0.475 | 0.450 | 0.487 | 0.458 | 0.479 | 0.446 | 0.501 | 0.466 | 0.570 | 0.546 | 0.565 | 0.515 | 0.491 | 0.469 | 0.481 | 0.459 | 0.778 | 0.659 | 0.459 | 0.465
720 | 0.492 | 0.474 | 0.503 | 0.491 | 0.481 | 0.470 | 0.500 | 0.488 | 0.653 | 0.621 | 0.594 | 0.558 | 0.521 | 0.500 | 0.519 | 0.516 | 0.836 | 0.699 | 0.506 | 0.507
Avg | 0.446 | 0.438 | 0.454 | 0.447 | 0.446 | 0.434 | 0.469 | 0.454 | 0.529 | 0.552 | 0.541 | 0.507 | 0.458 | 0.450 | 0.456 | 0.452 | 0.747 | 0.647 | 0.440 | 0.460
ETTh2 | 96 | 0.288 | 0.340 | 0.297 | 0.349 | 0.288 | 0.338 | 0.302 | 0.348 | 0.745 | 0.584 | 0.400 | 0.440 | 0.340 | 0.374 | 0.333 | 0.387 | 0.707 | 0.621 | 0.358 | 0.397
192 | 0.371 | 0.396 | 0.380 | 0.400 | 0.374 | 0.390 | 0.388 | 0.400 | 0.877 | 0.656 | 0.528 | 0.509 | 0.402 | 0.414 | 0.477 | 0.476 | 0.860 | 0.689 | 0.429 | 0.439
336 | 0.412 | 0.429 | 0.428 | 0.432 | 0.415 | 0.426 | 0.426 | 0.433 | 1.043 | 0.731 | 0.643 | 0.571 | 0.452 | 0.452 | 0.594 | 0.541 | 1.000 | 0.744 | 0.496 | 0.487
720 | 0.421 | 0.441 | 0.427 | 0.445 | 0.420 | 0.440 | 0.431 | 0.446 | 1.104 | 0.763 | 0.874 | 0.679 | 0.462 | 0.468 | 0.831 | 0.657 | 1.249 | 0.838 | 0.463 | 0.474
Avg | 0.373 | 0.401 | 0.383 | 0.407 | 0.374 | 0.398 | 0.387 | 0.407 | 0.942 | 0.684 | 0.611 | 0.550 | 0.414 | 0.427 | 0.559 | 0.515 | 0.954 | 0.723 | 0.437 | 0.449
ECL | 96 | 0.147 | 0.246 | 0.148 | 0.240 | 0.201 | 0.281 | 0.195 | 0.285 | 0.219 | 0.314 | 0.237 | 0.329 | 0.168 | 0.272 | 0.197 | 0.282 | 0.247 | 0.345 | 0.193 | 0.308
192 | 0.166 | 0.261 | 0.162 | 0.253 | 0.201 | 0.283 | 0.199 | 0.289 | 0.231 | 0.322 | 0.236 | 0.330 | 0.184 | 0.289 | 0.196 | 0.285 | 0.257 | 0.355 | 0.201 | 0.315
336 | 0.184 | 0.279 | 0.178 | 0.269 | 0.215 | 0.298 | 0.215 | 0.305 | 0.246 | 0.337 | 0.249 | 0.344 | 0.198 | 0.300 | 0.209 | 0.301 | 0.269 | 0.369 | 0.214 | 0.329
720 | 0.222 | 0.311 | 0.225 | 0.317 | 0.257 | 0.331 | 0.256 | 0.337 | 0.280 | 0.363 | 0.284 | 0.373 | 0.220 | 0.320 | 0.245 | 0.333 | 0.299 | 0.390 | 0.246 | 0.355
Avg |  0.179 | 0.274 | 0.178 | 0.270 | 0.219 | 0.298 | 0.216 | 0.304 | 0.244 | 0.334 | 0.251 | 0.344 | 0.192 | 0.295 | 0.212 | 0.300 | 0.268 | 0.365 | 0.214 | 0.327
Exchange | 96 | 0.085 | 0.204 | 0.086 | 0.206 | 0.093 | 0.217 | 0.088 | 0.205 | 0.256 | 0.367 | 0.094 | 0.218 | 0.107 | 0.234 | 0.088 | 0.218 | 0.267 | 0.396 | 0.148 | 0.278
192 | 0.174 | 0.298 | 0.177 | 0.299 | 0.184 | 0.307 | 0.176 | 0.299 | 0.470 | 0.509 | 0.184 | 0.307 | 0.226 | 0.344 | 0.176 | 0.315 | 0.351 | 0.459 | 0.271 | 0.315
336 | 0.317 | 0.411 | 0.331 | 0.417 | 0.351 | 0.432 | 0.301 | 0.397 | 1.268 | 0.883 | 0.349 | 0.431 | 0.367 | 0.448 | 0.313 | 0.427 | 1.324 | 0.853 | 0.460 | 0.427
720 | 0.820 | 0.681 | 0.847 | 0.691 | 0.886 | 0.714 | 0.901 | 0.714 | 1.767 | 1.068 | 0.852 | 0.698 | 0.964 | 0.746 | 0.839 | 0.695 | 1.058 | 0.797 | 1.195 | 0.695
Avg | 0.349 | 0.398 | 0.360 | 0.403 | 0.378 | 0.417 | 0.367 | 0.404 | 0.940 | 0.707 | 0.370 | 0.413 | 0.416 | 0.443 | 0.354 | 0.414 | 0.750 | 0.626 | 0.519 | 0.429
Traffic | 96 | 0.424 | 0.278 | 0.395 | 0.268 | 0.649 | 0.389 | 0.544 | 0.359 | 0.522 | 0.290 | 0.805 | 0.493 | 0.593 | 0.321 | 0.650 | 0.396 | 0.788 | 0.499 | 0.587 | 0.366
192 | 0.441 | 0.289 | 0.417 | 0.276 | 0.601 | 0.366 | 0.540 | 0.354 | 0.530 | 0.293 | 0.756 | 0.474 | 0.617 | 0.336 | 0.598 | 0.370 | 0.789 | 0.505 | 0.604 | 0.373
336 | 0.461 | 0.293 | 0.443 | 0.283 | 0.609 | 0.369 | 0.551 | 0.358 | 0.558 | 0.305 | 0.762 | 0.477 | 0.629 | 0.336 | 0.605 | 0.373 | 0.797 | 0.508 | 0.621 | 0.283
720 | 0.482 | 0.307 | 0.467 | 0.302 | 0.647 | 0.387 | 0.586 | 0.375 | 0.589 | 0.328 | 0.719 | 0.449 | 0.640 | 0.350 | 0.645 | 0.394 | 0.841 | 0.523 | 0.626 | 0.382
Avg | 0.452 | 0.291 | 0.428 | 0.282 | 0.626 | 0.378 | 0.555 | 0.362 | 0.550 | 0.304 | 0.760 | 0.473 | 0.620 | 0.336 | 0.625 | 0.383 | 0.804 | 0.509 | 0.610 | 0.376
Weather | 96 | 0.157 | 0.209 | 0.174 | 0.214 | 0.192 | 0.232 | 0.177 | 0.218 | 0.158 | 0.230 | 0.202 | 0.261 | 0.172 | 0.220 | 0.196 | 0.255 | 0.221 | 0.306 | 0.217 | 0.296
192 | 0.206 | 0.251 | 0.221 | 0.254 | 0.240 | 0.271 | 0.225 | 0.259 | 0.206 | 0.277 | 0.242 | 0.298 | 0.219 | 0.261 | 0.237 | 0.296 | 0.261 | 0.340 | 0.276 | 0.336
336 | 0.267 | 0.293 | 0.278 | 0.296 | 0.292 | 0.307 | 0.278 | 0.297 | 0.272 | 0.335 | 0.287 | 0.335 | 0.280 | 0.306 | 0.283 | 0.335 | 0.309 | 0.378 | 0.339 | 0.380
720 | 0.347 | 0.346 | 0.358 | 0.349 | 0.364 | 0.353 | 0.354 | 0.348 | 0.398 | 0.418 | 0.351 | 0.386 | 0.365 | 0.359 | 0.345 | 0.381 | 0.377 | 0.427 | 0.403 | 0.428
Avg | 0.244 | 0.274 | 0.258 | 0.279 | 0.272 | 0.291 | 0.259 | 0.281 | 0.259 | 0.315 | 0.271 | 0.320 | 0.259 | 0.287 | 0.265 | 0.317 | 0.292 | 0.363 | 0.309 | 0.360
Solar-Energy | 96 | 0.202 | 0.235 | 0.203 | 0.237 | 0.322 | 0.339 | 0.234 | 0.286 | 0.310 | 0.331 | 0.312 | 0.399 | 0.250 | 0.292 | 0.290 | 0.378 | 0.237 | 0.344 | 0.242 | 0.342
192 | 0.239 | 0.262 | 0.233 | 0.261 | 0.359 | 0.356 | 0.267 | 0.310 | 0.734 | 0.725 | 0.339 | 0.416 | 0.296 | 0.318 | 0.320 | 0.398 | 0.280 | 0.380 | 0.254 | 0.272
336 | 0.251 | 0.272 | 0.248 | 0.273 | 0.397 | 0.369 | 0.290 | 0.315 | 0.750 | 0.735 | 0.368 | 0.430 | 0.319 | 0.330 | 0.353 | 0.415 | 0.304 | 0.389 | 0.282 | 0.376
720 | 0.248 | 0.275 | 0.249 | 0.275 | 0.397 | 0.356 | 0.289 | 0.317 | 0.769 | 0.765 | 0.370 | 0.425 | 0.338 | 0.337 | 0.356 | 0.413 | 0.308 | 0.388 | 0.357 | 0.427
Avg | 0.235 | 0.261 | 0.233 | 0.262 | 0.369 | 0.270 | 0.307 | 0.641 | 0.6398 | 0.347 | 0.417 | 0.301 | 0.319 | 0.330 | 0.401 | 0.282 | 0.375 | 0.281 | 0.291 | 0.381
$\mathrm{1^{st}Count}$ | 22 | 20 | 11 | 11 | 3 | 11 | 5 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0

Table 2: Ablations of PDMLP. We replace the components of PDMLP one by one and
explore the performance of different MLPs. $Avg$ represents the average
results across all four forecast lengths. For clarity, the best results are
highlighted in bold.

Dateset | ECL | Solar-Energy | Traffic
---|---|---|---
Model | Metric | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg
PDMLP | MSE | 0.147 | 0.166 | 0.184 | 0.222 | 0.179 | 0.202 | 0.239 | 0.251 | 0.248 | 0.235 | 0.424 | 0.441 | 0.461 | 0.482 | 0.452
MAE | 0.246 | 0.261 | 0.279 | 0.311 | 0.274 | 0.235 | 0.262 | 0.272 | 0.275 | 0.261 | 0.278 | 0.289 | 0.293 | 0.307 | 0.291
PDMLP1 | MSE | 0.152 | 0.168 | 0.192 | 0.237 | 0.187 | 0.217 | 0.252 | 0.281 | 0.274 | 0.256 | 0.469 | 0.484 | 0.508 | 0.548 | 0.502
MAE | 0.252 | 0.265 | 0.291 | 0.325 | 0.283 | 0.260 | 0.282 | 0.300 | 0.291 | 0.283 | 0.324 | 0.332 | 0.354 | 0.365 | 0.343
PDMLP2 | MSE | 0.151 | 0.169 | 0.187 | 0.226 | 0.183 | 0.210 | 0.243 | 0.263 | 0.261 | 0.244 | 0.522 | 0.525 | 0.539 | 0.570 | 0.539
MAE | 0.254 | 0.268 | 0.285 | 0.319 | 0.281 | 0.256 | 0.276 | 0.290 | 0.288 | 0.277 | 0.342 | 0.353 | 0.360 | 0.451 | 0.376
PDMLP3 | MSE | 0.174 | 0.180 | 0.201 | 0.242 | 0.199 | 0.222 | 0.253 | 0.291 | 0.289 | 0.263 | 0.448 | 0.459 | 0.491 | 0.523 | 0.480
MAE | 0.257 | 0.263 | 0.283 | 0.317 | 0.280 | 0.258 | 0.277 | 0.300 | 0.297 | 0.283 | 0.290 | 0.292 | 0.309 | 0.327 | 0.304
DPMLP | MSE | 0.151 | 0.168 | 0.189 | 0.227 | 0.183 | 0.218 | 0.249 | 0.263 | 0.271 | 0.250 | 0.480 | 0.495 | 0.514 | 0.554 | 0.510
MAE | 0.251 | 0.265 | 0.289 | 0.320 | 0.281 | 0.270 | 0.292 | 0.304 | 0.299 | 0.291 | 0.328 | 0.339 | 0.346 | 0.362 | 0.343
PMLP | MSE | 0.153 | 0.170 | 0.194 | 0.242 | 0.189 | 0.219 | 0.257 | 0.274 | 0.267 | 0.254 | 0.474 | 0.482 | 0.512 | 0.557 | 0.506
MAE | 0.254 | 0.266 | 0.292 | 0.330 | 0.285 | 0.249 | 0.288 | 0.295 | 0.289 | 0.280 | 0.328 | 0.329 | 0.342 | 0.369 | 0.342
PMLP1 | MSE | 0.153 | 0.170 | 0.197 | 0.245 | 0.191 | 0.225 | 0.268 | 0.287 | 0.285 | 0.266 | 0.484 | 0.491 | 0.516 | 0.554 | 0.510
MAE | 0.252 | 0.268 | 0.296 | 0.335 | 0.287 | 0.262 | 0.290 | 0.303 | 0.301 | 0.289 | 0.334 | 0.336 | 0.350 | 0.370 | 0.347
PMLP2 | MSE | 0.153 | 0.169 | 0.188 | 0.227 | 0.184 | 0.221 | 0.266 | 0.267 | 0.269 | 0.255 | 0.526 | 0.531 | 0.528 | 0.556 | 0.535
MAE | 0.257 | 0.268 | 0.288 | 0.319 | 0.283 | 0.248 | 0.290 | 0.280 | 0.283 | 0.275 | 0.345 | 0.350 | 0.349 | 0.363 | 0.351
PMLP3 | MSE | 0.179 | 0.185 | 0.201 | 0.243 | 0.202 | 0.234 | 0.255 | 0.290 | 0.328 | 0.276 | 0.455 | 0.471 | 0.490 | 0.517 | 0.483
MAE | 0.260 | 0.267 | 0.284 | 0.317 | 0.282 | 0.269 | 0.278 | 0.299 | 0.321 | 0.291 | 0.293 | 0.298 | 0.308 | 0.322 | 0.305

###  3.4 Projection and Loss Function

We adopt a channel independent approach to implement
$\mathrm{Projection}:\mathbb{R}^{d_{model}}\to\mathbb{R}^{T}$ by MLP, where
each channel shares parameters.

Our loss function is calculated by the Mean Square Error (MSE) between the
model prediction $\hat{x}$ and the ground truth $x$:

$\mathcal{L}=\frac{1}{M}\sum_{M}^{i=1}\left\|\hat{x}^{i}_{L+1:L+T}-x^{i}_{L+1:L+T}\right\|^{2}_{2}$
(2)

and the loss is propagated back from the Projection output across the entire
model.

##  4 Experiments

We comprehensively evaluated the proposed PDMLP in various time series
forecasting applications, verifying the generality of the proposed framework,
and have further delved into the investigation of the effectiveness of
individual components of PDMLP when applied in LSTF tasks.

Datasets: We evaluated the performance of our proposed PDMLP on 8 commonly
used LSTF benchmark datasets: Solar Energy Lai et al. (2018), Weather,
Traffic, Electricity (ECL) Wu et al. (2021), and 4 ETT datasets (ETTh1, ETTh2,
ETTm1, ETTm2) Zhou et al. (2021).

Baselines and Metrics: We selected nine widely acknowledged SOTA forecasting
models for our benchmarking analysis, which comprises the Transformer-based
models: iTransformer Liu et al. (2023), PatchTST Nie et al. (2022),
Crossformer Zhang & Yan (2022), and FEDformer Zhou et al. (2022); the CNN-
based models: TimeNet Wu et al. (2022); and the significant MLP-based models:
DLinear Zeng et al. (2023), TiDE Das et al. (2023), and RLinear Li et al.
(2023a). To evaluate the performance of these models, we used widely used
evaluation metrics: MSE and Mean Absolute Error (MAE).

###  4.1 Multivariate Long-Term Forecasting

Table 1 presents the results of MTS forecasting, with lower MSE/MAE values
indicating greater predictive accuracy. Overall, the model we propose has
achieved a surprisingly efficacious outcome, realizing optimal performance
across all or the majority of forecast horizons. Out of a total of 90 metrics,
a remarkable count of 42 SOTA results were attained, surpassing the second-
ranked iTransformer by nearly 50%. This suggests that the Attention mechanism
may not necessarily be the optimal choice for LSTF tasks, as even simple
linear models are capable of achieving impressive results. Further analysis
indicates that channel independence models, represented by PatchTST, often
failed to perform to their potential, thus implying the indispensable role of
inter-variable interactions in MTS analysis. Notably, despite explicitly
capturing multivariate correlations, the performance of Crossformer did not
meet expectations, underscoring that inappropriate utilization of inter-
variable correlations can negatively impact model efficacy. Hence, we conclude
that skillful leverage of interactions between variables is vital and, in
certain scenarios, simple linear models may suffice to deliver outstanding
performance compared to the more complex Transformer architectures.

###  4.2 Model Analysis

####  4.2.1 Ablation Study

As shown in the Table 2, we investigate the impact of each module of PDMLP on
performance. DPMLP follows the conventional decomposition approach, which
involves decomposing the time series first and then embedding it before
separately handling the prediction of the trend and seasonal components. PMLP
eliminates the decomposition module, directly inputs the embedded original
sequence into the model for prediction. In both PDMLP1 and PMLP1, the MPE is
removed, and the embedding method is altered to a single linear layer. PDMLP2
and PMLP2, on the other hand, eliminate the dot product in the MLP among
variables and replace it with a simple residual connection, while PDMLP3 and
PMLP3 completely remove the MLP among variables, canceling the interaction
between them. We only change the corresponding modules and keep all other
settings unchanged, with the input length still set to 96.

It can be observed that conventional decomposition methods do not perform
well, which is attributable to the excessively complex temporal relationships
inherent in the original time series that cannot be effectively deconstructed
through simple decomposition techniques. In contrast, latent vector
decomposition can adeptly circumvent this issue. Furthermore, the observed
performance deterioration upon the removal of the MPE indicates that MPE plays
a significant role in learning the various temporal relationships.
Additionally, it is noteworthy that the dot product method outperforms simply
because it enhances the interactiveness between variables, while mere addition
does not confer this advantage. Correspondingly, models that lack variable
interactiveness experience a predictable decline in performance, which
reiterates the critical importance of appropriately leveraging the
interrelationships between variables.

####  4.2.2 Increasing look-back Window

Figure 5: Forecasting performance (MSE) with varying look-back windows on 3
datasets: ETTh1, ETTm2, and Weather. The look-back windows are selected to be
$L=\\{192,288,384,480,576,672,768\\}$, and the prediction horizons are
$T=\\{192,720\\}$.

In principle, a longer input sequence might enhance model performance,
primarily due to the provision of a richer historical context that assists the
model in more effectively learning and identifying long-term patterns within
time series. Generally, a robust LSTF model, equipped with substantial
capability to extract temporal relationships, is expected to yield improved
outcomes with extended input lengths. However, longer inputs necessitate the
model’s heightened ability to capture long-term dependencies, as a deficit in
this capacity could readily precipitate a decline in model performance.

As illustrated in Figure 5, there is a gradual augmentation in the performance
of all models with increasing input length, but as the input extends to
considerable lengths (768), some models begin to exhibit diminished
performance, which may also be attributed to the amplified noise accompanying
the expanded input. In contrast, our model and DLinear consistently
demonstrate steady improvements in performance as the input lengthens,
exemplifying the superiority of linear models in this context.

##  5 Conclusions

This paper provides an in-depth exploration of existing solutions for Long-
Term Time Series Forecasting (LTSF) tasks within the framework of the
Transformer architecture and introduces an innovative approach: Patch-based
Decomposed Multi-Layer Perceptron (PDMLP). Through empirical analysis, we
demonstrate the efficacy of the patch mechanism in time series forecasting and
address the current limitations of Transformer models by designing a novel,
simplified architecture. The PDMLP utilizes a simple moving average to
separate the smooth components from the noisy residuals in time series, and
employs a unique channel mixing strategy to enhance the interchange of
semantic information across variables. Additionally, we present the Multi-
Scale Patch Embedding method to enable the model to more effectively learn the
diverse associations among input sequences. Through extensive experimentation
on multiple real-world datasets, PDMLP shows superior performance compared to
existing technologies. This research not only validates the potential of
reducing complexity and simplifying network structures to improve performance
on LTSF tasks but also emphasizes the critical role of cross-variable
interactions in enhancing the accuracy of multivariate time series forecasting
. We anticipate that the success of PDMLP will not only advance further
research in the field of LSTF but also motivate future efforts toward
developing models that prioritize efficiency, simplicity, and
interpretability. Ultimately, we hope that this research will inspire the
creation of more innovative time series forecasting methods focused on
addressing specific problems rather than the pursuit of model complexity.

## References

  * Ba et al. [2016] Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bai et al. [2018] Bai, S., Kolter, J. Z., and Koltun, V.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ , 2018.
  * Borovykh et al. [2017] Borovykh, A., Bohte, S., and Oosterlee, C. W.  Conditional time series forecasting with convolutional neural networks.  _arXiv preprint arXiv:1703.04691_ , 2017.
  * Box & Pierce [1970] Box, G. E. and Pierce, D. A.  Distribution of residual autocorrelations in autoregressive-integrated moving average time series models.  _Journal of the American statistical Association_ , 65(332):1509–1526, 1970.
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2024] Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., and Guo, C.  Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting.  _arXiv preprint arXiv:2402.05956_ , 2024.
  * Chen et al. [2023] Chen, S.-A., Li, C.-L., Yoder, N., Arik, S. O., and Pfister, T.  Tsmixer: An all-mlp architecture for time series forecasting.  _arXiv preprint arXiv:2303.06053_ , 2023.
  * Cryer [1986] Cryer, J. D.  _Time series analysis_ , volume 286.  Duxbury Press Boston, 1986.
  * Das et al. [2023] Das, A., Kong, W., Leach, A., Sen, R., and Yu, R.  Long-term forecasting with tide: Time-series dense encoder.  _arXiv preprint arXiv:2304.08424_ , 2023.
  * Devlin et al. [2018] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. [2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Ekambaram et al. [2023] Ekambaram, V., Jati, A., Nguyen, N., Sinthong, P., and Kalagnanam, J.  Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting.  _arXiv preprint arXiv:2306.09364_ , 2023.
  * Feichtenhofer et al. [2022] Feichtenhofer, C., Li, Y., He, K., et al.  Masked autoencoders as spatiotemporal learners.  _Advances in neural information processing systems_ , 35:35946–35958, 2022.
  * Gong et al. [2023] Gong, Z., Tang, Y., and Liang, J.  Patchmixer: A patch-mixing architecture for long-term time series forecasting.  _arXiv preprint arXiv:2310.00655_ , 2023.
  * Han et al. [2023] Han, L., Ye, H.-J., and Zhan, D.-C.  The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting.  _arXiv preprint arXiv:2304.05206_ , 2023.
  * He et al. [2016] He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. [2022] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16000–16009, 2022.
  * Huang et al. [2024] Huang, Q. H., Shen, L., Zhang, R., Cheng, J., Ding, S., and Zhou, Zhengyang Wang, Y.  Hdmixer: Hierarchical dependency with extendable patch for multivariate time series forecasting.  _The 38th Annual AAAI Conference on Artificial Intelligence_ , 2024.
  * Ioffe & Szegedy [2015] Ioffe, S. and Szegedy, C.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pp. 448–456. pmlr, 2015.
  * Kingma & Ba [2014] Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kitaev et al. [2020] Kitaev, N., Kaiser, Ł., and Levskaya, A.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ , 2020.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Lai et al. [2018] Lai, G., Chang, W.-C., Yang, Y., and Liu, H.  Modeling long-and short-term temporal patterns with deep neural networks.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pp. 95–104, 2018.
  * Lee et al. [2023] Lee, S., Park, T., and Lee, K.  Learning to embed time series patches independently.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Li et al. [2019] Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  _Advances in neural information processing systems_ , 32, 2019.
  * Li et al. [2023a] Li, Z., Qi, S., Li, Y., and Xu, Z.  Revisiting long-term time series forecasting: An investigation on linear mapping.  _arXiv preprint arXiv:2305.10721_ , 2023a.
  * Li et al. [2023b] Li, Z., Rao, Z., Pan, L., and Xu, Z.  Mts-mixers: Multivariate time series forecasting via factorized temporal and channel mixing.  _arXiv preprint arXiv:2302.04501_ , 2023b.
  * Lim & Zohren [2021] Lim, B. and Zohren, S.  Time-series forecasting with deep learning: a survey.  _Philosophical Transactions of the Royal Society A_ , 379(2194):20200209, 2021.
  * Lim et al. [2020] Lim, B., Zohren, S., and Roberts, S.  Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8. IEEE, 2020.
  * Lim et al. [2021] Lim, B., Arık, S. Ö., Loeff, N., and Pfister, T.  Temporal fusion transformers for interpretable multi-horizon time series forecasting.  _International Journal of Forecasting_ , 37(4):1748–1764, 2021.
  * Lin et al. [2023] Lin, S., Lin, W., Wu, W., Wang, S., and Wang, Y.  Petformer: Long-term time series forecasting via placeholder-enhanced transformer.  _arXiv preprint arXiv:2308.04791_ , 2023.
  * Liu et al. [2018] Liu, L., Shen, J., Zhang, M., Wang, Z., and Tang, J.  Learning the joint representation of heterogeneous temporal events for clinical endpoint prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Liu et al. [2021a] Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S.  Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting.  In _International conference on learning representations_ , 2021a.
  * Liu et al. [2020] Liu, Y., Gong, C., Yang, L., and Chen, Y.  Dstp-rnn: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction.  _Expert Systems with Applications_ , 143:113082, 2020.
  * Liu et al. [2022] Liu, Y., Wu, H., Wang, J., and Long, M.  Non-stationary transformers: Rethinking the stationarity in time series forecasting.  _arXiv preprint arXiv:2205.14415_ , 2022.
  * Liu et al. [2023] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M.  itransformer: Inverted transformers are effective for time series forecasting.  _arXiv preprint arXiv:2310.06625_ , 2023.
  * Liu et al. [2021b] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 10012–10022, 2021b.
  * Nie et al. [2022] Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J.  A time series is worth 64 words: Long-term forecasting with transformers.  _arXiv preprint arXiv:2211.14730_ , 2022.
  * Oreshkin et al. [2019] Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y.  N-beats: Neural basis expansion analysis for interpretable time series forecasting.  In _International Conference on Learning Representations_ , 2019.
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Patton [2013] Patton, A.  Copula methods for forecasting multivariate time series.  _Handbook of economic forecasting_ , 2:899–960, 2013.
  * Qin et al. [2017] Qin, Y., Song, D., Chen, H., Cheng, W., Jiang, G., and Cottrell, G.  A dual-stage attention-based recurrent neural network for time series prediction.  _arXiv preprint arXiv:1704.02971_ , 2017.
  * Radford et al. [2018] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rangapuram et al. [2018] Rangapuram, S. S., Seeger, M. W., Gasthaus, J., Stella, L., Wang, Y., and Januschowski, T.  Deep state space models for time series forecasting.  _Advances in neural information processing systems_ , 31, 2018.
  * Salinas et al. [2020] Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T.  Deepar: Probabilistic forecasting with autoregressive recurrent networks.  _International Journal of Forecasting_ , 36(3):1181–1191, 2020.
  * Shen et al. [2023] Shen, L., Wei, Y., Wang, Y., and Qiu, H.  Take an irregular route: Enhance the decoder of time-series forecasting transformer.  _IEEE Internet of Things Journal_ , 2023.
  * Slutzky [1937] Slutzky, E.  The summation of random causes as the source of cyclic processes.  _Econometrica: Journal of the Econometric Society_ , pp. 105–146, 1937.
  * Sutskever et al. [2014] Sutskever, I., Vinyals, O., and Le, Q. V.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27, 2014.
  * Tang & Zhang [2022] Tang, P. and Zhang, X.  Mtsmae: Masked autoencoders for multivariate time-series forecasting.  In _2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI)_ , pp. 982–989. IEEE, 2022.
  * Tang & Zhang [2023] Tang, P. and Zhang, X.  Infomaxformer: Maximum entropy transformer for long time-series forecasting problem.  _arXiv preprint arXiv:2301.01772_ , 2023.
  * Tolstikhin et al. [2021] Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al.  Mlp-mixer: An all-mlp architecture for vision.  _Advances in neural information processing systems_ , 34:24261–24272, 2021.
  * Van der Maaten & Hinton [2008] Van der Maaten, L. and Hinton, G.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., Zhang, J. Y., and ZHOU, J.  Timemixer: Decomposable multiscale mixing for time series forecasting.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Wen et al. [2022] Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, L.  Transformers in time series: A survey.  _arXiv preprint arXiv:2202.07125_ , 2022.
  * Wold [1938] Wold, H.  _A Study in the Analisys of Stationery: Time Series_.  Almqvist & Wiksells Boktryckeri, 1938.
  * Wold et al. [1987] Wold, S., Esbensen, K., and Geladi, P.  Principal component analysis.  _Chemometrics and intelligent laboratory systems_ , 2(1-3):37–52, 1987.
  * Wu et al. [2021] Wu, H., Xu, J., Wang, J., and Long, M.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in Neural Information Processing Systems_ , 34:22419–22430, 2021.
  * Wu et al. [2022] Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  _arXiv preprint arXiv:2210.02186_ , 2022.
  * Wu et al. [2020] Wu, S., Xiao, X., Ding, Q., Zhao, P., Wei, Y., and Huang, J.  Adversarial sparse transformer for time series forecasting.  _Advances in neural information processing systems_ , 33:17105–17115, 2020.
  * Yin & Shang [2016] Yin, Y. and Shang, P.  Multivariate multiscale sample entropy of traffic time series.  _Nonlinear Dynamics_ , 86:479–488, 2016.
  * Yule [1927] Yule, G. U.  Vii. on a method of investigating periodicities disturbed series, with special reference to wolfer’s sunspot numbers.  _Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character_ , 226(636-646):267–298, 1927.
  * Zeng et al. [2023] Zeng, A., Chen, M., Zhang, L., and Xu, Q.  Are transformers effective for time series forecasting?  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 37, pp. 11121–11128, 2023.
  * Zhang & Yan [2022] Zhang, Y. and Yan, J.  Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhong et al. [2023] Zhong, S., Song, S., Li, G., Zhuo, W., Liu, Y., and Chan, S.-H. G.  A multi-scale decomposition mlp-mixer for time series analysis.  _arXiv preprint arXiv:2310.11959_ , 2023.
  * Zhou et al. [2021] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 11106–11115, 2021.
  * Zhou et al. [2022] Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International Conference on Machine Learning_ , pp. 27268–27286. PMLR, 2022.

##  Appendix A Related Work

###  A.1 Patch-Based Model

Patch-based methods have already demonstrated their immense potential across
various data modalities. In the field of Computer Vision (CV), traditional
Convolutional Neural Networks (CNNs) naturally segment images into patch and
process each patch to learn local features Krizhevsky et al. [2012]. Vision
Transformer (ViT) Dosovitskiy et al. [2020] was the first to integrate the
concept of patch into the Transformer Vaswani et al. [2017] model,
outperforming CNN-based approaches on large datasets and breaking the
inductive bias associated with CNNs. Subsequent models such as BEiT Huang et
al. [2024] and MAE He et al. [2022] also segment images into patch as input
for the Transformer.

In the realm of time series analysis, PatchTST Nie et al. [2022] innovatively
segments time series into multiple patch to extract localized semantic
information for subsequent Transformer processing. MTSMAE Tang & Zhang [2022]
introduced the masked autoencoder method to self-supervised pre-training for
time series, learning deep semantic associations through masked patches.
iTransformer Liu et al. [2023] further takes a reverse approach by treating an
entire time series as a single patch for processing. Additionally, patch
methods have been employed within CNN models, such as PatchMixer Gong et al.
[2023]. To address the potential loss of temporal information by permutation-
invariant self-attention mechanisms, it incorporates a novel permutational
variant convolutional structure to preserve temporal information. Relying
entirely on depthwise separable convolutions, PatchMixer extracts both local
features and global correlations within a single-scale architecture.

Furthermore, TSMixer Ekambaram et al. [2023] draws inspiration from the
success of the MLP-Mixer Tolstikhin et al. [2021] model in CV, combining patch
with MLP to propose a lightweight, all-MLP neural architecture specifically
designed for MTS and representational learning of patch time series.
Concurrently, HDMixer aims to capture semantically enriched patch through
effective modeling of hierarchical interactions, designing a Length-Extendable
Patch adaptable for MTS, enriching the boundary information of patch and
alleviating semantic incoherence within sequences.

###  A.2 Time Series Forecasting

Time series forecasting is integral to many fields and has seen the
development of a wide range of models Lim & Zohren [2021]. Classical time
series Yule [1927]; Slutzky [1937]; Wold [1938] models remain reliable
workhorses for forecasting, with many industrial applications continuing to
utilize them. These classic models boast good interpretability and can
effectively capture trends and seasonality in data Box & Pierce [1970].
However, their limitations become increasingly apparent as data volume and
complexity grow.

Recurrent Neural Networks (RNNs) have also been extensively used in time
series forecasting Salinas et al. [2020]. Their recursive structure gives them
an innate advantage in handling sequential data, leading deep learning methods
to typically rely on RNNs and their variants for developing sequence-to-
sequence forecasting paradigms, which have achieved significant advances
Rangapuram et al. [2018]; Lim et al. [2020]. CNNs designed for extracting
features from images that are principally localized and spatially invariant,
have been adapted for time series analysis through the design of multi-layer
causal convolutions, ensuring predictions only utilize past information
Borovykh et al. [2017]; Lai et al. [2018]; Bai et al. [2018]. Despite
meaningful progress with RNNs and their variants, the current algorithms are
still incremental sequence-to-sequence prediction models that struggle to
forecast long-torm time series accurately Liu et al. [2020]; Qin et al.
[2017]; Sutskever et al. [2014].

Transformers have exhibited stellar performance in domains like Natural
Language Processing (NLP) Radford et al. [2018]; Devlin et al. [2018]; Radford
et al. [2019] and CV Liu et al. [2021b]; He et al. [2016]. In time series
tasks, capturing long-torm dependencies and inter-sequence relations is
especially critical—a strength of the Transformer architecture Wen et al.
[2022]. Most of these models concentrate on innovating new means to simplify
the original attention mechanism, thus obtaining enhanced predictive
abilities, particularly for lengthier predictions Zhou et al. [2022]; Liu et
al. [2021a]; Li et al. [2019]; Kitaev et al. [2020]. While progress has been
made with Transformer-based multivariate forecasting models, studies such as
Dlinear Zeng et al. [2023] indicate that a simple univariate linear model can
surpass complex Transformers. Many subsequent MLP-based models are further
exploring the potential of multivariate linear models Li et al. [2023a, b];
Chen et al. [2023]; Ekambaram et al. [2023].

##  Appendix B Multivariate Time Series

Figure 6: Visualization of Multivariate Time Series Data. We have chosen PCA
and t-SNE methods to perform dimensionality reduction on three different types
of data. Due to the excessive number of variables in the data, we have opted
to visualize only the first 21 dimensions.

(a) The ETTh2 dataset.

(b) The Exchange dataset.

Figure 7: The off-diagonal subplots display the bivariate relationships
between multiple variables, while the subplots on the diagonal show the
univariate distribution of each variable individually.

To elucidate the interrelationships among variables within multivariate time
series data, we employed Principal Component Analysis (PCA) Wold et al. [1987]
and T-Distributed Stochastic Neighbor Embedding (T-SNE) Van der Maaten &
Hinton [2008] for dimensionality reduction. As illustrated in the Figure 6, it
is readily apparent that there is no distinct boundary between data of
different variables. Furthermore, on the ETTh2 and Exchange dataset, we
selected two distinct variables, assigning one to the x-axis and the other to
the y-axis, to more vividly demonstrate the inter-variable associations. It is
observable that certain pairs of variables display strong positive or negative
correlations (Figure 7(a)). In the Exchange dataset, many variables exhibit a
positive correlation with each other (Figure 7(b)).

A natural hypothesis arises: When employing channel mixing methods for
forecasting multivariate time series, the interactive effects between
different variables can be captured and utilized by the model, which can then
leverage these relationships to enhance forecasting accuracy. We posit that
the suboptimal performance of previous models employing channel mixing could
be attributed to their erroneous usage of cross-variable interaction
information. During the time series data collection process, noise can
significantly impact the quality of data, resulting in conspicuous noise
points (as shown in the Figure 7(a) and 7(b), where some points clearly
deviate from the established relationships among variables). Prior channel
mixing models based on temporal points were unable to effectively handle such
noise, thereby hindering the model’s ability to learn inter-variable
correlations.

##  Appendix C Time Series Decomposition

(a) Calculate the MSE after summing the trend component and the seasonal
component.

(b) Sum the MSE calculated separately for the trend component and the seasonal
component.

Figure 8: Visualization of input-96-predict-96 results of DLinear on the ETTm1
dataset.

(a) Calculate the MSE after summing the trend component and the seasonal
component.

(b) Sum the MSE calculated separately for the trend component and the seasonal
component.

Figure 9: Visualization of input-96-predict-96 results of PatchTST on the
ETTm1 dataset.

(a) Calculate the MSE after summing the trend component and the seasonal
component.

(b) Sum the MSE calculated separately for the trend component and the seasonal
component.

Figure 10: Visualization of input-96-predict-192 results of PatchTST on the
Exchange dataset.

(a) Calculate the MSE after summing the trend component and the seasonal
component.

(b) Sum the MSE calculated separately for the trend component and the seasonal
component.

Figure 11: Visualization of input-96-predict-336 results of PatchTST on the
Weather dataset.

Time series analysis is a vital approach to understanding complex data. It
involves the modeling and forecasting of series characterized by seasonal
fluctuations, trends, and other irregular influences. Numerous studies have
attempted to deconstruct these composite data sets and extract interpretable
structures through decomposition methods, by breaking down time series into
their constituent components, to achieve better understanding and prediction
Wu et al. [2021]; Zhou et al. [2022]; Tang & Zhang [2023]. We specifically
discusses the decomposition method employed in the Autoformer Wu et al.
[2021], accompanied by usage strategies of DLinear Zeng et al. [2023] and
PatchTST Nie et al. [2022]. The method initially utilizes a moving average
kernel to decompose the input raw data into trend and residual (seasonal)
components. Two identical yet parameter-independent modules are then applied
to each component, with the cumulative features subsequently added to yield a
final forecast.

As show in Figure 8(a), it is observable that the trend component predicted by
DLinear is comparatively rougher than the actual smooth trend, and its
forecast for the seasonal component appears disordered, deviating
significantly from the seasonal component of the original data, suggesting
minimal correlation. Furthermore, we visualized the trend and seasonal
components predicted by the PatchTST model (Figure 9(a)). The trend component
forecast by PatchTST appears somewhat smoother than that of DLinear, yet still
insufficiently smooth, and it can be stated that the model has not adeptly
learned the trend of the original time series.

Table 3: The impact of different components derived from the conventional
decomposition method on the predictive performance of the model. Patch is the
original method without using decomposition. DLinear and PatchTST_D are the
original decomposition method without any changes. DLinear1 and PatchTST1
represent the models where the loss has been modified to calculate the Mean
Squared Error (MSE) for the trend and seasonal components separately and then
sum them up. DLinear2 and PatchTST2 indicate that the loss is calculated only
for the trend component’s MSE, while DLinear3 and PatchTST3 indicate that the
loss is calculated only for the seasonal component’s MSE.

Model |  Prediction Lengths | ETTh2 | ETTm1 | Exchange | Weather
---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
DLinear | 96 | 0.333 | 0.387 | 0.345 | 0.372 | 0.088 | 0.218 | 0.196 | 0.255
192 | 0.477 | 0.476 | 0.380 | 0.389 | 0.176 | 0.315 | 0.237 | 0.296
336 | 0.594 | 0.541 | 0.413 | 0.413 | 0.313 | 0.427 | 0.283 | 0.335
720 | 0.831 | 0.657 | 0.474 | 0.453 | 0.839 | 0.695 | 0.345 | 0.381
Avg | 0.559 | 0.515 | 0.403 | 0.407 | 0.354 | 0.414 | 0.265 | 0.317
DLinear1 | 96 | 0.348 | 0.397 | 0.353 | 0.378 | 0.086 | 0.214 | 0.196 | 0.256
192 | 0.469 | 0.470 | 0.387 | 0.395 | 0.170 | 0.309 | 0.238 | 0.296
336 | 0.546 | 0.519 | 0.417 | 0.416 | 0.305 | 0.417 | 0.282 | 0.331
720 | 0.797 | 0.641 | 0.475 | 0.453 | 0.831 | 0.686 | 0.343 | 0.383
Avg | 0.540 | 0.507 | 0.408 | 0.411 | 0.348 | 0.407 | 0.265 | 0.317
DLinear2 | 96 | 0.406 | 0.446 | 0.430 | 0.432 | 0.091 | 0.222 | 0.213 | 0.273
192 | 0.527 | 0.513 | 0.459 | 0.447 | 0.178 | 0.319 | 0.253 | 0.309
336 | 0.601 | 0.556 | 0.489 | 0.466 | 0.306 | 0.418 | 0.300 | 0.349
720 | 0.859 | 0.677 | 0.541 | 0.498 | 0.834 | 0.687 | 0.359 | 0.396
Avg | 0.598 | 0.548 | 0.480 | 0.461 | 0.352 | 0.412 | 0.281 | 0.332
DLinear3 | 96 | 3.877 | 1.360 | 1.403 | 0.882 | 4.516 | 1.563 | 0.743 | 0.620
192 | 3.889 | 1.346 | 1.358 | 0.860 | 4.395 | 1.521 | 0.872 | 0.670
336 | 4.055 | 1.388 | 1.240 | 0.823 | 4.791 | 1.634 | 0.828 | 0.653
720 | 4.043 | 1.404 | 1.334 | 0.863 | 4.706 | 1.622 | 0.805 | 0.652
Avg | 3.966 | 1.375 | 1.334 | 0.857 | 4.602 | 1.585 | 0.812 | 0.649
PatchTST | 96 | 0.302 | 0.348 | 0.329 | 0.367 | 0.088 | 0.205 | 0.177 | 0.218
192 | 0.388 | 0.400 | 0.367 | 0.385 | 0.176 | 0.299 | 0.225 | 0.259
336 | 0.426 | 0.433 | 0.399 | 0.410 | 0.301 | 0.397 | 0.278 | 0.297
720 | 0.431 | 0.446 | 0.454 | 0.439 | 0.901 | 0.714 | 0.354 | 0.348
Avg | 0.387 | 0.407 | 0.387 | 0.400 | 0.367 | 0.404 | 0.259 | 0.281
PatchTST_D | 96 | 0.296 | 0.345 | 0.337 | 0.370 | 0.089 | 0.209 | 0.186 | 0.224
192 | 0.373 | 0.397 | 0.376 | 0.386 | 0.181 | 0.302 | 0.232 | 0.265
336 | 0.382 | 0.413 | 0.397 | 0.406 | 0.337 | 0.421 | 0.285 | 0.301
720 | 0.417 | 0.443 | 0.458 | 0.442 | 0.919 | 0.710 | 0.360 | 0.351
Avg | 0.367 | 0.400 | 0.392 | 0.401 | 0.382 | 0.411 | 0.266 | 0.285
PatchTST1 | 96 | 0.298 | 0.343 | 0.339 | 0.368 | 0.084 | 0.203 | 0.177 | 0.217
192 | 0.377 | 0.390 | 0.373 | 0.380 | 0.177 | 0.299 | 0.225 | 0.260
336 | 0.389 | 0.409 | 0.405 | 0.402 | 0.338 | 0.419 | 0.282 | 0.299
720 | 0.416 | 0.432 | 0.466 | 0.438 | 0.838 | 0.687 | 0.360 | 0.350
Avg | 0.370 | 0.394 | 0.396 | 0.397 | 0.359 | 0.402 | 0.261 | 0.282
PatchTST2 | 96 | 0.365 | 0.401 | 0.399 | 0.410 | 0.085 | 0.206 | 0.192 | 0.229
192 | 0.439 | 0.440 | 0.437 | 0.431 | 0.179 | 0.302 | 0.239 | 0.271
336 | 0.443 | 0.452 | 0.466 | 0.446 | 0.330 | 0.415 | 0.297 | 0.311
720 | 0.471 | 0.472 | 0.522 | 0.477 | 0.839 | 0.687 | 0.373 | 0.360
Avg | 0.430 | 0.441 | 0.456 | 0.441 | 0.358 | 0.403 | 0.275 | 0.293
PatchTST3 | 96 | 0.344 | 0.370 | 0.739 | 0.558 | 0.148 | 0.276 | 0.232 | 0.280
192 | 0.425 | 0.416 | 0.710 | 0.549 | 0.245 | 0.359 | 0.282 | 0.315
336 | 0.407 | 0.420 | 0.739 | 0.568 | 0.392 | 0.459 | 0.328 | 0.343
720 | 0.430 | 0.439 | 0.781 | 0.587 | 0.944 | 0.738 | 0.396 | 0.384
Avg | 0.402 | 0.411 | 0.742 | 0.566 | 0.432 | 0.458 | 0.310 | 0.331

Additionally, we investigated the influence of traditional time series
decomposition methods on the extracted trend and seasonal components
concerning model prediction performance. We selected the DLinear and PatchTST
models, and chose four datasets including ETTh2, ETTm1, Exchange, and Weather.
As shown in Table 3, we can see that models that modify the loss to calculate
the Mean Squared Error (MSE) separately for the trend and seasonal components
and then sum them achieve better performance. This indicates that computing
the loss separately constrains the model to learn these components more in
line with our expectations. However, for PatchTST, it seems that decomposing
the trend and seasonal components with traditional decomposition methods and
then calculating the loss after summing them does not appear to bring much
performance improvement. Moreover, as can be seen in Figure 9, the model does
not seem to have learned much information, especially with components like the
trend which appear to be very smooth. This suggests that traditional methods
of time series decomposition may not be very feasible. In some datasets (e.g.,
Figure 10), the model did not learn any information at all—both trend and
seasonality could essentially be described as predicting a constant.

##  Appendix D Attention score map of Transformer

(a) ECL

(b) ETTm1

(c) Traffic

Figure 12: The self-attention scores from a 2-layer Transformer trained on
different datasets, showing that time series data tends to be patch.

The traditional Transformer does not adopt the idea of patch but instead
directly embeds the time series at the point-level. Much research indicates
that this approach tends to lead to the Transformer processing a large amount
of redundant features Tang & Zhang [2022], resulting in lower attention values
for certain time points, exhibiting a long-tail distribution with attention
scores clearly showing a form of blockiness Zhou et al. [2021]. Figure 12
shows a typical attention score map of the original Transformer for time
series forecasting. We selected three datasets of different sizes based on
different sampling scales and trained a two-layer Transformer with eight
heads, ultimately selecting the attention scores from the second layer. We can
see that attention values tend to patch, i.e., close data points have similar
attention weights. This suggests that directly computing attention on time
points is not advisable for Transformers. However, as shown in Figure 1,
creating patch from the input time series can only mitigate this blockiness
and does not completely resolve it.

##  Appendix E Experiments

We further delved into the efficacy of each component of PDMLP and its
variants when applied to the LSTF task, and investigated the sensitivity of
PDMLP to a range of hyperparameters.

###  E.1 Datasets

Table 4: Detailed dataset descriptions. $Dim$ denotes the number of variables
in each dataset. $Dataset$ $Size$ indicates the total number of time points
split across the training, validation, and test sets, respectively.
$Prediction$ $Length$ refers to the future time points to be forecasted, with
each dataset encompassing four forecasting settings. $Frequency$ denotes the
sampling interval of time points for this dataset.

Dataset | Dim | Prediction Length | Dataset Size | Frequency | Information
---|---|---|---|---|---
ETTh1, ETTh2 | 7 | {96, 192, 336, 720} | (8545, 2881, 2881) | Hourly | Electricity
ETTm1, ETTm2 | 7 | {96, 192, 336, 720} | (34465, 11521, 11521) | 15min | Electricity
Exchange | 8 | {96, 192, 336, 720} | (5120, 665, 1422) | Daily | Economy
Weather | 21 | {96, 192, 336, 720} | (36792, 5271, 10540) | 10min | Weather
ECL | 321 | {96, 192, 336, 720} | (18317, 2633, 5261) | Hourly | Electricity
Traffic | 862 | {96, 192, 336, 720} | (12185, 1757, 3509) | Hourly | Transportation
Solar-Energy | 137 | {96, 192, 336, 720} | (36601, 5161, 10417) | 10min | Energy

We evaluated the performance of our proposed PDMLP on 8 commonly used LSTF
benchmark datasets: Solar Energy Lai et al. [2018], Weather, Traffic,
Electricity (ECL) Wu et al. [2021], and 4 ETT datasets (ETTh1, ETTh2, ETTm1,
ETTm2) Zhou et al. [2021]. The details of datasets are provided in Table 4.

  * •

ETT captured data incorporating seven factors pertinent to electric power
transformers from July 2016 to July 2018. It comprises four subsets: ETTh1 and
ETTh2, which record data hourly, and ETTm1 and ETTm2, which record data every
15 minutes.

  * •

Exchange compiled panel data of daily exchange rates from eight countries
spanning the period from 1990 to 2016.

  * •

Weather encompasses twenty-one meteorological factors collected every 10
minutes from the Max Planck Institute for Biogeochemistry’s weather station in
the year 2020.

  * •

ECL logged hourly electricity consumption data for 321 customers.

  * •

Traffic gathered data on hourly roadway occupancy rates measured by 862
sensors on the highways of the San Francisco Bay Area from January 2015 to
December 2016.

  * •

Solar Energy chronicled the power output from 137 photovoltaic power stations
in the year 2006, with samples taken every 10 minutes.

###  E.2 Implementation Details

Table 5: Default setting.

config | value
---|---
optimizer | Adam
learning rate | $10^{-4}$
batch size | 32
total epoch | 20
early stopping | 5
input length | 96
MLP layer | 1
d_model | 1024
patch size | 48,24,12,6
dropout | 0.1
loss function | MSE

All experiments were implemented using PyTorch Paszke et al. [2019] and
executed on a single NVIDIA V100 16GB GPU. Table 5 presents the experimental
setup. Optimization of the model was carried out using the Aadm optimizer
Kingma & Ba [2014], with an initial learning rate of $10^{-4}$ and L2 loss. We
standardized the batch size to 32, fixed the number of training epochs to 20,
and employed an early stopping criterion, where if the loss on the validation
set did not decrease for five consecutive epochs, training would be terminated
prematurely. In our proposed model, the number of MLP layers, denoted as N,
was set to 1. The dimension of series representations d_model was established
at 1024. Unless otherwise specified, the input sequence length for the models
was set to 96. For PDMLP, a consistent dropout probability of 0.1 was applied
across all positions. Regarding Multi-Scale Patch Embedding, we selected four
patch sizes: [48, 24, 12, 6].

All baseline models to which we compared our approach were based on the
iTransformer Liu et al. [2023] and TimesNet Wu et al. [2022] repositories,
constructed atop configurations provided by the original papers or official
codes of each respective model. Furthermore, except for results derived from
the iTransformer repository, all other results were obtained by conducting six
separate runs with different random seeds, and the mean of these six outcomes
was reported.

###  E.3 Hyperparameter Sensitivity

Figure 13: Hyperparameter sensitivity with respect to the learning rate, the
number of MLP blocks, and the hidden dimension of Embedding tokens. The
results are recorded with the lookback window length $L=96$ and the forecast
window length $T=\\{96,192\\}$

We evaluated the hypersensitivity of the PDMLP concerning the following
parameters: learning rate (lr), the number of blocks (layers) N in the MLP,
and the hidden dimension D of the embedding. The results are illustrated in
Figure 13. We observed that the performance of PDMLP is not particularly
sensitive to these hyperparameters, as evidenced by the relatively
inconspicuous variations in performance. It is advisable that both the number
of blocks and the hidden dimension size not be excessively large, which may be
attributable to the increase in the total number of parameters that require
learning with larger values of these hyperparameters. However, the hidden
dimension size should not be excessively small either, to avoid a
disproportionate compression of features into an insufficiently capacious
latent vector.

###  E.4 Layer Normalization and Batch Normalization

Table 6: The Impact of Different Normalization Techniques on the Predictive
Performance of Various Models. The notations 1 and 2 are employed to signify
that Layer Normalization (LN) and Batch Normalization (BN) are used in the
models, respectively. In the case of PDMLP, which incorporates two successive
normalization layers, PDMLP3 refers to a configuration wherein the first
normalization layer is composed of BN and the second of LN; conversely, PDMLP4
indicates that the initial normalization layer utilizes LN followed by a BN
layer. We compare the performance of identical models employing disparate
normalization techniques, with the optimal outcomes highlighted in bold
typeface.

Dateset | ETTh2 | ETTm1 | Exchange
---|---|---|---
Model | Metric | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg
PDMLP1 | MSE | 0.288 | 0.371 | 0.412 | 0.421 | 0.373 | 0.316 | 0.357 | 0.396 | 0.462 | 0.382 | 0.085 | 0.174 | 0.317 | 0.820 | 0.349
MAE | 0.340 | 0.396 | 0.429 | 0.441 | 0.401 | 0.357 | 0.378 | 0.402 | 0.439 | 0.394 | 0.204 | 0.298 | 0.411 | 0.681 | 0.398
PDMLP2 | MSE | 0.292 | 0.379 | 0.431 | 0.436 | 0.385 | 0.319 | 0.365 | 0.402 | 0.472 | 0.390 | 0.088 | 0.178 | 0.326 | 0.862 | 0.364
MAE | 0.344 | 0.400 | 0.438 | 0.450 | 0.408 | 0.358 | 0.383 | 0.406 | 0.444 | 0.398 | 0.209 | 0.302 | 0.413 | 0.699 | 0.406
PDMLP3 | MSE | 0.300 | 0.386 | 0.426 | 0.434 | 0.387 | 0.321 | 0.365 | 0.407 | 0.472 | 0.391 | 0.089 | 0.179 | 0.326 | 0.855 | 0.362
MAE | 0.347 | 0.398 | 0.435 | 0.447 | 0.407 | 0.360 | 0.382 | 0.407 | 0.443 | 0.398 | 0.207 | 0.301 | 0.413 | 0.698 | 0.405
PDMLP4 | MSE | 0.306 | 0.389 | 0.428 | 0.438 | 0.390 | 0.323 | 0.371 | 0.404 | 0.472 | 0.393 | 0.088 | 0.182 | 0.338 | 0.873 | 0.370
MAE | 0.354 | 0.404 | 0.434 | 0.452 | 0.411 | 0.362 | 0.387 | 0.409 | 0.445 | 0.401 | 0.206 | 0.303 | 0.423 | 0.703 | 0.409
iTransformer1 | MSE | 0.297 | 0.380 | 0.428 | 0.427 | 0.383 | 0.334 | 0.377 | 0.426 | 0.491 | 0.407 | 0.086 | 0.177 | 0.331 | 0.847 | 0.360
MAE | 0.349 | 0.400 | 0.432 | 0.445 | 0.407 | 0.368 | 0.391 | 0.420 | 0.459 | 0.410 | 0.206 | 0.299 | 0.417 | 0.691 | 0.403
iTransformer2 | MSE | 0.318 | 0.379 | 0.427 | 0.438 | 0.391 | 0.336 | 0.392 | 0.445 | 0.506 | 0.420 | 0.093 | 0.186 | 0.336 | 0.886 | 0.375
MAE | 0.350 | 0.398 | 0.433 | 0.448 | 0.407 | 0.371 | 0.399 | 0.429 | 0.466 | 0.416 | 0.215 | 0.310 | 0.420 | 0.710 | 0.414
PatchTST1 | MSE | 0.299 | 0.372 | 0.415 | 0.426 | 0.378 | 0.335 | 0.374 | 0.403 | 0.462 | 0.394 | 0.086 | 0.180 | 0.332 | 0.826 | 0.356
MAE | 0.347 | 0.397 | 0.430 | 0.444 | 0.405 | 0.370 | 0.390 | 0.409 | 0.447 | 0.404 | 0.208 | 0.303 | 0.417 | 0.680 | 0.402
PatchTST2 | MSE | 0.302 | 0.388 | 0.426 | 0.431 | 0.387 | 0.329 | 0.367 | 0.399 | 0.454 | 0.387 | 0.088 | 0.176 | 0.301 | 0.901 | 0.367
MAE | 0.348 | 0.400 | 0.433 | 0.446 | 0.407 | 0.367 | 0.385 | 0.410 | 0.439 | 0.400 | 0.205 | 0.299 | 0.397 | 0.714 | 0.404
TimesNet1 | MSE | 0.340 | 0.402 | 0.452 | 0.462 | 0.414 | 0.338 | 0.374 | 0.410 | 0.478 | 0.400 | 0.107 | 0.226 | 0.367 | 0.964 | 0.416
MAE | 0.374 | 0.414 | 0.452 | 0.468 | 0.427 | 0.375 | 0.387 | 0.411 | 0.450 | 0.406 | 0.234 | 0.344 | 0.448 | 0.746 | 0.443
TimesNet2 | MSE | 0.335 | 0.432 | 0.470 | 0.515 | 0.438 | 0.393 | 0.500 | 0.553 | 0.635 | 0.520 | 0.108 | 0.247 | 0.515 | 1.301 | 0.543
MAE | 0.372 | 0.426 | 0.466 | 0.496 | 0.440 | 0.413 | 0.456 | 0.486 | 0.527 | 0.471 | 0.236 | 0.361 | 0.529 | 0.877 | 0.501

Normalization is a critical processing technique to reduce internal covariate
shift while training deep neural networks. Although the choice between Layer
Normalization (LN) Ba et al. [2016] and Batch Normalization (BN) Ioffe &
Szegedy [2015] depends on the specific task, and PatchTST has demonstrated
certain advantages of BN in time series datasets, LN generally shows better
robustness and compatibility in sequence tasks.

We selected four models, PDMLP, iTransformer, PatchTST, and TimesNet, to
contrast their performance differences with the application of LN and BN over
three datasets: ETTh2, ETTm1, and Exchange. Among these, as stated in its
original publication, PatchTST emphasizes the use of BN. However, the TimesNet
approach, which is predicated on CNN architectures, primarily utilizes LN.
Table 6 illustrates that the BN method did not achieve the surprisingly high
performance delineated in the PatchTST paper. On the contrary, LN proved to
yield superior results in the majority of the experiments conducted. This
suggests that, relative to BN, LN remains more adept at handling time-series
tasks. For many models, the performance difference between LN and BN is not
particularly significant, but for TimesNet, BN performs 30% lower than LN.

###  E.5 Variants of PDMLP

Figure 14: A variant of Intra-Variable MLP or Inter-Variable MLP involves
modifying a single fully-connected layer into two fully-connected layers.
d_model and d_FN are the dimensions of the latent vector

In reference to the Intra-Variable MLP and the Inter-Variable MLP, our
previous work employed a single fully connected layer; however, we have now
flexibly adapted them as dual-layer fully connected structures, as depicted in
Figure 14. Moreover, in the previous versions of PDMLP, we utilized the same
MLP Layer to process both the Smooth and Residual components independently.
Currently, we have flexibly implemented variant MLP Layer to address these
disparate components. As illustrated in Table 7, the $\ast$ denotes the
previous single-layer MLP, $\Diamond$ denotes the dual-layer MLP with feature
number $d\\_FN=d\\_model/\times 2$, and $\triangle$ indicates a dual-layer MLP
with $d\\_FN=d\\_model/2$. Re-order signifies a change in the sequence of
Intra-Variable MLP and Inter-Variable MLP operations, prompting the model to
initiate cross-variable interactions prior to the within-variable
interactions.

Upon examination, it is apparent that different PDMLP variants do not
significantly alter the performance outcome. However, the integration of the
dual-layer MLP leads to an increase in the number of learning parameters
within the model. Even though the $\triangle$ type dual-layer MLP does not
modify the quantity of parameters, it potentially can result in information
loss due to dimensionality compression. Additionally, it is noteworthy that
altering the sequence of variable interactions, as seen in the $Re-order$
configuration, yields inferior performance. This suggests that intra-variable
interactions following the cross-variable interactions may lead to a loss of
the information obtained from the cross-variable interactions.

Table 7: Different variants of PDMLP. The part before the $+$ indicates the
type of MLP used for Intra-Variable, while the part after the $+$ indicates
the type of MLP adopted for Inter-Variable.

Smooth MLP Layer |  Residual MLP Layer |  Prediction Lengths | ETTh2 | ETTm1 | Exchange | Weather
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
$\ast+\ast$ | $\ast+\ast$ | 96 | 0.288 | 0.340 | 0.316 | 0.357 | 0.085 | 0.204 | 0.157 | 0.209
192 | 0.371 | 0.396 | 0.357 | 0.378 | 0.174 | 0.298 | 0.206 | 0.251
336 | 0.412 | 0.429 | 0.396 | 0.402 | 0.317 | 0.411 | 0.267 | 0.293
720 | 0.421 | 0.441 | 0.462 | 0.439 | 0.820 | 0.681 | 0.347 | 0.346
Avg | 0.373 | 0.401 | 0.382 | 0.394 | 0.349 | 0.398 | 0.244 | 0.274
Re-order | Re-order | 96 | 0.315 | 0.361 | 0.320 | 0.359 | 0.093 | 0.216 | 0.165 | 0.212
192 | 0.388 | 0.407 | 0.368 | 0.385 | 0.192 | 0.313 | 0.216 | 0.256
336 | 0.436 | 0.442 | 0.407 | 0.410 | 0.357 | 0.433 | 0.270 | 0.295
720 | 0.460 | 0.462 | 0.486 | 0.451 | 0.955 | 0.732 | 0.351 | 0.347
Avg | 0.400 | 0.418 | 0.395 | 0.401 | 0.399 | 0.424 | 0.251 | 0.278
$\ast+\Diamond$ | $\ast+\Diamond$ | 96 | 0.300 | 0.349 | 0.320 | 0.360 | 0.096 | 0.209 | 0.161 | 0.208
192 | 0.391 | 0.404 | 0.367 | 0.385 | 0.189 | 0.309 | 0.211 | 0.253
336 | 0.428 | 0.434 | 0.400 | 0.407 | 0.340 | 0.421 | 0.267 | 0.294
720 | 0.438 | 0.452 | 0.481 | 0.448 | 0.907 | 0.715 | 0.349 | 0.347
Avg | 0.389 | 0.410 | 0.392 | 0.400 | 0.383 | 0.414 | 0.247 | 0.276
$\Diamond+\ast$ | $\Diamond+\ast$ | 96 | 0.310 | 0.356 | 0.326 | 0.364 | 0.090 | 0.211 | 0.163 | 0.210
192 | 0.393 | 0.407 | 0.374 | 0.390 | 0.186 | 0.308 | 0.211 | 0.252
336 | 0.435 | 0.438 | 0.410 | 0.412 | 0.342 | 0.424 | 0.267 | 0.293
720 | 0.441 | 0.452 | 0.486 | 0.452 | 0.897 | 0.711 | 0.347 | 0.346
Avg | 0.395 | 0.413 | 0.399 | 0.405 | 0.379 | 0.414 | 0.247 | 0.275
$\Diamond+\Diamond$ | $\Diamond+\Diamond$ | 96 | 0.303 | 0.350 | 0.321 | 0.360 | 0.091 | 0.210 | 0.161 | 0.208
192 | 0.393 | 0.405 | 0.372 | 0.387 | 0.185 | 0.306 | 0.210 | 0.252
336 | 0.430 | 0.439 | 0.407 | 0.411 | 0.348 | 0.427 | 0.268 | 0.394
720 | 0.438 | 0.451 | 0.482 | 0.450 | 0.927 | 0.719 | 0.347 | 0.346
Avg | 0.391 | 0.411 | 0.396 | 0.402 | 0.388 | 0.416 | 0.247 | 0.300
$\ast+\Diamond$ | $\ast+\triangle$ | 96 | 0.297 | 0.347 | 0.325 | 0.362 | 0.090 | 0.210 | 0.163 | 0.210
192 | 0.386 | 0.400 | 0.365 | 0.382 | 0.185 | 0.306 | 0.212 | 0.253
336 | 0.437 | 0.438 | 0.395 | 0.403 | 0.345 | 0.424 | 0.269 | 0.294
720 | 0.439 | 0.450 | 0.472 | 0.445 | 0.904 | 0.715 | 0.348 | 0.346
Avg | 0.390 | 0.409 | 0.389 | 0.398 | 0.381 | 0.414 | 0.248 | 0.276
$\ast+\Diamond$ | $\ast+\ast$ | 96 | 0.303 | 0.351 | 0.325 | 0.362 | 0.091 | 0.210 | 0.164 | 0.210
192 | 0.387 | 0.402 | 0.367 | 0.384 | 0.185 | 0.306 | 0.213 | 0.253
336 | 0.428 | 0.435 | 0.402 | 0.407 | 0.345 | 0.425 | 0.268 | 0.294
720 | 0.445 | 0.454 | 0.476 | 0.447 | 0.909 | 0.715 | 0.347 | 0.345
Avg | 0.391 | 0.411 | 0.393 | 0.400 | 0.383 | 0.414 | 0.248 | 0.276
$\ast+\triangle$ | $\ast+\triangle$ | 96 | 0.301 | 0.349 | 0.322 | 0.361 | 0.089 | 0.207 | 0.167 | 0.213
192 | 0.387 | 0.399 | 0.363 | 0.382 | 0.186 | 0.306 | 0.213 | 0.253
336 | 0.426 | 0.434 | 0.402 | 0.408 | 0.342 | 0.422 | 0.271 | 0.395
720 | 0.440 | 0.453 | 0.473 | 0.446 | 0.873 | 0.702 | 0.348 | 0.345
Avg | 0.389 | 0.409 | 0.390 | 0.399 | 0.373 | 0.409 | 0.250 | 0.302
$\ast+\ast$ | $\ast+\triangle$ | 96 | 0.298 | 0.348 | 0.322 | 0.360 | 0.091 | 0.211 | 0.166 | 0.213
192 | 0.390 | 0.403 | 0.366 | 0.384 | 0.185 | 0.307 | 0.214 | 0.254
336 | 0.426 | 0.434 | 0.397 | 0.405 | 0.343 | 0.424 | 0.271 | 0.295
720 | 0.441 | 0.452 | 0.475 | 0.446 | 0.916 | 0.716 | 0.348 | 0.346
Avg | 0.389 | 0.409 | 0.390 | 0.399 | 0.384 | 0.415 | 0.250 | 0.277

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: The main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [No]

  9. Justification: The paper does not discuss the limitations of the work performed by the authors.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: The paper does not need to provide the full set of assumptions and a complete (and correct) proof.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent (the code are provided)

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: The data is open source, and the code has been provided

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: The paper provides these details

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [No]

  34. Justification: The paper provides the average results of the experiment

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: The paper provides the required computing resource information

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [N/A]

  49. Justification: There is no such issue with the paper.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: There is no such issue with the paper.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: The creators or original owners of assets have been fully referenced.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [N/A]

  64. Justification: The paper is not applicable to this.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: The paper is not applicable to this.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: The paper is not applicable to this.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.