# Combating Distribution Shift for Accurate Time Series Forecasting via
Hypernetworks

1st Wenying Duan  School of Mathematics and Computer Sciences
Nanchang University
Nanchang, China
<EMAIL_ADDRESS>2nd Xiaoxi He  Computer Engineering and Networks
Laboratory
ETH Zurich
Zurich, Switzerland
<EMAIL_ADDRESS>3rd Lu Zhou  Department of Civil and Environmental Engineering
The Hong Kong Polytechnic University
HongKong
<EMAIL_ADDRESS>4th Lothar Thiele  Computer Engineering and Networks
Laboratory
Zurich, Switzerland
Zurich, Switzerland
<EMAIL_ADDRESS>5th Hong Rao∗ school of software
Nanchang University
Nanchang, China
<EMAIL_ADDRESS>
###### Abstract

Time series forecasting has widespread applications in urban life ranging from
air quality monitoring to traffic analysis. However, accurate time series
forecasting is challenging because real-world time series suffer from the
distribution shift problem, where their statistical properties change over
time. Despite extensive solutions to distribution shifts in domain adaptation
or generalization, they fail to function effectively in unknown, constantly-
changing distribution shifts, which are common in time series. In this paper,
we propose Hyper Time-Series Forecasting (HTSF), a hypernetwork-based
framework for accurate time series forecasting under distribution shift. HTSF
jointly learns the time-varying distributions and the corresponding
forecasting models in an end-to-end fashion. Specifically, HTSF exploits the
hyper layers to learn the best characterization of the distribution shifts,
generating the model parameters for the main layers to make accurate
predictions. We implement HTSF as an extensible framework that can incorporate
diverse time series forecasting models such as RNNs and Transformers.
Extensive experiments on 9 benchmarks demonstrate that HTSF achieves state-of-
the-art performances.

###### Index Terms:

hypernetworks, time series forecasting, distribution shift

##  I Introduction

11footnotetext: Corresponding Author

Time-series forecasting is crucial for various data analytics domains,
including air quality monitoring [1], renewable energy production [2], human
activity recognition [3], electricity consumption planning [4], urban traffic
analysis [4] etc. The past years have witnessed diverse time-series
forecasting models ranging from the conventional statistical approaches _e.g._
, auto-regressive integrated moving average (ARIMA) [5] to the more recent
deep learning based models such as recurrent neural networks (RNNs) [6],
transformers, [7] and their variants [8, 9, 10, 4]. As deep learning based
models make few assumptions on the temporal structures of time series, they
are preferable in modeling complex, long-sequence time series. They have
demonstrated a notable improvement in prediction accuracy than statistical
models [8].

A unique challenge in time series forecasting is that they are often non-
stationary, resulting in temporal shifts in their distributions. This
distribution shift phenomenon causes discrepancies between the distributions
of the training and the testing data, which degrades the performance of
forecasting models [11, 12]. Such distribution shift also exists among the
input sequences of the training data, which makes it challenging to train a
model that generalizes well to unknown data [13, 14]. Figure 1 illustrates the
distribution shift problem in time series forecasting. Consider three sliding
windows 1, 2, and 3 to segment the time series for training. The probability
distributions of the time series $p_{1}$, $p_{2}$, $p_{3}$ may vary across
windows, and they are likely to differ from the distribution $p_{f}$ in the
unseen forecast window. That is, $p_{1}\neq p_{2}\neq p_{3}\neq p_{f}$, which
makes accurate forecasting for window $p_{f}$ challenging.

Figure 1: Illustration of distribution shifts in time series forecasting and
our solution. Given sliding windows win 1, 2, 3 and so on, the data
distributions $p_{1}$, $p_{2}$, $p_{3}$, $\ldots$ vary across windows, and
they are likely to differ from the distribution $p_{f}$ in the forecasting
window. Our solution is to explicitly model and learn the data distributions
from the historical sequences $\hat{\mathbf{x}}_{1}$, $\hat{\mathbf{x}}_{2}$,
$\hat{\mathbf{x}}_{3}$, $\ldots$ via hyper layers, which generate the model
parameters $W_{f}$ for the main layers. The main layers then take
$\mathbf{x}_{f}$ as input and generate the predictions $y_{f}$.

Distribution shift is usually tackled by domain generalization [15] or domain
adaptation [14, 16]. The idea is to learn the common knowledge transferable
between domains despite the differences in their distributions [15]. However,
it is non-trivial to apply these techniques to time series [17, 12]. This is
because the distributions in time series change constantly (_e.g._ , $p_{1}$,
$p_{2}$, $p_{3}$ in Figure 1), and it is unknown how to best characterize the
distributions upon which to learn the common knowledge [12]. The state-of-the-
art, ADARNN [12] proposes to characterize the worst-case distribution shift in
time series leveraging the principle of maximum entropy. Nevertheless, this
approach is sub-optimal because (i) the training data may not always contain
the worst-case distribution shift; and (ii) it under-utilizes the diversity of
the distribution shifts in the time series.

In this paper, we propose a hypernetwork-based framework called Hyper Time-
Series Forecasting (HTSF) to mitigate the distribution shift problem for
accurate time series forecasting. The idea is to jointly learn the common
knowledge and the corresponding distributions with an end-to-end hypernetwork
architecture. As shown in Figure 1, a hypernetwork consists of hyper and main
layers, where the hyper layers generate model parameters $W_{f}$ conditioned
on the inputs $\mathbf{x}_{f}$ for the main layers to make predictions
$y_{f}$. We exploit the hyper layers to learn the best characterization of the
distribution shifts and the main layers to learn the corresponding forecast
model. Our strategy is advantageous in the following aspects. (i) The end-to-
end learning scheme eliminates the assumptions on the worst-case distribution
shift in the training data and makes full use of the various distribution
shifts in the training data. (ii) The hypernetwork architecture is capable of
modeling complex feature representations in both the distributions and the
common transferable knowledge, and it applies to various deep learning based
time series models.

Our main contributions are summarized as follows:

  * •

We propose HTSF, a hypernetwork-based framework for accurate time series
forecasting. To the best of our knowledge, it is the first study that
addresses the distribution shift problem in time series via hypernetworks.

  * •

HTSF offers a generic, extensible, and end-to-end solution to the distribution
shift problem in time series that makes no assumptions on the training data
and applies to diverse deep learning based time series models such as RNNs and
transformers.

  * •

We evaluate HTSF on 9 time series forecasting benchmarks. Experimental results
show that HTSF achieves state-of-the-art accuracy in both short-term and long-
term time series forecasting. It also yields comparable performance on
spatiotemporal data to the state-of-the-art graph neural network based
solutions [18, 19].

In the rest of this paper, we review related work in Sec. II, present the
problem statement in Sec. III and introduce our method in Sec. IV. We report
the performance in Sec. V and ablation studies in Sec. VI, and finally
conclude in Sec. VII.

##  II Related Work

Our work is related to the following categories of research.

###  II-A Time Series Forecasting

For time series forecasting, we distinguish between statistic models and deep
learning based models. Statistic models[20, 21] are interpretable and
theoretically sound. But they often suffer from heavy crafting on data pre-
processing and labor-intensive feature engineering, which may fail to capture
complex patterns in time series. Recently, deep learning based models have
attracted increasing attention due to their outstanding performance in time
series analysis. For example, RNN-based models [2, 8, 22] prove effective in
capturing both short-term and long-term patterns. Transformer-based models
[23, 4] have shown remarkable performance for extremely long sequences.
However, they are not robust to the distribution shift problem and suffer from
performance degradation [12, 17].

###  II-B Distribution Shift

Domain adaptation [14, 16] and domain generalization [15] are generic
solutions to the distribution shift problem. Domain adaptation focuses on the
disparity difference between the source and target domain, whereas domain
generalization aims to learn a generalizable model from various source
domains. However, it is non-trivial to apply these techniques to time series.
For example, some research reports [24] that domain adversarial learning, a
prevailing domain adaptation technique, may be inappropriate for regression,
while time series forecasting is often a regression task. Furthermore, since
the distributions in time series change constantly, it is not straightforward
how to define multiple source domains in non-stationary time series [12].

A few pioneer studies [17, 12] explore solving the distribution shift problem
in time series forecasting. RevIN [17] proposes an instance-level
normalization technique to reduce the discrepancy in data distributions such
as mean and variance. Our work is orthogonal to RevIN by adopting
representation learning to enhance the generalization of forecasting models.
Our work is most related to ADARNN [12], which first attempts to characterize
the worst-case distribution shift in time series and then matches the
distributions to learn a generalizable model. We advance ADARNN by eliminating
the worst-case assumption and fully exploiting the diversity of the
distribution shifts in the training data.

###  II-C Hypernetworks

A hypernetwork [25] utilizes a primary neural network $g$ to generate weights
$\theta_{f}$ for a second network $f$. The network $f$ with the weights
$\theta_{f}$ can then be applied for specific inference tasks. Hypernetworks
have demonstrated superior performances on many benchmarks [26, 27], due to
their ability to adapt $f$ for different inputs, which allows the hypernetwork
to model tasks more effectively [28]. Hypernetworks have been applied in few-
shot learning [29], continual learning [27], efficient parameter fine-tuning
[30, 31] etc. A few studies have also introduced hypernetworks in time series
forecasting such as traffic prediction [32]. However, it does not explicitly
account for the distribution shift problem, and thus results in sub-optimal
accuracy. To the best of our knowledge, we are the first to adopt
hypernetworks to tackle the distribution shift problem in time series
forecasting.

##  III Problem Statement

We consider a standard multi-variate time series forecasting problem as
follows. A time series $\mathcal{X}$ with the corresponding label sequence
$\mathcal{Y}$, is segmented into a set of sequences with labels
$\mathcal{D}=\left\\{\mathbf{x}^{(i)},\mathbf{y}^{(i)}\right\\}_{i}^{N}$,
where $\mathbf{x}^{(i)}\in\mathbb{R}^{d_{\mathbf{x}}\times T_{\mathbf{x}}}$ is
a sub-sequence of $\mathcal{X}$,
$\mathbf{y}^{(i)}\in\mathbb{R}^{d_{\mathbf{y}}\times T_{\mathbf{y}}}$ is the
corresponding label form $\mathcal{Y}$. $N$, $d_{\mathbf{x}}$,
$d_{\mathbf{y}}$, $T_{\mathbf{x}}$, and $T_{\mathbf{y}}$ denote the number of
sequences, the dimension of input variables, the dimension of label variables,
the input length and the prediction length, respectively. Our objective is to
learn a model $\mathcal{M}:\mathrm{x}_{i}\rightarrow\mathrm{y}_{i}$ from
$\mathcal{D}_{train}$ for unseen sequences in $\mathcal{D}_{test}$, where
$\mathcal{D}_{train}$ and $\mathcal{D}_{test}$ are the training and test sets
from $\mathcal{D}$.

We aim to learn model $\mathcal{M}$ in presence of distribution shift in the
time series, which is described by the following assumptions.

  * •

There is discrepancy between the marginal probability distribution
$P_{\mathcal{D}_{train}}$ of the training set and the distribution
$\mathcal{D}_{test}$ of the test set, _i.e._ , $P_{\mathcal{D}_{train}}\neq
P_{\mathcal{D}_{test}}$. Note that
$P_{\mathcal{D}_{train}}(Y|X)=P_{\mathcal{D}_{test}}(Y|X)$ because the
underlying laws governing the inputs and the outputs of the time series
usually stay the same [12].

  * •

The marginal probability distribution of each $\mathbf{x}^{(i)}$ from
$\mathcal{D}_{train}$ may differ, although their conditional distributions are
the same. That is $P_{\mathcal{D}}(\mathbf{x}^{(i)})\neq
P_{\mathcal{D}}(\mathbf{x}^{(j)})$, $1\leq i\neq j\leq N$,
$P_{\mathcal{D}}\left(\mathbf{y}^{(i)}\mid\mathbf{x}^{(i)}\right)=P_{\mathcal{D}}\left(\mathbf{y}^{(j)}\mid\mathbf{x}^{(j)}\right)$.

The discrepancy between the training and test set highlights the necessity to
account for distribution shift for accurate predictions, while that among the
sub-sequences within the training set explains the challenges of effectively
learning the model $\mathcal{M}$.

##  IV Method

This section presents HTSF, our hypernetwork-based framework for time series
forecasting under distribution shift. HTSF is a model-agnostic framework that
can incorporate various deep learning based time series forecasting models
such as RNNs and Transformers. For ease of presentation, we explain HTSF using
a gated recurrent unit (GRU) as an example forecasting model.

###  IV-A HTSF Overview

Figure 2 depicts the overview of HTSF with GRUs as the underlying forecasting
model, which we call HyperGRU. It mainly consists of two modules.

  * •

Hyper Layers: These layers learn to characterize the distributions from the
historical sequences. The learned distribution characterizations are then used
to generate the weights of the main layers. We employ bidirectional GRU
(BiGRU) for the hyper layers (see Sec. IV-B).

  * •

Main Layers: These layers adapt their weights according to the learned
distribution characterizations from the hyper layers as well as their own
internal states through an attention mechanism. As a result, the main layers
can leverage the common knowledge extracted from the diverse distribution
shifts to make accurate predictions. The core part of the main layers is a
simple GRU cell (see Sec. IV-C).

Figure 2: An overview of HTSF, where we employ GRUs as the example forecasting
model. The corresponding hypernetwork is termed HyperGRU.

The hyper and main layers of HyperGRU interact as follows. Let
$\widehat{\mathcal{X}}^{(i)}$ be the historical data of $\mathbf{x}^{(i)}$.
$S=\left\\{\hat{\mathbf{x}}_{n}^{(i)}\right\\}_{n=1}^{L}$ is the historical
dataset generated by $\widehat{\mathcal{X}}^{(i)}$ with a sliding window of a
fixed length $T$, where
$\hat{\mathbf{x}}_{n}^{(i)}\in\mathbb{R}^{d_{\mathbf{x}}\times T}$,
$L=\left|\widehat{\mathcal{X}}^{(i)}\right|-T+1$. The hyper layers take all
$\hat{\mathbf{x}}_{n}^{(i)}$ from $S$ as inputs and output a set of
representations of the historical marginal probability distributions. These
representations and input at the current time step from $\mathbf{x}^{(i)}$ are
then combined through the attention mechanism to generate the weights for the
main layers at the next time step. Finally, the main layers make predictions
as conventional time series forecasting methods.

Note that our method makes full use of the various distribution shifts in
historical data $S$ and aggregates the common transferable knowledge from $S$
to mitigate the distribution shift problem for accurate time series
forecasting. In comparison, prior proposals either ignore the distribution
shift problem [8, 9, 4] or under-utilize the diverse distribution shifts in
the time series [12].

###  IV-B Hyper Layers

The hyper layers are responsible to learn the complex feature representations
of various distribution shifts. We use BiGRU to collectively encode all
sequences in the historical data
$S=\left\\{\hat{\mathbf{x}}_{n}^{(i)}\right\\}_{n=1}^{L}$.

  * •

We first apply an 1D average pooling with kernel size $k$ and stride $k$ over
$\hat{\mathbf{x}}_{n}^{(i)}$ on time dimension to down sample
$\hat{\mathbf{x}}_{n}^{(i)}$ into its $\frac{1}{k}$ slice:

$\medop{\overline{\mathbf{x}}_{n}^{(i)}=\operatorname{AvgPool1D}\left(\hat{\mathbf{x}}_{n}^{(i)}\right)}$
(1)

where
$\overline{\mathbf{x}}_{n}^{(i)}\in\mathbb{R}^{d_{\mathbf{x}}\times\frac{T}{k}}$.
Equ. (1) reduces the memory usage while preserving most of the information in
the historical data [4]. We empirically show that such down-sampling has
little impact on the prediction accuracy (see Sec.
LABEL:sec:analysis:average).

  * •

Then we feed $\overline{\mathbf{x}}_{n}^{(i)}$ into a biGRU to generate the
state sequence $\mathbf{h}_{n}^{(i)}\in\mathbb{R}^{\frac{T}{k}\times d_{h}}$:

$\displaystyle\medop{\mathbf{h}_{n}^{(i)}=\left\\{h_{n,(t)}^{(i)}\right\\}_{t=1}^{\frac{T}{k}}}$
$\displaystyle=\medop{\operatorname{BiGRU}\left(\overline{\mathbf{x}}_{n}^{(i)}\right)}$
(2)

where $h_{n,(t)}^{(i)}\in\mathbb{R}^{d_{h}}$ is a hidden state with feature
dimension $d_{h}$. Note that $\mathbf{h}_{n}^{(i)}$ can be considered as the
distribution characterization for $\overline{\mathbf{x}}_{n}^{(i)}$.

The hyper layer explicitly models and characterizes the distributions from the
historical data. Such information is then utilized to generate the weights of
the main layers, as explained below.

###  IV-C Main Layers

The main layers can adaptively change their parameters conditioned on
$\mathbf{h}_{n}^{(i)}$ and $\mathbf{x}^{i}$. The core part of a main layer is
a simple GRU cell. To illustrate its weights generation process, we first
present the standard formulation of the GRU cell:

$\displaystyle\medop{r_{t}}$
$\displaystyle=\medop{\sigma\left(W_{xr}{x}_{t}^{(i)}+W_{hr}{s}_{t-1}^{(i)}+b_{r}\right)}$
(3) $\displaystyle\medop{z_{t}}$
$\displaystyle=\medop{\sigma\left(W_{xz}{x}_{t}^{(i)}+W_{hz}{s}_{t-1}^{(i)}+b_{z}\right)}$
$\displaystyle\medop{n_{t}}$
$\displaystyle=\medop{\tanh\left(W_{xn}{x}_{t}^{(i)}+r_{t}\odot\left(W_{hn}\mathbf{s}_{t-1}^{(i)}++b_{n}\right)\right)}$
$\displaystyle\medop{{s}_{t}^{(i)}}$
$\displaystyle=\medop{\left(1-z_{t}\right)\odot
n_{t}+z_{t}\odot\mathbf{s}_{t-1}^{(i)}}$

where ${x}_{t}^{(i)}\in\mathbb{R}^{d_{\mathbf{x}}}$ denotes input of the GRU
cell from $\mathbf{x}^{(i)}$ at time step $t$,
${s}_{t}^{(i)}\in\mathbb{R}^{d_{s}}$ is the hidden state at $t$, $\sigma$ is
sigmoid function, $W_{xr}$, $W_{xz}$, $W_{xn}$, $W_{hr}$, $W_{hz}$, $W_{hn}$,
$b_{r}$, $b_{z}$ and $b_{n}$ are learnable parameters.

The initial hidden state ${s}_{0}^{(i)}$ is computed as follows:

$\medop{s_{0}^{(i)}=W_{init}h_{n,\left(\frac{T}{k}\right)}^{(i)}\mid+b_{init}}$
(4)

where $h_{n,\left(\frac{T}{k}\right)}^{(i)}$ is the last state in
$\mathbf{h}_{n}^{(i)}$; $W_{init}$ and $b_{init}$ are learnable parameters.

The weights of the GRU cell at current time step $t$ are controlled by a
vector ${v}_{t}$ that varies dynamically with $t$ :

$\medop{{W}_{Ih}={W}_{hv}{v_{t}},\,{W}_{Ix}={W}_{xv}{v_{t}},\,{W}_{Ib}={W}_{bv}{v_{t}}}$
(5)

where ${{W}_{hv}}$, ${{W}_{xv}}$ and ${{W}_{bv}}$ are learnable parameters.
${W}_{Ih}\in\mathbb{R}^{d_{Ih}}$, ${W}_{Ix}\in\mathbb{R}^{d_{Ix}}$, and
${W}_{Ib}\in\mathbb{R}^{d_{Ib}}$ are the weights of the GRU cell, ${v}^{t}$ is
determined by the previous state $s^{(i)}_{t-1}$ and the outputs of the hyper
layers $\mathbf{h}_{n}^{(i)}$, which is computed by the attention mechanism:

$\displaystyle\medop{{v}_{t}}=\medop{{W_{c}}{c}_{t}},\,\medop{{c}_{t}}=\medop{\sum_{p=1}^{\frac{T}{k}}\alpha_{p}{h}_{n,(p)}^{(i)}},\,\medop{\alpha_{p}}=\medop{\frac{\exp\left(\operatorname{score}\left({s}^{(i)}_{t-1},{h}_{n,(p)}^{(i)}\right)\right)}{\sum_{q=1}^{\frac{T}{k}}\exp\left(\mathrm{score}\left({s}^{(i)}_{t-1},{h}_{n,(q)}^{(i)}\right)\right)}}$
(6)

where $\operatorname{score}(\cdot)$ is a score function, according to Luong
attention [33]. The score function is given by:

$\medop{\operatorname{score}\left({s}^{(i)}_{t-1},{h}_{n,(p)}^{(i)}\right)=V^{\mathrm{T}}\operatorname{tanh}(W_{s}{s}^{(i)}_{t-1}+W_{h}{h}_{n,(p)}^{(i)}+b_{s})}$
(7)

where $V$, $W_{s}$, $W_{h}$, and $b_{s}$ are learnable parameters.

As mentioned above, all the parameters of the main layer are generated by
${{W}_{Ih}}$, ${{W}_{Ix}}$, and ${{W}_{Ib}}$:

$\displaystyle\medop{\left(W_{hr},W_{hz},W_{hn}\right)}$
$\displaystyle=\medop{\operatorname{Chunk}\left(W_{Ih}\right)}$ (8)
$\displaystyle\medop{\left(W_{xr},W_{xz},W_{xn}\right)}$
$\displaystyle=\medop{\operatorname{Chunk}\left(W_{Ix}\right)}$
$\displaystyle\medop{\left(b_{r},b_{z},b_{n}\right)}$
$\displaystyle=\medop{\operatorname{Chunk}\left(W_{Ib}\right)}$

where $W_{hr}$, $W_{hz}$, $W_{hn}\in\mathbb{R}^{d_{s}\times d_{s}}$,
$d_{s}\times 3d_{s}=d_{Ih}$ and $W_{xr}$, $W_{xz}$,
$W_{xn}\in\mathbb{R}^{d_{s}\times{d_{\mathbf{x}}}}$ , $d_{\mathbf{s}}\times
3d_{x}=d_{Ix}$ and $b_{r}$, $b_{z}$, $b_{n}\in\mathbb{R}^{d_{s}}$ ,
$3d_{s}=d_{Ib}$. $\operatorname{Chunk(\cdot)}$ is the operation to split a
tensor into a specific number of equal-sized chunks. Through Equ. (6), we can
adaptively attach different information in $\mathbf{h}_{n}^{(i)}$ to the state
of the GRU cell at the current time step $t$ to dynamically adjust the weights
of the main layers. By integrating Equ. (3), Equ. (5) and Equ. (6), we can
dynamically generate weights of the main layers.

###  IV-D Training

According to Sec. IV-B and Sec. IV-C, we implement HyperGRU with a simple
prediction layer(_e.g._ , sigmoid function) that takes
$\hat{\mathbf{x}}_{n}^{(i)}$ and $\mathbf{x}^{(i)}$ as inputs and outputs the
prediction:

$\medop{\hat{\mathbf{y}}^{(i)}_{n}=\operatorname{HyperGRU}(\mathbf{x}^{(i)},\hat{\mathbf{x}}_{n}^{(i)})}$
(9)

The loss of the $i$-th pair $(\mathbf{x}^{(i)},\mathbf{y}^{(i)})$ is
calculated as follows:

$\medop{\mathcal{L}_{\text{pred
}}(\theta)=\frac{1}{L}\sum_{n=1}^{L}\operatorname{criterion}\left(\hat{\mathbf{y}}^{(i)}_{n},\mathbf{y}^{i}\right)}$
(10)

where $\operatorname{criterion}\left(\cdot,\cdot\right)$ is an objective
function, and $\theta$ denotes the learnable parameters of HyperGRU. Then we
perform training using Equ. (10).

###  IV-E Extensions to Other Models

As mentioned, HTSF is a model-agnostic framework that can incorporate various
time series forecasting models. We briefly explain how to extend HTSF to other
RNNs such as vanilla RNN, LSTM[34], ConvLSTM[35]_et al._.

Extending HTSF to other RNNs is very straightforward. Take the LSTM as an
example, we can use Equ. (6) to calculate $v_{t}$, and use $v_{t}$ to generate
the weights of memory cell, input gate, forget gate and output gate. The
weights of other RNNs can be generated in a similar way.

##  V Experiments

In this section, we evaluate the performance of HTSF on 9 benchmarks and
different time series models covering short sequence time series forecasting
(Sec. V-A), grid-based spatiotemporal data forecasting (Sec. V-B), and graph-
based data spatiotemporal forecasting (Sec. V-C). All experiments were
conducted out on a single Nvidia RTX3090 GPU.

###  V-A Short Sequence Time Series Forecasting

We test HTSF for short sequence time series forecasting on three datasets from
diverse application domains.

####  V-A1 Datasets and Metrics

  * •

Air Quality [1]: This dataset contains hourly air quality data collected from
12 stations in Beijing, China, from March 2013 to February 2017. We use data
from the same four stations (Dongsi, Tiantan, Nongzhanguan, and Dingling) and
the same six features (PM2.5, PM10, SO2, NO2, CO, and O3) as [12]. We use the
data from 01/03/2013 to 30/06/2016 for training, data from 01/07/2016 to
31/10/2016 for validation, and data from 02/11/016 to 28/02/2017 for testing.
The Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) are used as
the performance metrics.

  * •

Electricity Consumption [36]: This dataset contains household electric power
consumption measurements of 2, 062, 346 valid samples (null values removed).
The measurements were collected between 16/12/2006 and 26/11/2010. Each
interval lasts 10 min. We use the data from 16/12/2006 to 24/04/2009 for
training, data from 25/02/010 to 26/11/2010 for testing, and the remaining for
validation. RMSE is used as the performance metric.

  * •

Human Activity [3]: This dataset contains smartphone sensor data
(accelerometer, gyroscope, and magnetometer) collected from 30 volunteers
performing six activities. We predict the activities of volunteers based on
the historical data. We use 7,352 instances for training and 2, 947 instances
for testing following [12]. We use Accuracy (ACC), Precision (P), Recall (R),
and F1 as the performance metrics.

TABLE I: Datasets for short sequence time series forecasting. Datasets | #Train | #Valid | #Test
---|---|---|---
Air quality | 29,232 | 2,904 | 2,832
Electricity consumption | 1,235,964 | 413,280 | 413,202
Human activity | 6,000 | 1,352 | 2,947
TABLE II: Performance of Air quality and Electricity Consumption. Methods | Dongsi | Tiantan | Nongzhanguan | Dingling | Electric Power
---|---|---|---|---|---
RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE
LSTM | 0.0502 | 0.0403 | 0.0511 | 0.0453 | 0.0341 | 0.0476 | 0.0329 | 0.0339 | 0.092
GRU | 0.0510 | 0.0380 | 0.0519 | 0.0475 | 0.0348 | 0.0459 | 0.0330 | 0.0347 | 0.093
LSTNet | 0.0544 | 0.0651 | 0.0519 | 0.0651 | 0.0548 | 0.0696 | 0.0599 | 0.0705 | 0.080
MMD-RNN | 0.0360 | 0.0267 | 0.0183 | 0.0133 | 0.0267 | 0.0197 | 0.0288 | 0.0168 | 0.082
DANN-RNN | 0.0356 | 0.0255 | 0.0214 | 0.0157 | 0.0274 | 0.0203 | 0.0291 | 0.0211 | 0.080
STRIPE | 0.0365 | 0.0216 | 0.0204 | 0.0148 | 0.0248 | 0.0154 | 0.0304 | 0.0139 | 0.086
ADARNN | 0.0295 | 0.0185 | 0.0164 | 0.0112 | 0.0196 | 0.0122 | 0.0233 | 0.0150 | 0.077
HyperLSTM | 0.0278 | 0.0178 | 0.0155 | 0.0107 | 0.0189 | 0.0120 | 0.0231 | 0.0134 | 0.075
HyperGRU | 0.0285 | 0.0182 | 0.0156 | 0.0111 | 0.0184 | 0.0119 | 0.0229 | 0.0138 | 0.075
TABLE III: Performance of human activity prediction Methods | ACC | P | R | F1 |
---|---|---|---|---|---
LightGBM | 84.11 | 83.73 | 83.63 | 84.91 |
GRU | 85.68 | 85.62 | 85.51 | 85.46 |
MMD-RNN | 86.39 | 86.80 | 86.26 | 86.38 |
DANN-RNN | 85.88 | 85.59 | 85.62 | 85.56 |
ADARNN | 88.44 | 88.71 | 88.59 | 88.63 |
HyperGRU | 89.61 | 89.64 | 89.45 | 89.43 |
|  |  |  |  |

####  V-A2 Baselines and Configurations

We adopt GRU/LSTM as the hyper layers and main layers for our HTSF framework
because they suffice for short sequence prediction in many applications [10,
18]. The corresponding models are termed HyperGRU and HyperLSTM. We compare
them with the following baselines:

  * •

LSTM : a standard long short memory network.

  * •

GRU : a standard gated recurrent unit.

  * •

LightGBM : a lightweight and efficient gradient boosting decision tree.

  * •

MMD-RNN : a domain generalization method, whose backbone is changed to RNNs
and uses maximum mean discrepancy measure as the final output of the RNN
layers.

  * •

DANN-RNN : the RNN version of a dynamic adversarial adaptation network to
dynamically learn domain-invariant representations [12].

  * •

LSTNet : a method combining Convolutional Neural Networks(CNNs) and LSTM for
time series forecasting.

  * •

STRIPE : an Encoder-Decoder model with a diversification mechanism from a
determinantal point process.

  * •

ADARNN : the state-of-the-art model to handle distribution shift in time
series forecasting.

These baselines are chosen for the following reasons. LSTM and GRU are basic
models in our HyperLSTM and HyperGRU. LightGBM is a popular and strong
traditional machine learning algorithm in Kaggle competitions. LSTNet and
STRIPE are state-of-the-art RNN-based methods without considering the problem
of distribution shift. MMR-RNN, DANN-RNN, and ADARNN are recent models
attempting to solve the distribution shift problem in time series, where
ADARNN is state-of-the-art.

For all datasets, we use Adam [37] with weight decay regularization as the
optimizer. The dataset-specific configurations are summarized below.

  * •

Air Quality: We use L2 loss as the training objective. The learning rate is
2e-4 with a weight decay of 0.01, and the batch size is 128. We set sliding
window size $T=672$ (28 consecutive days), $k=24$ for 1D average pooling.

  * •

Electricity Consumption: We use L2 loss as the training objective and 5e-5 as
the learning rate. We set sliding window size $T=1008$ (one week), $k=36$ for
1D average pooling, and the batch size is set to 64.

  * •

Human Activity: We use cross-entropy as the training objective and the
learning rate is 2e-5 with a weight decay of 0.01. We set sliding window size
$T=128$, $k=36$ for 1D average pooling, and the batch size is set to 256.

We set the prediction step $T_{\mathbf{y}}=1$ for all benchmarks as in [12].

####  V-A3 Results

The left part of Table II shows the RMSE and MAE of different methods for the
air quality prediction tasks. The right part of Table II presents the RMSE of
predicting the power consumption. Table III lists the ACC, P, R, and F1 scores
of different methods of activity classification. Since LSTNet and STRIPE are
built for regression tasks, they are not applicable to this dataset. We make
the following observations.

  * •

HyperLSTM outperforms LSTNet and STRIPE, the methods that ignore distribution
shift, by decreasing the RMSE/MAE by 61.4%/80.6% and 23.9% /34.4% in average.
This indicates the importance of addressing the distribution shift problem.

  * •

HyperLSTM and HyperGRU outperform MMD-RNN and DANN-RNN on all benchmarks,
suggesting that HTSF better handles distribution shift than existing domain
adaptation and generalization methods for time series forecasting.

  * •

Compared to ADARNN, one state-of-the-art method for handling distribution
shift, HyperLSTM and HyperGRU decrease the RMSE by 3.94% and 3.82% averaged on
the air quality dataset; and 2.60% and 2.60% on the electricity consumption
dataset. HyperGRU also outperforms ADARNN by increasing ACC by 1.32% and F1
score by 1.04% on the human activity dataset. These results indicate that HTSF
can better characterize the distribution shifts in data.

###  V-B Grid-based Spatiotemporal Forecasting

In this subsection, we demonstrate the applicability of HTSF to spatiotemporal
data (grid-based). We are interested in the gains by explicitly accounting for
the distribution shifts in spatiotemporal data forecasting.

####  V-B1 Datasets and Metrics.

We experiment with NYC-Taxi and NYC-Bike, two datasets of trip records
collected in New York City (NYC) with grid partitioned regions for traffic
prediction. NYC-Taxi contains taxi trip records of NYC from 01/01/2015 to
03/01/2015. NYC-Bike contains bike trip records of NYC from 07/01/2016 to
08/29/2016. For both datasets, The first 40 days are used as the training set.
We use the Mean Absolute Percentage Error (MAPE) and RMSE as the performance
metrics.

####  V-B2 Baselines and Configurations

A mainstream strategy to process spatiotemporal data with grid partition is to
combine CNNs and RNNs[38, 39]. We choose three representative methods adopting
this strategy as the baselines.

  * •

ConvLSTM [35]: an extension of LSTM using convolution operators to build
cells.

  * •

DMVST-Net [38]: it combines CNN and LSTM in a joint model that captures the
complex relations from both spatial and temporal perspectives.

  * •

STDN [39]: it designs a periodically shifted attention mechanism to capture
long-term dependency and applies LSTM to learn short-term dependency.

We pick these three methods because ConvLSTM is one of the earliest deep
learning models for grid-based spatiotemporal data analysis, while DMVST-Net
and STDN are very strong baselines for NYC-Taxi and NYC-Bike.

We adapt HTSF to spatiotemporal forecasting as follows.

  * •

We adopt ConvLSTM as both the hyper layers and main layers to build the
hypernetwork-based model for ConvLSTM, which we term HyperConvLSTM.

  * •

Extensions to STDN and DMVST-Net are more challenging because STDN and DMVST-
Net have dedicated components to learn spatial patterns, while HTSF is
designed only to learn temporal patterns. Therefore, we reuse the
corresponding spatial components in STDN and DMVST-Net but replace their
temporal components with HyperLSTM. The corresponding models are termed STDN
w/ HyperLSTM and DMVST-Net w/ HyperLSTM.

The region is split into $10\times 20$ grids with size $1km\times 1km$. The
Min-Max normalization is used to convert data value to the $[0,1]$ scale. The
time interval is set as half-hour. We set the sliding window size $T=1344$
(four weeks), and $k=48$ for 1D average pooling. L2 loss is used for training.
We optimize all the models by Adam, and the learning rate is set to 2e-4 with
a weight decay of 0.01.

TABLE IV: HTSF adapted for grid-based urban traffic prediction. Dataset | Method | pick-up | drop-off
---|---|---|---
RMSE | MAPE | RMSE | MAPE
NYC-Taxi | ConvLSTM | 28.13 | 20.50 | 23.67 | 20.70
DMVST-Net | 25.74 | 17.38 | 20.51 | 17.14
STDN | 24.10 | 16.30 | 19.05 | 16.25
HyperConvLSTM | 26.45 | 19.41 | 22.15 | 29.64
DMVST-Net w/HTSF | 24.26 | 16.59 | 19.13 | 16.53
STDN w/HTSF | 23.26 | 15.74 | 18.53 | 16.26
NYC-Bike | ConvLSTM | 10.40 | 25.10 | 9.22 | 23.20
DMVST-Net | 9.14 | 22.20 | 8.50 | 21.56
STDN | 8.85 | 21.84 | 8.15 | 20.87
HyperConvLSTM | 9.73 | 24.33 | 8.64 | 21.82
DMVST-Net w/HTSF | 8.66 | 21.51 | 7.97 | 20.45
STDN w/HTSF | 8.13 | 21.23 | 7.62 | 19.97

####  V-B3 Results

Table IV lists the results of traffic flow prediction on NYC-Taxi and NYC-
Bike. We make the following observations.

  * •

The best performance is achieved with HTSF, indicating the benefit of
mitigating distribution shift in traffic prediction.

  * •

For the three baselines, the corresponding hypernetwork-based architecture
reduces the pick-up RMSE by 3.49% to 6.27% on NYC-Taxi and 5.54% to 8.13% on
NYC-Bike; and reduces the drop-off RMSE by 2.73% to 6.73% on NYC-Taxi and
6.23% to 6.50% on NYC-Bike. Therefore, our hypernetwork-based framework is
easily applicable in grid-based spatiotemporal data forecasting and notably
improves prediction accuracy.

###  V-C Graph-based Spatiotemporal Forecasting

Since graph neural network (GNN) based models have achieved the state-of-the-
art performance in urban traffic forecasting[18, 19], we are interested in
whether HTSF can further improve the performance of GNN-based models.

####  V-C1 Datasets and Metrics.

We conduct experiments on PeMSD3, PeMSD4, PeMSD7, and PeMSD8, which are graph-
based traffic datasets collected by the Caltrans Performance Measurement
System [40] for traffic flow prediction. All the datasets are split with a
ratio of 6:2:2 for training, validation, and testing. The traffic flows are
aggregated into 5-minute interval. Table V summarizes the detailed information
of the datasets. RMSE, MAE, and MAPE are used as performance metrics.

TABLE V: Dataset description of PeMSD. Datasets | #Detectors | Range
---|---|---
PeMSD3 | 358 | 9/1/2018-11/30/2018
PeMSD4 | 307 | 1/1/2018-2/28/2018
PeMSD7 | 883 | 5/1/2017-8/31/2017
PeMSD8 | 170 | 7/1/2016-8/31/2016

####  V-C2 Baselines and Configurations

We use the baselines below.

  * •

DCRNN[41]: an extension of RNN using graph convolutional operators to build
cells.

  * •

STGCN [42]: it formulates traffic forecasting on graphs and builds the model
with GCNs.

  * •

ASTGCN [43]: an attention based spatiotemporal GCN, which can effectively
model the dynamic spatiotemporal correlations in traffic data.

  * •

GraphWaveNet [44]: it can capture the hidden spatial dependency via a novel
adaptive dependency matrix.

  * •

AGCRN [18]: an adaptive graph convolutional recurrent network to capture
spatial and temporal correlations.

  * •

STFGNN [19]: a spatiotemporal fusion GNN, which can effectively learn hidden
spatiotemporal patterns by a fusion operation of various spatial and temporal
graphs.

The selected GNN-based models include representative ([41, 42]) and state-of-
the-art ([18, 19]) models for traffic forecasting.

We compare these methods with our HTSF using pure RNN-based models (HyperLSTM
and HyperGRU) and an extension of HTSF to AGCRN, a GNN-based model.
Specifically, we adopt AG-
CRN as the hyper layers and main layers to HTSF, and the corresponding model
is termed HyperAGCRN. For all four datasets, we use the data of past 12 steps
to predict the future 12 steps. We set sliding window size $T=2016$ (one
week), and $k=24$ for 1D average pooling. We optimize all the models by Adam
with batch size 128. The learning rate is set to 3e-4 with a weight decay of
0.01.

TABLE VI: HTSF adapted for graph-based traffic prediction. Methods | PeMSD3 | PeMSD4 | PeMSD7 | PeMSD8
---|---|---|---|---
RMSE | MAE | MAPE | RMSE | MAE | MAPE | RMSE | MAE | MAPE | RMSE | MAE | MAPE
LSTM | 35.11 | 21.33 | 21.33 | 39.59 | 25.14 | 20.33 | 42.84 | 29.98 | 15.33 | 32.06 | 22.20 | 15.32
GRU | 36.23 | 21.47 | 21.06 | 38.87 | 25.36 | 20.49 | 41.93 | 30.14 | 15.71 | 32.24 | 23.03 | 15.62
DCRNN | 30.31 | 18.18 | 18.91 | 38.12 | 24.70 | 17.12 | 38.58 | 28.30 | 11.66 | 27.83 | 17.86 | 11.45
STGCN | 30.12 | 17.49 | 17.15 | 35.55 | 22.70 | 14.59 | 38.78 | 25.38 | 11.08 | 27.83 | 18.02 | 11.40
ASTGCN | 29.66 | 17.69 | 19.40 | 35.22 | 22.93 | 16.56 | 42.57 | 28.05 | 13.92 | 28.16 | 18.61 | 13.08
GraphWaveNet | 32.94 | 19.85 | 19.31 | 39.70 | 25.45 | 17.29 | 42.78 | 26.85 | 12.12 | 31.05 | 19.13 | 12.68
AGCRN | - | - | - | 32.30 | 19.83 | 12.97 | - | - | - | 25.22 | 15.95 | 10.09
STFGNN | 26.28 | 16.77 | 16.30 | 32.51 | 20.48 | 16.77 | 36.60 | 23.46 | 9.21 | 26.25 | 16.94 | 10.60
HyperLSTM | 32.36 | 19.21 | 19.04 | 37.19 | 23.65 | 18.46 | 38.67 | 27.18 | 13.17 | 28.12 | 19.20 | 13.68
HyperGRU | 32.18 | 19.42 | 18.79 | 36.94 | 23.47 | 19.14 | 38.75 | 27.31 | 12.73 | 27.84 | 19.33 | 14.23
HyperAGCRN | 25.68 | 15.85 | 14.37 | 29.93 | 18.89 | 12.11 | 34.37 | 21.64 | 8.89 | 24.37 | 15.26 | 9.64

####  V-C3 Results

Table VI summarizes the average results on graph-based spatiotemporal
forecasting. We have the following observations.

  * •

GNN-based models outperform LSTM and GRU since GNN-based models take advantage
of modeling spatial patterns while LSTM and GRU only model the temporal
patterns.

  * •

Our HyperLSTM and HyperGRU, despite modeling only the temporal patterns,
outperform multiple GNN-based models. For example, HyperLSTM and HyperLSTM
have better RMSE, MAE, and MAPE than GraphWaveNet on PeMSD3, PeMSD4, and
PeMSD7. The reason might be that the temporal distribution shift may be a more
important bottleneck than the spatial patterns in the accuracy of traffic
prediction.

  * •

HyperAGCRN achieves the best performance across all datasets. This
demonstrates that HTSF can improve the performance of GNN-based models.

##  VI Ablation Studies

###  VI-A Visual Analysis of Distribution Shift

####  VI-A1 Setups

In this study, we demonstrate the problem of distribution shift in time series
via visual analysis of the Air Quality dataset. For each station, we randomly
sample one instance $\mathbf{x}^{(i)}$ with its corresponding historical
dataset $S=\left\\{\hat{\mathbf{x}}_{n}^{(i)}\right\\}_{n=1}^{L}$, where $S$
contains 128 sequences, _i.e._ , $L=128$. We use t-SNE [45] to visualize the
learned distributions (via the hyper layers) of all sequences in $S$ and their
original distributions.

####  VI-A2 Results

From Figure 3, we observe that the original distributions are random while the
learned distributions can be approximately fitted by a linear model,
indicating that common knowledge among all historical sequences has been
learned by HTSF.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: Visualization of distribution shifts in the Air Quality dataset. The
upperparts (a, b, c, d) present the original distribution visualization of
four stations (Dongsi, Tiantan, Nongzhangguan, Dingling). The lower parts (e,
f, g, h) present the learned distribution visualization of four stations
(Dongsi, Tiantan, Nongzhangguan, Dingling). One scatter presents a sequence in
$S$.

###  VI-B Impact of Sliding Window Length

####  VI-B1 Setups

The length of the sliding window $T$ is an important hyper-parameter in HTSF.
We experiment with different lengths using HyperGRU on the datasets of Air
Quality, NYC-Taxi and NYC-Bike. As mentioned in Sec. IV-B, the actual input
length of the hyper layers is $\frac{T}{k}$. Let $T_{k}=\frac{T}{k}$, to
observe how sensitive the model is to $T_{K}$, we vary $T_{k}$ from 2 to 256.
Length 2 is the shortest length of a sequence, and Length 256 is challenging
for optimization of GRU.

####  VI-B2 Results

Figure 4 plots the performance variations of HyperGRU on Air-quality, NYC-Taxi
and NYC-Bike as the sliding window length increases. We observe high error (in
RMSE) when $T_{k}=2$. One potential reason may be that it is difficult to
learn complex feature representations of the distribution shifts given very
short sequences. The RMSE only experiences a small variance when $T_{k}$
increases from 4 to 128. When $T_{k}\geq 192$, the RMSE starts to grow again.
One potential reason is that when considering a very long length, the training
becomes harder. This results show that HTSF is insensitive to sliding window
length, except in extreme cases($T_{k}=2$ or $T_{k}\geq 192$).

(a) Air Quality

(b) NYC

Figure 4: Impact of sliding window size $T$ on the forecasting accuracy.

###  VI-C Impact of Prediction Horizon

####  VI-C1 Setups

Multi-step prediction capability is crucial for traffic flow forecasting. We
analyze the performance of HyperGRU for multi-step prediction on PeMSD4 and
PeMSD8 datasets.

####  VI-C2 Results

Figure 5 shows the prediction performance of various methods as the horizon
increases. Overall, the errors (in MAE) grow with the forecast step. However,
for both datasets, the errors of AGCRN and HyperAGCRN increase much more
slowly than GRU and HyperGRU. HTSF significantly improves the performance of
GRU and HyperAGCRN on multi-step prediction, while our HyperAGCRN achieves the
best results for all horizons. These results indicate that mitigating the
distribution shift problem using HTSF is effective in mining the dynamic
patterns of spatiotemporal data.

(a) PeMSD4

(b) PeMSD8

Figure 5: Multi-step prediction performance on PeMSD.

##  VII Conclusion

In this paper, we investigate the distribution shift problem in the time
series forecasting problem, which causes discrepancies between the
distributions of the training and the testing data. To this end, we propose
HTSF, a novel hypernetwork-based framework which applies for time series
forecasting under distribution shift. Specifically, HTSF exploits the hyper
layers to learn the best characterization of the distribution shifts,
generating the model parameters for the main layers to make accurate
predictions. Moreover, HTSF is implemented as an extensible framework that can
incorporate diverse time series forecasting models such as RNNs and
Transformers. Extensive experiments show that HTSF outperforms other state-of-
the-art methods on 9 benchmarks.

## References

  * [1] S. Zhang, B. Guo, A. Dong, J. He, Z. Xu, and S. X. Chen, “Cautionary tales on air-quality improvement in beijing,” _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 473, no. 2205, p. 20170457, 2017.
  * [2] D. Lee and K. Kim, “Recurrent neural network-based hourly prediction of photovoltaic power output using meteorological information,” _Energies_ , vol. 12, no. 2, pp. 1–22, 2019.
  * [3] B. Almaslukh, J. AlMuhtadi, and A. Artoli, “An effective deep autoencoder approach for online smartphone-based human activity recognition,” _Int. J. Comput. Sci. Netw. Secur_ , vol. 17, no. 4, pp. 160–165, 2017.
  * [4] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_. Palo Alto, CA, USA: AAAI Press, 2021, pp. 11 106–11 115.
  * [5] J. D. Hamilton, _Time series analysis_. Princeton, NJ, USA: Princeton university press, 2020.
  * [6] J. Connor, L. E. Atlas, and D. R. Martin, “Recurrent networks and narma modeling,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 1992, pp. 301–308.
  * [7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2017, pp. 6000–6010.
  * [8] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-and short-term temporal patterns with deep neural networks,” in _Proceedings of the International SIGIR Conference on Research & Development in Information Retrieval_. New York, NY, USA: ACM, 2018, pp. 95–104.
  * [9] V. Le Guen and N. Thome, “Probabilistic time series forecasting with structured shape and temporal diversity,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2020, pp. 4427–4440.
  * [10] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski, “Deepar: Probabilistic forecasting with autoregressive recurrent networks,” _International Journal of Forecasting_ , vol. 36, no. 3, pp. 1181–1191, 2020\.
  * [11] V. Kuznetsov and M. Mohri, “Generalization bounds for time series prediction with non-stationary processes,” in _Proceedings of the International Conference on Algorithmic Learning Theory_. Berlin, Germany: Springer, 2014, pp. 260–274.
  * [12] Y. Du, J. Wang, W. Feng, S. Pan, T. Qin, and C. Wang, “Adarnn - adaptive learning and forecasting of time series,” in _Proceedings of the International Conference on Information and Knowledge Management_. New York, NY, USA: ACM, 2021, pp. 402–411.
  * [13] A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in _Proceedings of the Conference on Computer Vision and Pattern Recognition_. Piscataway, NJ, USA: IEEE Press, 2011, pp. 1521–1528.
  * [14] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in _Proceedings of the Conference on Computer Vision and Pattern Recognition_. Piscataway, NJ, USA: IEEE Press, 2017, pp. 2962–2971.
  * [15] J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin, “Generalizing to unseen domains: A survey on domain generalization,” in _Proceedings of the International Joint Conference on Artificial Intelligence_. Burlington, MA, USA: Morgan Kaufmann, 2021, pp. 4627–4635.
  * [16] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, and P. S. Yu, “Visual domain adaptation with manifold embedded distribution alignment,” in _Proceedings of the International Conference on Multimedia_. New York, NY, USA: ACM, 2018, pp. 402–410.
  * [17] T. Kim, J. Kim, Y. Tae, C. Park, J.-H. Choi, and J. Choo, “Reversible instance normalization for accurate time-series forecasting against distribution shift,” Proceedings of the International Conference on Learning Representations, 2022.
  * [18] L. Bai, L. Y. andCan Li, X. Wang, and C. Wang, “Adaptive graph convolutional recurrent network for traffic forecasting,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2020, pp. 17 804–17 815.
  * [19] M. Li and Z. Zhu, “Spatial-temporal fusion graph neural networks for traffic flow forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_. Palo Alto, CA, USA: AAAI Press, 2021, pp. 4189–4196.
  * [20] P. J. Brockwell, R. A. Davis, J. O. Berger, S. E. Fienberg, and B. Singer, _Time Series: Theory and Methods_. Berlin, Germany: Springer, 1987.
  * [21] A. A. Ariyo, A. O. Adewumi, and C. K. Ayo, “Stock price prediction using the ARIMA model,” in _Proceedings of the International Conference on Computer Modelling and Simulation_. Piscataway, NJ, USA: IEEE Press, 2014, pp. 106–112.
  * [22] V. Le Guen and N. Thome, “Shape and time distortion loss for training deep time series forecasting models,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2019, pp. 4189–4201.
  * [23] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, “Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2019, pp. 5243–5253.
  * [24] J. Jiang, Y. Ji, X. Wang, Y. Liu, J. Wang, and M. Long, “Regressive domain adaptation for unsupervised keypoint detection,” in _Proceedings of the Conference on Computer Vision and Pattern Recognition_. Piscataway, NJ, USA: IEEE Press, 2021, pp. 6776–6785.
  * [25] D. Ha, A. M. Dai, and Q. V. Le, “Hypernetworks,” Proceedings of the International Conference on Learning Representations, 2017.
  * [26] C. Zhang, M. Ren, and R. Urtasun, “Graph hypernetworks for neural architecture search,” Proceedings of the International Conference on Learning Representations, 2019.
  * [27] J. von Oswald, C. Henning, J. Sacramento, and B. F. Grewe, “Continual learning with hypernetworks,” Proceedings of the International Conference on Learning Representations, 2020.
  * [28] T. Galanti and L. Wolf, “On the modularity of hypernetworks,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2020, pp. 10 409–10 419.
  * [29] L. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi, “Learning feed-forward one-shot learners,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2016, pp. 523–531.
  * [30] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson, “Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks,” in _Proceedings of the Annual Meeting of the Association for Computational Linguistics_. Stroudsburg, PA, USA: ACL, 2021, pp. 565–576.
  * [31] W. Duan, X. He, Z. Zhou, H. Rao, and L. Thiele, “Injecting descriptive meta-information into pre-trained language models with hypernetworks,” in _Proceedings of the Conference of the International Speech Communication Association_. ISCA, 2021, pp. 3216–3220.
  * [32] Z. Pan, Y. Liang, W. Wang, Y. Yu, Y. Zheng, and J. Zhang, “Urban traffic prediction from spatio-temporal data using deep meta learning,” in _Proceedings of the SIGKDD International Conference on Knowledge Discovery & Data Mining_. New York, NY, USA: ACM, 2019, pp. 1720–1730.
  * [33] M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” arXiv preprint arXiv:1508.04025, 2015.
  * [34] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [35] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo, “Convolutional LSTM network: A machine learning approach for precipitation nowcasting,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates Inc., 2015, pp. 802–810.
  * [36] Kaggle, “Household electric power consumption dataset,” https://www.kaggle.com/uciml/electric-power-consumption-data-set, 2021.
  * [37] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Proceedings of the International Conference on Learning Representations, 2015\.
  * [38] H. Yao, F. Wu, J. Ke, X. Tang, Y. Jia, S. Lu, P. Gong, J. Ye, and Z. Li, “Deep multi-view spatial-temporal network for taxi demand prediction,” in _Proceedings of the AAAI Conference on Artificial Intelligence_. Palo Alto, CA, USA: AAAI Press, 2018, pp. 2588–2595.
  * [39] H. Yao, X. Tang, H. Wei, G. Zheng, and Z. Li, “Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction,” in _Proceedings of the AAAI Conference on Artificial Intelligence_. Palo Alto, CA, USA: AAAI Press, 2019, pp. 5668–5675.
  * [40] C. Chen, K. Petty, A. Skabardonis, P. Varaiya, and Z. Jia, “Freeway performance measurement system: Mining loop detector data,” _Transportation Research Record_ , vol. 1748, no. 1, pp. 96–102, 2001.
  * [41] Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” Proceedings of the International Conference on Learning Representations, 2018.
  * [42] B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting,” in _Proceedings of the International Joint Conference on Artificial Intelligence_. Burlington, MA, USA: Morgan Kaufmann, 2018, pp. 3634–3640.
  * [43] S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan, “Attention based spatial-temporal graph convolutional networks for traffic flow forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_. Palo Alto, CA, USA: AAAI Press, 2019, pp. 922–929.
  * [44] Z. Wu, S. Pan, G. Long, J. Jiang, and C. Zhang, “Graph wavenet for deep spatial-temporal graph modeling,” in _Proceedings of the International Joint Conference on Artificial Intelligence_. Burlington, MA, USA: Morgan Kaufmann, 2019, pp. 1907–1913.
  * [45] V. D. M. Laurens and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, no. 2605, pp. 2579–2605, 2008.