# Listening to the city, attentively: A Spatio-Temporal Attention Boosted
Autoencoder for the Short-Term Flow Prediction Problem

Stefano Fiorini  Michele Ciavotta  Andrea Maurino

###### Abstract

In recent years, studying and predicting alternative mobility (e.g., sharing
services) patterns in urban environments has become increasingly important as
accurate and timely information on current and future vehicle flows can
successfully increase the quality and availability of transportation services.
This need is aggravated during the current pandemic crisis, which pushes
policymakers and private citizens to seek social-distancing compliant urban
mobility services, such as electric bikes and scooter sharing offerings.
However, predicting the number of incoming and outgoing vehicles for different
city areas is challenging due to the nonlinear spatial and temporal
dependencies typical of urban mobility patterns. In this work, we propose
STREED-Net, a novel deep learning network with a multi-attention (spatial and
temporal) mechanism that effectively captures and exploits complex spatial and
temporal patterns in mobility data. The results of a thorough experimental
analysis using real-life data are reported, indicating that the proposed model
improves the state-of-the-art for this task.

###### keywords:

spatial correlation , temporal correlation , attention mechanism , vehicle
flow prediction.

††journal: Information Sciences

[label1]organization=University of Milano-Bicocca, Department of Informatics,
Systems and Communication, addressline=Viale Sarca, 336, city=Milan,
postcode=20126, country=Italy

ANN

    Artifical Neural Network
CMU

    Cascade Multiplicative Unit
CNN

    Convolutional Neural Network
GRU

    Gated Recurrent Unit
HA

    Historical Average
LSTM

    Long Short-Term Memory
MU

    Multiplicative Unit
ReLU

    Rectified Linear Unit
RNN

    Recurrent Neural Network
SVR

    Support Vector Regression
BN

    Batch Normalization
DNN

    Deep Neural Network

##  1 Introduction

In recent years, many researchers have studied and created models to describe
and predict mobility dynamics, or flow prediction in urban areas. This
interest is motivated by the need to comprehend displacement dynamics, which
are also rapidly changing due to alternative electric and shared public
transport systems, to define effective regulatory strategies for human
mobility and freight transport in the smart city [1]. A clear example of the
public decision-maker’s regulatory action concerns the capacity constraints on
public transport (in Europe) imposed to control the spread of the SARS-CoV-2
virus that significantly reshaped mobility habits in large urban areas.

Mobility models are traditionally deployed to plan actions over the long term,
but acting in a timely (even preventive) manner is becoming increasingly
necessary to achieve a high quality of service and availability. This need has
grown in recent years, mainly due to the introduction of new sharing services,
which according to the latest Moovit report [2], are increasingly appreciated
by consumers for the shorter commuting time and the greater flexibility to
reach areas not well served by public transport.

In this regard, this research draws on the following two guiding scenarios. In
the first one, we consider a company, which runs shared mobility services,
that exploits demand forecasting models to improve short-term resource
planning. In the second scenario, a public administration wants to predict the
number of vehicles entering the different areas of the city to identify in
advance possible traffic jam conditions. In both cases, the area of interest
is divided into a grid, each element of which is a region. The number of
vehicles entering (Inflow) and exiting (Outflow) each region must be
predicted. This problem has inherently spatio-temporal characteristics;
evidently, the vehicular flow entering (exiting) a region does not only
present temporal dependencies (time of day, flow in the previous hours) but
also spatial dependencies as it strongly depends on the traffic leaving
(entering) adjacent areas. Formally, such considerations relate to two widely
recognized properties in the study of displacement dynamics [3], namely
temporal and spatial correlation. Mobility data are innately continuous time
series generally not associated with abrupt changes. This means that
displacement dynamics in temporally close periods share similarities, and this
phenomenon is all the more true when the data sampling frequency increases.
Moreover, different neighboring areas featuring similar functional
characteristics (e.g., residential, commercial and industrial areas) often
show correlated traffic patterns. However, the models proposed in the
literature for predicting vehicular flows entering or leaving a particular
area generally tend to consider all areas adjacent to the one considered as
equivalently predictive. In practice, the district structure of the city tends
to be irregular, which calls for a mechanism that can identify these zones and
exploit this information to improve the forecast.

Finally, external factors have also a profound impact on the use of vehicles.
For instance, it is well known in the literature that weather conditions and
the days of the week (workdays vs. weekend) affect displacement dynamics,
especially for lightweight transport means like bikes.

These considerations have steered the design of our proposal, STREED-Net
(Spatio Temporal REsidual Encoder-Decoder Network) novel effective flow
prediction deep learning network.

The main contributions of this paper can be summarized as follows:

  1. 1.

We propose a novel prediction architecture that includes two different
attention blocks to acquire customized temporal and spatial information, i.e.
able to adapt specifically to the city and the means of transport considered,
identifying districts in the city.

  2. 2.

Moreover, to the best of our knowledge, STREED-Net is the first autoencoder
architecture that combines the use of convolutional blocks with residual
connections, a series of Cascade Multiplicative Unit (CMU) and two different
attention mechanisms.

  3. 3.

Finally, this work presents a methodologically sound comparative performance
assessment of various models from the literature on real-life datasets. The
analyses presented consider different types of loss functions and KPIs.
Results indicate that STREED-Net outperforms the considered state-of-the-art
approaches.

The rest of this paper is organized as follows. In section 2, the literature
on techniques used in flow and traffic prediction is analyzed. Section 3
defines the flow prediction problem in urban areas, while in section 4, the
core deep learning techniques exploited in this work and the proposed
framework are described in detail. In section 5 data and results of
experiments are presented and analyzed. Finally, the conclusions and
recommendations for future work are discussed in section 6.

##  2 Related Work

Several studies have addressed the problem of predicting vehicle flows in
urban environments. This problem has been initially modelled as a time series
prediction problem for each city area and approached through classical
statistical methods at first, and Artifical Neural Networks (e.g., deep
learning) later. In particular, different statistical methods have been
applied, including autoregressive integrated moving average (ARIMA) [4],
Kalman filtering [5], and their variants, as well as other classical
approaches such as Bayesian networks [6], Markov chain [7], and Support Vector
Regression (SVR) models [8]. Others approaches have used k-means clustering,
principal component analysis, and self-organizing maps to mine spatio-temporal
performance trends [9]. However, classical statistics models shows some
weaknesses when applied to the flow prediction problem, namely they are unable
to capture the spatial dependencies between the various areas because data for
each region of the city are considered as independent time series, and ii)
they fail to capture the nonlinear relationship between space and time, which
is essential for reliable prediction. Further studies overcame these downsides
by considering spatial relationships [10] and external factors (e.g.,
environment and weather conditions [11]) within traditional time-series
prediction methods.

ANNs have exploited in flow predictions for their capability of capturing the
non-linear spatial and temporal relationships within data. Initial works using
ANNs followed two main approaches. The first one exploits variants of
Recurrent Neural Networks [12] such as i) Long Short-Term Memory (LSTM) [13]
and ii) Gated Recurrent Unit (GRU) [14], whose architectures can effectively
capture both the long-term pattern and short-term fluctuation of time series.
The second research line applies models based on Convolutional Neural Networks
to identify spatial dependencies in traffic networks, treating dynamic traffic
data as a sequence of frames [15]. However, these neural networks in their
standard configuration (derived from the image recognition field) can only
identify either spatial or temporal patterns of the traffic flow data,
respectively.

Spatial and temporal information are inherent in traffic data, making it
essential to consider both aspects at the same time when predicting mobility
dynamics. In this direction, deep learning-based approaches have been recently
proposed, which exploits architectures able to capture spatial and temporal
patterns, including 2D convolutions and residual units [16, 17], 3D
convolutions [18], 3D convolutions and LSTM [19], a combination of 2D and 3D
convolutions [20], autoencoder architecture with Multiplicative Cascade Unit
(CMU) [21]. In recent years, with the development of graph convolutional
networks [22], which can be used to capture the structural characteristics of
the graph network, we are witnessing their use in the field of traffic
prediction. In [23] the authors propose DCRNN, it is a model that captures the
characteristic space through random walks on the graphs, and the temporal
feature through the encoder-decoder architecture, while in [24] they apply the
temporal graph convolutional network (T-GCN) model, which is in combination
with the graph convolutional network (GCN) and gated recurrent unit (GRU).
Finally, in [25] the authors propose a method of forecasting the traffic flow
based on dynamic graphs: the traffic network is modeled by dynamic probability
graphs. The convolution of the graph is performed on the dynamic graphs to
learn the spatial features, which are then combined with the LSTM units to
learn the temporal features. We provide a more detailed description of a
selection of the approaches mentioned above in section 5.

##  3 Problem Statement

Given a tessellation of the area of interest (henceforth referred to as city)
in regularly-shaped regions, a set of historical observations regarding
trajectories of vehicles within the city and, possibly, other spatial and non-
spatial data sources for a reference time horizon $T_{H}$ of $H$ time points,
the citywide vehicle flow prediction problem [18] is defined as the problem of
minimizing the prediction error for vehicle Inflow and Outflow at time
$t^{\prime}$ that is the first time point after $T_{H}$.

In the literature, there are several definitions of location/region with
different granularity and different semantic meaning [26]. However, when it
comes of traffic forecasts, the majority of works uses a rectangular
tessellation, which maximizes the number of neighboring areas. Similarly, in
this study, the geographical space of interest (city) is logically partitioned
into a regular grid of size $N\times M$ oriented by longitude and latitude
[16]. Each element of the grid is termed region and is addressable through a
pair of coordinates $(n,m)$ corresponding to the $n^{th}$ row and the $m^{th}$
column of the grid.

The term Inflow (Outflow, respectively) refers to the number of vehicles
entering (leaving) a specific region in the considered time unit (1(a)) [18].
More specifically, the Inflow (Outflow) indicates the number of pedestrians,
cars, public transport and sharing vehicles entering (leaving) the region in a
certain time period. As shown in 1(b), by analyzing the movement data of the
vehicles, it is possible to obtain the Inflow and Outflow matrix, which
encompass the information about displacements between the areas of the city at
each time $t$. More in detail, let $\tau_{i}=\\{s^{1}_{i},s^{2}_{i},\ldots
s^{t}_{i}\\}$ be a trajectory where $s_{i}^{t}$ represents the position of
vehicle $i$ at time $t$, and let $\mathcal{T}$ be a collection of
trajectories. The Inflow (Outflow, respectively) of a region $(n,m)$ at time
$t$, namely $\iota^{t}_{n,m}$ ($\omega^{t}_{n,m}$) can be formally defined as
in equation (1) (2, respectively).

$\footnotesize\iota^{t}_{n,m}=\sum_{\tau_{i}\in\mathcal{T}}\phi_{\iota}(\tau_{i},t,n,m)$
(1)
$\footnotesize\omega^{t}_{n,m}=\sum_{\tau_{i}\in\mathcal{T}}\phi_{\omega}(\tau_{i},t,n,m)$
(2)

where

$\footnotesize\phi_{\iota}(\tau_{i},t,n,m)=\begin{cases}1,\quad\text{if}\;s_{i}^{t-1}\notin(n,m)\wedge
s^{t}_{i}\in(n,m)\\\ 0,\quad\text{otherwise.}\end{cases}$

and

$\footnotesize\phi_{\omega}(\tau_{i},t,n,m)=\begin{cases}1,\quad\text{if}\;s_{i}^{t-1}\in(n,m)\wedge
s^{t}_{i}\notin(n,m)\\\ 0,\quad\text{otherwise.}\end{cases}$

(a) Inflow and Outflow

(b) Measurement of flows

Finally, the state of the vehicular flow at time $t$ can be represented by a
tensor (also referred to as frame in what follows)
$F_{t}\in\mathbb{R}^{N\times M\times C}$, where $C$ indicates the number of
flow variables considered in the analysis, in this specific case $C=2$
(Inflow/Outflow), whereas $N\times M$ is the total number of regions in the
city. Then, to take into account the temporal dependence, over the time
horizon $T$ (divided into $H$ time points), the flow representation is
extended to a tensor of four dimensions $F\in\mathbb{R}^{H\times N\times
M\times C}$, which represents the main input to our problem. The problem at
issue then becomes predicting $F_{t}$ given a volume, that is a sequence of
past tensors $\mathcal{V}\subset F$. It is worth noting that the resulting
problem shows several similarities with the frame prediction problem [19]
since the tensor $F$ can be seen as a four dimensional volume composed of $H$
consecutive images, each of which featuring $C$ channels.

##  4 STREED-Net

STREED-Net is a Autoencoder-based deep learning model that combines
convolutions and CMUs with two different types of Attentions (spatial and
temporal). This section presents STREED-Net, detailing its components and
relationships, prefacing it with a brief introduction to the main underpinning
concepts, namely the autoencoder architecture and the attention mechanism.

###  4.1 Prerequisites

Autoencoder architecture. Given a set of unlabeled training examples
$\\{x^{1},x^{2},x^{3},...\\}$, where $x^{i}\in\mathbb{R}^{n}$, an autoencoder
neural network is an unsupervised learning algorithm that applies
backpropagation setting the target values to be equal to the inputs
$y(i)=x(i)$. It is a neural network that is trained to learn a function
$h_{W,b}(x)=\hat{x}\approx x$, where $W$ and $b$ are weights and biases of the
ANN, respectively. In other words, an autoencoder is a learned approximation
of the identity function, so as to output $\hat{x}$ that is as much as
possible similar to $x$. The overall network can be decomposed into two parts:
an encoder function $h=f(x)$, which maps the input vector space onto an
internal representation, and a decoder that transforms it back, that is
$\hat{x}=g(h)$. This type of architecture has been applied successfully to
different difficult tasks, including traffic prediction [21].

Attention mechanism. In Deep Neural Networks Attention Mechanism helps focus
on important features of the input, shadowing the others. This paradigm is
inspired by the human neurovisual system, which quickly scans images and
identifies sub-areas of interest, optimizing the usage of the limited
attention resources [27]. Similarly, the attention mechanism in DNNs
determines and stresses on the most informative features in the input data
that are likely to be most valuable to the current activity.

Recently, attention has been widely applied to different areas of deep
learning, such as natural language processing [28], image recognition [29],
image captioning [30], image generation [31] and traffic prediction [32].

Figure 2: STREED-Net Architecture

###  4.2 Encoder

The encoder structure depicted in Figure 3 is the first block of the STREED-
Net architecture. It is composed of an initial convolutional layer, a series
of residual units, and a final convolution layer. Unlike similar approaches
(e.g., STAR [17]), the proposed encoder structure introduces three novel
aspects: i) each layer is time-distributed, meaning that the model learns from
a sequence of frames (for time coherence) instead of focusing on each frame
singularly. ii) it applies further convolutions after the residual unit, so
that to reduce the frame size and iii) it applies a Batch Normalization (BN)
after each convolution to avoid gradient disappear/explode problems and
achieve faster and more efficient reported optimization [33, 34].

Figure 3: Encoder

Unlike other works from the literature, where the distant temporal information
is also used (from the previous day and previous week), the encoder takes as
input a four-dimensional tensor $F\in\mathbb{R}^{H\times N\times M\times 2}$.
This tensor is a sequence of consecutive three-dimensional frames conveying
flow information of nearby periods (with regard to the prediction time
$t^{\prime}$). Such a tensor (also referred to as closeness in the literature
[18]) is obtained by selecting $p$ points preceding the prediction time
$t^{\prime}$, i.e., the sequence
$[F_{t^{\prime}-p},F_{t^{\prime}-(p+1)},....,F_{t^{\prime}-1}]$. In this way,
STREED-Net can focus on the most recent dynamics only. Each frame in $F$ is
processed by the convolution layer to extrapolate spatial information. It is
worth noting that in Figure 2, the encoder is represented by a collection of
identical blocks in parallel execution on the input frames instead of (as in
reality) a single convolution applied sequentially. Such a representation is
used to highlight that a time-distributed layer is trained by taking into
account all input frames simultaneously. The use of this approach leads the
model to identify temporal (that is, inter-frame) dynamics, rather than
looking only to spatial dependencies within each frame.

Each convolutional layer is followed by a Rectified Linear Unit (ReLU)
activation function and a BN layer. Formally, we have:

$E_{t}^{(0)}=\textit{BN}(\textit{ReLU}(W_{e}^{(0)}\ast F_{t}+b_{e}^{(0)}))$
(3)

where $E_{t}^{(0)}$ corresponds to the output of the first convolutional
layer, $F_{t}$ is one of $p$ frames in input to the model and $\ast$ the
convolution operator. $W_{e}^{(0)}$ and $b_{e}^{(0)}$ are the weights and
biases of the respective convolutional operation. Next, $\mathbf{L}$ encoder
blocks (see Figure 3) are placed. Each of these blocks is composed of a
residual unit followed by a downsampling layer:

$E_{t}^{(l)}=\textit{Downsampling}(\textit{ResUnit}(E_{t}^{(l-1)}))$ (4)

where $E_{t}^{(l-1)}$ and $E_{t}^{(l)}$ correspond respectively to the input
and output of the encoder block; $l$ takes values in $\\{1,...,\mathbf{L}\\}$.
The residual units have been implemented as a sequence of two convolutional
layers whose output is eventually summed to the block input. Mathematically,
in STREED-Net the residual unit are defined as follows:

$\displaystyle c_{1}=\textit{BN}(\textit{ReLU}(W^{(l)}_{1}\ast
E_{t}^{(l-1)}+b_{1}^{(l)}))$ (5) $\displaystyle
c_{2}=\textit{BN}(\textit{ReLU}(W_{2}^{(l)}\ast c_{1}+b_{2}^{(l)}))$ (6)
$\displaystyle ERU_{t}^{(l)}=E_{t}^{(l-1)}+c_{2}$ (7)

where $E_{t}^{(l-1)}$ is the residual unit input and $ERU_{t}^{(l)}$ (Encoder
Residual Unit) is used to indicate the result of
$\textit{ResUnit}(E_{t}^{(l-1)})$. $\ast$ is the convolution operator,
$W_{1}^{(l)}$, $W_{2}^{(l)}$ and $b_{1}^{(l)}$, $b_{2}^{(l)}$ are the weights
and biases of the respective convolutional operations.

For what concerns the Downsampling, it has been implemented as:

$E_{t}^{(l)}=\textit{BN}(\textit{ReLU}(W_{ds}^{(l)}\ast
ERU_{t}^{(l)}+b_{ds}^{(l)})$ (8)

where $W_{ds}^{(l)}$, $b_{ds}^{(l)}$ and $\ast$ indicate a convolutional layer
with kernel size and stride parameters set to halve the height and width of
the input frame.

The rationale behind the design of this architecture is threefold: i) a deep
structure is needed for the model to grasp dependencies not only among
neighboring regions but also among distant areas; ii) Deep networks are
difficult to train as they present both the problem of the explosion or
disappearance of the gradient and a greater tendency to overfitting due to the
large number of parameters. To try to avoid these obstacles and to make the
training model more efficient, we introduced residual units. Finally, iii) the
downsampling layers were introduced to ensure translational equivariance [35].

Finally, the encoder structure ends with a closing convolution- Rectified
Linear Unit (ReLU)- Batch Normalization (BN) sequence, which has as its main
objective to reduce the number of feature maps. In this way, the next
architectural component (i.e., the Cascading Hierarchical Block) will receive
and process a smaller input, reducing the computational cost of the CMU array.
The encoder output is:

$E_{t}^{(L+1)}=\textit{BN}(\textit{ReLU}(W_{e}^{(L)}\ast
E_{t}^{(L)}+b_{e}^{(L)}))$ (9)

The output of the encoder is a tensor $E^{(L+1)}\in\mathbb{R}^{H\times
N/2^{\mathbf{L}}\times M/2^{\mathbf{L}}\times C^{\prime}}$, where $C^{\prime}$
is the number of feature maps generated by the last convolution of the
encoder.

###  4.3 Cascading Hierarchical Block

A connection section between the encoder and the decoder is provided to handle
the temporal relationships among the frames. Unlike what is proposed in other
works that combine the use of CNN with the use of RNN such as LSTM [36],
STREED-Net implements a Cascading Hierarchical Block with Cascade
Multiplicative Units (CMU) [21], which computes the hidden representation of
the current state directly using the input frames of both previous and current
time steps, rather than what happens in recurrent networks that model the
temporal dependency by a transition from the previous state to the current
state. This solution is designed to explicitly model the dependency between
different time points by conditioning the current state on the previous state,
improving the model accuracy; incidentally, it also reduces training times.

The fundamental constituent of CMU architecture is the Multiplicative Unit
(MU) [37], which is a non-recurrent convolutional structure whose neuron
connectivity, except for the lack of residual connections, is quite similar to
that of LSTM [38]; the output, however, only depends on the single input frame
$h$. Formally, Multiplicative Unit (MU) is defined by the following equation
set:

$\displaystyle g_{1}=\sigma(W_{1}*h+b_{1})$ (10) $\displaystyle
g_{2}=\sigma(W_{2}*h+b_{2})$ (11) $\displaystyle g_{3}=\sigma(W_{3}*h+b_{3})$
(12) $\displaystyle u=tanh(W_{4}*h+b_{4})$ (13) $\displaystyle
MU(h;W)=g_{1}\odot tanh(g_{2}\odot h+g_{3}\odot u)$ (14)

where $\sigma$ is the sigmoid activation function, $*$ the convolution
operator and $\odot$ the element-wise multiplication operator. $W_{1}\sim
W_{4}$ and $b_{1}\sim b_{4}$ are the weights and biases of the respective
convolutional gates and $W$ denotes all MU parameters.

CMU incorporates three MUs. Unlike MU, CMU accepts two consecutive frames as
input to model explicitly the temporal dependencies between them. The more
recent frame in time is inputted to a MU to capture the spatial information of
the current representation. The older frame is instead processed by two MUs in
sequence to overcome the time gap. The partial outputs are then added together
and finally, thanks to two gated structures containing convolutions along with
non-linear activation functions, the output of the CMU ($X_{t}^{l+1}$) is
generated. CMU is described by the following equations:

$\displaystyle h_{1}=MU(MU(E^{l}_{(t-1)};W_{1});W_{1})$ (15) $\displaystyle
h_{2}=MU(E_{t}^{l};W_{2})$ (16) $\displaystyle h=h_{1}+h_{2}$ (17)
$\displaystyle o=\sigma(W_{o}*h+b_{o})$ (18) $\displaystyle X_{t}^{l+1}=o\odot
tanh(W_{h}*h+b_{h})$ (19)

where $W_{1}$ and $W_{2}$ are the parameters of the MU in the left branch and
of the MU in the right branch respectively, $W_{o}$, $W_{h}$, $b_{o}$ and
$b_{h}$ are the weights and biases of the corresponding convolutional gates.
The cascading hierarchical block uses CMUs to process all frames at the same
time (see Figure 4):

$\displaystyle X_{cmu}=CascadeCMU(E^{(L+1)})$ (20)

where $X_{cmu}\in\mathbb{R}^{N\times M\times C^{\prime}}$.

Figure 4: Cascading Hierarchical Block

###  4.4 External Factors

To integrate external information, such as the day of the week, holidays, and
weather conditions STREED-Net features a specific input branch. The input is a
one-dimensional vector that contains information that refers to prediction
time $t^{\prime}$.

Through the use of two fully connected overlapping layers, this information is
conveyed, encoded, into the mainstream of the network. The first level is used
to embed each sub-factor, while the second reshapes the external factors
embedding space to match the size of the CMU output vector.

###  4.5 Decoder

The decoder is the last component of STREED-Net and its task is to generate
the flow prediction starting from the latent representation that corresponds
to the output of the cascading hierarchical block.

As shown in Figure 5, the decoder takes as input a tensor $z=X_{cmu}+X_{ext}$,
where $z\in\mathbb{R}^{N\times M\times C^{\prime}}$, which is the result of
the sum of the outputs of the hierarchical structure and the network dedicated
to incorporate external factors. $X_{ext}$ is added at this point of the
network to allow the model to use the information extracted from the external
factors during the reconstruction phase.

Figure 5: Decoder

The decoder architecture features a structure that is somehow symmetrical to
that of the encoder with an array of residual units preceded and followed by a
convolutional layer (Equation 21).

$\small D^{(0)}=\textit{BN}(\textit{ReLU}(W_{d}^{(0)}*z+b_{d}^{(0)}))$ (21)

Nevertheless, this symmetry is breached by the presence of two significant
differences. The first one is the presence of a long skip connection before
every residual unit. The long skip connection is used to improve the accuracy
and to recover the fine-grained details from the encoder. Another benefit is a
significant speed up in model convergence [39]. A generic decoding block
$D^{(l)},\;\forall l\in\\{1\dots L\\}$ can be formally defined as the
sequential application of the following three operations:

$\displaystyle sc^{(l)}=\textit{Conv2DTranspose}(D^{(l-1)})+ERU^{(L+1-l)}_{1}$
(22) $\displaystyle U^{(l)}=\textit{BN}(\textit{ReLU}(sc^{(l)})$ (23)
$\displaystyle D^{(l)}=\textit{ResUnit}(U^{(l)})$ (24)

where $D^{(l-1)}$ corresponds to the input block, Conv2DTranspose indicates
the transposed convolution operation (also known as deconvolution), which
doubles the height and width of the input, and $sc^{(l)}$ (skip connection) is
the sum of u with $ERU_{1}^{(L+1-l)}$, i.e., the output of the remaining
encoder unit at level $L+1-l$ for the most recent frame. The residual units of
the decoder are structured exactly like those of the encoder.

The second difference is the presence of two attention blocks (viz. Channel
and Temporal Attention) before the final convolution layer. More details are
provided in the following subsections.

#### Channel Attention

After the convolutional stage of the decoder, a three-dimensional tensor,
referred to as $D^{(L)}\in\mathbb{R}^{N\times M\times C^{\prime}}$, is
obtained with the channel size $C^{\prime}$. Since the dimension of the
channel also includes the temporal aspects compressed by the cascading
hierarchical block, the channel attention has been introduced to identify and
emphasize the most valuable channels.

Figure 6: Channel Attention Block

Figure 6 shows the structure of the channel attention block. Given the input
tensor, $D^{(L)}$ a channel attention map $A_{c}\in\mathbb{R}^{1\times 1\times
C^{\prime}}$ is created by applying attention block deduction operations on
the channels. More in details, through the operation of global average pooling
and global max pooling performed simultaneously, two different feature maps
($X^{max}$ and $X^{avg}$) of size $1\times 1\times C^{\prime}$ each are
spawned. They go through two fully connected layers that allow the model to
learn (and assess) the importance of each channel. The first layer performs a
dimensionality reduction, downsizing the input feature maps to $1\times
1\times\frac{C\prime}{s}$, based on the choice of the reduction ratio $s$; the
second layer restores the feature maps to their original size. This approach
has proven to increase the model efficiency without accuracy reduction [32].
Once these two steps have been completed, the two resulting feature maps are
combined into a single tensor through a weighted summation as:

$A_{c}=\sigma(\Lambda_{1}\otimes X^{max}+\Gamma_{1}\otimes X^{avg})$ (25)

where $\Lambda_{1}$ and $\Gamma_{1}$ are two trainable tensors with the same
size as the two feature maps and $\sigma$ is the activation function.
$\Lambda$ and $\gamma$ are set during the training phase and weight the
relative importance of each element of the two feature maps. Finally, the
process of getting channel attention can be summarized as:

$D^{{}^{\prime}}=A_{c}\otimes D^{(L)}$ (26)

where $D^{{}^{\prime}}$ is the operation output and $\otimes$ denotes the
element-wise multiplication.

#### Spatial attention

Cities are made up of a multitude of different functional areas. Areas have
different vehicle concentrations and mobility patterns; thus, the spatial
attention mechanism has the task of identifying where are located the most
significant areas and scale their contribution to improve the prediction.
Figure 7 presents the main internals involved in the calculation of the
spatial attention Map.

Figure 7: Spatial Attention Block

The spatial attention map $A_{s}\in\mathbb{R}^{N\times M\times 1}$ can be
calculated by applying pooling operations along the axes of the channel to
highlight informative regions [40]. Therefore, first the global average
pooling and global max pooling operations are applied along the channel axes
and, as in the channel attention block, two distinct feature maps of size
$N\times M\times 1$ are obtained. Subsequently, each single feature map passes
through a convolution layer with a filter size of $4\times 4$. unlike what is
done in other approaches from the literature [41] where the filter is set to
$7\times 7$. It is worth noting that the filter size depends on the size of
the areas that make up the city. For the case studies addressed in this work
(see Section 5), which feature rather large regions, the proposed model does
not need to focus on large area clusters.

Finally, the sigmoid activation function is applied.

Also for the spatial attention block, the two feature maps are combined into a
single tensor through a weighted summation:

$A_{s}=\sigma(\Lambda_{2}\otimes X^{max}+\Gamma_{2}\otimes X^{avg})$ (27)

Finally, the process of getting spatial attention can be summarized as:

$D^{{}^{\prime\prime}}=A_{s}\otimes D^{{}^{\prime}}$ (28)

##  5 Experimental Analysis

This section reports on an extensive experimental evaluation of the proposed
model by comparing it against several reference models (see Section 5.1) using
three different performance metrics and three different case studies (detailed
in Section 5.2). An ablation study and the analysis of computational
complexity complete the section.

###  5.1 Reference methods

The proposed model is compared against the following state-of-the-art methods
expressly devised to to solve the citywide vehicle flow prediction problem
[18]:

ST-ResNet [16]: it is one of the first deep learning approaches to traffic
prediction. It predicts the flow of crowds in and out each individual region
of activity. ST-ResNet uses three residual networks that model the temporal
aspects of proximity, period, and trend separately.

MST3D [18]: this model is architecturally similar to ST-ResNet. The three time
dependencies and the external factors are independently modeled and
dynamically merged by assigning different weights to different branches to
obtain the new forecast. Differently from ST-ResNet, MST3D learns to identify
space-time correlations using 3D convolutions.

ST-3DNet [20]: the network uses two distinct branches to model the temporal
components of closeness and trend, while the daily period is left out. Both
branches start with a series of 3D convolutional layers used to capture the
spatio-temporal dependencies among the input frames. In the closeness branch,
the output of the last convolutional layer is linked to a sequence of residual
units to further investigate the spatial dependencies between the frames of
the closeness period. The most innovative architectural element is the
Recalibration Block. It is a block inserted at the end of each of the two main
branches to explicitly model the contribution that each region makes to the
prediction.

3D-CLoST [19]: the model uses sequential 3D convolutions to capture spatio-
temporal dependencies. Afterwards, a fully-connected layer encloses the
information learned in a one-dimensional vector that is finally passed to an
LSTM block. LSTM layers in sequence allow the model to dwell on the temporal
dependencies of the input. The output of the LSTM section is added to the
output produced by the network for external features. The output is multiplied
by a mask, which allows the user to introduce domain knowledge: the mask is a
matrix with null values in correspondence with the regions of the city that
never have Inflow or Outflow values greater than zero (such areas can exist or
not depending on the conformation of the city) while it contains $1$ in all
other locations.

STAR [17]: this approach aims to model temporal dependencies by extracting
representative frames of proximity, period and trend. However, unlike other
solutions, the structure of the model consists of a single branch: the frames
selected for the prediction are concatenated along the axis of the channels to
form the main input to the network. In STAR as well, there is a sub-network
dedicated to external factors and the output it generates is immediately added
to the main network input. Residual learning is used to train the deep network
to derive the detailed outcome for the expected scenarios throughout the city.

PredCNN [21]: this network builds on the core idea of recurring models, where
previous states in the network have more transition operations than future
states. PredCNN employs an autoencoder with CMU, which proved to be a valid
alternative to RNN. Unlike the models discussed above, this approach considers
only the temporal component of closeness but has a relatively complex
architecture. The key idea of PredCNN is to sequentially capture spatial and
temporal dependencies using CMU blocks.

Historical Average (HA): the algorithm generates Inflow and Outflow forecasts
by performing the arithmetic average of the corresponding values of the same
day of the week at the same time as the instant in time to be predicted. This
classical method represents a baseline in our comparative analysis as it has
not been developed specifically for the flow prediction problem.

Excluding MST3D, which has been entirely reimplemented following the
indications of the original paper strictly, and PredCNN, whose original code
has been completed of some missing parts, for all the other models the
implementation released by the original authors has been used. The STREED-Net
code, together with all the code realized for this research work, is freely
available111https://github.com/UNIMIBInside/Smart-Mobility-Prediction.

We conclude this section by pointing out that, although the literature offers
numerous proposals for deep learning models based on graphs with performances
often superior to those of convolutional models, for the problem addressed in
this paper, preliminary experiments that we conducted with graph-based models
did not lead to satisfactory results. The main reason for this apparent
contradiction resides in the nature of the problem considered, whose basic
assumption is to be able to observe only the inflow and outflow across all
areas of the city. Such a scenario is feasible and more realistic than one in
which the trajectory or origin-destination pair of all vehicles is known but
makes it impossible to create graphs with nontrivial connections (i.e., not
between adjacent areas) for the problem under consideration. Graph-based
models under such conditions have not proven to be sufficiently accurate for
the case studies considered.

###  5.2 Case studies

Three real-life case studies are considered for the experimental analysis,
which differ in both the city considered (New York and Beijing) and the type
of vehicle considered (bicycle and taxi). This choice allow the models to be
assess on usage patterns that are expected to be significantly distinct.
Follows a brief description of the considered case studies:

BikeNYC. In this first case study the behavior of bicycles in New York city is
analyzed. The data has been collected by the NYC Bike system in 2014, from
April 1 to September 30. Records from the last 10 days form the testing data
set, while the rest is used for training. The length of each time period is of
1 hour.

TaxiBJ. In the second case study, a fleet of cabs and the city of Beijing are
considered. Data have been collected in 4 different time periods: July 1 2013
- October 30 2013, March 1 2014 - June 30 2014, March 1 2015 - Jun 30 2015,
November 1 2015 - April 15 2016. The last four weeks are test data and the
others are used for training purposes. The length of each time period is set
to 30 minutes.

TaxiNYC. Finally, a data set containing data from a fleet of taxicabs in New
York is considered. Data have been collected from January 1, 2009 to December
31, 2014. The last four weeks are test data and the others are used for
training purposes. The length of each time period is set to one hour. This
case study has been specifically created to perform a more thorough and sound
experimental assessments than those presented in the literature.

The city of New York has been tessellated into $16\times 8$ regions, while the
city of Beijing has been divided into $32\times 32$ areas; the discrepancy in
the number of regions considered is due to the large difference in extension
between the two cities. The Beijing area (16,800 km2) is 22 times bigger than
the New York area (781 km2).

The Beijing taxi data set (TaxiBJ) and New York Bike data set (BikeNYC) are
available via [16]; they are already structured to carry out the experiments
reported in this work. As for the TaxiNYC dataset, available for experiments
on GitHub222https://github.com/UNIMIBInside/Smart-Mobility-
Prediction/tree/master/data/TaxiNYC, it has been expressly built for this work
by processing and structuring data available from the NYC government
website333https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page.

A Min-Max normalization has been applied to all data sets to convert traffic
values based on the scale [-1, 1]. Note, however, that in the experiments a
denormalization is applied to the expected values to be used in the
evaluation.

In the three experiments, public holidays, metadata (i.e. DayOfWeek,
Weekday/Weekend) and weather have been considered as external factors.
Specifically, the meteorological information reports the temperature, the wind
speed, and the specific atmospheric situation (viz., sun, rain and snow).

###  5.3 Analysis of results

This section presents and discusses the results of experiments performed by
running STREED-Net and the models presented in Section 5.1 on the three case
studies. Moreover, three different evaluation metrics are used in this study
to compare the results obtained: Root Mean Square Error (RMSE), Mean Absolute
Percentage Error (MAPE) and Absolute Percentage Error (APE), which are defined
as follows:

$\displaystyle
RMSE=\sqrt{\frac{\displaystyle\sum_{n=1}^{N}\sum_{m=1}^{M}[\left(\hat{\iota}_{n,m}-\iota_{n,m}\right)^{2}+\left(\hat{\omega}_{n,m}-\omega_{n,m}\right)^{2}]}{N\times
M}}$ (29) $\displaystyle
MAPE=100\cdot\frac{\displaystyle\sum_{n=1}^{N}\sum_{m=1}^{M}\left|\frac{\left(\hat{\iota}_{n,m}-\iota_{n,m}\right)+\left(\hat{\omega}_{n,m}-\omega_{n,m}\right)}{\left(\hat{\iota}_{n,m}-\iota_{n,m}\right)}\right|}{N\times
M}$ (30) $\displaystyle
APE=100\cdot\displaystyle\sum_{n=1}^{N}\sum_{m=1}^{M}\left|\frac{\left(\hat{\iota}_{n,m}-\iota_{n,m}\right)+\left(\hat{\omega}_{n,m}-\omega_{n,m}\right)}{\left(\hat{\iota}_{n,m}-\iota_{n,m}\right)}\right|$
(31)

where $\hat{\iota}_{n,m}$ and $\hat{\omega}_{n,m}$ are, respectively, the
predicted Inflow and Outflow for region $(n,m)$ at time $t^{\prime}$ and
$N\times M$ is the total number of regions in the city.

Finally, it is worth noting that to account for and reduce the inherent
stochasticity of learning-based models, each experiment was repeated ten times
(replicas) using a different random seed in each replica. Mean and standard
deviation are reported for each metric to provide a robust indication of the
overall behavior of the compared methods.

####  5.3.1 BikeNYC

For the BikeNYC case study STREED-Net parameters have been set as follow. The
number $n$ of input frames has been set to 4, the number $L$ of encoding and
decoding blocks has been set to 2. This decision has been dictated by the size
of the grid ($16\times 8$): setting $L$ greater than 2 (for example 3) would
result in an encoder output tensor of size $4\times 2\times 1\times C$, which
would be too small to allow the CMU block to effectively capture the time
dependencies in the section located between the encoder and decoder. After
some preliminary tests, the number of convolutional filters has been set to 64
in the first layer of the encoder and in the subsequent blocks, while in the
last layer it has been set equal to 16. In this way, the dimensionality of the
input vector goes from $I\in\mathbb{R}^{4\times 16\times 8\times 2}$ to
$O\in\mathbb{R}^{4\times 4\times 2\times 16}$ as the encoder output.
Symmetrically, the convolutions within the decoder use 64 filters, except for
the final layer which uses only 2 filters to generate the prediction of the
Inflow and Outflow channels. The parameters corresponding to the
dimensionality of the convolution kernel (kernel size equal to 3, batch size
equal to 16 and learning rate equal to 0.0001. The number of epochs is set to
150), to the batch size and to the learning rate have been optimized with the
Bayesian optimization technique.

As for the models from the literature, they have been arranged and trained
following carefully the parameter values and indications reported in the
respective publications.

Table 1: Results obtained for the Bike NYC data set Model | RMSE | MAPE | APE
---|---|---|---
HA | 6.56 | 26.46 | 4.09E+05
ST-ResNet | 5.01±0.07 | 21.97±0.26 | 3.40E+05±4.06E+03
MST3D | 4.98±0.05 | 22.03±0.47 | 3.41E+05±7.26E+03
3D-CLoST | 4.90±0.04 | 21.38±0.20 | 3.31E+05±3.12E+03
PredCNN | 4.81±0.04 | 21.38±0.24 | 3.31E+05±3.76E+03
ST-3DNet | 4.75±0.06 | 21.42±0.28 | 3.31E+05±4.36E+03
STAR | 4.73±0.05 | 20.97±0.13 | 3.24E+05±2.02E+03
STREED-Net | 4.67±0.03 | 20.85±0.15 | 3.23E+05±2.31E+03

As shown in Table 1, STREED-Net outperforms all other considered approaches
for all evaluation metrics. In addition, the small standard deviation values
are evidence of the robustness of the proposed approach. Nonetheless, it is
worth observing that all learning-based approaches return similar results. We
believe this is mainly due to the reduced size of the data set that does not
allow the models to be adequately trained. Moreover, the tessellation used in
this case study (widely used in the literature), with a small grid of
dimensions ($16\times 8$), tends to level off the metrics and hinder a more
precise performance assessment.

####  5.3.2 TaxiBJ

As with the experiment discussed above, for the TaxiBJ case study the
parameters of the models have been set according to the specifications given
in the respective publications. In the case of STREED-Net, the hyperparameters
are kept unchanged in the two experiments, except for the number $L$ of
encoding and decoding blocks, which has been increased to 3 because the grid
is larger ( $32\times 32$) in this experiment and more convolutional layers
are needed to map the input tensor of the model. Also for this experiment, the
kernel size, batch size, and learning rate parameters have been optimized with
Bayesian optimization and the best values found were 3, 16, and 0.0001
respectively. The number of epochs has been set at 150. Notice that these
values are the same used in the BikeNYC experiment.

Table 2: Results obtained for the Taxi Beijing data set Model | RMSE | MAPE | APE
---|---|---|---
HA | 40.93 | 30.96 | 6.77E+07
ST-ResNet | 17.56±0.91 | 15.74±0.94 | 3.45E+07±2.05E+06
MST3D | 21.34±0.55 | 22.02±1.40 | 4.81E+07±3.03E+05
3D-CLoST | 17.10±0.23 | 16.22±0.20 | 3.55E+07±4.39E+05
PredCNN | 17.42±0.12 | 15.69±0.17 | 3.43E+07±3.76E+05
ST-3DNet | 17.29±0.42 | 15.64±0.52 | 3.43E+07±1.13E+06
STAR | 16.25±0.40 | 15.40±0.62 | 3.38E+07±1.36E+06
STREED-Net | 15.61±0.11 | 14.73±0.21 | 3.22E+07±4.51E+05

As can be seen from Table 2, STREED-Net outperforms all other methods,
reducing by 6%, 4.4%, and 1.5% RMSE, MAPE, and APE respectively, compared to
the second-best approach. The difference in performance, in favor of the
proposed model, in this experiment is more appreciable because the data set
used for the training process is more significant but also because the number
of regions is higher. This last consideration highlights how the proposed
model seems suitable to be applied in real-world scenarios, i.e., where high
model accuracy and dense tessellation are required (i.e., the city is
partitioned into a large number of small regions).

####  5.3.3 TaxiNYC

As mentioned earlier, the TaxiNYC case study was created specifically to be
able to evaluate the behavior of the proposed model in a wider set of
scenarios than the literature. Consequently, in order to make a fair
comparison, it was necessary to search for the best configuration of
hyperparameters not only for the STREED-Net model but also for all the other
approaches considered. The optimized parameters and the relative values used
in the training phase are briefly summarized below for each model. Unreported
configuration values have been set as for the BikeNYC case study since the two
experiments share the same map size ($16\times 8$). The parameters for each
model are as follows:

  * •

ST-ResNet*. Optimized parameters: number of residual units, batch size and
learning rate. Optimal values found: 2, 16 and 0.0001.

  * •

MST3D. Optimized parameters: batch size and learning rate. Optimal values
found: 16 and 0.00034.

  * •

PredCNN. Optimized parameters: encoder length, decoder length, number of
hidden units, batch size and learning rate. Optimal values found: 2, 3, 64, 16
and 0.0001.

  * •

ST-3DNet. Optimized parameters: number of residual units, batch size and
learning rate. Best values found: 5, 16 and 0.00095.

  * •

STAR*. Optimized parameters: number of remaining units, batch size and
learning rate. Optimal values found: 2, 16 and 0.0001.

  * •

3D-CLoST. Optimized parameters: number of LSTM layers, number of hidden units
in each LSTM layer, batch size, and learning rate. Optimal values found: 2,
500, 16, and 0.00076.

  * •

STREED-Net. Optimized parameters: kernel size, batch size, and learning rate.
Optimal values found: 3, 64 and 0.00086.

It is worth noting that, preliminary experiments showed a convergence issue
for the training phase of both STAR and ST-ResNet models. In particular, they
were unable to converge for any combination of parameters. This behavior is
due to the strong presence of outliers and to the concentration of the
relevant Inflow and Outflow values in a few central regions of the city. To
overcome this issue, Batch Normalization layers have been inserted in the
structure of the two models. In particular, Batch Normalization layers have
been added after each convolution present in the residual units (a possibility
that has already been foreseen in the original implementations) and after the
terminal convolution of the networks (an option not considered in the source
code provided by the original authors). For this reason, ST-ResNet and STAR
are marked with an asterisk in the Table 3, which summarizes the experimental
results.

Table 3: Results obtained for the Taxi New York data set Model | RMSE | MAPE | APE
---|---|---|---
HA | 164.31 | 27.19 | 7.94E+05
ST-ResNet* | 35.87±0.60 | 22.52±3.43 | 6.57E+05±1.00E+05
MST3D | 48.91±1.98 | 23.98±1.30 | 6,98E+05±1.34E+04
3D-CLoST | 48.17±3.16 | 22.18±1.05 | 6.48E+05±3.08E+04
PredCNN | 40.91±0.51 | 25.65±2.16 | 7.49E+05±6.32E+04
ST-3DNet | 41.62±3.44 | 25.75±6.11 | 7.52E+05±1.78E+05
STAR* | 36.44±0.88 | 25.36±5.24 | 7.41E+05±1.53E+05
STREED-Net | 36.22±0.72 | 20.29±1.48 | 5.93E+05±4.31E+04

As it can be seen from Table 3, STREED-Net achieved excellent results in this
experiment as well, ranked as the best model for two out of three evaluation
metrics. In particular, as far as the RMSE is concerned, the performances
obtained are very close to the best one (achieved by ST-ResNet*, which is
considerably different from the original ST-ResNet), while the MAPE and APE
values position it as the best model.

###  5.4 Ablation study

In this section, an ablation study conducted on STREED-Net is presented in
which variations in the input structure and in the network architecture are
analyzed. The study, for reasons of space refers only to BikeNYC case study
and does not involves the full combinatorics of all possible variants of the
proposed model but aims to assess the impact on performance metrics of some
parameters (namely, the number of input time points $n$) and specific
architectural choices (viz., long skip connection, attention blocks, and
external factors input branch), while maintaining all other conditions. More
precisely, in what follows STREED-Net is compared against the 5 different
variations described below:

  * •

STREED-Net_N3. Same architecture as STREED-Net, but input volumes with 3
frames ($[X_{t-3},X_{t-2},X_{t-1}]$).

  * •

STREED-Net_N5. Same architecture as STREED-Net, but input volumes with 5
frames ($[X_{t-5},X_{t-4},X_{t-3},X_{t-2},X_{t-1}]$).

  * •

STREED-Net_NoLSC. STREED-Net by removing the long skip connection between
encoder and decoder.

  * •

STREED-Net_NoAtt. STREED-Net without the attention blocks.

  * •

STREED-Net_NoExt. STREED-Net without the external factors.

Notice that the study does not consider the variations with $n=1$ and $n=2$ as
such values would not allow the network to capture meaningful temporal
patterns between traffic flows.

Table 4: Results obtained from ablation studies Model | RMSE | MAPE | APE
---|---|---|---
STREED-Net_N3 | 4.75±0.04 | 21.18±0.18 | 3.28E+05±2.73E+03
STREED-Net_N5 | 4.74±0.03 | 21.03±0.22 | 3.26E+05±3.43E+03
STREED-Net_NoLSC | 4.84±0.04 | 21.53±0.24 | 3.33E+05±3.71E+03
STREED-Net_NoAtt | 4.78±0.04 | 20.95±0.27 | 3.25E+05±4.20E+03
STREED-Net_NoExt | 4.76±0.04 | 20.99±0.29 | 3.26E+05±4.55E+03
STREED-Net | 4.67±0.03 | 20.85±0.15 | 3.23E+05±2.31E+03

Table 4 reports the results of the ablation study conducted. Each data point
in the table has been obtained performing 10 times the training procedure for
each model variation changing the random seed, and evaluating the resulting
network on the test set. The mean and standard deviation are reported.

The results show that regarding the time horizon, for the BikeNYC case study,
$n=4$ allows the model to obtain better results. This means that, considering
the particular setup, for the city of New York 4 hours of data allow to
predict more accurately the dynamics of bicycle mobility whereas considering a
greater amount of information ($n=5$) would reduce the accuracy of the
network. It is plausible to believe that considering a larger number of
temporal instants would lead the network to grow in the number of parameters
to be trained and thus require a larger amount of data to identify possible
longer-term patterns.

From the architectural point of view, the two components attention block and
long skip connection, confirm their importance in improving the performance of
the proposed model, accounting for a 2.36% and 3.64% increase in RMSE,
respectively. In particular, as regards the attention block, not only STREED-
Net reaches lower average error values, but also the standard deviation is
reduced, proving that the attention blocks are effective in helping the
network single out the most meaningful information and in making the training
process more stable. Finally, the experiment shows also a strong impact of the
long skip connection mechanism, which, as illustrated in subsection 4.5,
connects the encoder to the decoder to convey fine-grained details through the
network.

###  5.5 Number of trainable parameters and FLOPs

A a brief analysis of the number of trainable parameters and the computational
complexity (measure in number of FLOPs) of each model for the different case
studies is reported in this section. For what concerns the number of trainable
parameters, as shown in Table 5, STREED-Net has a generally low number
compared to other models as only STAR features lesser parameters to train.
Such a reduced number of parameters is due to the fact that the dimensionality
of the input is reduced by the encoder downsampling mechanism. The model with
the highest number of parameters is 3D-CLoST, which uses both 3D convolutions
and LSTM.

Finally, Table 6 provide the computational complexity of each model in terms
floating point operations (FLOPs) as in [42] for each case study. As can be
seen from the results obtained, the model with the higher computational
complexity is PredCNN, which is based on CMUs. While, 3DCLoST is the model
with the shortest forward and backward times. STREED-Net, instead, has a
middle-range computational complexity compared to the other models, despite
its autoencoder structure, the use of attention blocks, and CMUs. This occurs
because although the number of network parameters is small, the network
employs high complexity operators. However, the training and execution times
of STREED-Net are compatible with its applicability in full-scale real-world
scenarios.

Table 5: Number of trainable parameters Model | BikeNYC | TaxiNYC | TaxiBJ
---|---|---|---
ST-ResNet | 906,272 | 458,304 | 2,696,992
MST3D | 668,218 | 668,378 | 8,674,370
3D-CLoST | 13,099,090 | 19,477,648 | 72,046,714
PredCNN | 3,967,906 | 3,967,906 | 4,827,842
ST-3DNET | 540,696 | 617,586 | 903,242
STAR | 161,052 | 310,076 | 476,388
STREED-Net | 582,673 | 582,673 | 765,497
Table 6: Computational complexity (in number of FLOPs) Model | BikeNYC | TaxiNYC | TaxiBJ
---|---|---|---
ST-ResNet | 230,849,450 | 115,735,786 | 5,459,663,018
MST3D | 33,042,250 | 33,042,570 | 272,483,226
3D-CLoST | 29,613,094 | 9,601,920 | 338,148,804
PredCNN | 1,015,468,288 | 1,015,468,288 | 9,883,813,888
ST-3DNET | 171,242,496 | 190,130,922 | 1,823,295,898
STAR | 40,449,706 | 78,231,530 | 928,100,922
STREED-Net | 130,047,738 | 130,047,738 | 1,067,063,882

##  6 Conclusions

Predicting vehicular flow is one of the central topics in the domain of
intelligent mobility. It is a challenging task, influenced by several complex
factors, such as spatio-temporal dependencies and external factors. In this
study, we have developed a new deep learning architecture, based on
convolutions and CMU to forecast the Inflow and Outflow in each region of the
smart city. A comprehensive experimental campaign has been conducted on three
different real-world case studies. The results showed that STREED-Net
consistently outperforms state-of-the-art models in predicting dynamics in all
the experiments conducted on the three performance metrics considered. This
work also reports and analyzes the results from an ablation study and
complexity analysis.

For future developments, the possible integration of other external factors,
such as the territorial characteristics of each geographical area, should be
tested. Moreover, it would be appropriate to increase the granularity of the
city tessellation, as well as conduct transfer learning experiments to study
the applicability of the proposed model to scenarios with a reduced amount of
data available.

## References

  * [1] Y. Zheng, L. Capra, O. Wolfson, H. Yang, Urban computing: concepts, methodologies, and applications, ACM Transactions on Intelligent Systems and Technology (TIST) 5 (3) (2014) 1–55.
  * [2] Moovit, https://moovitapp.com/insights/en/Moovit_Insights_Public_Transit_Index-countries, accessed: 2021-02-22.
  * [3] Y. Liu, C. Lyu, A. Khadka, W. Zhang, Z. Liu, Spatio-Temporal Ensemble Method for Car-Hailing Demand Prediction, IEEE Transactions on Intelligent Transportation Systems 21 (12) (2019) 1–6.
  * [4] H. Z. Moayedi, M. Masnadi-Shirazi, Arima model for network traffic prediction and anomaly detection, in: 2008 International Symposium on Information Technology, Vol. 4, IEEE, 2008, pp. 1–6.
  * [5] J. Guo, W. Huang, B. M. Williams, Adaptive kalman filter approach for stochastic short-term traffic flow rate prediction and uncertainty quantification, Transportation Research Part C: Emerging Technologies 43 (2014) 50–64.
  * [6] S. Sun, C. Zhang, G. Yu, A bayesian network approach to traffic flow forecasting, IEEE Transactions on intelligent transportation systems 7 (1) (2006) 124–132.
  * [7] Y. Qi, S. Ishak, A hidden markov model for short term prediction of traffic conditions on freeways, Transportation Research Part C: Emerging Technologies 43 (2014) 95–111.
  * [8] C.-H. Wu, J.-M. Ho, D.-T. Lee, Travel-time prediction with support vector regression, IEEE transactions on intelligent transportation systems 5 (4) (2004) 276–281.
  * [9] M. T. Asif, J. Dauwels, C. Y. Goh, A. Oran, E. Fathi, M. Xu, M. M. Dhanya, N. Mitrovic, P. Jaillet, Spatiotemporal patterns in large-scale traffic speed prediction, IEEE Transactions on Intelligent Transportation Systems 15 (2) (2013) 794–804.
  * [10] Y. Tong, Y. Chen, Z. Zhou, L. Chen, J. Wang, Q. Yang, J. Ye, W. Lv, The simpler the better: A unified approach to predicting original taxi demands based on large-scale online platforms, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 1653–1662.
  * [11] X. Qian, S. V. Ukkusuri, Spatial variation of the urban taxi ridership using gps data, Applied Geography 59 (2015) 31 – 42.
  * [12] A. Azzouni, G. Pujolle, A long short-term memory recurrent neural network framework for network traffic matrix prediction, arXiv preprint arXiv:1705.05690 (2017).
  * [13] R. Yu, Y. Li, C. Shahabi, U. Demiryurek, Y. Liu, Deep learning: A generic approach for extreme condition traffic forecasting, in: Proceedings of the 2017 SIAM international Conference on Data Mining, SIAM, 2017, pp. 777–785.
  * [14] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1724–1734.
  * [15] X. Ma, Z. Dai, Z. He, J. Ma, Y. Wang, Y. Wang, Learning traffic as images: a deep convolutional neural network for large-scale transportation network speed prediction, Sensors 17 (4) (2017) 818.
  * [16] J. Zhang, Y. Zheng, D. Qi, R. Li, X. Yi, T. Li, Predicting citywide crowd flows using deep spatio-temporal residual networks, Artificial Intelligence 259 (2018) 147–166.
  * [17] H. Wang, H. Su, Star: A concise deep learning framework for citywide human mobility prediction, in: 2019 20th IEEE International Conference on Mobile Data Management (MDM), IEEE, 2019, pp. 304–309.
  * [18] C. Chen, K. Li, S. G. Teo, G. Chen, X. Zou, X. Yang, R. C. Vijay, J. Feng, Z. Zeng, Exploiting spatio-temporal correlations with multiple 3d convolutional neural networks for citywide vehicle flow prediction, in: 2018 IEEE international conference on data mining (ICDM), IEEE, 2018, pp. 893–898.
  * [19] S. Fiorini, G. Pilotti, M. Ciavotta, A. Maurino, 3d-clost: A cnn-lstm approach for mobility dynamics prediction in smart cities, in: 2020 IEEE International Conference on Big Data (Big Data), 2020, pp. 3180–3189.
  * [20] S. Guo, Y. Lin, S. Li, Z. Chen, H. Wan, Deep spatial–temporal 3d convolutional neural networks for traffic data forecasting, IEEE Transactions on Intelligent Transportation Systems 20 (10) (2019) 3913–3926.
  * [21] Z. Xu, Y. Wang, M. Long, J. Wang, M. KLiss, Predcnn: Predictive learning with cascade convolutions., in: IJCAI, 2018, pp. 2940–2947.
  * [22] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, arXiv preprint arXiv:1609.02907 (2016).
  * [23] Y. Li, R. Yu, C. Shahabi, Y. Liu, Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, arXiv preprint arXiv:1707.01926 (2017).
  * [24] L. Zhao, Y. Song, C. Zhang, Y. Liu, P. Wang, T. Lin, M. Deng, H. Li, T-gcn: A temporal graph convolutional network for traffic prediction, IEEE Transactions on Intelligent Transportation Systems 21 (9) (2019) 3848–3858.
  * [25] H. Peng, B. Du, M. Liu, M. Liu, S. Ji, S. Wang, X. Zhang, L. He, Dynamic graph convolutional network for long-term traffic flow prediction with reinforcement learning, Information Sciences 578 (2021) 401–416.  doi:https://doi.org/10.1016/j.ins.2021.07.007.
URL https://www.sciencedirect.com/science/article/pii/S0020025521006976

  * [26] K. Kemp, et al., Encyclopedia of geographic information science, Sage, 2008.
  * [27] S. K. Ungerleider, L. G, Mechanisms of visual attention in the human cortex, Annual review of neuroscience 23 (1) (2000) 315–341.
  * [28] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014).
  * [29] H. Zheng, J. Fu, T. Mei, J. Luo, Learning multi-attention convolutional neural network for fine-grained image recognition, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5209–5217.
  * [30] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International conference on machine learning, 2015, pp. 2048–2057.
  * [31] K. Gregor, I. Danihelka, A. Graves, D. Rezende, D. Wierstra, Draw: A recurrent neural network for image generation, in: International Conference on Machine Learning, PMLR, 2015, pp. 1462–1471.
  * [32] Y. Liu, Z. Liu, C. Lyu, J. Ye, Attention-based deep ensemble net for large-scale online taxi-hailing demand prediction, IEEE Transactions on Intelligent Transportation Systems 21 (11) (2020) 4798–4807.  doi:10.1109/TITS.2019.2947145.
  * [33] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, CoRR abs/1502.03167 (2015).  arXiv:1502.03167.
  * [34] S. Santurkar, D. Tsipras, A. Ilyas, A. Madry, How does batch normalization help optimization?, in: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 31, Curran Associates, Inc., 2018, pp. 2483–2493.
URL
https://proceedings.neurips.cc/paper/2018/file/905056c1ac1dad141560467e0a99e1cf-
Paper.pdf

  * [35] I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep learning, Vol. 1, MIT press Cambridge, 2016.
  * [36] N. Ranjan, S. Bhandari, H. Zhao, H. Kim, P. Khan, City-wide traffic congestion prediction based on cnn, lstm and transpose cnn, IEEE Access PP (2020) 1–1.  doi:10.1109/ACCESS.2020.2991462.
  * [37] N. Kalchbrenner, A. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, K. Kavukcuoglu, Video pixel networks, in: International Conference on Machine Learning, PMLR, 2017, pp. 1771–1779.
  * [38] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [39] C. Szegedy, S. Ioffe, V. Vanhoucke, A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, AAAI Press, 2017, p. 4278–4284.
  * [40] N. Komodakis, S. Zagoruyko, Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer, in: ICLR, 2017.
  * [41] S. Woo, J. Park, J.-Y. Lee, I. So Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  * [42] S. Bianco, R. Cadene, L. Celona, P. Napoletano, Benchmark analysis of representative deep neural network architectures, IEEE Access 6 (2018) 64270–64277.

  *[CMU]: Cascade Multiplicative Unit
  *[SVR]: Support Vector Regression
  *[ANNs]: Artifical Neural Network
  *[LSTM]: Long Short-Term Memory
  *[GRU]: Gated Recurrent Unit
  *[CMUs]: Cascade Multiplicative Unit
  *[ANN]: Artifical Neural Network
  *[DNNs]: Deep Neural Network
  *[BN]: Batch Normalization
  *[ReLU]: Rectified Linear Unit
  *[CNN]: Convolutional Neural Network
  *[RNN]: Recurrent Neural Network
  *[MU]: Multiplicative Unit
  *[MUs]: Multiplicative Unit
  *[HA]: Historical Average