# Convolutional and Deep Learning based techniques for Time Series Ordinal
Classification

Rafael Ayllón-Gavilán, David Guijo-Rubio∗, Pedro Antonio Gutiérrez, Anthony
Bagnall, César Hervás-Martínez ∗: Corresponding author.
R. Ayllón-Gavilán, P.A. Gutiérrez and C. Hervás-Martínez are with the Dept. of
Computer Science and Numerical Analysis, University of Cordoba, Spain. E-mail:
<EMAIL_ADDRESS>
D. Guijo-Rubio is with both the Dept. of Computer Science and Numerical
Analysis, University of Cordoba, Spain and the School of Computing Sciences,
University of East Anglia, United Kingdom. E-mail<EMAIL_ADDRESS>
A. Bagnall is with the School of Computing Sciences, University of East
Anglia, United Kingdom. E-mail<EMAIL_ADDRESS>received April 19,
2005; revised August 26, 2015.

###### Abstract

Time Series Classification (TSC) covers the supervised learning problem where
input data is provided in the form of series of values observed through
repeated measurements over time, and whose objective is to predict the
category to which they belong. When the class values are ordinal, classifiers
that take this into account can perform better than nominal classifiers. Time
Series Ordinal Classification (TSOC) is the field covering this gap, yet
unexplored in the literature. There are a wide range of time series problems
showing an ordered label structure, and TSC techniques that ignore the order
relationship discard useful information. Hence, this paper presents a first
benchmarking of TSOC methodologies, exploiting the ordering of the target
labels to boost the performance of current TSC state-of-the-art. Both
convolutional- and deep learning-based methodologies (among the best
performing alternatives for nominal TSC) are adapted for TSOC. For the
experiments, a selection of $18$ ordinal problems from two well-known archives
has been made. In this way, this paper contributes to the establishment of the
state-of-the-art in TSOC. The results obtained by ordinal versions are found
to be significantly better than current nominal TSC techniques in terms of
ordinal performance metrics, outlining the importance of considering the
ordering of the labels when dealing with this kind of problems.

###### Index Terms:

time series machine learning, time series analysis, time series
classification, ordinal classification

##  1 Introduction

A time series is an ordered sequence of values. This type of data can be found
in a wide variety of domains, such as medicine [1], financial analysis [2], or
agriculture [3], among others. For instance, electrocardiogram signals[4] are
ordered by time and spectrograms [5] are ordered by frequency. Time Series
Classification (TSC) is one of the most popular machine learning tasks, with
hundreds of different approaches presented in the literature [6, 7, 8, 9].
Moreover, other time series related domains are time series clustering [10]
and forecasting [11], with an increasing interest in the last years.

TSC involves predicting a discrete output variable for a given time series.
Depending on the number of variables observed at each time point, time series
are univariate (only one channel or dimension) or multivariate (two or more
channels) [12, 13, 14]. The publication of the University of California
Riverside (UCR) TSC archive [15]111https://timeseriesclassification.com/
allowed the development of effective methods for TSC. This archive provides an
heterogeneous problem set leading to a fair comparison of novel proposals.
Recently, a large set of $33$ TSC methods has been developed and tested on
this archive [16]. This study showed that the latest algorithms represent a
significant advance in the state of the art for TSC.

While there has been significant progress in the algorithmic development for
TSC almost no attention has been paid to Time Series Ordinal Classification
(TSOC). We aim to address this ansence. Some problems in the UCR archive are
ordinal in nature, i. e. labels associated with samples follow an ordinal
relationship among them. Up to now, these problems have been tackled by
nominal TSC methods, which hinder the learning process: nominal methods
generally require more data or iterations to achieve the same performance than
an ordinal classifier [17].

We can define ordinal classification (also known as ordinal regression) as a
classification problem where the output labels exhibit a natural ordering.
This characteristic can be found in many and varied domains: finances [18],
human age prediction [19], climatological applications [20] or medical
research [21]. All these previous works take advantage of information related
to the ordering of the categorical labels to increase the performance of the
models being applied. The most prominent type of models in the literature on
ordinal classification are the so-called threshold-based models. In these
methodologies the existence of a real-valued variable underlying the ordinal
response is assumed. Hence, the training process focuses on modelling the real
variable and learning the optimal thresholds determining which intervals
corresponds to each ordinal label. The Cumulative Link Model (CLM) approach
belongs to this family of threshold-based models, and works with cumulative
probabilities of the input belonging to a certain class or inferior classes in
the ordinal scale.

An example of ordinal classification can be seen in [21], where a set of
patients infected with the SaRS-COV 2 coronavirus is studied and grouped into
three levels of illness severity: moderate, severe, and critical. These labels
follow an ordinal relationship, in that a critical patient is sicker than a
moderate or a severe patient. The main characteristic of an ordinal variable,
following this example, is that it is worse to misclassify a critical patient
as moderate, than it is to misclassify them as severe. The magnitude of the
error should be higher in the first case. As stated in [22], two types of
ordinal variables are distinguished: 1) grouped continuous variables, and 2)
assessed ordered categorical variables. The former is based on the existence
of an underlying continuous variable, in such a way that the labels are the
result of the discretisation of this variable. In the latter case, it is a
domain expert who is responsible for assigning labels to the patterns,
establishing the ordering at his/her own judgment. As can be seen, first group
of problems are more objective than those belonging to the second group, as no
influence from human opinion is included.

In this paper, our focus is on TSOC problems, which involve classifying time
series data based on ordinal categories. An example dataset used for
experimental validation in this study is the DistalPhalanxOutlineAgeGroup
dataset. It is specifically designed to assess the effectiveness of detecting
hand and bone outlines and determine if they can aid in predicting bone age.
The dataset focuses on distal phalanges of the middle finger, and the labels
correspond to different age groups: 0-6 years old, 7-12 years old, and 13-19
years old. Figure 1 illustrates three patterns from each of the classes,
showcasing the observable ordinality of the labels in certain parts of the
time series.

Figure 1: Example of time series extracted from the DistalPhalanxOAG dataset.
The target ordinal scale represents different age ranges and the time series
are the outlines of the middle finger.

Our contributions can be listed as follows: 1) the introduction of five novel
TSOC techniques, focusing on the state-of-the-art approaches in TSC. 2) The
identification of $18$ ordinal time series problems from different data
sources. 3) A benchmarking study of all these five techniques over the whole
set of 18 ordinal datasets, comparing them against both similar nominal TSC
approaches and standard Machine Learning (ML) approaches. The results
demonstrate that TSOC techniques significantly enhance existing state-of-the-
art nominal TSC methods in ordinal problems. And finally, 4) contributing to
the literature with the first baseline study for TSOC, which aims to encourage
time series community for further enhancement in this novel field.

The remainder of this paper is organised as follows: related works are
described in Section 2; Section 3 formalises preliminary definitions needed to
present the different proposals; Section 4 defines the proposed TSOC methods;
Section 5 specifies the experimental setup and the datasets considered;
finally, in Section 6, we provide the conclusions and future research of our
work.

##  2 Related Works

This Section revises the literature of the two main areas involved in this
work: Time Series Classification (TSC) and Ordinal Classification.

###  2.1 Time Series Classification (TSC)

A range of TSC methodologies have been developed in recent years. The first
taxonomy grouping these approaches by their typology was proposed in [6], and
extended in [16]. In the following, we provide a summarised, yet updated,
version of the main categories of this taxonomy:

Distance-based techniques: algorithms belonging to this group aim to estimate
the class of a time series using the well-known $k$-Nearest Neighbours ($k$NN)
combined with a distance measure. The most common distance measures are the
Euclidean Distance (ED) and the Dynamic Time Warping (DTW) [23]. A more recent
proposal was the Elastic Ensemble (EE), introduced in [24], in which the
classification is performed according to a combination of several distance
measures. More recent approaches, such as Proximity Forest [25], achieve a
faster training time than previous techniques as well as a superior accuracy.
A second version of this algorithm called Proximity Forest 2.0 (PF2) [26] has
been recently developed. PF2 significantly outperforms previous distance-based
techniques, introducing a novel distance metric called Amerced Dynamic Time
Warping and implementing early stopping and pruning methods to speed up
similarity computations.

Interval-based techniques: the aim of the interval-based approaches is to
extract significant phase dependent features from the time series. Methods
belonging to this group are generally based on the random forest algorithm.
The first approach presented in this group was the Time Series Forest (TSF)
[27], which is similar to the traditional Random Forest, but each decision
tree is trained with different features extracted from intervals sampled from
the time series. This approach has been lately improved in the Canonical
Interval Forest (CIF) [28] approach, in which the feature extraction process
is extended with the so-called catch22 algorithm [29], which computes 22
relevant time series features. In the same line, a Supervised TSF (STSF)
algorithm was proposed in [30]. STSF reduces the input space to a set of
subseries, which are selected through a top-down process employing feature
ranking metrics. The results in terms of accuracy are comparable to state-of-
the-art techniques, while being way less computationally expensive. Finally,
the HIVE-COTE2 (HC2) ensemble introduced Diverse representation CIF (DrCIF)
[31], an improved version of the CIF proposal that employs the diverse
representation strategy, which trains each classifier in the ensemble with
different time series intervals and different feature subsets. Currently,
DrCIF is the state-of-the-art interval-based approach.

Shapelet-based techniques: this family of models focuses on finding short
phase independent patterns in the time series, called shapelets. Shapelet
Transform (ST) [32] is the main algorithm in this group. ST transforms the
time series dataset into a distance matrix in which cells represent distances
between shapelets and the original time series. This distance information is
used to make predictions on new input time series. In this line, we find the
current state-of-the-art of shapelet-based techniques, which is the Random
Dilated Shapelet Transform (RDST) [33]. This method uses randomly dilated
shapelets to improve both in accuracy and training time over the other
shapelet-based techniques.

Dictionary-based techniques: this type of models constructs a histogram with
the frequency of occurrence of sub-series within a time series. The first
method developed was the Bag of Symbolic-Fourier Approximation Symbols (BOSS)
[34], which builds a symbolic representation called bag-of-words, used to
compute distance measurements between different time series. This approach has
been later extended in the Word ExtrAction for time SEries cLassification
(WEASEL) [35] method, in which an ANalysis Of VAriance (ANOVA) test is used to
build a more significant bag-of-words from each time series. In this line, the
Temporal Dictionary Ensemble (TDE) method was presented in [36]. Similarly to
BOSS and WEASEL, TDE replaces the bag-of-word extraction algorithm by a more
efficient approach based on the introduction of decision trees thresholds to
compute the word representation of the time series. Finally, the novel version
of the WEASEL algorithm (WEASEL 2.0) [37] is the current state-of-the-art of
dictionary-based classifiers, it combines time series dilation with random
hyper-parameters search to build both an scalable and accurate technique.

Convolutional-based techniques: this family of methods tries to extract
features from the input time series by applying a set of convolutional
kernels. The RandOm Convolutional KErnel Transform (ROCKET) method [38] was
the first approach in this group of algorithms. It consists in training a
Ridge regressor with features extracted from the application of the kernels to
the time series. In addition, two enhanced versions of this method have also
been presented to the literature: MiniROCKET [39], which proposes a
computationally lighter kernel extraction process while maintaining a high
accuracy, and MultiROCKET [40], which extends the kernel extraction by
applying it to the first order difference of the time series and performing
more pooling operators. MultiROCKET is significantly better than MiniROCKET in
accuracy, but it is computationally more expensive. A recently established
method in the literature is the HYbrid Dictionary–Rocket Architecture (HYDRA)
[41]. HYDRA is a combination of dictionary and convolutional techniques that
achieves a highly competitive performance in comparison with existing
approaches. HYDRA can be used in combination with any ROCKET-based
methodology. In the current state-of-the-art, MultiROCKET combined with HYDRA
(HYDRA+MultiROCKET) stands as the best TSC methodology when accounting for
both accuracy and training time.

Deep Learning-based (DL) models: DL has also been exploited for TSC, matching
or eventually increasing the performance of current state-of-the-art
techniques. The best methodology belonging to this group is InceptionTime
[42], an ensemble of $5$ Inception Networks, which, as other proposals like
ResNet [43], is based on residual networks. Other InceptionTime-based
methodologies are found in the literature, such as InceptionFCN, which
combines the Fully Connected Network (FCN) with the InceptionTime architecture
[44]. The last proposal in this field has been made in [45], which employs
hand-crafted filters in the convolutional layers of the network. This helps to
reduce the training time and to focus on the extraction of specific features
of the time series like increase, decrease or peaks, among others.

Hybrid methods: Hybrid techniques refer to those methods that combine many of
the methodologies explained above. They focus on capturing more sources of
possible discriminatory features in time series. The two versions of the
HIerarchical VotE Collective Of Transformation-based Ensembles (HIVE-COTE, or
HC for simplicity) [46] are the most representative methods of this group. The
first version combines five different approaches: the EE (distance-based), a
Shapelet-based ensemble, the BOSS (dictionary-based), and the TSF and RIF-ACF
(both interval-based). The second version, which is the state-of-the-art
approach, not only regarding this group but also considering all TSC
techniques, is the HC2 [31], an improved version of the HC recently developed.
Specifically, HC2 combines four approaches from different domains:
convolutional-based (Arsenal, an ensemble of ROCKET models), interval-based
(DrCIF), dictionary-based (TDE) and shapelet-based (ST). Another model
proposed following the same concept is the Time Series Combination of
Heterogeneous and Integrated Embedding Forest (TS-CHIEF) [47], which achieves
a competitive performance using tree-structured classifiers to do so
efficiently.

This variety of algorithms has pushed the TSC field to new limits and conforms
the current overview. However, the performance for ordinal problems included
in the archives could be significantly improved using appropriate approaches.
Following, we specifically focus on ordinal classifiers.

###  2.2 Ordinal classification

Over the last few years, several ordinal classification techniques based on
nominal methodologies have been developed to exploit ordinality in the
training process. In [48], a novel learning algorithm based on large margin
rank boundaries was proposed for ordinal classification tasks, and the result
can be seen as an adaptation of the Support Vector Machine (SVM) algorithm to
ordinal problems. Subsequently, methods such as Support Vector Ordinal
Regression with Explicit Constraints (SVOREX) or SVOR with Implicit
Constraints (SVORIM) [49] have been proposed. Both methods are ordinal SVMs
but with different constraints definition criteria in the optimisation
process. In the same way, an extended kernel discriminant learning [50]
algorithm has been proposed to work as an ordinal classifier by using a
ranking constraint. This method is known as Kernel Discriminant Learning for
Ordinal Regression (KDLOR) and has been proposed as an alternative to the
ordinal SVM methods mentioned above. The key difference between these
approaches is that SVOR methods do not take advantage of the global
information of the data (especially SVOREX as only considers adjacent classes
when determining the thresholds) and also suffers from higher computational
complexity.

Other approaches to ordinal classification, such as [51], rely on Gaussian
Processes (GP) to develop an ordinal regressor: GP for Ordinal Regression
(GPOR). The performance of this new method has been compared against SVORs
techniques in several datasets, and the results showed a good generalisation
capacity and competitive performance. Furthermore, inspired by this approach
and previous SVOR methods, Srijith et al. in [52] have proposed a sparse GPOR
using Leave-One-Out cross-validation to perform model selection (LOO-GPOR), a
probabilistic least squares ordinal regressor (PLSOR) [53] and a semi-
supervised ordinal regressor using GP [54].

Deep learning for ordinal classification problems is an area that has been
less explored up to now, with some works such as [55] or [56], which studied
the application of a new output layer based on Cumulative Link Models (CLMs)
together with a Quadratic Weighted Kappa (QWK) loss function. This approach
has been proposed for tackling ordinal image classification problems, for
which excellent results are achieved in comparison to nominal approaches. A
recent work [57] has taken this idea to approach the field of Aesthetic
Quality Control (AQC), developing a DL architecture with CLM as the output
layer and cross-entropy as the cost function. The results have demonstrated
the capacity of the network to take advantage of the ordinal nature of AQC
problems, improving performance over previous nominal approximations.

At this point, once both main fields tackled in this work have been
introduced, Time Series Ordinal Classification (TSOC) can be defined. TSOC
consists in the application of ordinal classifiers to ordinal time series
problems, in such a way that the ordinal classifiers are able to consider and
exploit the ordinal information present in the label space. TSOC is a recently
established field only covered by a small set of conference papers [58, 59,
60]. The first two works have proposed an ordinal shapelet transformer in
which the ordinal information of the data is exploited in two ways: 1) by
introducing a novel shapelet quality measure in which ordinal information is
taken into account, and 2) by using an ordinal classifier instead of a nominal
one, specifically the Proportional Odds Model (POM) [61] and SVORIM [49] have
been tested as final classifiers. Results have demonstrated that ordinal
approaches significantly outperform nominal TSC approaches. Later, in [60],
the effect of using different $L_{p}$ norms for the computation of the
shapelet quality has been studied, concluding that the best p-norm value is
$p=1.9$ for the proposed experiments. As can be seen, TSOC has room for
significant improvement.

##  3 Preliminary definitions

In this section, a formal definition of the concepts related to TSOC is
provided, including time series and ordinal classification. Moreover, we also
define a family of probabilistic ordinal classifiers (Cumulative Link Models,
CLMs), which will be used for the TSOC methods proposed in this paper.

###  3.1 Time series and related supervised learning problems

In supervised learning time series problems, we are provided a training
dataset including a set of labeled time series,
$D=\\{(\mathbf{t}_{1},y_{1}),(\mathbf{t}_{2},y_{2}),\ldots,(\mathbf{t}_{N},y_{N})\\}$,
where $N$ is the number of available time series, $\mathbf{t}_{i}$ is the
$i$-th time series, $y_{i}$ is the label assigned to it, and
$i\in\\{1,\ldots,N\\}$. In general, the objective is to learn a mapping
function (model) able to accurately predict the labels for the time series of
the test set. Every time series is a set of $C$ ordered sets (also known as
channels or dimensions) of real values. Although the most common type of time
series are univariate ($C=1$), there is an increasing interest in multivariate
time series ($C>1$). A multivariate time series with $C$ channels is defined
as $\mathbf{t}_{i}=(\mathbf{t}_{i}^{1},\ldots,\mathbf{t}_{i}^{C})$, where the
$c$-th channel is denoted as
$\mathbf{t}_{i}^{c}=(t^{c}_{i,1},t^{c}_{i,2},\ldots t^{c}_{i,T})$,
$c\in\\{1,\ldots,C\\}$, and $T$ is the length of the time series. In this
paper, we focus on time series with a constant spacing of observation times,
in contrast to unevenly spaced time series, and we consider datasets where all
the time series are equal-length.

Depending on the nature of the labels, the supervised time series problems can
be categorised in Time Series Extrinsic Regression (TSER) [62], nominal Time
Series Classification (TSC) [16] or Time Series Ordinal Classification (TSOC).
TSER covers those problems where the label to be predicted is a real number,
i.e. $y_{i}\in\mathbb{R}$. For TSC, the label of each time series takes values
in a discrete set of categories,
$y_{i}\in\\{\mathcal{C}_{1},\mathcal{C}_{2},\ldots,\mathcal{C}_{q},\ldots,\mathcal{C}_{Q}\\}$,
where $q\in\\{1,\ldots,Q\\}$, and $Q$ is the number of classes of the problem.
Finally, TSOC problems are TSC problems (with more than two classes) which
include an order constraint in the labels.

###  3.2 Time series ordinal classification (TSOC)

The focus of this paper is TSOC, where the categories show a natural order, in
such a way that a relation
$\mathcal{C}_{1}\prec\mathcal{C}_{2}\prec\ldots\prec\mathcal{C}_{Q}$ is
present in label space. This relation is also present in TSER problems, where
the $\prec$ operator is directly $<$ because we are working with real numbers
instead of categories. However, the difference between TSOC and TSER is that
the distance between categories is not known.

To provide a deeper understanding of this sort of problems, let us consider
the example depicted in Figure 1. In this case, the classification problem
involves determining the age ranges using the outlines of the middle finger.
The ranges exhibit a natural relationship and can be categorised into three
levels: 0-6 years old, 7-12 years old, and 13-19 years old.

TSOC problems involve two important characteristics: 1) misclassification
costs are different depending on the error (an error confusing the classes 0-6
years old and 13-19 years old should be more penalised than one confusing the
classes 0-6 years old and 7-12 years old); and 2) including the order
information during learning should boost convergence.

####  3.2.1 Cumulative Link Models

CLM belong to a broad family of models known as threshold-based models, which
are built on two main ideas: 1) The existence of a real variable underlying
the (discrete) output response, such that each class $\mathcal{C}_{q}$ belongs
to a certain interval in $\mathbb{R}$. This hidden variable will be denoted as
$y^{*}$ and is often referred to in the literature as latent variable. 2) The
use of a function $f$ that transforms a pattern of the input space
$\mathbf{x}\in\mathbf{X}\subset\mathbb{R}^{D}$ (where $D$ is the input
dimensionality) into the one-dimensional real space corresponding to the
latent variable,
$f:\mathbf{x}\in\mathbf{X}\subset\mathbb{R}^{C}\to\mathbb{R}$. The aim is to
segment the real line into $Q$ consecutive non-overlapping ordered intervals,
so that each one is mapped to a certain label $\mathcal{C}_{q}$. The values
defining those intervals are represented by the vector
$\bm{\theta}=(\theta_{1},\ldots,\theta_{Q-1})$, where
$\bm{\theta}\in\mathbb{R}^{Q-1}$, which is called the threshold vector and
satisfies $\theta_{1}\leq\theta_{2}\leq...\leq\theta_{Q-1}$. Considering this
setting, a given output class $\mathcal{C}_{q}$ will be observed for a certain
input $\mathbf{x}_{i}$, if $f(\mathbf{x}_{i})\in[\theta_{q-1},\theta_{q}]$.
For the sake of simplicity, $\theta_{0}$ and $\theta_{Q}$ are set to $-\infty$
and $+\infty$ respectively, and are considered to be outside the vector
$\bm{\theta}$.

In view of the foregoing, the special case of CLM consider the latent variable
model: $y^{*}=f(\mathbf{x})+\epsilon$, where $\epsilon$ is the random error
component with zero expectation, for which a certain distribution
$F_{\epsilon}$ is assumed. Moreover, it assumes a linear latent variable
$f(\mathbf{x})=\mathbf{w}^{T}\mathbf{x}$. On that account, it is satisfied
that:

$\displaystyle P(y\preceq\mathcal{C}_{q}|\mathbf{x})$
$\displaystyle=P(y^{*}\leq\theta_{q})$ (1)
$\displaystyle=P(\mathbf{w}^{T}\mathbf{x}+\epsilon\leq\theta_{q})$
$\displaystyle=P(\epsilon\leq\theta_{q}-\mathbf{w}^{T}\mathbf{x}),$

taking $g$ as the probability distribution function (p.d.f) of the assumed
$F_{\epsilon}$, we get to:

$g(\theta_{q}-\mathbf{w}^{T}\mathbf{x})=P(y\preceq\mathcal{C}_{q}|\mathbf{x}).$
(2)

Therefore, the learning objective of the CLM is to find the set of optimal
thresholds $\bm{\theta}^{*}$, as well as the optimal parameters
$\mathbf{w^{*}}$ of the function $f$, in such a way that, given a pattern of
the test set $(\mathbf{x}_{i},y_{i})$, the cumulative probabilities for each
class $\mathcal{C}_{q}$, i.e. $P(y_{i}\preceq\mathcal{C}_{q}|\mathbf{x}_{i})$,
are as close as possible to the observed ones.

To this end, we need to define a loss function $\ell$ that represents the
disagreement between the target label assigned to an input $\mathbf{x}_{i}$
and the output given by a prediction function $\text{pred}(\mathbf{p}_{i})$,
where $\mathbf{p}_{i}=(p_{i_{1}},p_{i_{2}},...,p_{i_{Q}})$ is the vector of
predicted probabilities, where
$p_{i_{q}}=P(y_{i}=\mathcal{C}_{q}|\mathbf{x}_{i})=P(y_{i}\preceq\mathcal{C}_{q}|\mathbf{x}_{i})-P(y_{i}\preceq\mathcal{C}_{q-1}|\mathbf{x}_{i})$.
Hence, our objective is to find the function pred that minimises the expected
risk:

$\mathcal{L}(\text{pred})=\text{E}(\ell(y_{i},\text{pred}(\mathbf{p}_{i}))),$

where $\mathcal{L}$ is the risk function and E represent the expected value.
There are different options for the expected risk within the context of
ordinal regression (e.g. the absolute difference in number of categories
between the predicted and true labels). However, we cannot deal directly with
this expected risk, mainly for two reasons: 1) the probability distribution
that generates the patterns $(\mathbf{x_{i}},y_{i})$ of a dataset $D$ is
usually unknown; 2) the function $\ell$ is naturally discontinuous since its
two arguments belong to the discrete space defined by the output $y$. This, as
stated in [63], can lead to an NP-hard problem. Therefore, the common practice
is to approximate $\ell$ by the so called surrogate risk, which, considering
the CLM setting, can be defined as:

$\mathcal{A}(f)=E(\psi(\bm{\theta},f(\mathbf{x})),$

where $\psi:\mathbb{R}^{Q-1}\times\mathbb{R}\to\mathbb{R}$ is generally
referred to as the surrogate loss function. The need to introduce this
surrogate terms is the main motivation for the existence of the previously
introduced latent variable. As the values of $y^{*}$ are not a priori known,
we work with the vector $\bm{\theta}$, so that, for a pair
$(\mathbf{x}_{i},y_{i})$, where $y_{i}$ takes a value $\mathcal{C}_{q}$, we
want the output of $f(\mathbf{x}_{i})$ to be as close as possible to the
$\theta_{q}$ threshold. To ensure the ordering of $\bm{\theta}$ introduced
above, we essentially have two options, one is to generate its elements
through an incremental function, as for example, the one used in the CLM
activation layer (see Section 3.2.3). The other option is to design the cost
function in such a way that it converges to a point where this ordering is
assured, this is the case of the Logistic All-Threshold [63] method being used
in our work.

####  3.2.2 Logistic All-Threshold (LogisticAT)

The LogisticAT method is a special case of CLM where the considered
$F_{\epsilon}$ distribution is the logistic function. This leads to an
interesting property that characterises the so-called proportional odds models
(POM). This property states that the ratio of the odds for two patterns inputs
$\mathbf{x}_{1}$, $\mathbf{x}_{2}$ is proportional [64]:

$\frac{odds(y\preceq\mathcal{C}_{q}|\mathbf{x}_{1})}{odds(y\preceq\mathcal{C}_{q}|\mathbf{x}_{2})}=\text{exp}(-\mathbf{w}^{T}(\mathbf{x}_{2}-\mathbf{x}_{1})).$

The main distinguishing feature of LogisticAT with respect to the POM is its
(surrogate) loss function, which is presented below:

$\displaystyle\psi_{\mathrm{AT}}=$
$\displaystyle\sum_{i=1}^{N}\left(\sum_{q=1}^{y_{i}-1}h(\theta_{q}-\mathbf{w}^{T}\mathbf{x}_{i})+\sum_{q=y_{i}}^{Q}h(\mathbf{w}^{T}\mathbf{x}_{i}-\theta_{q})\right)+$
(3) $\displaystyle\frac{\lambda}{2}\mathbf{w}^{T}\mathbf{w},$

where $(\mathbf{x}_{i},y_{i})$ is a pattern of the training set,
$h:\mathbb{R}\to\mathbb{R}$, $h(z)=\text{log}(1+\text{exp}(z))$, $\mathbf{w}$
is the parameter of the function $f(\mathbf{x})=\mathbf{w}^{T}\mathbf{x}$
presented above, and $\lambda$ is the regularization term, which is adjusted
by cross-validation in our experiments (see Section 5). Observe that in
$\psi_{AT}$, by taking into account all the thresholds in $\bm{\theta}$ for
each input $(\mathbf{x}_{i},y_{i})$, we ensure that at the point of
convergence the ordering of $\bm{\theta}$ is satisfied.

This model is used for the ordinal convolutional-based techniques (O-ROCKET,
O-Mini-Rocket and O-MultiROCKET) discussed in Section 4.1.

####  3.2.3 CLM Activation Layer

The previously described mathematical structure of the cumulative link models
can be utilised as link function in deep learning architectures [55]. On this
wise, the one dimensional output of a deep network is mapped to a set of
predicted probabilities through a CLM, for which we considered (as in the case
of the LogisticAT method) a logistic distribution for the error component
$\epsilon$. In order to ensure the ordering of the vector $\bm{\theta}$, the
next definition is considered:
$\theta_{q}=\theta_{1}+\sum_{i=1}^{q-1}\alpha_{i}^{2}$, with
$q\in\\{2,...,Q-1\\}$, then $\theta_{1}$ and $\alpha_{2},\ldots,\alpha_{Q-1}$
can be learnt without constraints.

In addition, a cost function that takes greater account of ordinality is
considered [65]. This function, denoted as $\psi_{\text{\tiny QWK}}$, is based
in the Quadratic Weighted Kappa (QWK) metric, and it is defined in terms of
the predicted probabilities:

$\psi_{\text{\tiny
QWK}}=\frac{\sum\limits_{i=1}^{N}\sum\limits_{q=1}^{Q}\omega_{y_{i},q}P(y=\mathcal{C}_{q}|\mathbf{x}_{i})}{\sum\limits_{q=1}^{Q}\frac{N_{q}}{N}\sum\limits_{j=1}^{Q}(\omega_{j,q}\sum\limits_{i=1}^{N}P(y=\mathcal{C}_{j}|\mathbf{x}_{i}))},$

where $\psi_{\text{\tiny QWK}}\in[0,2]$, ($\mathbf{x}_{i},y_{i}$) is the i-th
sample of the data, $N_{q}$ is the number of patterns labeled with the $q$-th
class, and $\omega_{j,q}$ are the elements of the penalization matrix, where
$\omega_{j,q}=\frac{(j-q)^{2}}{(Q-1)^{2}}$.

This setting is implemented in the ordinal deep learning methodologies
(O-ResNet and O-InceptionTime) discussed in Section 4.2.

##  4 Proposed TSOC methods

In this Section, we present the TSOC methodologies proposed according to the
category they belong.

###  4.1 Convolutional-based techniques

This family of TSC algorithms was opened in [38] with the development of the
ROCKET method for TSC. The main idea is to extract features from the input
time series by applying a set of convolutional kernels. This convolution
process involves a sliding dot product that depends on the properties of the
kernel being applied, which are: the values of the kernels or weights
($\mathbf{w}$), the length ($n$), which sets the kernel extension (or number
of weights); dilation ($d$), that spreads the kernel over the time series so
that higher values of dilation help to more easily capture long-term patterns
in the series [66]; padding ($p$), which adds to the beginning and end of the
series a vector filled with zeros to control where the middle weight of the
first and last kernel to be applied falls; and bias ($b$), a real value that
is added to the kernel convolution result.

In addition to the original version of ROCKET, the authors presented two
improved versions: MiniROCKET and MultiROCKET. All of them have the same
architecture, shown in Figure 2, in which we can distinguish four sequentially
applied phases: 1) a kernel convolution transform, 2) a set of pooling
operations, 3) a standard scaler, and 4) a final classifier.

In this work, we propose the use of the LogisticAT method introduced in
Section 3.2.1 as final classifier, with a built-in crossvalidation of the
$\lambda$ regularization parameter. With respect to the standardization
process and the final classifier, both remain unchanged. Conversely, the first
two phases (kernel convolution and pooling operations), are different for each
of the three alternatives. A comparison between the configuration of the three
methods is presented in Table I.

####  4.1.1 O-ROCKET

ROCKET was a significant contribution to the state-of-the-art in TSC, as it
was capable of achieving an excellent performance in a fraction of
computational time. ROCKET applies to the input time series a large set of
kernels generated with random properties: length $n$ is randomly sampled from
$\\{7,9,11\\}$ with equal probability; weights $\mathbf{w}$ are sampled from a
normal distribution, $w_{k}\sim\mathcal{N}(0,1)$; bias $b$ is sampled from an
uniform distribution, $b\sim\mathcal{U}(-1,1)$; dilation
$d=2^{x},x\sim\mathcal{U}(0,A),A=log_{2}\frac{T-1}{n-1}$ where $T$ denotes the
time series length, and $n$ the kernel length; and padding, which is denoted
by $p$ and is applied or not with the same probability in each kernel. In the
case it is applied, its value is computed with $p=(n-1)*d$.

Once the kernel convolution is finished, two real-valued features are
extracted from each kernel, one being the result of a Global Max Pooling
(GMP), which consists in taking the highest value, and the second being the
Proportion of Positives Values (PPV), being
$\text{PPV}=\frac{1}{n}\sum_{i=0}^{n-1}[z_{i}>0]$, where $z_{i}$ is the output
of a single kernel convolution. Hence, PPV is a real value ranging from zero
to one that represent the percentage of kernel convolutions that are greater
than zero. These extracted features are then fed as input to the LogisticAT
classifier.

Figure 2: Generalised ordinal convolutional-based methods architecture. The
final set of features obtained from the kernel convolution and pooling
operations is standardised and then fed to a LogisticAT classifier, which
produces the final output.

####  4.1.2 O-MiniROCKET

MiniROCKET [39] is a later version of ROCKET which is up to $75$ times faster
than the original version, achieving similar performances. The main novelties
with respect to ROCKET are:

  * •

The kernel length $n$ is set to a fixed value of $9$.

  * •

Kernel weights $\mathbf{w}$ are restricted to two possible values, $-1$ and
$2$.

  * •

A fixed dilation $d$ is computed according to the input length. The $d$ value
can be in $\\{2^{0},\ldots,2^{a}\\}$, where $a=\text{log}_{2}(T-1)/(n-1)$.

  * •

Padding $p$ is applied or not alternatively for each kernel/dilation
combination. This way, half of the kernels are computed with padding and the
other half not. Whenever applied, its value is computed in the same way as in
the O-ROCKET method.

  * •

Only the PPV aggregated feature is computed for each kernel.

With these changes, MiniROCKET becomes mostly (sometimes fully) deterministic,
achieving both better accuracy and lower training time than comparable
methodologies, such as its predecessor, ROCKET, InceptionTime [42],
dictionary-based methods like Temporal Dictionary Ensemble (TDE) [36] or tree-
based techniques as the Canonical Interval Forest (CIF) [28].

####  4.1.3 O-MultiROCKET

This is the last version of the ROCKET family. In this novel variant, known as
MultiROCKET [40], two main adjustments are made to MiniROCKET: 1) the
convolution is also done on the first order difference of the time series; and
2) three additional aggregated features are computed: the Mean Positives
Values (MPV), Mean of Indices of Positives Values (MIPV) and the Longest
Stretch of Positive Values (LSPV).

These modifications substantially increase the computational expense of the
algorithm but in turn achieves a significant increase in terms of accuracy.

| O-ROCKET | O-MiniROCKET | O-MultiROCKET
---|---|---|---
Kernel length | {7, 9, 11} | {9} | {9}
Weights | $\mathcal{N}(0,1)$ | {-1, 2} | {-1, 2}
Dilation | $\mathcal{U}(-1,1)$ | $\\{2^{0},\ldots,2^{a}\\}$ | $\\{2^{0},\ldots,2^{a}\\}$
Use of Padding | random | alternative | alternative
Pooling operations | [PPV, GMP] | [PPV] |  | [PPV, GMP, MPV,
---
MIPV, LSPV]
Num. Features | 20,000 | 10,000 | 50,000
Table I: Differences between O-ROCKET, O-MiniROCKET and O-MultiROCKET
methodologies.

###  4.2 Deep Learning techniques

We propose two types of deep learning architectures: the Ordinal ResNet
(O-ResNet) methodology based on the residual network architecture [43], and
the Ordinal InceptionTime (O-InceptionTime) methodology based on the
InceptionTime architecture [42].

####  4.2.1 O-ResNet

The original residual architecture was proposed in [67]. The main idea behind
this sort of architectures is to add shortcut connections between non-adjacent
layers, which facilitates the flow of the gradient through the network. This
work was extended to the TSC paradigm in [43], in which the input time series
is passed sequentially through a set of three residually connected
convolutional blocks, which, for ease of understanding, are referred to as
super-blocks (SBlocks). The reason for this notation is given by the fact that
each super-block $\text{SBlock}_{k}$ is composed of three simple convolutional
blocks with $k$ filters each. In Figure 3, we refer to this fact with the x3
notation. In each $\text{SBlock}_{k}$, the convolution result is added to the
original time series, and finally, a Batch Normalization layer (BN) and a
Rectified Linear Unit (ReLU) activation layer are applied. In our proposal,
after the concatenated three SBlocks convolution, the final output of the
network is obtained by applying a Global Average Pooling (GAP) layer followed
by a CLM activation layer (see Figure 3). The complete O-ResNet convolution
pipeline is given by:

$\displaystyle h_{i_{1}}$ $\displaystyle=\text{SBlock}_{64}(\mathbf{x}_{i}),$
(4) $\displaystyle h_{i_{2}}$ $\displaystyle=\text{SBlock}_{128}(h_{i_{1}}),$
$\displaystyle h_{i_{3}}$ $\displaystyle=\text{SBlock}_{128}(h_{i_{2}}),$
$\displaystyle h_{i_{4}}$ $\displaystyle=\text{GAP}(h_{i_{3}}),$
$\displaystyle\mathbf{p}_{i}$ $\displaystyle=\text{CLM}(h_{i_{4}}),$

where $\mathbf{p}_{i}$ is the final output of the network, i. e. the set of
predicted probabilities.

Figure 3: O-ResNet architecture. The notation x3 means that we have 3 stacked
convolutional blocks inside each residual block. The final activation layer is
the CLM (see Section 3.2.1).

####  4.2.2 O-InceptionTime

The O-InceptionTime architecture takes some of the main ideas of the ResNet
architecture and the originial InceptionTime methodology [42], which includes
concatenated convolution blocks together with residual connections between
them. This method itself is an ensemble of $5$ O-Inception Networks (O-IN). An
O-IN comprises two residual blocks, each consisting of three Inception Modules
(IMs) that remain unchanged from the initial InceptionTime proposal (see
Figure 4). O-IN applies a dimensionality reduction layer known as bottleneck,
whose main goals are reducing the model complexity and avoiding overfitting.
The convolution operation of a bottleneck layer consists in sliding $m$
filters of length $1$ with a stride equal to $1$. Figure 5 presents the O-IN
architecture.

Figure 4: InceptionTime Module architecture. Two pipelines are applied in
parallel to the input time series. On the first one, the bottleneck layer is
applied, followed by $3$ sliding filter operations with sizes $40$, $20$ and
$10$, respectively. On the second one, a Max Pooling is performed, followed by
another bottleneck layer. The result is a multivariate 4-channels time series.

Specifically, an O-IN is composed of two residual blocks, each with 3 IMs. A
GAP layer is applied to the output of the second block. Finally, the resulting
features are fed to a fully connected layer, and finally, to the CLM
activation layer.

As mentioned above, O-InceptionTime consists of $5$ O-INs initialised with
random weights, where the final prediction is performed by a majority voting
system. This ensemble is built with the purpose of reducing the variability
inherent to the ResNet architectures [68].

Figure 5: O-Inception Network architecture, two blocks of three IMs with
residual connections are stacked, the resulting multivariate time series is
passed to a GAP layer, followed by a fully connected and a CLM activation
layer.

##  5 Experimental settings and results

This Section firstly introduces the naïve methods used for comparison together
with the datasets considered in the experiments. Then, it analyses the
developed ordinal methodologies and presents the ordinal performance metrics.
The section finishes by benchmarking all the methods compared. Note that we
provide open source scikit-learn-compatible implementations of all the methods
benchmarked, a guide on how to reproduce experiments, and all the results
obtained in an associated website222https://uco.es/ayrna/grupos/tsoc-dl-conv.
All ordinal approaches will be available in the aeon toolkit
333https://github.com/aeon-toolkit/aeon/, which has also been used for
benchmarking and analysing the results obtained.

###  5.1 Naïve methodologies

In order to to better understand how the proposed TSOC techniques perform,
three baseline approaches have been run as a sanity check. The first two are a
logistic regression [69] (LogReg) and a Extreme Gradient Boosting (XGBoost)
[70] classifier. For these, time series are flattened into a vector
concatenating all the channels. Thus, a multivariate time series with $C$
channels and length $T$ is transformed into a single vector of length $C\times
T$. The third baseline method is the Time Series Forest (TSF) [27]. As
explained above, these techniques are considered to enrich the
experimentation, since they have been proven to be competitive in a wide range
of problems.

###  5.2 TSOC Datasets

Given that we are covering the TSOC paradigm, all of the datasets considered
must ordinal in nature. To this end, we made a dataset selection from two time
series repositories, the UCR TSC
archive444https://timeseriesclassification.com, and the Monash TSER
archive555http://tseregression.org/. For the UCR TSC repository, we analysed
the output variable of each problem to select those with an ordinal nature. We
have identified seven univariate, and two multivariate datasets. Moreover, we
have chosen two additional datasets from the TSER archive. Given that it is a
regression repository, we have discretised the original continuous output
variable, grouping it into five uniformly distributed bins. Even though there
are more regression datasets, we could only select these two as it is
difficult to achieve a reasonable class distribution for the rest (the output
values are too close to obtain more than three representative classes).
Furthermore, the datasets with unequal length time series and missing values
are also discarded.

Other five TSOC problems built from historical price data from five of the
most important companies in the stock market are considered: Apple (AAPL),
Amazon (AMZN), Google (GOOG), META and Microsoft (MSFT). The data has been
collected from Yahoo Finance666https://es.finance.yahoo.com/ website,
extracting weekly price data from each stock earliest available date to March
2023. The time series of each stock data are formed by its price returns over
53 weeks (around one year) before a varying date $t$. The output ordinal label
corresponds to the price return in $t$. This value is discretised according to
a set of predefined symmetrical thresholds
$(-\infty,-0.05,-0.02,0.02,0.05,\infty)$. Finally, two more TSOC problems were
built from data collected from the National Data Buoy Center
(NDBC)777https://www.ndbc.noaa.gov/. Input multivariate time series are seven
climatological variables collected from a reanalysis model, as detailed in
[71]. Some of these variables are the air temperature, the pressure, or the
relative humidity. Each time series comprises four values per day during four
weeks, i.e. the length of the time series is $112$. The output label indicates
the energy fluctuation level during the data collection period, with $0$ being
the minimum and $3$ the highest level. The second problem also comprises
multivariate time series. Same seven climatological variables are considered
for this problem. This time, the length of the time series is $28$, as
collects values during only one week (four values per day). The purpose is to
estimate the wave height level during that period of time, from $0$ (the
lowest height) to $3$ (the highest height). Note that the outputs of both
datasets have been discretised in bins with identical widths.

| Dataset | #Train | #Test | $Q$ | $T$ | $C$
---|---|---|---|---|---|---
TSC | AtrialFibrillation | 15 | 15 | 3 | 640 | 2
DistalPhalanxOAG | 400 | 139 | 3 | 80 | 1
DistalPhalanxTW | 400 | 139 | 6 | 80 | 1
EthanolConcentration | 261 | 263 | 4 | 1751 | 3
EthanolLevel | 504 | 500 | 4 | 1751 | 1
MiddlePhalanxOAG | 400 | 154 | 3 | 80 | 1
MiddlePhalanxTW | 399 | 154 | 6 | 80 | 1
ProximalPhalanxOAG | 400 | 205 | 3 | 80 | 1
ProximalPhalanxTW | 400 | 205 | 6 | 80 | 1
TSER | AppliancesEnergy | 95 | 42 | 5 | 144 | 24
Covid3Month | 140 | 61 | 5 | 84 | 1
new TSOC | AAPL | 1720 | 431 | 5 | 53 | 1
AMZN | 1035 | 259 | 5 | 53 | 1
GOOG | 732 | 183 | 5 | 53 | 1
META | 408 | 103 | 5 | 53 | 1
MSFT | 1501 | 376 | 5 | 53 | 1
USASouthwestEnergyFlux | 327 | 141 | 4 | 112 | 7
USASouthwestSWH | 1310 | 562 | 4 | 28 | 7
Table II: Characteristics of the datasets used in this work. #Train stands for
the number of training patterns, #Test that of the test set, $Q$ the number of
classes, $T$ for the time series length, and $C$ is the number of
channels/dimensions.

###  5.3 Experimental settings

All methodologies detailed in Section 3 together with the naïve approaches
have been applied to the datasets shown in Table II. Each dataset has a
default train/test partition. Given the stochastic nature of the methodologies
and to avoid overfitting the training data of the default partition, every
experiment is repeated 30 times. Specifically, the first experiment is carried
out with the default partition. Subsequent experiments are performed with
stratified resamples of the original datasets, keeping the same train/test
proportion as the original. We have to crossvalidate only one hyperparameter
in our experiments, $\lambda$, which is the regularisation term of the
LogisticAT classifier used in the ordinal convolutional-based methodologies.
The set of $\lambda$ values to be tested are obtained according to
$10^{-3+\frac{6i}{9}}$, where $i\in\\{0,1,\ldots,9\\}$. A 5-fold stratified
crossvalidation approach is used. The best $\lambda$ value is selected by MAE,
defined in Equation 6, as it best represents the performance of ordinal
approaches.

As this work deals with ordinal classification, specific ordinal metrics
should be considered. Concretely, MAE, 1-Off accuracy (1-OFF) and Quadratic
Weighted Kappa (QWK) are considered. All these metrics aim to quantify how
close the predictions are to the actual variable on the ordinal scale. In
addition, the Correct Classification Rate (CCR), widely used in nominal
problems, is also computed. We formalise these metrics below:

  * •

CCR computes the percentage of correctly classified instances following the
next equation:

$\text{CCR}=\frac{1}{N}\cdot\sum_{i=1}^{N}\delta(\hat{y}_{i}-y_{i}),$ (5)

where $\delta$ express the zero-one loss function, $\hat{y}_{i}$ and $y_{i}$
are the predicted and the actual labels, respectively.

  * •

MAE express the mean deviation of the obtained predictions from the target
labels considering the ordinal scale:

$\text{MAE}=\frac{1}{N}\cdot\sum_{i=1}^{N}|\hat{y}_{i}-y_{i}|.$ (6)

Higher values of MAE indicate poor performance since it means that our
predictions are on average farther from the actual class.

  * •

QWK is based on the kappa index [72]. This index establishes different weights
to the levels of disagreement between the actual and the predicted classes.
This feature makes it an appropriate metric for ordinal problems, as it
penalises more severely predictions that are farther away from the true labels
on the ordinal scale. QWK is formalised as follows:

$\text{QWK}=1-\frac{\sum^{N}_{i,j}\omega_{i,j}O_{i,j}}{\sum^{N}_{i,j}\omega_{i,j}E_{i,j}},$
(7)

where $\omega$ denotes the penalization matrix for which quadratic weights are
considered, $\omega_{i,j}=\frac{(i-j)^{2}}{(Q-1)^{2}}$,
$\omega_{i,j}\in[0,1]$, $O_{i,j}$ is the $ij$-th element of the confusion
matrix, and $E_{i,j}=\frac{O_{i\bullet}O_{\bullet j}}{N}$, where
$O_{i\bullet}$ are the marginal frequencies of the confusion matrix for the
target class $\mathcal{C}_{i}$ and $O_{\bullet j}$ are the marginal
frequencies of the confusion matrix for predicted class $\mathcal{C}_{j}$.

  * •

1-OFF metric measures the proportion of samples classified in the target class
or in any of the adjacent classes. It is defined as:

$\text{OFF1}=\frac{S(\mathbf{M})}{S(\mathbf{O})}$ (8)

where the $S$ function returns the sum of the elements of a vector or matrix,
$\mathbf{O}$ is the confusion matrix, and $\mathbf{M}$ is the result of
multiplying $\mathbf{O}$ by a matrix where all elements are zero, except the
elements of the main diagonal and the adjacent diagonals, which are filled
with ones.

Finally, the comparison of the different classifiers has been performed using
the pairwise Wilcoxon signed-rank tests (see Table III) for the comparison
between nominal and ordinal techniques, and an adaptation of the critical
difference diagram [73] for a global comparison between ordinal and baseline
methodologies (see Figure 6). In this case, the cliques are built using the
Holm correction [74] and a significance level of $0.05$.

###  5.4 Results

The first assessment is to determine whether the ordinal algorithms outperform
the nominal techniques. Specifically, we compare ordinal versus nominal
versions for the InceptionTime, ResNet, ROCKET, MiniROCKET, and MultiROCKET
classifiers. The results are presented in Table III. In general, ordinal
methods are significantly better than nominal techniques. The exception is
ResNet, where there is no discernible difference, InceptionTime in terms of
QWK, and MultiROCKET in terms of CCR.

|  InceptionTime vs O-InceptionTime |  ResNet vs O-ResNet |  ROCKET vs O-ROCKET |  MiniROCKET vs O-MiniROCKET |  MultiROCKET vs O-MultiROCKET
---|---|---|---|---|---
MAE | $\mathbf{<0.01}$ | 0.320 | $\mathbf{<0.01}$ | $\mathbf{<0.01}$ | $\mathbf{0.033}$
CCR | $\mathbf{<0.01}$ | 0.085 | $\mathbf{0.011}$ | $\mathbf{0.020}$ | 0.557
QWK | 0.276 | 0.948 | $\mathbf{<0.01}$ | $\mathbf{<0.01}$ | $\mathbf{<0.01}$
1-OFF | $\mathbf{<0.01}$ | 0.231 | $\mathbf{<0.01}$ | $\mathbf{<0.01}$ | $\mathbf{<0.01}$
Table III: $p$-values of the Wilcoxon pair tests for nominal versus ordinal
versions of the methodologies proposed in this work.

(a) TSOC and baseline methods ranked by MAE.

(b) TSOC and baseline methods ranked by CCR.

(c) TSOC and baseline methods ranked by 1-OFF.

(d) TSOC and baseline methods ranked by QWK.

Figure 6: A comparison between the presented TSOC methodologies and baselines
methods for the four metrics considered. A CDD is represented where cliques
are built using the wilcoxon signed rank test with holm correction and a
significance level of 0.05.

The second experiment will determine which is the state-of-the-art approach in
TSOC. Specifically, the five ordinal versions of the proposed approaches are
benchmarked along with baseline approaches: XGBoost, LogisticRegression and
TSF. The results for the testing sets are graphically presented in Figure 6.
We observe that O-MiniROCKET remains the best in three out of the four
metrics, being only slightly outperformed by O-ROCKET in terms of 1-OFF. In
general, it is noted that convolution-based methodologies have the best
performance across all ordinal metrics. For MAE, the top clique is only
composed of the four top-ranked ordinal approaches (note that the LogReg
method is outside this clique). Focusing now on the performance of the deep
learners, the O-InceptionTime remains the best method if we keep convolution-
based techniques aside, except for the CCR metric. The reason behind this
behaviour is that O-InceptionTime requires more training patterns to improve
its performance. Note that TSF and LogReg achieve their best results for CCR.
This is understandable as the goal of these approaches along with XGBoost is
achieving the best results in the nominal performance metric (CCR). Moreover,
since most of the top-performing approaches are in the same clique, and no
significant differences are achieved among them, it can be said that there is
still room for better algorithms to be adapted and developed for TSOC
problems. Furthermore, another way of improvement could be extending the
problems archive, to include a wider range of datasets.

With the goal of assessing the computational load of the approaches, Figure 7
compares the execution times with respect to the MAE. It can be observed that
ROCKET-based methodologies have the best balance between time and performance,
being the O-MiniROCKET the best.

Figure 7: Run time in milliseconds (log scale average over all datasets)
plotted against mean MAE obtained in the test sets.

Additionally, the performance of each algorithm is compared in terms of
relative MAE. The relative MAE is calculated by scaling the MAE of each
approach with the median MAE for each dataset. Figure 8 presents the relative
MAE for each algorithm using boxplots. In this figure, values greater than
$0.5$ indicate that the approach performs worse than the average method, while
values smaller than $0.5$ indicate that the approach performs better than the
average method. It can be observed that the ROCKET family exhibits a larger
spread in the values, with most values being smaller than $0.5$. On the other
hand, O-ResNet, the two standard approaches (LogReg and XGBoost), and the TSF
approaches have values around $0.5$, indicating that they perform similarly to
the average algorithm. Finally, in Figure 9, a global comparison in terms of
MAE between the presented ordinal techniques is provided in the form of a full
pairwise Multi-Comparison Matrix (MCM) [75]. As can be observed, the
MiniROCKET method is positioned as the best performing technique, obtaining a
significant p-value,lower than $0.05$, against the rest of methodologies,
except for the ROCKET.

Figure 8: Relative MAE. Lower and greater values than $0.5$ indicate that the
method is better and worse, respectively, than the average method. Figure 9:
Multi-Comparison Matrix between ordinal methodologies in MAE. In each cell,
three values are provided: 1) Average of differences in MAE. 2)
Win/Ties/Losses counts. And 3) p-value of a Wilcoxon signed rank test.

##  6 Conclusion

This paper presents the first approach of convolutional and deep learning
based techniques to Time Series Ordinal Classification (TSOC), up-to-the-
knowledge of the authors. This area remains largely unexplored in comparison
against nominal Time Series Classification (TSC). In this work, one of the
contributions is the first archive of TSOC problems, including a total of 18
datasets of various domains. We also contributes to the literature ordinal
version of the two main categories in TSC: convolutional (ROCKET, MiniROCKET
and MultiROCKET) and deep learning-based approaches (InceptionTime and
ResNet). The ordinal versions of these techniques have resulted in significant
performance improvements on the selected ordinal datasets against the nominal
ones. Specifically, O-MiniROCKET is the best approach, achieving the best
performance in terms of ordinal measures in an acceptable computational time.
O-InceptionTime is the most promising deep learner, improving significantly
the results achieved by the O-ResNet. All results, source code and guidance on
reproducing experiments are available.

We recognise the potential for further advancements in TSOC algorithms.
Incorporating remaining TSC techniques can offer additional opportunities to
explore this area and can serve as a baseline for future developments in this
field. Another way of improvement is extending the experiments to consider
heterogeneous ensembles, such as HIVE-COTE2 [31], the state-of-the-art
approach in TSC. It may be possible to adapt and apply this approach to TSOC
with careful consideration. We also would like to extend our repository with
new problems. We greatly appreciate any contributions to this archive.

## Acknowledgments

This work has been partially subsidised by “Agencia Española de Investigación
(España)” (grant ref.: PID2020-115454GB-C22/AEI/10.13039/501100011033) and by
EPSRC (grant ref.: EP/W030756/1). David Guijo-Rubio’s research has been
subsidised by the University of Córdoba for the requalification of the Spanish
university system of the Ministry of Universities, financed by the European
Union - NextGenerationEU (grant ref.: UCOR01MS).

## References

  * [1] R. Khasha, M. M. Sepehri, and N. Taherkhani, “Detecting asthma control level using feature-based time series classification,” _Applied Soft Computing_ , vol. 111, 2021.
  * [2] O. B. Sezer, M. U. Gudelek, and A. M. Ozbayoglu, “Financial time series forecasting with deep learning : A systematic literature review: 2005-2019,” _Applied Soft Computing_ , vol. 90, 2020.
  * [3] D. P. Roy and L. Yan, “Robust landsat-based crop time series modelling,” _Remote Sensing of Environment_ , vol. 238, no. SI, 2020.
  * [4] G. Moody, “Spontaneous termination of atrial fibrillation: A challenge from physionet and computers in cardiology,” in _Computers in Cardiology 2004_ , ser. Computing in Cardiology Series, vol. 31. IEEE, 2004, pp. 101–104.
  * [5] J. Large, E. K. Kemsley, N. Wellner, I. Goodall, and A. Bagnall, “Detecting forged alcohol non-invasively through vibrational spectroscopy and machine learning,” in _Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018_. Springer, 2018, pp. 298–309.
  * [6] A. Bagnall, J. Lines, A. Bostrom, J. Large, and E. Keogh, “The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances,” _Data Mining and Knowledge Discovery_ , vol. 31, no. 3, pp. 606–660, 2017.
  * [7] Z. Gong, H. Chen, B. Yuan, and X. Yao, “Multiobjective learning in the model space for time series classification,” _IEEE transactions on cybernetics_ , vol. 49, no. 3, pp. 918–932, 2018.
  * [8] Q. Ma, Z. Chen, S. Tian, and W. W. Ng, “Difference-guided representation learning network for multivariate time-series classification,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 6, pp. 4717–4727, 2020.
  * [9] A. Jastrzebska, G. Nápoles, W. Homenda, and K. Vanhoof, “Fuzzy cognitive map-driven comprehensive time-series classification,” _IEEE Transactions on Cybernetics_ , 2021.
  * [10] D. Guijo-Rubio, A. M. Durán-Rosal, P. A. Gutiérrez, A. Troncoso, and C. Hervás-Martínez, “Time-series clustering based on the characterization of segment typologies,” _IEEE transactions on cybernetics_ , vol. 51, no. 11, pp. 5409–5422, 2020.
  * [11] Q. Liu, H. Peng, L. Long, J. Wang, Q. Yang, M. J. Pérez-Jiménez, and D. Orellana-Martín, “Nonlinear spiking neural systems with autapses for predicting chaotic time series,” _IEEE Transactions on Cybernetics_ , pp. 1–13, 2023.
  * [12] A. Jastrzebska, “Time series classification through visual pattern recognition,” _Journal of King Saud University - Computer and Information Sciences_ , vol. 34, no. 2, pp. 134–142, 2022.
  * [13] E. S. García-Treviño and J. A. Barria, “Structural generative descriptions for time series classification,” _IEEE transactions on cybernetics_ , vol. 44, no. 10, pp. 1978–1991, 2014.
  * [14] N. Asadi, A. Mirzaei, and E. Haghshenas, “Creating discriminative models for time series classification and clustering by hmm ensembles,” _IEEE transactions on cybernetics_ , vol. 46, no. 12, pp. 2899–2910, 2015.
  * [15] H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh, “The ucr time series archive,” _IEEE-CAA Journal of Automatica SINICA_ , vol. 6, no. 6, pp. 1293–1305, 2019.
  * [16] M. Middlehurst, P. Schäfer, and A. Bagnall, “Bake off redux: a review and experimental evaluation of recent time series classification algorithms,” _arXiv preprint arXiv:2304.13029_ , 2023.
  * [17] E. F. Harrington, “Online ranking/collaborative filtering using the perceptron algorithm,” in _Proceedings of the Twentieth International Conference on Machine Learning (ICML2003)_ , 2003.
  * [18] F. Fernandez-Navarro, P. Campoy-Munoz, M. de la Paz-Marin, C. Hervas-Martinez, and X. Yao, “Addressing the eu sovereign ratings using an ordinal regression approach,” _IEEE Transactions on Cybernetics_ , vol. 43, no. 6, pp. 2228–2240, 2013.
  * [19] C. Li, Q. Liu, W. Dong, X. Zhu, J. Liu, and H. Lu, “Human age estimation based on locality and ordinal information,” _IEEE Transactions on Cybernetics_ , vol. 45, no. 11, pp. 2522–2534, 2015.
  * [20] D. Guijo-Rubio, P. A. Gutierrez, C. Casanova-Mateo, J. Sanz-Justo, S. Salcedo-Sanz, and C. Hervas-Martinez, “Prediction of low-visibility events due to fog using ordinal classification,” _Atmospheric Research_ , vol. 214, pp. 64–73, 2018.
  * [21] K. Xu, M. Zhou, D. Yang, Y. Ling, K. Liu, T. Bai, Z. Cheng, and J. Li, “Application of ordinal logistic regression analysis to identify the determinants of illness severity of covid-19 in china,” _Epidemiology and Infection_ , vol. 148, 2020.
  * [22] J. A. Anderson, “Regression and ordered categorical variables,” _Journal of the Royal Statistical Society. Series B (Methodological)_ , vol. 46, no. 1, pp. 1–30, 1984.
  * [23] E. Keogh and C. A. Ratanamahatana, “Exact indexing of dynamic time warping,” _Knowledge and information systems_ , vol. 7, pp. 358–386, 2005.
  * [24] A. Bagnall, “Time series classification with ensembles of elastic distance measures,” _Data Mining and Knowledge Discovery_ , vol. 29, 2014.
  * [25] B. Lucas, A. Shifaz, C. Pelletier, L. O’Neill, N. Zaidi, B. Goethals, F. Petitjean, and G. I. Webb, “Proximity forest: an effective and scalable distance-based classifier for time series,” _Data Mining and Knowledge Discovery_ , vol. 33, no. 3, pp. 607–635, feb 2019.
  * [26] M. Herrmann, C. W. Tan, M. Salehi, and G. I. Webb, “Proximity forest 2.0: A new effective and scalable similarity-based classifier for time series,” 2023\.
  * [27] H. Deng, G. Runger, E. Tuv, and M. Vladimir, “A time series forest for classification and feature extraction,” _Information Sciences_ , vol. 239, pp. 142–153, 2013.
  * [28] M. Middlehurst, J. Large, and A. Bagnall, “The canonical interval forest (cif) classifier for time series classification,” in _2020 IEEE International Conference on Big Data_. IEEE, 2020, pp. 188–195.
  * [29] C. H. Lubba, S. S. Sethi, P. Knaute, S. R. Schultz, B. D. Fulcher, and N. S. Jones, “catch22: Canonical time-series characteristics,” _Data Mining and Knowledge Discovery_ , vol. 33, no. 6, pp. 1821–1852, 2019.
  * [30] N. Cabello, E. Naghizade, J. Qi, and L. Kulik, “Fast and accurate time series classification through supervised interval search,” 11 2020, pp. 948–953.
  * [31] M. Middlehurst, J. Large, M. Flynn, J. Lines, A. Bostrom, and A. Bagnall, “Hive-cote 2.0: a new meta ensemble for time series classification,” _Machine Learning_ , vol. 110, no. 11-12, pp. 3211–3243, 2021.
  * [32] J. Hills, J. Lines, E. Baranauskas, J. Mapp, and A. Bagnall, “Classification of time series by shapelet transformation,” _Data mining and knowledge discovery_ , vol. 28, no. 4, pp. 851–881, 2014.
  * [33] A. Guillaume, C. Vrain, and W. Elloumi, “Random dilated shapelet transform: A new approach for time series shapelets,” in _Pattern Recognition and Artificial Intelligence_. Springer International Publishing, 2022, pp. 653–664.
  * [34] P. Schäfer, “The boss is concerned with time series classification in the presence of noise,” _Data Mining and Knowledge Discovery_ , vol. 29, no. 6, pp. 1505–1530, 2015.
  * [35] P. Schäfer and U. Leser, “Fast and accurate time series classification with weasel,” in _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , 2017, pp. 637–646.
  * [36] M. Middlehurst, J. Large, G. Cawley, and A. Bagnall, “The temporal dictionary ensemble (tde) classifier for time series classification,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2020, pp. 660–676.
  * [37] P. Schäfer and U. Leser, “Weasel 2.0 – a random dilated dictionary transform for fast, accurate and memory constrained time series classification,” 2023.
  * [38] A. Dempster, F. Petitjean, and G. I. Webb, “Rocket: exceptionally fast and accurate time series classification using random convolutional kernels,” _Data Mining and Knowledge Discovery_ , vol. 34, pp. 1454–1495, 2020.
  * [39] A. Dempster, “A very fast (almost) deterministic transform for time series classification angus dempster, daniel f. schmidt, geoffrey i. webb,” _arXiv preprint arXiv:2012.08791_ , 2020.
  * [40] C. W. Tan, A. Dempster, C. Bergmeir, and G. I. Webb, “Multirocket: multiple pooling operators and transformations for fast and effective time series classification,” _Data Mining and Knowledge Discovery_ , pp. 1–24, 2022\.
  * [41] A. Dempster, D. F. Schmidt, and G. I. Webb, “Hydra: Competing convolutional kernels for fast and accurate time series classification,” 2022.
  * [42] H. I. Fawaz, B. Lucas, G. Forestier, C. Pelletier, D. F. Schmidt, J. Weber, G. I. Webb, L. Idoumghar, P.-A. Muller, and F. Petitjean, “InceptionTime: Finding AlexNet for time series classification,” _Data Mining and Knowledge Discovery_ , vol. 34, no. 6, pp. 1936–1962, 2020.
  * [43] Z. Wang, W. Yan, and T. Oates, “Time series classification from scratch with deep neural networks: A strong baseline,” in _2017 International joint conference on neural networks (IJCNN)_. IEEE, 2017, pp. 1578–1585.
  * [44] N. M. Foumani, L. Miller, C. W. Tan, G. I. Webb, G. Forestier, and M. Salehi, “Deep learning for time series classification and extrinsic regression: A current survey,” 2023.
  * [45] A. Ismail-Fawaz, M. Devanne, J. Weber, and G. Forestier, “Deep learning for time series classification using new hand-crafted convolution filters,” in _2022 IEEE International Conference on Big Data_ , 2022, pp. 972–981.
  * [46] J. Lines, S. Taylor, and A. Bagnall, “Time series classification with hive-cote: The hierarchical vote collective of transformation-based ensembles,” _ACM Transactions on Knowledge Discovery from Data_ , vol. 12, no. 5, 2018.
  * [47] A. Shifaz, C. Pelletier, F. Petitjean, and G. Webb, I, “Ts-chief: a scalable and accurate forest algorithm for time series classification,” _Data Mining and Knowledge Discovery_ , vol. 34, no. 3, pp. 742–775, 2020.
  * [48] R. Herbrich, T. Graepel, and K. Obermayer, “Support vector learning for ordinal regression,” in _Ninth International Conference on Artificial Neural Networks (ICANN99)_ , no. 470. IEE, 1999, pp. 97–102.
  * [49] W. Chu and S. Keerthi, “Support vector ordinal regression,” _Neural computation_ , vol. 19, pp. 792–815, 2007.
  * [50] D. Olsson, P. Georgiev, and P. M. Pardalos, “Kernel principal component analysis: Applications, implementation and comparison,” in _Models, Algorithms, and Technologies for Network Analysis_ , vol. 59, 2013, pp. 127–148.
  * [51] W. Chu and Z. Ghahramani, “Gaussian processes for ordinal regression,” _Journal of Machine Learning Research_ , vol. 6, pp. 1019–1041, 2005.
  * [52] P. K. Srijith, S. Shevade, and S. Sundararajan, “Validation based sparse gaussian processes for ordinal regression,” in _Neural Information Processing, ICONIP 2012_ , vol. 7664. IEEE, 2012, pp. 409–416.
  * [53] P. Srijith, S. Shevade, and S. Sundararajan, “A probabilistic least squares approach to ordinal regression,” in _AI 2012: Advances in Artificial Intelligence_. Springer, 2012, pp. 683–694.
  * [54] P. K. Srijith, S. Shevade, and S. Sundararajan, “Semi-supervised gaussian process ordinal regression,” in _Machine Learning and Knowledge Discovery in Databases_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 144–159.
  * [55] V. M. Vargas, P. A. Gutierrez, and C. Hervas-Martinez, “Cumulative link models for deep ordinal classification,” _Neurocomputing_ , vol. 401, pp. 48–58, 2020.
  * [56] V. M. Vargas, P. A. Gutiérrez, and C. Hervás-Martínez, “Deep ordinal classification based on the proportional odds model,” in _From Bioinspired Systems and Biomedical Applications to Machine Learning_ , vol. 11487, 2019, pp. 441–451.
  * [57] R. Rosati, L. Romeo, V. M. Vargas, P. A. Gutiérrez, C. Hervás-Martínez, and E. Frontoni, “A novel deep ordinal classification approach for aesthetic quality control classification,” _Neural Computing and Applications_ , 2022\.
  * [58] D. Guijo-Rubio, P. A. Gutierrez, A. Bagnall, and C. Hervas-Martinez, “Time series ordinal classification via shapelets,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020.
  * [59] D. Guijo-Rubio, P. A. Gutiérrez, A. Bagnall, and C. Hervás-Martínez, “Ordinal versus nominal time series classification,” in _International Workshop on Advanced Analytics and Learning on Temporal Data_. Springer, 2020, pp. 19–29.
  * [60] D. Guijo-Rubio, V. M. Vargas, P. A. Gutiérrez, and C. Hervás-Martínez, “Studying the effect of different lp norms in the context of time series ordinal classification,” in _Conference of the Spanish Association for Artificial Intelligence_. Springer, 2021, pp. 44–53.
  * [61] P. McCullagh, “Regression models for ordinal data,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 42, no. 2, pp. 109–127, 1980.
  * [62] D. Guijo-Rubio, M. Middlehurst, G. Arcencio, D. F. Silva, and A. Bagnall, “Unsupervised feature based algorithms for time series extrinsic regression,” _arXiv preprint arXiv:2305.01429_ , 2023.
  * [63] F. Pedregosa, F. Bach, and A. Gramfort, “On the consistency of ordinal regression methods,” _The Journal of Machine Learning Research_ , vol. 18, no. 1, pp. 1769–1803, 2017.
  * [64] P. A. Gutierrez, M. Perez-Ortiz, J. Sanchez-Monedero, F. Fernandez-Navarro, and C. Hervas-Martinez, “Ordinal regression methods: Survey and experimental study,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 28, no. 1, pp. 127–146, 2016.
  * [65] J. de La Torre, D. Puig, and A. Valls, “Weighted kappa loss function for multi-class classification of ordinal data in deep learning,” _Pattern Recognition Letters_ , vol. 105, pp. 144–154, 2018.
  * [66] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” _arXiv preprint arXiv:1803.01271_ , 2018.
  * [67] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [68] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep neural network ensembles for time series classification,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–6.
  * [69] D. R. Cox, “The regression analysis of binary sequences,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 20, no. 2, pp. 215–232, 1958.
  * [70] T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen _et al._ , “Xgboost: extreme gradient boosting,” _R package version 0.4-2_ , vol. 1, no. 4, pp. 1–4, 2015.
  * [71] A. Gómez-Orellana, D. Guijo-Rubio, P. Gutiérrez, and C. Hervás-Martínez, “Simultaneous short-term significant wave height and energy flux prediction using zonal multi-task evolutionary artificial neural networks,” _Renewable Energy_ , vol. 184, pp. 975–989, 2022.
  * [72] J. Cohen, “A coefficient of agreement for nominal scales,” _Educational and psychological measurement_ , vol. 20, no. 1, pp. 37–46, 1960.
  * [73] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” _Journal of Machine Learning Research_ , vol. 7, pp. 1–30, 2006.
  * [74] S. García and F. Herrera, “An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons,” _Journal of Machine Learning Research_ , vol. 9, pp. 2677–2694, 2008.
  * [75] A. Ismail-Fawaz, A. Dempster, C. W. Tan, M. Herrmann, L. Miller, D. F. Schmidt, S. Berretti, J. Weber, M. Devanne, G. Forestier, and G. I. Webb, “An approach to multiple comparison benchmark evaluations that is stable under manipulation of the comparate set,” 2023.

|  Rafael Ayllón-Gavilán received the BSc degree in Computer Science in 2021
and the MSc degree in Artificial Intelligence in 2022. He is currently
pursuing the PhD degree in Computer Science at the University of Córdoba. His
current research concerns the time series classification domain, addressing
the ordinal case together with its possible real-world applications.
---|---
|  David Guijo-Rubio received the PhD degree in Computer Science from the
University of Córdoba (UCO), Spain, in 2021. He is working as a Postdoctoral
Researcher at both UCO and University of East Anglia (UK). His current
interests include ordinal classification, pattern recognition and different
tasks applied to time series (classification -nominal and ordinal, clustering
and regression).
---|---
|  Pedro A. Gutiérrez (M’07-SM’15) received the PhD degree in Computer Science
and Artificial Intelligence from the University of Granada, Spain, in 2009. He
is currently a Full Professor of Computer Science at University of Córdoba,
Spain. His research interests are in the areas of supervised learning and
ordinal classification. He serves on the Editorial board of the IEEE
Transaction on Neural Networks and Learning Systems and on the organisation of
several computational intelligence conferences.
---|---
|  Anthony Bagnall received the PhD degree in Computer Science from the
University of East Anglia (UEA), UK in 2001. He is currently a Full Professor
of Computer Science at UEA, where he leads the time series machine learning
group. His primary research interest is in time series machine learning, with
a historic focus on classification, but more recently looking at clustering
and regression. He has a side interest in ensemble design.
---|---
|  César Hervás-Martínez (M’07-SM’14) received the PhD in Mathematics from the
University of Seville, Spain, in 1986. He is currently a Full Professor of
Computer Science at University of Córdoba, Spain, leading the AYRNA reseach
group. His current research interests include neural networks, evolutionary
computation, and the modelling of natural systems.
---|---