# United We Pretrain, Divided We Fail!
Representation Learning for Time Series by Pretraining on 75 Datasets at Once

Maurice Kraus  Felix Divo  David Steinmann  Devendra Singh Dhami  Kristian
Kersting

###### Abstract

In natural language processing and vision, pretraining is utilized to learn
effective representations. Unfortunately, the success of pretraining does not
easily carry over to time series due to potential mismatch between sources and
target. Actually, common belief is that multi-dataset pretraining does not
work for time series! Au contraire, we introduce a new self-supervised
contrastive pretraining approach to learn one encoding from many unlabeled and
diverse time series datasets, so that the single learned representation can
then be reused in several target domains for, say, classification.
Specifically, we propose the _XD-MixUp_ interpolation method and the _Soft
Interpolation Contextual Contrasting_ (SICC) loss. Empirically, this
outperforms both supervised training and other self-supervised pretraining
methods when finetuning on low-data regimes. This disproves the common belief:
We can actually learn from multiple time series datasets, even from 75 at
once.

machine learning, representation learning, pretraining, time series, multi-
dataset training

Arrow, FD-A, ECG, … XD-MixUpEncoder $\mathcal{F}_{\bm{\theta}}$Classifier
$\mathcal{C}_{2}$Classifier $\mathcal{C}_{1}$Classifier $\mathcal{C}_{3}$
Epilepsy Arrow Beef Figure 1:  The core idea of our method XIT is to learn a
single encoder from multiple datasets. The resulting representation can then
be used to train classifiers on datasets seen during the pretraining phase and
to be transferred to entirely new ones.

##  1 Introduction

The recent success of large language models (Devlin et al., 2019; Brown et
al., 2020) as well as diffusion models (Rombach et al., 2022) has shown that
leveraging vast amounts of text and image data can dramatically improve the
performance of deep learning models. This has led to impressive advancements
in several application areas, such as translation, interactive chat
assistants, and text-conditioned image generation. However, there is still a
need for a methodology to apply the same principles to the time domain, as
significant amounts of _unlabeled_ time domain data are available, yet they
are seldom used for successfully training models for different tasks. Most
classification systems on time series are purely supervised and, therefore,
still rely on expensive and complete labels per dataset (Shi et al., 2021; Nie
et al., 2023; Zeng et al., 2023). Overcoming this is crucial in contemporary
real-world situations, such as healthcare, where lack of labels is not the
only limitation. There is often an additional overall scarcity of sufficient
data points, for instance, due to privacy constraints. However, this clashes
with the requirements of current deep learning models, which typically require
large single-source datasets (Iwana & Uchida, 2021). Fortunately, multiple
small datasets exist that, in combination, can be leveraged even if they are
unlabeled. For instance, the UCR/UEA Time Series Classification Archive (Dau
et al., 2019) contains 57% of datasets with 300 or fewer training samples,
which are usually not enough to be applicable to the tasks. Additionally,
there are datasets with only unlabeled data, such as the M4 Competition
(Makridakis et al., 2018) or recordings of meteorological, financial,
industrial, traffic, and other signals. Combining them is a new perspective on
this collection of valuable data.

We address these challenges by utilizing transfer learning, and specifically,
by training a representation on unlabeled source datasets (Eldele et al.,
2020). As shown in Figure 1, the learned features are then used as a starting
point for finetuning on a target dataset with typically much fewer labeled
instances. In the natural language processing (NLP) and the vision domain, one
can leverage pretrained weights from models trained on a general dataset, even
in largely different domains. However, while multiple works have shown the
fundamental feasibility of pretraining on time series (Ma et al., 2023), the
source and target domain currently still need to be quite similar and follow
the same underlying temporal dynamics (Zhang et al., 2022) for it to succeed.
Thus, a similar generalized representation taking advantage of a diverse
collection of source datasets would prove very beneficial for use in several
target domains.

Contributions To this end, we make several important contributions:  (1) We
show how up to 75 unlabeled time series datasets can be combined effectively
into a single pretraining collection.  (2) We propose a novel interpolation
method called Cross-Dataset MixUp (XD-MixUp) that induces a shared latent
representation for multiple datasets.  (3) We propose the Soft Interpolation
Contextual Contrasting (SICC) loss function, which we combine with the Time
Series Temporal and Contextual Contrasting (TS-TCC) (Eldele et al., 2020)
framework using XD-MixUp. Overall, we call our new architecture
XIT111pronounced ¡¡exit¿¿ (XD-MixUp + SICC + Temporal Contrasting).  (4) We
demonstrate good transfer classification performance on multiple small labeled
target datasets without requiring extensive retraining for each. In
particular, we outperform supervised training and other self-supervised
pretraining methods.

Structure of the Paper We start with explaining our proposed XIT method,
including introducing the XD-MixUp and the SICC loss, before moving on to the
empirical demonstration of its efficacy. Before concluding, we present related
work.

##  2 Multi-Dataset Pretraining with XIT

In this section, we present our pretraining method XIT, where the overall goal
is to learn a $D$-dimensional latent representation
$\bm{z}_{i}\in\mathbb{R}^{D}$ of some time series
$\bm{x}_{i}\in\mathbb{R}^{T}$ of length $T$. For clarity, we focus on
univariate time series, while the method can be readily extended to
multivariate tasks. We train the parameters $\bm{\theta}$ of an encoder
$\mathcal{F}_{\bm{\theta}}(\cdot)$ to compress $\bm{x}_{i}$ into a more
abstract representation $\bm{z}_{i}=\mathcal{F}_{\bm{\theta}}(\bm{x}_{i})$.
That representation can subsequently be used for downstream tasks, such as
supervised training of a classifier. Note that in the pretraining phase, the
encoder $\mathcal{F}$ is trained in a self-supervised fashion without access
to any labels. We base our method on the work of Eldele et al. (2020) and
adapt their _TS-TCC_ framework to enable the training on multiple datasets.
While TS-TCC works well for its designed application to single-dataset
scenarios, it cannot leverage the availability of multiple datasets. On the
contrary, its performance severely drops in those settings, as we will show in
Section 3.1. Similar to their work, we start with a simple 1D convolutional
model with three layers as the encoder $\mathcal{F}$. As shown in Figure 2, we
merge a pair of time series through interpolation and then derive two
augmented variants to calculate two distinct pretraining losses, namely the
Temporal Contrastive (TC) and Soft Interpolation Contextual Contrastive (SICC)
loss, which we specifically designed to solve the challenges of multi-dataset
pretraining and thereby overcome the limitations of TS-TCC. Eventually, the
two losses are combined into a single training objective
$\mathcal{L}_{\text{Total}}$ and optimized jointly, yielding the XIT
architecture. The complete procedure to perform both pretraining and
subsequent finetuning is given in Algorithm 1.

Figure 2:  Our proposed XIT architecture. From two time series of a mini-
batch, we generate a randomly interpolated variant that gets augmented twice
and projected several times along with the original time series. Eventually,
we compute two losses to define the overall pretraining objective.  Algorithm
1 The XIT method consists of the pretraining and finetuning phases.

Input: Datasets for PT $D_{\text{PT}}$ (unlabeled) and FT $D_{\text{FT}}$
(labeled); batch size $B$; constants $\alpha$, $\beta$, and $\tau$; random
strong and weak augmentations $A_{s}$ and $A_{w}$; initialized models
$\mathcal{F}_{\bm{\theta}}$, $\mathcal{S}$, $g$, and $h$; optimizer
$O_{\text{PT}}$ over all four models; initialized classifier $\mathcal{C}$;
optimizer $O_{\text{FT}}$ over $\mathcal{C}$

Output: learned models $\mathcal{F}_{\bm{\theta}}$ and $\mathcal{C}$

if no parameters $\bm{\theta}$ of $\mathcal{F}$ are known then

# Pretraining phase (loop until convergence)

for mini-batch $\\{\bm{x}_{i}\\}_{i=1}^{B}\sim D_{\text{PT}}$ do

for all $i\in\\{1,\ldots,B\\}$ do

$\bm{z}_{i}\leftarrow\mathcal{F}_{\bm{\theta}}(\bm{x}_{i})$ # Projections

$\bm{\kappa}_{i}\leftarrow
h\left(\mathcal{S}\left(\bm{z}_{i}^{1:(K-1)}\right)\right)$

$\lambda_{i}\sim\operatorname{Beta}(\alpha,\alpha)$ # XD-MixUp

$j\leftarrow(i+1)\mod B$

$\bm{\widetilde{x}}_{i}\leftarrow\lambda_{i}\bm{x}_{i}+(1-\lambda_{i})\bm{x}_{j}$

$\bm{\widetilde{x}}_{i,s}\sim A_{s}(\bm{\widetilde{x}}_{i})$ # Strong
augmentation

$\bm{\widetilde{z}}_{i,s}\leftarrow\mathcal{F}_{\bm{\theta}}(\bm{\widetilde{x}}_{i,s})$

$\bm{\widetilde{c}}_{i,s}\leftarrow\mathcal{S}\left(\bm{\widetilde{z}}_{i,s}^{1:(K-1)}\right)$

$\bm{\kappa}_{i,s}\leftarrow h(\bm{\widetilde{c}}_{i,s})$

$\bm{\widetilde{x}}_{i,w}\sim A_{w}(\bm{\widetilde{x}}_{i})$ # Weak
augmentation

$\bm{\widetilde{z}}_{i,w}\leftarrow\mathcal{F}_{\bm{\theta}}(\bm{\widetilde{x}}_{i,w})$

$\bm{\widetilde{c}}_{i,w}\leftarrow\mathcal{S}\left(\bm{\widetilde{z}}_{i,w}^{1:(K-1)}\right)$

$\bm{\kappa}_{i,w}\leftarrow h(\bm{\widetilde{c}}_{i,w})$

# Compute the TC loss

$\mathcal{L}_{\text{TC}}^{s}=-\frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{g\left(\bm{\widetilde{c}}_{i,w},\bm{\widetilde{z}}_{i,s}^{K}\right)}{\sum_{j=1}^{B}g\left(\bm{\widetilde{c}}_{i,w},\bm{\widetilde{z}}_{j,s}^{K}\right)}\right)$

$\mathcal{L}_{\text{TC}}^{w}=-\frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{g\left(\bm{\widetilde{c}}_{i,s},\bm{\widetilde{z}}_{i,w}^{K}\right)}{\sum_{j=1}^{B}g\left(\bm{\widetilde{c}}_{i,s},\bm{\widetilde{z}}_{j,w}^{K}\right)}\right)$

$\mathcal{L}_{\text{TC}}\leftarrow\frac{1}{2}\left(\mathcal{L}_{\text{TC}}^{s}+\mathcal{L}_{\text{TC}}^{w}\right)$

# Compute the SICC loss

$\mathfrak{B}^{s}=\left(\bm{\kappa}_{1,l},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,l},\bm{\kappa}_{1,s},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,s},\bm{\kappa}_{1,r},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,r}\right)$

$\mathfrak{B}^{w}=\left(\bm{\kappa}_{1,l},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,l},\bm{\kappa}_{1,w},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,w},\bm{\kappa}_{1,r},\mathinner{\kern-1.00006pt\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp\kern-1.00006pt},\bm{\kappa}_{B,r}\right)$

Compute $\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{s}\right)$ and
$\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{w}\right)$ by Eq. 3

$\mathcal{L}_{\text{SICC}}\leftarrow\frac{1}{2}\left(\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{s}\right)+\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{w}\right)\right)$

$\mathcal{L}_{\text{Total}}\leftarrow\beta\mathcal{L}_{\text{TC}}+(1-\beta)\mathcal{L}_{\text{SICC}}$

Update all model parameters with $O_{\text{PT}}$ to minimize
$\mathcal{L}_{\text{Total}}$

Store learned parameters $\bm{\theta}$ of $\mathcal{F}$

# Finetuning phase (loop until convergence)

for mini-batch $\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{B}\sim D_{\text{FT}}$ do

$\bm{z}_{i}\leftarrow\mathcal{F}_{\bm{\theta}}(\bm{x}_{i})$

$\bm{\widehat{y}}_{i}\leftarrow\mathcal{C}(\bm{z}_{i})$ # Obtain class
probabilities

$\mathcal{L}_{\text{Class}}\leftarrow\operatorname{CE}(\bm{\widehat{y}}_{i},y_{i})$
# Use cross-entropy as criterion

Update model parameters with $O_{\text{FT}}$ to minimize
$\mathcal{L}_{\text{Class}}$

return learned composed model $\mathcal{F}_{\bm{\theta}}\circ\mathcal{C}$

###  2.1 XD-MixUp and Data Augmentation

We now construct a pretraining task to train the joint encoder $\mathcal{F}$.
Given two different time series $\bm{x}_{i}$ and $\bm{x}_{j}$, we first
generate a pointwise interpolated variant
$\bm{\widetilde{x}}_{i}\in\mathbb{R}^{T}$ by

$\bm{\widetilde{x}}_{i}=\lambda_{i}\bm{x}_{j}+(1-\lambda_{i})\bm{x}_{k},\hskip
7.11317pt\lambda_{i}\sim\operatorname{Beta}(\alpha,\alpha).$ (1)

Here, $\alpha>0$ is the shape parameter of the symmetric $\operatorname{Beta}$
distribution. Since $\lambda_{i}\in[0,1]$, $\bm{\widetilde{x}}_{i}$ is a
convex combination of $\bm{x}_{j}$ and $\bm{x}_{k}$. We will later show how to
select them appropriately. This approach is inspired by the label smoothing
method MixUp (Zhang et al., 2022), which is known to improve robustness to
outliers, training stability, and calibration (Thulasidasan et al., 2019) in
the image domain and has already been applied to time series by Wickstrøm et
al. (2022).

Besides the usual benefits of data augmentation, we use it to generate data
points between the different clusters of time series induced by training on
multiple datasets, typically each consisting of one or many sub-clusters. This
is especially relevant since the eventual downstream task might consist of
some time series not seen in the pre-training phase and, therefore, might lie
outside the clusters.

A mini-batch consists of randomly sampled time series from each of the
included datasets. We select the pairs for interpolation from a single mini-
batch of size $B$ on the fly while optimizing. We build distinct pairs by
interpolating between consecutive pairs of time series, like
$\left(\bm{x}_{1},\bm{x}_{2}\right),\left(\bm{x}_{2},\bm{x}_{3}\right),\dots,\left(\bm{x}_{B},\bm{x}_{1}\right)$
with independently sampled $\lambda_{i}$. We are then left with a batch of $B$
interpolated time series $\bm{\widetilde{x}}_{i}$, where
$i\in\\{1,\dots,B\\}$, in addition to the original time series. We
deliberately do not consider _all_ possible combinations within a mini-batch
to not dramatically increase the batch size, which would incur high
computational costs in the loss computations. Following TS-TCC, we then apply
a strong and a weak augmentation to our specifically crafted time series
$\bm{\widetilde{x}}_{i}$ to obtain $\bm{\widetilde{x}}_{i,s}$ and
$\bm{\widetilde{x}}_{i,w}$.

###  2.2 Pretraining Loss

The strongly and weakly augmented samples are then used to compute the loss
$\mathcal{L}_{\text{Total}}$, which consists of a weighted combination of the
temporal contrastive loss $\mathcal{L}_{\text{TC}}$ and the soft interpolation
contextual contrastive loss $\mathcal{L}_{\text{SICC}}$:

$\mathcal{L}_{\text{Total}}=\beta\mathcal{L}_{\text{TC}}+(1-\beta)\mathcal{L}_{\text{SICC}}.$
(2)

The weight $\beta\in[0,1]$ is determined by hyperparameter search. The TC loss
guarantees that a time series representation captures its unique, relevant
features and differentiates from others. The SICC loss in combination with
MixUp ensures meaningful connections between the various datasets seen during
pretraining by enforcing a well-structured latent space between them.

As shown in Figure 2, the next step after XD-MixUp is to encode each time
series $\bm{\widetilde{x}}_{i,s}$ into a sequence of $K$ embedding vectors
$\left(\bm{z}_{i,s}^{1},\dots,\bm{z}_{i,s}^{K}\right)=\bm{z}_{i,s}=\mathcal{F}(\bm{x}_{i,s},\bm{\theta})$
of size $Z$ each using the convolutional encoder $\mathcal{F}$, where
$\bm{z}_{i,s}^{k}\in\mathbb{R}^{Z}$. The time series
$\bm{\widetilde{x}}_{i,w}$ and the original $\bm{x}_{i}$ are encoded into
$\bm{\widetilde{z}}_{i,w}$ and $\bm{z}_{i}$, respectively. We then use a
learned summarization model $\mathcal{S}(\cdot)$ to condense the first $K-1$
embedding vectors $\bm{z}^{1:(K-1)}$ into a single context
$\bm{c}\in\mathbb{R}^{C}$, for all time series individually. Working with
these summary contexts instead of with the individual embedding vectors
greatly simplifies the computation of the pretraining losses and makes it more
efficient. In addition, it promotes learning more high-level features that can
compress all embedding vectors into smaller representations (Eldele et al.,
2020). We will now provide an in-depth explanation of both TC and SICC loss.

####  2.2.1 Temporal Contrasting

The TC loss (Eldele et al., 2020) is computed by solving a cross-forecasting
task. The context derived from the weakly augmented embedding
$\bm{\widetilde{c}}_{w}$ is used to predict the last embedding vector
$\bm{\widetilde{z}}_{s}^{K}$ of the strong embedding, and vice versa. This is
done using a similarity measure
$g\left(\bm{\widetilde{c}},\bm{\widetilde{z}}^{K}\right)=\exp\left(\bm{\widetilde{c}}^{T}\bm{W}\bm{\widetilde{z}}^{K}\right)$
with jointly learned $\bm{W}\in\mathbb{R}^{C\times Z}$. The task is to
maximize the similarity to the differently augmented time series embedding of
the same time series while minimizing the similarity to the other embeddings
from the mini-batch. This favors representations that are invariant to the
augmentations being applied. Overall, we compute the loss as follows:

$\displaystyle\mathcal{L}_{\text{TC}}^{s}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{g\left(\bm{\widetilde{c}}_{i,w},\bm{\widetilde{z}}_{i,s}^{K}\right)}{\sum_{j=1}^{B}g\left(\bm{\widetilde{c}}_{i,w},\bm{\widetilde{z}}_{j,s}^{K}\right)}\right)$
$\displaystyle\mathcal{L}_{\text{TC}}^{w}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{g\left(\bm{\widetilde{c}}_{i,s},\bm{\widetilde{z}}_{i,w}^{K}\right)}{\sum_{j=1}^{B}g\left(\bm{\widetilde{c}}_{i,s},\bm{\widetilde{z}}_{j,w}^{K}\right)}\right)$

We then average them to obtain
$\mathcal{L}_{\text{TC}}=\frac{1}{2}\left(\mathcal{L}_{\text{TC}}^{s}+\mathcal{L}_{\text{TC}}^{w}\right)$.
This loss is called InfoNCE (from Noise Contrastive Estimation) since van den
Oord et al. (2018) have shown that it optimizes a lower bound on the mutual
information $I(\bm{\widetilde{c}};\bm{\widetilde{z}})$ between the context
vectors $\bm{\widetilde{c}}$ and corresponding embeddings
$\bm{\widetilde{z}}$.

####  2.2.2 Soft Interpolation Contextual Contrasting

Our novel SICC loss aligns the information in the augmented time series
context vector to the non-augmented contexts. This ensures that the encoder
$\mathcal{F}$ remains invariant to the chosen augmentations, allowing it to
capture abstract concepts that can be applied to multiple datasets. We want
the context to contain the information of the source time series pair
$(\bm{x}_{i},\bm{x}_{j})$ used to form the interpolated time series
$\bm{\widetilde{x}_{i}}$ depending on the interpolation coefficient
$\lambda_{i}$. Namely, if $\lambda\approx 0$ then
$(\bm{\widetilde{x}_{i}},\bm{x}_{i})$ should be a positive pair and
$(\bm{\widetilde{x}_{i}},\bm{x}_{j})$ a negative pair. If $\lambda\approx 1$,
then the positive/negative relations switch. We, therefore, extend the normal
notion of _hard_ positive-negative examples to a _soft_ variant that treats
pairing within the interpolated time series group proportional to $\lambda$
and still considers the rest of the mini-batch to be hard negative samples.
Our approach thereby differs from the loss of Sohn (2016) and Chen et al.
(2020) used in TS-TCC, where the contextual alignment is solely performed
between the two augmented time series. Our approach allows learning
representations that are aligned across multiple datasets.

To enforce the entire embeddings $\bm{z}$ to be aligned, we directly use the
contexts $\bm{c}$ that we computed with the summarization model $\mathcal{S}$.
Since we want the information content of a positive pair to match but do not
require the concrete representation to be exactly the same, we further project
the contexts $\bm{c}$ using a two-layer learned $\operatorname{MLP}$,
obtaining
$\bm{\kappa}=h(\bm{c})=\operatorname{Linear}(\operatorname{ReLU}(\operatorname{BatchNorm1D}1D(\operatorname{Linear}(\bm{c}))))$.
Here, $\operatorname{Linear}(\bm{\xi})=\bm{W}\bm{\xi}+\bm{b}$ such that the
resulting vector has half the dimension of the input vector, i.e., that
$\bm{\kappa}\in\mathbb{R}^{C/4}$. Let
$(\bm{\kappa}_{i,l},\bm{\kappa}_{i,s},\bm{\kappa}_{i,r})$ with
$i\in\\{1,\dots,B\\}$ be the mini-batch of $B$ triples consisting of either
the strongly or weakly augmented time series projection $\bm{\kappa}_{i}^{s}$
or $\bm{\kappa}_{i}^{w}$ along with the projection of the left
$\bm{\kappa}_{i,l}$ and right time series $\bm{\kappa}_{i,r}$ that were
interpolated between with $\lambda_{i}$ to form the augmented one in Equation
1. We arrange these into two sets of vectors of length $3B$:

$\displaystyle\mathfrak{B}^{s}$
$\displaystyle=\left(\bm{\kappa}_{1,l},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,l},\bm{\kappa}_{1,s},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,s},\bm{\kappa}_{1,r},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,r}\right),$
$\displaystyle\mathfrak{B}^{w}$
$\displaystyle=\left(\bm{\kappa}_{1,l},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,l},\bm{\kappa}_{1,w},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,w},\bm{\kappa}_{1,r},\,\mathinner{\ldotp\kern-1.99997pt\ldotp\kern-1.99997pt\ldotp},\,\bm{\kappa}_{B,r}\right).$

The loss function is then computed as the average
$\mathcal{L}_{\text{SICC}}=\frac{1}{2}\left(\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{s}\right)+\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}^{w}\right)\right)$,
where the individual parts are defined as:

$\displaystyle\mathcal{L}_{\text{SICC}}\left(\mathfrak{B}\right)=\frac{1}{B}\sum_{i=1}^{B}\Big{[}$
$\displaystyle\ell(\mathfrak{B},i,B+i,1-\lambda_{i})+$ (3)
$\displaystyle\ell(\mathfrak{B},B+i,i,1-\lambda_{i})+$
$\displaystyle\ell(\mathfrak{B},B+i,2B+i,\lambda_{i})+$
$\displaystyle\ell(\mathfrak{B},2B+i,B+i,\lambda_{i})\;\Big{]},$
$\ell\left(\mathfrak{B},i,j,\mu\right)=-\log\left(\frac{\exp(\mu\operatorname{sim}(\mathfrak{B}_{i},\mathfrak{B}_{j}))}{\sum_{k=1}^{3B}\mathbbm{1}_{k\neq
i}\exp(\operatorname{sim}(\mathfrak{B}_{i},\mathfrak{B}_{k}))}\right).$ (4)

Here,
$\operatorname{sim}(\bm{x},\bm{x}^{\prime})=\frac{\bm{x}^{T}\bm{x}^{\prime}}{\tau\lVert\bm{x}\rVert_{2}\lVert\bm{x}^{\prime}\rVert_{2}}$
denotes the scaled cosine similarity with temperature parameter $\tau$,
$\mathbbm{1}$ is an indicator variable that is $1$ if $k\neq i$ and $0$
otherwise. In Equation 1, $\lambda_{i}$ scaled the _distance_ of
$\bm{\widetilde{x}}_{i}$ to $\bm{x}_{i}$ and $1-\lambda$ the distance to the
other time series, $\bm{x}_{j}$. Since we now want to scale _similarities_
proportionally, we have to reverse the roles of $\lambda$ and $1-\lambda$. In
Equation 4, since we minimize the negative of the fraction, we optimize for
maximizing the numerator (i.e., the similarity of the positive pair $(i,j)$)
and minimizing the denominator (i.e., the similarity of all other negative
pairs $(i,k)$). The computation of $\ell$ can be numerically stabilized using
the log-sum-exp trick.

We have now defined the three core components of our approach and how they
connect. First, we interpolate between datasets with XD-MixUp, to then apply
the TC and SICC losses. Together, they form the complete XIT pretraining
procedure, whose efficacy we now assess in the next section.

##  3 Experiments

After having laid out the motivation and formal foundation of XIT, we will
answer these key research questions with our experiments:  (Q1) Do multiple
datasets help to learn a more general representation that is easier to
transfer to seen and unseen datasets?  (Q2) Is an encoder pretrained in a
self-supervised fashion more helpful than directly learning a classifier,
especially for small target datasets?  (Q3) Which key components of our
proposed XIT procedure cause the improvements that we observe?  (Q4) How
discriminative are the learned representations regarding inter- and intra-
dataset structures?  The remainder of this section will provide an overview of
our experimental setup, with further details provided in Section A.1.
Subsequently, we will present the results and discoveries.

Datasets and Augmentations We compare our method with the baselines on a
diverse set of classification datasets from the large _UCR Time Series
Classification_ repository (Dau et al., 2019). At the time of writing, it
contained 128 datasets of a wide range of domains, time series lengths, sample
counts, and numbers of classes. We used all datasets of up to 600 time steps,
resulting in 100 datasets as shown in Table 3 in the Appendix. We did not
include longer ones to limit the discrepancy with shorter datasets, which
would contain increasingly more padding. In the case of multivariate time
series, we exclusively used the first variate for both pretraining and
classification. For a level comparison, we ensure that all time series
datasets are equal in length by prepending zeros. We use the same
augmentations as in the supplementary TS-TCC implementation: magnitude scaling
as weak augmentation and _permutation-and-jitter_ as strong augmentation.

Baselines We compare XIT against multiple baselines to evaluate its
effectiveness. To this end, we employed the following five self-supervised
pretraining methods:  (1) TS-TCC (Eldele et al., 2020), which uses temporal
and contextual contrasting between weakly and strongly augmented time series
to learn an embedding.  (2) TF-C (Zhang et al., 2022), which learns and aligns
a time and frequency domain encoding into a joint representation, again with
contrastive methods.  (3) TS2Vec (Yue et al., 2022), which performs masking
and contrasting hierarchically.  (4) TNC (Tonekaboni et al., 2020) aligns
neighboring windows with their respective latent representation.  (5) T-Loss
(Franceschi et al., 2019) utilizes time-based negative sampling in conjunction
with a triplet loss to learn an encoding.  Additionally, the _supervised_
model performs no pretraining, i.e., it uses a randomly initialized encoder
that is then used for finetuning.

Evaluation The primary objective of the evaluation is to determine the
effectiveness of the representations learned by $\mathcal{F}_{\bm{\theta}}$.
We therefore follow the _linear probing_ experiment of van den Oord et al.
(2018), i.e. learning a shallow neural classifier head $\mathcal{C}$ with a
single linear layer while the encoder $\mathcal{F}_{\bm{\theta}}$ is frozen.
The limited capacity of the classifier heads in linear probing allows for a
better evaluation of the underlying representations than more powerful
classifiers like SVMs or deep neural networks. We evaluate the classification
mainly with AUROC and macro-averaged F1 scores since the accuracy metric is
unreliable in the face of uneven class distributions (cf. Table 3 in the
appendix).

###  3.1 Self-supervised Pretraining on Multiple Datasets: (Q1) & (Q2)

To determine whether we can learn a single encoder on multiple datasets, we
pretrain time series classification models on large portions of the UCR
repository. Here, 100 datasets have a fixed sequence length of up to 600. We
pretrained on up to 75 different datasets, where we subsequently finetuned
both on all 100 datasets (Figure 3(a)) and 25 of the held-out ones (Figure
3(b)). We sample each domain with the same frequency (see Table 3 in the
Appendix). Within each domain, we sampled time series data points for training
according to the size of the respective origin datasets. We create three folds
of each dataset selection, where we each perform the pretraining and
finetuning steps with independently initialized models. We first observe that
in Figure 3(a), increasing the number of datasets in the pretraining phase
increases the benefit of pretraining over supervised training since the Macro
F1 transfer becomes increasingly positive. The effect is even more pronounced
when only transferring to entirely unseen datasets, where the increased
diversity aids in learning more general representations. In the case of
pretraining on 75 datasets, XIT outperforms direct supervised training in
64.7% of all datasets, while in the hold-out evaluation, this is still true
for an impressive 62.5% of datasets. In conclusion, XIT effectively learns
time series representations from multiple, diverse datasets.

(a) Finetuning on the 100 UCR datasets.

(b) Finetuning on a hold-out set of 25 datasets each.

Figure 3:  This box plot shows the transfer surplus over supervised training
measured in Macro F1 score difference after pretraining XIT on increasingly
large subsets of the UCR repository with three folds each. Higher is better.

To further position XIT among existing approaches, we compare it to five
popular baselines. We assess the performance on 25 difficult UCR datasets that
were not used for pretraining, in addition to the other 75 datasets. It is
crucial to emphasize, as the other methods have demonstrated limitations in
this context, that the conventional pretraining approaches do not work
effectively for this scenario. Table 1 shows the competitive and state-of-the-
art performance of XIT. For ease of comparison, we follow Demšar (2006) and
append ranks for each method. In general, we outperform the purely
constrastive methods (TS-TCC, TF-C, TNC, and T-Loss). Even when compared to
the mixed reconstructive-contrastive method TS2Vec, we outperform it overall.

These experiments show that—in contrast to the findings of Zhang et al.
(2022)—pretraining on more than one dataset is indeed feasible and beneficial
when compared to supervised given an appropriate model is employed.

Table 1:  This table shows the classification performance of XIT in comparison to baselines when pretrained on 75 datasets of the UCR repository. The shown Macro F1 scores were computed on three independent seeds and 25 datasets not seen during pretraining (similar to the hold-out in Figure 3(b)). Bold denotes the best model on each dataset, underlined the runner-up.  Target Dataset | XIT | TS-TCC | TF-C | TS2Vec | TNC | T-Loss
---|---|---|---|---|---|---
ArrowHead | 50.5 ±9.2 | 47.5 ±2.1 | 36.9 ±11.9 | 57.4 ±0.4 | 36.6 ±8.1 | 29.5 ±2.2
BeetleFly | 77.6 ±6.4 | 66.5 ±11.0 | 46.3 ±7.6 | 68.0 ±5.9 | 67.2 ±14.5 | 61.7 ±6.9
BirdChicken | 66.6 ±12.6 | 63.0 ±5.8 | 56.4 ±20.0 | 88.0 ±8.1 | 57.8 ±8.2 | 61.5 ±7.7
Car | 47.7 ±2.1 | 60.2 ±10.6 | 25.4 ±6.0 | 73.0 ±2.2 | 29.9 ±14.1 | 23.5 ±6.0
Crop | 53.7 ±0.7 | 53.7 ±1.2 | 47.1 ±0.8 | 40.4 ±2.5 | 19.6 ±9.8 | 14.1 ±1.1
Fish | 59.4 ±1.9 | 55.0 ±7.9 | 31.8 ±6.9 | 44.6 ±7.3 | 12.5 ±6.3 | 08.5 ±4.6
FreezerRegularTrain | 77.2 ±0.3 | 76.2 ±0.3 | 76.9 ±1.2 | 76.9 ±0.5 | 73.1 ±8.0 | 58.3 ±3.0
GunPoint | 72.1 ±6.3 | 78.9 ±2.1 | 58.7 ±17.0 | 94.4 ±2.8 | 62.4 ±22.4 | 64.6 ±12.0
GunPointMaleV.Female | 92.7 ±2.2 | 95.7 ±3.7 | 76.2 ±4.8 | 86.7 ±3.4 | 75.5 ±8.0 | 74.3 ±3.8
Lightning7 | 65.9 ±1.2 | 50.5 ±3.1 | 17.5 ±3.6 | 78.8 ±4.7 | 35.6 ±17.2 | 37.3 ±2.7
MedicalImages | 22.2 ±1.0 | 06.8 ±0.0 | 12.2 ±3.0 | 06.8 ±0.0 | 09.7 ±4.3 | 08.2 ±2.5
MiddlePh.O.AgeGroup | 31.8 ±4.7 | 20.2 ±7.3 | 39.6 ±1.8 | 21.1 ±2.3 | 23.9 ±13.1 | 25.4 ±14.9
MiddlePh.O.Correct | 36.3 ±0.0 | 36.3 ±0.0 | 36.3 ±0.0 | 36.3 ±0.0 | 36.3 ±0.0 | 36.3 ±0.0
MiddlePhalanxTW | 19.9 ±3.7 | 14.3 ±2.3 | 24.3 ±2.0 | 15.6 ±0.7 | 14.3 ±7.4 | 14.6 ±8.1
OSULeaf | 41.9 ±0.6 | 41.7 ±3.0 | 34.1 ±2.8 | 46.1 ±0.6 | 14.6 ±6.6 | 12.3 ±2.0
ProximalPh.O.Correct | 41.0 ±0.7 | 41.0 ±0.7 | 46.8 ±6.2 | 40.6 ±0.0 | 41.4 ±1.8 | 40.6 ±0.0
ShapesAll | 70.9 ±0.2 | 66.4 ±0.8 | 28.4 ±6.9 | 50.1 ±3.9 | 08.0 ±5.5 | 03.0 ±0.9
SonyAIBORobotSur.1 | 30.3 ±0.2 | 30.0 ±0.0 | 76.0 ±10.3 | 38.1 ±4.7 | 47.8 ±13.1 | 45.7 ±14.0
SonyAIBORobotSur.2 | 51.6 ±11.8 | 50.9 ±11.6 | 86.2 ±1.1 | 70.4 ±4.2 | 63.7 ±5.7 | 65.4 ±1.3
Symbols | 72.0 ±0.5 | 74.7 ±1.3 | 36.7 ±3.0 | 90.7 ±1.9 | 45.1 ±19.5 | 53.6 ±16.7
Tiselac | 29.3 ±0.3 | 26.3 ±0.7 | 19.4 ±0.5 | 18.5 ±1.2 | 15.3 ±2.7 | 12.3 ±0.2
ToeSegmentation2 | 69.6 ±7.4 | 74.3 ±9.6 | 46.9 ±18.4 | 76.0 ±3.0 | 56.3 ±8.4 | 57.8 ±0.3
UWaveGestureLib.X | 77.1 ±0.3 | 69.7 ±0.5 | 55.1 ±5.7 | 64.1 ±0.2 | 32.3 ±10.6 | 21.1 ±2.6
UWaveGestureLib.Z | 68.3 ±1.3 | 61.3 ±1.9 | 51.4 ±3.2 | 55.5 ±1.1 | 27.1 ±11.8 | 22.9 ±3.1
WordSynonyms | 35.1 ±4.4 | 17.3 ±0.6 | 06.6 ±1.1 | 03.3 ±0.5 | 04.0 ±1.3 | 03.1 ±1.7
Average Macro F1 $\uparrow$ | 54.4 ±3.2 | 51.1 ±3.5 | 42.9 ±5.8 | 53.7 ±2.5 | 36.4 ±9.1 | 34.2 ±4.7
Rank $\downarrow$ | 2.10 ±1.2 | 3.06 ±1.6 | 3.56 ±1.8 | 2.82 ±1.6 | 4.48 ±1.0 | 4.98 ±1.1

###  3.2 Ablation Studies: (Q3)

Table 2:  The ablation studies we performed to determine the key components driving the increase in performance observed earlier (Q3). We followed the same evaluation as in Table 1. Lower ranks are better, bold denotes best. The results demonstrate that each of the key components, XD-MixUp, TC loss, and SICC loss, contribute to the overall performance of XIT.  Pretraining Component | AUROC rank $\downarrow$ | Accuracy rank $\downarrow$ | Macro F1 rank $\downarrow$
---|---|---|---
XD-MixUp + SICC + TC (XIT) | 1.800 ±0.91 | 1.780 ±0.89 | 1.780 ±0.84
XD-MixUp + SICC | 3.560 ±0.87 | 3.700 ±0.56 | 3.580 ±0.79
XD-MixUp + TC | 2.060 ±0.94 | 1.920 ±0.84 | 1.980 ±0.91
TC | 2.580 ±0.91 | 2.600 ±0.78 | 2.660 ±0.81

We conduct a series of ablation experiments to determine if the observed
effects were due to the key components of XIT. We pretrained on the same three
folds of 75 UCR datasets and finetuned on all 100 datasets. We ablated by
systematically omitting our method’s three key components XD-MixUp, TC loss,
and SICC loss. Note that SICC is inherently linked to the presence of XD-
Mixup, so SICC in isolation is not feasible. Similarly, only XD-MixUp without
a loss is not a valid training procedure. The results are presented in Table
2. Our method XIT is superior to any ablated models when looking at the ranks
for accuracy, AUROC, and Macro F1 scores. In particular, the high AUROC score
means we have a high overall probability of correctly classifying classes,
whereas the high Macro F1 score shows we can also correctly classify
underrepresented classes. Therefore, we conclude that all three building
blocks of our combined pretraining procedure XIT are relevant for the
performance increases observed in the previous section.

###  3.3 Inspecting the Learned Representations: (Q4)

To gain insight into the effects of our proposed method, we examined the
structure of multiple embedded time series datasets. We achieve a mean Davies-
Bouldin Index (DBI) (Davies & Bouldin, 1979) of 7.63 when embedded with a
newly initialized encoder. After pretraining on 75 datasets, this score
improved to 7.32. This relatively small decrease suggests that our method does
not need to alter the overall structure drastically but rather rearranges the
intra-dataset structure. This effect can be seen qualitatively in Figure 4,
which shows a projection of the latent representation learned by XIT of
multiple unseen datasets. The figure was generated by performing a reduction
to two dimensions via Principal Component Analysis on the output of the
encoder $\mathcal{F}$. We observe a major structuring effect without any
finetuning or the introduction of any labels. This demonstrates that in the
majority of cases, the pretraining losses induce beneficial structure in the
latent space, which is consistent with the results of Figure 3.

Figure 4: Low-dimensional embedding visualization from a single pretrained XIT
model trained on 75 UCR datasets, evaluated on three hold-out datasets.
Furthermore, we include the DBI to support our visual observation further.
Lower DBI $\rightarrow$ more separated clusters.

##  4 Related Work

Pretraining is a key element of current deep learning, allowing state-of-the-
art outcomes in areas with scarce labels or data by utilizing a shared
representation as the basis for adapting to the target domain. Although it has
been extensively studied for domains such as natural language processing and
computer vision, it still remains a challenge for the time series domain (Ma
et al., 2023). We can generally differentiate between supervised,
unsupervised, and self-supervised pretraining. The core idea for the former is
to utilize labels to steer the representation learning, while the latter two
approaches work without any labels. Due to the wide availability of unlabeled
time series datasets, we will focus on self-supervised methods in this paper.

Self-Supervised Time Series Pretraining Several methods have been proposed to
pretrain models on unlabeled datasets. The three main types of losses to
optimize are reconstruction, pseudo-labels, and contrastive methods (Ma et
al., 2023). The SimMTM framework (Dong et al., 2023) reconstructs time series
from multiple masked variants and series-wise similarities within and across
domains. Further models based on reconstructions are Ti-MAE Li et al. (2023)
and its extension TimeMAE Cheng et al. (2023). Contrastive methods, on the
other hand, need means to generate view pairs, e.g., as proposed in LEAVES (Yu
et al., 2022), by Tang et al. (2020), or in PAITS (Beebe-Wang et al., 2023).
Shi et al. (2021) learn long-term dependencies using Dynamic Time Warping
(DTW) (Sakoe & Chiba, 1978). Kiyasseh et al. (2021) specifically apply their
contrastive learning method CLOCS to ECG signals. In TS2Vec by Yue et al.
(2022), instance-wise and temporal hierarchical contrasting is used to capture
multiscale contextual information and dynamics. TS-TCC (Eldele et al., 2020)
is a pretraining framework involving two views generated by weak and strong
augmentations, which we also use in XIT. This is then fed into a temporal and
contextual contrasting module using the NTXent loss (Sohn, 2016) for learning
robust and discriminative representations. Furthermore, the authors indicate
that it is effective in transfer learning scenarios with few-labeled targets.
To use spectral information in contrasting, Zhang et al. (2022) developed
TF-C, allowing the model to align the time and frequency domains with the
respective views. Wickstrøm et al. (2022) propose MixUp

(Zhang et al., 2018) for the time domain, where two samples are combined by a
sampled parameter $\lambda$, which is predicted as a pseudo-label.
Furthermore, the two mixed views are aligned via the same contrastive NTXent
loss used in TS2Vec, TS-TCC, TF-C, and our method XIT. Tonekaboni et al.
(2020) propose TNC, utilizing time series windows where ones with close-
proximity share similar latent representations. In addition, (Franceschi et
al., 2019) propose T-Loss, which employs time-based negative sampling along
with a triplet loss to learn an encoding.

Multi-Dataset Pretraining While it is common to pretrain on a single source
dataset, there is very little research in the area of multi-dataset
pretraining, especially in the context of time series classification (Ma et
al., 2023). This arises from the fact that applying multiple datasets in a
suboptimal setting may drastically decrease the performance (Zhang et al.,
2022), leading to a so-called _negative transfer_. Other works (Gikunda &
Jouandeau, 2021; Tseng et al., 2023; Brüsch et al., 2023) leverage multiple
datasets in homogeneous settings where source and target distributions match.
Some works like OFA (Zhou et al., 2023), Time-LLM (Jin et al., 2023), and
LLMTime (Gruver et al., 2023) leverage existing large and diverse text
datasets, however, doing so at the expense of using extremely resource-hungry
large language models. As far as we are aware, Kashiparekh et al. (2019) and
Zhang et al. (2022) are the only works that investigate proper multi-dataset
pretraining. However, the former applies it in a supervised and very
inflexible way, using one encoding head per source dataset, and the latter
reports significant challenges when applying their method TF-C to multiple
datasets at once, which they call _many-to-one_ setting. They note a clear
drop in performance when increasing the number of datasets from one to two,
three, and four.

##  5 Conclusion & Future Work

Our research presents a paradigm shift in time series pretraining. Contrary to
prevailing beliefs, our findings illustrate the possibility and effectiveness
of multi-dataset pretraining for time series. By introducing XIT, consisting
of XD-MixUp along with the SICC and TC losses, we have carved a promising path
in self-supervised contrastive pretraining for time series. Our empirical
evaluations showcased the efficacy of this method, especially in low-data
regimes, against both supervised training and other self-supervised
pretraining techniques. In essence, not only have we debunked the myth that
multi-dataset pretraining is infeasible for time series, but we have also
opened the door for further advancements in leveraging multiple
datasets—beyond simultaneously using 75 datasets.

While our study has advanced time series pretraining, several promising
directions beckon further exploration. The versatility of our approach needs
evaluation on further tasks like forecasting and anomaly detection. We are
eager to explore model reprogramming (Yang et al., 2021) to enhance
adaptability and further decrease negative transfer. While we utilized MixUp
augmentation, exploring specialized interpolations like DTW may yield further
insights. Furthermore, we want to explore the potential of our SICC loss and
integrated interpolation mechanism in other time series models and different
modalities. Much like in NLP, future work might consider creating compound
datasets with special attention to the types and distribution of the contained
data.

## Acknowledgments

This work received funding by the EU project EXPLAIN, funded by the German
Federal Ministry of Education and Research (BMBF) (grant 01—S22030D). It was
further funded by the BMBF within the “The Future of Value Creation – Research
on Production, Services and Work” program (grant 02L19C150) managed by the
Project Management Agency Karlsruhe (PTKA). Additionally, it was funded by the
projects “The Adaptive Mind” and “The Third Wave of Artificial Intelligence –
3AI” from the Hessian Ministry of Science and the Arts (HMWK), the “ML2MT”
project from the Volkswagen Stiftung, the ACATIS Investment KVG mbH project
“Temporal Machine Learning for Long-Term Value Investing”, the ICT-48 Network
of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA No 952215), and
the Collaboration Lab with Nexplore “AI in Construction” (AICO). The authors
are responsible for the content of this publication.

## References

  * Akiba et al. (2019) Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M.  Optuna: A Next-generation Hyperparameter Optimization Framework.  In _Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)_ , 2019.
  * Beebe-Wang et al. (2023) Beebe-Wang, N., Ebrahimi, S., Yoon, J., Arik, S. O., and Pfister, T.  PAITS: Pretraining and augmentation for irregularly-sampled time series.  _ArXiv:2308.13703_ , 2023.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language models are few-shot learners.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., and Lin, H.-T. (eds.), _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Brüsch et al. (2023) Brüsch, T., Schmidt, M. N., and Alstrøm, T. S.  Multi-view self-supervised learning for multivariate variable-channel time series.  _ArXiv:2307.09614_ , 2023.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Cheng et al. (2023) Cheng, M., Liu, Q., Liu, Z., Zhang, H., Zhang, R., and Chen, E.  TimeMAE: Self-supervised representations of time series with decoupled masked autoencoders.  _ArXiv:2303.00320_ , 2023.
  * Dau et al. (2019) Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., and Keogh, E.  The UCR time series archive.  _IEEE/CAA Journal Automatica Sinica (JAS)_ , 2019.
  * Davies & Bouldin (1979) Davies, D. L. and Bouldin, D. W.  A Cluster Separation Measure.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 1979.
  * Demšar (2006) Demšar, J.  Statistical Comparisons of Classifiers over Multiple Data Sets.  _Journal of Machine Learning Research (JMLR)_ , 2006.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Burstein, J., Doran, C., and Solorio, T. (eds.), _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019.
  * Dong et al. (2023) Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., and Long, M.  SimMTM: A simple pre-training framework for masked time-series modeling.  _ArXiv:2302.00861_ , 2023.
  * Eldele et al. (2020) Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., and Guan, C.  Time-Series Representation Learning via Temporal and Contextual Contrasting.  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2020.
  * Franceschi et al. (2019) Franceschi, J.-Y., Dieuleveut, A., and Jaggi, M.  Unsupervised scalable representation learning for multivariate time series.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Gikunda & Jouandeau (2021) Gikunda, P. and Jouandeau, N.  Homogeneous Transfer Active Learning for Time Series Classification.  In _Proceedings of the International Conference on Machine Learning and Applications (ICMLA)_ , 2021.
  * Gruver et al. (2023) Gruver, N., Finzi, M. A., Qiu, S., and Wilson, A. G.  Large Language Models Are Zero-Shot Time Series Forecasters.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2023.
  * Iwana & Uchida (2021) Iwana, B. K. and Uchida, S.  An empirical survey of data augmentation for time series classification with neural networks.  _PLOS ONE_ , 2021.
  * Jin et al. (2023) Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., and Wen, Q.  Time-LLM: Time Series Forecasting by Reprogramming Large Language Models.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2023.
  * Kashiparekh et al. (2019) Kashiparekh, K., Narwariya, J., Malhotra, P., Vig, L., and Shroff, G.  ConvTimeNet: A pre-trained deep convolutional neural network for time series classification.  In _Proceedings of the International Joint Conference on Neural Networks (IJCNN)_ , 2019.
  * Kiyasseh et al. (2021) Kiyasseh, D., Zhu, T., and Clifton, D. A.  CLOCS: Contrastive learning of cardiac signals across space, time, and patients.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Li et al. (2023) Li, Z., Rao, Z., Pan, L., Wang, P., and Xu, Z.  Ti-MAE: Self-supervised masked time series autoencoders.  _ArXiv:2301.08871_ , 2023.
  * Ma et al. (2023) Ma, Q., Liu, Z., Zheng, Z., Huang, Z., Zhu, S., Yu, Z., and Kwok, J. T.  A survey on time-series pre-trained models.  _ArXiv:2305.10716_ , 2023.
  * Makridakis et al. (2018) Makridakis, S., Spiliotis, E., and Assimakopoulos, V.  The M4 Competition: Results, findings, conclusion and way forward.  _International Journal of Forecasting (IJF)_ , 2018.
  * Nie et al. (2023) Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J.  A time series is worth 64 words: Long-term forecasting with transformers.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2023.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-Resolution Image Synthesis with Latent Diffusion Models.  In _Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Sakoe & Chiba (1978) Sakoe, H. and Chiba, S.  Dynamic programming algorithm optimization for spoken word recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)_ , 1978.
  * Shi et al. (2021) Shi, P., Ye, W., and Qin, Z.  Self-Supervised Pre-training for Time Series Classification.  In _Proceedings of the International Joint Conference on Neural Networks (IJCNN)_ , 2021.
  * Sohn (2016) Sohn, K.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2016.
  * Tang et al. (2020) Tang, C. I., Perez-Pozuelo, I., Spathis, D., and Mascolo, C.  Exploring contrastive learning in human activity recognition for healthcare.  _ArXiv:2011.11542_ , 2020.
  * Thulasidasan et al. (2019) Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., and Michalak, S.  On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Tonekaboni et al. (2020) Tonekaboni, S., Eytan, D., and Goldenberg, A.  Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Tseng et al. (2023) Tseng, G., Zvonkov, I., Purohit, M., Rolnick, D., and Kerner, H.  Lightweight, pre-trained transformers for remote sensing timeseries.  _ArXiv:2304.14065_ , 2023.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _ArXiv:1807.03748_ , 2018.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is All you Need.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Wang et al. (2017) Wang, Z., Yan, W., and Oates, T.  Time series classification from scratch with deep neural networks: A strong baseline.  In _Proceedings of the International Joint Conference on Neural Networks (IJCNN)_ , 2017.
  * Wickstrøm et al. (2022) Wickstrøm, K., Kampffmeyer, M., Mikalsen, K., and Jenssen, R.  Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series.  _Pattern Recognition Letters_ , 2022.
  * Yang et al. (2021) Yang, C.-H. H., Tsai, Y.-Y., and Chen, P.-Y.  Voice2Series: Reprogramming Acoustic Models for Time Series Classification.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Yu et al. (2022) Yu, H., Yang, H., and Sano, A.  LEAVES: Learning views for time-series data in contrastive learning.  _ArXiv:2210.07340_ , 2022.
  * Yue et al. (2022) Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., and Xu, B.  TS2Vec: Towards Universal Representation of Time Series.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2022.
  * Zeng et al. (2023) Zeng, A., Chen, M., Zhang, L., and Xu, Q.  Are Transformers Effective for Time Series Forecasting?  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2023.
  * Zhang et al. (2018) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond Empirical Risk Minimization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * Zhang et al. (2022) Zhang, X., Zhao, Z., Tsiligkaridis, T., and Zitnik, M.  Self-supervised contrastive pre-training for time series via time-frequency consistency.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Zhou et al. (2023) Zhou, T., Niu, P., Wang, X., Sun, L., and Jin, R.  One Fits All: Power General Time Series Analysis by Pretrained LM.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2023.

##  Appendix A Appendix

###  A.1 Experimental Details

This section gives more details for the experimental evaluation that should
aid in reproducing our results. Table 3 compares the datasets in the UCR
repository we used, which are freely available from
https://timeseriesclassification.com.

Table 3: The distribution of datasets from the UCR repository used in our experiments. Domain | Datasets | Sequence Length | Train Size | Balanced
---|---|---|---|---
|  | min | mean | max | min | median | max |
Audio | 2 | 270 | 337.5 | 405 | 60 | 132 | 204 | 50.0%
Device | 2 | 96 | 120.0 | 144 | 180 | 4553 | 8926 | 50.0%
Ecg | 4 | 82 | 113.5 | 140 | 23 | 61 | 500 | 50.0%
Eeg | 5 | 50 | 230.0 | 510 | 56 | 316 | 5890 | 40.0%
Har | 9 | 30 | 127.2 | 315 | 30 | 151 | 2238 | 55.6%
Image | 30 | 23 | 226.1 | 512 | 16 | 399 | 81714 | 20.0%
Meg | 1 | 200 | 200.0 | 200 | 727 | 727 | 727 | 0.0%
Motion | 14 | 8 | 229.8 | 343 | 36 | 332 | 7494 | 57.1%
Other | 2 | 36 | 118.5 | 201 | 18 | 1238 | 2459 | 0.0%
Sensor | 14 | 24 | 273.1 | 577 | 20 | 85 | 3636 | 42.9%
Simulated | 8 | 15 | 159.4 | 500 | 20 | 93 | 1000 | 75.0%
Sound | 1 | 217 | 217.0 | 217 | 3315 | 3315 | 3315 | 100.0%
Spectro | 7 | 234 | 382.0 | 570 | 28 | 57 | 613 | 28.6%
Traffic | 1 | 24 | 24.0 | 24 | 20 | 20 | 20 | 0.0%
Total | 100 | 8 | 221.1 | 577 | 16 | 180 | 81714 | 40.0%

##### Implementation details

We used _PyTorch_ (Paszke et al., 2019) version 2.0 as the base framework for
all models. We performed almost all training in 16-bit mixed precision to save
resources. For TS2Vec, we stayed very close to the reference implementation
and therefore trained in 32-bit precision, used the existing hyperparameters
for pretraining (see Table 4), and no early stopping in finetuning. Due to
initially poor results with both TNC and T-Loss, we searched for optimal
learning rates in $\\{10^{-3},10^{-4},10^{-5}\\}$ (results are in Table 4). We
used early stopping after four training epochs without improvements in the
AUROC score for all other finetuning/supervised experiments. We ensured that
we trained for at least 40 steps before stopping and up to a maximum of 2000
steps. The Adam optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ was used
to optimize all models with the hyperparameters given in Table 4 and gradient
clipping to $1.0$. For the special case of the disproportionally large
_Tiselac_ dataset, we increased the batch size to 256 for faster completion.
For the data interpolation in Equation 1, we set $\alpha=0.2$ as determined by
a hyperparameter search. In Equation 2 we set $\beta=0.25$ according to our
hyperparameter search, effectively giving three times more weight to
$\mathcal{L}_{\text{SICC}}$ than to $\mathcal{L}_{\text{TC}}$. In $h(\bm{c})$,
$\operatorname{BatchNorm1D}1D$ normalizes the vectors per dimension by
subtracting the mean and dividing by the empirical standard deviation. We used
the default PyTorch configuration, maintaining a running average of the
elements within the mini-batches. We set the temperature $\tau$ in Equation 4
to 0.2. We based our implementation of the baselines on the official
repositories of:

  * •

TS-TCC (https://github.com/emadeldeen24/TS-TCC),

  * •

TF-C (https://github.com/mims-harvard/tfc-pretraining),

  * •

TNC (https://github.com/sanatonek/TNC_representation_learning),

  * •

T-Loss (https://github.com/White-
Link/UnsupervisedScalableRepresentationLearning
TimeSeries), and

  * •

TS2Vec (https://github.com/yuezhihan/ts2vec), respectively.

Table 4: This table shows the differing hyperparameters we chose by search using _Optuna_ (Akiba et al., 2019). In particular, we also considered larger batch sizes of up to 1,024 for the pretraining phase but did not find them to be beneficial, much like in the works of TS-TCC and TF-C. | Pretraining | Finetuning
---|---|---
Model | Batch size | LR | Weight decay | Batch size | LR | Weight decay
XIT | 64 | 0.0001 | 0.0003 | 64 | 0.00014 | 0.0016
TS-TCC | 128 | 0.0003 | 0.0003 | 64 | 0.00014 | 0.0016
TF-C | 64 | 0.0003 | 0.0005 | 64 | 0.0003 | 0.0003
TS2Vec | 16 | 0.001 | 0.0005 | 64 | 0.00014 | 0.0016
TNC | 64 | 0.001 | 0.0005 | 64 | 0.00014 | 0.0016
T-Loss | 20 | 0.001 | 0.0005 | 64 | 0.00014 | 0.0016
Supervised | – | – | – | 64 | 0.00014 | 0.0016

Encoder & Summarization Model We used a simple and efficient encoder
$\mathcal{F}$ with three residual convolution layers and configured it as in
the work of Wang et al. (2017) and TS-TCC. For the model $\mathcal{S}$ used to
obtain the context vectors $\bm{c}$, we used the exact same transformer model
configuration as in TS-TCC: A token dimension of $64$ obtained from a linear
projection with bias term, multi-head attention with four heads, and a total
of four pre-norm transformer layers. The feedforward $\operatorname{MLP}$s
each consist of two layers with hidden dimensions $64$, with a
$\operatorname{ReLU}$ activation and subsequent dropout layer in between and a
final dropout layer at the end. Both dropout probabilities were set to $10\%$.
We employed a transformer model (Vaswani et al., 2017) $\mathcal{S}$ to
calculate the context vectors $\bm{c}$, similar to TS-TCC. We observed in the
supplementary implementation that Eldele et al. (2020) did not include
positional encodings in their transformer tokens and thus used a set model
instead of a sequence model. Nevertheless, we decided not to use it either
since adding a sine-cosine positional encoding (Vaswani et al., 2017) did not
noticeably affect the results.

Finetuning To evaluate the utility of the learned representation for
classification, we trained simple classifiers $\mathcal{C}$ on top of all
encoder models $\mathcal{F}_{\bm{\theta}}$. This is embedded into the complete
procedure as shown in Algorithm 1. We perform linear probing (van den Oord et
al., 2018), i.e., learning a single linear layer classifier $\mathcal{C}$ on
top of the embeddings learned by $\mathcal{F}_{\bm{\theta}}$. The final output
is transformed with a softmax, and the training criterion is cross-entropy.
The hyperparameters for finetuning are given in Table 4.

##### Details on TF-C

We changed the frequency transformation via the FFT to be orthonormal by
scaling the result of length $T$ by $\sqrt{T}$. This preserves the signal’s
magnitude, allowing us to perform training and inference in 16-bit mixed
precision without numerical issues. Furthermore, we use the complete
pretraining datasets instead of only subsets for the N-to-one settings (Zhang
et al., 2022, Appendix K). We use mostly the same hyperparameters as in
Appendix E when pretraining TF-C. However, we deviate slightly to follow the
linear probing evaluation. This means that in finetuning, we only train a
single-layer classifier head instead of a deeper MLP. We only optimize the
classifier and classifier loss instead of training the encoder as well or
optimizing the pretraining and classifier losses jointly. We use the very same
encoder config for all datasets for a more direct comparison, especially in
the multi-dataset experiments.

###  A.2 Reproducibility

We acknowledge the significance of reproducibility in scientific research and
have taken multiple steps to ensure the strength and replicability of our
work.

  * •

Code: We have used publicly available software and libraries to guarantee
accessibility and have comprehensively described the architecture, software,
versions, and hyperparameters in the Section A.1. Our code is deterministic,
incorporating seeds for all random number generators to guarantee the
replicability of results. Furthermore, we will make our implementation
available in a public repository at a later point in time for other
researchers to access, review, and build upon.

  * •

Datasets: This study only utilizes publicly available datasets that have been
correctly cited. Furthermore, the authors contribute to an open-source
repository containing all the datasets used in this work, which will be made
available upon acceptance.

  * •

Architecture and Algorithm Details: We have provided thorough descriptions and
formulations of our architecture in the main text, supplemented by additional
clarifications and implementation details in the Section A.1, ensuring a clear
understanding of our contributions and facilitating reproduction. This
documentation is intended to provide researchers with all the necessary
information to replicate our experiments accurately.