# Simple Yet Surprisingly Effective Training Strategies for LSTMs in Sensor-
Based Human Activity Recognition

Shuai Shao  Department of Computer Science
University of Warwick
Coventry, UK
<EMAIL_ADDRESS>Yu Guan  Department of Computer Science
University of Warwick
Coventry, UK
<EMAIL_ADDRESS>Xin Guan  School of Computing
Newcastle University
Newcastle upon Tyne, UK
<EMAIL_ADDRESS>Paolo Missier  School of Computing
Newcastle University
Newcastle upon Tyne, UK
<EMAIL_ADDRESS>Thomas Ploetz  School of Interactive Computing
Georgia Institute of Technology
Atlanta, USA
<EMAIL_ADDRESS>
###### Abstract

Human Activity Recognition (HAR) is one of the core research areas in mobile
and wearable computing. With the application of deep learning (DL) techniques
such as CNN, recognizing periodic/static activities (e.g, walking, lying,
cycling, etc.) has become a well-studied problem. What remains a major
challenge though is the sporadic activity recognition (SAR) problem, where
activities of interest tend to be non-periodic, and occur less frequently when
compared with the often large amount of irrelevant “background” activities.
Recent works suggested that sequential DL models (such as LSTMs) have great
potential for modeling non-periodic behaviours, and in this paper we studied
some LSTM training strategies for SAR. Specifically, we proposed two simple
yet effective LSTM variants, namely delay model and inverse model, for two SAR
scenarios (with and without time-critical requirement). For time-critical SAR,
the delay model can effectively exploit pre-defined delay intervals (within
tolerance) in form of contextual information for improved performance. For
regular SAR task, the second proposed, inverse model can learn patterns from
the time-series in an inverse manner, which can be complementary to the
forward model (i.e., LSTM), and combining both can boost the performance.
These two LSTM variants are very practical, and they can be deemed as training
strategies without alteration of the LSTM fundamentals. We also studied some
additional LSTM training strategies, which can further improve the accuracy.
We evaluated our models on two SAR and one non-SAR datasets, and the promising
results demonstrated the effectiveness of our approaches in HAR applications.

###### Index Terms:

Human Activity Recognition; LSTM; Wearable; Deep Learning

##  I Introduction

Human activity recognition (HAR) represents one of the core pillars of mobile,
wearable, and pervasive computing, with many practical applications in fields
such as health and sleep monitoring [1, 2, 3, 4, 5, 6, 7], behaviour and skill
assessment [8, 9, 10, 11], or sports coaching[12, 13, 14], to name but a few.
For traditional machine learning based HAR approaches, feature engineering
plays an important role [15, 16, 17, 18]. With expert-designed features,
classification can be performed using conventional classifiers such as SVM or
KNN [12, 1]. However, designing effective features tends to be a somewhat
tedious trial-and-error process, and discriminant features may vary from task
to task, making system-developing expensive and less sustainable. Recently,
deep learning (DL) became the major approach for HAR and behaviour modeling,
mainly reasoned by its ability to learn high-level representation in an end-
to-end manner [19]. Based on fixed-length windows/ frames, activity features
can be learned via a range of frame-wise DL methods (e.g., RBM [20], CNN [21],
DeepConvLSTM [22], etc.), with substantial performance improvement when
compared with the traditional approaches.

Periodic |  Activities or gestures exhibiting periodicity, such as walking,running, rowing, biking, etc.
---|---
Static |  Static postures or static pointing gestures.
Sporadic |  The activity or gesture occurs sporadically, interspersed with other activities or gestures.
TABLE I: Categorization of activities according to [16].

In [16], activities were categorized into three types: i) static activities
ii) periodic activities; and iii) sporadic activities, with related
descriptions shown in Table I. Out of these types, the problem of recognizing
static or periodic activities (such as lying, standing, walking, running,
cycling, ascending stairs, etc.) has been well-addressed by recent frame-wise
DL approaches (e.g., [21, 22]). Recognizing sporadic activities, where
activities of interest tend to be non-periodic, and occur less frequently in
contrast to the large amount of irrelevant background activities, still remain
a major challenge in the HAR research community. Recent works suggested that
sequential DL models (e.g., sample-wise LSTM [21, 23]) have great potential
for modeling the non-periodic activities (e.g., opening the door, closing the
drawer, etc.), which is worth exploring on solving the challenging SAR
problems.

In this paper, we studied some LSTM training strategies for HAR. Specifically,
two simple yet effective LSTM variants, namely delay model, and inverse model,
were developed and studied for two SAR scenarios (i.e, with and without time-
critical requirement). For delay model, a time delay interval $\Delta$ is
defined, based on which the model performs LSTM-like, sample-wise
classification at timestamp $t$ based on the signal received at and before
timestamp $t+\Delta$. Performance gains are achieved by taking advantage of
the ”$\Delta$ timestamps of future information”, which can be set by the users
for the best trade-off between (decision-making) latency and effectiveness.

For regular SAR tasks (without time-critical requirement), the second proposed
model– the inverse model–performs sample-wise classification in an inverted
processing manner (in contrast to LSTM). This model learns patterns from the
time-series inversely, which, as we found, is complementary to the forward
model (i.e., LSTM), and combining them effectively boosts the overall
classification accuracy. Both LSTM variants are simple yet effective, and they
can be deemed as training strategies without adding the need to modify the
underlying LSTM modeling foundations. As such, they are universally applicable
with virtually no added costs or efforts.

Based on these two LSTM variants, we also studied additional training
strategies, i.e., hyper-parameter as variable strategy (HYPAV) and epoch-wise
bagging strategy [23] on some SAR/non-SAR datasets, and the improved
performance suggested they are generic, and can be employed as building blocks
for other advanced frameworks in HAR research.

##  II Related Work

###  II-A Human Activity Recognition(HAR)

HAR is one of the core research areas in wearable computing, and deep learning
(DL) based modeling approaches are now considered the state-of-the-art and the
most effective method for extracting behaviour patterns. Classical DL-based
HAR works include the application of CNN [24], LSTM[21], Bi-directional
LSTM(Bi-LSTM) [21], Convolutional LSTM[22], Attention models [25, 26, 27],
etc. To address the problem caused by inconsistent labels, an efficient and
flexible modeling approach named dense sampling with full convolution neural
network was proposed [28]. In [29], a deep learning approach was designed to
learn statistical, spatial, and temporal features for HAR. To improve the
performance of HAR, various fusion techniques (e.g., via sensors or
classifiers [30, 23]) were also developed.

Recently, and in addition to, for example, cross-modality transfer techniques
[31, 32, 33], more advanced DL methods were proposed and employed to solve the
core challenges, e.g., lack of annotations, large subject variability,
scalable training in HAR. To address the lack of annotation problem, self-
supervised learning were employed in [34, 35, 36, 37, 38, 39, 40] which can
take advantage of the unlabelled data for performance improvement. In [41, 42,
43, 44], adversarial learning methods were developed to reduce the variability
among subjects for user-independent HAR. Federated learning approaches were
studied in [45, 46] for privacy-preserving training and personalized HAR.

###  II-B Sample-wise Long Short Term Memory (LSTM)

Most of the DL-base HAR approaches performed frame-wise classification. Here
we also review sample-wise LSTM, which suggested their effectiveness in
modeling sequential activities [21, 23]. Based on LSTM, the proposed LSTM
variants can be easily derived.

LSTM is a hidden unit in recurrent neural networks(RNN), and it consists of
three gates (i.e., forget gate, input gate, and output gate) and one memory
cell [47]. For an exemplary one-layer LSTM, it takes current input signal
$\boldsymbol{x}_{t}$, the previous hidden and cell state
$\boldsymbol{h}_{t-1}$, $\boldsymbol{c}_{t-1}$, and outputs the current hidden
and cell state $\boldsymbol{h}_{t}$, $\boldsymbol{c}_{t}$. Specifically, the
forward pass can be written as follows:

$\boldsymbol{f}_{t}=\sigma\left(\boldsymbol{W}^{T}_{\boldsymbol{xf}}\boldsymbol{x}_{t}+\boldsymbol{W}^{T}_{\boldsymbol{hf}}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{\boldsymbol{f}}\right)$
$\boldsymbol{i}_{t}=\sigma\left(\boldsymbol{W}^{T}_{\boldsymbol{xi}}\boldsymbol{x}_{t}+\boldsymbol{W}^{T}_{\boldsymbol{hi}}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{\boldsymbol{i}}\right)$
$\boldsymbol{o}_{t}=\sigma\left(\boldsymbol{W}^{T}_{\boldsymbol{xo}}\boldsymbol{x}_{t}+\boldsymbol{W}^{T}_{\boldsymbol{ho}}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{\boldsymbol{o}}\right)$
$\tilde{\boldsymbol{c}}_{t}=\tanh\left(\boldsymbol{W}^{T}_{\boldsymbol{xc}}\boldsymbol{x}_{t}+\boldsymbol{W}^{T}_{\boldsymbol{hc}}\boldsymbol{h}_{t-1}+\boldsymbol{b}_{\boldsymbol{c}}\right)$
$\boldsymbol{c}_{t}=\boldsymbol{f}_{t}\odot\boldsymbol{c}_{t-1}+\boldsymbol{i}_{t}\odot\tilde{\boldsymbol{c}}_{t}$
$\boldsymbol{h}_{t}=\boldsymbol{o}_{t}\odot\tanh\left(\boldsymbol{c}_{t}\right)$

where $\sigma$ is the sigmoid function;$\odot$ is the entry-wise
multiplication;
$\boldsymbol{f}_{t},\boldsymbol{i}_{t},\boldsymbol{o}_{t},\tilde{\boldsymbol{c}}_{t}$
are the internal outputs of the three gates and the memory cell;
$\boldsymbol{c}_{t},\boldsymbol{h}_{t}$ are the outputs of the LSTM unit. With
$\boldsymbol{h}_{t}$ at timestamp $t$, for $C$ classes (e.g., with label
$\boldsymbol{y}_{t}=\\{1,2,...,C\\}$) the output of the sample-wise feed-
forward pass can be written as:

$\boldsymbol{p}^{LSTM}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t-1},\boldsymbol{c}_{t-1})=\operatorname{softmax}\left(\boldsymbol{W}^{T}_{\boldsymbol{h}\boldsymbol{C}}\boldsymbol{h}_{t}+\boldsymbol{b}_{\boldsymbol{C}}\right)$
(1)

which is the class probability vector. Given that, label $\hat{y}_{t}$ is
assigned to the class with the largest entry, i.e.,

$\hat{y}_{t}=\arg\max_{\\{1,...,C\\}}\boldsymbol{p}^{LSTM}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t-1},\boldsymbol{c}_{t-1}).$
(2)

For model parameters, $\boldsymbol{W}_{\boldsymbol{xf}}$,
$\boldsymbol{W}_{\boldsymbol{hf}}$, $\boldsymbol{W}_{\boldsymbol{xi}}$,
$\boldsymbol{W}_{\boldsymbol{hi}}$, $\boldsymbol{W}_{\boldsymbol{xo}}$,
$\boldsymbol{W}_{\boldsymbol{ho}}$, $\boldsymbol{W}_{\boldsymbol{xc}}$,
$\boldsymbol{W}_{\boldsymbol{hc}}$, $\boldsymbol{W}_{\boldsymbol{hC}}$ are
weight matrices, and
$\boldsymbol{b}_{\boldsymbol{f}}$,$\boldsymbol{b}_{\boldsymbol{i}}$,$\boldsymbol{b}_{\boldsymbol{o}}$,$\boldsymbol{b}_{\boldsymbol{c}}$,
$\boldsymbol{b}_{\boldsymbol{C}}$ are bias vectors, which can be trained using
backpropagation through time (BPTT)[47].

###  II-C Sample-wise LSTM Ensemble

In [23], epoch-wise bagging scheme was applied to generate a number of base
sample-wise LSTM learners to be aggregated for HAR. To improve the diversity
of ensemble, some hyper-parameters were modelled as variables, e.g., window
length, batch size, initial sampling point, which were dynamically changing in
an epoch(or batch) manner to generate less correlated epoch-wise base
classifiers. If we simplify LSTM’s class probability vector (i.e., Eq. 1) as
$\boldsymbol{p}^{LSTM}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t})$, for an
ensemble of $M$ epoch-wise LSTM classifiers, the aggregated score (via a
score-level fusion) can be expressed as:

$\boldsymbol{p}^{LSTM}_{fusion}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t})=\frac{1}{M}\Sigma_{m=1}^{M}\boldsymbol{p}^{LSTM}_{m}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t}),$
(3)

where $\boldsymbol{p}^{LSTM}_{m}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t})$
represents the $m^{th}$ base learner in the LSTM ensemble. In [23], based on
the inequality of arithmetic and geometric means, it was proved that if cross
entropy (CE) was used, the fusion loss $L^{LSTM}_{fusion}$ is always less or
equal than the average loss of base learners
$\frac{1}{M}\Sigma_{m=1}^{M}L^{LSTM}_{m}$, indicating the benefit of employing
ensemble approaches for performance improvement. Motivated by this, in this
work we also studied the fusion of base learners generated from the proposed
LSTM variants.

Figure 1: The delay LSTM model for time-critical applications (with pre-
defined delay interval $\Delta$)

##  III Methodology

In literature, the mainstream DL-base HAR methods performed frame-wise
classification. Signals within each pre-defined (sliding) window is associated
with an activity label, before DL was employed to learn the mapping. It has
achieved its success in the general HAR modeling since the information of most
activities (especially periodic/static types) can be preserved by using short
windows (e.g., 1s), and thus can be modelled via the independent window-label
pairs.

However, there might exist prolonged sequential activities to be recognized,
which requires more information beyond the length of pre-defined windows. It
was found that sample-wise stateful LSTM (short for LSTM in this paper) can
carry information across windows/frames [21], with promising results in
sequential activities modeling [21, 23], making it a potential solution for
the challenging SAR tasks. Moreover, compared with frame-wise approaches, it
can perform activity segmentation directly, which might be practical in many
applications (e.g., to predict the exact activity duration).

Given that, here we develop some LSTM variants, which can be considered as
training strategies without alternation of the LSTM fundamentals.

###  III-A The Delay LSTM Model

For some time-critical SAR applications such as health monitoring, it is
crucial to have the events or incidents detected in low latency. For example,
the Freeze-of-Gait incidents from patients with Parkinson’s disease should be
detected in less than 2s after its onset [48]. The delay model proposed in
this paper can be used to determine the latency for optimal accuracy in such
tasks.

Specifically there is a user-defined hyper-parameter, delay interval
($\Delta$), and at timestamp $t+\Delta$ the delay model will output the
”delayed predicted label $\hat{y}_{t}$”, as shown in Fig. 1.

Figure 2: The Inverse LSTM Model

In essence, it is to give the model $\Delta$ timestamps of ”additional
response time” (hence delay model) to better ”predict the past”. Therefore,
the delay model is expected to be able to exploit the $\Delta$ timestamps of
additional signal information for performance gain.

In contrast to vanilla LSTM, which can map signals to labels at the same
timestamp (i.e., from $\boldsymbol{x}_{t}$ to $\hat{y}_{t}$ as shown in Eq.
2), the delay LSTM model written as

$\hat{y}_{t}=\arg\max_{\\{1,...,C\\}}\boldsymbol{p}^{DLY}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t+\Delta},\boldsymbol{h}_{t+\Delta-1},\boldsymbol{c}_{t+\Delta-1})$
(4)

maps the current signal $\boldsymbol{x}_{t+\Delta}$ to prediction with
$\Delta$ timestamps of delay (i.e.,$\hat{y}_{t}$). In Eq. 4,
$\boldsymbol{h}_{(.)}$ and $\boldsymbol{c}_{(.)}$ correspond to hidden and
cell states.

###  III-B The Inverse LSTM Model

Compared with LSTM, which maps signal-label pairs in a forward way, i.e., from
the order $\\{(\boldsymbol{x}_{1},y_{1}),...,$
$(\boldsymbol{x}_{T},y_{T})\\}$, the inverse model performs the mapping in an
inverted order, i.e., $\\{(\boldsymbol{x}_{T},y_{T}),...,$
$(\boldsymbol{x}_{1},y_{1})\\}$, aiming at modeling the sequence from a
different view. Fig. 2 illustrates the inference process of the inverse model.
For timestamp $t$ we can define it as:

$\hat{y}_{t}=\arg\max_{\\{1,...,C\\}}\boldsymbol{p}^{INV}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t+1},\boldsymbol{c}_{t+1}),$
(5)

where $\boldsymbol{h}_{t+1}$,$\boldsymbol{c}_{t+1}$ are the ”previous” (at
timestamp $t+1$) hidden and cell states. Motivated by the concept of multi-
view learning [49], the introduction of inverse model is to analyze the
activity signals from a different perspective, i.e., looking at activities
from the inverse order, which may be complementary to LSTM. For better
performance in the challenging SAR tasks, we aggregate the inverse model and
LSTM via a score-level fusion:

$\begin{split}\hat{y}_{t}=\arg\max_{\\{1,...,C\\}}&\frac{1}{2}\\{\boldsymbol{p}^{INV}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t+1},\boldsymbol{c}_{t+1})\\\
&+\boldsymbol{p}^{LSTM}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t-1},\boldsymbol{c}_{t-1})\\},\end{split}$
(6)

from which we can see
$\boldsymbol{p}^{LSTM}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t-1},\boldsymbol{c}_{t-1})$
is based on the past hidden/cell states, while
$\boldsymbol{p}^{INV}(\boldsymbol{y}_{t}|\boldsymbol{x}_{t},\boldsymbol{h}_{t+1},\boldsymbol{c}_{t+1})$
is based on the ”future” (i.e., ”inverted past”) hidden/cell states. In this
case, it can be beneficial to combine these two complementary information for
improved performance.

(a) OPP dataset

(b) DG dataset

(c) PAMAP2 dataset

Figure 3: Class distributions of the three datasets used in this paper; OPP
and DG cover SAR tasks, whereas PAMAP2 targets Non-SAR tasks.

##  IV Experiments

To evaluate the effectiveness of the proposed LSTM variants, we conducted
experiments on three public HAR datasets, including i) Opportunity (OPP) [50],
ii) Daphnet Freeze of Gait (DG) [48] for SAR tasks, and iii) PAMAP2 [51] for
Non-SAR tasks. Moreover, based on the two LSTM variants, we also studied some
additional training strategies, i.e., Hyper-parameter as variable (HYPAV)
strategy and Epoch-wise bagging strategy for performance improvement.

###  IV-A Datasets

In our experiments, datasets with both SAR and Non-SAR tasks were used.
Specifically, OPP and DG datasets (corresponding to SAR tasks) cover low-
frequent, sporadic activities, contrasted by large (over-represented) portion
of “other” (null) class activities. The resulting class imbalance is
substantial: $>75\%$ of the data are NULL for OPP, and $>90\%$ belong to the
non-target class in DG, rendering the analysis task challenging. We also
evaluated our models on the Non-SAR PAMAP2 dataset – primarily for robustness
and generalization evaluation. The class distributions of these three datasets
are illustrated in Fig. 3. These datasets were detailed as follows.

####  IV-A1 Opportunity (OPP)

OPP [50] is probably the most challenging HAR dataset. From four participants,
$17$ pre-defined kitchen activities such as opening the door, closing the
drawer, cleaning, etc. as well as the background activity (referred to as the
Null class) were recorded. The pre-defined activities tend to be non-periodic,
and most of them cover only $<2\%$ each of the overall dataset, in contrast to
$>75\%$ of the null class. Following [21, 23], we used the 79-dimensional
sensor data with sampling rate at 30Hz.

(a) on OPP dataset (regular SAR)

(b) on DG dataset (time-critical SAR)

(c) on PAMAP2 dataset (Non-SAR)

Figure 4: Box-plots of delay models’ performance on three datasets in leave-
one-subject-out cross validation (LOSO-CV) setting; Mean-F1 scores were used
on OPP and PAMAP2, while sensitivity (with specificity fixed at $0.9$) was
used for DG dataset; The red circles indicate the average/mean values, and the
horizontal dashed line is the average performance of LSTM model (i.e., with
delay interval $\Delta=0$)

####  IV-A2 Daphnet Freeze of Gait (DG)

In contrast to OPP dataset which contains regular daily sporadic activities,
DG is a health monitoring dataset including time-critical SAR tasks. The
sporadic activity in DG is Freeze-of-Gait (FoG) symptom [48], which is a
common complication in Parkinson’s Disease(PD), with sudden and transient
inability to walk. In order to recognize the FoG incidents from the normal
walking or other activities (i.e., no freeze class), ten PD patients who
suffered from FoG were recruited, and wearable sensors were placed above the
ankle, knee, and waist to collect the 9-dimensional accelerometer data [48].

####  IV-A3 PAMAP2

For Non-SAR tasks, we used the popular PAMAP2 dataset [51], which includes 12
periodic/static activities such as walking, running, cycling, lying, sitting,
etc. From nine subjects, IMU(including accelerometer, gyroscope,
magnetometer), temperature, and heart rate data were recorded via body-worn
sensors attached to hand, chest and ankle, yielding a 52-dimensional time-
series dataset. We downsampled the dataset to 33Hz in our experiments.

###  IV-B Settings

####  IV-B1 Evaluation Protocols

In this work, we evaluated the models primarily using hold-out validation. In
addition, leave one subject out cross validation (LOSO-CV) was also applied in
some experiments as supplementary .

For OPP dataset, in the LOSO-CV setting data from all four participants were
used; while in the hold-out setting, we used the protocol defined in [21, 23].
That is, the second run from subject 1 was used as validation data, runs 4 and
5 from subjects 2 and 3 were used as test set, and the remaining data were
used for training.

For DG dataset, since subject 4 and 10 do not have the FoG incidents, they
were excluded in our experiments. Therefore, data from the rest eight subjects
were used in both LOSO-CV and hold-out settings. In the hold-out setting, we
followed the protocol used in [21]. That is, we used run 1 from subject 9 as
validation set, runs 1 and 2 from subject 2 as test set, and used the rest for
training.

For PAMAP2 dataset, in LOSO-CV, eight subjects were used, and in the hold-out
setting we followed the protocol used in [21][23]. That is, we used runs 1 and
2 from subject 5 for validation, and runs 1 and 2 from subject 6 as test while
using the rest data for training.

- | OPP | DG | PAMAP2
---|---|---|---
Metric | mean-F1 | Sensitivity | mean-F1 | mean-F1
LSTM | 0.673 | 0.821 | 0.668 | 0.795
Delay($\Delta=0.1s$) | 0.669 | 0.832 | 0.671 | 0.798
Delay($\Delta=0.3s$) | 0.678 | 0.834 | 0.668 | 0.801
Delay($\Delta=0.5s$) | 0.688 | 0.848 | 0.683 | 0.804
Delay($\Delta=0.7s$) | 0.685 | 0.839 | 0.684 | 0.806
Delay($\Delta=1.0s$) | 0.667 | 0.812 | 0.662 | 0.802
Delay($\Delta=1.5s$) | 0.645 | 0.776 | 0.642 | 0.796
TABLE II: Performance of Delay models in hold-out setting on three datasets

To measure the performance, for OPP and PAMAP2 datasets we used the popular
mean-F1 score. For DG dataset, since the FoG detection task is a typical
binary health monitoring problem, which requires suitable thresholds to
minimize the false negative predictions, we used sensitivity (with a fixed
specificity $0.9$) as the main metric, and we also reported the corresponding
mean-F1 score to make it in line with other HAR literature.

Since we focus on sample-wise activity recognition (as per [21, 23]), for any
frame-wise baselines (e.g., CNN [24], DeepConvLSTM [22]), we applied a post-
processing procedure that maps the frame-wise prediction to sample-level
results.

####  IV-B2 Network Architectures and Hyper-parameters

For comparison to the unmodified LSTM baseline, we followed [23] and used a
two-layer LSTM, with 256 hidden units for each layer. Dropout was used for
both the first/ second hidden layers with a probability of $50\%$. Cross
entropy loss was used and the learning rate was set to $0.001$. In the default
settings (i.e., when without HYPAV strategy), we set the batch size to $128$,
and window length to $1$ second for all datasets.

###  IV-C Delay Models

Based on LOSO-CV setting, in Fig. 4 we reported the delay models’ performance
using box-plots. Specifically, on the three datasets (OPP, DG, and PAMAP2) we
showed the results corresponding to different delay intervals
$\Delta=\\{0s,0.1s,0.3s,0.5s,0.7s,1s,1.5s\\}$. Note LSTM model is a special
case when we set $\Delta=0$ in the delay model. In Fig. 4, we also reported
the average performance using red circles, and plotted LSTM’s average
performance in dashed line to compare with other delay models.

For SAR tasks, in Fig. 4(a), and 4(b), we can observe when there is a small
delay interval $\Delta=0.1s$, the delay mechanism can generally improve the
average performance, indicating the ”short-term future information” is
important to the sequential modeling in SAR tasks. However, the results may
start to drop when a larger $\Delta$ was applied, suggesting information from
the ”long future” may be less relevant. This trend is more obvious on OPP
dataset than DG dataset.

On the other hand, on the Non-SAR PAMAP2 dataset, performance does not vary
much when with different $\Delta$ (as shown in Fig. 4(c)), indicating the
contextual information may not be as useful as the ones in SAR tasks.
Nevertheless, they are not harmful to the performance, indicating the
generalization capability of the delay models.

We also ran these delay models in hold-out settings on these three datasets,
and the results were reported in Table II. We see the performance is generally
consistent with the LOSO-CV settings. That is, the performance gains can be
achieved when suitable delay intervals were used in SAR tasks (i.e., on OPP
and DG datasets), while this trend is not obvious for the Non-SAR PAMAP2
dataset. These results suggest that the periodic or static activities (i.e.,
in Non-SAR tasks) can’t provide as much sequential information as the sporadic
activities for delay models.

In SAR tasks, it is interesting to see that the optimal delay interval
$\Delta$ may vary. For example, in LOSO-CV settings, best (average)
performance can be achieved when $\Delta=0.1s$ on OPP and DG datasets, while
in hold-out setting the optimal results were based on $\Delta=0.5s$ or
$\Delta=0.7s$. One possible explanation can be $\Delta=0.1s$ is the optimal
for the whole population (i.e., the subjects in the dataset), which is not
necessarily the best one for the unseen individuals. This indicates that
personalized or adaptive delay interval should employed, which will be studied
in the future.

In practical time-critical health monitoring scenarios (e.g., on DG dataset),
we can see the benefits from the delay models. In both LOSO-CV and hold-out
settings, the detection rate (i.e., sensitivity) can be improved when with
small delay intervals (e.g., $\Delta=\\{0.1s,0.3s,0.5s,0.7s\\}$), indicating
delay models can take advantage of the additional contextual information at a
cost of short latency $\Delta$. Since the FoG incidents should be detected as
early as possible (with an acceptable tolerance $<2s$ [48]), our delay model
(i.e., with $\Delta\leq 0.7s$)) can be served as an effective health
monitoring system in this time-critical SAR application.

For the rest of the paper, for simplicity all experimental results reported
were based on hold-out settings.

###  IV-D Combining LSTM and Inverse Models

We also ran the inverse LSTM model on three datasets, which can perform
sequential labelling in the backward way. Compared with the forward sequential
labelling models such as LSTM, inverse model look at the same data from a
different view, which may be complementary. Hence, we combined both models
using a simple score-level fusion, and compared it with baseline LSTM and
delay models (with the optimal $\Delta$).

- | OPP | DG | PAMAP2
---|---|---|---
Metric | mean-F1 | Sensitivity | mean-F1 | mean-F1
LSTM | 0.673 | 0.821 | 0.668 | 0.795
Delay | 0.688 | 0.848 | 0.683 | 0.806
LSTM$\&$Inverse | 0.705 | 0.832 | 0.674 | 0.804
TABLE III: Performance of LSTM$\&$Inverse models in hold-out setting on three
datasets; For delay models, only the ones with the best delay intervals were
reported here.

From Table III we can see on OPP dataset, the mean F1 score can be improved
from $0.673$ (LSTM) to $0.705$ (LSTM$\&$Inverse), and the combined model is
also $1.7\%$ higher than the best delay model. On DG and PAMAP2 datasets,
however, the performance gains are not as significant as the simple delay
models. Moreover, compared with LSTM or delay models, the combined
LSTM$\&$Inverse sacrifices the (near-)real-time property since the whole
inference sequence should be acquired before inverse model can be applied.
Given that, it is less practical for time-critical health monitoring tasks.

Nevertheless, from the modeling perspective we can see the benefit of
employing inverse model under the fusion scheme, which motivates us to explore
more fusion strategies on these LSTM variants.

###  IV-E Additional Training Strategies

The proposed LSTM variants can be deemed as training strategies without
alteration of the LSTM fundamentals, making them very practical. Moreover, we
studied additional training strategies for performance improvement, i.e., i)
Hyper-parameter as variable (HYPAV) strategy and ii) Epoch-wise bagging
strategy.

####  IV-E1 Hyper-parameter as variable (HYPAV) strategy

In previous sample-wise HAR work [23], some hyper-parameters such as window
length, batch size were modelled as random variables, which were dynamically
changing in different training batches or epochs in order to increase the
diversity of the LSTM ensemble system for higher performance.

Different from [23] which has an ensemble diversity emphasis, here we mainly
study how the proposed single LSTM variants can benefit from the ”hyper-
parameter as variable (HYPAV)” strategy.

Following [23], we set window length, batch size, and the sampling position as
random numbers. Due to these introduced random factors, we ran each model 30
times with the corresponding results reported in the form of mean$\pm$std or
box-plotting. In some experiments, we further employed two-tailed independent
t-tests and reported significance levels through p-values, where p $\leq$
0.05, p $\leq$ 0.01 and p $\leq$ 0.001 correspond to *, **, and ***,
respectively.

Fig. 5 shows the box-plot of performance w.r.t. delay interval ($\Delta$) of
our delay model on OPP dataset when HYPAV strategy was used, and we can see
the performance gain when with $\Delta\leq 1s$. Similar to the previous
results where HYPAV strategy was not used, it shows that a longer delay (e.g.,
$1.5s$) may not contribute positively to the performance. This is sensible
since signal and label may be less correlated for timestamps far-apart.
Specifically, when with a short delay $\Delta$, the delay model can
effectively exploit the additional contextual information for improved
results, and with $\Delta=0.7s$ it has the best mean-F1 $0.705\pm 0.02$, which
is nearly $2\%$ higher than the best one with fixed hyper-parameters (i.e., as
shown in Table II), suggesting the effectiveness of the HYPAV strategy in
delay models.

Figure 5: Box-plot of mean-F1 w.r.t. $\Delta$ of the delay models on OPP.
Figure 6: Box-plot of mean-F1 scores for LSTM, the inverse model, and their
fusion (LSTM$\&$Inverse) on OPP dataset; $*$ indicates p-value $p<0.05$.

We also applied HYPAV training strategy to inverse model on OPP dataset, and
report the related results in Fig. 6. Although we do not observe any
performance gain (over LSTM) if only inverse model was applied, under HYPAV
strategy it shows its powerful complementary capability with LSTM. By
performing a simple score-level fusion, we can see the combined model
LSTM$\&$Inverse achieves mean-F1 $0.733\pm 0.01$, much higher than single
models (with p-value $p<0.05$). Compared with the one with fixed window length
and batch size (with mean F1 score $0.705$ as reported in Table III), this
HYPAV training strategy can also bring nearly $3\%$ performance gain in the
LSTM$\&$Inverse scheme.

Similarly, we also conducted experiments on DG and PAMAP2 datasets, and the
results of these three datasets were summarized in Table IV. For delay models
with multiple results, only the best ones were reported, and we can see the
our models can benefit from the HYPAV training strategies, and performance
gain can be achieved if the hyper-parameters (e.g., window length, batch size)
were modelled as variables.

- | OPP (mean-F1) | DG (Sensitivity / mean-F1) | PAMAP2 (mean-F1)
---|---|---|---
Strategy | w/ HYPAV | w/o | w/ HYPAV | w/o | w/ HYPAV | w/o
Delay Model | 0.705$\pm$0.02 | 0.688 | 0.865$\pm$0.04 / 0.719$\pm$0.01 | 0.848 /0.683 | 0.829$\pm$0.04 | 0.806
LSTM$\&$Inverse | 0.733$\pm$0.01 | 0.705 | 0.833$\pm$0.03 / 0.702$\pm$0.01 | 0.832 /0.674 | 0.829$\pm$0.05 | 0.804
TABLE IV: Performance evaluation of the Hyper-parameter as variable (HYPAV)
strategy; For DG dataset, both sensitivity (with specificity fixed at 0.9) and
the corresponding mean F1 scores were reported; For delay models with
different delay intervals $\Delta$, only the best results were presented here
Figure 7: Box-plot of mean-F1 scores for different ensembles on Opp; DLY/INV
represents the delay/inverse models.

####  IV-E2 Epoch-wise bagging strategy

The delay model is very practical, and we can easily achieve some performance
gain at a cost of (the pre-defined) latency, making it is a flexible tool for
near-real-time prediction applications. On the other hand, the results also
suggested the effectiveness of using score-level fusion of two different LSTMs
(e.g., LSTM$\&$Inverse), which motivates us to employ ensemble or more fusion
approaches for additional performance gain. In [23], Guan and Ploetz proposed
an epoch-wise bagging scheme for LSTM ensemble, and with the HYPAV strategy,
the models trained from different epochs were used as base learners to be
aggregated.

Here on OPP dataset, we also studied this epoch-wise bagging framework (with
HYPAV) based on the three LSTM types (i.e., LSTM/Delay(DLY)/Inverse(INV)).
Specifically, we generated different ensembles (with a fixed size of 20 base
classifiers) and compared their results in Fig. 7, from which we can observe:

  * •

for single source ensemble, DLY(20) has much better results than other
LSTM(20) or INV(20)

  * •

multi-source ensembles tend to have higher performance, and the best results
can be achieved when fusing the base classifiers from three sources
LSTM(6)+DLY(7)+INV(7), significantly higher than any single/dual-source
ensembles.

Although base learner’s performance is important, it is also crucial to
combine diversified base classifiers. Although HYPAV strategy can be used to
inject diversity in the epoch-wise bagging scheme, the base learners are
derived from the same training process and still are highly correlated [23].
On the other hand, the multi-source ensemble in Fig. 7 are derived from three
different training processes (i.e., based on LSTM/Inverse/Delay models) which
may yield diversified base learners with improved results. These results
indicate that our proposed LSTM variants can be used as effective building
blocks of other advanced schemes for performance boosting.

###  IV-F Model Comparison

On the three HAR datasets, we also compare our LSTM models (Delay and
LSTM$\&$Inverse, and multi-source ensemble) with other state-of-the-art
approaches.

In Table V we listed the results of other approaches on OPP dataset, including
Dense Labeling [28], DeepConvLSTM [22], LSTM-S [21], Bi-LSTM [21], Attention
Model [52], LSTM Ensemble [23], and Attend$\&$Discriminate[53]. We can see our
LSTM variants (Delay ($\Delta=0.7s$) and LSTM$\&$Inverse) yield very promising
results, indicating their effectiveness. They are very practical due to the
simplicity nature, and with the epoch-wise bagging strategy (including HYPAV),
these LSTM variants can be combined with the highest result on the OPP
dataset, which was considered as one of the most challenging dataset.

We also compared our approaches with other baselines on DG and PAMAP2
datasets, and we can see substantial performance gain when with our LSTM
variants and training strategies. It is worth noting that on DG dataset,
although single delay model had lower performance than the multi-source
ensemble, it can be performed in near-real-time (with a delay interval of
$0.5s$), making it a practical solution in health monitoring applications. On
the other hand, on the Non-SAR PAMAP2 dataset, although previous results
suggested no significant performance gains can be achieved by using LSTM
variants only (as shown in Table III), we can see the benefits of applying the
additional training scheme (i.e., epoch-wise bagging with HYPAV) to these LSTM
variants, suggesting their generalization capability and robustness.

Model | Mean F1
---|---
Dense Labelling [28] | 0.624
DeepConvLSTM [22] | 0.672
LSTM-S [21] | 0.684
Bi-LSTM [21] | 0.697
Attention Model [52] | 0.707
LSTM Ensemble [23] | 0.726 $\pm$ 0.01
Attend$\&$Discriminate [53] | 0.746
Delay($\Delta=0.7s$) | 0.705 $\pm$ 0.02
LSTM$\&$Inverse | 0.733 $\pm$ 0.01
LSTM(6)$\&$DLY(7)$\&$INV(7) | 0.778 $\pm$ 0.01
TABLE V:  Model Comparison on OPP dataset. Model | Sensitivity | Mean F1
---|---|---
CNN[24] | 0.804 | 0.658
DeepConvLSTM[22] | 0.823 | 0.665
Bi-LSTM[21] | 0.814 | 0.682
Delay($\Delta=0.5s$) | 0.865 $\pm$ 0.04 | 0.719 $\pm$ 0.01
LSTM&Inverse | 0.833 $\pm$ 0.03 | 0.702 $\pm$ 0.01
LSTM(6)$\&$DLY(7)$\&$INV(7) | 0.882 $\pm$ 0.03 | 0.735 $\pm$ 0.01
TABLE VI: Model Comparison on DG dataset (in both sensitivity and Mean-F1 score); Sensitivities correspond to a fixed specificity of 0.9. Model | Mean F1
---|---
CNN[24] | 0.771
DeepConvLSTM[22] | 0.765
Bi-LSTM[21] | 0.802
Delay($\Delta=1.5s$) | 0.829 $\pm$ 0.04
LSTM&Inverse | 0.829 $\pm$ 0.05
LSTM(6)$\&$DLY(7)$\&$INV(7) | 0.857 $\pm$ 0.03
TABLE VII: Model Comparison on PAMAP2 dataset

###  IV-G Discussion

In the tasks of recognizing the challenging sporadic activities, sample-wise
LSTM demonstrated its great potential in previous works. In this work, we
studied several training strategies (including the two LSTM variants) to
further boost the performance for SAR, which can also generalize well to Non-
SAR tasks.

####  IV-G1 LSTM variants as Simple Training Strategies

Our delay and inverse models can be regarded as training strategies, which can
be realized without changing the code of LSTM. Specifically, the delay model
can be trained by feeding the shifted signal sequences(i.e., backward by
$\Delta$ timestamps, as shown in Fig. 1), while the inverse model can be
trained by feeding the inverted sequences, as shown in Fig. 2. Without
alteration of LSTM, these two variants are simple, and can be very practical.

####  IV-G2 LSTM$\&$Inverse and Bi-LSTM

It is worth mentioning that our LSTM$\&$Inverse model and Bi-LSTM [21] are two
different methods. Although Bi-LSTM [21] can also leverage the backward
sequential information, it is limited to the sliding-window level and tends to
be short-term. In contrast, inverse model can learn the long-term backward
sequential information without the aforementioned limitation. In essence, Bi-
LSTM is to combine forward/backward LSTM units in the network, while
LSTM$\&$Inverse is to perform score-level fusion of two separated LSTMs, which
can exploit long-term (in both forward/backward) sequential information.

####  IV-G3 Fusion Scheme for the General HAR

Based on the epoch-wise bagging scheme (with HYPAV)[23], much higher
performance can be achieved if base learners are from multiple training
sources (e.g., LSTM/Delay/Inverse models), than single source (e.g., LSTM
ensemble [23]). In the multi-source ensemble, higher diversity can be easily
achieved since the fusion system is formed by base classifiers generated from
different epoch-wise bagging schemes, which would also increase the training
cost by a factor of source number. Nevertheless, for some challenging HAR
cases this could be a potential way to boost the performance. It is worth
noting the multi-source fusion can be very flexible, e.g., it can also be
achieved by fusing epoch-wise classifiers from different network
architectures.

##  V Conclusion

In this work, we proposed two simple yet effective LSTM variants, namely delay
and inverse models for tackling the challenging SAR problems. They can be
deemed as training strategies without alteration of the LSTM fundamentals. For
time-critical SAR scenarios, the delay model can effectively take advantage of
the delay interval for performance improvement. For regular SAR tasks without
time-critical requirement, the inverse model can be used to analyze the
activities in an inverse manner, which is complementary to LSTM. Based on LSTM
and its variants, we also studied additional training strategies, i.e., hyper-
parameter as variable (HYPAV) and epoch-wise bagging [23], and we found
significant performance gains can be achieved when multi-source ensemble was
applied in both SAR and non-SAR tasks, indicating their effectiveness in
challenging HAR scenarios.

## References

  * [1] T. Plötz, N. Y. Hammerla, A. Rozga, A. Reavis, N. Call, and G. D. Abowd, “Automatic assessment of problem behavior in individuals with developmental disabilities,” in _Proceedings of the 2012 ACM Conference on Ubiquitous Computing_ , ser. UbiComp ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 391–400.
  * [2] D. Liaqat, M. Abdalla, P. Abed-Esfahani, M. Gabel, T. Son, R. Wu, A. Gershon, F. Rudzicz, and E. D. Lara, “Wearbreathing: Real world respiratory rate monitoring using smartwatches,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 3, no. 2, jun 2019.
  * [3] Y. Bai, Y. Guan, and W.-F. Ng, “Fatigue assessment using ecg and actigraphy sensors,” in _Proceedings of the 2020 International Symposium on Wearable Computers_ , ser. ISWC ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 12–16.
  * [4] S. Halloran, L. Tang, Y. Guan, J. Q. Shi, and J. Eyre, “Remote monitoring of stroke patients’ rehabilitation using wearable accelerometers,” ser. ISWC ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 72–77.
  * [5] B. Zhai, I. Perez-Pozuelo, E. A. D. Clifton, J. Palotti, and Y. Guan, “Making sense of sleep: Multimodal sleep stage classification in a large, diverse population using movement and cardiac sensing,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 4, no. 2, Jun. 2020.
  * [6] S. Bhattacharya, R. Adaimi, and E. Thomaz, “Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 6, no. 2, jul 2022.
  * [7] G. Ogbuabor and R. La, “Human activity recognition for healthcare using smartphones,” in _Proceedings of the 2018 10th International Conference on Machine Learning and Computing_ , ser. ICMLC 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 41–46.
  * [8] X. Qingxin, A. Wada, J. Korpela, T. Maekawa, and Y. Namioka, “Unsupervised factory activity recognition with wearable sensors using process instruction information,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 3, no. 2, pp. 1–23, 2019.
  * [9] Y. Gao, Y. Long, Y. Guan, A. Basu, J. Baggaley, and T. Ploetz, “Towards reliable, automated general movement assessment for perinatal stroke screening in infants using wearable accelerometers,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 3, no. 1, Mar. 2019.
  * [10] N. Yoshimura, T. Maekawa, T. Hara, and A. Wada, “Acceleration-based activity recognition of repetitive works with lightweight ordered-work segmentation network,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 6, no. 2, jul 2022.
  * [11] A. Khan, S. Mellor, E. Berlin, R. Thompson, R. McNaney, P. Olivier, and T. Plötz, “Beyond activity recognition: Skill assessment from accelerometer data,” ser. UbiComp ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 1155–1166.
  * [12] A. Khan, J. Nicholson, and T. Plötz, “Activity recognition for quality assessment of batting shots in cricket using a hierarchical representation,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 1, no. 3, Sep. 2017.
  * [13] C. Ladha, N. Y. Hammerla, P. Olivier, and T. Plötz, “Climbax: Skill assessment for climbing enthusiasts,” in _Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing_ , ser. UbiComp ’13. New York, NY, USA: Association for Computing Machinery, 2013, p. 235–244.
  * [14] D. Schuldhaus, “Human activity recognition in daily life and sports using inertial sensors,” Ph.D. dissertation, 01 2019.
  * [15] T. Plötz, P. Moynihan, C. Pham, and P. Olivier, “Activity recognition and healthier food preparation,” in _Activity Recognition in Pervasive Intelligent Environments_. Atlantis Press, 2011, pp. 313–329.
  * [16] A. Bulling, U. Blanke, and B. Schiele, “A tutorial on human activity recognition using body-worn inertial sensors,” _ACM Comput. Surv._ , vol. 46, no. 3, Jan. 2014.
  * [17] H. Haresamudram, D. V. Anderson, and T. Plötz, “On the role of features in human activity recognition,” in _Proceedings of the 23rd International symposium on wearable computers_ , 2019, pp. 78–88.
  * [18] N. Y. Hammerla, R. Kirkham, P. Andras, and T. Plötz, “On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution,” in _Proc. Int. Symposium on Wearable Computers (ISWC)_. ACM, 2013, pp. 65–68.
  * [19] T. Plötz and Y. Guan, “Deep learning for human activity recognition in mobile computing,” _Computer_ , vol. 51, no. 5, pp. 50–59, 2018.
  * [20] T. Plötz, N. Y. Hammerla, and P. Olivier, “Feature learning for activity recognition in ubiquitous computing,” ser. IJCAI’11. AAAI Press, 2011, p. 1729–1734.
  * [21] N. Y. Hammerla, S. Halloran, and T. Plötz, “Deep, convolutional, and recurrent models for human activity recognition using wearables,” _arXiv preprint arXiv:1604.08880_ , 2016.
  * [22] F. J. Ordóñez and D. Roggen, “Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition,” _Sensors_ , vol. 16, no. 1, p. 115, 2016.
  * [23] Y. Guan and T. Plötz, “Ensembles of deep lstm learners for activity recognition using wearables,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 1, no. 2, Jun. 2017. [Online]. Available: https://doi.org/10.1145/3090076
  * [24] J. Yang, M. N. Nguyen, P. P. San, X. L. Li, and S. Krishnaswamy, “Deep convolutional neural networks on multichannel time series for human activity recognition,” in _Twenty-fourth international joint conference on artificial intelligence_ , 2015.
  * [25] J. Morales, N. Yoshimura, Q. Xia, A. Wada, Y. Namioka, and T. Maekawa, “Acceleration-based human activity recognition of packaging tasks using motif-guided attention networks,” in _2022 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2022, pp. 1–12.
  * [26] B. Khaertdinov, E. Ghaleb, and S. Asteriadis, “Deep triplet networks with attention for sensor-based human activity recognition,” in _IEEE PerCom_ , 2021.
  * [27] A. Abedin, M. Ehsanpour, Q. Shi, H. Rezatofighi, and D. C. Ranasinghe, “Attend and discriminate: beyond the state-of-the-art for human activity recognition using wearable sensors,” _ACM IMWUT_ , 2021.
  * [28] R. Yao, G. Lin, Q. Shi, and D. C. Ranasinghe, “Efficient dense labelling of human activity sequences from wearables using fully convolutional networks,” _Pattern Recognition_ , vol. 78, pp. 252–266, 2018.
  * [29] H. Qian, S. J. Pan, B. Da, and C. Miao, “A novel distribution-embedded neural network for sensor-based activity recognition.” in _IJCAI_ , 2019, pp. 5614–5620.
  * [30] S. Liu, S. Yao, J. Li, D. Liu, T. Wang, H. Shao, and T. Abdelzaher, “Giobalfusion: A global attentional deep learning framework for multisensor information fusion,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 1, pp. 1–27, 2020.
  * [31] H. Kwon, C. Tong, H. Haresamudram, Y. Gao, G. D. Abowd, N. D. Lane, and T. Ploetz, “Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition,” _Proc. ACM Interactive, Mobile, Wearable and Ubiquitous Computing (IMWUT)_ , vol. 4, no. 3, 2020.
  * [32] H. Kwon, B. Wang, G. D. Abowd, and T. Plötz, “Approaching the real-world: Supporting activity recognition training with virtual imu data,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 5, no. 3, pp. 1–32, 2021.
  * [33] H. Kwon, G. D. Abowd, and T. Plötz, “Complex deep neural networks from large scale virtual imu data for effective human activity recognition using wearables,” _Sensors_ , vol. 21, no. 24, p. 8337, 2021.
  * [34] C. I. Tang, I. Perez-Pozuelo, D. Spathis, S. Brage, N. Wareham, and C. Mascolo, “Selfhar: Improving human activity recognition through self-training with unlabeled data,” _ACM IMWUT_ , 2021.
  * [35] A. Mohamed, F. Lejarza, S. Cahail, C. Claudel, and E. Thomaz, “Har-gcnn: Deep graph cnns for human activity recognition from highly unlabeled mobile sensor data,” in _IEEE PerCom Workshops_ , 2022.
  * [36] H. Yuan, S. Chan, A. P. Creagh, C. Tong, D. A. Clifton, and A. Doherty, “Self-supervised learning for human activity recognition using 700,000 person-days of wearable data,” 2022.
  * [37] T. P. Harish Haresamudram, Irfan Essa, “Assessing the state of self-supervised human activity recognition using wearables,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 6, no. 3, 2022\.
  * [38] H. Haresamudram, A. Beedu, V. Agrawal, P. L. Grady, I. Essa, J. Hoffman, and T. Plötz, “Masked reconstruction based self-supervision for human activity recognition,” in _Proceedings of the 2020 international symposium on wearable computers_ , 2020, pp. 45–49.
  * [39] H. Haresamudram, I. Essa, and T. Plötz, “Contrastive predictive coding for human activity recognition,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 5, no. 2, pp. 1–26, 2021\.
  * [40] H. Haresamudram, I. Essa, and T. Ploetz, “Investigating enhancements to contrastive predictive coding for human activity recognition,” _arXiv preprint arXiv:2211.06173_ , 2022.
  * [41] S. Suh, V. F. Rey, and P. Lukowicz, “Adversarial deep feature extraction network for user independent human activity recognition,” in _IEEE PerCom_ , 2022.
  * [42] L. Bai, L. Yao, X. Wang, S. S. Kanhere, B. Guo, and Z. Yu, “Adversarial multi-view networks for activity recognition,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 2, pp. 1–22, 2020.
  * [43] H. Qian, S. J. Pan, C. Miao, H. Qian, S. Pan, and C. Miao, “Latent independent excitation for generalizable sensor-based cross-person activity recognition,” in _AAAI_ , vol. 35, no. 13, 2021, pp. 11 921–11 929.
  * [44] J. Su, Z. Wen, T. Lin, and Y. Guan, “Learning disentangled behaviour patterns for wearable-based human activity recognition,” _ACM IMWUT_ , 2022.
  * [45] R. Presotto, G. Civitarese, and C. Bettini, “Fedclar: Federated clustering for personalized sensor-based human activity recognition,” in _2022 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2022, pp. 227–236.
  * [46] H. Cho, A. Mathur, and F. Kawsar, “Flame: Federated learning across multi-device environments,” _ACM IMWUT_ , 2022.
  * [47] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [48] M. Bachlin, M. Plotnik, D. Roggen, I. Maidan, J. M. Hausdorff, N. Giladi, and G. Troster, “Wearable assistant for parkinson’s disease patients with the freezing of gait symptom,” _IEEE Transactions on Information Technology in Biomedicine_ , vol. 14, no. 2, pp. 436–446, 2010.
  * [49] J. Zhao, X. Xie, X. Xu, and S. Sun, “Multi-view learning overview: Recent progress and new challenges,” _Information Fusion_ , vol. 38, pp. 43–54, 2017.
  * [50] R. Chavarriaga, H. Sagha, A. Calatroni, S. T. Digumarti, G. Tröster, J. d. R. Millán, and D. Roggen, “The opportunity challenge: A benchmark database for on-body sensor-based activity recognition,” _Pattern Recognition Letters_ , vol. 34, no. 15, pp. 2033–2042, 2013.
  * [51] A. Reiss and D. Stricker, “Introducing a new benchmarked dataset for activity monitoring,” in _Wearable Computers (ISWC), 2012 16th International Symposium on_ , 2012, pp. 108–109.
  * [52] V. S. Murahari and T. Plötz, “On attention models for human activity recognition,” in _Proceedings of the 2018 ACM International Symposium on Wearable Computers_ , ser. ISWC ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 100–103.
  * [53] A. Abedin, M. Ehsanpour, Q. Shi, H. Rezatofighi, and D. C. Ranasinghe, “Attend and discriminate: Beyond the state-of-the-art for human activity recognition using wearable sensors,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 5, no. 1, Mar. 2021.