# Know Thy Neighbors: A Graph Based Approach for Effective Sensor-Based Human
Activity Recognition in Smart Homes

Srivatsa P<EMAIL_ADDRESS>School of Interactive Computing, Georgia
Institute of Technology, AtlantaE1566B. CODA 15th Floor, Georgia Institute of
TechnologyAtlantaGeorgiaUSA30308 and  Thomas Plötz<EMAIL_ADDRESS>0000-0002-1243-7563 School of Interactive Computing, Georgia Institute of
Technology, AtlantaE1564B. CODA 15th Floor, Georgia Institute of
TechnologyAtlantaGeorgiaUSA30308

###### Abstract.

There has been a resurgence of applications focused on Human Activity
Recognition (HAR) in smart homes, especially in the field of ambient
intelligence and assisted living technologies. However, such applications
present numerous significant challenges to any automated analysis system
operating in the real world, such as variability, sparsity, and noise in
sensor measurements. Although state-of-the-art HAR systems have made
considerable strides in addressing some of these challenges, they especially
suffer from a practical limitation: they require successful pre-segmentation
of continuous sensor data streams before automated recognition, i.e., they
assume that an oracle is present during deployment, which is capable of
identifying time windows of interest across discrete sensor events. To
overcome this limitation, we propose a novel graph-guided neural network
approach that performs activity recognition by learning explicit co-firing
relationships between sensors. We accomplish this by learning a more
expressive graph structure representing the sensor network in a smart home, in
a data-driven manner. Our approach maps discrete input sensor measurements to
a feature space through the application of attention mechanisms and
hierarchical pooling of node embeddings. We demonstrate the effectiveness of
our proposed approach by conducting several experiments on CASAS datasets,
showing that the resulting graph-guided neural network outperforms the state-
of-the-art method for HAR in smart homes across multiple datasets and by large
margins. These results are promising because they push HAR for smart homes
closer to real-world applications.

smart-home, human activity recognition, pattern recognition

††copyright: rightsretained††journal: IMWUT††journalyear: 2023††journalvolume:
0††journalnumber: 0††article: 0††publicationmonth: 0††doi:
XXXXXXX.XXXXXXX††ccs: Human-centered computing Ubiquitous and mobile
computing††ccs: Computing methodologies Machine learning††ccs: Human-centered
computing Empirical studies in ubiquitous and mobile computing

##  1\. Introduction

Human activity recognition (HAR) in the context of smart homes has recently
been regaining interest (Röcker et al., 2011). Two trends have been major
drivers for the resurgence of interest: The first is the proliferation of
inexpensive yet accessible, engaging, and helpful (Wood, 2021a, b; na2, 2022)
smart home products such as Google Home and Amazon Alexa in a large number of
households. Secondly, with a rapidly growing aging population (Organization,
2015), many applications have focused on the field of ambient intelligence and
assisted living (AAL) technologies, aimed at improving the quality of life for
seniors through the use of ubiquitous sensors (Chen et al., 2012). HAR is one
pertinent process in incorporating ambient intelligence in a smart home
environment. It comprises modeling, reasoning, and decision-making procedures
(Augusto et al., 2010; Cook et al., 2009) with the aim of detecting and
identifying complex human activities. Hence, improving the quality of life
with AAL technologies depends on how well an interconnected network of
sensors, which are capable of communicating and learning from user habits, can
be used to detect and then identify a plethora of complex human activities in
real-world settings. This can be achieved through processing collected spatial
and temporal information (Cook et al., 2009; Ranasinghe et al., 2016).

A successful HAR system is one that is able to learn a user’s behavior during
everyday life, often through a network of domotic sensors (e.g., motion
sensors, etc.). The real-world settings of such a system might pose several
challenges: (1) Variability – Sensors can stop working at any time, different
sensors might record different values at different points in time, and
observations of sensors might not necessarily be aligned in time (Horn et al.,
2020; Wang et al., 2011). This can be attributed to practical issues such as
cost-saving measures, intermittent and unexpected failure of sensors, external
forces in systems, etc. (Choi et al., 2020). (2) Sparsity – Some activities
occur rarely and/or activity occurrences may translate into a signal with a
single value from one sensor. (3) Noise and redundancy – The same set of
sensors may be triggered for different activities. (4) Limited data – It is
challenging to collect large amounts of labeled data due to, for example,
privacy concerns (Guhr et al., 2020). The aforementioned challenges motivate
the implementation of machine learning (ML) techniques that are able to
discover knowledge from data and make consistent predictions about human
behavior (Ramasamy Ramamurthy and Roy, 2018). Focusing solely on data-driven
approaches that depend on large real-world datasets (Yuen and Torralba, 2010)
is difficult because of limited data. On the other hand, taking a primarily
knowledge-driven approach, which involves making numerous assumptions, tends
not to be as robust because the assumptions might not hold across different
smart homes (Ye et al., 2015).

HAR in smart homes is a problem requiring the solving of both segmentation and
classification concurrently. Segmentation refers to identifying windows of
interest (contiguous windows of sensor events relevant to an activity) before
performing classification (i.e., recognizing activities associated with the
identified window of interest). This dual problem is hard for several reasons:
(1) different activities have very different time windows (e.g., walking out
of home only takes 5 seconds, while sleeping can take as long as 10 hours),
(2) sparsity – most sensors are inactive in a typical day, (3) sensors have
varying sampling rates – every home sensor potentially has a varied sampling
rate. Consequently, defining an ideal window size is extremely challenging,
leading to poorer performance, especially for HAR applications in smart homes.

Prior works have attempted to address this by (a) using fixed sliding windows
(with limited success) or assuming the presence of an oracle (which is
impractical) and/or (b) relying on alternative encoding techniques (Liciotti
et al., 2020) (Bouchabou et al., 2021b, a). Unfortunately, HAR systems that
rely on the assumption that an oracle can manually segment sensor streams are
not very applicable to real-world scenarios because it is very time-consuming
for residents to identify and filter segments in the deployment scenario
(PlÖtz, 2021). Although a popular alternative in literature is to rely on a
fixed-length sliding window (Ye et al., 2023), we show that modeling discrete
sensor events can outperform approaches with fixed-length sliding windows.

Building on the work by Ye at al. (Ye et al., 2023), we propose a novel graph
neural network that is able to capture relationships between sensors by
pooling node embeddings in a hierarchical manner. Since graphs are a natural
way of representing networks, we posit that learning an expressive graph
structure that models (1) the dynamics of sensor dependencies; (2) how those
relationships evolve over time is key to addressing the main challenges
associated with HAR. Our approach outperforms the existing state-of-the-art
approaches on several publicly available smart home datasets. We also show how
(1) it is more robust to intermittently faulty sensors; and (2) it can be a
preliminary step in building more explainable HAR systems. Our compelling
results across several scenarios and ablations strongly advocate for graph-
based approaches to human activity recognition in smart homes.  The key
contributions of this paper can be summarized as follows:

  1. (1)

A novel and more expressive graph-based activity recognition system for smart
homes that better models inter-sensor relationships;

  2. (2)

A discussion on how to address an irregular sampling of sensor data streams as
it is prevalent in smart homes;

  3. (3)

A thorough experimental evaluation on challenging, real-world smart home
problems which are encapsulated by a variety of CASAS smart home datasets.

##  2\. Background

The primary task of HAR in smart homes consists of recognizing what a human is
doing at what time, thereby analyzing sensor data that have been captured
using a range of modalities, from the ’Internet of Things (IoT)’ to
environmental sensors (Plötz et al., 2011), wearables, and cameras (Hussain et
al., 2019). Given the trends of decreasing sensor costs and increasing demand
for home automation, we focus here on IoT-based HAR.

###  2.1. Problem

IoT-based HAR is an inherently hard problem as it requires solving two sub-
problems concurrently: segmentation (identifying contiguous–in time–sensor
events that correspond to, yet unknown, activities), and classification
(recognizing the resident’s actual activity covered by the previously
identified segment of contiguous sensor events) (Plötz and Guan, 2018). Many
HAR approaches, especially with wearables, rely primarily on the sliding
window method, which is essentially a workaround that circumvents the explicit
segmentation step. Specifically, contiguous sequences of sensor events in
fixed time windows are treated as input to a HAR system. The window sizes are
typically determined by guessing window lengths that most likely capture a
complete activity (e.g., a user wearing a smartwatch running) (Li et al.,
2018). The time window is shifted along with some overlap. It is then
typically, yet not always correctly (Hammerla and Ploetz, 2015), assumed that
sliding windows are independent and identically distributed (i.i.d.).

Unlike wearables HAR, IoT-based HAR in smart homes is different along several
dimensions. Primarily, the sensor data tends to be sparse (i.e., most sensors
remain inactive most of the day) and contain activities that span over a more
varied range of durations. Consequently, no single window length can
effectively capture all activities. Furthermore, unlike wearables, smart home
sensors typically do not have a fixed sampling rate, making it more difficult
to define an ideal window size. For these reasons, sliding window approaches
do not work as well for many HAR applications in smart homes.

Prior works have attempted to improve the effectiveness of HAR in smart homes
by (a) relying on a guessed fixed-length window (a workaround), (b) assuming
access to an oracle that effectively guides the classification systems towards
the portions of a sensor data stream that shall be classified, and/or (c)
using different encoding techniques. We can group works into three broad
categories: (1) Traditional approaches, (2) General deep learning approaches,
and (3) Graph Neural Network (GNN) based methods. Traditional approaches tend
to ignore temporal relationships of sensor activity. More recent deep learning
approaches (Liciotti et al., 2020; Bouchabou et al., 2021b, a) tend to assume
access to an oracle during deployment – the HAR system takes as input manually
segmented windows of sensor events. Though feasible for developing and
studying activity classification tasks, the assumption of having access to
segmented sensor event streams is not valid for practical applications. Recent
GNN methods (Ye et al., 2023), while promising, require fixed-length windows
of sensor events which might not fully cover the varied durations of
activities.

###  2.2. Traditional Approaches to HAR in Smart Homes

A plethora of (conventional) algorithms have been proposed for the task of
sensor-based human activity recognition (HAR), ranging from naive Bayes (NB)
and decision trees (SEDKY et al., 2018) to conditional random fields (CRF),
hidden Markov models (HMM) and support vector machines (SVM) (Cook, 2010).
Clustering-based classification methods have also been proposed (Fahad et al.,
2014) where the k-nearest neighbors algorithm was used to determine resident
activity. Beyond this, several variants of SVM have been shown to outperform
traditional machine learning algorithms such as NB, CRF, and HMM (Cook et al.,
2013). Some authors have developed behavior classification models derived from
SVM classifiers to differentiate/identify residents (Chen et al., 2011). An
SVM classifier using multiple kernels was also proposed to identify individual
activities of residents (Fatima et al., 2013). Some authors focused on time-
space feature importance relevance and used random forests and SVMs to
distinguish relevant features for activity classification (Chinellato et al.,
2016). More complex probabilistic methods have also been proposed that
leverage Bayesian networks (Nazerfard and Cook, 2015), and methods that
estimate prior probabilities of activities happening at different points in
time (Coppola et al., 2016), relying on Gaussian mixture models. A
disadvantage associated with many of these methods is that they typically
require handcrafted feature extraction methods (Baccouche et al., 2011) or
approximate kernel fusion methods (Fatima et al., 2013) – an issue directly
addressed by deep learning methods.

###  2.3. Deep Learning Approaches to HAR in Smart Homes

The key benefit of Deep Learning (DL) methods lies in their ability to uncover
features from raw data (e.g., sensor measurements) (Ramasamy Ramamurthy and
Roy, 2018). Early successful DL works in the field of sensor-based human
activity recognition proposed the use of a Restricted Boltzmann Machine (RBM)
(Plötz et al., 2011), which relied on a single-layer feed-forward network for
feature extraction. Later works go beyond a simple-feed forward network by
suggesting the use of one-dimensional convolutional networks (CNNs) (Hammerla
et al., 2016) and two-dimensional CNNs (Gochoo et al., 2018; Mohmed et al.,
2020) that capture local dependencies in a temporal sequence. Local
dependencies are captured through what is known as parameter sharing across
time – the convolutional kernel used has the same weights across time.
Consequently, CNNs have a restricted ability to capture dependencies between
data samples (Murad and Pyun, 2017).

LSTM networks have become more popular recently because of their ability to
model long-term dependencies in sequences. In fact, they have been shown to be
a very successful variant of recurrent neural networks (RNN) in automatically
learning temporal information from raw data and also achieving reasonable
performance in HAR in smart home settings (Liciotti et al., 2020). The authors
also demonstrate the effectiveness of extensions of LSTMs such as
bidirectional LSTMs and cascading LSTMs, attributing their success to explicit
modeling of multi-modal channels of domotic sensors. Recent work has also
shown the importance of good feature representations (Tahir et al., 2020; Yan
et al., 2019). Bouchabou et al. (Bouchabou et al., 2021a) combine this insight
with a popular natural language encoding technique, namely term frequency
encoding, in generating good feature representations fed to a convolutional
network. Although they seem to achieve state-of-the-art HAR results, many of
these deep-learning methods have drawbacks: A key limitation is their reliance
on oracle segmented sensor data, as outlined earlier. These segmented windows
are often used as inputs to a classification model that identifies a specific
human activity (Liciotti et al., 2020; Bouchabou et al., 2021a). Yet, HAR
systems that rely on manually segmented data are not very applicable to real-
world scenarios because it is not practically possible to identify and filter
segments that can be used for HAR in a deployment scenario.

###  2.4. Graph Neural Networks for HAR

In formulating the problem of HAR in smart home settings, most prior works
implicitly ignore the heterogeneous structure of the data collected from a
network of sensors. At any point in time, only a subset of all sensors will be
activated whereas other sensor measurements are not necessarily meaningful –
which is in stark contrast to, for example, activity recognition scenarios
using wearable sensors. Since different combinations of sensors can fire
together at different points in time when a resident engages in an activity,
sensor data associated with each resident activity is implicitly variable.
Prior works ignore this heterogeneous structure. Apart from TLGAT (Ye et al.,
2023), which attempts to construct a graph attention network across space and
time, the state of the art methods for HAR in smart homes are convolutional
neural networks (CNNs) (Bouchabou et al., 2021b, a) and/or variants of
recurrent neural networks (RNNs) (Liciotti et al., 2020), which do not factor
in relational inductive biases and assume a fixed input size. In this context,
relational inductive biases refer to assumptions that impose constraints on
relationships and interactions between sensors. For example, when a resident
of a smart home decides to watch television in the living room, their activity
would probably trigger the living room light sensor and binary sensor
indicating if a couch is being used. In the context of this example, imposing
relational inductive biases translates into encoding that the couch sensor and
light sensor co-fired, and hence are both correlated (i.e., dependent on each
other).

In real-world settings, where only limited labeled data is available and
sensor data is highly variable, inductive biases are an excellent way to train
models that generalize well (Battaglia et al., 2018); better generalization
leads to better adjustment to variations in resident behavior, leading to more
robust activity recognition systems in smart homes. When working with
heterogeneous data, relative inductive bias can be introduced through guiding
models in learning dependencies between sensors. Several works in HAR use
wearable sensor data and skeleton data that exemplify this: variants of
spatial-temporal graph convolution network (Hedegaard et al., 2023; Han et
al., 2019), variants of residual graph convolutional networks (Yan et al.,
2022; Mondal et al., 2020). While graph convolution networks have been
successful in HAR for wearables, we show in this work that they tend not to be
as successful as attention-based graph neural networks in HAR for smart homes,
especially since attention-based models are able to leverage the prior
knowledge indicating that some neighbors might be more informative than
others.

Knowing how a set of sensors are co-firing (e.g., sensors that typically
trigger and don’t trigger while a resident is watching the television) can
provide more context when deciding the activity of a resident even when only a
subset of the sensors are functioning at a given point in time. Since graphs
are an effective way to model dependencies in a network of sensors, one
meaningful problem formulation involves defining a sensor network as a graph
with nodes representing individual sensors and edges representing
relationships between sensors. This leads to one of the key ideas pertaining
to GNNs – we want to build representations of each sensor, which is some
learned combination of the sensor’s observed data (encoded as a node feature
vector) and the feature vectors of correlated sensors.

Unlike prior works, we extend the work of Graph Neural Networks in the domain
of human activity recognition in smart homes in several ways:

  * •

Our approach does not rely on designing separate networks for individual
features such as time, timestamp, and location. Our approach learns all the
features automatically from raw sensor observations directly. We believe this
is helpful because by not splitting raw sensor data into separate features and
training a unified graph neural network (Section 3) on less sparse data,
learning is improved. This translates into better generalization, which is
shown by the improved performance using our method.

  * •

Secondly, our approach learns correlation relations between sensors across
different features where we explicitly prune edges with low attention weights,
allowing the learned graph structure to retain more pertinent information,
allowing it to generalize better. More details in Section 3.4.1.

  * •

Thirdly, we do away with the use of convolutional filters often found in prior
works (Ye et al., 2023) by relying on the use of residual connections and
hierarchical pooling (section 3.5). Our experiments find that the use of
convolution filters can be extremely sensitive to sensor behavior, window
size, and chosen kernel size, potentially explaining worse performance
(Section 4.2) compared to our approach.

  * •

Another very important distinction lies in evaluation. Most prior works use a
70-30% split for training and test data with a random shuffle (Bouchabou et
al., 2021b; Baccouche et al., 2011; Ye et al., 2023). Unfortunately, this
method is not representative of practical scenarios and appropriate for time
series data because of temporal dependencies (Hammerla et al., 2016). We show
evaluations across several pertinent prior works using forward chaining. More
details in Section 4.1.2.

##  3\. Graph-guided networks for activity recognition in Smart Homes

###  3.1. Overview

A successful HAR system in a smart home is one that is able to leverage a
multitude of such sensors to effectively recognize user activities despite the
challenges posed by a real-world deployment. We argue that explicitly modeling
relationships and interactions between sensors is a crucial step in addressing
issues arising from variability, noise, redundancy, and sparsity in sensor
measurements, especially when labeled user activity data is scarce. Knowing
how different sets of sensors are co-firing can provide more context when
determining the activity of a user even when sensors malfunction (e.g., due to
poor network connectivity) or get activated for a short duration (e.g., a door
sensor when the resident is leaving the home).

An elegant way to model relationships between an arbitrary number of entities
is through graphs (cf. Sec. 2.4 ). We propose a novel graph-based method to
model relationships between sensors and then leverage the graph to perform HAR
in smart homes (see Fig. 1 for an overview). Our approach consists of four
main components:

  1. (1)

Encoding. What: A way to encode each sensor’s values into a vector that
encodes both temporal information as well as sensor measurements into a
vector. Why: Mapping observations to a high dimensional space using a non-
linear transformation allows for sufficient expressive power (Veličković et
al., 2017).

  2. (2)

Sensor Embedding Generation. What: A sensor-specific transformation to capture
the unique characteristics of each sensor. Why: Values recorded by each sensor
can follow different distributions and have distinct patterns – hereon
referred to as ’sensor behavior’.

  3. (3)

Attention-based Graph Structure Learning. What: A directed graph to represent
the dependency relationships between sensors. It is learned through an
attention function that attempts to quantify strength of relationships between
sensors. Why: Modeling dependency relations can provide more context for the
task of human activity recognition. The graph is directed because dependency
patterns need not necessarily be symmetric. Attention is needed since not all
sensor dependencies are the same; some are perhaps more important compared to
others.

  4. (4)

Activity Recognition. What: A modified fully connected neural network that
uses hierarchically pooled information encoded in the learned graph to predict
user’s activity. Why: The resultant graph from the previous step contains
several dimensions of information: each node encodes temporal information and
measurements from one sensor as well as weighted contributions from its
neighbors; each edge encodes the strength and presence of dependence between
sensors. By iteratively clustering and aggregating node embeddings using a
fully connected neural network, the HAR system would be better able to piece
all the dimensions of data encoded in the graph to make an informed prediction
on a user’s activity.

In what follows we provide detailed descriptions for each of the components of
our approach.

Figure 1. Overview of the proposed GNN-based approach to human activity
recognition in smart homes. Inputs $\\{x_{1},\ldots x_{N}\\}$ correspond to
each of the $N$ sensor observations. (1) Preprocessing: Perform forward
imputation if necessary. (2) Encoding: In the encoding step, an encoder
applies non-linear transformations to inputs $\\{x_{1},\ldots x_{N}\\}$ to
generate representation vectors, $\\{h_{1},\ldots h_{N}\\}$. (3) Sensor
Embedding Generation: The encoded inputs are then used to generate sensor-
specific embeddings $\\{s_{1},\ldots s_{N}\\}$. (4) Attention-based Graph
Structure Learning: The sensor-specific embeddings are used in learning
dependency relations i.e. the edges between nodes. (5) Activity Recognition:
All sensor embeddings are combined with the learned graph structure
$\\{z_{1},\ldots z_{N}\\}$ using a modified one-layer feed-forward neural
network to predict user activity (we perform a hierarchical pooling of node
embeddings so as to maximize the learning of global and local sensor
relations).

###  3.2. Encoding

As outlined as one of the challenges in Section 1, variability in sensor
measurements can manifest as signals that may be missing at random periods and
for arbitrary durations. Several ways to address the issue of irregular
sampling of sensor signals have been developed. Traditional imputation
techniques, which are based primarily on averaging (Liao et al., 2009) or
linear regression (Aydilek and Arslan, 2013), do not necessarily encode the
complexity that plagues smart home sensor signals. More robust approaches tend
to be: (a) probabilistic, such as Gaussian Processes (Li and Marlin, 2015,
2016; Futoma et al., 2017); (b) kernel-based (Lu et al., 2008); or (c) deep
learning based, such as Generative Adversarial Networks (GANs) (Lin et al.,
2020; Luo et al., 2018; Yoon et al., 2018a), Recurrent Neural Networks (RNN)
(Cao et al., 2018; Yoon et al., 2018b; Che et al., 2016), and attention
mechanisms (Liu et al., 2019). A major challenge associated with both
probabilistic and kernel-based approaches for imputation is the complexity
that arises from designing appropriate kernel functions or covariance
functions for Gaussian processes in the multivariate case. In practice,
because there is often limited labeled data when it comes to HAR in the
context of smart homes, training GAN-like generative approaches to convergence
is extremely challenging (Mescheder et al., 2018). End-to-end deep learning
approaches such as Neural Ordinary Differential Equation (ODE) networks (Chen
et al., 2018; Kidger et al., 2020) and Gated Recurrent Unit (GRU) networks
with hidden states decayed towards zero (Che et al., 2016) add too much model
complexity, making the HAR system more prone to overfitting to a specific
resident or smart home environment.

We found that a straightforward solution that adds minimal complexity and
works well is forward imputation, i.e., assuming any missing value is the same
as its last measurement and thus use the most recent sensor measurement.
Hence, the main data transformation we perform to raw sensor events is to
perform forward imputation (the first step in Fig. 1). Discrete sensor events
are then fed into an encoder that is able to capture contextual information
across time. The encoder converts $x_{i}$ to $h_{i}$ as shown in the second
step in Fig. 1. Since variants of Recurrent Neural Networks (RNN) such as LSTM
and Bidirectional LSTMs are known to be effective in capturing long-term
dependencies, we choose to use an RNN-based encoder to extract feature
vectors, $\\{h_{1},\ldots h_{N}\\}$, from the raw sensor data,
$\\{x_{1},\ldots x_{N}\\}$, as shown in step 2 of Fig. 1.

###  3.3. Embedding Generation

Across various smart homes, different sensors might have different behaviors
and characteristics along several dimensions: (i) frequency of activation,
e.g., sensors placed along hallways might trigger more frequently compared to
sensors at the door for a resident who mostly stays at home; (ii) periodicity
of triggering, e.g., bedroom sensors might get activated daily at night while
the guest bathroom sensors might only activate during the holiday season when
a guest visits; and (iii) range of measured values – discrete-value sensors
such as binary sensors with two values (On and Off) versus continuous-value
sensors such as temperature sensors and air quality sensors which can take a
wide range of values. We refer to variations along these dimensions as ‘sensor
behavior‘.

In order to capture the aforementioned dimensions of variations in ‘sensor
behavior‘, our approach includes an embedding layer with a size, $N$ – the
number of sensors. Each sensor’s embedding, $s_{i}$, is defined as follows:

(1) $\displaystyle s_{i}$ $\displaystyle=ReLU(h_{i}\cdot W_{i})$ (2)
$\displaystyle s_{i}$ $\displaystyle\in\mathbb{R}^{d},\text{for
}i\in\\{1,2,3,\ldots N\\}$

where $ReLU$ is a non-linearity function, $W_{i}$ is a sensor-specific
trainable weight vector, and $d$ refers to the dimension of the embedding.
Together, the non-linear transformation projects feature vector
$h_{i}\in\\{h_{1},\ldots h_{N}\\}$ from the encoding step (Section 3.2) to a
new feature space. In this new feature space, sensors that have similar
’sensor behaviors’ share similar embeddings, indicating a high tendency to be
related to one another. The learned embeddings are used to construct the graph
structure, which maps the relationship between sensors. They are also used to
compute attention scores, which determine how much each sensor affects the
values of related sensors – a way to address variability in sensor
measurements, e.g., when sensors malfunction. Initialized randomly, the sensor
embeddings, which are essentially semantic clustering, are learned alongside
the rest of the model.

###  3.4. Attention-Based Graph Structure Learning

Knowing how a set of sensors are co-firing can provide more information when
determining the activity of a user in real-world settings. By explicitly
modeling relationships and interactions between sensors using graphs, and then
leveraging the graphs to recognize human activities, we believe that HAR
systems would be better equipped in addressing issues arising from
variability, noise, redundancy, and sparsity. For example, when sensors
malfunction due to poor network connectivity or get activated for an extremely
short duration (e.g., a door sensor activating only when a user leaves home),
being able to rely on information from other sensors would allow a HAR system
to have sufficient context in predicting user activity. Concretely, we refer
to the process of modeling relationships between sensors using graphs as
attention-based graph structure learning.

Figure 2. Layout of CASAS Milan smart home with labels (with permission from
(Cook et al., 2012)) with the corresponding subgraph for the workspace &
television room - remaining nodes and edges of the graph are hidden for
brevity.

A graph is a collection of nodes and edges. In the context of our proposed HAR
model, the graph comprises nodes that represent features associated with
specific sensors and edges that indicate relations between sensors. A directed
edge $\overrightarrow{e_{uv}}$ from a sensor $u$ to another sensor $v$ can be
interpreted as sensor $u$ potentially influencing the behavior of sensor $v$.
Since the dependency between $u$ and $v$ might not be symmetric, we explicitly
model a directed graph implemented as an adjacency matrix $A$, where
$A_{uv}\in\\{0,1\\}$ represents the presence of a relationship between sensor
$u$ and sensor $v$: when sensor $u$ captures an observation, sensor $v$ will
receive a neural message. If there is no edge connecting sensor $u$ and sensor
$v$, there is no exchange of neural information between both sensors,
indicating that the sensors are unrelated. As detailed in Section 2.4, neural
message passing is equivalent to incorporating the hidden state (i.e., feature
vectors) of any given sensor $u$ the feature vectors of all potentially
dependent (i.e. co-firing) sensors using a weighted aggregation function.
Serving as an illustrative example, consider the CASAS Milan smart home that
has a regular layout. Each sensor (e.g., M07, M08, and M26) in the house (Fig.
2) has a corresponding node in the subgraph on the right representing its
feature vector. The presence of directed edges, e.g., between M07 and M26
indicates that M07 influences M26, and vice versa. Consequently, the neural
message-passing algorithm incorporates to the feature vector of M26 the
features of M07.

####  3.4.1. Edge Pruning

Since it is reasonable to assume that not every sensor will influence every
other sensor, the dependency relations between sensors need to be determined.
Unlike prior works (Ye et al., 2023), we include an explicit edge pruning step
where at each training step we prune edges between nodes with small attention
weights, allowing the learned graph to primarily focus on sensor relations
that are actually pertinent for the task of activity recognition. We believe
this directly addresses the high level of sparsity present in smart home
sensor observations.  Starting with a fully connected graph where every sensor
$i$ has directed edges going to and coming from all other sensors, we can
remove unnecessary connections – relations that do not necessarily provide
additional information. Note we can also start with a partially connected
graph if we have prior information about sensor relationships. The importance
of edges can be determined by comparing the similarity of sensor embeddings
which were designed to encode sensor behaviors.

Figure 3. Overview of how we perform edge pruning on a 4-node sample graph.
(1) A fully connected or partially connected graph is first constructed. (2)
$sim_{ij}$ between sensors $i$ and $j$ are computed. (3) The similarity scores
of all sensors are sorted in descending order. (4) Only the edges
corresponding to top-k similarity scores are kept and the rest are discarded.

More formally, we use the normalized dot product of sensor embeddings
generated from the embedding layer, which is computed between every pair of
candidate relations (step 2 in Fig. 3) as follows:

(3)
$\text{sim}_{ij}=\frac{s_{i}^{\intercal}s_{j}}{\|s_{i}\|\cdot\|s_{j}\|}\;\text{for}\;j\in
C_{i}$

where $s_{i}$ is the embedding of sensor $i$, and $C_{i}$ is a set of all
potentially related sensors to $i$ (i.e., sensors that co-fire). For each
sensor $i$, we sort and only keep edges corresponding to the top $k$
similarity scores, where $k$ is a hyperparameter (steps 3 and 4 in Fig. 3),
which we have chosen to be 5 through empirical study. Discarding non-top-k
edges translates into updating the adjacency matrix $A$ of the graph.

####  3.4.2. Attention Mechanism

Some sensor pairs can be more closely related (i.e., correlated) to each other
than others. For instance, sensors associated with a ceiling light and fan in
the living room are probably more likely to co-fire when a resident is sitting
and watching television in the living room. Compare this to a ’weaker’
dependence between sensors associated with a bed lamp in the bedroom and
hallway; A resident walking through the hallway could be on their way to carry
out any activity: to the bedroom to read, to the living room to watch
television, or to the kitchen to cook, to name but a few examples. Clearly, it
is reasonable to expect the dependence between light and fan sensors in the
living room to be stronger than the dependence between sensors associated with
the hallway and bed lamp in the bedroom. In order to effectively learn such
nuanced sensor relations, we include attention-based (Vaswani et al., 2017)
weights for every directed edge $\overrightarrow{e_{uv}}$ between sensors $u$
and $v$ where $\\{u,v\\}\in S$ and $S$ is the set of all sensors in a given
smart home.

More specifically, a graph attention-based feature extractor performs a
weighted aggregation of feature vectors from related neighbors to update each
node’s representation $\mathbf{z}_{i}$, i.e., sensor $i$’s hidden state. It is
defined as follows:

(4) $\displaystyle\mathbf{z}_{i}^{(t)}$
$\displaystyle=\mathsf{ReLU}\left(\beta_{i,i}\mathbf{W}\mathbf{x}_{i}^{(t)}+\sum_{j\in\mathcal{N}(i)}\beta_{i,j}\mathbf{W}\mathbf{x}_{j}^{(t)}\right),$

where $x_{i}^{(t)}\in\mathbb{R}^{N\times v}$ and $N$ is the number of sensors,
and $v$ is the window size on input sensor readings. $\mathcal{N}(i)$ is the
set of neighboring sensors of (i.e., sensors related to) sensor $i$, obtained
from the adjacency matrix $A$. $W$ is a trainable weight matrix, which is used
to apply a shared linear transformation to the model input $x_{i}$.
$\beta_{i,j}$ are the normalized (using softmax) attention weights, which are
computed using the following equations:

(5) $\displaystyle\beta_{i,j}$
$\displaystyle=\frac{\exp\left(\mathbf{\phi}\left(i,j\right)\right)}{\sum_{k\in\mathcal{N}(i)\cup\\{i\\}}\exp\left(\mathbf{\phi}\left(i,k\right)\right)}$
(6) $\displaystyle\phi\left(i,j\right)$
$\displaystyle=\mathsf{LeakyReLU}\left(b^{\top}\left(g_{i}^{(t)}\oplus
g_{j}^{(t)}\right)\right)$ (7) $\displaystyle g_{i}^{(t)}$
$\displaystyle=s_{i}\oplus Wx_{i}^{(t)}$

where $g_{i}^{(t)}$ is the concatenation of sensor $i$’s embeddings with the
corresponding transformed feature vector of sensor $i$ and $b$ is a set of
learned coefficients. Note that our approach specifically factors each
sensor’s behavior encoded in the embeddings, $s_{i}$, unlike prior graph
attention mechanisms.

The attention mechanism in conjunction with the edge-pruning strategy allows
us to learn a directed graph that encodes both sensor-specific features as
well as inter-sensor relationships. In the next subsection, we outline how
this rich representation can be used to recognize the activity a user is
engaged in.

###  3.5. Activity Recognition

An effective HAR system that learns dependencies between sensors ought to
answer several questions:

  1. (1)

Are sensors co-firing in patterns we have observed before?

  2. (2)

Which sensors have been active?

  3. (3)

What can we learn about user activities by looking at sensors that are active
as well as those that are not?

These questions can be answered by querying the learned graph structure. More
specifically, we apply a function $f_{pool}$, which pools the information
encoded in the graph to predict user activity. This function ought to
meaningfully map the set of node feature vectors $\\{z_{1},\ldots z_{N}\\}$ in
a learned directed graph to a specific user activity (Step 5 in Fig. 1). A
natural implementation of $f_{pool}$ would be to use a single fully connected
layer since they are known to be universal function approximators (Hornik et
al., 1989). Unlike prior works, we modify the standard fully connected layer
using a differentiable soft assignment of the graph, by mapping nodes to sets
of clusters based on their learned embeddings (in a method similar to (Ying et
al., 2018)). Essentially this translates into nodes being pooled iteratively
to aggregate both local and global graph embeddings. The aggregated embeddings
are then used to learn to map features from sensors and information encoded in
the directed graph to specific user activities. Also, unlike prior works, each
of the $N$ representations
$\\{\mathbf{z}_{1}^{(t)},\cdots,\mathbf{z}_{N}^{(t)}\\}$ are element-wise
multiplied with the corresponding sensor embedding $s_{i}$ and fed into the
fully connected layer of a size equivalent to the total number of desired
activities for classification. This is equivalent to having a skip connection,
minimizing effects of over-smoothing (Oono and Suzuki, 2019). Consequently, we
have a complete HAR system that takes as input discrete sensor measurements
$\\{x_{1},\ldots x_{N}\\}$ (data preprocessing is outlined in Section 4.1.1)
and predicts user activity for that duration. In practice, our system can be
used as is without the need for any oracle manually segmenting windows or for
having to guess the ideal window size of input window sequences.

##  4\. Experimental Evaluation

In this section, we report on our extensive and rigorous experimental
evaluation that aims to test the effectiveness of our proposed approach in
plausible, real-world scenarios, and compare our approach to state-of-the-art
methods. The natural choice for datasets to be explored in a rigorous
experimental evaluation is the CASAS datasets (Cook et al., 2012). They
contain diverse collections of daily activities data collected in both single
and multi-person households with real users thereby covering time spans
between two and eight months and covering a range of demographics – from young
adults to older adults with dementia and pets. The houses from which data was
collected featured a varied set of sensors, including temperature sensors,
motion sensors, and binary sensors such as door sensors (which capture whether
or not a door was opened). Together, the diversity of residents (and their
behaviors) as well as the variety of sensors used make the CASAS datasets
reflective of real-world settings.

The subset of CASAS datasets we utilize for evaluations portray the previously
mentioned four main challenges that are pervasive in real-world settings:
sensor variability, sparsity in measurements, presence of noise, and limited
availability of activity data. In fact, they even contain months of labeled
activities that are severely imbalanced, i.e., certain activities occur way
more frequently than others, rendering the problem more challenging. What
follows are evaluations done using several CASAS datasets: Aruba, Cairo,
Kyoto7, Kyoto8, Milan (Table 1).

Table 1. Details of CASAS datasets used for experimental evaluation.

Properties | Aruba | Cairo | Kyoto7 | Kyoto8 | Milan
---|---|---|---|---|---
Residents | 1 | 2+pet | 2 | 2 | 1+pet
Number of sensors | 39 | 27 | 58 | 61 | 33
Number of activities | 12 | 13 | 13 | 12 | 16
Number of days | 219 | 56 | 46 | 58 | 82

###  4.1. Evaluation Setup

####  4.1.1. Data Preprocessing.

For each of the datasets, we follow a process of cleaning up, which is similar
to Bouchabou et al. (Bouchabou et al., 2021b). Specifically, we address the
following major anomalies found in each of the raw datasets: (i) the presence
of duplicate data, for example, sequences of sensor activations for some days
are repeated; and (2) incorrect ordering of sensor activations, i.e., some
sensor activations do not appear chronologically. Duplicate data are removed,
and incorrectly ordered sensor activations are reordered to be chronologically
consistent. Furthermore, since the raw data only contains the ”start” and
”end” of activity labels at specific timestamps, we process raw data into
steps, where a step for a dataset is defined as the smallest interval in time
between consecutive sensor events. For steps during which no observation data
is available, we use forward imputation, i.e., we use the last known
observation data. More details are provided in the appendix.

####  4.1.2. Evaluation Methodology

Figure 4. Overview of evaluation methodology involving forward-chaining for a
three-fold evaluation.

Unlike prior works (Bouchabou et al., 2021b, a; Liciotti et al., 2020), we are
unable to use stratified K-fold cross-validation method (Mullin and
Sukthankar, 2000) since adjacent segments of observed data are not
statistically independent. Relying on stratified K-fold cross-validation
results in bias towards approaches that preserve similarity between adjacent
segments of time-series data, affecting the generalizability of classifiers
(Hammerla and Ploetz, 2015). Hence, we use a more principled procedure:
forward chaining (Tashman, 2000; Bergmeir and Benítez, 2012). We split each
dataset into three consecutive segments for three-fold evaluation as
illustrated in Fig. 4. In each fold, a subsequence of sensor events is used in
the training process – called train subsequence (highlighted in yellow in Fig.
4). The train subsequence is further split into a training subset (highlighted
in purple in Fig. 4) and a validation subset – 10% of the train subsequence
(highlighted in green in Fig. 4). In the same fold, an adjacent unseen
subsequence is identified for testing (highlighted in red in Fig. 4). In the
next fold, the train and test subsequences from a previous fold are used for
training in the next fold, and an unseen subsequence of sensor events is used
for testing. This process is repeated three times to obtain three different F1
scores that cover all samples. The reported scores are the average of all
three folds of evaluation over several runs.  We compare validation and
training losses to determine if training should terminate early, i.e., before
overfitting. This is achieved through early stopping (Goodfellow et al.,
2016), where training is terminated if the validation loss does not decrease
for $S$ epochs; we chose $S$ to be 15. Setting a larger value for $S$ led to
models overfitting. With early stopping, all models, even though were set to
train for 300 epochs, terminated early – between 50 to 80 epochs, depending on
the dataset. The learning rate is set to 0.001 using the Adam optimizer with a
weight decay of 0.5%.

Model hyperparameters. We have used a batch size of 128. The number of nodes
in the graph is dependent on the number of sensors in the dataset (Refer to
Table 1). We start with fully connected graphs and keep top-5 edges in each
step of edge pruning (Refer to Section 3.4.1). . The window size we use is 20
with a 50% overlap. A window size of 20 in our case means that in each window
there are 20 timesteps, where the duration of each timestep is determined to
be the shortest interval for which any sensor has observations.

###  4.2. Results for Different Evaluation Settings

####  4.2.1. Setting 1: Classic time series classification

The first experiment evaluates how well the proposed approach is able to
identify resident activities using sensor measurements across time windows.
Each of the chosen datasets is pre-processed as outlined in section 4.1.1.

Table 2.  Classification performance on multiple CASAS datasets, on a per-sample basis. Comparisons are made across 5 categories of works: (1) Common recurrent baselines - Long Short Term Memory (LSTM) and 1-dimensional Convolutional Neural Network (CNN), (2) recent Graph Neural Network approaches used in HAR for wearable devices, (3) Language embedding based methods single-resident daily activity recognition in smart homes (4) all other benchmarks for single-resident daily activity recognition in smart homes, and (5) Our proposed approach - which does not require fixed windows or oracle specified time windows. We report the mean and standard deviation of the 3-fold test F1-score across three runs. Methods \Datasets | Kyoto8 | Milan | Kyoto7 | Aruba | Cairo
---|---|---|---|---|---
| F1 score | F1 score | F1 score | F1 score | F1 score
CNN1D | 26.6 $\pm$ 0.66 | 36.6 $\pm$ 0.36 | 74.9 $\pm$ 0.29 | 65.6 $\pm$ 0.43 | 80.5 $\pm$ 1.67
LSTM | 22.7 $\pm$ 1.95 | 30.6 $\pm$ 0.85 | 76.4 $\pm$ 0.88 | 83.7 $\pm$ 0.26 | 81.1 $\pm$ 1.07
TCN-AE (Zamani et al., 2023) | 26.2 $\pm$ 1.22 | 43.5 $\pm$ 2.32 | 69.4 $\pm$ 2.84 | 85.5 $\pm$ 1.01 | 80.44 $\pm$ 1.75
LSTM-CNN (Xia et al., 2020) | 24.9 $\pm$ 0.22 | 40.1 $\pm$ 0.39 | 81.5 $\pm$ 0.72 | 88.9 $\pm$ 0.40 | 83.95 $\pm$ 0.48
GraphConvLSTM (Han et al., 2019) | 30.1 $\pm$ 1.56 | 41.0 $\pm$ 1.98 | 77.3 $\pm$ 0.73 | 87.6 $\pm$ 0.12 | 84.2 $\pm$ 0.11
ResGCNN (Yan et al., 2022) | 52.3 $\pm$ 2.26 | 55.6 $\pm$ 1.34 | 68.6 $\pm$ 1.17 | 85.7 $\pm$ 1.44 | 81.1 $\pm$ 1.03
ST-GCN (Yan et al., 2018) | 55.3 $\pm$ 0.48 | 54.4 $\pm$ 1.03 | 73.2 $\pm$ 0.77 | 85.5 $\pm$ 0.44 | 82.6 $\pm$ 1.40
A3TGCN (Bai et al., 2021) | 57.5 $\pm$ 0.75 | 61.3 $\pm$ 0.57 | 77.4 $\pm$ 0.96 | 85.6 $\pm$ 1.12 | 84.0 $\pm$ 0.16
ELMOBiLSTM(Bouchabou et al., 2021b) | 50.6 $\pm$ 0.83 | 53.1 $\pm$ 0.22 | 70.5 $\pm$ 1.55 | 84.2 $\pm$ 0.99 | 81.8 $\pm$ 1.42
E-FCN (Bouchabou et al., 2021a) | 61.6 $\pm$ 0.34 | 69.9 $\pm$ 1.11 | 78.5 $\pm$ 1.31 | 90.7 $\pm$ 0.33 | 84.6 $\pm$ 0.73
Bi-LSTM(Liciotti et al., 2020) | 27.5 $\pm$ 2.05 | 45.5 $\pm$ 2.69 | 77.5 $\pm$ 1.56 | 90.1 $\pm$ 0.89 | 84.2 $\pm$ 0.47
TLGAT (Ye et al., 2023) | 72.8 $\pm$ 0.63 | 74.5 $\pm$ 1.09 | 83.4 $\pm$ 1.72 | 91.8 $\pm$ 0.17 | 85.1 $\pm$ 0.83
Our GNN Approach | 78.3 $\pm$ 0.95 | 80.4 $\pm$ 0.52 | 88.7 $\pm$ 0.60 | 92.4 $\pm$ 0.34 | 88.7 $\pm$ 0.22

Our graph-guided approach outperforms prior works across several CASAS
datasets. We compare our approach to several other approaches: (1) Common
recurrent baselines - Long Short Term Memory (LSTM) and 1-dimensional
Convolutional Neural Network (CNN), (2) recent Graph Neural Network approaches
used in HAR for wearable devices, (3) Language embedding based methods single-
resident daily activity recognition in smart homes (4) several other
benchmarks for single-resident daily activity recognition in smart homes.
Note: Several of the baselines had to be modified to be suitable for
comparison (e.g. A3TGCN (Bai et al., 2021) was designed for forecasting but
had to be modified to be suitable for classification - change loss function
and replace forecasting head with a classification layer. There are some
related works such as Cook et al. (Cook et al., 2013) who evaluate only on
private datasets to which we do not have access, hence, we cannot compare with
these works.

We have had several key findings. Firstly, several prior works that focused on
deep learning methods relying on natural language processing (NLP) techniques
(Bouchabou et al., 2021b, a), though seemingly effective they require an
oracle to segment windows of sensor events; Fundamentally, these methods
require external segmentation information in form of text labels (e.g., ON,
START, etc.) during deployment. It is unrealistic and impractical to expect
residents to label time windows before performing activity recognition.
Secondly, graph-based methods typically tend to outperform common recurrent
approaches such as Convolutional Neural Networks and LSTMs as can be seen from
the F1 scores in the first two sections of Table 2.

Thirdly, Graph-based methods that rely on graph convolutions tend to perform
worse than graph-based methods that use self-attention at their core. This is
evident from the significant improvement of at least $\approx$ 10% in F1
scores (Table 2) on CASAS Kyoto8, Milan, and Kyoto7 by using Graph Attention
Networks (TLGAT (Ye et al., 2023) and our approach) as opposed to Graph
Convolutional Networks such as A3TGCN (Bai et al., 2021), ST-GCN (Yan et al.,
2018) and, GraphConvLSTM (Han et al., 2019). Unlike TLGAT (Ye et al., 2023),
our approach takes the benefits of a graph attention network a step further in
several ways: 1. our approach is a combination of LSTM encoders combined with
a single unified graph instead of performing 1D convolutions over several
different networks, 2. our approach explicitly relies on learned attention
weights at each training iteration to prune edges do not strongly represent
the correlation relationship between any pair of sensors, and 3. we propose
the use of residual connections (connecting sensor embeddings directly to
graph embeddings), instead of using a set of convolution layers to aggregate
node embeddings. We attribute the improvement in performance to architecture
modifications that allowed our network to minimize the effects of over-
smoothing (Oono and Suzuki, 2019).  Over-smoothing occurs when node-specific
information becomes lost after iterations of message passing (i.e., node
representations converge to indistinguishable vectors). In incorporating skip
connections between sensor encodings (before and after message passing), our
GNN approach doesn’t seem to suffer from over-smoothing. Furthermore, the
hierarchical pooling of node embeddings allows for learning multiple
relationships that leverage the graph structure. Unlike prior works, we do not
just concatenate all node embeddings randomly, we perform soft clustering of
node embeddings iteratively allowing for the co-firing relationships to
persist after pooling. Consequently, our approach is able to achieve
significantly higher than state-of-the-art F1 scores across several CASAS
datasets.

Fourthly, the gain in performance by using our approach as measured by the F1
score is largest in Kyoto8 and lowest in Cairo. This phenomenon can be
explained in part by the difficulty of the datasets. There are approximately 4
times more sensor activations in the Aruba dataset as opposed to the Milan
dataset. Having limited collected data exacerbates the difficulty of HAR since
fewer examples are available to effectively learn user behavior. The fact that
our approach significantly outperforms the state-of-the-art method on both
more challenging datasets such as Milan and Kyoto8, as well as on less
challenging CASAS datasets, is a testament to the robustness of our proposed
approach.

####  4.2.2. Setting 2: Fixed sensors dropped

In smart homes, it is reasonable to expect missing observation data because
sensors can suddenly stop working – perhaps because they malfunction when
exposed to external forces or their battery runs out. We posit that learning
dependencies between sensors can be extremely beneficial in such scenarios. To
this end, we evaluate if our approach is robust to scenarios where observation
data from a subset of sensors is not available. One way to test this would be
to permanently hide all measurements from the most informative sensors in the
test data, keeping the training data unchanged. Training data is (still)
required to learn dependencies between sensors. Informativeness is determined
by how often each sensor is activated.

As shown in Table 3 (under ’Set. 2 F1’ columns), our approach (evaluated on
three CASAS datasets) is very effective. For instance, on Milan and Kyoto7,
our approach achieves F1 scores of 53.8 and 79.9, even when observation data
from 40% of the most informative sensors are missing. Another recent graph-
based baseline (Ye et al., 2023) also remains robust to sensor dropping which
makes the case for graph-based methods, although it still performs worse than
our approach. In comparison, the RNN-based state-of-the-art approach collapses
to an F1 score of only 8.00 and 35.1 correspondingly. Another finding is that
the drop in effectiveness is not as significant (less than 10%) for our
approach (and (Ye et al., 2023)) when the percentage of missing sensor data is
10%, 20%, or 30%) across all three datasets. On the other hand, the decrease
in the effectiveness of the prior state-of-the-art approach is much larger.
For instance, when just 10% of sensor data is missing (Milan), the F1 score of
Deep CASAS BiLSTM drops from 45.5 to 31.7 while graph-based approaches like
ours only drops by less than 5 points, from 80.4 to 75.5.

####  4.2.3. Setting 3: Random sensors dropped

Instead of only hiding observation data from the most informative sensors, as
described in Setting 2 (Section 4.2.2), it is also possible for different
subsets of sensors to malfunction over time. We test this by hiding the
observation data of randomly selected subsets of sensors for each window in
the test set, again with several CASAS datasets. Results are reported under
’Set. 3 F1’ of Table 3. Evaluation for each missing sensor ratio is repeated
10 times to prevent selection bias from strongly influencing the results.

We found that, generally, the effectiveness of both methods was better in
Setting 3 than in Setting 2 because in the former it was still possible for
data from the more informative sensors to help in activity recognition. In the
latter, we systematically removed data from the most informative sensors.
Similar to our finding in section 4.2.2, our approach continues to be robust
even when we randomly remove observation data from up to 40% of the sensors.
In contrast, the effectiveness of non-graph-based approaches such as BiLSTM
drops steeply. In both scenarios, our approach still outperforms or at least
performs as well as other smart home HAR baselines.

Table 3. Classification performance on samples with a fixed set of left-out
sensors (Setting 2) or random missing sensors (Setting 3) on several CASAS
datasets. Set. 2 F1 stands for F1 score obtained in Setting 2, and Set. 3 F1
stands for F1 score obtained in Setting 3. We report the mean and standard
deviation of F1 scores across ten runs.

Missing Sensor ratio | Approach \Datasets | Milan | Kyoto8 | Kyoto7
---|---|---|---|---
Set. 2 F1 | Set. 3 F1 | Set. 2 F1 | Set. 3 F1 | Set. 2 F1 | Set. 3 F1
0% | BiLSTM (Liciotti et al., 2020) | 45.5 $\pm$ 1.42 | 45.5 $\pm$ 1.42 | 27.5 $\pm$ 2.05 | 27.5 $\pm$ 2.05 | 77.5 $\pm$ 1.56 | 77.5 $\pm$ 1.56
TLGAT (Ye et al., 2023) | 74.5 $\pm$ 1.09 | 74.5 $\pm$ 1.09 | 72.8 $\pm$ 0.63 | 72.8 $\pm$ 0.63 | 83.4 $\pm$ 1.72 | 83.4 $\pm$ 1.72
| Our GNN Approach | 80.4 $\pm$ 1.23 | 80.4 $\pm$ 4.68 | 78.3 $\pm$ 0.95 | 78.3 $\pm$ 0.95 | 88.7 $\pm$ 0.60 | 88.7 $\pm$ 0.60
10% | BiLSTM (Liciotti et al., 2020) | 31.7 $\pm$ 2.48 | 32.1 $\pm$ 5.15 | 21.3 $\pm$ 2.11 | 26.2 $\pm$ 2.19 | 66.4 $\pm$ 1.26 | 69.2 $\pm$ 3.56
TLGAT (Ye et al., 2023) | 69.7 $\pm$ 1.89 | 70.2 $\pm$ 4.34 | 67.9 $\pm$ 1.11 | 63.8 $\pm$ 2.53 | 81.3 $\pm$ 0.93 | 82.0 $\pm$ 2.63
| Our GNN Approach | 75.5 $\pm$ 2.85 | 77.6 $\pm$ 4.74 | 75.3 $\pm$ 1.96 | 71.3 $\pm$ 3.96 | 85.8 $\pm$ 0.58 | 86.4 $\pm$ 2.04
20% | BiLSTM (Liciotti et al., 2020) | 24.4 $\pm$ 3.62 | 30.5 $\pm$ 4.93 | 12.0 $\pm$ 2.46 | 20.1 $\pm$ 2.45 | 50.1 $\pm$ 1.34 | 61.7 $\pm$ 3.78
TLGAT (Ye et al., 2023) | 67.1 $\pm$ 1.73 | 67.0 $\pm$ 2.13 | 55.6 $\pm$ 1.33 | 60.4 $\pm$ 3.11 | 80.8 $\pm$ 1.42 | 85.8 $\pm$ 1.98
| Our GNN Approach | 71.3 $\pm$ 2.36 | 72.4 $\pm$ 4.42 | 65.2 $\pm$ 2.33 | 67.4 $\pm$ 3.87 | 85.6 $\pm$ 0.99 | 86.1 $\pm$ 0.39
30% | BiLSTM (Liciotti et al., 2020) | 17.1 $\pm$ 2.95 | 22.7 $\pm$ 4.98 | 10.3 $\pm$ 1.26 | 12.2 $\pm$ 3.74 | 44.3 $\pm$ 1.97 | 59.1 $\pm$ 2.70
TLGAT (Ye et al., 2023) | 62.2 $\pm$ 2.41 | 65.1 $\pm$ 3.38 | 50.0 $\pm$ 1.87 | 51.1 $\pm$ 2.99 | 80.4 $\pm$ 0.88 | 84.5 $\pm$ 0.91
| Our GNN Approach | 67.8 $\pm$ 2.86 | 67.6 $\pm$ 4.87 | 58.5 $\pm$ 2.01 | 55.5 $\pm$ 4.56 | 84.2 $\pm$ 1.66 | 84.5 $\pm$ 0.67
40% | BiLSTM (Liciotti et al., 2020) | 8.0 $\pm$ 3.42 | 11.7 $\pm$ 5.18 | 8.1 $\pm$ 2.33 | 12.2 $\pm$ 4.67 | 35.1 $\pm$ 1.32 | 57.8 $\pm$ 0.89
TLGAT (Ye et al., 2023) | 53.7 $\pm$ 3.02 | 53.5 $\pm$ 4.26 | 42.4 $\pm$ 1.67 | 40.5 $\pm$ 4.73 | 79.8 $\pm$ 2.07 | 80.9 $\pm$ 0.24
| Our GNN Approach | 53.8 $\pm$ 2.17 | 62.6 $\pm$ 4.88 | 47.5 $\pm$ 1.95 | 40.5 $\pm$ 4.95 | 79.9 $\pm$ 2.17 | 80.9 $\pm$ 0.17

Through both evaluation scenarios (Settings 2 and 3), we show that our graph-
based approach is more robust to different degrees of missing sensor data as
opposed to the state-of-the-art. One takeaway from this is that graph-based
approaches can be more useful than RNN-based methods in real-world deployments
where sensor data can be missing for several reasons not limited to frequent
power cuts and external forces of nature.

###  4.3. Ablation study

Table 4. Ablation studies on CASAS datasets - Kyoto8, Milan, Kyoto7, Aruba and Cairo. The baseline is a 2 layer fully connected network comparable to the graph in terms of the network size (measured by the number of parameters). The graph approach involves the use of the neural message-passing framework (outlined in Section 2.4). Embedding refers to adding the embedding layer (outlined in Section 3.3). Attn refers to the inclusion of the attention mechanism (outlined in Section 3.4.2). We report the mean and standard deviation of F1 scores across five runs. Methods \Datasets | Kyoto8 | Milan | Kyoto7 | Aruba | Cairo
---|---|---|---|---|---
F1 score | F1 score | F1 score | F1 score | F1 score
Baseline | 20.2 $\pm$ 2.47 | 41.3 $\pm$ 1.71 | 54.9 $\pm$ 1.84 | 66.7 $\pm$ 3.07 | 72.6 $\pm$ 0.88
Graph | 71.5 $\pm$ 0.62 | 70.9 $\pm$ 0.89 | 76.8 $\pm$ 0.83 | 83.7 $\pm$ 1.68 | 78.4 $\pm$ 0.84
Graph + Embedding | 73.5 $\pm$ 0.24 | 72.2 $\pm$ 2.11 | 80.6 $\pm$ 0.09 | 87.3 $\pm$ 0.94 | 82.2 $\pm$ 1.11
Graph + Embedding + Attn | 78.3 $\pm$ 0.95 | 80.4 $\pm$ 0.52 | 88.7 $\pm$ 0.60 | 92.4 $\pm$ 0.34 | 88.7 $\pm$ 0.22

Considering that our approach contains several components as outlined in
Section 3.1, we investigated the contributions of the different components to
the effectiveness of HAR. Table 4 summarizes our findings from conducting
several ablations. For example, the graph structure seems to contribute the
most towards being effective in recognizing user activity (measured by an
$\approx$ 30% improvement in F1 score). Although the inclusion of an embedding
layer seems only to improve the F1 score by approximately 2% across several
datasets (comparing Graph and Graph + Embedding in Table 4), the inclusion of
attention mechanism seems to improve F1 scores by at least $\approx 5\%$.
Table 4 shows that all components are necessary and that the proposed graph
structure help improve the effectiveness of our proposed approach across
several diverse CASAS datasets.

##  5\. Discussion

In designing and evaluating the graph-guided neural network presented in
Section 3, we have gained several insights with respect to design decisions
and what future research directions could entail. For brevity, we only discuss
some of these insights that contextualize our work. We have deferred
additional discussion to the appendix.

###  5.1. What are some limitations & future work?

Evaluation. As detailed in Section 4.1.2, our evaluation methodology uses
nested forward-chaining. Specifically, we create several train/test splits and
average the results across all the splits in order to reduce bias in
evaluation and factor temporal dependencies in sensor events. However, in our
evaluation setup there is a possibility for bias arising from delay (in days)
between train and test splits. We minimize the bias by repeating the
evaluation with different values of K in K-fold and averaging it.

GNN architecture limitation(s). The message-passing framework (detailed in
Section 2.4) is a crucial component in learning the graph structure (Step 4 of
Figure 1). One of the unintended effects of message passing is over-smoothing
(Oono and Suzuki, 2019), where a single node’s representation becomes
dominated by too much information from all of its neighboring nodes. In the
context of a sensor network, this is equivalent to irrelevant sensors
corrupting the representation of any single sensor. As a corollary, the
representations encoded by each node in the graph are less useful, leading to
poorer accuracy in recognizing resident activity. One solution is to include
skip connections to minimize loss of information, as detailed in Section 3.5.
Another intuitive solution we incorporate, unlike prior works, is to constrain
information between a node and its top-k most important neighbors – where
importance is determined using attention scores (or attention weights) as
detailed in Section 3.4.2.

Generalizability. Theoretically, our proposed graph approach can be easily
used to bootstrap homes with similar layouts. However, since transfer learning
is out of scope for this work, we do not conduct any studies where we test how
well, learned graph structures can be used in new home layouts. We leave it
for future work.

###  5.2. How scalable is the proposed approach?

One of the main strengths of our approach is its scalability. Since every node
in the graph represents a sensor’s embedding and every edge is the co-firing
relationship between a pair of sensors, removing sensors in a smart home would
be equivalent to removing nodes in the learned graph representation. From
experiments in Sections 4.2.2 and 4.2.3, it is evident that our graph-based
HAR system is pretty robust to both random and specific sensors being removed,
without any retraining. In scenarios where only 1-2 random sensors are
removed, we observed that, with retraining, we can improve the F1 score to
within 2% of the setup where no sensors are removed. The caveat is that if a
specific sensor that is removed is important for a specific activity, we
observe a sharp drop in the effectiveness of activity recognition. For
instance, removing the bedroom sensors altogether resulted in the HAR system
barely being able to recognize the resident activity of ’sleep’ on the CASAS
Milan dataset.

In the case where new sensors need to be added to the system, it is also easy
to add them without having to retrain everything from scratch; only finetuning
is needed to refine the edge connections of the new sensors to all existing
sensors in the learned graph representation. No modifications are necessary to
the training paradigm. If we know that a new sensor added is similar in
behavior to an existing sensor, we can bootstrap the new sensor’s embedding
using the similar sensor’s embedding. We can further improve the effectiveness
of the HAR system by finetuning the graph structure with the existing
attention-based graph structure learning paradigm by starting with a partially
trained graph. In future work, we aim to create datasets with realistic
removal and addition of sensors, using which we can robustly quantify the
scalability of our and other similar works in the context of HAR in smart
homes.

###  5.3. Can we explain the classifications performed by our proposed HAR
system?

Most activity recognition systems are not perfectly accurate. Current HAR
systems are not able to recognize every activity of every resident in
households without any error. Although it is reasonable to expect activity
recognition systems to improve, expecting them to be perfectly accurate seems
unrealistic, specifically in the smart home setting where there is a lot of
diversity in terms of occupants and their behaviors. Unexplained, inconsistent
behavior of activity recognition systems can give rise to smart home
operations that tend to be surprising or even inappropriate to the resident.
Given this context, we investigate the potential for explainability that our
proposed approach might carry.

(a) Attention map

(b) Milan layout (with permission from (Cook et al., 2012))

Figure 5. Explanation for one occurrence of sleeping activity of a resident
from the Milan dataset. The attention map on the left (Figure 5(a)) shows that
sensor 28 (M028) has the highest attention score with the darkest cell on the
map. Looking at the Milan layout on the right (Figure 5(b)), M028 is the
sensor identifying if a resident is on the bed in the Master Bedroom. We can
infer from this the following: Since the resident is on the bed, the HAR
system recognizes that the resident is sleeping. Being able to attribute
recognized activities to its associated sensors through attention can help
shed some light on the decision-making process of the HAR system, in turn,
building trust with such systems.

The graph-guided model inherently allows for more explainable classifications
because we are explicitly modeling both the presence of a relationship between
any two sensors in a sensor network and the strength of such a relationship
using attention. The strength of relationships between sensors, as measured by
the attention score can be very informative in explaining classifications. For
instance, in accurately recognizing the activity of sleeping for the resident
from the Milan dataset, we can identify which sensors were deemed important by
the model. To do so we rely on attention scores (or attention weights), which
are similarity scores computed by taking the normalized dot product of
observations for all pairs of input sensor events as well as between any two
sensors’ feature vectors. If we were to plot the normalized attention scores
(a proxy for relationship strength) between all pairs of sensors in Milan, we
get an attention map as shown in Fig. 5(a). It is evident that sensor 28
(M028) has the highest attention score.

Looking at the annotated layout of the household used in Milan (Fig. 5(b)), we
can see that sensor M028 is located in the master bedroom. From this, we could
infer that for an input window of sensor measurements, the model predicts
sleep as the resident’s activity based on the high sensor activity of the
sensor in the master bedroom.

Explanations have also been useful in determining the need for curriculum
learning. When analyzing incorrect activity predictions, we found that they
often corresponded to high attention weights associated with incorrect
sensors. For example, the activity of leaving home was often incorrectly
identified and the sensors deemed most pertinent were the hallway sensors
(M011 and M019). What makes this outcome not surprising is that not only were
the wrong sensors identified but the corresponding activity was a more
frequently occurring activity such as doing chores. We used this insight to
structure the training process such that we reintroduce difficult training
samples–training samples that more intensely showed the issue of noise and
redundancy–as training evolved. We hope that such explanations not only help
improve training processes but also instill more trust in HAR systems,
especially in extenuating circumstances where such a system is bound to fail.

##  6\. Conclusion

A growing number of applications have focused on Human Activity Recognition
(HAR) in smart homes, specifically in the field of ambient intelligence and
assisted living technologies. Real-world deployments of HAR systems pose
several challenges such as variability, sparsity, and noise in sensor
measurements. The complexity of the aforementioned challenges motivates the
implementation of machine learning (ML) techniques that are able to discover
knowledge from data and recognize human activities. Although deemed effective
in recognizing resident activities, most recent works assume that during
deployment, smart home residents are able to segment sequences of discrete
sensor events before the HAR system is able to perform activity recognition;
We call this the oracle-guided segmentation problem. Unfortunately, relying on
such data is not very applicable to real-world scenarios since it is not
practical to expect residents to identify and filter sequences of sensor data.
Similarly, relying on fixed window lengths is another common limitation of
prior works.

In an attempt to address the aforementioned issues, we have developed a graph-
guided neural network for HAR thereby avoiding the previously mentioned
limitations. The key insights leveraged in our work: (1) explicitly learning
dependencies between sensors in a hierarchical manner can make for a robust
HAR system; (2) discrete sensor events can be effectively used for smart home
HAR without the need for oracle-specified segmentation or fixed window
lengths. In our evaluations, we demonstrated the effectiveness of our proposed
HAR system in several settings. We also showed how our proposed approach
allows for more explainable classifications. Ultimately, we hope that graph-
based approaches become integral in designing HAR systems in real-world
deployments of smart homes.

## References

  * (1)
  * na2 (2022) 2022\.  Smart Home - United States.  https://www.statista.com/outlook/dmo/smart-home/united-states.  [Online; accessed 06-Jan-2023].
  * Augusto et al. (2010) Juan Carlos Augusto, Hideyuki Nakashima, and Hamid Aghajan. 2010\.  Ambient intelligence and smart environments: A state of the art.  _Handbook of ambient intelligence and smart environments_ (2010), 3–31.
  * Aydilek and Arslan (2013) Ibrahim Berkan Aydilek and Ahmet Arslan. 2013.  A hybrid method for imputation of missing values using optimized fuzzy c-means with support vector regression and a genetic algorithm.  _Information Sciences_ 233 (2013), 25–35.
  * Baccouche et al. (2011) Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, and Atilla Baskurt. 2011\.  Sequential deep learning for human action recognition. In _International workshop on human behavior understanding_. Springer, 29–39.
  * Bai et al. (2021) Jiandong Bai, Jiawei Zhu, Yujiao Song, Ling Zhao, Zhixiang Hou, Ronghua Du, and Haifeng Li. 2021.  A3t-gcn: Attention temporal graph convolutional network for traffic forecasting.  _ISPRS International Journal of Geo-Information_ 10, 7 (2021), 485.
  * Battaglia et al. (2018) Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018\.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_ (2018).
  * Bergmeir and Benítez (2012) Christoph Bergmeir and José M Benítez. 2012.  On the use of cross-validation for time series predictor evaluation.  _Information Sciences_ 191 (2012), 192–213.
  * Bouchabou et al. (2021a) Damien Bouchabou, Sao Mai Nguyen, Christophe Lohr, Benoit Leduc, and Ioannis Kanellos. 2021a.  Fully convolutional network bootstrapped by word encoding and embedding for activity recognition in smart homes. In _International Workshop on Deep Learning for Human Activity Recognition_. Springer, 111–125.
  * Bouchabou et al. (2021b) Damien Bouchabou, Sao Mai Nguyen, Christophe Lohr, Benoit LeDuc, and Ioannis Kanellos. 2021b.  Using language model to bootstrap human activity recognition ambient sensors based in smart homes.  _Electronics_ 10, 20 (2021), 2498.
  * Cao et al. (2018) Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. 2018.  Brits: Bidirectional recurrent imputation for time series.  _Advances in neural information processing systems_ 31 (2018).
  * Che et al. (2016) Z Che, S Purushotham, K Cho, D Sontag, and Y Liu. 2016.  Recurrent neural networks for multivariate time series with missing values, CoRR abs/1606.01865.  _URL: http://arxiv. org/abs/1606.01865_ (2016).
  * Chen et al. (2012) Liming Chen, Jesse Hoey, Chris D Nugent, Diane J Cook, and Zhiwen Yu. 2012.  Sensor-based activity recognition.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 42, 6 (2012), 790–808.
  * Chen et al. (2011) Liming Chen, Chris D Nugent, Jit Biswas, and Jesse Hoey. 2011\.  _Activity recognition in pervasive intelligent environments_. Vol. 4.  Springer Science & Business Media.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018.  Neural ordinary differential equations.  _Advances in neural information processing systems_ 31 (2018).
  * Chinellato et al. (2016) Eris Chinellato, David C Hogg, and Anthony G Cohn. 2016\.  Feature space analysis for human activity recognition in smart environments. In _2016 12th International Conference on Intelligent Environments (IE)_. IEEE, 194–197.
  * Choi et al. (2020) Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. 2020.  Learning the graphical structure of electronic health records with graph convolutional transformer. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 606–613.
  * Cook (2010) Diane J Cook. 2010\.  Learning setting-generalized activity models for smart spaces.  _IEEE intelligent systems_ 2010, 99 (2010), 1\.
  * Cook et al. (2009) Diane J Cook, Juan C Augusto, and Vikramaditya R Jakkula. 2009\.  Ambient intelligence: Technologies, applications, and opportunities.  _Pervasive and Mobile Computing_ 5, 4 (2009), 277–298.
  * Cook et al. (2012) Diane J Cook, Aaron S Crandall, Brian L Thomas, and Narayanan C Krishnan. 2012.  CASAS: A smart home in a box.  _Computer_ 46, 7 (2012), 62–69.
  * Cook et al. (2013) Diane J Cook, Narayanan C Krishnan, and Parisa Rashidi. 2013\.  Activity discovery and activity recognition: A new partnership.  _IEEE transactions on cybernetics_ 43, 3 (2013), 820–828.
  * Coppola et al. (2016) Claudio Coppola, Tomas Krajnik, Tom Duckett, Nicola Bellotto, et al. 2016\.  Learning Temporal Context for Activity Recognition.. In _ECAI_. 107–115.
  * Fahad et al. (2014) Labiba Gillani Fahad, Syed Fahad Tahir, and Muttukrishnan Rajarajan. 2014.  Activity recognition in smart homes using clustering based classification. In _2014 22nd International conference on pattern recognition_. IEEE, 1348–1353.
  * Fatima et al. (2013) Iram Fatima, Muhammad Fahim, Young-Koo Lee, and Sungyoung Lee. 2013.  A unified framework for activity recognition-based behavior analysis and action prediction in smart homes.  _Sensors_ 13, 2 (2013), 2682–2699.
  * Futoma et al. (2017) Joseph Futoma, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara O’brien. 2017\.  An improved multi-output gaussian process rnn with real-time validation for early sepsis detection. In _Machine Learning for Healthcare Conference_. PMLR, 243–254.
  * Gochoo et al. (2018) Munkhjargal Gochoo, Tan-Hsu Tan, Shing-Hong Liu, Fu-Rong Jean, Fady S Alnajjar, and Shih-Chia Huang. 2018.  Unobtrusive activity recognition of elderly people living alone using anonymous binary sensors and DCNN.  _IEEE journal of biomedical and health informatics_ 23, 2 (2018), 693–702.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep learning_.  MIT press.
  * Guhr et al. (2020) Nadine Guhr, Oliver Werth, Philip Peter Hermann Blacha, and Michael H Breitner. 2020.  Privacy concerns in the smart home context.  _SN Applied Sciences_ 2, 2 (2020), 1–12.
  * Hammerla and Ploetz (2015) Nils Hammerla and Thomas Ploetz. 2015.  Let’s (not) Stick Together: Pairwise Similarity Biases Cross-Validation in Activity Recognition. In _Proc. UbiComp_.
  * Hammerla et al. (2016) Nils Y Hammerla, Shane Halloran, and Thomas Plötz. 2016\.  Deep, convolutional, and recurrent models for human activity recognition using wearables.  _arXiv preprint arXiv:1604.08880_ (2016).
  * Han et al. (2019) Jindong Han, Yuan He, Juan Liu, Qianqian Zhang, and Xiaojun Jing. 2019.  Graphconvlstm: spatiotemporal learning for activity recognition with wearable sensors. In _2019 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 1–6.
  * Hassan et al. (2018) Mohammed Mehedi Hassan, Md Zia Uddin, Amr Mohamed, and Ahmad Almogren. 2018.  A robust human activity recognition system using smartphone sensors and deep learning.  _Future Generation Computer Systems_ 81 (2018), 307–313.
  * Hedegaard et al. (2023) Lukas Hedegaard, Negar Heidari, and Alexandros Iosifidis. 2023\.  Continual spatio-temporal graph convolutional networks.  _Pattern Recognition_ 140 (2023), 109528.
  * Horn et al. (2020) Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. 2020.  Set functions for time series. In _International Conference on Machine Learning_. PMLR, 4353–4363.
  * Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989\.  Multilayer feedforward networks are universal approximators.  _Neural networks_ 2, 5 (1989), 359–366.
  * Hussain et al. (2019) Zawar Hussain, Michael Sheng, and Wei Emma Zhang. 2019\.  Different approaches for human activity recognition: A survey.  _arXiv preprint arXiv:1906.05074_ (2019).
  * Kidger et al. (2020) Patrick Kidger, James Morrill, James Foster, and Terry Lyons. 2020.  Neural controlled differential equations for irregular time series.  _Advances in Neural Information Processing Systems_ 33 (2020), 6696–6707.
  * Li et al. (2018) Hong Li, Gregory D Abowd, and Thomas Plötz. 2018.  On specialized window lengths and detector based human activity recognition. In _Proceedings of the 2018 ACM international symposium on wearable computers_. 68–71.
  * Li and Marlin (2015) Steven Cheng-Xian Li and Benjamin M Marlin. 2015.  Classification of Sparse and Irregularly Sampled Time Series with Mixtures of Expected Gaussian Kernels and Random Features.. In _UAI_. 484–493.
  * Li and Marlin (2016) Steven Cheng-Xian Li and Benjamin M Marlin. 2016.  A scalable end-to-end gaussian process adapter for irregularly sampled time series classification.  _Advances in neural information processing systems_ 29 (2016).
  * Liao et al. (2009) Zaifei Liao, Xinjie Lu, Tian Yang, and Hongan Wang. 2009\.  Missing data imputation: a fuzzy K-means clustering algorithm over sliding window. In _2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery_ , Vol. 3. IEEE, 133–137.
  * Liciotti et al. (2020) Daniele Liciotti, Michele Bernardini, Luca Romeo, and Emanuele Frontoni. 2020.  A sequential deep learning application for recognising human activities in smart homes.  _Neurocomputing_ 396 (2020), 501–513.
  * Lin et al. (2020) Suwen Lin, Xian Wu, Gonzalo Martinez, and Nitesh V Chawla. 2020\.  Filling missing values on wearable-sensory time series data. In _Proceedings of the 2020 SIAM International Conference on Data Mining_. SIAM, 46–54.
  * Liu et al. (2019) Zongtao Liu, Yang Yang, Wei Huang, Zhongyi Tang, Ning Li, and Fei Wu. 2019\.  How do your neighbors disclose your information: Social-aware time series imputation. In _The World Wide Web Conference_. 1164–1174.
  * Lu et al. (2008) Zhengdong Lu, Todd K Leen, Yonghong Huang, and Deniz Erdogmus. 2008.  A reproducing kernel Hilbert space framework for pairwise time series distances. In _Proceedings of the 25th international conference on Machine learning_. 624–631.
  * Luo et al. (2018) Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. 2018\.  Multivariate time series imputation with generative adversarial networks.  _Advances in neural information processing systems_ 31 (2018).
  * Mescheder et al. (2018) Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018\.  Which training methods for GANs do actually converge?. In _International conference on machine learning_. PMLR, 3481–3490.
  * Mohmed et al. (2020) Gadelhag Mohmed, Ahmad Lotfi, and Amir Pourabdollah. 2020\.  Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data. In _Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments_. 1–7.
  * Mondal et al. (2020) Riktim Mondal, Debadyuti Mukherjee, Pawan Kumar Singh, Vikrant Bhateja, and Ram Sarkar. 2020\.  A new framework for smartphone sensor-based human activity recognition using graph neural network.  _IEEE Sensors Journal_ 21, 10 (2020), 11461–11468.
  * Mullin and Sukthankar (2000) Matthew D Mullin and Rahul Sukthankar. 2000.  Complete Cross-Validation for Nearest Neighbor Classifiers.. In _ICML_. 639–646.
  * Murad and Pyun (2017) Abdulmajid Murad and Jae-Young Pyun. 2017.  Deep recurrent neural networks for human activity recognition.  _Sensors_ 17, 11 (2017), 2556.
  * Nazerfard and Cook (2015) Ehsan Nazerfard and Diane J Cook. 2015.  CRAFFT: an activity prediction model based on Bayesian networks.  _Journal of ambient intelligence and humanized computing_ 6, 2 (2015), 193–205.
  * Oono and Suzuki (2019) Kenta Oono and Taiji Suzuki. 2019.  Graph neural networks exponentially lose expressive power for node classification.  _arXiv preprint arXiv:1905.10947_ (2019).
  * Organization (2015) World Health Organization. 2015\.  _World report on ageing and health_.  World Health Organization.
  * PlÖtz (2021) Thomas PlÖtz. 2021\.  Applying machine learning for sensor data analysis in interactive systems: Common pitfalls of pragmatic use and ways to avoid them.  _ACM Computing Surveys (CSUR)_ 54, 6 (2021), 1–25.
  * Plötz and Guan (2018) Thomas Plötz and Yu Guan. 2018.  Deep learning for human activity recognition in mobile computing.  _Computer_ 51, 5 (2018), 50–59.
  * Plötz et al. (2011) Thomas Plötz, Nils Y Hammerla, and Patrick L Olivier. 2011\.  Feature learning for activity recognition in ubiquitous computing. In _Twenty-second international joint conference on artificial intelligence_.
  * Ramasamy Ramamurthy and Roy (2018) Sreenivasan Ramasamy Ramamurthy and Nirmalya Roy. 2018.  Recent trends in machine learning for human activity recognition—A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 8, 4 (2018), e1254.
  * Ranasinghe et al. (2016) Suneth Ranasinghe, Fadi Al Machot, and Heinrich C Mayr. 2016\.  A review on applications of activity recognition systems with regard to performance and evaluation.  _International Journal of Distributed Sensor Networks_ 12, 8 (2016), 1550147716665520\.
  * Röcker et al. (2011) Carsten Röcker, Martina Ziefle, and Andreas Holzinger. 2011\.  Social inclusion in ambient assisted living environments: Home automation and convenience services for elderly user. In _Proceedings on the International Conference on Artificial Intelligence (ICAI)_. The Steering Committee of The World Congress in Computer Science, Computer …, 1.
  * Rubanova et al. (2019) Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. 2019\.  Latent odes for irregularly-sampled time series (2019).  _arXiv preprint arXiv:1907.03907_ (2019).
  * Salakhutdinov (2015) Ruslan Salakhutdinov. 2015\.  Learning deep generative models.  _Annual Review of Statistics and Its Application_ 2 (2015), 361–385.
  * Schafer and Graham (2002) Joseph L Schafer and John W Graham. 2002.  Missing data: our view of the state of the art.  _Psychological methods_ 7, 2 (2002), 147.
  * SEDKY et al. (2018) Mohamed SEDKY, Christopher HOWARD, Talal Alshammari, and Nasser Alshammari. 2018.  Evaluating machine learning techniques for activity classification in smart home environments.  _International Journal of Information Systems and Computer Sciences_ 12, 2 (2018), 48–54.
  * Shukla and Marlin (2019) Satya Narayan Shukla and Benjamin M Marlin. 2019.  Interpolation-prediction networks for irregularly sampled time series.  _arXiv preprint arXiv:1909.07782_ (2019).
  * Tahir et al. (2020) Syed Fahad Tahir, Labiba Gillani Fahad, and Kashif Kifayat. 2020.  Key feature identification for recognition of activities performed by a smart-home resident.  _Journal of Ambient Intelligence and Humanized Computing_ 11, 5 (2020), 2105–2115.
  * Tashman (2000) Leonard J Tashman. 2000\.  Out-of-sample tests of forecasting accuracy: an analysis and review.  _International journal of forecasting_ 16, 4 (2000), 437–450.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need.  _CoRR_ abs/1706.03762 (2017).  arXiv:1706.03762 http://arxiv.org/abs/1706.03762
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Wang et al. (2019) Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019.  Deep learning for sensor-based activity recognition: A survey.  _Pattern recognition letters_ 119 (2019), 3–11.
  * Wang et al. (2011) Zhen Wang, Yang Zhang, Ai Jiang, Ji Zhang, Zhao Li, Jun Gao, Ke Li, and Chenhao Lu. 2011.  DAMA-Net: A Novel Predictive Model for Irregularly Asynchronously and Sparsely Sampled Multivariate Time Series.  (2011).
  * Wood (2021a) Laura Wood. 2021a.  Global Smart Home Market Research Report (2021 to 2026) - by Product, Technologies, Service, Deployment and Region - ResearchAndMarkets.com.  https://www.businesswire.com/news/home/20211207005627/en/Global-Smart-Home-Market-Research-Report-2021-to-2026.  [Online; accessed 06-Jan-2023].
  * Wood (2021b) Laura Wood. 2021b.  Smart Home Market Size by Product - Global Forecast to 2026.  https://www.marketsandmarkets.com/Market-Reports/smart-homes-and-assisted-living-advanced-technologie-and-global-market-121.html.  [Online; accessed 06-Jan-2023].
  * Xia et al. (2020) Kun Xia, Jianguang Huang, and Hanyu Wang. 2020.  LSTM-CNN architecture for human activity recognition.  _IEEE Access_ 8 (2020), 56855–56866.
  * Yan et al. (2019) Surong Yan, Kwei-Jay Lin, Xiaolin Zheng, and Wenyu Zhang. 2019\.  Using latent knowledge to improve real-time activity recognition for smart IoT.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 3 (2019), 574–587.
  * Yan et al. (2018) Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018.  Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition.  arXiv:1801.07455 [cs.CV]
  * Yan et al. (2022) Yan Yan, Tianzheng Liao, Jinjin Zhao, Jiahong Wang, Liang Ma, Wei Lv, Jing Xiong, and Lei Wang. 2022.  Deep transfer learning with graph neural network for sensor-based human activity recognition.  _arXiv preprint arXiv:2203.07910_ (2022).
  * Ye et al. (2023) Jiancong Ye, Hongjie Jiang, and Junpei Zhong. 2023\.  A Graph-Attention-Based Method for Single-Resident Daily Activity Recognition in Smart Homes.  _Sensors_ 23, 3 (2023), 1626.
  * Ye et al. (2015) Juan Ye, Graeme Stevenson, and Simon Dobson. 2015\.  KCAR: A knowledge-driven approach for concurrent activity recognition.  _Pervasive and Mobile Computing_ 19 (2015), 47–70.
  * Ying et al. (2018) Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. 2018.  Hierarchical Graph Representation Learning with Differentiable Pooling.  _CoRR_ abs/1806.08804 (2018).  arXiv:1806.08804 http://arxiv.org/abs/1806.08804
  * Yoon et al. (2018a) Jinsung Yoon, James Jordon, and Mihaela Schaar. 2018a.  Gain: Missing data imputation using generative adversarial nets. In _International conference on machine learning_. PMLR, 5689–5698.
  * Yoon et al. (2018b) Jinsung Yoon, William R Zame, and Mihaela van der Schaar. 2018b.  Estimating missing data in temporal data streams using multi-directional recurrent neural networks.  _IEEE Transactions on Biomedical Engineering_ 66, 5 (2018), 1477–1490.
  * Yuen and Torralba (2010) Jenny Yuen and Antonio Torralba. 2010.  A data-driven approach for event prediction. In _European Conference on Computer Vision_. Springer, 707–720.
  * Zamani et al. (2023) Somayeh Zamani, Hamed Talebi, and Gunnar Stevens. 2023\.  Time Series Anomaly Detection in Smart Homes: A Deep Learning Approach.  _arXiv preprint arXiv:2302.14781_ (2023).
  * Zhang et al. (2021) Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. 2021.  Graph-guided network for irregularly sampled multivariate time series.  _arXiv preprint arXiv:2110.05357_ (2021).

##  Appendix A Aggregated sensor triggers to explain the difficulty of CASAS
datasets

(a) Kyoto7

(b) Kyoto8

(c) Cairo

Figure 6. Sensor triggers aggregated over a 24-hour period across all days.
Without regard for the function and location of each sensor, if we were to
compare sensor activity in Kyoto7, Kyoto8, and Cairo, it is evident that
sensor activity is more evenly spread in the Cairo dataset - indicated by the
more even spread of colors on the map in Subfigure 6(c). This illustrates how
each of the datasets has a different distribution of sensor triggers over
time, influencing how easy or hard the task of HAR is.

##  Appendix B On the need for forward imputation

The need for forward imputation arises from the irregularity of the collected
sensor data, caused by missing observations, different sampling rates, sensor
failures, etc. (Choi et al., 2020). For instance, irregularities could mean
that data collected from various sensors might not be aligned in time, and/or
different numbers of observations might be recorded at different time
intervals for each sensor. As a consequence, the raw data collected from
sensors in smart homes tend to be irregularly sampled time series data.
Although there are several ways to deal with irregularly sampled time series
data using kernel methods, interpolation, etc. (Schafer and Graham, 2002; Che
et al., 2016; Horn et al., 2020), we chose to use forward imputation between
the smallest intervals in time between consecutive sensor events. This is
because it is a straightforward method to achieve a reasonably effective HAR
system which avoids the need for (1) complex interpolation methods (Shukla and
Marlin, 2019); and (2) overly complex model designs (Rubanova et al., 2019; Li
and Marlin, 2016). Note: there is no reliance on oracle provided segment data
or use of fixed-length sliding windows.

##  Appendix C Additional studies

###  C.1. Does including an auxiliary task improve classification performance?

Considering that it is one of the main challenges associated with real-world
deployments of smart home systems, sparsity is a problem that is worth
addressing. Recall that the sparsity problem refers to the phenomenon where
some activities occur rarely and/or activity occurrences may translate into a
signal with a single value from one sensor. Prior works (Liciotti et al.,
2020; Bouchabou et al., 2021b, a) have circumvented this issue by using a
combination of oracle provided segmentation and weighted resampling of data to
extract meaningful subsets of data for HAR. As highlighted earlier, oracle-
guided segmentation is not viable in deployment scenarios. Furthermore, our
empirical results have shown that weighted resampling (both upsampling and
downsampling) does little to alleviate the problem (at most $\approx 1\%$ on
our proposed GNN and several baselines).

Table 5. Contribution of the auxiliary task of forecasting to the effectiveness of HAR using Our GNN Approach . We report the mean and standard deviation of the 3-fold test F1-score across five runs. Methods \Datasets | Kyoto8 | Milan | Kyoto7 | Aruba | Cairo
---|---|---|---|---|---
F1 score | F1 score | F1 score | F1 score | F1 score
No Forecasting | 78.3 $\pm$ 0.95 | 80.4 $\pm$ 0.52 | 88.7 $\pm$ 0.60 | 92.4 $\pm$ 0.34 | 88.7 $\pm$ 0.22
With Forecasting | 80.7 $\pm$ 0.13 | 81.3 $\pm$ 0.09 | 89.8 $\pm$ 0.37 | 92.7 $\pm$ 0.20 | 88.9 $\pm$ 0.39

Another way to resolve the sparsity problem is to introduce auxiliary task(s).
The intuition is that auxiliary tasks could act as self-supervisory signals to
guide the model during training to focus on useful signals while ignoring
irrelevant sensor measurements. One such auxiliary task is forecasting future
measurements based on historical measurements of each sensor. The hypothesis
was that it is easier to learn better representations for each sensor by
concurrently attempting to classify resident activity and predict future
sensor measurements based on historical measurements. Also, better
representations translate into a higher accuracy on the task of HAR. Although
including such a task gives an improvement in effectiveness, we found that the
accuracy gains are not uniform across datasets. For example, it was more
significant on the Kyoto8 dataset with an approximate 2% improvement in F1
score but negligible ($\sim 0.3\%$) on the less noisy Aruba dataset. We
attribute the difference to the predictability of resident behavior.
Forecasting seems to improve activity recognition results in scenarios where
the resident behavior has more repeated patterns. We have omitted the
forecasting task from our final architecture because it adds unnecessary
complexity to the model especially when there is no guaranteed (significant)
improvement.

###  C.2. Do we really need forward imputation while preprocessing data?

While preprocessing the raw data, we chose to use forward imputation. In
Section 4.1.1, we explained that the reason for using imputation can be traced
back to the irregularity of the collected sensor data. If data collected from
various sensors are not aligned in time or different numbers of observations
get recorded at different time intervals for each sensor, it is reasonable to
expect that the raw data might have intervals of time where there are no
observations. In order to determine the extent to which the lack of imputation
affects the effectiveness of the HAR system, we compared three different ways
of addressing the irregularity issue: (1) No preprocessing – Use the raw data
with no interpolation. (2) A deep learning method, IP-Net (Shukla and Marlin,
2019), which learns to interpolate missing values using a combination of low
pass, high pass, and cross-channel interpolations, relying on historical
trends; and (3) Forward imputation – use the last known observation data. Note
that we were not able to try the more recent state-of-the-art methods for
dealing with irregularly sampled time series data (Horn et al., 2020; Che et
al., 2016; Zhang et al., 2021; Rubanova et al., 2019) since they do not offer
a solution to impute missing values; instead, they propose a complex model
that is not necessarily applicable to the problem of HAR in smart homes.

Table 6. Classification performance on multiple CASAS datasets using different preprocessing techniques - applying forward imputation, using a deep learning based interpolation network IP-Net(Shukla and Marlin, 2019), and applying no preprocessing. We report the mean and standard deviation of the 3-fold test F1-score across three runs. Methods \Datasets | Kyoto8 | Milan | Kyoto7 | Aruba | Cairo
---|---|---|---|---|---
F1 score | F1 score | F1 score | F1 score | F1 score
No preprocessing | 50.9 $\pm$ 0.45 | 64.6 $\pm$ 1.05 | 76.1 $\pm$ 2.10 | 75.3 $\pm$ 1.22 | 69.4 $\pm$ 0.89
IP-Net (Shukla and Marlin, 2019) | 55.3 $\pm$ 0.42 | 68.8 $\pm$ 1.43 | 82.7 $\pm$ 1.84 | 90.4 $\pm$ 0.64 | 83.8 $\pm$ 0.42
Forward imputation (Our approach) | 78.3 $\pm$ 0.95 | 80.4 $\pm$ 0.52 | 88.7 $\pm$ 0.60 | 92.4 $\pm$ 0.34 | 88.7 $\pm$ 0.22

The results in Table 6 show that forward imputation provides a significant
increase in the effectiveness of our approach across several CASAS datasets.
For instance, we see an increase of $\approx 28\%$ in the F1 score on the
Kyoto8 dataset. The deep learning-based approach, IP-Net, seems only to get
close to using forward imputation but does not seem to be better than using
forward imputation. This should not be surprising because IP-Net relies
heavily on historical data to learn how to interpolate. Given that the
duration for which residents’ data is collected in each of the five CASAS
datasets is limited and resident behavior is quite dynamic, it was probably
challenging to effectively learn to interpolate effectively. Despite all the
complexity introduced by using additional deep learning modules in IP-Net it
was not possible to improve the effectiveness of the HAR system. The stark
difference in F1 scores between not performing any preprocessing and using
forward imputation is also expected. Without using forward imputation most
sliding windows would have no recorded sensor activities. It is difficult for
a HAR system to learn any resident behavior when a significant percentage of
input windows have missing sensor readings. A direct implication of this study
is that forward imputation is a straightforward method to achieve a reasonably
effective HAR system, which avoids the need for (1) complex interpolation
methods (Shukla and Marlin, 2019); and (2) overly complex model designs
(Rubanova et al., 2019; Li and Marlin, 2016).

###  C.3. How to address poor performance arising from uneven distribution of
activities in training data?

Not all activities carried out by residents are equally challenging to
recognize automatically. This is because the issues associated with real-world
settings–variability, noise, redundancy, sparsity, and uneven distribution of
activity labels–do not necessarily manifest evenly across all time windows.
For example, in households with a child and a parent, detecting that the
parent is watching television is probably more difficult than if they are
sleeping. When the parent is watching the television, sensors in the bedroom
might get activated because the child might be playing in the bedroom. The
activation of bedroom sensors adds noise to the parent’s activity of watching
television. On the other hand, when both the parent and child are sleeping at
night in the same room, we can expect only a single-bedroom sensor to be
activated. Not activating unrelated sensors translates into lesser confusion
to the HAR system, especially when learning the dependence between sensors. If
some activities are more challenging to recognize than others, e.g., due to
noise, how might we facilitate the learning process to factor in this
difficulty?

We performed a preliminary experiment where we designed a tailored training
process that focuses learning on more difficult examples, improving the
effectiveness of the HAR system. Concretely, it is a variant of curriculum
learning, wherein each epoch of model training, a subset of training samples
is selected based on their difficulty and/or informativeness – quantified by
feedback provided during training (i.e. value of loss during model training).
We define a training sample as a tuple consisting of: (1) all sensor values at
a given time stamp OR a window size $w$ for baselines ending at time $t$; and
(2) its corresponding activity label. At each training step, we record the
loss of each training sample. We define the difficulty of each training sample
as the magnitude of the loss function, i.e., cross-entropy loss between the
model’s predicted activity and the ground-truth label. In the next step,
training examples are sampled for training with a probability value that is
proportional to its difficulty, i.e., the loss value in the previous step.
This process is repeated until the training procedure is completed. Our
preliminary experiments show that the aforementioned curriculum learning
algorithm boosts F1 scores of our graph-based HAR system by at least 5% on
CASAS Milan, CASAS Kyoto7 and CASAS Kyoto8 datasets. In future work, we intend
to further explore curriculum learning-based training paradigms for graph-
based HAR systems.

###  C.4. Do we really need a sequential encoder?

Many turn to sequential models for HAR tasks (Wang et al., 2019;
Salakhutdinov, 2015; Hassan et al., 2018). Given that smart home HAR is based
on the analysis of sequential data, sequential models such as LSTMs and other
Recurrent Neural Network (RNN) variants are popular choices. The often-cited
benefits of LSTMs include the ability to extract highly discriminative
features and learn long-term dependencies. Keeping in mind the benefits
sequential models confer, the initial choice of encoder in our proposed
approach was an LSTM. We evaluated a variety of widely used sequential models
on five CASAS datasets with a fixed window size that corresponds to
20-timestep windows, where each timestep is the shortest duration between
which sensor observations are observed. The reason for this choice is that
most daily activities in the dataset range from short events such as leaving
the house to long activities such as sleeping which can take place for hours,
empirically we have found this window size to be able to capture the diverse
range of activity durations. We keep the comparison fair by ensuring each
encoder had a similar number of parameters to a two-layer LSTM with a hidden
dimension size of 64.

Table 7. Evaluating choice of the encoder in our proposed approach on several CASAS datasets. The mean and standard deviation of F1 scores across 5 runs are reported. Methods \Datasets | Kyoto8 | Milan | Kyoto7 | Aruba | Cairo
---|---|---|---|---|---
F1 score | F1 score | F1 score | F1 score | F1 score
GRU |  75.5 $\pm$ 1.43 |  80.3 $\pm$ 0.08 | 88.7 $\pm$ 0.84 |  91.2 $\pm$ 0.18 |  88.3 $\pm$ 2.94
LSTM |  74.6 $\pm$ 2.86 |  79.2 $\pm$ 0.23 |  84.7 $\pm$ 0.99 |  90.1 $\pm$ 0.23 |  88.3 $\pm$ 2.33
Transformer |  75.5 $\pm$ 1.74 |  79.8 $\pm$ 0.47 |  84.6 $\pm$ 1.01 |  90.4 $\pm$ 0.04 |  88.4 $\pm$ 2.63
Linear |  75.5 $\pm$ 0.25 |  80.1 $\pm$ 0.36 |  85.6 $\pm$ 0.95 |  91.1 $\pm$ 0.15 |  88.2 $\pm$ 2.56
BiLSTM | 78.3 $\pm$ 0.95 | 80.4 $\pm$ 0.52 | 88.7 $\pm$ 0.60 | 92.4 $\pm$ 0.34 | 88.7 $\pm$ 0.22

From Table 7, it is evident that a Linear layer connected to the message-
passing framework of the graph-guided network is as good as or even better as
a choice for encoding raw sensor data as compared to some deep sequential
approaches. For example, using a linear encoder for Kyoto8 dataset is better
than using an LSTM or even a Transformer (Table 7). Although a Linear layer is
better than several deep sequential approaches (LSTM and GRU), the BiLSTM
encoder is still the best choice for encoder across several CASAS datasets,
with the linear encoder not too far behind. The strong performance of a linear
layer hints at an interesting finding: For smart home HAR systems, it is
probably more important to know how sensors are related and the measurements
observed from related sensors instead of needing to capture longer-term
context.