[1]Tanvir Islam

[1]Peter Washington

[1]Information and Computer Sciences Department, University of Hawai’i at
Manoa, 1680 East-West Road, Honolulu, 96822, Hawaii, USA

# Personalization of Stress Mobile Sensing using Self-Supervised Learning

<EMAIL_ADDRESS><EMAIL_ADDRESS>*

###### Abstract

Stress is widely recognized as a major contributor to a variety of health
issues. Stress prediction using biosignal data recorded by wearables is a key
area of study in mobile sensing research because real-time stress prediction
can enable digital interventions to immediately react at the onset of stress,
helping to avoid many psychological and physiological symptoms such as heart
rhythm irregularities. Electrodermal activity (EDA) is often used to measure
stress. However, major challenges with the prediction of stress using machine
learning include the subjectivity and sparseness of the labels, a large
feature space, relatively few labels, and a complex nonlinear and subjective
relationship between the features and outcomes. To tackle these issues, we
examine the use of model personalization: training a separate stress
prediction model for each user. To allow the neural network to learn the
temporal dynamics of each individual’s baseline biosignal patterns, thus
enabling personalization with very few labels, we pre-train a 1-dimensional
convolutional neural network (CNN) using self-supervised learning (SSL). We
evaluate our method using the Wearable Stress and Affect prediction (WESAD)
dataset. We fine-tune the pre-trained networks to the stress prediction task
and compare against equivalent models without any self-supervised pre-
training. We discover that embeddings learned using our pre-training method
outperform supervised baselines with significantly fewer labeled data points:
the models trained with SSL require less than 30% of the labels to reach
equivalent performance without personalized SSL. This personalized learning
method can enable precision health systems which are tailored to each subject
and require few annotations by the end user, thus allowing for the mobile
sensing of increasingly complex, heterogeneous, and subjective outcomes such
as stress.

###### keywords:

Mobile Sensing, Affective Computing, Personalized Machine Learning, Self-
Supervised Learning, Biosignals, Stress Prediction

##  1 Introduction

Chronic stress can drastically affect an individual’s health across several
dimensions. Continuous stress increases the risk of cardiovascular disease,
hypertension, reduced immunity, and cancer [1]. Despite these detrimental
physical effects, stress is often left unmanaged. A growing body of work in
the field of human-computer interaction (HCI) is focusing on mobile sensing of
stress using signals from wearable consumer devices [2, 3, 4, 5, 6, 7]. Real-
time detection of stress via wearable devices can enable stress interventions
which can provide therapeutic support via immediate prompting when stress is
detected. Until recently, these mobile sensing methods of subjective outcomes
have required large amounts of training data from each user, making the
deployment of such models infeasible in practice. We present a novel
methodology for personalizing mobile sensing models using a minimal number of
human labels.

Consumer wearable devices offer a range of biosignal measurements, which may
include but are not limited to EDA, electrocardiograms (ECG), electromyography
(EMG), respiration rate, and galvanic skin resistance (GSR). EDA is a
physiological measurement of the electric current that flows through the skin.
EDA is particularly sensitive to changes in skin ionic conductivity caused by
even minor perspiration that is not visible on the surface of the skin [8]. In
contrast to other organs in the body that are connected to the parasympathetic
branch, human skin, including its sweat glands and blood vessels, is
exclusively supplied by the sympathetic nervous system [9]. In comparison to
other physiological measurements like heart rate variations and blood
pressure, EDA is a robust way to gauge physiological arousal and, by
extension, the response to stress. In the field of affective computing, EDA is
widely recognized as a key marker of physiological arousal and stress
responses [10][11]. Therefore, an EDA-based stress quantification approach is
the primary focus of this investigation.

Deep neural networks (DNNs) have been a driving force behind major machine
learning progress in recent years [12]. DNNs are multi-layer computational
models that may learn progressively complex and high-level representations of
the input data to accomplish a prediction goal. DNNs have been revolutionary
in fields as diverse as speech recognition [13], computer vision [14], NLP,
strategic games [15], and medical domain [16]. DNNs are an especially
promising solution for analyzing biosignals due to their ability to spot
patterns and learn meaningful features from unprocessed data inputs without
requiring extensive data preprocessing or manual feature engineering.

Creating a stress prediction model using biosignal data that is generalizable
is impractical due to the varying physiological responses to stress among
individuals. As a result, stress monitoring systems must possess the
flexibility to accommodate these differences [17]. Rather than building a one-
size-fits-all stress prediction model, we propose a novel approach aimed at
developing individual-specific models, with the goal of achieving greater
personalization. Such models can enable precision health mobile sensing
systems to reach clinically useful performance.

A major hurdle in developing personalized mobile sensing models through
biosignal data is the procurement of adequate supervisory data (i.e., ground
truth labels) to train DNNs, which require large training sets for the
training process to converge. Self-reported stress labels are often generated
by laborious and faulty manual human labeling owing to human subjectivity and
weariness [18].

Current research on EDA-based stress assessment tends to focus on supervised
deep learning. Achieving clinically useful model performance is hindered,
however, by the fact that traditional supervised learning does not make use of
all of the unlabeled data that are available. Individuals who use wearable
devices have copious amounts of unlabeled longitudinal body signal data, but
only a few labels to indicate important health events. These labels are used
for training machine learning models to recognize the events of interest.

To bridge this gap, we propose the use of SSL to learn neural network weights
which are optimized to make predictions about the baseline temporal dynamics
of a user’s biosignals. Such weights can then be efficiently fine-tuned to
personalized prediction tasks using that user’s biosignal data. We implement
an instantiation of this SSL paradigm and then fine-tune the resulting models
to the stress prediction task using EDA signals. In particular, we make the
following contributions:

  * •

We introduce a novel “personalized SSL” paradigm to address the common real-
world situation of large datasets with few training labels. This solution
enables mobile sensing systems to predict traditionally subjective classes
such as stress. This interaction paradigm is similar to how Apple’s Face ID
works: once the face recognition system is provided with only a few examples
provided by the end users, it can be used repeatedly to unlock the device. In
this case, the labels are stress predictions rather than identity predictions
and the model input is wearable biosignals rather than a face image.

  * •

We evaluate this novel mobile sensing methodology on a real-world biosignals
dataset with stress labels: the Wearable Stress and Affect Detection (WESAD).

Once the model has been calibrated, it can be used repeatedly to predict how
stressed that person will be in different scenarios. The stress prediction
task requires less computing power because the model uses the model weights
learned during the training process to efficiently fine-tune towards stress
prediction. This reduced need for computing resources means that the model can
be processed faster, making it easier to use in real-world settings.

##  2 Related Work

###  2.1 Stress Assessment using Biosignals

Hyperactivity of the nervous system is associated with chronic stress and is
responsible for its wide range of adverse effects on health and behavior.
Neuroimaging research has shown a possible connection between HRV and areas of
the brain that are responsible for assessing stressful situations such as the
ventromedial prefrontal cortex [19]. It was hypothesised by He et al. that
HRV-based approaches may be used to identify acute cognitive stress, who
demonstrated that acute cognitive stress may be detected in real-time using
HRV [20].

Stress and brain activity are closely correlated [21]. Single-channel
electroencephalography (EEG) headsets, mental strain, and a public-speaking
assignment serve as stimuli in research which attempts to quantify the range
of mental stress experienced by study participants [22]. Statistical analysis
of EEG data reveals the existence of the alpha and beta bands which are useful
for classifying stress. To quantify mental strain, researchers have suggested
using EEG and the Hilbert-Huang transform as input features for a support
vector machine (SVM) classifier [23]. It has been demonstrated by Halim et al.
that there exists a framework for measuring driver stress using EEG [24]. EEG
data were captured to monitor the driver’s continuing brainwave activity while
driving with the goal of establishing a relationship between the driver’s
brain activity and their emotional reaction [24].

EMG is a biological signal that measures the electrical charge generated by
engaging human muscles to reflect the neuromuscular activity. Karthikeyan et
al. discuss the results of an EMG investigation that analyzed the link between
a person’s fluctuating stress levels and the tension in their muscles [25].

ECG is the most common biosignal used for monitoring heart activity. ECG is
not only a reliable indicator of heart rate but also of stress levels in
humans [26]. A method for gauging human stress using a mono-fuzzy index
derived from ECG data is described in [27]. In [28], Walocha et al. looked at
ECG data and presented a person-independent and a person-specific strategy to
stress prediction, respectively. The authors studied subjects to stress while
driving manually while multitasking, and while driving autonomously. Hannun et
al. show that single-lead ECGs can be used to identify rhythms using an end-
to-end deep learning method. They demonstrate that the 1D CNN model
architecture can be successfully applied towards making predictions from
biosignal data without manual feature engineering and feature selection [29].

Variations in the skin’s electrical characteristics are reflected by EDA,
making it a helpful stress indicator for the nervous system. Sweating is a
natural response to stressful situations and may further enhance the skin’s
conductivity [30]. Spathis et al. describe a stress prediction framework that
enables consumers to receive feedback on their stress levels through speech-
enabled consumer devices like smartphones and smart speakers [31]. Using data
from the Empatica E4 device included in the publicly available WESAD dataset
[32], Spathis et al. offer a stress prediction framework triggered by issues
such as money, work, and personal responsibility. The design of a wearable
sensor system for mental stress prediction is presented in [33], which
includes a two-channel EDA sensor and a two-channel ECG sensor. EDA sensors
are located on the hands, whereas ECG sensors are located on the chest. Data
from the EDA sensors is wirelessly transferred to the ECG sensor through WiFi.
This method attempts to improve stress prediction by capturing two EDA
channels to eliminate motion artifacts. The designed sensor yields excellent
linearity and jitter performance and exhibits minimal delays during wireless
transmission. Zhu et al. studied the possibility of detecting stress using EDA
data acquired from wearable sensors[34]. The study makes use of two publicly
accessible datasets: VerBIO [35] and WESAD [32], both of which comprise of EDA
signals from research-grade wearable sensors. The experimental results
demonstrate that Random Forests differentiate stress from non-stress
conditions with an accuracy of 85.7%, highlighting the potential for wearable
devices with EDA sensors to anticipate stress. The study also evaluates the
dependability of training with extracted statistical and EDA-related features
from EDA signals, discovering no consistent trend between employing complete
features and chosen features. The findings indicate the effectiveness of
placing EDA sensors into wearable devices for stress prediction, instilling
confidence that incorporating EDA sensors into smartwatches is a potential
option for future product development. Sanchez-Reoli et al. review stress
prediction systems based on EDA and machine learning (ML). Sanchez-Reoli et
al. present two basic ways for evaluating an individual’s emotional state
after reviewing 58 chosen articles: utilizing only EDA for prediction and
integrating EDA with additional physiological data [36]. Both approaches have
pros and cons, with the former providing the advantages of compact non-
invasive equipment and high precision, while the latter provides an improved
mapping of a subject’s physical, psychological, and cognitive condition.

In addition to research grade equipment, EDA is also frequently measured using
consumer wearable devices. Using low-resolution Electrodermal Activity (EDA)
data from consumer-grade wearable devices, Ninh et al. explore the stress
prediction capabilities of user-dependent and user-independent models [37].
This work reveals that stress prediction models employing low-resolution EDA
data from wrist-worn and finger-mounted sensors have high balanced accuracy
scores (66.10% to 100%). This finding indicates that low-resolution EDA
signals from consumer-grade devices may be used to efficiently create user-
dependent stress prediction models, enabling the collection of in-the-wild
data for mental health tracking and analysis. Anusha et al. show that EDA
collected from a wrist device can be used to automatically recognize stress
before surgery [38]. The strategy uses a supervised machine learning method to
find motion glitches in the EDA data and a localized supervised learning model
to predict stress. Anusha et al. demonstrate a novel method for the use of
physiological changes to measure stress in a hospital setting, achieving a
high level of accuracy in detecting stress.

EDA is often used in conjunction with other biosignals. Eren et al. suggest a
way to detect stress by using sensors from the Empatica E4 device used in the
WESAD [32] dataset to measure blood volume pulse (BVP) and electrodermal
activity (EDA) [39]. The suggested neural network model is 96.26% accurate.
The study also calls for more research to test the method’s usefulness and
general flexibility. Pakarinen et al. [40] explored the capacity of
electrodermal activity (EDA) to precisely categorize mental states, including
relaxation, arousal, and stress, in addition to self-perceived stress and
arousal, utilizing a three-phase modified MIST test. The research demonstrated
high levels of accuracy in identifying these mental states, indicating that
EDA has the potential for long-term evaluation of stress and arousal in the
workplace. They explored the utilization of machine learning models for stress
detection and classification, employing features derived from physiological
signals, including EDA and BVP. By achieving an F1 score of 0.71, the study
highlights the potential to create a seamless, real-world stress monitoring
system and promotes continued investigation through the integration of machine
learning and signal processing methodologies in a professional context.
Kalimeri et al. propose an advanced multimodal framework to assess the
emotional and cognitive experiences of visually impaired individuals while
navigating unknown indoor environments, leveraging mobile monitoring and the
combination of EEG and EDA signals [41]. The goal is to identify the
environmental factors that increase stress and cognitive load, thereby guiding
the creation of emotionally intelligent mobility systems that can adapt to
challenging environments using real-time biosensor information.

The study of stress prediction using biosignals is a growing area of interest
in the HCI community. Lee et al. created a wearable in- and over-ear gadget
that monitors EEG and ECG signals concurrently [42]. Convolutional neural
network (CNN) was utilized to distinguish ”stressed” and ”relaxed” based on
EEG and ECG data with an accuracy of around 75%. Hernandez et al. devised a
continuous monitoring method for stress [43]. In this research, two types of
stress have been examined: cognitive load and subjective and personal stress.
This study involved the use of a pressure-sensitive keyboard and a capacitive
mouse. They concluded that in stressed conditions, typing pressure increases
dramatically and there is greater contact with the mouse’s surface. Sarker et
al. introduced a time-series pattern matching approach to identify major
stress occurrences in a time series with discontinuous and quickly variable
stress data [44]. This is the first method for analyzing stress data in the
time-series domain, which helps determine the timing of just-in-time stress
interventions, which are the ultimate goal of stress mobile sensing systems.

###  2.2 Mobile Sensing Systems

There are several mobile sensing systems in the HCI literature for detecting
psychiatric and mental health outcomes using mobile and wearable data streams.
FEEL is a personalized mobile phone application developed by Ayzenberg et al.
which tracks the user’s social interactions and biological data [45]. They
develop customized stress identification software using real-time data and
assign anxiety levels to every occurrence. Both environmental data and stress
levels are pooled in an accessible notebook in which the user may represent
his/her physiological reactions. This study is focused on stress vs. non-
stress conditions, and the evaluation was done by a single user. More subjects
would do a better job of evaluating personalization.

Hinckley et al. discussed sensing strategies inspired by special elements of
HCI with mobile devices [46]. Hinckley et al. conducted experiments on various
mobile device interactions, such as recording memos, switching between
portrait and landscape orientation, and auto-powering the device, and
collected data on how people responded to these functions. It is hard to
personalize as these things move forward to generalization.

MoodScope is a smartphone application that determines the user’s mood based on
their use behavior [47]. LiKamWa et al. demonstrate that a personalized model
is a sole alternative for determining a user’s emotional state, but a generic
model fails when personalization is the essential component. By learning about
its user and linking smartphone use patterns with certain emotions, MoodScope
seeks to utilize trends such as where the user has been, with whom they speak,
and what apps they use. However, it is not always possible to sense the
emotion with the type of using a phone. There is a possibility to fake this
kind of data. In those cases, biosensors, directly connected to the human body
help as it is very hard to fake.

Experience sampling techniques (ESM) are a common technique for collecting
emotional state data from users. Kang et al. conducted the first research on
the consequences of ESM-introduced interrupts [48], demonstrating that 38% of
participants’ emotional state was influenced by the ESM surveys while they
were completing them [48]. The challenge of ESM is that the records can be
biased due to fluctuating emotional states.

Figure 1: Overall process of our personalized stress prediction framework. The
raw signal is segmented into distinct but overlapping data points. A
forecasting model is used for self-supervised pre-training. These weights are
fine-tuned to the stress prediction task (right model). We compare this model
against a supervised learning model without self-supervised pre-training
(left).

###  2.3 SSL on Biosignals

The field of SSL on biosignals is an emerging field of study. Two popular SSL
approaches, SimCLR [49] and BOYL [50], are contrastive learning approaches
which rely on data augmentation. This methodology contains limitations such as
invariance, which can obscure valuable detailed information necessary for
specific downstream tasks. Additionally, predicting which augmentations will
yield optimal benefits can prove to be challenging [51]. As a result, in our
study, we have excluded the use of data augmentation and chosen instead to
focus on pre-training and downstream tasks with raw data. Robert et al. offer
a contrastive learning-based stress prediction system that uses modified EDA
signals to classify stress and non-stress circumstances [52]. Nevertheless,
the current project differs from this approach because it does not require any
signal transformation. Instead of only stress vs. non-stress categorization,
our presented study allows for the learning of a wide range of stress levels.
The Step2Heart system is a self-supervised multimodal wearable data feature
extractor [31]. Using the representations of heart rate in activity data, this
model makes predictions about individual health outcomes using transfer
learning. With the limitations of the biosignals dataset, such as the presence
of noisy labels and small sample size, Cheng et al. turned to a self-
supervised based on contrastive learning [53]. By using the temporal and
spatial information included within ECG impulses, CLOCS uses contrastive
learning to acquire unique representations of each patient [54]. This novel
concept is to consider a positive pair as a representation of two changed
occurrences that both correspond to an identical patient. As a result, the
model automatically adjusts the learnt representations to the specifics of
every patient’s case. Sarkar et al. offer a self-supervised method for
learning ECG representations for the purpose of emotion classification [55].
Banville et al. attempted to learn a patient representation from EHR data and
proposed employing a stacked denoising autoencoder called DeepPatient [56]. In
this paper, Banville et al. reported the first extensive evaluation of self-
supervised representation learning using 12-lead electrocardiogram data in
clinical settings. To accomplish this objective, the researchers utilize
state-of-the-art self-supervised algorithms that rely on instance
discrimination and latent forecasting in the field of electrocardiogram (ECG)
analysis. Banville et al. proposed a novel self-supervised for extracting
comprehensive EEG feature sets [56]. The findings of this research [56]
demonstrate that, particularly under limited data conditions, incorporating
two key features, namely ”relative positioning” and ”temporal shuffling,” can
outperform conventional unsupervised and supervised techniques in a downstream
sleep pacing task.

##  3 Methodology

We propose a two-step process for training mobile sensing models which can
predict subjective labels such as stress using only a few training examples
per user: (1) personalized SSL for representation learning of temporal EDA
dynamics per user, and (2) fine-tuning the pre-trained model to the stress
prediction task. During SSL, the pretext task helps the model learn interim
data representations without ground truth labels. The pretext task is designed
to predict the missing segments of the input data given the remaining
segments. After this representation is learned, the downstream task involves
fine-tuning the weights from the pretext task towards training the neural
network model to predict stress. An overall architecture is shown in Figure 1.

As depicted in Figure 1, the personalized SSL model is designed to accept the
raw EDA signal for each subject. The raw EDA signal is divided into chunks of
7000-dimensional vectors, as described in Section 3.3, generating a large
dataset of 7910 data points to pre-train the model. The pre-training phase
involves the use of a 1D CNN model to predict 40 data points, as detailed in
Section 3.3. The weights learned in the pre-training phase are transferred to
another 1D CNN model which is fine-tuned for the stress prediction task.

In addition, a second 1D CNN model with the same architecture as the fine-
tuned model is employed for stress prediction, but without any pre-training.
This is our control model which we use as a baseline to quantify the benefits
of our SSL over personalization without SSL.

We compare these two models using varying numbers of labeled data points, as
described in Section 4.2 and Section 4.3

We perform a subject-dependent study by directly comparing the following two
models trained and evaluated separately per subject:

  * •

Model A: Stress prediction via supervised learning

  * •

Model B: Stress prediction with a model pre-trained with SSL and fine-tuned
using the same supervised learning strategy as Model A

###  3.1 Dataset

We use the WESAD [32] dataset to evaluate our methodology. WESAD [32] is a
freely accessible multimodal physiological dataset containing ECG, EDA, EMG,
respiration (RESP), core body temperature (TEMP), and three-axis acceleration
(ACC). Biosignals were collected from 15 healthy adults (mean age 27.5;
standard deviation (SD) = 2.4)) when they were subjected to one of three
conditions: neutral, stress, or amusement in a controlled laboratory setting.
The researchers used the Trier Social Stress Test (TSST) to induce anxiety.
Two devices: Empatica E4 and RspiBAN, were used to collect signals for this
dataset. Using the RespiBAN device, data was acquired from each participant
for around 33 minutes at a sample rate of 700 Hz. Multiple sensors, including
an ECG, EDA, EMG, RESP, TEMP, and ACC are included in the RespiBAN device. The
Empatica E4 records sensor data on BVP at 64 Hz, EDA at 4 Hz, TEMP at 4 Hz,
and ACC at 32 Hz. Participants were asked six questions from the State-Trait
Anxiety Inventory (STAI) to derive insight into their degree of anxiety. Each
question was answered using a four-point Likert scale. The STAI questions are
as follows:

  * •

Question 1: I feel at ease

  * •

Question 2: I feel nervous

  * •

Question 3: I am jittery

  * •

Question 4: I am relaxed

  * •

Question 5: I am worried

  * •

Question 6: I feel pleasant

In this study, we used stress data for baseline condition only. Our models
were trained using EDA signal data from the RespiBAN device with a sampling
rate of 700 HZ.

###  3.2 Label Representation

We convert the categorical ordinal outcome variables (i.e., the Likert scale
responses to the questionnaires) into a representation which maintains the
categorical nature of the responses as well as the ordinal nature. In the
WESAD dataset, participants were given six questions from STAI (see Section
3.1 for details). Participants were asked to rate this question on a four-
point Likert scale. Therefore, each question was labelled between $1$ to $4$.
To model the ordinal regression task, we convert the original labels of (1, 2,
3, 4) into the probabilities of (0.25, 0.5, 0.75, 1.0). Because the four-point
Likert chart used in the original evaluation was equally spaced, these labels
are quantified and distributed uniformly. These labels are subjective since
models use only within-subject data and participants are internally consistent
in their labeling (i.e., we assume that a rating of 3 consistently indicates
higher severity than a rating of 2 on the same question within each
participant’s set of ratings).

Figure 2: 1D CNN architecture for pretext task.

###  3.3 Self-supervised Pre-training

The use of self-supervised pre-training, an unsupervised approach in which
data is self-labeled and subsequently fine-tuned using supervised learning
approaches to extract important feature representations, has proven to be more
effective than supervised training alone in an increasing number of
applications [49]. The self-supervised pre-training approach focuses on
learning a high-level representation of data without using ground truth labels
which is also known as the pretext task. A pre-trained model enables efficient
transfer to a target task since the representation it has learned is already
suited (or readily transferable) for a given job. SSL in this manner is the
same technology which powers large-scale language models like ChatGPT and
natural language training tasks powered by BERT [57, 58, 59].

We pre-train our model through forecasting, a similar technique used to pre-
train OpenAI’s GPT language model [60]. A separate model was pre-trained for
each human subject using only the data from that subject. To pre-train our
model, we have employed a 1D CNN, a standard neural network architecture for
signal prediction [61, 62, 63]. Prior research has demonstrated the robustness
of 1D CNN in 1D signal processing tasks, particularly in the domain of
biosignals or 1D signals [64, 29, 65]. One of the key advantages of 1D CNNs is
their ability to automatically learn relevant features from the input signal,
eliminating the need for manual feature engineering. This can save time and
effort while also improving the model’s performance, as it can adaptively
learn the most discriminative features for a given task.

Moreover, 1D CNNs can capture information at different time scales due to the
use of multiple neural layers and multiple independent learned kernels. This
makes this family of architectures particularly helpful when dealing with
signals with complicated hierarchical patterns or events that occur at
different time scales. Additionally, 1D CNNs can handle input data with noise
better than other machine learning algorithms, as the convolutional layers can
learn to focus on the most important features and ignore noise or irrelevant
signals. Figure 2 illustrates the architecture that has been used in this
study for pre-training the model as a knowledge representation of the raw EDA
signal. To maintain consistency, we have used the same model architecture for
both self-supervised pre-training and supervised training. We use Root Mean
Squared Error (RMSE) as our evaluation metric.

The pre-text 1D CNN model is implemented with an input shape of (7000, 1),
consisting of four 1D convolution layers, each utilizing a leaky ReLU
activation function. The first convolution layer has 40 filters, followed by a
layer with 30 filters, and then a layer with 18 filters. The fourth
convolution layer has 30 filters and uses a leaky ReLU activation function.
Moreover, the model included two dense layers, one with 70 nodes and the other
with 30 nodes. The final layer is an output layer with a linear activation
function. The convolution layers are responsible for extracting important
features from the EDA signal, while the dense layers utilize these features to
make a stress prediction.

The model is pre-trained on the entirety of the EDA signal for each
participant. We divide the whole signal into fixed windows (each containing a
7000-dimensional vector) (Figure 3). Figure 3 illustrates an example of data
points in the training set with overlapping information. Assuming that the
signal is represented by $X_{1}\cdots\cdots X_{n}$, each training data point
includes the sequence of $X_{1}\cdots\cdots X_{7000}$, and the corresponding
target is $X_{7001}\cdots\cdots X_{7040}$. The training data points overlap by
100 indices, meaning that the next training data point will be
$X_{100}\cdots\cdots X_{7100}$ while the corresponding target will be
$X_{7101}\cdots\cdots X_{7140}$. Using this process, we produce a total of
7,910 data points for pre-training per human subject. We predict a 40-point
frame using a 10-second window of the original data. The rationale of this
approach is to enable the model to predict stress levels using 10-second time
segments.

Figure 3:  We use signal forecasting as our self-supervised pretext task. A
sliding window is used to predict the following chunk of the EDA signal.

###  3.4 Downstream Stress Prediction Task

Self-supervised pre-training offers the benefit of optimizing data efficiency
during the fine-tuning process for subsequent downstream activities. The
difficulty in sourcing high-quality labels makes this a critical scenario for
use in mobile sensing research. We conduct a thorough quantitative analysis of
this added benefit by contrasting the efficiency of freshly trained, solely
supervised models against that of those pre-trained using self-supervised
representations.

In this stage, the feature extractor layers are frozen, and the rest of the
layers are fine-tuned to the stress prediction task. For this downstream task,
we have modified the model with $3$ new dense layers for the final prediction.
The architecture for stress quantifying is shown in Figure 4. As illustrated
in Figure 4, the fine-tuned model shares the same set of convolution layers
and activation functions as the pre-text 1D CNN model. However, it has three
dense layers, the first one with $50$ nodes, followed by a dense layer with
$30$ nodes, and the final dense layer with $10$ nodes. The output layer has a
linear activation function, and the predicted output is stress.

In the case of purely supervised training, the same architecture as the fine-
tuned model is utilized to facilitate direct comparison between methods.

Figure 4: 1D CNN architecture for fine-tuning the pre-trained network towards the downstream stress prediction task.  Table 1: RMSE for the signal forecasting pre-training task for all subjects. The low RMSE indicates that the model learned the baseline dynamics of each user’s EDA biosignals. S2 |  S3 |  S4 |  S5 |  S6 |  S7 |  S8 |  S9 |  S10 |  S11 |  S13 |  S14 |  S15 |  S16 |  S17
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
0.024 |  0.045 |  0.018 |  0.027 |  0.027 |  0.050 |  0.032 |  0.113 |  0.017 |  0.031 |  0.015 |  0.019 |  0.026 |  0.021 |  0.030

###  3.5 Experimental Procedures

For our experiment, we select a fixed number of random data points having
window size of $7,000$ which corresponds to $10$ seconds of data to train our
model in both SSL and solely supervised manners with exactly the same set of
data points. In both cases, we utilize the same model architecture and the
same sampled data points. To train using SSL, we transferred the weights from
the pre-trained model to the downstream task (Section 3.4). The average RMSE
of the trained model using 10 data points is shown in Figure 5. In this
experiment, all models are subjected to an identical test set.

##  4 Results

We use RMSE to evaluate our models, as regression models commonly utilize RMSE
as a statistic for assessing model performance. For a given test set, a
model’s RMSE measures the average distance between the expected values and the
actual values. It is measured as the square root of the average of the squared
differences between the predicted and actual values. RMSE is helpful because
it penalizes large mistakes over smaller ones and quantifies far apart the
mistakes in the original units of the dataset. A smaller RMSE indicates that
the model performed with less error.

###  4.1 Pre-trained Models

For pre-training purposes, all the models have been trained using data coming
from a single subject. Table 1 demonstrates the RMSE of the pre-trained models
of all $15$ subjects. The RMSE score is a measure of how well a model
performed in predicting values compared to the actual values. For example,
Subject 2’s pre-trained model achieves an RMSE score of $0.024$, meaning that
the model’s predictions are, on average, $0.024$ units away from the actual
values. In other words, the model’s predictions have an average error of
$0.024$ units. The lower the RMSE score, the better the model’s performance. A
smaller RMSE value indicates that the model is making more accurate
predictions, while a higher RMSE value indicates that the model is making
larger errors. The low RMSE values we achieved across subjects suggest that
our pre-trained models are well-trained and have learned the underlying
dynamics of each subject’s EDA signals.

###  4.2 Evaluation using pre-trained models

Figure 5 displays the RMSE score of all participants for each question in the
WESAD dataset while training on a randomly sampled subset of $10$ data points
in both SSL and supervised modes. We conduct bootstrapped sampling using $5$
independent samples, and the same samples are used to compare the SSL pre-
trained and purely supervised models. From Figure 5, we observe that in most
of the cases, the RMSE score is lower for SSL-based models than purely
supervised trained models. We observe that SSL models outperform the purely
supervised models in approximately $90\%$ of cases.

Figure 5: Comparison of the RMSE between a model fine-tuned after SSL vs.
purely supervised training for all subjects and for all questions.

###  4.3 Results for a demonstrative user

We compare purely supervised training and SSL as shown in Figure 6. Our study
shows that models that use SSL techniques require much fewer data to perform
as well as models that do not use SSL techniques. We observe that models that
employ SSL techniques require less than 30% of the data needed by models that
do not use SSL techniques.

We see that models that only use fully supervised training methods need more
data to get to the same RMSE as similar SSL models that are trained with the
same number of labeled data points. This illustrates how SSL methods can speed
up data-heavy applications by making them more efficient.

In addition to the benefits of preserving data, our findings demonstrate that
SSL methods are stable across training runs. The RMSE of the purely supervised
method varies drastically (has a high standard deviation) across bootstrapped
samples, but the performance of SSL models is more stable. The fluctuation of
the purely supervised model indicates that its performance (as measured by
RMSE or other metrics) can vary significantly across different training runs
or data samples. This variability can be problematic in real-world situations
where there may be limited labeled data or data of varying quality. In
contrast, SSL methods offer a more stable way of learning, as the performance
of SSL models is less susceptible to fluctuations and remains more consistent
across training runs or data samples. This stability is particularly useful in
situations where data is limited or noisy, as it allows for more efficient and
reliable model training. Therefore, the finding that SSL models are more
stable than non-SSL models is an important contribution of this study,
highlighting the potential benefits of using SSL methods in real-world
applications.

Figure 6: Comparison of the performance between SSL pre-training vs. purely
supervised training for Subject 2 for all questions over different sets of
labeled data points. We show these results for all participants in the WESAD
dataset in our Supplementary Material.

##  5 Discussion

###  5.1 Implications for Mobile Sensing Systems

Mobile sensing systems have traditionally been unable to detect psychiatric
and mental health outcomes such as stress due to the inherent subjectivity of
the labels and variability across users. SSL on biosignals, such as the
methodology introduced and evaluated here, provides an opportunity for mobile
sensing solutions which detect complex psychiatric events in a personalized
manner. Personalization of models is one of the leading challenges and
opportunities in machine learning for healthcare and digital psychiatry [66,
67]. From a user interaction standpoint, users are only required to provide a
few (on the order of ten) labeled examples of the adverse event of interest.

Through the use of a limited number of labeled examples for the targeted
adverse event, this methodology effectively reduces the user labeling burden
while simultaneously facilitating the development of more tailored and precise
models. This approach not only improves the overall user experience but also
amplifies the probability of user engagement and adherence to digital
therapies due to the improved user experience required to achieve a high-
performing personalized mobile sensing model.

Personalization through SSL is likely to work in other domains as well. There
are numerous digital healthcare research efforts which are attempting to build
machine learning models for recurring conditions and events which are
traditionally infeasible for machine learning due to the subjectivity of the
classes. For example, some digital therapies for pediatric autism use computer
vision models to predict facial expressions evoked by conversational partners
[68, 69, 70, 71, 72, 73, 74, 75]. Often, such digital therapeutic systems
involve the curation of user-centric data [76, 77, 78, 79, 80, 81, 82, 83],
frequently with the end goal of developing a predictive model which analyzes
the behavior of the end user [84, 85, 86, 87, 88, 89]. Such systems provide a
clear opportunity for self-supervised personalization as described here.
Personalizing the emotion recognition model powering such digital therapies
towards frequent conversational partners by the child is possible with the
methods described here.

The interaction paradigm demonstrated here is similar to how Apple’s Face ID
works. Once the face recognition system is set up with only a few examples
provided by the end users, it can be used repeatedly to unlock the device. Our
primary goal of this paper is to demonstrate that this user interaction
workflow can be adapted to mobile and wearable health settings.

In terms of implementation platforms, the HCI paradigm demonstrated here can
be applied to a wide range of systems, such as smartphone apps, wearable
devices, and laptops. For example, a smartphone app could use our technique to
provide real-time stress monitoring and personalized tips for dealing with
stress, while a wearable device could use the model to identify stress levels
during physical activities or to monitor stress levels throughout the day.

The choice of platform depends on a number of factors, such as the specific
use case, user preferences, and the desired amount of user activity. For
example, a smartphone app might be preferred by individuals who want to
actively track their stress levels and receive feedback, while a wearable
device might be preferred by people who want to passively track their stress
levels and have less contact.

###  5.2 Limitations and Future Work

While this work demonstrates promise for personalized learning for mobile
sensing, our evaluation involved only a single dataset (WESAD) and a single
modality (EDA). In future work, we would like to see our evaluation expanded
to numerous biosignal datasets and health outcomes. Moving forward, an
important direction for further investigation is to examine the integration of
multiple signals within a single model, potentially enhancing its predictive
capabilities and overall performance. Additionally, the importance of
explainability in the medical informatics domain cannot be overstated. In
future research, there is considerable potential to extend this study by
exploring how different features influence our personalized model, thus
enhancing its interpretability and applicability throughout the field. It is
unclear whether the personalized SSL we presented will generalize to other
health events which are not as strongly correlated as EDA and stress. If the
method does generalize, then this work has the potential to become widely
adopted in several HCI fields including wearable computing, affective
computing, and mobile sensing.

There are inherent limitations to the WESAD dataset which we used for
evaluation. WESAD provides data from controlled experimental situations where
realistic emotional engagement is a challenge. As the participants were
required to assess their emotional state after each condition (i.e. baseline,
amusement, stress, meditation), there is no continuous annotation of the
participant’s emotional state in this dataset. Therefore, there is a
possibility of not obtaining the participant’s actual affective state after
each task. During data collection, some variables, such as a participant’s
proficiency in spoken English and familiarity with the provided subject under
the stress condition, may introduce bias. Also, the participants’ ages ranged
from 25 to 30, which may have introduced bias into the findings.

In future work, we recommend testing the system using real-time, unimodal data
obtained from participants in a more comprehensive and unconstrained
experimental setup. This approach could potentially lead to a better
understanding of the system’s performance and generalizability in real-world
scenarios, further strengthening the practical applicability of our findings.
Nevertheless, WESAD is a frequently used dataset in the field of affective
computing using wearable data [90, 91, 92, 93].

##  6 Conclusion

We present a novel method for personalizing deep learning models for the
prediction of subjective outcomes such as stress using continuous biosignals.
We have demonstrated the low labeling requirement for fine-tuning the model
after self-supervised pre-training. This method can enable high-performance
personalized modeling with minimal manual annotation effort from end users.
This technique enables deep learning models to be tailored to an individual’s
specific physiological reactions, resulting in more precise and personalized
stress prediction. This has the potential to significantly increase the
efficacy of stress management and preventive digital interventions while also
reducing the load of human annotation that is generally necessary for tailored
models. Overall, our findings show that self-supervised learning can enable
increasingly individualized mobile sensing solutions.

##  7 Declarations

###  7.1 Ethics approval and consent to participate

Not applicable

###  7.2 Consent for publication

Not applicable

###  7.3 Availability of data and materials

The dataset analysed during the current study is available in Dataset Link

###  7.4 Competing interests

Not applicable

###  7.5 Funding

Not applicable

###  7.6 Authors’ contributions

Both authors have equal contributions to this paper.

###  7.7 Acknowledgements

The technical support and advanced computing resources from the University of
Hawaii Information Technology Services – Cyberinfrastructure, funded in part
by the National Science Foundation CC* awards # 2201428 and # 2232862 are
gratefully acknowledged.

###  7.8 Author’s information

Tanvir Islam is a Masters’s student in the Department of Information and
Computer Sciences at the University of Hawaii at Manoa. He is primarily
focused on conducting research within the healthcare sector, specifically in
the areas of Natural Language Processing (NLP) and Bio-signals. He is employed
as a student researcher at both the Hawaii Digital Health Lab and the
Laboratory for Advanced Visualization & Applications (LAVA) at the University
of Hawaii at Manoa.

Peter Washington is a tenure-track Assistant Professor in the Department of
Information and Computer Sciences at the University of Hawaii at Manoa. His
work spans digital health and machine learning innovations for healthcare with
a particular focus on underserved populations in Hawaii. Prior to starting the
Hawaii Digital Health Lab, he completed his PhD in Bioengineering from
Stanford University, MS in Computer Science from Stanford University, and BA
in Computer Science from Rice University.

## References

  *   * [1] Cohen, S., Janicki-Deverts, D. & Miller, G. E.  Psychological stress and disease.  _Jama_ 298, 1685–1687 (2007).
  * [2] Paredes, P. & Chan, M.  Calmmenow: Exploratory research and design of stress mitigating mobile interventions (2011).  URL https://doi.org/10.1145/1979742.1979831.
  * [3] Sarsenbayeva, Z. _et al._ Measuring the effects of stress on mobile interaction.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 3, 1–18 (2019).
  * [4] Hänsel, K., Poguntke, R., Haddadi, H., Alomainy, A. & Schmidt, A.  What to put on the user: Sensing technologies for studies and physiology aware systems (2018).
  * [5] Wang, W. _et al._ Social sensing: assessing social functioning of patients living with schizophrenia using mobile phone sensing (2020).
  * [6] Liapis, A., Katsanos, C., Sotiropoulos, D., Xenos, M. & Karousos, N.  Recognizing emotions in human computer interaction: studying stress using skin conductance (2015).
  * [7] Zakaria, C., Balan, R. & Lee, Y.  Stressmon: Scalable detection of perceived stress and depression using passive sensing of changes in work routines and group interactions.  _Proceedings of the ACM on Human-Computer Interaction_ 3, 1–29 (2019).
  * [8] Giannakakis, G. _et al._ Review on psychological stress detection using biosignals.  _IEEE Transactions on Affective Computing_ 13, 440–460 (2019).
  * [9] Boucsein, W.  _Electrodermal activity_ (Springer Science & Business Media, 2012).
  * [10] Healey, J. & Picard, R.  Detecting stress during real-world driving tasks using physiological sensors.  _IEEE Transactions on Intelligent Transportation Systems_ 6, 156–166 (2005).
  * [11] Setz, C. _et al._ Discriminating stress from cognitive load using a wearable eda device.  _IEEE Transactions on Information Technology in Biomedicine_ 14, 410–417 (2010).
  * [12] LeCun, Y., Bengio, Y. & Hinton, G.  Deep learning.  _nature_ 521, 436–444 (2015).
  * [13] Amodei, D. _et al._ Balcan, M. F. & Weinberger, K. Q. (eds) _Deep speech 2 : End-to-end speech recognition in english and mandarin_.  (eds Balcan, M. F. & Weinberger, K. Q.) _Proceedings of The 33rd International Conference on Machine Learning_ , Vol. 48 of _Proceedings of Machine Learning Research_ , 173–182 (PMLR, New York, New York, USA, 2016).  URL https://proceedings.mlr.press/v48/amodei16.html.
  * [14] He, K., Zhang, X., Ren, S. & Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015).
  * [15] Silver, D. _et al._ Mastering the game of go with deep neural networks and tree search.  _Nature_ 529, 484–489 (2016).
  * [16] Gulshan, V. _et al._ Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.  _JAMA_ 316, 2402–2410 (2016).  URL https://doi.org/10.1001/jama.2016.17216.
  * [17] Schmidt, P., Reiss, A., Duerichen, R. & Van Laerhoven, K.  Wearable affect and stress recognition: A review.  _arXiv preprint arXiv:1811.08854_ (2018).
  * [18] Geiger, R. S. _et al._ “Garbage in, garbage out” revisited: What do machine learning application papers report about human-labeled training data?  _Quantitative Science Studies_ 2, 795–827 (2021).
  * [19] Kim, H.-G., Cheon, E.-J., Bai, D.-S., Lee, Y. H. & Koo, B.-H.  Stress and heart rate variability: A meta-analysis and review of the literature.  _Psychiatry investigation_ 15, 235 (2018).
  * [20] He, J., Li, K., Liao, X., Zhang, P. & Jiang, N.  Real-time detection of acute cognitive stress using a convolutional neural network from electrocardiographic signal.  _IEEE Access_ 7, 42710–42717 (2019).
  * [21] Dharmawan, Z.  Analysis of computer games player stress level using eeg data (2007).
  * [22] Secerbegovic, A., Ibric, S., Nisic, J., Suljanovic, N. & Mujcic, A.  Mental workload vs. stress differentiation using single-channel eeg (2017).
  * [23] Vanitha, V. & Krishnan, P.  Real time stress detection system based on eeg signals 2016, S271–S275 (2016).
  * [24] Halim, Z. & Rehan, M.  On identification of driving-induced stress using electroencephalogram signals: A framework based on wearable safety-critical scheme and machine learning.  _Information Fusion_ 53, 66–79 (2020).  URL https://www.sciencedirect.com/science/article/pii/S1566253518303221.
  * [25] Karthikeyan, P., Murugappan, M. & Yaacob, S.  Emg signal based human stress level classification using wavelet packet transform (2012).
  * [26] Ahn, J. W., Ku, Y. & Kim, H. C.  A novel wearable eeg and ecg recording system for stress assessment.  _Sensors_ 19, 1991 (2019).
  * [27] Charbonnier, S. _et al._ A multi-feature fuzzy index to assess stress level from bio-signals (2018).
  * [28] Walocha, F., Drewitz, U. & Ihme, K.  Activity and stress estimation based on openpose and electrocardiogram for user-focused level-4-vehicles.  _IEEE Transactions on Human-Machine Systems_ (2022).
  * [29] Hannun, A. Y. _et al._ Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network.  _Nature medicine_ 25, 65–69 (2019).
  * [30] Wickramasuriya, D. S., Qi, C. & Faghih, R. T.  A state-space approach for detecting stress from electrodermal activity (2018).
  * [31] Spathis, D., Perez-Pozuelo, I., Brage, S., Wareham, N. J. & Mascolo, C.  Self-supervised transfer learning of physiological representations from free-living wearable data (2021).
  * [32] Schmidt, P., Reiss, A., Duerichen, R., Marberger, C. & Van Laerhoven, K.  Introducing wesad, a multimodal dataset for wearable stress and affect detection (2018).
  * [33] Affanni, A.  Wireless sensors system for stress detection by means of ecg and eda acquisition.  _Sensors_ 20, 2026 (2020).
  * [34] Zhu, L. _et al._ Feasibility study of stress detection with machine learning through eda from wearable devices (2022).
  * [35] Yadav, M. _et al._ Exploring individual differences of public speaking anxiety in real-life and virtual presentations.  _IEEE Transactions on Affective Computing_ PP, 1–1 (2020).
  * [36] Sánchez-Reolid, R., López, M. T. & Fernández-Caballero, A.  Machine learning for stress detection from electrodermal activity: A scoping review (2020).
  * [37] Ninh, V.-T., Smyth, S., Tran, M.-T. & Gurrin, C.  Analysing the performance of stress detection models on consumer-grade wearable devices.  _arXiv preprint arXiv:2203.09669_ (2022).
  * [38] Anusha, A. _et al._ Electrodermal activity based pre-surgery stress detection using a wrist wearable.  _IEEE journal of biomedical and health informatics_ 24, 92–100 (2019).
  * [39] Eren, E. & Navruz, T. S.  Stress detection with deep learning using bvp and eda signals (2022).
  * [40] Pakarinen, T., Pietilä, J. & Nieminen, H.  Prediction of self-perceived stress and arousal based on electrodermal activity (2019).
  * [41] Kalimeri, K. & Saitis, C.  Exploring multimodal biosignal features for stress detection during indoor mobility (2016).
  * [42] Lee, J. H. _et al._ Stress monitoring using multimodal bio-sensing headset (2020).
  * [43] Hernandez, J., Paredes, P., Roseway, A. & Czerwinski, M.  Under pressure: sensing stress of computer users (2014).
  * [44] Sarker, H. _et al._ Finding significant stress episodes in a discontinuous time series of rapidly varying mobile sensor data (2016).
  * [45] Ayzenberg, Y., Hernandez Rivera, J. & Picard, R.  Feel: frequent eda and event logging–a mobile social interaction stress monitoring system (2012).
  * [46] Hinckley, K., Pierce, J., Sinclair, M. & Horvitz, E.  Sensing techniques for mobile interaction (2000).
  * [47] LiKamWa, R., Liu, Y., Lane, N. D. & Zhong, L.  Moodscope: Building a mood sensor from smartphone usage patterns (2013).
  * [48] Kang, S., Park, C. Y., Kim, A., Cha, N. & Lee, U.  Understanding emotion changes in mobile experience sampling (2022).
  * [49] Chen, T., Kornblith, S., Norouzi, M. & Hinton, G.  A simple framework for contrastive learning of visual representations (2020).
  * [50] Grill, J.-B. _et al._ Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ 33, 21271–21284 (2020).
  * [51] Zhang, J. & Ma, K.  Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views (2022).
  * [52] Matton, K., Lewis, R. A., Guttag, J. & Picard, R.  Contrastive learning of electrodermal activity representations for stress detection (2022).  URL https://openreview.net/forum?id=bSC_xo8VQ1b.
  * [53] Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O. & Azemi, E.  Subject-aware contrastive learning for biosignals.  _arXiv preprint arXiv:2007.04871_ (2020).
  * [54] Kiyasseh, D., Zhu, T. & Clifton, D. A.  Clocs: Contrastive learning of cardiac signals across space, time, and patients (2021).
  * [55] Sarkar, P. & Etemad, A.  Self-supervised ecg representation learning for emotion recognition.  _IEEE Transactions on Affective Computing_ (2020).
  * [56] Banville, H. _et al._ Self-supervised representation learning from electroencephalography signals (2019).
  * [57] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * [58] Lee, M., Liang, P. & Yang, Q.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities (2022).
  * [59] Karinshak, E., Liu, S. X., Park, J. S. & Hancock, J. T.  Working with ai to persuade: Examining a large language model’s ability to generate pro-vaccination messages.  _Proceedings of the ACM on Human-Computer Interaction_ 7, 1–29 (2023).
  * [60] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. _et al._ Improving language understanding by generative pre-training (2018).
  * [61] Ancillon, L., Elgendi, M. & Menon, C.  Machine learning for anxiety detection using biosignals: a review.  _Diagnostics_ 12, 1794 (2022).
  * [62] Kiranyaz, S., Ince, T., Abdeljaber, O., Avci, O. & Gabbouj, M.  1-d convolutional neural networks for signal processing applications (2019).
  * [63] Rim, B., Sung, N.-J., Min, S. & Hong, M.  Deep learning in physiological signal data: A survey.  _Sensors_ 20, 969 (2020).
  * [64] Kim, S.-H., Geem, Z. W. & Han, G.-T.  Hyperparameter optimization method based on harmony search algorithm to improve performance of 1d cnn human respiration pattern recognition system.  _Sensors_ 20, 3697 (2020).
  * [65] Tang, W. _et al._ Rethinking 1d-cnn for time series classification: A stronger baseline.  _arXiv preprint arXiv:2002.10061_ 1–7 (2020).
  * [66] Washington, P. _et al._ Challenges and opportunities for machine learning classification of behavior and mental state from images.  _arXiv preprint arXiv:2201.11197_ (2022).
  * [67] Washington, P. & Wall, D. P.  A review of and roadmap for data science and machine learning for the neuropsychiatric phenotype of autism.  _Annual Review of Biomedical Data Science_ 6 (2023).
  * [68] Washington, P. _et al._ Superpowerglass: a wearable aid for the at-home therapy of children with autism.  _Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies_ 1, 1–22 (2017).
  * [69] Washington, P. _et al._ A wearable social interaction aid for children with autism (2016).
  * [70] Kline, A. _et al._ Superpower glass.  _GetMobile: Mobile Computing and Communications_ 23, 35–38 (2019).
  * [71] Voss, C. _et al._ Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial.  _JAMA pediatrics_ 173, 446–454 (2019).
  * [72] Voss, C. _et al._ Designing a holistic at-home learning aid for autism.  _arXiv preprint arXiv:2002.04263_ (2020).
  * [73] Voss, C. _et al._ Superpower glass: delivering unobtrusive real-time social cues in wearable systems (2016).
  * [74] Daniels, J. _et al._ Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism.  _NPJ digital medicine_ 1, 32 (2018).
  * [75] Daniels, J. _et al._ Feasibility testing of a wearable behavioral aid for social learning in children with autism.  _Applied clinical informatics_ 9, 129–140 (2018).
  * [76] Kalantarian, H. _et al._ Guess what? towards understanding autism from structured video using facial affect.  _Journal of healthcare informatics research_ 3, 43–66 (2019).
  * [77] Penev, Y. _et al._ A mobile game platform for improving social communication in children with autism: a feasibility study.  _Applied clinical informatics_ 12, 1030–1040 (2021).
  * [78] Washington, P. _et al._ Training an emotion detection classifier using frames from a mobile therapeutic game for children with developmental disorders.  _arXiv preprint arXiv:2012.08678_ (2020).
  * [79] Kalantarian, H. _et al._ A gamified mobile system for crowdsourcing video for autism research (2018).
  * [80] Hou, C., Kalantarian, H., Washington, P., Dunlap, K. & Wall, D. P.  Leveraging video data from a digital smartphone autism therapy to train an emotion detection classifier.  _medRxiv_ 2021–07 (2021).
  * [81] Kalantarian, H. _et al._ Labeling images with facial emotion and the potential for pediatric healthcare.  _Artificial intelligence in medicine_ 98, 77–86 (2019).
  * [82] Kalantarian, H., Jedoui, K., Washington, P. & Wall, D. P.  A mobile game for automatic emotion-labeling of images.  _IEEE transactions on games_ 12, 213–218 (2018).
  * [83] Kalantarian, H. _et al._ The performance of emotion classifiers for children with parent-reported autism: quantitative feasibility study.  _JMIR mental health_ 7, e13174 (2020).
  * [84] Washington, P. _et al._ Improved digital therapy for developmental pediatrics using domain-specific artificial intelligence: Machine learning study.  _JMIR Pediatrics and Parenting_ 5, e26760 (2022).
  * [85] Deveau, N. _et al._ Machine learning models using mobile game play accurately classify children with autism.  _Intelligence-Based Medicine_ 6, 100057 (2022).
  * [86] Chi, N. A. _et al._ Classifying autism from crowdsourced semistructured speech recordings: Machine learning model comparison study.  _JMIR Pediatrics and Parenting_ 5, e35406 (2022).
  * [87] Washington, P. _et al._ Data-driven diagnostics and the potential of mobile artificial intelligence for digital therapeutic phenotyping in computational psychiatry.  _Biological Psychiatry: Cognitive Neuroscience and Neuroimaging_ 5, 759–769 (2020).
  * [88] Banerjee, A., Washington, P., Mutlu, C., Kline, A. & Wall, D. P.  Training and profiling a pediatric emotion recognition classifier on mobile devices.  _arXiv preprint arXiv:2108.11754_ (2021).
  * [89] Banerjee, A. _et al._ Training and profiling a pediatric facial expression classifier for children on mobile devices: machine learning study.  _JMIR formative research_ 7, e39917 (2023).
  * [90] Liapis, A., Faliagka, E., Katsanos, C., Antonopoulos, C. & Voros, N.  Ardito, C. _et al._ (eds) _Detection of subtle stress episodes during ux evaluation: Assessing the performance of the wesad bio-signals dataset_.  (eds Ardito, C. _et al._) _Human-Computer Interaction – INTERACT 2021_ , 238–247 (Springer International Publishing, Cham, 2021).
  * [91] Gu, F. _et al._ A survey on deep learning for human activity recognition.  _ACM Computing Surveys (CSUR)_ 54, 1–34 (2021).
  * [92] Bajpai, D. & He, L.  Evaluating knn performance on wesad dataset (2020).
  * [93] Rovinska, S. & Khan, N.  Affective state recognition with convolutional autoencoders (2022).