# Multimodal Stress Detection Using Facial Landmarks and Biometric Signals
††thanks: Corresponding Author: Raju Gottumukkala,
<EMAIL_ADDRESS>
Majid Hosseini, Morteza Bodaghi, Ravi Teja Bhupatiraju, Anthony Maida, Raju
Gottumukkala*
University of Louisiana at Lafayette
USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The development of various sensing technologies is improving measurements of
stress and the well-being of individuals. Although progress has been made with
single signal modalities like wearables and facial emotion recognition,
integrating multiple modalities provides a more comprehensive understanding of
stress, given that stress manifests differently across different people.
Multi-modal learning aims to capitalize on the strength of each modality
rather than relying on a single signal. Given the complexity of processing and
integrating high-dimensional data from limited subjects, more research is
needed. Numerous research efforts have been focused on fusing stress and
emotion signals at an early stage, e.g., feature-level fusion using basic
machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal
learning approach for stress detection that integrates facial landmarks and
biometric signals. We test this multi-modal integration with various early-
fusion and late-fusion techniques to integrate the 1D-CNN model from biometric
signals and 2-D CNN using facial landmarks. We evaluate these architectures
using a rigorous test of models’ generalizability using the leave-one-subject-
out mechanism, i.e., all samples related to a single subject are left out to
train the model. Our findings show that late-fusion achieved 94.39% accuracy,
and early-fusion surpassed it with a 98.38% accuracy rate. This research
contributes valuable insights into enhancing stress detection through a multi-
modal approach. The proposed research offers important knowledge in improving
stress detection using a multi-modal approach.

_K_ eywords Stress Detection, Deep Neural Network, Multimodal Model,
Convolutional Neural Network, Decision Layer Fusion, early-fusion, late-fusion

##  1 Introduction

Stress has gained importance in recent years as people are placing more
emphasis on holistic well-being, i.e., physical health, mental health,
emotional and social well-being. Moreover, prolonged exposure to stress can in
turn, affect physical and mental health, leading to conditions such as heart
disease, depression, and anxiety [1]. Recent research has shown the potential
of employing computer vision and biometric data for stress detection [28].
Biometric indicators such as Heart Rate (HR), Electrodermal Activity (EDA),
and Skin Temperature (ST/TEMP) have the capacity to reflect an individual’s
physiological reactions to stress, making them reliable objective measures of
stress levels. Conversely, facial landmarks represent crucial points on the
face that provide valuable insights into facial expressions, enabling
inference of emotional states and stress. Conventional stress detection
methods mostly focus on a single modality, such as physiological signals or
facial expressions. It is well known that Stress is a complex interplay of
psychological and physiological responses to external or internal stimuli,
making it difficult to predict accurately. One of the key challenges of
predicting stress is that it is transient, meaning it can change quickly and
is influenced by a variety of factors. These factors include environmental
stimuli, prior conditioning or past experience with stress, emotional states,
and external factors such as work demands, social interactions, and life
events. These factors can significantly impact stress levels and vary widely
between individuals. Stress manifests differently in different people, leading
to individual differences in stress responses [68]. This makes it difficult to
develop a single, accurate method for stress prediction. Researchers are
investigating new approaches to stress prediction that use multiple data
sources such as heart rate, temperature, and facial expressions [14, 13, 11,
12]. The idea of multimodal stress prediction is inspired by the human senses,
which work together to provide us with a comprehensive understanding of the
world around us.

It is also important to differentiate between "multivariate" and "multimodal"
learning for neural networks. Multivariate networks focus on processing
multiple variables or predictors but often the same kind of data. For example,
it might analyze time series data signals such as ECG, temperature, and heart
rate, all intended for stress detection. In contrast, multimodal neural
networks are designed to integrate information from different types of data
sources or modalities. For example, a multimodal network is one that processes
voice and image data simultaneously, determining how one might be related to
one another. Multimodal stress prediction models combine data from multiple
sensors to learn complex patterns that are associated with stress.

This paper presents a novel multimodal approach for stress detection,
combining facial landmarks analysis with biometric signals comparing early and
late fusion techniques. The overarching theme of stress detection is pre-
established, and various fusion methodologies, both late and early, have been
explored previously [12, 11, 16, 17]. Earlier research typically treats both
modalities as time series signals, modeling each with a 1D CNN. In our
approach, we model facial landmarks using a 2D CNN. The integration of 2D CNN
and 1D CNN models for stress detection has not seen studied before. We study
the performance of this against a wide range of both late and early fusion
techniques that employ facial landmarks and biometric feedbacl. We also
analyze the effects of various signal components and model architectures,
including 2079 video and biometric signal features, EDA tonic and phasic
components, and the performance of six deep learning models with different
feature selection methods, along with eleven multimodal models using these
features.

We first discuss related work in the area of stress detection, focusing on
physiological signals from wearables and computer vision, as well as multi-
modal stress detection approaches. We provide a comprehensive summary of
biomedical features documented in the literature. Next, we offer an in-depth
overview of the dataset and elaborate on the methodologies employed to extract
various features from physiological and computer vision datasets. Following
this discussion, we present early and late-fusion models and their
architectures. Finally, we evaluate and discuss the outcomes of the study,
weighing various model architectures against both accuracy and computational
cost.

##  2 Related Work

Researchers have studied stress detection through the use of wearable devices
and facial expressions. Sioni et al. [19] discuss how physiological signals
such as HR and EDA can detect stress. Alberdi et al. [28] studied automatic,
continuous, and unobtrusive early stress detection methods for office workers.
They found that ECG, especially using HRV features, and EDA are the most
accurate physiological signals for recognizing stress. It is desirable to have
better accuracy when incorporating data from multiple modalities, such as
behavioral responses. This approach would lead to the development of a less
intrusive and widely applicable monitoring system, significantly enhancing
practicality. Well-engineered multimodal methods should offer reliable models
through diverse signal information and enhanced robustness against noise. We
review previous research in the fields of physiological signals, computer
vision, and multi-modal signals.

Table 1: Summarize of Reviewed Paper

Author |  Signals |  Classification Performance
---|---|---
Schmidt, Philip, et al. [35] |  BVP, ECG, EDA, EMG, RR, TEMP, and ACC. |  Acc 80% (3-class) 93% (2-class)
Koldijk, et al. [36] |  Facial Expressions, Posture, Physiology. |  Acc 90% (2-class)
Giannakakis, et al. [38] |  Facial Cues and camera-based HR. |  Acc 91.68% (3 -class)
Padmaja, B., et al. [42] |  FITBIT wearable device data |  Acc 62% (3-class)
Shu, Lin, et al. [45] |  HR |  Acc 84% (3-class)
Hsu, Yu Liang, et al. [46] |  ECG |  Acc 82.78% (2-class)
Sharma, et al. [48] |  EEG signals |  Acc 82.01% (4-class)
Zhu, Lili, et al. [49] |  EDA |  Acc 85.7% (2-class)
Zhu, Lili, et al. [50] |  EDA, ECG, and PPG |  Acc 86.4% (2-class)
Jeon, Taejae, et al. [51] |  Facial Landmarks |  Acc 64.63% (3-class)
Zhang, Huijun, et al. [52] |  Facial Expressions |  Acc 85.42% (2-class)
Cardone, Daniela, et al. [53] |  Thermal Infrared Imaging |  AUC 80% (2-class)
Kopaczka, Marcin, et al. [55] |  Facial Expressions |  Acc 65.75% (4-class)
Seo, Wonju, et al. [56)] |  ECG, RR, and Facial Landmarks |  Acc 73.3% (2-class)
Bobade, Pramod et al. [11] |  ACC, ECG, BVP, TEMP, RR, EMG and EDA |  Acc 84.32 % (3-class) 95.21% (2-class)
Walambe, Rahee, et al. [58] |  Facial Expressions, Posture, HR ,Computer Interaction |  Acc 96.09% (2-class)
Naegelin, Mara, et al. [59] |  HR and Behavioral data |  F1 score 77.5% (3-class)
Kuttala, Radhika et al. [17] |  EDA and ECG |  Acc 97.6% (2-class)
Hosseini, Majid, et al. [15] |  HR, TEMP, EDA, ACC and Facial Expression |  Prec 85.04 (3-class)

###  2.1 Physiological Signals

Schmidt et al. [35] introduced the WESAD dataset consisting of BVP, ECG, EDA,
EMG, RR, ST, and three-axis acceleration (ACC) to detect neutral, stress, and
amusement using self-reports validation surveys. The benchmark achieved
classification accuracies up to 80% for three classes (baseline, stress,
amusement) and up to 93% for binary (stress, non-stress) classification.
Padmaja, B. et al. [42] propose a method for detecting stress levels using
data from a physical activity tracker device (Fitbit), using Logistic
Regression to evaluate the impact of individual stressors. They achieved 62%
accuracy for three stress levels. Hsu, Yu Liang, et al. [46] introduced an
automatic algorithm for emotion recognition based on ECG signals. The
classification rates for positive/negative valence, high/low arousal, and four
emotion classification tasks were 82.78%, 72.91%, and 61.52%, respectively.
Shu, Lin, et al. [45] achieved 84% accuracy for emotion recognition based on
HR data from a wearable smart bracelet using the Gradient Boosted Decision
Trees (GBDT) model. Sharma et al. [48] achieved an accuracy of 82.01% with
four-labeled emotion classes using 10-fold cross-validation on online
recognition of human emotions using EEG signals on the DEAP dataset [61]. Zhu,
Lili, et al. [49] examined the potential of utilizing EDA from two publicly
available datasets collected from research-grade wearable devices. The
experimental results demonstrated that Random Forests(RF) achieve an 85.7%
accuracy rate in classifying stress from non-stress states. Zhu, Lili, et al.
[50] used signals of EDA, ECG, and photoplethysmography (PPG) from
smartwatches. They investigated machine learning methods for two-level stress
classification. The results indicate that EDA, particularly when used with
Stacking Ensemble Learning, achieves the highest accuracy of 86.4% compared to
other signals and combinations. Kuttala, Radhika, et al. [17] used EDA and ECG
signals for the hierarchical feature fusion stress detection model and
achieved 97.6% accuracy for binary classification.

###  2.2 Imaging/Video Signals

Giannakakis, Giorgos, et al. [38] introduces a framework for detecting stress
and anxiety emotional states using facial cues captured in video recordings.
Features like eye-related events, mouth activity, head motion parameters, and
camera-based HR estimation were investigated and classified into three
classes: neutral, relaxed, and stressed/anxious. They got 91.68% accuracy
using the AdaBoost classifier. Kopaczka, Marcin, et al. [55] developed a
system consisting of face detection and facial landmark detection and employed
Histogram of Oriented Gradients (HOG) features using a random forest
classifier, achieving a classification accuracy of 65.75% for four basic
emotions (namely, Neutral-Happy-Sad-Surprised). [52] addressed the issue of
stress detection using video-based camera sensors. They used facial
expressions and action motions from videos to identify stress with a
performance of 85.42% accuracy for stress binary classification. Cardone,
Daniela, et al. [53] used thermal infrared images for stress detection. In
this study, they achieved 80% for the area under the curve (AUC), sensitivity
of 77%, and specificity of 78% for binary classification based on a non-linear
support vector regression (SVR). Jeon, Taejae, et al. [51] proposes a stress
recognition algorithm using face images and facial landmarks to understand
eye, mouth, and head movements. They achieved an accuracy of 64.63%, detecting
three levels of no, weak, and strong stress.

###  2.3 Multimodal Signals

The SWELL project [36] focuses on multimodal sensor data (computer logging,
facial expressions, posture, and physiology). The study distinguishes neutral
and stressful working conditions with 90% accuracy using a support vector
machine (SVM) model, using posture and facial expressions to provide valuable
information. Bobade, Pramod, et al. [11] used motion sensors alongside
physiological signals (ECG, BVP, TEMP, RR, EMG, and EDA) to detect stress. The
results showed that machine learning techniques achieve accuracies of up to
81.65% and 93.20% for emotion detection(amusement, baseline, and stress) and
binary stress detection(stress vs. non-stress), respectively. Deep learning
achieves accuracies of up to 84.32% and 95.21% for the two and three-class
classifications. Seo, Wonju, et al. [56)] used ECG, RR, and facial landmarks
extracted from video data and achieved an accuracy of 73.3% for binary
classification and 54.4% for three-level stress classification using
multivariate deep neural networks. A multimodal AI-based framework is proposed
by Walambe, Rahee, et al. [58]. The methodology involves fusing data of facial
expressions, posture, HR, and computer interaction (Mouse activity, Left
click, etc.) to detect workload stress and achieve an accuracy of 96.09%.
Naegelin, Mara, et al. [59] utilize mouse, keyboard, and HRV features to
detect three levels of perceived stress, valence, and arousal, and they
achieved a maximum F1 score of 77.5%.

Some of these recent findings show the promise of multimodal approaches with
respect to improving the accuracy and reliability of real-time stress
detection. However, these studies are evaluated in small datasets and
evaluated in laboratory conditions. More datasets and methods are needed to
improve the efficacy of these methods. Table 1 summarizes some reviewed
papers, listing the signals they used and their performance in terms of
accuracy (Acc), area under the curve (AUC), precision (Prec), and F1 score.

##  3 Methods

###  3.1 Dataset

The EmpathicSchool [15] dataset contains video-based facial expressions and
corresponding physiological signals, HR, EDA, TEMP, and ACC from Empatica E4.
26 hours of data were collected from 20 participants from two universities,
Tampere University, Finland, and the University of Louisiana at Lafayette,
United States. The stress levels were determined via National Aeronautics and
Space Administration Task Load Index questionnaires. The dataset includes
seven different signal types, including both computer vision and physiological
features. The video dataset provides facial landmarks. To the best of our
knowledge, facial information of 68 landmarks has never been studied before.
Additionally, 30 unique landmark features were incorporated to improve further
stress prediction accuracy, such as the difference and distance of two or more
landmark points on the face.

####  3.1.1 Biometric Data

The following biometric signals were measured via the E4 wristband and are
provided in the dataset:

• HR: Healthy resting heart rate (HR) ranges from 60 to 100 beats per minute,
regardless of gender. However, HR is influenced by activity and emotional
state. Wearable HR sensors enable the analysis of patterns and fluctuations,
providing insights into stress, exertion, and cardiovascular health.

• TEMP: Skin temperature is influenced by skin blood flow and can reveal
stress, fever, and changes in blood flow. Wearable ST sensors can capture
data, enabling analysis of temperature patterns and correlation with other
physiological parameters.

• EDA: EDA (Electrodermal Activity), also known as GSR (Galvanic Skin
Response), is a signal that quantifies sweat gland activity and offers
insights into emotional and psychological arousal. It reflects reactions to
stress, excitement, anxiety, and other emotional states. Wearable EDA sensors
detect changes in skin conductance, providing valuable information about an
individual’s physiological and emotional responses.

• Accelerometers (ACC): ACC data, measured by sensors, records object
acceleration. These sensors have diverse applications, such as human action
recognition and step counting. Wearable ACCs track movement patterns, offering
information about activity levels.

####  3.1.2 Facial Landmarks

Facial landmarks are key points on a face that are used to identify features
on a face, such as corners of eyes, eyebrows, tip of the nose, and lips. Image
processing systems typically use these features to detect facial orientation,
expression, or general facial patterns. 2D pixel values on the other hand are
raw signals stored as pixel values of the entire image in a 2D array. Compared
to a 2D image, facial landmarks provide a compact representation of facial
features, capturing essential and relevant features. Extracting 68 facial
landmarks is consisted of two steps face detection and identifying smaller
features of the face, like eyes. We used Dlib 68-point face detection [8] with
OpenCV [7] to extract facial landmarks. Figure 1 shows 68 specific points on
the face, which we refer to as landmarks. Analysis of spatial and temporal
changes in these landmarks enables machine learning models to detect and
classify stress.

Figure 1: Illustration of the 68 facial landmarks

###  3.2 Pre-processing

The original signal data was collected with a frequency of 4 Hz in the
EmphaticSchool [15] dataset. We down-sample the data to a frequency of 1 Hz to
reduce training and testing times. To reduce the effect of noise and artifacts
on the stress data, we separated the EDA signal into its phasic and tonic
components, and several features of each component were derived. Table 2 shows
the different features of EDA signals.

Table 2: EDA signals EDA Signals | Description
---|---
EDA_Clean |  Filtered version of the raw EDA signal
EDA_Tonic |  Sustained changes in skin conductance
EDA_Phasic |  Rapid changes in skin conductance
SCR_Onsets |  Starting points of SCR
SCR_Peaks |  Highest points of SCR
SCR_Height |  Intensity or strength of the response
SCR_Amplitude |  Magnitude of the response
SCR_RiseTime |  Duration for a SCR
SCR_Recovery |  Period after a SCR
SCR_Rec.Time |  Recovery time after an arousing event

The phasic component of the EDA signal represents the rapid and temporary
changes in skin conductance. Phasic changes are characterized by short-
duration and high-amplitude fluctuations in the EDA signal. The tonic
component of the EDA signal represents the relatively slower and more
sustained changes in skin conductance. Unlike phasic changes, the tonic
component is not directly linked to specific events or stimuli but rather
reflects the overall arousal level of an individual over a longer period.
Figure 2 shows a part of the raw EDA signal with relative phasic and tonic
components [60].

Figure 2: Phasic and Tonic components of raw EDA signal

The statistical features of each signal are generated using a rolling window
of 40 seconds with a sliding of 20 seconds due to the data characteristic,
computational efficiency, and training data availability. The average stress
level of each sliding window was obtained from the raw signal and binned into
three different bins, namely no-stress 0(0 to 6), medium-stress 1(7 to 13),
and stressful 2(14 to 19). The statistical features of each participant were
extracted using the tsfresh [20] Python library and normalized independently.
The statistical features were extracted, and the missing data were removed. We
used the data of 10 participants at different sessions for 26 hours. The dlib
facial expressions model [8] consumes the video signals and generates the
facial landmarks. We also extract several facial features and investigate the
correlation of mentioned features using different feature selection methods.
To evaluate their performance, the resulting multimodal data of biometric
signals and facial landmarks consisted of 1897 features and the raw signals
consumed by three deep learning networks: multivariate, early-fusion, and
late-fusion.

####  3.2.1 Multivariate models

The signals and generated features were given to a deep-learning model. One
hundred seventy-five features, including 15 statistical features (Absolute
Energy, Fourier Entropy, Skewness, Autocorrelation, Quantile, kurtosis, Count
Above Mean, Count Below Mean, Variation Coefficient, Root Mean Square,
Variance, Mean, Standard Deviation, Maximum, Minimum ) of four different
signals (EDA, Temp, HR, ACC), and 1904 features of 68 facial landmarks were
generated and used in the feature selection model. Biometric signal and facial
landmark features were decreased to 30 and 100 using lasso regularization
[65], respectively.

####  3.2.2 Multimodal Early-Fusion model

The multimodal data comprising 2079 features combining biometric and facial
landmarks data was decreased to 100 features using lasso regularization [65].
The added unique features were found to be crucial in this study.

####  3.2.3 Multimodal Late-Fusion model

The multimodal fusion model consumed the same data as our early-fusion
networks, including 30 biometric and 100 landmark features with 1Hz frequency.
The model showed promising results and was comparable to the early-fusion
approach. In addition, the concatenation outperforms the early-fusion models.

####  3.2.4 Feature analysis

We employed three different types of feature selection, namely filter methods,
namely Spearman [64], Pearson [63], and Variance Threshold [66], wrapper
method namely Recursive Feature Elimination (RFE) [67] and embedded methods of
Random Forest (RF) Importance [62], and Lasso Regularization [65]. The top 10
features are presented in Table 3. The Spearman and Pearson analyses yielded
similar results, emphasizing facial landmark features. Conversely, the RF
Importance and RFE analysis identified a combination of facial landmarks, ST,
and EDA (Tonic and Phasic) features are the top contributors. The tonic
component of EDA appears to have the highest variance using the variance
threshold. Comparing the six feature selection methods, we observed that lasso
regularization provided the most favorable outcome for running the models in
this study, selecting features related to EDA and facial landmarks. The second
and third best outcomes were from RF importance and RFE, respectively. We
provided SHAP values and feature contributions using the Lasso regularization
model. Figures 3,4 show the explainable information of the model using SHAP
library [69]. Among all features, the Max_EDA had the maximum effect on the
prediction results of all three stress levels, while some features, e.g.,
min_EDA and quantile_EDA, only discriminate between two stress levels. In
Figure 4, red signals represent the signals pushing higher stress levels
positively associated with stress (e.g., Max_EDA, EDA_Clean, and phasic
Max_EDA). In contrast, the blue features represent lower stress levels
negatively associated with stress (e.g., Max_EDA_clean, Energy_EDA, standard
deviation of tonic EDA).

Figure 3: Explainable AI features and feature contribution for stress level
discrimination Table 3: Top ten features using different feature selection
techniques

Rank | Spearmans | Pearson | Variance Threshold
---|---|---|---
1 | energy_X2 | energy_X2 | below_mean_EDA_Tonic
2 | energy_X1 | energy_X3 | above_mean_EDA_Tonic
3 | energy_X3 | energy_X4 | skew_EDA_Tonic
4 | energy_X4 | energy_X1 | kurtosis_EDA_Tonic
5 | RMS_X1 | RMS_X2 | energy_EDA_Tonic
6 | energy_Y9 | RMS_X3 | max_EDA_Tonic
7 | RMS_X2 | avg_X2 | EDA_Tonic
8 | avg_X1 | avg_X3 | avg_EDA_Tonic
9 | Y9 | RMS_X4 | RMS_EDA_Tonic
10 | energy_X5 | RMS_X1 | quantile_EDA_Tonic
Rank | RF Importance | RFE | Lasso Regularization
1 | energy_X2 | std_EDA_Clean | max_EDA
2 | energy_EDA_Tonic | max_EDA | max_EDA_Clean
3 | max_TEMP | energy_EDA | variance_X21
4 | variation_EDA_Phasic | variation_EDA | std_Y56
5 | skew_EDA_Tonic | energy_X5 | energy_Y31
6 | energy_X32 | energy_X4 | energy_EDA
7 | max_EDA_Clean | avg_X27 | std_X3
8 | energy_EDA_Phasic | avg_X25 | variation_Y22
9 | max_Y24 | avg_X61 | variation_Y28
10 | RMS_EDA_Phasic | energy_X65 | energy_Y30

Figure 4: force plot SHAP values

##  4 Stress detection models

Using a sliding window to combine statistical features of data with a random
train-test split can easily result in an overfit model. For stress detection
with biometric signals, Leave One Out Cross Validation (LOOCV) is crucial. It
prevents overfitting and offers a realistic gauge of the model’s ability to
generalize, as it tests the model with new data at each iteration. We fine-
tuned our proposed stress detection network using the LOOCV approach to
circumvent potential pitfalls. Moreover, stress detection models often
encounter data from new individuals they were not trained on. LOOCV ensures
the model’s performance is evaluated reliably and meaningfully. Our
investigation focused on three distinct network architectures: multivariate,
early-fusion, and late-fusion.

###  4.1 Multivariate

We analyzed biometric signals and facial landmarks for feature extraction.
Then, each model consumed the features separately. The one-dimensional
Convolutional Neural Network (1D-CNN), two-dimensional convolutional neural
network (2D-CNN), and pure fully connected, deep neural network (FCDNN) was
trained and tested using the LOOCV paradigm. By employing this approach, we
aimed to obtain separate results for biometric signals and facial landmarks,
allowing us to independently evaluate the models’ performance for each type of
data.

In our study, when using the 1D-CNN model to analyze facial landmarks, We
treat the 68 facial landmarks as time series data, similar to how we handle
ACC data that tracks coordinate changes over time. We focused on understanding
how these landmarks change sequentially rather than capturing complex spatial
relationships. On the other hand, using the 2D-CNN model, We pay more
attention to how facial landmarks are arranged in space, which helps us
capture overall spatial features.

###  4.2 Early-Fusion

Figure 5 shows the early-fusion stress detection model. The model consumes the
concatenation of selected facial landmarks and biometric signal features to
detect stress. We compared three early-fusion models, namely 1DCNN, 2D-CNN,
and FCDNN. The 1DCNN and 2DCNN models use a one or two-dimensional
convolutional layer to extract the spatiotemporal information of the
representation of the signals for fully connected neural network layers,
respectively. However, the FCDNN model directly consumes the concatenated
features to detect stress. This integration enables the model to
simultaneously analyze the biometric signals and visual information following
the LOOCV paradigm.

Figure 5: 1D-CNN Early-Fusion Network

###  4.3 Late-Fusion

The proposed network architecture Figure 6 consists of two components that
incorporate biometric signals and 68 landmarks in parallel. The late-fusion
models consume the same features extracted using feature extraction modules.
The biometric signals are fed into a 1D-CNN model, allowing us to capture
temporal patterns and dependencies. Meanwhile, the landmarks data is processed
using a 2D-CNN model to extract spatial information. We employ the decision-
level Fusion technique to merge the results and produce a final output using
the Softmax classifier (Figure 6). In the second experiment, we implemented a
concatenation-based approach, wherein we fused the outputs of the two
mentioned models, effectively giving rise to a novel composite model. We also
used a late-fusion concatenation model, combining one and two-dimension
convolutional networks (Figure 7) before the decision layer (SoftMax). The
model concatenates the representation of facial landmarks and biometric
signals before the decision layer for stress detection. The concatenation
late-fusion model showed promising results compared to other networks.
However, the decision-level concatenation model outperforms the late-fusion
network.

Figure 6: decision-level Fusion Network Figure 7: late-fusion Concatenation
Network

We used accuracy, precision, recall, F1 score, and computational cost to
perform model comparisons.

  * •

Accuracy: The ratio of the number of correct predictions to the total number
of predictions

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (1)

  * •

Precision: Measures the proportion of correctly predicted positive instances
out of all instances predicted as positive

$Precision=\frac{TP}{TP+FP}$ (2)

Precision is a measure of how accurately the model can classify a specific
class and in cases where the work is severely stressful, like military,
firefighting, and nursing.

  * •

Recall: Measures the proportion of correctly predicted positive instances out
of all actual positive instances

$Recall=\frac{TP}{TP+FN}$ (3)

Recall is important when misclassifying the stress is not crucial (e.g., the
wellness, meditation, and relaxation tasks). We prefer models with higher
recall scores in stress treatment tasks.

  * •

F1_score: Provides a balanced measure of a classification model’s performance
by combining the precision and recall scores.

$F1\\_Score=2\times\frac{Precision\times Recall}{Precision+Recall}$ (4)

TP, TN, FP, and FN are True Positive, True Negative, False Positive, and False
Negative, respectively.

F1_score is needed in mental health applications where both stress onset and
relaxation are important. Furthermore, the models with higher f1_score are
desired in physiological signal analysis than the precision and recall score.

Table 4: Multimodal and multivariate neural network models performance (%) comparison  Network | Multi-Variate | Early-Fusion | Late-Fusion
---|---|---|---
Model | 1D-CNN | 2D-CNN | FCDNN | 1D-CNN | 2D-CNN | FCDNN | Decision | Concat
Data | BIO | LND | BIO | LND | BIO | LND
Acc | 93.01 | 91.09 | 91.28 | 95.02 | 92.08 | 92.42 | 98.38 | 94.13 | 94.62 | 94.39 | 92.67
Prc | 90.00 | 86.30 | 88.30 | 88.30 | 87.33 | 83.51 | 96.73 | 93.65 | 93.08 | 94.94 | 91.11
Rec | 85.44 | 83.80 | 84.91 | 86.74 | 84.80 | 81.52 | 95.61 | 87.31 | 89.26 | 86.35 | 86.53
F1 | 85.71 | 82.62 | 84.83 | 86.45 | 84.92 | 81.91 | 95.70 | 88.63 | 90.12 | 88.63 | 87.54

##  5 Results

The performance of trained deep neural networks with regard to multi-variate,
early, and late-fusion approaches was evaluated on facial landmarks, biometric
signals, and a combination of their features. We compared the performance and
computational costs of the networks using training and testing times. We
trained and tested ten networks using six feature selection methods and
compared the results of the networks in detail. Furthermore, Table 4
summarizes the performance of each network in terms of the above-mentioned
metrics.

###  5.1 Multivariate Results

In the first experiment, we use the standard multivariate signals for a single
neural network architecture. Biometric signals are multi-variate, but each
signal sequence is still 1-dimensional, and 1DCNN has been shown to capture
patterns within the individual signals and cross-channel relationships [6,
54]. The 1D-CNN model captures temporal patterns and biometric signals, while
the 2D-CNN extracts spatial features of facial landmarks. We can observe in
Table 4 that 1D-CNN provides good performance for biometric signals. This is
on par with prior studies where we achieved 93.01% accuracy [42, 45, 46, 48,
49, 50, 17, 35]. Landmark data (LND) on the other hand are vector data (like
images), and we observe that 2D-CNN models provide better performance for
facial landmarks. The 2D CNN results are close to what is observed in [38, 52,
53, 55, 51]. Fully connected DNN models, on the other hand, are good at
capturing non-linear relationships, which has good overall performance when
you look at both signals. Fully connected DNN models treat data in their true
form and do not capture temporal dependencies, making these models vulnerable
to capturing false positives as true. In other words, the fully connected DNN
model cannot distinguish between decreasing and increasing trends. For
example, when stress has passed, the signs from the tonic EDA signal still
remain. As expected, the 1D-CNN model works better with biometric signals,
resulting in 93.01% accuracy, and 2D-CNN works best with facial landmarks,
showing 95.02% accuracy compared to fully connected deep neural network model
with around 92% for both biometric and facial landmarks signals.

###  5.2 Late-Fusion Results

In the case of late fusion, each modality is processed independently, and
their outputs are combined at a later stage, usually closer to the final
decision or prediction stage. The main advantage of this approach is its
flexibility because each modality is processed independently, accommodating
the unique characteristics of each data source. Also, the spatial and temporal
scales need not perfectly align for late-fusion. We experiment with late-
fusion models that combine one and two-dimensional convolutional neural
networks. The first late-fusion approach involved combining the outcomes of
the 1D-CNN and 2D-CNN models through decision-level fusion. We can observe in
Table-4 that results from late-fusion are better than the multi-variate model
but lower than early-fusion for both decision-based fusion and concatenation-
based fusion.

The concatenation-based approach concatenates the input and output of every
mentioned model into a single composite model. The fusion of two distinct
models, each specialized in a different modality, contributed to the overall
accuracy of 94.39% and 92.67% for stress detection using decision-level and
concatenation fusion, respectively. We can observe that the late fusion model
provides better performance compared to the multivariate model, but the model
performance is slightly lower compared to the 1D CNN-based early fusion model.

###  5.3 Early-Fusion Results

Early fusion, or feature-level fusion, combines features at the early
processing stage in the end-to-end model architecture. This is most suitable
when data from modalities correspond to the same instance or context [10].
Stress signals may exhibit some form of temporal alignment, but not always.
There is high inter-subject variability with respect to how individuals
perceive and react to stress [9]. Also, acute stress leads to immediate and
observable physiological and behavioral changes [3] compared to chronic stress
resulting from subtle and long-term changes. It is also observed by [4, 14]
that physiological signals such as HRV, skin conductance, and facial
expressions due to the stressful event also occur within the close temporal
context. In our case, the dataset includes two individual signals that have
been shown to have temporal proximity. We observe that Early fusion performs
better with accuracy, recall, and F1-score in most cases for 1D-CNN, 2D-CNN,
and FCDNN compared to multi-variate models, but 1D-CNN-based early fusion has
the best performance overall. The overall accuracy is 98.38%, which I better
compared to previous studies which are based on multi-variate signals [56),
58, 59, 15, 36, 11]. In the context of this dataset, the model is able to
focus on the most relevant and informative features from both modalities. The
analysis of False Positive (FP) and False Negative (FN) rates for biometric
data, landmark data, and their combination (fusion) for the 1D-CNN model is
illustrated in Figure8. Using biometric signals and facial landmarks showed
good accuracy in identifying non-stressful instances but did not perform well
in distinguishing stressed instances. However, when both signals are combined,
the model achieves better accuracy in identifying all three instances.
Reducing the FN rate is essential to ensure no high-stress situations are
overlooked and individuals receive appropriate warning signals. The combined
biometric and landmark data fusion approach showed promising results in
decreasing the FN rate and enhancing the reliability of the stress detection
system for individuals experiencing high stress.

Figure 8: FP, and FN rate for each class in 1D-CNN Early-Fusion Network

We investigated various feature selection methods, and the following points
summarize the results with the 1D-CNN model performance on combining both data
types. Some of the results are as follows:

  * •

Using RF importance resulted in an Accuracy of 96.94%, Precision of 95.72%,
Recall of 91.20%, and F1-score of 92.93%.

  * •

Using RFE resulted in an Accuracy of 96.63%, Precision of 95.34%, Recall of
92.54%, and F1-score of 92.97%.

  * •

Using Variance Threshold resulted in an Accuracy of 92.76%, Precision of
92.40%, Recall of 87.52%, and F1-score of 88.19%.

We investigated the importance of using the Tonic and Phasic components of the
EDA signal. Our experiment showed that using these components in the data in
the 1D-CNN model resulted in 3.57%, 9.23%, 10.47%, and 10.71% higher Accuracy,
Precision, Recall, and F1-score, respectively.

Table 5: The computational cost of the models

Network Multi-Variate Early-Fusion Late-Fusion Model 1D-CNN 2D-CNN FCDNN
1D-CNN 2D-CNN FCDNN decision Concat Data BIO LND Bio LND BIO LND Trainable
Parameters 23,931 79,931 4,923 27,323 32,703 46,703 79,931 27,323 46,703
51,254 51,176 Train Time (s) 2,610 2,590 2,870 2,930 2,290 2,220 2,820 2,870
2,310 5,810 3,850 Test Time (s) 4.19 4.01 3.4 3.65 3.27 3.14 4.07 3.64 3.35
3.73 5.86

###  5.4 Computational Cost

Table 5 presents the computational cost of these models with respect to the
trainable parameters, training time, and prediction time. We used a
two-28-core (56 CPUs) node with Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz CPU
and L1d, L1i, L2, and L3 cache of 32K, 32K, 256K, 35840K, respectively. The
node used a P100 Tesla GPU for deep-learning computations. Early-fusion models
show lower training and test times, while late-fusion models require more
training time. The early fusion model needs lower training time due to the
model only needs to be trained once on the combined feature set. However, in
the late-fusion model, Each model needs to be trained separately, which can
take more time, especially because there are many features. Moreover, The
complexity of the overall system is higher because it includes two different
models. The 2D-CNN model on biometric signals had the lowest number of
parameters. Because all the signals were concatenated before the convolutional
model, it resulted in lower training time and accuracies due to not capturing
enough information. The FDCNN model using facial landmarks requires lesser
training time than late fusion models; however, the overall performance of
these models drops about 8%. However, the 1D-CNN model has the maximum number
of trainable parameters but has a lower training time than the late-fusion
level neural networks. The 1D-CNN model is the most accurate with combined
data but has the most trainable parameters. The balance between accuracy,
complexity, and computational efficiency varies across models, showing the
importance of modality fusion and pre-processing choices for effective stress
detection.

##  6 Conclusion

This paper studies stress detection through multimodal learning that
encompasses biometric signals and facial landmarks. We used the EmpathicSchool
dataset that contains video-based facial expressions and physiological signals
when subjects were asked to perform stressful student tasks. We experimented
with various early and late-fusion techniques based on 1-D CNN, 2D-CNN, and
fully connected DNN models for multi-modal stress detection. We observe that
early-fusion techniques with a 1D-CNN model provide the best performance with
an accuracy of 98.38%. This model performs better than multi-variate unimodal
or multi-modal signals with late fusion. The fully connected DNN demonstrates
strong performance, exceeding the 1D-CNN model on landmarks data alone and
enhancing overall precision. Conversely, the 2D-CNN model performed well with
facial landmarks data due to its ability to capture both spatial relationships
and temporal dynamics of facial landmarks.

This study’s results are essential in creating reliable stress detection
methods, especially in healthcare, education, and the workplace. The study
should be further expanded to include a more diverse population and tasks. The
source code and datasets used in this study are for public use through the
University of Louisiana CPHS-lab GitHub repository [18].

## References

  * [1] Cohen, S., Janicki-Deverts, D., & Miller, G. E. Psychological stress and disease. Jama 2007, 298(14), 1685–1687.
  * [2] Ghazal, T. M., Hasan, M. K., Alshurideh, M. T., Alzoubi, H. M., Ahmad, M., Akbar, S. S., Al Kurdi, B., & Akour, I. A. IoT for smart cities: Machine learning approaches in smart healthcare—A review. Future Internet 2021, 13(8), 218.
  * [3] Morgado, P., & Cerqueira, J. J. The impact of stress on cognition and motivation. Frontiers in Behavioral Neuroscience 2018, 12, 326.
  * [4] Soni, A., & Rawal, K. A review on physiological signals: Heart rate variability and skin conductance. In Proceedings of First International Conference on Computing, Communications, and Cyber-Security (IC4S 2019) 2020, 387–399.
  * [5] Hosseini, S., Gottumukkala, R., Katragadda, S., Bhupatiraju, R. T., Ashkar, Z., Borst, C. W., & Cochran, K. A multimodal sensor dataset for continuous stress detection of nurses in a hospital. Scientific Data 2022, 9(1), 255.
  * [6] Kiranyaz, S., Ince, T., Abdeljaber, O., Avci, O., & Gabbouj, M. 1-D convolutional neural networks for signal processing applications. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2019, 8360–8364.
  * [7] Bradski, G. The OpenCV Library. Dr. Dobb’s Journal of Software Tools 2000.
  * [8] King, D. E. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 2009, 10, 1755–1758.
  * [9] Healey, J. A., & Picard, R. W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on intelligent transportation systems 2005, 6(2), 156–166.
  * [10] Dong, Y., Gao, S., Tao, K., Liu, J., & Wang, H. Performance evaluation of early and late fusion methods for generic semantics indexing. Pattern Analysis and Applications 2014, 17, 37–50.
  * [11] Bobade, P., & Vani, M. Stress detection with machine learning and deep learning using multimodal physiological data. In 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA) 2020, 51–57.
  * [12] Zhang, J., Yin, H., Zhang, J., Yang, G., Qin, J., & He, L. Real-time mental stress detection using multimodality expressions with a deep learning framework. Frontiers in Neuroscience 2022, 16, 947168.
  * [13] Islam, T., & Washington, P. Personalized Prediction of Recurrent Stress Events Using Self-Supervised Learning on Multimodal Time-Series Data. arXiv preprint arXiv:2307.03337 2023.
  * [14] Hosseini, L., Sharif Nia, H., & Ashghali Farahani, M. Development and psychometric evaluation of family caregivers’ hardiness scale: a sequential-exploratory mixed-method study. Frontiers in Psychology 2022, 13, 807049.
  * [15] Hosseini, M., Sohrab, F., Gottumukkala, R., Bhupatiraju, R. T., Katragadda, S., Raitoharju, J., Iosifidis, A., & Gabbouj, M. EmpathicSchool: A multimodal dataset for real-time facial expressions and physiological data analysis under different stress conditions. arXiv preprint arXiv:2209.13542 2022.
  * [16] Montesinos, V., Dell’Agnola, F., Arza, A., Aminifar, A., & Atienza, D. Multi-modal acute stress recognition using off-the-shelf wearable devices. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2019, 2196–2201.
  * [17] Kuttala, R., Subramanian, R., & Oruganti, V. R. M. Multimodal Hierarchical CNN Feature Fusion for Stress Detection. IEEE Access 2023, 11, 6867–6878.
  * [18] Majid Satya Ravi. Stress-Detection-in-Nurse. GitHub 2021.
  * [19] Sioni, R., & Chittaro, L. Stress detection using physiological sensors. Computer 2015, 48(10), 26–33.
  * [20] Christ, M., Braun, N., Neuffer, J., & Kempa-Liehr, A. W. Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package). Neurocomputing 2018, 307, 72–77.
  * [21] Ho, T. K. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition 1995, 278–282.
  * [22] Kleinbaum, D. G., Dietz, K., Gail, M., Klein, M., & Klein, M. Logistic regression. Springer 2002.
  * [23] Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., et al. Xgboost: extreme gradient boosting. R package version 0.4-2 2015, 1(4), 1–4.
  * [24] Noble, W. S. What is a support vector machine? Nature biotechnology 2006, 24(12), 1565–1567.
  * [25] Windeatt, T. Accuracy/diversity and ensemble MLP classifier design. IEEE Transactions on Neural Networks 2006, 17(5), 1194–1211.
  * [26] Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review 1958, 65(6), 386.
  * [27] Montavon, G., Samek, W., & Müller, K. R. Methods for interpreting and understanding deep neural networks. Digital signal processing 2018, 73, 1–15.
  * [28] Alberdi, A., Aztiria, A., & Basarab, A. Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review. Journal of biomedical informatics 2016, 59, 49–75.
  * [29] Fukazawa, Y., Yamamoto, N., Hamatani, T., Ochiai, K., Uchiyama, A., & Ohta, K. Smartphone-based mental state estimation: A survey from a machine learning perspective. Journal of Information Processing 2020, 28, 16–30.
  * [30] Zhai, J., & Barreto, A. Stress detection in computer users based on digital signal processing of noninvasive physiological variables. In 2006 international conference of the IEEE engineering in medicine and biology society 2006, 1355–1358.
  * [31] Kurniawan, H., Maslov, A. V., & Pechenizkiy, M. Stress detection from speech and galvanic skin response signals. In Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems 2013, 209–214.
  * [32] Abouelenien, M., Burzo, M., & Mihalcea, R. Human acute stress detection via integration of physiological signals and thermal imaging. In Proceedings of the 9th ACM international conference on pervasive technologies related to assistive environments 2016, 1–8.
  * [33] Wijsman, J., Grundlehner, B., Liu, H., Hermens, H., & Penders, J. Towards mental stress detection using wearable physiological sensors. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2011, 1798–1801.
  * [34] Sandulescu, V., Andrews, S., Ellis, D., Bellotto, N., & Mozos, O. M. Stress detection using wearable physiological sensors. In Artificial Computation in Biology and Medicine: International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2015, Elche, Spain, June 1-5, 2015, Proceedings, Part I 6 2015, 526–532.
  * [35] Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., & Van Laerhoven, K. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction 2018, 400–408.
  * [36] Koldijk, S., Neerincx, M. A., & Kraaij, W. Detecting work stress in offices by combining unobtrusive sensors. IEEE Transactions on affective computing 2016, 9(2), 227–239.
  * [37] Rizwan, M. F., Farhad, R., Mashuk, F., Islam, F., & Imam, M. H. Design of a biosignal based stress detection system using machine learning techniques. In 2019 international conference on robotics, electrical and signal processing techniques (ICREST) 2019, 364–368.
  * [38] Giannakakis, G., Pediaditis, M., Manousos, D., Kazantzaki, E., Chiarugi, F., Simos, P. G., Marias, K., & Tsiknakis, M. Stress and anxiety detection using facial cues from videos. Biomedical Signal Processing and Control 2017, 31, 89–101.
  * [39] Garcia-Ceja, E., Osmani, V., & Mayora, O. Automatic stress detection in working environments from smartphones’ accelerometer data: a first step. IEEE journal of biomedical and health informatics 2015, 20(4), 1053–1060.
  * [40] Gjoreski, M., Gjoreski, H., Luštrek, M., & Gams, M. Continuous stress detection using a wrist device: in laboratory and real life. In proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing: Adjunct 2016, 1185–1193.
  * [41] Kim, J., & André, E. Emotion recognition based on physiological changes in music listening. IEEE transactions on pattern analysis and machine intelligence 2008, 30(12), 2067–2083.
  * [42] Padmaja, B., Prasad, V. V. R., & Sunitha, K. V. N. A machine learning approach for stress detection using a wireless physical activity tracker. International Journal of Machine Learning and Computing 2018, 8(1), 33–38.
  * [43] Chen, S., Jiang, K., Hu, H., Kuang, H., Yang, J., Luo, J., Chen, X., & Li, Y. Emotion recognition based on skin potential signals with a portable wireless device. Sensors 2021, 21(3), 1018.
  * [44] Feng, X., Lu, X., Li, Z., Zhang, M., Li, J., & Zhang, D. Investigating the Physiological Correlates of Daily Well-being: A PERMA Model-Based Study. The Open Psychology Journal 2020, 13(1).
  * [45] Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J., & Xu, X. Wearable emotion recognition using heart rate data from a smart bracelet. Sensors 2020, 20(3), 718.
  * [46] Hsu, Y. L., Wang, J. S., Chiang, W. C., & Hung, C. H. Automatic ECG-Based Emotion Recognition in Music Listening. IEEE Transactions on Affective Computing 2020, 11(1), 85–99.
  * [47] Siirtola, P. Continuous stress detection using the sensors of commercial smartwatch. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers 2019, 1198–1201.
  * [48] Sharma, R., Pachori, R. B., & Sircar, P. Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomedical Signal Processing and Control 2020, 58, 101867.
  * [49] Zhu, L., Ng, P. C., Yu, Y., Wang, Y., Spachos, P., Hatzinakos, D., & Plataniotis, K. N. Feasibility study of stress detection with machine learning through eda from wearable devices. In ICC 2022-IEEE International Conference on Communications 2022, 4800–4805.
  * [50] Zhu, L., Spachos, P., & Gregori, S. Multimodal physiological signals and machine learning for stress detection by wearable devices. In 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA) 2022, 1–6.
  * [51] Jeon, T., Bae, H., Lee, Y., Jang, S., & Lee, S. Stress recognition using face images and facial landmarks. In 2020 International Conference on Electronics, Information, and Communication (ICEIC) 2020, 1–3.
  * [52] Zhang, H., Feng, L., Li, N., Jin, Z., & Cao, L. Video-based stress detection through deep learning. Sensors 2020, 20(19), 5552.
  * [53] Cardone, D., Perpetuini, D., Filippini, C., Spadolini, E., Mancini, L., Chiarelli, A. M., & Merla, A. Driver stress state evaluation by means of thermal imaging: A supervised machine learning approach based on ECG signal. Applied Sciences 2020, 10(16), 5673.
  * [54] Khoshkhahtinat, A., Zafari, A., Mehta, P. M., Akyash, M., Kashiani, H., & Nasrabadi, N. M. Multi-Context Dual Hyper-Prior Neural Image Compression. arXiv preprint arXiv:2309.10799 2023.
  * [55] Kopaczka, M., Breuer, L., Schock, J., & Merhof, D. A modular system for detection, tracking and analysis of human faces in thermal infrared recordings. Sensors 2019, 19(19), 4135.
  * [56)] Seo, W., Kim, N., Park, C., & Park, S. M. Deep Learning Approach for Detecting Work-Related Stress Using Multimodal Signals. IEEE Sensors Journal 2022, 22(12), 11892–11902.
  * [57] Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. Driver stress detection via multimodal fusion using attention-based CNN-LSTM. Expert Systems with Applications 2021, 173, 114693.
  * [58] Walambe, R., Nayak, P., Bhardwaj, A., & Kotecha, K. Employing multimodal machine learning for stress detection. Journal of Healthcare Engineering 2021, 1–12.
  * [59] Naegelin, M., Weibel, R. P., Kerr, J. I., Schinazi, V. R., La Marca, R., von Wangenheim, F., Hoelscher, C., & Ferrario, A. An interpretable machine learning approach to multimodal stress detection in a simulated office environment. Journal of Biomedical Informatics 2023, 139, 104299.
  * [60] Bailey, R. L. Electrodermal activity (EDA). The international encyclopedia of communication research methods 2017, 1–15.
  * [61] Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 2011, 3(1), 18–31.
  * [62] Breiman, L. Random forests. Machine learning 2001, 45, 5–32.
  * [63] Cohen, I., Huang, Y., Chen, J., Benesty, J., Benesty, J., Chen, J., Huang, Y., & Cohen, I. Pearson correlation coefficient. In Noise reduction in speech processing 2009, 1–4.
  * [64] Myers, L., & Sirois, M. J. Spearman correlation coefficients, differences between. Encyclopedia of statistical sciences 2004, 12.
  * [65] Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 1996, 58(1), 267–288.
  * [66] Guyon, I., & Elisseeff, A. An introduction to variable and feature selection. Journal of machine learning research 2003, 3(Mar), 1157–1182.
  * [67] Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. Gene selection for cancer classification using support vector machines. Machine learning 2002, 46, 389–422.
  * [68] Berger, R. Stress, trauma, and posttraumatic growth: Social context, environment, and identities. Routledge 2015.
  * [69] Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S. I. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2020, 2(1), 56–67.


*[ allpages, angle=55, scale=17, xpos=-70, ypos=50 ] PREPRINT

# How to Quantify the Security Level of Embedded Systems? A Taxonomy of
Security Metrics

1st Ángel Longueira-Romero, 2nd Rosa Iglesias, 3rd David Gonzalez  Industrial
Cybersecurity
Ikerlan Technology Research Centre (BRTA)
Arrasate/Mondragón, Spain
{alongueira, riglesias<EMAIL_ADDRESS>4th Iñaki Garitano  Dept. of
Electronics and Computing
Mondragon Unibertsitatea
Arrasate/Mondragón, Spain
<EMAIL_ADDRESS>
###### Abstract

Embedded Systems (ES) development has been historically focused on
functionality rather than security, and today it still applies in many sectors
and applications. However, there is an increasing number of security threats
over ES, and a successful attack could have economical, physical or even human
consequences, since many of them are used to control critical applications. A
standardized and general accepted security testing framework is needed to
provide guidance, common reporting forms, and the possibility to compare the
results along the time. This can be achieved by introducing security metrics
into the evaluation or assessment process. If carefully designed and chosen,
metrics could provide a quantitative, repeatable and reproducible value that
would reflect the level of security protection of the ES. This paper analyzes
the features that a good security metric should exhibit, introduces a taxonomy
for classifying them, and finally, it carries out a literature survey on
security metrics for the security evaluation of ES. In this review, more than
500 metrics were collected and analyzed. Then, they were reduced to 169
metrics that have the potential to be applied to ES security evaluation. As
expected, the $77.5\%$ of them is related exclusively to software, and only
the $0.6\%$ of them addresses exclusively hardware security. This work aims to
lay the foundations for constructing a security evaluation methodology that
uses metrics to quantify the security level of an ES.

###### Index Terms:

embedded systems, security metrics, quantitative security, security level,
security measurement, taxonomy, assurance, evaluation

##  I Introduction

The current society is driven by Embedded Systems (ES). All kinds of
industries, including automotive, healthcare, avionics, telecommunications,
and military defense, take advantage of ES to sense and measure the
environment, control different processes and perform all types of operations
[1, 2]. The development of Industry 4.0 heavily depends on the characteristics
of these ES since they represent the intelligence needed for automation and
digitalization.

As stated, there is clearly an increasing number of security threats over ES
[2] and countermeasures are being applied to protect them against known
threats. As such, there are some questions that arise and cannot be answered
yet: Do the countermeasures work properly? Is their coverage enough to
undermine threats? And, are they correctly implemented? In this situation, a
standardized and general accepted security testing methodology is needed to
provide guidance, confidence, structured procedures, common reporting forms,
and the possibility to compare results along time and among different systems.
This last factor can be achieved by introducing security metrics into the
evaluation or assessment process. Furthermore, if carefully designed and
chosen, metrics would provide a quantitative, repeatable and reproducible
value that would reflect the level of security protection of the ES. These
metrics can shed some light on measuring the degree of confidence in the
effectiveness of the implementation of security measures in an ES. Metrics
would allow the comparison of different evaluations of the same device (same
hardware, same software, and same versions of both) over time, and even
evaluations of different versions or devices.

Security has been widely analyzed in Information Technology (IT) systems, but
ES have some particular properties, such as resource limitation, physical
protection difficulty, use of insecure industrial protocols, cost limitations,
among others, which make it very challenging to secure them. Countermeasures,
such as encryption, authentication, firewalls, secure protocols (e.g.,
TLS/SSL), intrusion detection, and intrusion prevention systems are commonly
used in networks and PCs, but these cannot be directly applied to solve
security issues of ES. On the other hand, most ES are not upgraded regularly,
but have much longer lifespan and a successful attack can be easily escalated
on other devices of the same kind. The aim of this paper is to provide a
broader set of criteria (metrics) as a part of a more comprehensive process
for evaluating the security of ES, that is, that allows to measure the degree
of confidence in the effectiveness of protections and countermeasures.

This paper structure is as follows: Section II describes current security
metrics in literature and standards, raised issues, and proposed taxonomies.
Section III describes the characteristics of a good security metric. In
Section IV, a new taxonomy for metrics based on the characteristics of ES is
proposed, along with an analysis of the metrics collected in the literature.
In Section V, conclusions and future work are introduced.

##  II Previous Work on Security Metrics

In general, most evaluation criteria reflect two different aspects of
security: functionality and assurance. Functionality refers to the means that
are used, e.g., types of access control, or authentication mechanisms. On the
other hand, security assurance is mostly based on development practices,
documentation, analysis and configuration, and less in testing because it is
more difficult, time-consuming and costly. Thus, current approaches are more
focused on subjective than objective measures, that is, they are mainly based
on the reviews of a group of evaluators. Without good metrics and
corresponding evaluation methods, security cannot be accurately diagnosed and
quantitatively measured.

First, to understand what a security metric is, it is important to distinguish
between “metric” and “measurement”. A measurement is a concrete and objective
attribute that provides a single-point-in-time view of a specific and discrete
factor; whereas, a metric is generated from the analysis of the raw data
provided by measurements [3]. An example of measurement would be the number of
spams detected. A corresponding metric would be the number of spams detected
last month compared to the number of spam detected during this month.

This section highlights the importance of security metrics, describes the
existing metrics proposed in international standards and analyzes the proposed
taxonomies.

###  II-A Metrics in the literature

In [4], the authors conclude that measurements and metrics are necessary to
justify investments in security and to manage security. On the other hand, in
[5], Pendleton et al. provide a comprehensive survey on quantitative system
security metrics. In their work, they analyze software, network, economic, and
effectiveness security metrics. To classify the surveyed metrics, a
hierarchical ontology is proposed, and then, each metric is briefly described.
A survey that focuses on model-based quantitative security metrics is
presented in [6]. The authors review the state of the art of network security
metrics in depth, more specifically, in the realm of model-based quantitative
metrics. They present a complete and thorough review of metrics proposals,
remarking the pros and cons of each metric.

In 2019, it is highlighted the lack of consistent security metrics towards a
common certification approach [7]. These metrics, together with a suitable
certification approach would help to assess and compare the security of
different ES, and would increase the trust of the end user.

###  II-B Metrics in standards and methodologies

Over time, standards have also proposed the use of security metrics to
evaluate the degree of compliance achieved.

The IEC 62443-1-3 standard [8] includes cybersecurity conformance metrics for
managing a security program throughout the life cycle of an industrial
automation control system. These metrics provide evidence of conformance for
verifiability, completeness, and accuracy. They can also be of assistance to
others for assessing the cyber robustness of their control system solutions.
This standard also provides a list with the characteristics of a good metric,
a metrics development and implementation process, and the steps to be followed
for creating metrics. The 62443-4-1 standard also includes some examples of
metrics that can be used at the component level, that is, for ES, network
components, host software and applications, e.g., functional security,
deployment security and current backlog. In Annex A of [8], possible metrics
are described to be used in the case of highest maturity level for an
organization. Some of these metrics are: results of static code analysis,
analysis of attack surface and deviations from Secure Coding Guidelines. They
show the effectiveness of the development process with measurable
improvements.

The Common Criteria for Information Technology Security Evaluation (Common
Criteria or CC for short) is an international standard (ISO/IEC 15408) for
computer security certification [9]. CC is a framework which provides
assurance that the process of specification, implementation and evaluation of
a computer security product has been conducted in a rigorous and standard and
repeatable manner at a level that is commensurate with the target environment
for use. To describe the rigor and depth of an evaluation, the CC defines the
Evaluation Assurance Levels (EALs) as an increasing scale. CC list seven
levels, from EAL1 (the most basic one) to EAL7 (the most stringent security
level). It is important to notice that the EAL levels do not measure security
itself. Instead, the emphasis is given to “functional testing”, confirming the
overall security architecture and design, and performing some testing
depending on the EAL to be achieved. The testing methodology defined in CC for
each EAL is underspecified. As a result, evaluations can vary from one
laboratory to another, and results are difficult to be reproduced. It can be
said that the highest level means that more testing was done, but not
necessarily that the product is more secure.

The Common Criteria for Information Technology Evaluation (CEM) [10] suggests
the use of an effective and measurable life-cycle model that addresses the
development and maintenance processes and that uses some quantitative
evaluation. It explicitly mentioned three metrics: source code complexity,
defect density (errors per size of code), and mean time to failure. The CEM
also suggests that generated documentation should include numerical values of
each metric used, as well as the actions taken as a result of the
measurements, but there is no reference about how to compute these metrics.

The National Institute of Standards and Technology (NIST) has done a lot of
research on security metrics, and has proposed nine security metrics for three
different aspects: (1) implementation, (2) effectiveness/efficiency, and (3)
impact [11]. NIST presents its security metrics taxonomy in [12] and [13]. The
taxonomy is comprehensive, presenting three security categories, (1)
management, (2) technical, and (3) operational; however these metrics address
security at the organization level, and do not apply to ES. A similar approach
is proposed in the security framework developed by the Industrial Internet
Consortium (IIC) [14]. According to IIC, metrics should be used from the
moment systems are conceived, from their design, creation to operation. Such
metrics help identify security problems early and assist in faster and more
efficient management and governance.

OSSTMM (Open Source Security Testing Methodology Manual) [15] describes a
complete security auditing methodology, offering fairly good tools to report
the result set. It is designed to audit the operational security of physical
locations, workflows, human security testing, physical security testing,
wireless security testing, telecommunication security testing, data networks
security testing and compliance. OSSTMM defines the rav, a scale measurement
of the attack surface. In this scale, 100 rav means perfect balance and a
lower value indicates insufficient controls and therefore a greater attack
surface. A value greater than 100 rav shows more controls than necessary,
which might be inefficient and sometimes a problem, since excessive controls
often increase complexity and maintenance issues. The rav does not measure
risk for an attack surface, rather it enables the measurement of it.

###  II-C Current Issues with Security Metrics

There are many recurrent problems associated with security metrics that are
identified in the literature. In [16] and [17], the authors identified that
hardly any security indicator has a solid theoretical foundation or empirical
evidence in support of the claimed correlation. They also highlighted that
many security metrics lack an adequate description of the scale, unit and
reference values for comparing and interpreting results, to name a few.
Moreover, only a few implementations or programs were available, and only one
performed some kind of benchmarking or comparison to similar metrics. The
authors concluded to a large extent that the information provided in the
papers is insufficient to directly apply the method, especially by non-
experts.

Even though metrics and methodologies have been proposed in the literature, it
is difficult to understand whether they are applicable in a given context, and
how to use them. It seems reasonable that future research should be focused on
the development of a convincing theoretical foundation, empirical evaluation,
and systematic improvement of existing approaches. As stated in this section,
there is clearly a growing interest in adopting and using security metrics,
but this interest confronts with a lack of widely accepted solutions.

###  II-D Taxonomies for security metrics

In [18, 19], Savola reviews the most common taxonomies for security metrics,
and also provides a new model to taxonomize security metrics for technical
systems to systematize and organize development activities. Although there are
many taxonomies for security metrics proposed in the literature, the most
common classification for metrics divides them as (1) organizational (to
describe and track how effectively organizational programs and processes
achieve cybersecurity goals); (2) technical (to describe and compare technical
objects -e.g., algorithms, specifications and architectures- and to indicate
the security level a specific system exhibits); and (3) operational (to
describe and manage the risk to an operational environment, including used
systems and operating practices) [20, 21].

##  III Features of a Good Metric

Metrics are important for quantifying the security level achieved; however,
computing metrics can be exhausting and time-consuming resulting in not being
practical. Furthermore, it is not resource-effective to measure everything,
thus using the appropriate metrics is critical. In this section, a conceptual
mapping of related criteria, based on the most three relevant research works
is described. This map will help in identifying security metrics that show the
degree of confidence in security effectiveness.

Measuring security, both qualitatively and quantitatively, is not an easy task
[4, 22, 23]. It is a long-standing open problem to the research community and
it is of practical importance to software industry today. Suitable metrics are
needed to achieve a foundational science to guide system design and
development, and to reveal the safety, security, and possible fragility of
complex systems. Using metrics allows systems to be compared and evaluated, so
we can increase accountability, demonstrate compliance and determine how much
our investments in products and processes are making our systems more secure
[21, 24].

Some authors have tried to characterize which criteria should meet a good
metric. Among their efforts, three main ideas can be found in the literature:
(1) the SMART criterion [25], (2) the PRAGMATIC criterion [26] and (3) the
characteristics identified in the work carried out by Savola [27].

Since Doran proposed the “SMART” (Specific, Measurable, Attainable,
Repeatable, Time-dependent) criteria in [25], it has been widely used,
developed and extended, and also adapted for security metrics development
[28]. On the other hand, Krag et al. [26] propose a set of nine criteria for
assessing and selecting metrics, using the acronym “PRAGMATIC” (Predictive,
Relevant, Actionable, Genuine, Meaningful, Accurate, Timely, Independent,
Cheap).

In [27, 29], Savola analyzes the literature to identify the quality criteria
and dimensions of security metrics and measurement process. In [29], the pre-
existing goodness criteria are listed. Afterwards, in [27], Savola takes
research one step further by conducting expert surveys and interviews to 141
security experts from 21 different countries, and extracting 19 quality
criteria. According to his results, correctness, measurability, and
meaningfulness are the main quality criteria along with usability.

The quality criteria that Savola proposes are intended to support human
decision-making in every step of the life cycle, e.g., security engineering
activities at design and development practices or in security management
activities at run time. In his work, Savola concludes that there is no
security approach that would allow the measurement of security as an universal
property, since there is no security metric that could fulfill all the quality
criteria, so security metrics cannot be used to measure security as a whole.
He finally points out that there is a need for widely-accepted approaches for
security measuring.

Figure 1 shows the mapping among the three different concepts explained in
each criterion. This mapping was carried out using the definition and the data
given by the authors, to find the common points among them. This figure can be
better visualized at https://embeddedsecuritymetrics.github.io/.

Figure 1: Mapping of criteria for identifying a good metric, according to the
SMART, and PRAGMATIC criteria; and the survey carried out by Savola. Each
black circle represents a common point between each one of the properties.

Savola’s approach classified properties into 19 items; whereas the other two
criteria have 9 and 5 criteria resulting in a higher mapping ratio. With this
figure it is not possible to deduce which property is more important in any
context, but it is possible to conclude that cost, accuracy, genuineness and
repeatability are the most common properties in all three criteria.

It should be noted that none of these criteria explicitly mentions that
metrics should be quantitative. This may be because, depending on the context,
a nominal scale could be used instead. Some authors explicitly avoid this
approach and prefer metrics to be quantitative [30]. In this mapping, many
criteria have associated the cheap feature. This is because there are
properties that directly affect the cost of computing a metric (the more
properties it meets, the more expensive it is).

On the other hand, there are two properties identified by Savola - i.e.,
contextual specific and representativeness - that do not have any
correspondence with the rest of the criteria.

In the context of ES, the quality criteria that Security Metrics (SM) should
address are (in no particular order):

  * •

Comparability: SM should support comparison of the targets that they
represent.

  * •

Cost effectiveness: measurement gathering and approaches should be cost
effective.

  * •

Measurability: SM are capable of having dimensions, quantity or capacity
ascertained in the ES.

  * •

Repeatability: The same results are achieved if a measurement is repeated in
the same context, with exactly the same conditions.

  * •

Reproducibility: The same results are achieved if a measurement is repeated in
the same context, with exactly the same conditions, and different persons.

##  IV Security Metrics and Taxonomy

In this section, the results of a literature review about security metrics,
and a new taxonomy for security metrics are described.

###  IV-A Selecting metrics for embedded systems

There is neither a single system metric nor “one-perfect” set of metrics that
could be suitable for every single case. The set of metrics that will be most
suitable depends on multiple factors. For that reason, it is convenient to
analyze whether the existing metrics can be applied to ES. In this research
work, a systematic review of the state of the art is carried out in order to
search for the currently available security metrics. The objective is to
analyze to what extent the proposed metrics can be applicable to the security
evaluation of ES. The details of this search are explained in the following
points.

####  IV-A1 Search and selection strategy

The data sources used are online databases, conference proceedings and
academic journals of IEEE Xplore, Elsevier, AMC Digital Library, Springer,
along with Google Scholar search engine. The search terms included the
keywords: “security”, “metric”, “measure”, “evaluation” and “assessment”,
considering another synonyms.

The inclusion criteria included:

  * •

Security measurements or metrics are the main subjects.

  * •

Surveys collecting security metrics were preferred.

  * •

The paper is primarily related to measuring security.

The initial number of metrics analyzed was 531, obtained mainly from [5, 31,
32, 33]. For each one, the following data was collected:

  * •

Definition: Definition of the metric given by the author.

  * •

Scale: Nominal, ordinal, interval, ratio, absolute and distribution [5].

  * •

Scope: User, software, hardware, device or organization.

  * •

Automation: Automatic, semi-automatic or manual.

  * •

Measurement: Static or dynamic measurement.

####  IV-A2 Filtering and exclusion criteria

In order to identify the subset of metrics that can be also applied to ES from
that initial set, the collected metrics were filtered according to the
following criteria:

  * •

Organizational specific metrics.

  * •

Network specific metrics.

  * •

Metrics without a clear definition.

  * •

Repeated metrics.

When the concept of a metric was found to be applicable with some
modification, it was considered to be eligible.

####  IV-A3 Data extraction and interpretation

After filtering the data, the final count of metrics was 169. An interesting,
albeit not very surprising fact, is that most of the metrics ($77.5\%$ of
them) were related exclusively to software - e.g., lines of code, number of
functions and so on. On the other hand, only $0.6\%$ of them were related
exclusively to hardware - e.g., side-channel vulnerability factor metric; and
finally, $14.8\%$ of them could be applied to both software and hardware -
e.g., the historically exploited vulnerability metric that measures the number
of vulnerabilities exploited in the past. The remaining $7.1\%$ is focused on
other aspects, such as user usability. This shows that there is a clear lack
of hardware security metrics in the literature, and main efforts are centered
in developing software-related metrics. It is worth commenting these metrics
are not only valid for ES, but could also be applied to other systems.

All the data extracted from the articles, including the 169 security metrics,
as well as the filtering and selection processes, are available at
https://embeddedsecuritymetrics.github.io/.

###  IV-B A new taxonomy for security metrics

Although Section II discussed the most commonly used taxonomy for classifying
security metrics, to the best of the authors knowledge, there is no taxonomy
that focuses on the specific assessment needs and limitations of ES. On the
other hand, the taxonomies proposed in the literature are usually at a very
high level, and therefore not applicable to ES. Thus, a new taxonomy is
proposed that is based on assets to be protected in ES. Figure 2 shows the
structure of the proposed taxonomy.

According to the research work in [34], embedded security cannot be solved at
a single level of abstraction; instead must be addressed at all abstraction
levels. The main structure of the proposed taxonomy is built on three main
blocks that aim to address all those abstraction levels: (1) Assets: elements
of the ES that can be evaluated, as hardware, software, data, services, crypto
keys, or communications; (2) Security Dimension: which attribute is to be
protected (foundational requirements from the IEC 62443); and (3) Metric
Properties: which are the characteristic parameters of the metric (such as
automation, or computing cost). It is important to notice that a metric might
cover more than one asset, and/or more than one security dimension, but at
least, one value for each one have to be assigned (dashed line in Figure 2).
Nevertheless, all values in metric properties are needed for each metric, as
they represent the way the metric behaves, the cost associated to a metric and
the information it returns (solid line in Figure 2).

Figure 2: Taxonomy structure for security evaluation metrics for ES. Dashed
line indicates AND/OR condition, and solid line AND condition.

The use of this taxonomy can be further explained using the Side-Channel
Leakage (SCL) metric as an example. This metric uses statistical tests to
measure if a device is prone to a side channel attack by comparing two sets of
data; a random one and a fixed one. The SCL metric can be classified as
follows:

  * •

Assets: system hardware.

  * •

Security dimension: data confidentiality.

  * •

Metric properties:

    * –

Automation: semi-automation.

    * –

Scale: nominal (yes/no).

    * –

Measurement: dynamic.

    * –

Computing cost: constant.

As can be seen, the main feature of the proposed taxonomy is that considers
all assets in ES. Unlike other proposals in the literature that are at a high
level - e.g., enterprise level - this one is specifically designed to evaluate
all aspects that comprise ES. Furthermore, it includes not only the CIA triad
(i.e. Confidentiality, Integrity and Availability), but also other dimensions,
such as, access control, use control, restrict data flow and timely-response
to events. These dimensions are foundational requirements in IEC 62443 that
need to be evaluated in ES.

##  V Conclusions and Future Work

Security metrics are not standardized, and they would help in quantifying the
security level of ES, that is, in giving the degree of confidence in the
effectiveness of protections and countermeasures. This research work proposes
a new taxonomy for security metrics, which takes into account the different
assets of ES. Since it is critical to identify proper and non-exhausting
security metrics for this security assurance, a conceptual mapping of criteria
for choosing the most appropriate metrics is given. On the other hand, the
literature has been analyzed, and 169 security metrics have been identified.
$77.5\%$ of them are dedicated exclusively to evaluating software security,
while only $0.6\%$ of them are oriented to hardware evaluation. This
difference clearly shows the wider adoption by the community of software-
related security metrics.

As a future work, a reduced set of metrics from those that have been analyzed
will be selected, based on the 5 proposed criteria, refined, and applied to a
real ES. This will be the foundation for proposing a security assessment
methodology specifically designed for ES, which will provide qualitative and
quantitative results. These results will give trust to end-users, make it
possible to monitor the evolution of the security level of a device over time,
and to compare different versions or devices.

## References

  * [1] P. Marwedel, Embedded System Design: Embedded Systems Foundations of Cyber-Physical Systems.  Springer, 01 2010.
  * [2] D. Kleidermacher and M. Kleidermacher, Embedded Systems Security: Practical Methods for Safe and Secure Software and Systems Development.  Newton, MA, USA: Newnes, 1st ed., 2012.
  * [3] T. Zeb, M. Yousaf, H. Afzal, and M. R. Mufti, “A quantitative security metric model for security controls: Secure virtual machine migration protocol as target of assessment,” China Communications, vol. 15, no. 8, pp. 126–140, 2018.
  * [4] A. Atzeni and A. Lioy, “Why to adopt a security metric? a brief survey,” Advances in Information Security, vol. 23, pp. 1 – 12, 2006.
  * [5] M. Pendleton, R. Garcia-Lebron, J.-H. Cho, and S. Xu, “A survey on systems security metrics,” ACM Comput. Surv., December 2016.
  * [6] A. Ramos, M. Lazar, R. H. Filho, and J. J. P. C. Rodrigues, “Model-based quantitative network security metrics: A survey,” IEEE Communications Surveys Tutorials, vol. 19, pp. 2704–2734, Fourthquarter 2017\.
  * [7] S. N. Matheu, J. L. Hernandez-Ramos, and A. F. Skarmeta, “Toward a cybersecurity certification framework for the internet of things,” IEEE Security Privacy, vol. 17, no. 3, pp. 66–76, 2019.
  * [8] I. E. Commission, “Security for industrial automation and control systems – Network and system security: System security compliance metrics,” standard, IEC.
  * [9] CC, “The Common Criteria for Information Technology Security Evaluation - Introduction and General Model.”
  * [10] CC, “Common Methodology for Information Security Technology Evaluation.”
  * [11] E. Chew, M. Swanson, K. Stine, N. Bartol, A. Brown, and W. Robinson, “Performance measurement guide for information security (draft),” July 2008\.
  * [12] M. Swanson, “Security self-assessment guide for information technology systems,” November 2001.
  * [13] M. Swanson, N. Bartol, J. Sabato, J. Hash, and L. Graffo, “Security metrics guide for information technology systems,” July 2003.
  * [14] IIC, “Industrial Internet of Things Volume G4: Security Framework,” 2016.
  * [15] ISECOM, The Open Source Security Testing Methodology Manual (OSSTMM).  ISECOM, 2010.
  * [16] M. Rudolph and R. Schwarz, “A critical survey of security indicator approaches,” in 2012 Seventh International Conference on Availability, Reliability and Security, pp. 291–300, Aug 2012.
  * [17] S. Sentilles, E. Papatheocharous, and F. Ciccozzi, “What do we know about software security evaluation? a preliminary study,” in QuASoQ@APSEC, 2018\.
  * [18] R. Savola, “A security metrics taxonomization model for software-intensive systems,” Journal of Information Processing Systems, 2009.
  * [19] R. Savola, “Towards a security metrics taxonomy for the information and communication technology industry,” in International Conference on Software Engineering Advances (ICSEA 2007), pp. 60–60, Aug 2007.
  * [20] W. H. Sanders, “Quantitative security metrics: Unattainable holy grail or a vital breakthrough within our reach?,” IEEE Security Privacy, Mar 2014\.
  * [21] S. Stolfo, S. M. Bellovin, and D. Evans, “Measuring security,” IEEE Security Privacy, May 2011.
  * [22] S. M. Bellovin, “On the brittleness of software and the infeasibility of security metrics,” IEEE Security & Privacy, vol. 4, no. 4, pp. 96–96, 2006.
  * [23] V. Verendel, “Quantified security is a weak hypothesis: A critical survey of results and assumptions,” in Proceedings of the 2009 Workshop on New Security Paradigms Workshop, NSPW ’09, (New York, NY, USA), pp. 37–50, ACM, 2009\.
  * [24] S. Pfleeger and R. Cunningham, “Why measuring security is hard,” IEEE Security Privacy, July 2010.
  * [25] G. T. Doran, “There’s a s.m.a.r.t. way to write managements’s goals and objectives.,” Management Review, vol. 70, no. 11, pp. 35–36, 1981.
  * [26] W. Krag Brotby and Gary Hinson, PRAGMATIC Security Metrics. Applying Metametrics to Information Security.  Auerbach Publications, 01 2013.
  * [27] R. M. Savola, “Quality of security metrics and measurements,” Computers & Security, vol. 37, pp. 78 – 90, 2013.
  * [28] G. Jelen, “SSE-CMM Security Metrics,” in National Institute of Standards and Techonology (NIST) and Computer System Security and Privacy Advisory Board (CSSPAB) Workshop, June 2000.
  * [29] R. Savola, “On the feasibility of utilizing security metrics in software-intensive systems,” International Journal of Computer Science and Network Security, vol. 10, pp. 230–239, 01 2010.
  * [30] A. Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt.  Addison-Wesley Professional, 01 2007.
  * [31] A. Ramos, M. Lazar, R. H. Filho, and J. J. P. C. Rodrigues, “Model-based quantitative network security metrics: A survey,” IEEE Communications Surveys Tutorials, vol. 19, no. 4, pp. 2704–2734, 2017.
  * [32] S. Yusuf Enoch, J. Hong, M. Ge, and D. Kim, “Composite metrics for network security analysis,” vol. 2017, pp. 137–160, 02 2017.
  * [33] P. Morrison, D. Moye, and L. L. Williams, “Mapping the field of software security metrics,” 2014.
  * [34] D. D. Hwang, P. Schaumont, K. Tiri, and I. Verbauwhede, “Securing embedded systems,” IEEE Security Privacy, vol. 4, no. 2, pp. 40–49, 2006\.