singlelinecheck=on

Predicting Driver Fatigue in Automated Driving with Explainability

Feng Zhou

Department of Industrial and Manufacturing Systems Engineering,

The University of Michigan, Dearborn, MI, USA

email<EMAIL_ADDRESS>
Areen Alsaid

Department of Industrial and Systems Engineering,

The University of Wisconsin, Madison, WI, USA

email<EMAIL_ADDRESS>
Mike Blommer, Reates Curry, Radhakrishnan Swaminathan, Dev Kochhar, Walter
Talamonti, and Louis Tijerina

Ford Motor Company Research and Advanced Engineering, Dearborn, MI, USA

email: {mblommer, rcurry4, sradhak1, dkochhar, wtalamo1<EMAIL_ADDRESS>
Manuscript type: Research Article
Running head: Fatigue Prediction with Explainability
Word count: 5117

Acknowledgment: This work was supported by Ford Summer Sabbatical Program

Corresponding author: Feng Zhou, 4901 Evergreen Road, Dearborn, MI 48128,
Email<EMAIL_ADDRESS>
##  1 Abstract

Research indicates that monotonous automated driving increases the incidence
of fatigued driving. Although many prediction models based on advanced machine
learning techniques were proposed to monitor driver fatigue, especially in
manual driving, little is known about how these black-box machine learning
models work. In this paper, we proposed a combination of eXtreme Gradient
Boosting (XGBoost) and SHAP (SHapley Additive exPlanations) to predict driver
fatigue with explanations due to their efficiency and accuracy. First, in
order to obtain the ground truth of driver fatigue, PERCLOS (percentage of
eyelid closure over the pupil over time) between 0 and 100 was used as the
response variable. Second, we built a driver fatigue regression model using
both physiological and behavioral measures with XGBoost and it outperformed
other selected machine learning models with 3.847 root-mean-squared error
(RMSE), 1.768 mean absolute error (MAE) and 0.996 adjusted $R^{2}$. Third, we
employed SHAP to identify the most important predictor variables and uncovered
the black-box XGBoost model by showing the main effects of most important
predictor variables globally and explaining individual predictions locally.
Such an explainable driver fatigue prediction model offered insights into how
to intervene in automated driving when necessary, such as during the takeover
transition period from automated driving to manual driving.

Keywords: Driver fatigue prediction, explainability, automated driving,
physiological measures

##  2 Introduction

A driver may become fatigued or drowsy because of sleep deprivation, boredom,
or monotony, time-on-driving tasks, medication side-effects, or a combination
of such factors. Research indicates that the incidence of driver fatigue can
be increased by monotonous automated driving (Vogelpohl ., 2019). This can be
dangerous in SAE Level 2 - Level 4 (SAE, 2018) automated vehicles after the
driver is out of the control loop for prolonged periods (Hadi ., 2020).
Depending on the automation level of the vehicle, drivers need a high level of
situation awareness in SAE Level 2 (partial automation) automated vehicles and
good capabilities to respond to emerging hazards for takeover requests in SAE
Level 3 (conditional automation) and Level 4 (high automation) automated
vehicles (Collet Musicant, 2019). For example, takeover requests will be
issued in conditional automated driving, when the vehicle hits the operational
limit, such as adverse weather conditions and construction zones (Ayoub .,
2019; Du, Zhou ., 20201, 20202), which require the driver to safely take over
control from automated driving. Therefore, it is important to make sure that
the driver is available and ready in certain situations in automated driving
(Zhou ., 2019; Du, Yang Zhou, 2020; Du, Zhou ., 20203).

Although driver fatigue has been widely studied in manual driving (e.g., see
(Dong ., 2010; Sikander Anwar, 2018)), the probe into the fatigue prediction
in automated driving seems limited. In automated driving, many researchers
instead focus on the influence of performing non-driving related tasks (e.g.,
cognitive workload, engagement, and distraction) on takeover performance (Du,
Zhou ., 20201; Clark ., 2017). On the other hand, if drivers are not involved
in non-driving related tasks, they would quickly show signs of fatigue
(Vogelpohl ., 2019), which could potentially influence their takeover
performance, too. For example, Gonçalves . (2016) found that participants felt
subjectively fatigued even after as short as 15 minutes of a monitoring task
in automated driving and Feldhütter . (2017) identified fatigue indicators
among 31 participants in a 20-minute automated driving scenario using eye-
tracking data. Furthermore, in conditional automated driving, Hadi . (2020)
found that the higher the degree of fatigue was, the worse the takeover
performance. Hence, it is critical to detect and predict driver fatigue in
monotonous automated driving between SAE Level 2 and Level 4.

Another phenomenon witnessed is that increasingly more researchers applied
advanced machine learning models in driver fatigue detection and prediction in
order to improve the performance of the models (see (Sikander Anwar, 2018))
due to their great successes in learning hidden patterns and making
predictions of unobserved data, such as deep learning models based on
convolutional neural networks (CNNs) and long short-term memory (LSTM). For
example, Dwivedi . (2014) used CNNs to explicitly capture various latent
facial features to detect driver drowsiness. Nagabushanam . (2019) proposed a
two-layer LSTM and four-layer improved neural network deep learning algorithm
for driver fatigue prediction and their method outperformed other machine
learning models.

However, the trust and acceptance of such models can be compromised without
revealing the domain knowledge, i.e., explainability or interpretations,
contained in the data (Doshi-Velez Kim, 2017). Unlike other domains, the
importance of explainable machine learning models in decision making with high
risks is even greater, such as medicine (Lundberg, Nair ., 2018) and
transportation (Zhou ., 2020). This is also advocated by Mannering . (2020) in
safety analysis to consider both predictability and causality using advanced
machine learning model. Furthermore, the domain knowledge captured by the
machine learning models can be further used as guidelines to address the
issues at hand. For example, Caruana . (2015) built a generalized additive
model with pairwise interactions to predict pneumonia risks and found that
those with asthma were less likely to die from pneumonia, which was counter-
intuitive. However, by examining the data and the model, the researchers found
that those with asthma were intensively cared, which was effective at reducing
the likelihood of dying from pneumonia compared to the general population.
Such knowledge explained the model behavior. Thus, similar knowledge can be
potentially identified and used in driver fatigue prediction using explainable
models in manual and automated driving to help provide effective intervention
measures.

Towards this end, we proposed an explainable machine learning model to predict
driver fatigue using XGBoost (eXtreme Gradient Boosting) (Chen Guestrin, 2016)
and SHAP (SHapley Additive exPlanations) (Lundberg, Nair ., 2018; Lundberg .,
2020) in automated driving. First, XGBoost is a highly effective and efficient
algorithm based on tree boosting and it is one of the most successful machine
learning algorithms in various areas, including driver fatigue prediction
(Kumar ., 2017). In order to understand the hidden patterns captured by the
XGBoost model, SHAP (Lundberg, Nair ., 2018; Lundberg ., 2020) was used to
explain the XGBoost model by examining the main effects of the most important
measures globally and explaining individual prediction instances locally. SHAP
uses the Shapley value from cooperative game theory (Shapley ., 1953) to
calculate individual contributions of the features in the prediction model and
satisfies many desirable properties in explaining machine learning models,
including local accuracy, missingness, and consistency (Lundberg ., 2020).
However, it is challenging to compute the exact Shapley values for features of
machine learning models, especially deep learning models. Lundberg, Erion Lee
(2018) proposed the SHAP algorithm to reduce the complexity of calculating
Shapley value in algorithms based on tree ensembles from $O(TL2^{M})$ to
$O(TLD^{2})$, where $T$ is the number of trees, $L$ is the largest number of
leaves in the trees, $M$ is the number of the features, and $D$ is the maximum
depth of the trees. Hence, XGBoost and SHAP were used in this paper to predict
driver fatigue and uncover the hidden patterns in the machine learning model.

##  3 RELATED WORK

###  3.1 Driver Fatigue Detection and Prediction

####  3.1.1 Manual Driving

Driver fatigue has been studied widely in manual driving and previous studies
examined driver fatigue from two main types of measures, including driving
behavioral measures and physiological measures. Driving behavior measures
mainly include steering motion and lane deviation (Koesdwiady ., 2016). For
example, Feng . (2009) found that driver fatigue was negatively correlated
with steering micro-corrections. Sayed Eskandarian (2001) proposed a fatigue
prediction model with drivers’ steering angles based on an artificial neural
network that classified fatigued drivers and non-fatigued driver with 88% and
90% accuracy among 12 drivers. Using a sleep deprivation study ($n$ = 12),
Krajewski . (2009) extracted features from slow drifting and fast corrective
counter steering to predict driver fatigue and their best prediction accuracy
was 86.1% in terms of classifying slight fatigue from strong fatigue. Li .
(2017) detected driver fatigue ($n$ = 10) by calculating approximate entropy
features of steering wheel angles and yaw angles within a short sliding window
with 88.02% accuracy. McDonald . (2014) applied a random forest steering
algorithm to detect drowsiness indicated by lane departure among 72
participants and it performed better than other algorithms (e.g., neural
networks, SVMs, boosted trees). Though driving behavioral measures are easier
to collect, it is still challenging to obtain high prediction accuracy
(McDonald ., 2014).

Many studies investigated driver fatigue using physiological measures, which
have proven to be highly correlated with driver fatigue (Dong ., 2010;
Sikander Anwar, 2018). First, many researchers used head- and eye-related
physiological data to detect fatigue (Ji ., 2004; Watta ., 2007). For
instance, Khan Mansoor (2008) extracted features from the driver’ face and
eyes to detect driver fatigue (indicated by eye closure) with a normalized
cross-correlation function, which had 90% accuracy. PERCLOS (percentage of
eyelid closure over the pupil over time) and the average eye closure speed
were used to detect driver fatigue using neural networks (Chang Chen, 2014).
The system was able to detect fatigue with a success rate of 97.8% among 4
participants. However, it dropped to 84.8% when the participants wore glasses,
and the reliability was susceptible to lighting, motion, and occlusion (e.g.,
sunglasses). Second, other popular methods used measures derived from EEG
(electroencephalogram), EOG (electrooculography), ECG (electrocardiography),
and EMG (electromyography) for fatigue detection. For example, Jung . (2014)
examined heart rate variability to monitor driver fatigue. Zhang . (2013)
extracted entropy and complexity measures from EEG, EMG, and EOG data of 20
subjects. Lee Chung (2012) combined both photoplethysmography (PPG) signals
and facial features to detect driver fatigue ($n$ = 10) and the model had a
true and false detection rate of 96% and 8%, respectively, using a dynamic
Bayesian network.

####  3.1.2 Automated Driving

Although these previous research endeavors provided insights into the
progression of driver fatigue in manual driving, limited research is conducted
in detecting and predicting driver fatigue in automated driving. Gonçalves .
(2016) found that participants were easily fatigued due to underload in
automated driving for 15 minutes of a monitoring task. Similarly, Feldhütter .
(2017) found fatigue signs due to underload in 31 participants using eye-
tracking data for as short as 20-minute automated driving. Moreover, Körber .
(2015) found that participants ($n$ = 20) experienced substantial passive
fatigue due to monotony after 42 minutes of automated driving using eye-
related data. Hadi . (2020) demonstrated that drivers’ ($n$ = 12) takeover
performance was significantly worse in various scenarios for fatigued driver
in conditional automated driving. Vogelpohl . (2019) ($n$ = 60) indicated that
compared to sleep-deprived drivers in manual driving, drivers in automated
driving exhibited facial indicators of fatigue 5 to 25 minutes earlier and
their takeover performance was significantly jeopardized. Therefore, fatigued
drivers could be one of the safety issues in takeover transition periods where
a high level of situation awareness is needed. These studies indicate the
necessity for driver fatigue detection and prediction in SAE Level 2 - Level 4
automated driving.

###  3.2 Explainable Machine Learning Models

To detect and predict driver fatigue, it is extremely important to develop
accurate machine learning models in both manual driving and automated driving.
For example, CNN was used to extract spatial facial features in detecting
driver fatigue (Dwivedi ., 2014) and LSTM was used to model temporal relations
of physiological measures to detect driver fatigue (Nagabushanam ., 2019).
Nevertheless, it can be _difficult_ to trust and accept such black-box models
without revealing its domain knowledge captured by the models, especially with
the risks associated with the decisions based on the models are high.
Therefore, the choice between simple, easier to interpret models and complex,
black-box models is one of the important factors to consider in deploying such
models. Usually there are two types of explainable models, i.e., model-based
and post-hoc explainability (Murdoch ., 2019). Typical examples of the model-
based explainability include linear regression models, logistic regression
models, and single decision tree models. However, their performance is usually
inferior compared to complex black-box models. Post-hoc explainability is then
used to explain the behaviors and working mechanisms of black-box models
approximately, such as SHAP (Lundberg, Nair ., 2018; Lundberg ., 2020) and
LIME (Local Interpretable Model-agnostic Explanations) (Ribeiro ., 2016). For
example, LIME was used to explain neural network models in credit scoring
applications (Munkhdalai ., 2019) and SHAP was used to explain ensemble
machine learning models to identify risk factors during general anesthesia
(Lundberg, Nair ., 2018). Such explanation not only identified the key
variables in modeling, but also increased trust in real applications (Ayoub .,
2021). Compared to LIME, SHAP was better in explaining machine learning models
in terms of local accuracy and consistency (Lundberg ., 2020).

##  4 EXPERIMENT DESIGN

###  4.1 Participants

We excluded participants who had caffeine consumption, sleep disorders or
other factors that might have impacted driver fatigue. Finally, twenty
participants were recruited in this study (14 males and 6 females between 20
and 70 years old). In order to elicit fatigue, unlike the common sleep
deprivation method, we made use of the nature of underload and monotony in
automated driving to elicit passive fatigue in the experiment in the
afternoons. According to previous studies (Gonçalves ., 2016; Feldhütter .,
2017; Körber ., 2015), participants were expected to show passive fatigue
signs as soon as in 15 minutes without doing any secondary tasks and such
fatigue was more prevalent in automated driving.

###  4.2 Apparatus

The study took place in the VIRTTEX (VIRtual Test Track EXperiment) driving
simulator at Ford (see Figure 1b), a large six degree-of-freedom motion base
simulator that uses a hydraulically powered Stewart platform to reproduce
vehicle motion. The visual environment consists of a 240° front field-of-view
and a 120° rear field-of-view. Drivers were seated in a Ford Edge cab with 3D
simulated sound to provide realistic interior and exterior environment sounds,
as well as a steering control loader for accurate road feedback and tire
forces to the driver. The simulator was configured with an SAE Level 3
automated driving system and auditory-visual displays to indicate automated
system status throughout the drive. The driver wore ISCAN© eye-tracking
goggles (ISCAN, Inc., MA, USA) outfitted with an eye camera, a dichroic
mirror, and a scene camera to track percent pupil occlusion in real time in
order to calculate the PERCLOS measure (see Figure 1c). Before entering the
simulator, the participant was outfitted with a BioHarness 3.0 Wireless Heart
Rate Physiological Monitor (Zephyr Technology, MD, USA) with Bluetooth to
capture physiological measures (see Figure 1d).

Figure 1: (a) Example PERCLOS time series data for a participant. (b) The
VIRTTEX driving simulator and driving scenarios with different views. (c) The
configuration of ISCAN Eye Tracker for PERCLOS measure collection. (d)
BioHarness model for physiological data collection.

###  4.3 Simulator Scenario

The simulated drive was on a 4-lane undivided rural roadway with light traffic
flowing with and opposing the drivers. Drivers were instructed to initially
stay in the right lane and follow a lead vehicle that was varying its speed
between 50 – 70 mph (80 – 115 kph). The driver then engaged the automated
driving system using the steering wheel controls. A key feature of this
scenario was that, after the automation was engaged for the data collection
portion of the study, there were no secondary tasks at all or no interaction
with the research staff until the end of the session in order to elicit
passive driver fatigue in automated driving. There were also no hazardous
events in the automated driving session, which lasted about 60 minutes. With
essentially no disruptions, some drivers entered the fatigued state quickly
(Gonçalves ., 2016; Feldhütter ., 2017; Körber ., 2015).

###  4.4 Predictor Variables and Response Variables

The response variable to be predicted was PERCLOS obtained from the ISCAN eye
tracker sampled at 60Hz. PERCLOS was operationally defined as the average
percent of the time the eyelids occluding the pupil (larger than 80%) using a
1-minute moving window at any point in the data collection session during the
experiment (Zhou ., 2020). A typical example of a fatigued participant was
shown in Figure 1a.

The reason that we used PERCLOS as the ground truth of our prediction model
was that it was a reliable indicator of driver fatigue (Zhou ., 2020), but it
was intrusive to measure in real applications (Figure 1 (c)). Therefore, we
collected 11 less intrusive measures as predictor variables as shown in Table
1 to predict driver fatigue indicated by PERCLOS. A low pass filter was used
to remove baseline wander noises in ECG data (Kher, 2019). Breathing wave
signals were filtered using a moving average filter to remove noise. Steering
wheel angles (swa), torque applied on steering wheel (intertq), and posture
data were also filtered by a low pass filter. Other measures, such as
hr_avg60, were then calculated based on the filtered signals.

Table 1: Predictor variables used in modeling driver fatigue Features | Unit |  Explanation
---|---|---
heart_rate_variability | millisecond (ms) |  Standard deviation of inter-beat interval
hr_avg60 | beats/minute |  Average heart rate with a 60s sliding window
br_avg60 | breaths/minute |  Average breathing rate with a 60s sliding window
br_std60 | breaths/minute |  Standard deviation of breathing rate with a 60s sliding window
hr_std60 | beats/minute |  Standard deviation of heart rate with a 60s sliding window
heart rate | beats/minute |  Number of beats in one minute
breathing | bits |  Breathing waveform (16Hz)
ECG | mV |  ECG waveform (250Hz)
intertq | Nm |  Torque applied to steering wheel (200Hz)
swa | degree |  Steering wheel angle (200Hz)
posture | degree |  Degree from subject vertical (1Hz)

###  4.5 Experimental Procedure

Once participants had signed the informed consent form, they were given a
brief study introduction via a PowerPoint® presentation. The study
introduction highlighted the study objective to examine driver reactions to an
automated driving system. Participants were then given an overview of the
automated driving system’s capabilities and training on how to engage and
disengage the system. Prior to entering the VIRTTEX simulator, the participant
donned the BioHarness 3.0 belt and Bluetooth connectivity was verified. Once
seated in the cab, the participant put on the eye-tracking goggles. Up to 10
minutes was spent on calibrating the eye-tracker (typically less than 5
minutes for drivers without glasses). Once the training was complete, the
participants transitioned right into the main drive. The driver was instructed
to engage the automation, monitor the driving task, and not to do any
secondary tasks or communicate with the experimenter.

##  5 Driver Fatigue Prediction and Explanation

###  5.1 XGBoost

XGBoost is a highly efficient and effective machine learning model both for
regression and classification (Chen Guestrin, 2016). In our driver fatigue
prediction, we used a one-second time window to discretize all the predictor
variables, $\mathbf{X}=\\{\mathbf{x}_{k}\\}$, and response variable,
$Y=\\{y_{k}\\}$, of all the participants, where $k=1,...,n$. The training data
set is indicated as $D={\\{\mathbf{x}_{k},y_{k},\mathbf{x}_{k}\in
R^{m},y_{k}\in R\\}}$. In this research, $n$ is the total number of the
samples and $m$ is the number of the features (i.e., predictor variables), and
$n=58846,m=11$. Let $\hat{y}_{k}$ denote the predicted result of a tree-based
ensemble model, $\hat{y}_{k}=\phi(x_{k})=\sum_{s=1}^{S}f_{s}(x_{k})$, where
$S$ is number of the trees in the regression model, $f_{s}\big{(}x_{k}\big{)}$
is the $s$-th tree. For XGBoost, the objective function is regularized to
prevent over-fitting as follows:

$L\big{(}\phi\big{)}=\sum_{k}l\big{(}y_{k},\hat{y}_{k}\big{)}+\sum_{s}\Omega\big{(}f_{s}\big{)},$
(1)

where $l$ is the loss function and in this research, we used root-mean-squared
error (RMSE). The penalty term $\Omega$ has the following form:

$\Omega\big{(}f_{s})=\gamma T+\frac{1}{2}\lambda\|w\|^{2},$ (2)

where $\gamma$ and $\lambda$ are the penalty parameters to control the number
of leaves $T$ and the magnitude of leaf weights $w$. In the training process,
XGBoost used an iterative process to minimize the objective function and for
the $i$-th step when adding a tree, $f_{i}$, to the model as follows:

$L^{i}\big{(}\phi\big{)}=\sum_{k}l\big{(}y_{k},\hat{y}_{k}^{(i-1)}+f_{i}(x_{k})\big{)}+\Omega\big{(}f_{i}\big{)},$
(3)

This formula was approximated with a 2nd order Taylor expansion by
substituting the loss function with mean-squared error and after tree
splitting from a given node, we have:

$\displaystyle L_{split}=\frac{1}{2}\big{(}\frac{(\sum_{k\in
K_{L}}g_{k})^{2}}{(\sum_{k\in K_{L}}h_{k}+\lambda)}+\frac{(\sum_{k\in
K_{R}}g_{k})^{2}}{(\sum_{k\in K_{R}}h_{k}+\lambda)}-\frac{\sum_{k\in
K}g_{k})^{2}}{(\sum_{k\in K}h_{k}+\lambda)}\big{)}-\gamma,$ (4)

where $K$ is a subset of observations for the given node and $K_{L},K_{R}$ are
subsets of observations in the left and right trees, respectively. $g_{k}$ and
$h_{k}$ are the first and second order gradient statistics on the loss
function and are defined as
$g_{k}=\partial_{\hat{y}^{(j-1)}}l\big{(}y_{k},\hat{y}_{k}^{(j-1)}\big{)},h_{k}=\partial^{2}_{\hat{y}^{(j-1)}}l\big{(}y_{k},\hat{y}_{k}^{(j-1)}\big{)}$.
For a given tree structure, the algorithm pushed $g_{k}$ and $h_{k}$ to the
leaves they belong to, summed the statistics together, and used Eq. (4) to
identify the optimal splitting, which was similar to the impurity measure in a
decision tree, except that XGBoost also considered model complexity in the
training process.

###  5.2 SHAP

SHAP uses Shapley values (Shapley ., 1953) based on coalitional game theory to
calculate individual contributions of each feature, which is named as SHAP
values. In this research, SHAP was used to explain the main effects in the
XGBoost model and individual predictions. According to (Lundberg ., 2020;
Lundberg, Nair ., 2018), SHAP values are consistent and locally accurate
individualized features that obey the missingness property. The definition of
the SHAP value of a feature-value set is calculated for a model $f$ below:

$\varphi_{i}(v)=\sum_{S\subseteq
N\setminus\\{i\\}}{\frac{|S|!(m-|S|-1)!}{m!}}(f_{x}(S\cup\\{i\\})-f_{x}(S)),$
(5)

where $f_{x}(S)=E[f(x)|x_{S}]$ is the contribution of coalition $S$ in
predicting driver fatigue, indicated by PERCLOS in this study, $S$ is a subset
of the input features, $N$ is the set of all the input features, and $m=11$ is
the total number of features. The summation extends over all subsets $S$ of
$N$ that does not contain feature $i$. However, it is challenging to estimate
the value of $f_{x}(S)$ efficiently due to the exponential complexity in Eq.
(5). Lundberg, Nair . (2018) proposed an algorithm to approximate the values
of $E[f(x)|x_{S}]$ for tree-based models, such as XGBoost, in $O(TLD^{2})$
time, where $T$ is the number of the trees, $L$ is the number of maximum
leaves in any tree, and $D=logL$. SHAP uses the difference between individual
fatigue prediction against the average fatigue prediction, which is fairly
distributed among all the feature-value sets in the data (see Figure 5).
Hence, it has a solid theory foundation in explaining our XGBoost model.

##  6 Results

###  6.1 Prediction Results by XGBoost

First, we compared the performance of the XGBoost prediction model with other
six regression models, including linear regression, linear SVM, quadratic SVM,
Gaussian SVM, decision trees, random forest. The setting of the XGBoost was as
follows: max depth = 10, learning rate = 0.1, objective = reg:squarederror,
number of estimators = 150, regularization parameter alpha = 1, subsample =
0.9, and colsample = 0.9. All the 11 predictor variables were included in all
the models with 10-fold cross validation, except the last entry for XGBoost
(best), where only 5 most important features were selected to obtain the best
performance (see Figure 3). We reported the results in predicting PERCLOS
(0-100) with the following three performance metrics, including RMSE (the
smaller the better), MAE (i.e., mean absolute error, the smaller the better),
and adjusted $R^{2}$ (the closer to 1, the better) defined as follows:

$RMSE=\sqrt{\frac{\sum_{k=1}^{n}\big{(}y_{k}-\hat{y}_{k}\big{)}^{2}}{n}},$ (6)
$MAE=\frac{\sum_{k=1}^{n}|y_{k}-\hat{y}_{k}|}{n},$ (7)
$Adj.\leavevmode\nobreak\ {R}^{2}=1-(1-R^{2})\frac{n-1}{n-m-1},$ (8)

where $n$ is the total number of the samples, $\hat{y}_{k}$ is the predicted
value of the ground truth, $y_{k}$, and
$R^{2}=\frac{SS_{regression}}{SS_{total}}=\frac{\sum_{k=1}^{n}\big{(}\hat{y}_{k}-\overline{y}\big{)}^{2}}{\sum_{k=1}^{n}\big{(}y_{k}-\overline{y}\big{)}^{2}}$,
and $m$ is the total number of the predictor variables. The results are shown
in Table 2. It can be seen that XGBoost outperformed other machine learning
models.

Table 2: Comparisons of prediction results of different machine learning
models

Model |  RMSE |  MAE |  Adj. $R^{2}$
---|---|---|---
Linear Regression |  26.429 |  20.189 |  0.250
Linear SVM |  28.972 |  18.793 |  0.102
Quadratic SVM |  25.995 |  16.537 |  0.269
Gaussian SVM |  18.027 |  11.915 |  0.653
Fine Tree |  6.753 |  2.516 |  0.951
Random Forest |  6.700 |  3.910 |  0.950
XGBoost (all) |  4.788 |  2.316 |  0.993
XGBoost (best) |  3.847 |  1.768 |  0.996

Note XGBoost (all) indicates all the 11 predictors were included in the model
and XGBoost (best) indicate only five of the most important features were
included (see Figure 3).

###  6.2 SHAP Explanation

####  6.2.1 Feature Importance

During the 10-fold cross-validation process, we used the test data in each
fold to calculate the SHAP values in order to improve its generalizability so
that each sample was calculated its SHAP value for exactly once. We used the
SHAP values in Eq. 5 to identify the most important features as shown in
Figure 2. The feature importance was sorted by their global impact
$\sum_{k=1}^{n}|\varphi_{k}^{m}|$ identified by SHAP plotted vertically as
follows: hr_avg60 is the most important, followed by heart_rate_variability,
br_avg60, br_std60, hr_std60, and so on. Every sample in the data was run
through the model and each dot (i.e., $\varphi_{k}^{m}$) was created for each
feature value and was plotted horizontally. The more important the feature is,
the more impact on the model output. For example, hr_avg60 had a range of SHAP
value (i.e., PERCLOS) between -40 and 60. Note that the SHAP value was
computed with regard to the base average output (see Figure 5) and hr_avg60
could push some extreme output 40 lower than the average and push some other
extreme output 60 higher than the average. It tended to show that the higher
the value of hr_avg60, the lower the predicted PERCLOS.

Figure 2: Importance ranking of 11 features identified by SHAP summary plot.
The higher the SHAP value of a feature, the higher the predicted PERCLOS.

(a)  (b)

Figure 3: How performance changes when the model added one feature at a time
from the most important one to the least important one: (a) RMSE and MAE; (b)
Adjusted $R^{2}$. Note the error bar was the standard deviation obtained in
the ten-fold cross validation process.

####  6.2.2 Prediction Results with an Optimal Subset of Features

We further added one feature at a time to the XGBoost model starting from the
most important identified as shown in Figure 2. Figure 3 shows that the
performance was increasing when more features were added until when there were
5 features (i.e., hr_avg60, heart_rate_variability, br_avg60, br_std60, and
hr_std60) in the prediction model. The performance was better than that
obtained by the model when 11 predictors were included, i.e., a subset of
important features had the optimal performance (see Table 2).

####  6.2.3 Main Effects

We also examined the main effects of the top five most important features when
the model had the best performance in Figure 3. Figure 4 shows the main
effects. Consistent with Figure 2, the overall trend is that the larger the
value of hr_avg60, the smaller the SHAP values (i.e., predicted PERCLOS), but
not in an exact linear fashion (see Figure 4c). The slope is much larger in a
narrow interval around 50 and 55 beats/min than others and between 55 and 63
beats/min, it is almost flat. When it is larger than 63 beats/min, the larger
the value of hr_avg60, the smaller the predicted PERCLOS. For
heart_rate_variability, it has a V-shape relationship with the predicted
PERCLOS (see Figure 4b). The predicted PERCLOS is decreasing when the value of
heart_rate_variability is smaller than about 50 ms while the predicted PERCLOS
is increasing when it is going up from 50 ms to 140ms. The overall trend for
br_avg60 is that the larger the value of br_avg60, the smaller the predicted
PERCLOS, and this trend was not obvious until the value of br_avg60 is larger
than 15 breaths/min (see Figure 4c). The overall trend for br_std60 is that
the predicted PERCLOS is decreasing when the value of br_std60 is increasing
until it reaches around 0.8 breaths/min, after which the predicted PERCLOS
seems flat (see Figure 4d). The predicted PERCLOS decreases when the value of
br_avg60 increases from 1 beat/min to 2 beats/min. Then the trend tends to be
reversed, i.e., the larger the value of br_avg60, the larger the value of the
predicted PERCLOS (see Figure 4e). Note the importance or the global impact of
each individual feature is also noticeable in the range of predicted PERCLOS,
where hr_avg60 has the maximum range, followed by heart_rate_variability,
while the rest have similar ranges.

(a)  (b)  (c)  (d)  (e)

Figure 4: Main effects of the most important features. (a) hr_avg60; (b)
heart_rate_variability; (c) br_avg60; (d) br_std60; (e) hr_std60. Note only
data between the 2.5th percentile and the 97.5th percentile were included in
the figures. Figure 5: SHAP individual explanations. (a) An example of low
predicted driver fatigue; (b) An example of high predicted driver fatigue.
Note the base value, 22.92, indicated the average predicted PERCLOS by all the
training data in the model and those in blue push the predicted value lower
and those in red push the predicted value higher.

####  6.2.4 Individual Explanations

SHAP is also able to produce individual explanations locally to show the
contributions of each input feature. Figure 5 shows two examples. The base
value is the averaged output from the model, which was 22.92, the feature-
value sets in blue push the output (predicted PERCLOS) lower, and the feature-
value sets in red push the output higher. For example, the feature-value sets
in Figure 5a push the output value to be 25.90, which was very close to the
ground-truth value at 26.16. hr_avg60 and br_std60 push the predicted PERCLOS
higher while br_avg60, heart_rate_variability push the predicted PERCLOS
lower. The feature hr_avg60 contributes the most to such an output. All the
feature-value sets in Figure 5b push the output higher to be 88.93, which was
also close to the ground true value at 90.91. Among them, br_std60 contributes
the most, followed by br_avg60, heart_rate_variability, and hr_avg60.
Therefore, despite the fact that globally hr_avg60 is the most important
feature, its contribution might not always be the largest locally.

##  7 Discussions

###  7.1 Fatigue Prediction

The proposed model used both physiological and behavioral measures to predict
driver fatigue indicated by the PERCLOS measure. The model used 11 predictor
variables as input and the model with best performance only made use of heart
data and breathing data and was able to predict driver fatigue with high
accuracy in real time. With Python 3.8 on a MacBook Pro with 2.3GHz Quad-Core
Intel Core i7 and macOS Catalina, the average prediction time for one sample
was only $3.6\times 10^{-6}$ seconds. Compared with the fatigue-related
studies in automated driving, our work used prediction models rather than
simply described the fatigue progression in automated driving. Moreover, we
identified the most important features in predicting driver fatigue in
automated driving using SHAP, including hr_avg60, heart_rate_variability,
br_avg60, br_std60, and hr_std60. Therefore, the included physiological
measures were found to be more important than the included behavioral
measures. The prediction model provided one good way to monitor, quantify,
detect, and predict driver fatigue in real time. Despite the fact that fatigue
has many components in terms of its bodily representation, multiple
physiological measures were able to predict driver fatigue with RMSE = 3.847,
MAE = 1.768, and adjusted $R^{2}=0.996$. During automated driving, wearable
physiological sensors can be potentially used to detect and predict drivers’
fatigued state in real time in a minimally invasive manner. Such insights give
us informed design guidelines in customizing driver fatigue models by tuning
only the most critical physiological measures.

###  7.2 Model Explanation

Unlike previous driver fatigue prediction studies, the most important findings
in this study are the relationships between the five most important measures
identified by SHAP and driver fatigue indicated by PERCLOS. For example,
hr_avg60 and br_avg60 tended to be negatively correlated with predicted driver
fatigue except at some specific, narrow intervals (see Figures 4a and 4c).
This could be caused by the individual differences or noises involved in the
dataset. However, such domain knowledge captured in the driver fatigue
prediction model can be used to help design systems that fight driver fatigue
in automated driving.

First, consistent with previous studies (e.g., (Ünal ., 2013)), a low heart
rate can be indicative of a low level of arousal with low vigilance. In order
to fight monotony in automated driving, music with high tempos, for example,
can be used to increase drivers’ heart rate to help drivers stay at an optimal
level of arousal to improve driving performance (Dalton ., 2007). Compared to
a control group with no music, participants with self-selected music increased
3 beats/min on average, which could decrease PERCLOS by as much as 40 (see
Figure 4a). Second, consistent with previous studies (e.g., (Sun ., 2011)), a
decreasing breathing rate was also a sign of the onset of fatigue. To fight
monotonous driver fatigue in automated driving, a breath booster system based
on haptic guidance was proposed to increase breathing rate and heart rate in
order to increase driver alertness and focus (Balters ., 2018). However, what
is less known is that a smaller br_std60 was also associated with driver
fatigue. A variable breath pattern could also be used to fight driver fatigue.
Third, heart_rate_variability was computed as the standard deviation of inter-
beat intervals while hr_std60 was calculated as the standard deviation of
heart rate (see Table 1). In addition, both had a V-shape relationship with
the predicted PERCLOS (see Figures 4b and 4e). In this sense, they described
the same heart rate activity and its association with driver fatigue.
Increases in heart_rate_variability could be associated with decreases in
mental workload, which often occurred in sleepy drivers with monotonous
driving (Horne Reyner, 1995). This was consistent with our finding when
heart_rate_variability was between 50 and 140 ms or when hr_std60 was between
2 beats/min and 8 beats/min (see Figures 4b and 4e). However, in other
intervals, increases in heart_rate_variability led to decreases in driver
fatigue, especially when it was smaller than 50 ms or hr_std60 was smaller
than 2 beats/min. This was not reported previously and could be potentially
explained by the different measures used for heart rate variability. For
example, Fujiwara . (2018) specifically included a feature named NN50, which
was defined as the number of adjacent inter-beat intervals whose difference
was more than 50 ms within a period of time. This is consistent with our
results, where 50 ms was the turning point in the V-shape relationship between
heart_rate_variability and the predicted PERCLOS.

###  7.3 Limitations and Future Work

First, we used XGBoost to predict driver fatigue without considering the
temporal relationships among the training data. A model, such as LSTM, can
potentially improve the performance of the model further by examining the
temporal relationships in the data in the future. However, the cost of better
performance of LSTM is that it would be difficult to explain the captured
knowledge by LSTM using SHAP. Moreover, it might still be not adequate to
detect and predict driver fatigue in real time and more research should be
devoted to predicting driver fatigue ahead of time in order for the driver to
prepare possible hazards in the takeover process in automated driving (Zhou .,
2020). Second, it should be cautious to generalize our results to other
situations because driver fatigue in this study mainly refers to passive
fatigue due to monotonous automated driving, which can be different from
fatigue caused by sleep-deprivation in traditional manual driving. In this
sense, our model is more appropriate for fatigue monitoring in automated
driving rather than in manual driving.

##  8 Conclusion

In this study, we built a fatigue prediction model using XGBoost in automated
driving. In order to understand the black-box XGBoost model, we utilized SHAP
based on coalitional game theory. First, SHAP was used to identify the most
important measures among the 11 predictor variables and using only the top
five most important predictor variables, the XGBoost was able to predict
driver fatigue indicated by PERCLOS accurately. Second, SHAP was able to
identify the main effects of the important predictor variables in the XGBoost
model globally. Third, SHAP also offered individual prediction explanations to
understand the contributions of each predictor variable locally. These
insights can potentially help automotive manufacturers design more acceptable
and trustworthy fatigue detection and prediction models in automated vehicles.

## References

  * Ayoub . (2021) AYOUB2021102Ayoub, J., Yang, XJ. Zhou, F.  2021\.  Modeling dispositional and initial learned trust in automated vehicles with predictability and explainability Modeling dispositional and initial learned trust in automated vehicles with predictability and explainability.  Transportation Research Part F: Traffic Psychology and Behaviour77102 - 116.
  * Ayoub . (2019) ayoub2019manualAyoub, J., Zhou, F., Bao, S. Yang, XJ.  2019\.  From manual driving to automated driving: A review of 10 years of autoui From manual driving to automated driving: A review of 10 years of autoui.  Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Proceedings of the 11th international conference on automotive user interfaces and interactive vehicular applications ( 70–90).
  * Balters . (2018) balters2018breathBalters, S., Murnane, EL., Landay, JA. Paredes, PE.  2018\.  Breath Booster! Exploring In-car, Fast-paced Breathing Interventions to Enhance Driver Arousal State Breath booster! exploring in-car, fast-paced breathing interventions to enhance driver arousal state.  Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare Proceedings of the 12th eai international conference on pervasive computing technologies for healthcare ( 128–137).
  * Caruana . (2015) caruana2015intelligibleCaruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M. Elhadad, N.  2015\.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.  Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining ( 1721–1730).
  * Chang Chen (2014) chang2014driverChang, TH. Chen, YR.  2014\.  Driver fatigue surveillance via eye detection Driver fatigue surveillance via eye detection.  17th International IEEE Conference on Intelligent Transportation Systems (ITSC) 17th international ieee conference on intelligent transportation systems (itsc) ( 366–371).
  * Chen Guestrin (2016) chen2016xgboostChen, T. Guestrin, C.  2016\.  Xgboost: A scalable tree boosting system Xgboost: A scalable tree boosting system.  Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining ( 785–794).
  * Clark . (2017) clark2017performanceClark, H., McLaughlin, AC., Williams, B. Feng, J.  2017\.  Performance in takeover and characteristics of non-driving related tasks during highly automated driving in younger and older drivers Performance in takeover and characteristics of non-driving related tasks during highly automated driving in younger and older drivers.  Proceedings of the Human Factors and Ergonomics Society Annual Meeting Proceedings of the human factors and ergonomics society annual meeting ( 61, 37–41).
  * Collet Musicant (2019) collet2019associatingCollet, C. Musicant, O.  2019\.  Associating vehicles automation with drivers functional state assessment systems: A challenge for road safety in the future Associating vehicles automation with drivers functional state assessment systems: A challenge for road safety in the future.  Frontiers in human neuroscience13131.
  * Dalton . (2007) dalton2007effectsDalton, BH., Behm, DG. Kibele, A.  2007\.  Effects of sound types and volumes on simulated driving, vigilance tasks and heart rate Effects of sound types and volumes on simulated driving, vigilance tasks and heart rate.  Occupational Ergonomics73153–168.
  * Dong . (2010) dong2010driverDong, Y., Hu, Z., Uchimura, K. Murayama, N.  2010\.  Driver inattention monitoring system for intelligent vehicles: A review Driver inattention monitoring system for intelligent vehicles: A review.  IEEE transactions on intelligent transportation systems122596–614.
  * Doshi-Velez Kim (2017) doshi2017towardsDoshi-Velez, F. Kim, B.  2017\.  Towards a rigorous science of interpretable machine learning Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608.
  * Du, Yang Zhou (2020) du2020psychophysiologicalDu, N., Yang, XJ. Zhou, F.  2020\.  Psychophysiological responses to takeover requests in conditionally automated driving Psychophysiological responses to takeover requests in conditionally automated driving.  Accident Analysis & Prevention148105804.
  * Du, Zhou . (20203) du2020predicting1Du, N., Zhou, F., Pulver, E., Tilbury, D., Robert, LP., Pradhan, AK. Yang, XJ.  20203\.  Predicting Takeover Performance in Conditionally Automated Driving Predicting takeover performance in conditionally automated driving.  Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended abstracts of the 2020 chi conference on human factors in computing systems ( 1–8).
  * Du, Zhou . (20201) du2020examiningDu, N., Zhou, F., Pulver, EM., Tilbury, DM., Robert, LP., Pradhan, AK. Yang, XJ.  20201\.  Examining the effects of emotional valence and arousal on takeover performance in conditionally automated driving Examining the effects of emotional valence and arousal on takeover performance in conditionally automated driving.  Transportation research part C: emerging technologies11278–87.
  * Du, Zhou . (20202) du2020predictingDu, N., Zhou, F., Pulver, EM., Tilbury, DM., Robert, LP., Pradhan, AK. Yang, XJ.  20202\.  Predicting driver takeover performance in conditionally automated driving Predicting driver takeover performance in conditionally automated driving.  Accident Analysis & Prevention148105748.
  * Dwivedi . (2014) dwivedi2014drowsyDwivedi, K., Biswaranjan, K. Sethi, A.  2014\.  Drowsy driver detection using representation learning Drowsy driver detection using representation learning.  2014 IEEE international advance computing conference (IACC) 2014 ieee international advance computing conference (iacc) ( 995–999).
  * Feldhütter . (2017) feldhutter2017durationFeldhütter, A., Gold, C., Schneider, S. Bengler, K.  2017\.  How the duration of automated driving influences take-over performance and gaze behavior How the duration of automated driving influences take-over performance and gaze behavior.  Advances in ergonomic design of systems, products and processes Advances in ergonomic design of systems, products and processes ( 309–318).  Springer.
  * Feng . (2009) feng2009boardFeng, R., Zhang, G. Cheng, B.  2009\.  An on-board system for detecting driver drowsiness based on multi-sensor data fusion using Dempster-Shafer theory An on-board system for detecting driver drowsiness based on multi-sensor data fusion using dempster-shafer theory.  2009 International Conference on Networking, Sensing and Control 2009 international conference on networking, sensing and control ( 897–902).
  * Fujiwara . (2018) fujiwara2018heartFujiwara, K., Abe, E., Kamata, K., Nakayama, C., Suzuki, Y., Yamakawa, T.others  2018\.  Heart rate variability-based driver drowsiness detection and its validation with EEG Heart rate variability-based driver drowsiness detection and its validation with eeg.  IEEE Transactions on Biomedical Engineering6661769–1778.
  * Gonçalves . (2016) gonccalves2016drowsinessGonçalves, J., Happee, R. Bengler, K.  2016\.  Drowsiness in conditional automation: proneness, diagnosis and driving performance effects Drowsiness in conditional automation: proneness, diagnosis and driving performance effects.  2016 IEEE 19th international conference on intelligent transportation systems (ITSC) 2016 ieee 19th international conference on intelligent transportation systems (itsc) ( 873–878).
  * Hadi . (2020) hadi2020influenceHadi, AM., Li, Q., Wang, W., Yuan, Q. Cheng, B.  2020\.  Influence of Passive Fatigue and Take-Over Request Lead Time on Drivers’ Take-Over Performance Influence of passive fatigue and take-over request lead time on drivers’ take-over performance.  International Conference on Applied Human Factors and Ergonomics International conference on applied human factors and ergonomics ( 253–259).
  * Horne Reyner (1995) horne1995sleepHorne, JA. Reyner, LA.  1995\.  Sleep related vehicle accidents Sleep related vehicle accidents.  Bmj3106979565–567.
  * Ji . (2004) ji2004realJi, Q., Zhu, Z. Lan, P.  2004\.  Real-time nonintrusive monitoring and prediction of driver fatigue Real-time nonintrusive monitoring and prediction of driver fatigue.  IEEE transactions on vehicular technology5341052–1068.
  * Jung . (2014) jung2014driverJung, SJ., Shin, HS. Chung, WY.  2014\.  Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel.  IET Intelligent Transport Systems8143–50.
  * Khan Mansoor (2008) khan2008realKhan, MI. Mansoor, AB.  2008\.  Real time eyes tracking and classification for driver fatigue detection Real time eyes tracking and classification for driver fatigue detection.  International Conference Image Analysis and Recognition International conference image analysis and recognition ( 729–738).
  * Kher (2019) kher2019signalKher, R.  2019\.  Signal processing techniques for removing noise from ECG signals Signal processing techniques for removing noise from ecg signals.  J. Biomed. Eng. Res31–9.
  * Koesdwiady . (2016) koesdwiady2016recentKoesdwiady, A., Soua, R., Karray, F. Kamel, MS.  2016\.  Recent trends in driver safety monitoring systems: State of the art and challenges Recent trends in driver safety monitoring systems: State of the art and challenges.  IEEE transactions on vehicular technology6664550–4563.
  * Körber . (2015) korber2015vigilanceKörber, M., Cingel, A., Zimmermann, M. Bengler, K.  2015\.  Vigilance decrement and passive fatigue caused by monotony in automated driving Vigilance decrement and passive fatigue caused by monotony in automated driving.  Procedia Manufacturing32403–2409.
  * Krajewski . (2009) krajewski2009steeringKrajewski, J., Sommer, D., Trutschel, U., Edwards, D. Golz, M.  2009\.  Steering wheel behavior based estimation of fatigue Steering wheel behavior based estimation of fatigue.  Proceedings of the… international driving symposium on human factors in driver assessment, training and vehicle design Proceedings of the… international driving symposium on human factors in driver assessment, training and vehicle design ( 5, 118–124).
  * Kumar . (2017) kumar2017predictiveKumar, S., Kalia, A. Sharma, A.  2017\.  Predictive analysis of alertness related features for driver drowsiness detection Predictive analysis of alertness related features for driver drowsiness detection.  International Conference on Intelligent Systems Design and Applications International conference on intelligent systems design and applications ( 368–377).
  * Lee Chung (2012) lee2012driverLee, BG. Chung, WY.  2012\.  Driver alertness monitoring using fusion of facial features and bio-signals Driver alertness monitoring using fusion of facial features and bio-signals.  IEEE Sensors Journal1272416–2422.
  * Li . (2017) li2017automaticLi, Z., Chen, L., Peng, J. Wu, Y.  2017\.  Automatic detection of driver fatigue using driving operation information for transportation safety Automatic detection of driver fatigue using driving operation information for transportation safety.  Sensors1761212.
  * Lundberg . (2020) lundberg2020localLundberg, SM., Erion, G., Chen, H., DeGrave, A., Prutkin, JM., Nair, B.Lee, SI.  2020\.  From local explanations to global understanding with explainable AI for trees From local explanations to global understanding with explainable ai for trees.  Nature machine intelligence212522–5839.
  * Lundberg, Erion Lee (2018) lundberg2018consistentLundberg, SM., Erion, GG. Lee, SI.  2018\.  Consistent individualized feature attribution for tree ensembles Consistent individualized feature attribution for tree ensembles.  arXiv preprint arXiv:1802.03888.
  * Lundberg, Nair . (2018) lundberg2018explainableLundberg, SM., Nair, B., Vavilala, MS., Horibe, M., Eisses, MJ., Adams, T.others  2018\.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  Nature biomedical engineering210749–760.
  * Mannering . (2020) mannering2020bigMannering, F., Bhat, CR., Shankar, V. Abdel-Aty, M.  2020\.  Big data, traditional data and the tradeoffs between prediction and causality in highway-safety analysis Big data, traditional data and the tradeoffs between prediction and causality in highway-safety analysis.  Analytic methods in accident research25100113.
  * McDonald . (2014) mcdonald2014steeringMcDonald, AD., Lee, JD., Schwarz, C. Brown, TL.  2014\.  Steering in a random forest: Ensemble learning for detecting drowsiness-related lane departures Steering in a random forest: Ensemble learning for detecting drowsiness-related lane departures.  Human factors565986–998.
  * Munkhdalai . (2019) munkhdalai2019advancedMunkhdalai, L., Wang, L., Park, HW. Ryu, KH.  2019\.  Advanced Neural Network Approach, Its Explanation with LIME for Credit Scoring Application Advanced neural network approach, its explanation with lime for credit scoring application.  Asian Conference on Intelligent Information and Database Systems Asian conference on intelligent information and database systems ( 407–419).
  * Murdoch . (2019) murdoch2019definitionsMurdoch, WJ., Singh, C., Kumbier, K., Abbasi-Asl, R. Yu, B.  2019\.  Definitions, methods, and applications in interpretable machine learning Definitions, methods, and applications in interpretable machine learning.  Proceedings of the National Academy of Sciences1164422071–22080.
  * Nagabushanam . (2019) nagabushanam2019eegNagabushanam, P., George, ST. Radha, S.  2019\.  EEG signal classification using LSTM and improved neural network algorithms Eeg signal classification using lstm and improved neural network algorithms.  Soft Computing1–23.
  * Ribeiro . (2016) ribeiro2016shouldRibeiro, MT., Singh, S. Guestrin, C.  2016\.  ” Why should i trust you?” Explaining the predictions of any classifier ” why should i trust you?” explaining the predictions of any classifier.  Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining ( 1135–1144).
  * SAE (2018) sae2018taxonomySAE.  2018\.  Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles.  SAE International in United States, J3016–201806.
  * Sayed Eskandarian (2001) sayed2001unobtrusiveSayed, R. Eskandarian, A.  2001\.  Unobtrusive drowsiness detection by neural network learning of driver steering Unobtrusive drowsiness detection by neural network learning of driver steering.  Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering2159969–975.
  * Shapley . (1953) shapley1953contributionsShapley, LS., Kuhn, H. Tucker, A.  1953\.  Contributions to the Theory of Games Contributions to the theory of games.  Annals of mathematics studies282307–317.
  * Sikander Anwar (2018) sikander2018driverSikander, G. Anwar, S.  2018\.  Driver fatigue detection systems: A review Driver fatigue detection systems: A review.  IEEE Transactions on Intelligent Transportation Systems2062339–2352.
  * Sun . (2011) sun2011vehicleSun, Y., Yu, X., Berilla, J., Liu, Z. Wu, G.  2011\.  An in-vehicle physiological signal monitoring system for driver fatigue detection An in-vehicle physiological signal monitoring system for driver fatigue detection.  3rd International Conference on Road Safety and SimulationPurdue UniversityTransportation Research Board. 3rd international conference on road safety and simulationpurdue universitytransportation research board.
  * Ünal . (2013) unal2013drivingÜnal, AB., de Waard, D., Epstude, K. Steg, L.  2013\.  Driving with music: Effects on arousal and performance Driving with music: Effects on arousal and performance.  Transportation research part F: traffic psychology and behaviour2152–65.
  * Vogelpohl . (2019) vogelpohl2019asleepVogelpohl, T., Kühn, M., Hummel, T. Vollrath, M.  2019\.  Asleep at the automated wheel—Sleepiness and fatigue during highly automated driving Asleep at the automated wheel—sleepiness and fatigue during highly automated driving.  Accident Analysis & Prevention12670–84.
  * Watta . (2007) watta2007nonparametricWatta, P., Lakshmanan, S. Hou, Y.  2007\.  Nonparametric approaches for estimating driver pose Nonparametric approaches for estimating driver pose.  IEEE transactions on vehicular technology5642028–2041.
  * Zhang . (2013) zhang2013automatedZhang, C., Wang, H. Fu, R.  2013\.  Automated detection of driver fatigue based on entropy and complexity measures Automated detection of driver fatigue based on entropy and complexity measures.  IEEE Transactions on Intelligent Transportation Systems151168–177.
  * Zhou . (2020) zhou2020driverZhou, F., Alsaid, A., Blommer, M., Curry, R., Swaminathan, R., Kochhar, D.Lei, B.  2020\.  Driver Fatigue Transition Prediction in Highly Automated Driving Using Physiological Features Driver fatigue transition prediction in highly automated driving using physiological features.  Expert Systems with Applications113204.
  * Zhou . (2019) zhou2019takeoverZhou, F., Yang, XJ. Zhang, X.  2019\.  Takeover Transition in Autonomous Vehicles: A You Tube Study Takeover transition in autonomous vehicles: A You Tube study.  International Journal of Human–Computer Interaction1–12.