# Towards safe, explainable, and regulated autonomous driving

Shahin Atakishiyev,1 Mohammad Salameh, 2Hengshuai Yao,1, 2Randy Goebel1

###### Abstract

There has been growing interest in the development and deployment of
autonomous vehicles on modern road networks over the last few years,
encouraged by the empirical successes of powerful artificial intelligence
approaches (AI), especially in the applications of deep and reinforcement
learning. However, there have been several road accidents with “autonomous”
cars that prevent this technology from being publicly acceptable at a wider
level. As AI is the main driving force behind the intelligent navigation
systems of such vehicles, both the stakeholders and transportation
jurisdictions require their AI-driven software architecture to be safe,
explainable, and regulatory compliant. We present a framework that integrates
autonomous control, explainable AI architecture, and regulatory compliance to
address this issue and further provide several conceptual models from this
perspective, to help guide future research directions.

## Introduction

Autonomous driving as a rapidly growing field has attracted increasing
attention over the last decade. According to a recent report by Intel, the
deployment of autonomous cars will reduce approximately 250 million hours of
on-road travel and save about 585,000 lives between the years 2035 and 2045
just in the USA (Lanctot et al. 2017). While these advantages encourage the
use of autonomous vehicles, there is also a major societal concern about the
safety of this technology. This concern mainly emits from the recent accidents
(Stanton et al. 2019; Yurtsever et al. 2020; NTSB 2020) with the involvement
of autonomous or semi-autonomous cars, mainly due to improper “autonomous”
functions. This issue is a major drawback, impeding self-operating vehicles to
be acceptable by road users and society at a broader level. As artificial
intelligence techniques power autonomous vehicles’ real-time decisions and
actions, a malfunction of the vehicle’s intelligent system is considered the
main focus of analysis in such mishaps. Hence, both road users and regulators
require that the AI systems of autonomous vehicles should be “explainable,”
meaning that real-time decisions of such cars, especially in critical
scenarios, should be intelligible in addition to being robust and safe. In
this case, explainability needs not only to provide transparency on an
individual failure, but also more broadly in the process of public
transportation regulatory compliance (Slagter and Voster 2017; Dentons 2021).
Urging the right of an explanation, several regulators have established safety
and compliance standards for current and next-generation intelligent
transportation systems. We are developing a general framework that can make
autonomous vehicle manufacturers and involved users more safety and regulatory
compliant and help autonomous vehicles be more publicly trustable and
acceptable. In addition, we provide a prioritization of relevant AI approaches
that can help achieve such a framework.

## A brief overview of autonomous driving

Autonomous cars (also known as self-driving cars or driverless cars) are
vehicles capable of perceiving their environment and making real-time driving
decisions with the help of intelligent driving systems, without the
involvement of humans. To capture the operational environment, autonomous
vehicles leverage a variety of passive (e.g., collecting information from the
surrounding without emitting a wave, such as visible spectrum cameras) and
active (e.g., sending a wave and receiving a reflected signal from the
objects, such as the LIDAR) sensors. Sensor devices detect changes in the
environment and enable the driving system of the car to make real-time
decisions on the captured information (Campbell et al. 2018; Yeong et al.
2021). Current autonomous vehicles deployed on real roads have different
levels of automation. The SAE International (previously known as the Society
of Automotive Engineers) have defined six different levels of autonomous
driving based on the in-vehicle technologies and level of intelligent system,
namely Level 0 - No automation, Level 1 - Driving assistance, Level 2 -
Partial automation, Level 3 - Conditional automation, Level 4 - High
automation, and Level 5 - Full automation (Shuttleworth 2019). The increase of
automation reinforces reliance on an intelligent driving system rather than a
human driver, particularly in Level 3 and higher levels. However, such
vehicles have recently caused several road accidents, cited above, that led to
severe injuries or even loss of humans lives. Why did the accident happen?
What malfunction of the driving system led to the crash or hitting? These
questions naturally raise serious ethical and safety issues and are the
motivation for explanatory AI systems. So, both societal requirements and
regulatory compliance require the AI system of autonomous vehicles to be safe,
explainable and regulated. The General Data Protection Regulation (GDPR) of
the European Union (EU) established guidelines to promote a “right to
explanation” for users, enacted in 2016 and carried into effect in May 2018
(GDPR 2016). Another example, The National Highway Traffic Safety
Administration (NHTSA) of the US Department of Transportation have issued a
federal guideline on automated vehicle policy to attain enhanced road safety
(NHTSA 2016). Thus, the current and future generation of autonomous vehicles
must comply with such regulations, and their intelligent driving system should
be explainable, transparent, and acceptably safe. In this regard, we propose a
high-level but straightforward framework that considers these requirements and
shows what computational and legislative components are necessary for
transparent autonomous driving.

## An XAI framework for autonomous driving

We intend to articulate a framework in which methods for developing end-to-end
autonomous systems, XAI, and regulatory compliance are connected. We hope to
place existing work, including surveys, into a context that helps set
priorities on research in our overall framework. We intend, for example, that
our framework helps determine priorities for the development of XAI systems
coupled with machine learning methods for the development of autonomous
control methods that can inform general processes of regulatory compliance.
Our immediate task here is to describe a high-level framework in which the
concepts of end-to-end autonomous systems, XAI, and regulatory compliance all
have a role. We propose simple notations to bring some precision to the
description of this framework by providing an economic vocabulary.

### End-to-end autonomous systems component

We need at least a high-level but precise description of what we mean by end-
to-end autonomous systems. To start, we need to be able to refer to the set of
actions that any autonomous vehicle is capable of executing. We consider the
set of possible autonomous actions of automated vehicles as

$A=\\{a_{1},a_{2},...a_{n}\\}$

Notice we consider the list of executable actions a finite repertoire of
actions that can be selected by a predictive model, whether that model is
constructed by hand or by a machine learning algorithm. Example actions are
things like “turn right,” or “accelerate.” For now, we refrain from
considering complex compound actions like “decelerate and turn right” but
acknowledge such actions will be possible in any given set $A$ of actions,
depending on the vehicle.

We will use the notation $C$ to denote an autonomous system controller and $E$
to denote the set of all possible autonomous system operating environments.
The overall function of an end-to-end controller is to map an environment $E$
to an action $A$; we have an informal description of the role of the
autonomous system controller as

$C:E\mapsto A$

This mapping is intended to denote how any controller’s responsibility is to
map a perceived or sensed environment to an autonomous system function.
Autonomous driving researchers and practitioners have already attempted to use
the concept of end-to-end learning in recent studies. (Bojarski et al. 2016)
have used convolutional neural networks (CNNs) to map camera inputs to
steering and obtained impressive results with minimal training data. Their CNN
architecture learned entire driving operations for driving in situations with
and without lane markings in sunny, rainy, and cloudy weather conditions on a
highway. Another empirically successful end-to-end learning example is
(Toromanoff, Wirbel, and Moutarde 2020)’s work. The authors use reinforcement
learning to drive an autonomous car safely while considering road users (i.e.,
pedestrians) and respecting driving rules (i.e., lane keeping, collision
prevention, and traffic light detection). Their ablation study of the proposed
method proved its effectiveness by winning the “Camera Only” track of the
CARLA competition challenge (CARLA’s blog 2019). So regardless of choices for
a real-time computational decision-making method (i.e., deep learning (DL),
RL), the end-to-end learning approach corroborates its effectiveness.

Now, before we provide a descriptive basis for the XAI component, we can
already provide a descriptive definition of an end-to-end autonomous
controller: A control system $C$ is an end-to-end control system or eeC, if
$C$ is a total function that maps every instance $e\in E$ to an action $a\in
A$ where $A$ in an autonomous system’s repertoire of actions.

Even with such a high-level description, we can note that the most important
attribute of a eeC is that it provides a complete mapping from any sensed
environment to an action selection. We want this simple definition because we
are interested only in autonomous controllers that provide a complete control
system for any particular autonomous system rather than subcomponents (e.g.,
we are not directly interested in sub-controllers for sub-actions).

### Safety-regulatory compliance component

Figure 1: A simple conceptual framework of autonomous driving that combines
end-to-end autonomous control, explainability, and regulatory compliance. The
end-to-end control system (eeC) maps the perceived environment to relevant
actions. The safety-regulatory component (srC) confirms the combination of
actions with the eeC. These constituents form up explainable AI for the
intelligent driving system that backs up safe, end-to-end, transparent,
regulated, and explainable autonomous driving.

We first provide a high-level description of the safety regulatory component
or srC, and then how the XAI component connects $eeC$ and $srC$. The role of
our framework’s safety-regulatory component $srC$ is to represent the function
of a regulatory agency, one of whose main roles is to verify the safety of any
combination of $eeC$ and autonomous vehicle with $A$ as a repertoire of
actions.

We first note that safety compliance is a function that confirms the safety
and reliability of an $eeC$ system. This could be as pragmatic as some
inspection of individual vehicle safety (e.g., some jurisdiction processes
that certify basic safety functions of individual vehicles for re-licensing),
but more and more is associated with sophisticated compliance testing of $eeC$
components from manufacturers, to establish their public safety approval
(e.g., (Transport Canada 2021)).

The $srC$ components will vary widely from jurisdiction to jurisdiction (e.g.,
(Dentons 2021)). Also, AI and machine learning techniques have largely
emphasized on building the predictive models for $eeC$ systems. Hence, this
leads the confirmation of compliance to focus on confirming the safety of the
mapping from any $eeC$ to any vehicle’s compendium of actions $A$.

All we know about methods for software testing applies to this $srC$ task, and
we expect the evolution of $srC$ processes will increasingly rely on the
automation of compliance testing against all $eeC$ systems. The complexity of
$srC$ systems lies within the scope of testing methods that are certified to
confirm a threshold of safety. For example, from a compliance repertoire of
$N$ safety tests, a regulator may require a 90% performance of any particular
$eeC$. Ultimately, $srC$ will rely on data sampling and structural analysis of
any $eeC$ system to be used by regulatory software systems to achieve target
safety thresholds. That this component of the overall framework is so closely
related to the overall challenge of proving software correctness is not
unexpected. Ideas from software correctness must be coupled with $eeC$ and
$srC$ practices.

### XAI component

We finally get to the last component of our high-level framework, which
describes the role of a general explainable AI (XAI) component. At the highest
level, the XAI component of our framework will be responsible for providing
transparency on how an $eeC$ makes a selection from a repertoire of actions
$A$. The difficulty of the challenge is directly related to those methods used
to construct the control mechanism $C$ of the $eeC$. That is coupled with the
demands created for the “explainee,” which may range from natural language
interaction with a human driver (e.g., why did the system suggest having the
driver take over steering?), to the potential very complex setting of
compliance safety parameters for regulatory compliance testing software. The
graphical illustration of the aforementioned framework is depicted in Figure
1.

## XAI: what to expect?

In general, autonomous vehicles make decisions by perceiving their environment
(e.g., using computer vision). They map this perception to relevant actions
using various DL approaches, often in terms of CNN methods and their augmented
variations. On the other hand, RL-based approaches have proven computationally
robust and adequate to map real-time sensing of states to relevant actions.
Based on this intuition, we can infer that explainable autonomous driving can
be achieved by explaining the decisions of its learned software architecture.
To conform to the framework presented, we provide directions for 1)
explainable vision, 2) explainable RL, 3) representation of knowledge acquired
from the operational environment, and 4) question-driven software hierarchy
for comprehensible action selection.

### Use of explainable deep learning

Explainable vision-directed actions for autonomous vehicles are based on how
high-level features are used to detect and identify objects. As CNNs, as deep
learning architectures, are “black-box” models, there is a need to develop
explainable CNN (XCNN) architectures to comply with the requirements of the
proposed framework. There have recently been works on explaining predictions
of CNNs, such as DeepLift (Shrikumar, Greenside, and Kundaje 2017), Class
Activation Map (CAM) (Zhou et al. 2016) and its extended versions such as
Grad-CAM (Selvaraju et al. 2017), Grad-CAM++ (Chattopadhay et al. 2018), and
Guided Grad-CAM (Tang et al. 2019). So, it is worthwhile to further test XCNN
architectures on self-driving vehicles. XCCNs providing intelligible and
trustable explanations coupled with accurate environmental perception may be a
major step towards attaining interpretable vision models in autonomous
machines.

### Focusing on model-based reinforcement learning

Throughout their motion trajectory, autonomous vehicles make sequential
decisions within a setting formally characterized as Markov Decision Processes
(MDP). An agent’s interaction with the environment as an MDP can be
implemented as either model-free or model-based RL. In a model-free setting
(such as Q-learning), the algorithm does not have access to the dynamics of
the environment (i.e., transition or reward function). It estimates the value
function directly from a sensed experience (Sutton and Barto 2018). On the
other hand, the model-based RL approach uses an estimated transition function
to compute the optimal policy (Yao and Szepesvári 2012; Yao et al. 2014;
Sutton and Barto 2018; Moerland, Broekens, and Jonker 2020). A model-based RL
method usually has a planning component, which learns and uses a model to
approximate value functions. The idea of planning in RL is essential for
decision making, e.g., the Dyna architectures (Sutton 1991; Sutton et al.
2008; Yao et al. 2009). In a model-free RL agent, it directly learns a policy
with the environment through interaction. The Dyna and linear-Dyna style
architectures simultaneously learn a world model whilst learning the optimal
policy through interactions with the world. Planning is naturally explainable.
In planning, a real state from the world or an imaginary state is given, the
planning process essentially generates a predicted future trajectory starting
from it. At each of the planning steps, an optimal action according to the
model projection is generated, giving us a predicted state and a predicted
reward. The predicted states and rewards can be analyzed, summed and
visualized for the planned trajectory, giving explanation to why the agent
favours the choice of a particular action at a particular time step. That
being said, the potential of using Dyna architectures and model-based RL is
huge for XAI and autonomous driving.

### Leveraging predictive knowledge

It is important to specify how an agent could represent and use the knowledge
collected through the interaction with the environment. This knowledge can be
considered a compendium of predictions that the agent makes in anticipation of
a selection from possible actions. This kind of knowledge is usually called
“predictive knowledge” or “prediction as knowledge” (PAK), and has received
significant attention in RL studies (Drescher 1991; Sutton and Tanner 2004;
Sutton et al. 2011; White 2015). An agent regularly expects a response from
the environment by making many predictions about the dynamics of the
environment in accordance with the autonomous system’s behavior. Therefore,
PAK is one of the essential notions of reinforcement learning and can be
considered as prior knowledge about the possible worlds in which the
corresponding actions might be taken. An important aspect to consider a
prediction as knowledge is that a prediction should comply with the
requirements of knowledge, carrying fundamental elements of epistemology -
justification and truth, in itself (Kearney and Pilarski 2019). General Value
Functions (GVFs) are an initial proposal for the robust representation of
predictive knowledge; GVFs are computationally scalable (White 2015) and have
proven their value in perception problems (Graves, Rezaee, and Scheideman
2019) and learning policies in real-world for autonomous driving (Graves et
al. 2020). So it is important to investigate the PAK approach further and its
representation with GVFs to fully exploit environmental knowledge for safe
decision making of automated machines.

### Temporal questions and question hierarchies

If there is one single common design principle that is shared by the
autonomous driving industry, that will be arranging the software in a
hierarchical structure. To have a self-explainable decision procedure for
autonomous driving, it is important that this hierarchy becomes question
driven. In particular, the questions frequently coming to us in driving are
those like “Am I going to see the traffic lights turning yellow shortly?”,
“Will the front car brake?”, “Will the car in the front right of me cut to my
lane?”, etc. All these questions seem to come to us naturally when we drive.
This concept, however, receives very little attention from current research on
AI methods for autonomous driving. Understanding why human beings have this
ability is important to advance the safety of autonomous machines. These
questions don’t emerge randomly but subconsciously and appear as dynamically
relevant based on convenience about what we observe; considering answers for
these questions prepare us for a safe drive. An interesting approach to
generate temporal questions based on the ongoing actions would be using the
concept of options (Sutton, Precup, and Singh 1999). Options are the
generalized concept of actions that has a policy for taking action with
terminal conditions. The option-critic architecture, recently proposed by
(Bacon, Harb, and Precup 2017), and motivated by the concept of options, is
capable of learning internal policies and terminal conditions of options and
has proven to be experimentally successful in end-to-end learning of options
in the Arcade Learning Environment (ALE). The option-critic architecture is
indeed useful for temporal questions in autonomous driving: because driving-
related questions are often temporal, new questions can be generated for the
subsequent actions after just several seconds. Hence, it is important to study
the formulation of questioning, with research efforts on the generation,
evaluation, selection, topology, and organization of questions for transparent
autonomous driving (Zablocki et al. 2021). The time sensitivity of driving
decisions varies dynamically in real time, creating different levels of risks
for the vehicle. In general, low-risk actions and options are preferred.
However, in safety stringent situations, we need to explore efficiently
(possibly in real time) in order for the vehicle to curb the danger. This
requires a representation of the risks in decision making with a principled
way of evaluating the risks to decide which risk level for the vehicle to
undertake. Experiments by Zhang et al. (2018) showed that considering no risks
but only acting according to the mean of return as in traditional RL is not
always the best decision and can fail to discover the optimal policy. In
contrast, acting according to different levels of risks can find the optimal
policy in environments with a diverse range of different transition and reward
dynamics. We can thus infer and conclude that decision making for autonomous
vehicles is time-sensitive, often in a part of seconds, and a well-composed
question hierarchy may not only help to select the present action but also
subsequent actions that help safe driving. With such a benefit, the
hierarchical structure can also provide temporal, informative and reliable
explanations in critical traffic situations.

## Conclusions

In our ongoing study, we propose a simple end-to-end autonomous driving
framework incorporating the essential constituents of transparent and
regulated autonomous driving. We also advocate for prioritized choices on
selecting AI approaches to attain a proper vision-action mapping for
autonomous vehicles. We hope that further empirical studies based on the
proposed framework can help autonomous driving be safer, explainable, and
legally governed, making this technology publicly acceptable.

## Acknowledgments

We acknowledge support from the Alberta Machine Intelligence Institute (Amii),
from the Computing Science Department of the University of Alberta, and the
Natural Sciences and Engineering Research Council of Canada (NSERC). Shahin
Atakishiyev also acknowledges support from the Ministry of Education of the
Republic of Azerbaijan.

## References

  * Bacon, Harb, and Precup (2017) Bacon, P.-L.; Harb, J.; and Precup, D. 2017.  The option-critic architecture.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Bojarski et al. (2016) Bojarski, M.; Del Testa, D.; Dworakowski, D.; Firner, B.; Flepp, B.; Goyal, P.; Jackel, L. D.; Monfort, M.; Muller, U.; Zhang, J.; et al. 2016.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_.
  * Campbell et al. (2018) Campbell, S.; O’Mahony, N.; Krpalcova, L.; Riordan, D.; Walsh, J.; Murphy, A.; and Ryan, C. 2018.  Sensor technology in autonomous vehicles: A review.  In _2018 29th Irish Signals and Systems Conference (ISSC)_ , 1–4. IEEE.
  * CARLA’s blog (2019) CARLA’s blog. 2019.  The CARLA Autonomous Driving Challenge.  (Accessed on November 1, 2021).
  * Chattopadhay et al. (2018) Chattopadhay, A.; Sarkar, A.; Howlader, P.; and Balasubramanian, V. N. 2018.  Grad-Cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE winter conference on applications of computer vision (WACV)_ , 839–847. IEEE.
  * Dentons (2021) Dentons. 2021.  Global Guide to Autonomous Vehicles 2021.
  * Drescher (1991) Drescher, G. L. 1991.  _Made-up minds: a constructivist approach to artificial intelligence_.  MIT press.
  * GDPR (2016) GDPR. 2016.  Regulation EU 2016/679 of the European Parliament and of the Council of 27 April 2016.  _Official Journal of the European Union_.
  * Graves et al. (2020) Graves, D.; Nguyen, N. M.; Hassanzadeh, K.; and Jin, J. 2020.  Learning predictive representations in autonomous driving to improve deep reinforcement learning.  _arXiv preprint arXiv:2006.15110_.
  * Graves, Rezaee, and Scheideman (2019) Graves, D.; Rezaee, K.; and Scheideman, S. 2019.  Perception as prediction using general value functions in autonomous driving applications.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 1202–1209. IEEE.
  * Kearney and Pilarski (2019) Kearney, A.; and Pilarski, P. M. 2019.  When is a Prediction Knowledge?  _arXiv preprint arXiv:1904.09024_.
  * Lanctot et al. (2017) Lanctot, R.; et al. 2017.  Accelerating the future: The economic impact of the emerging passenger economy.  _Strategy analytics_ , 5.
  * Moerland, Broekens, and Jonker (2020) Moerland, T. M.; Broekens, J.; and Jonker, C. M. 2020.  Model-based reinforcement learning: A survey.  _arXiv preprint arXiv:2006.16712_.
  * NHTSA (2016) NHTSA. 2016.  _Federal automated vehicles policy: Accelerating the next revolution in roadway safety_.  US Department of Transportation.
  * NTSB (2020) NTSB. 2020.  Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator Mountain View, California.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-Cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Shrikumar, Greenside, and Kundaje (2017) Shrikumar, A.; Greenside, P.; and Kundaje, A. 2017.  Learning important features through propagating activation differences.  In _International Conference on Machine Learning_ , 3145–3153. PMLR.
  * Shuttleworth (2019) Shuttleworth, J. 2019.  SAE Standard News: J3016 Automated-Driving Graphic Update.  https://www.sae.org/news/2019/01/sae-updates-j3016-automated-driving-graphic.  Accessed online on August 16, 2021.
  * Slagter and Voster (2017) Slagter, R.; and Voster, R. 2017.  Autonomous Compliance Standing on the shoulders of RegTech!  _Compact_.
  * Stanton et al. (2019) Stanton, N. A.; Salmon, P. M.; Walker, G. H.; and Stanton, M. 2019.  Models and methods for collision analysis: a comparison study based on the Uber collision with a pedestrian.  _Safety science_ , 120: 117–128.
  * Sutton (1991) Sutton, R. S. 1991.  Dyna, an integrated architecture for learning, planning, and reacting.  _ACM Sigart Bulletin_ , 2(4): 160–163.
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement learning: An introduction_.  MIT press, Second edition.
  * Sutton et al. (2011) Sutton, R. S.; Modayil, J.; Delp, M.; Degris, T.; Pilarski, P. M.; White, A.; and Precup, D. 2011.  Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction.  In _The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2_ , 761–768.
  * Sutton, Precup, and Singh (1999) Sutton, R. S.; Precup, D.; and Singh, S. 1999.  Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2): 181–211.
  * Sutton et al. (2008) Sutton, R. S.; Szepesvári, C.; Geramifard, A.; and Bowling, M. H. 2008.  Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping.  In McAllester, D. A.; and Myllymäki, P., eds., _UAI 2008, Proceedings of the 24th Conference in Uncertainty in Artificial Intelligence, Helsinki, Finland, July 9-12, 2008_ , 528–536. AUAI Press.
  * Sutton and Tanner (2004) Sutton, R. S.; and Tanner, B. 2004.  Temporal-difference networks.  _Advances in neural information processing systems_ , 17.
  * Tang et al. (2019) Tang, Z.; Chuang, K. V.; DeCarli, C.; Jin, L.-W.; Beckett, L.; Keiser, M. J.; and Dugger, B. N. 2019.  Interpretable classification of Alzheimer’s disease pathologies with a convolutional neural network pipeline.  _Nature communications_ , 10(1): 1–14.
  * Toromanoff, Wirbel, and Moutarde (2020) Toromanoff, M.; Wirbel, E.; and Moutarde, F. 2020.  End-to-end model-free reinforcement learning for urban driving using implicit affordances.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7153–7162.
  * Transport Canada (2021) Transport Canada. 2021.  Guidelines for testing automated driving systems in Canada.
  * White (2015) White, A. 2015.  Developing a predictive approach to knowledge.  _PhD thesis, University of Alberta_.
  * Yao et al. (2009) Yao, H.; Bhatnagar, S.; Diao, D.; Sutton, R. S.; and Szepesvári, C. 2009.  Multi-Step Dyna Planning for Policy Evaluation and Control.  In Bengio, Y.; Schuurmans, D.; Lafferty, J.; Williams, C.; and Culotta, A., eds., _Advances in Neural Information Processing Systems_ , volume 22. Curran Associates, Inc.
  * Yao and Szepesvári (2012) Yao, H.; and Szepesvári, C. 2012.  Approximate policy iteration with linear action models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 26.
  * Yao et al. (2014) Yao, H.; Szepesvári, C.; Pires, B. A.; and Zhang, X. 2014.  Pseudo-MDPs and factored linear action models.  In _2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)_ , 1–9. IEEE.
  * Yeong et al. (2021) Yeong, D. J.; Velasco-Hernandez, G.; Barry, J.; Walsh, J.; et al. 2021.  Sensor and sensor fusion technology in autonomous vehicles: A review.  _Sensors_ , 21(6): 2140.
  * Yurtsever et al. (2020) Yurtsever, E.; Lambert, J.; Carballo, A.; and Takeda, K. 2020.  A survey of autonomous driving: Common practices and emerging technologies.  _IEEE access_ , 8: 58443–58469.
  * Zablocki et al. (2021) Zablocki, É.; Ben-Younes, H.; Pérez, P.; and Cord, M. 2021.  Explainability of vision-based autonomous driving systems: Review and challenges.  _arXiv preprint arXiv:2101.05307_.
  * Zhang et al. (2018) Zhang, S.; Mavrin, B.; Kong, L.; Liu, B.; and Yao, H. 2018.  QUOTA: The Quantile Option Architecture for Reinforcement Learning.  arXiv:1811.02073.
  * Zhou et al. (2016) Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2921–2929.