# Robustness and Adaptability of Reinforcement Learning based Cooperative
Autonomous Driving in Mixed-autonomy Traffic

Rodolfo Valiente1, Behrad Toghi1, Ramtin Pedarsani2, Yaser P. Fallah1 *This
material is based upon work partially supported by the National Science
Foundation under Grant No. CNS-1932037.1 Rodolfo Valiente, Behrad Toghi, and
Yaser P. Fallah are with the Department of Electrical and Computer
Engineering, University of Central Florida, Orlando, FL, USA.
<EMAIL_ADDRESS>Ramtin Pedarsani is with the Department of
Electrical and Computer Engineering, University of California, Santa Barbara,
CA, USA.

###### Abstract

Building autonomous vehicles (AVs) is a complex problem, but enabling them to
operate in the real world where they will be surrounded by human-driven
vehicles (HVs) is extremely challenging. Prior works have shown the
possibilities of creating inter-agent cooperation between a group of AVs that
follow a social utility. Such altruistic AVs can form alliances and affect the
behavior of HVs to achieve socially desirable outcomes. We identify two major
challenges in the co-existence of AVs and HVs. First, social preferences and
individual traits of a given human driver, e.g., selflessness and
aggressiveness are unknown to an AV, and it is almost impossible to infer them
in real-time during a short AV-HV interaction. Second, contrary to AVs that
are expected to follow a policy, HVs do not necessarily follow a stationary
policy and therefore are extremely hard to predict. To alleviate the above-
mentioned challenges, we formulate the mixed-autonomy problem as a multi-agent
reinforcement learning (MARL) problem and propose a decentralized framework
and reward function for training cooperative AVs. Our approach enables AVs to
learn the decision-making of HVs implicitly from experience, optimizes for a
social utility while prioritizing safety and allowing adaptability;
robustifying altruistic AVs to different human behaviors and constraining them
to a safe action space. Finally, we investigate the robustness, safety and
sensitivity of AVs to various HVs behavioral traits and present the settings
in which the AVs can learn cooperative policies that are adaptable to
different situations.

###### Index Terms:

Behavior Planning, Cooperative Driving, Mixed-autonomy, Reinforcement
Learning, Robustness

##  I INTRODUCTION

The development of autonomous vehicles (AVs) is on the verge of passing beyond
the laboratory and simulation tests and is shifting towards addressing the
challenges that limit their practicality in today’s society. While there is
still need for further technological improvements to enable safe and smooth
operation of a single AV, a great deal of research attention is being focused
on the emerging challenge of operating multiple AVs and the co-existence of
AVs and human-driven vehicles (HVs) [1, 2]. A realistic outlook for the
adoption of autonomous vehicles on the roads is a mixed-traffic scenario in
which human drivers with different driving styles and social preferences share
the road with AVs that are perhaps built by different manufacturers and hence
follow different policies [3, 4]. In this work, we seek a solution that can
ensure the safety and robustness of AVs in the presence of human drivers with
heterogeneous behavioral traits.

Connected & autonomous vehicles (CAVs) via vehicle-to-vehicle (V2V)
communication allow vehicles to directly communicate with their neighbors,
creating an extended perception that enables explicit coordination among
vehicles to overcome the limitations of an isolated agent [5, 6, 7, 8, 9, 10,
11]. While planning in a fully AV scenario is relatively easy to achieve,
coordination in the presence of HVs is a significantly more challenging task,
as the AVs not only need to react to road objects but also need to consider
the behaviors of HVs [12, 3, 4]. We start by identifying the major challenges
in the domain of behavior planning and prediction for AVs in mixed-autonomy
traffic. As a preliminary, it is important to distinguish between the
individual traits of a human driver, e.g., aggressiveness, conservativeness,
risk-tolerance, and their social preferences, e.g., egoism and altruism.
Despite the correlation between the two categories, they arise from different
natures and also lead to different behaviors in mixed traffic. As an example,
an aggressive driver is not necessarily egoistic and selfish, but their
aggression might limit their capability to cooperate with other drivers and
take part in a socially desirable co-existence with AVs [13, 14, 15]. First,
one major challenge is that human drivers are heterogeneous in their
individual traits and social preferences, which makes the autonomous vehicle
behavior planning extremely difficult, as it is challenging for the AV to
predict the type of behavior it is going to face when dealing with a human
driver. Additionally, relying on real-time inference of HVs’ behaviors is not
always viable as the interaction time between vehicles can be short-lived,
e.g., two vehicles that meet in an intersection. Second, the driving task
involves complex interactions of agents in a partially observable and non-
stationary environment, as HVs do not follow a stationary policy and change
their policies in real-time according to the other vehicles’ behaviors.

Figure 1: An overview of our methodology for ensuring the safety and
robustness of cooperative autonomous vehicles in interaction with human-driven
vehicles.

In a pursuit to alleviate the challenges of this co-existence and enable
social navigation for AVs, existing works either rely on models of human
behavior derived from pre-recorded driving datasets [16, 17] or defining
social utilities that can enforce a cooperative behavior among AVs and HVs [2,
18]. Other works focus on rule-based methods that use heuristics and hand-
coded rules to guide the AVs [19] or probabilistic driver modeling [20, 21,
22]. While this is feasible for simple situations, these methods become
impractical in complex scenarios. The majority of the existing literature
relies on simulated environments or human-in-the-loop simulations, which
limits the capabilities of modeling the interactions of human drivers with AVs
and implementing the heterogeneity of human behaviors. This shortcoming
hinders the applicability of the resulting solutions as they are often limited
to the human behaviors with which they have interacted during the training. In
order to accommodate for this, some of the proposed policies in the literature
take an overly-conservative approach when interacting with humans [23]. Not
only this approach leaves the AVs vulnerable to other aggressive drivers,
especially in competitive scenarios such as intersections, but can also cause
traffic congestion and potential safety threats [24, 1].

This study builds on our prior work in [3] and aim to develop a safe and
robust training regimen that enables the AVs to work together and influence
the behavior of human drivers to create socially desirable traffic outcomes,
regardless of humans’ driving individual traits and social preferences.
Furthermore, we emphasize the importance of safety in social settings and
constrain the AVs’ policies to remove high-risk actions that can cause safety
threats. Our work in this paper is built on the following key insights. First,
we rely on learning from experience in a decentralized reinforcement learning
framework that optimizes for a social utility, and exposes the learning agents
to a wide spectrum of driver behaviors. By doing so, the resulting agents
become robust to the behavior of human drivers and are able to handle
cooperative-competitive behaviors regardless of HV’s level of aggression and
social preference. Second, a safety prioritizer is proposed to avoid high-risk
actions that can undermine driving safety. The overview of our methodology is
presented in Figure 1.

Ultimately, the focus of this paper is on exploring the safe and robust
decision-making problem in a mixed-autonomy MARL environment, in inherently
competitive driving scenarios, such as the ones illustrated in Figure 2, where
cooperation is required for safety and efficiency. The purpose is not to fully
solve the autonomous driving problem, but instead, use it as an example to
investigate the effectiveness of societal concepts from psychology literature
within the MARL domain. Further work is required to use these ideas on real
roads. Nevertheless, we are encouraged to see altruistic AVs that are safe,
robust, and can learn to influence humans in a desired way without the
limitations of current solutions [25, 26, 27, 3, 28, 29].

Our main contributions are:

  * •

We begin by formulating the mixed-autonomy traffic as a stochastic game and
introduce a general decentralized framework for training cooperative AV, that
optimizes for a social utility while prioritizing safety and allowing
adaptability.

  * •

A novel training regimen is introduced that robustifies the AVs’ capability in
creating socially desirable outcomes with regards to human drivers’ behavior.

  * •

We proposed a safety prioritizer that constrained the policy of cooperative
AVs to ensure the safety of their behavior via masking the Q-states that lead
to high-risk outcomes.

Figure 2: Highway merging and exiting scenarios in mixed traffic where the
road is shared by AVs (green) and aggressive HVs (red). Altruistic AVs must
learn to coordinate to allow for a safe and efficient merging/exiting while
also being robust to different scenarios and behaviors and ensuring safety in
decision-making.

##  II Literature Review

MARL has received a lot of attention from the research community in recent
years. MARL algorithms that assume separately trained agents perform poorly
due to the intrinsic non-stationarity of the environment [30]. Some efforts to
solve this issue assume that all agents observe the global state [31] or that
they can share their states with their neighbors [32]. These assumptions
address the non-stationarity challenge; however, they are unfeasible on real
roads[33]. Authors in [34, 35, 36] take steps to address this challenge and
[34] propose a centralized critic that reduces the influence of non-
stationarity during the learning process. Xie  _et al._ consider a MARL
approach that learns latent representations of the agent’s policies, modeling
agent strategies that depend on long interactions, alleviating the non-
stationary effect, and enabling better performance and co-adaptation [37].
[38] further investigates the impact of interactions on agent’s modeling.
Authors in [39] present a RL agent that learns to acquire social norms from
public sanctions using a decentralized framework.

###  II-A Driver Behavior and Social Navigation

Social navigation in mixed autonomy has shown the potential of collaboration
among AVs and HVs [40]. Current works in social navigation tackle the MARL
cooperation by assuming the nature of agent interactions [41, 42] or by
directly modeling or classifying human driver behaviors [43, 44, 45].
Different methods to predict or classify driver behaviors are based on driver
attributes [46], graph theory [47], game theory [1] and data mining [48].
Toghi  _et al._ release a maneuver-based dataset and propose a model that can
be used to classify driving maneuvers [28]. Authors in [44] present an
approach to modeling and predicting human behavior in situations with several
humans interacting in highly multi-modal scenarios that could allow AVs to
predict human reactions. [43] can be referred for a comprehensive study on
modeling and prediction in multi-agent traffic scenarios.

In [49] and [4] the driving patterns of humans are learned from demonstration
through inverse RL. An approach based on imitation learning is proposed by
[16] to learn a reward function for human drivers and demonstrates how AVs can
manipulate human behaviors. In [50] a centralized game-theory model for
cooperative inverse reinforcement learning is proposed. Several works take a
more abstract and traffic-level perspective [51, 52, 53]. Differently, we rely
on implicitly learning from experience altruistic behaviors that facilitate
AVs’ coordination without the need for a human model or counting on their
collaboration.

###  II-B Safe and Robust Decision-Making

In addition to the socially advantageous behavior of altruistic AVs, it is
important to consider robustness and safety. Safety is essential for AVs and
is particularly important for AVs trained using RL. We need to prioritize
safety; as coordination is often coupled with risk, in many situations there
exists a safe action that produces lower rewards, and a risky action that
produces higher rewards if agents coordinate [54, 55], however, the risky
action increases the probability of crashes when synchronization fails. In
particular, AVs using trained RL algorithms may not always behave safely as
the trained models may choose unsafe actions [23]. In that direction, several
works take a pure reward shaping approach to avoid collisions. While this is a
common practice in RL, safety is not implicitly prioritized and AVs using
those RL algorithms may not behave safely in some scenarios, as the agents
could choose dangerous actions due to function approximation.

To address this challenge, the idea of safe RL is proposed in [23] to improve
safety in unseen driving environments in which the RL algorithm behaves
unsafely. [56] presents a rule-based decision-making framework that examines
the trajectories given by the controller and replaces the actions causing
collisions. Nageshrao  _et al._ [57] includes a short-horizon safety
supervisor to substitute risky actions with safer ones. Nevertheless, these
studies consider oversimplified and non-realistic environments. The work in
[58, 59] utilizes a Q-masking approach to prevent collisions, removing the
actions that could result in a crash. Chen  _et al._ present a novel priority-
based safety supervisor to significantly reduce collisions [60].

In this work, we leverage these approaches to improve the safety of the
altruistic agents while also training the cooperative agents to be robust to
different driver behaviors and scenarios using a decentralized reward
function, local actions, and assuming partial observability. We consider a
general setup where AVs and HVs with different behaviors coexist as in Figure
2. Particularly, the figure represents two common traffic situations where
vehicles are either required to efficiently merge to a lane or exit the
highway without colliding with other vehicles. In an ideal cooperative
setting, the vehicles should proactively decelerate or accelerate to make
adequate space for the vehicles to safely merge/exit and avoid deadlock
situations, while also being robust to different scenarios and behaviors and
ensuring safety in decision-making.

##  III Problem Formulation

The MARL problem can be formulated as a centralized or decentralized problem.
A centralized controller that assigns a central joint reward and joint action
is straightforward. Nevertheless, such assumptions are impractical in the real
world. Therefore, this study focuses on a decentralized controller where
agents have partial observability and consequently, the problem is formulated
as a partially observable stochastic game (POSG) defined by
$\langle\mathcal{I},\mathcal{S},\\{\mathcal{A}_{i}\\}_{i\in\\{1,...,N\\}},\\{\mathcal{O}_{i}\\}_{i\in\\{1,...,N\\}},P,\\{R_{i}\\},\gamma\rangle$
where

  * •

$\mathcal{I}$: a finite set of agents $N\geq 2$.

  * •

$\mathcal{S}$ : a set of possible states that contains all configurations that
$N$ AVs can take (probably infinite).

  * •

$\mathcal{A}_{i}$: a set of possible actions for agent $i$.

  * •

$\mathcal{O}_{i}$: a set of observations for agent $i$.

  * •

$P$: a state transition probability function from state $s\in\mathcal{S}$ to
state $s^{\prime}\in\mathcal{S}$, $P(S=s^{\prime}|S=s,A=a)$.

  * •

$R_{i}$: a reward function for the $i^{th}$ agent, $R_{i}(s,a)$.

  * •

$\gamma$: a discount factor, $\gamma\in[0,1]$.

It should be noted that the agents have no access to the exact environmental
state but only a local observation which is correlated with the state,
increasing the difficulty of solving the POSG. The POSG can be described as
follows: at every time step $t$, $s_{t}$ is the state of the environment, each
agent senses the environment and obtains a local observation
$o_{i}:\mathcal{S}\rightarrow\mathcal{O}_{i}$, based on $o_{i}$ and its
stochastic policy $\pi_{i}$, it selects an action from the action space
$a_{i}\in\mathcal{A}_{i}$. As a result, the agent moves to the next state
$s^{\prime}$ and obtains a decentralized reward
$r_{i}:\mathcal{S}\times\mathcal{A}_{i}\rightarrow\mathbb{R}$. The goal of
each agent $i$ is to optimally solve the POSG by deriving a probability
distribution over actions in $\mathcal{A}$ at a given state, that maximizes
its cumulative discounted reward over an infinite time horizon, and find the
corresponding optimal policy $\pi^{*}:\mathcal{S}\rightarrow\mathcal{A}$.

An optimal policy maximizes the action-value function, i.e.,
$\pi^{*}(s)=\arg\max_{a}Q^{*}(s,a)$ where
$Q^{\pi}(s,a)\coloneqq\mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^{k}R_{k}(s,a)|s_{0}=s,a_{0}=a]$.
The optimal action-value function is determined by solving the Bellman
equation
$Q^{*}(s,a)=\mathbb{E}\left[R(s,a)+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})|s_{0}=s,a_{0}=a\right]$.

###  III-A Double Deep Q-Network (DDQN)

We use Double Deep Q-Network (DDQN) as the function approximator to estimate
the action-value function, i.e., $\tilde{Q}(.;\textbf{w})\cong Q(.)$ (with
weights w) [61]. DDQN improves Deep Q-Network (DQN) by decomposing the max
operation in the target into action selection and action evaluation,
mitigating the over-estimation problem. The idea is to periodically sample
data from a buffer and compute an estimate of the Bellman error or loss
function, written as

$\mathcal{L}(\textbf{w})=\mathbb{E}_{s,a,r,s^{\prime}\sim\mathcal{RM}}[(Target-\tilde{Q}(s,a;\textbf{w}))^{2}]$
(1)
$Target=R(s,a)+\gamma\tilde{Q}(s^{\prime},\underset{a^{\prime}}{\arg\max}\tilde{Q}(s^{\prime},a^{\prime};\textbf{w});\hat{\textbf{w}}))$
(2)

The DDQN algorithm then applies mini-batch gradient descent as
$\textbf{w}_{k+1}=\textbf{w}_{k}-\alpha\hat{\nabla}_{\textbf{w}}\mathcal{L}(\textbf{w})$,
on the loss $\mathcal{L}$ to learn the approximation of the value function
($\tilde{Q}(.)$). The $\hat{\nabla}_{\textbf{w}}$ operator denotes an estimate
of the gradient at $\textbf{w}_{k}$, w are the weights of the online network
and $\hat{\textbf{w}}$ are the weights of the target network which are updated
at a lower frequency ($Target_{update}$).

###  III-B Driving Scenarios

Our goal is to study driving scenarios where the absence of coordination by
the AVs compromises safety and efficiency. Additionally, we aim to investigate
adaptability among competitive scenarios and driver behaviors. For this
purpose, we define a set of scenarios $\mathcal{F}$ and choose highway merging
and exiting ramp as the base scenarios where a mission vehicle
(merging/exiting) attempts to complete its task in a mixed-traffic environment
as in Figure 2.

We design the merging and exit scenarios such that cooperation is required for
safety. AVs must coordinate and neither of them alone can realize a safe and
smooth traffic flow, i.e., only one AV will not make the merging/exiting
possible without the coordination of the other AVs. The altruistic AVs must
learn to account for the interests of all the vehicles, coordinate, make
sacrifices and guide the behavior of humans to allow for a safe
merging/exiting while also adapting to different traffic situations safely.
For instance, in Figure 2, the AV1 must learn to sacrifice and slow down
(compromising its own utility) to guide the traffic of aggressive HVs,
creating a safe corridor for the merging/exiting vehicle; while the other AVs
have to accelerate to make space for the mission vehicle. The merging and
exiting scenarios are defined as $f_{m},f_{e}\in\mathcal{F}$ respectively. We
select such situations because of their intrinsic closeness and competitive
characteristics, since the merging/exiting vehicle’s local utility conflicts
with that of the highway vehicles.

###  III-C Social Value Orientation and Altruistic AVs

Social Value Orientation (SVO) characterizes the individual’s preference to
account for the interests of others vs. their own interest [1]. The behavior
of a human or an AV can fluctuate from egoistic to absolutely altruistic based
on the importance given to the utility of others. The SVO of humans is
uncertain, therefore we depend on AVs instead to guide the traffic towards
more socially advantageous goals. Formally, the SVO angle $\phi$ of an AV,
determines how the AV balances its own benefit against that of others. In
terms of rewards, we can define the total reward $R_{i}$ of an AV as:
$R_{i}(s,a)=\cos\phi_{i}r^{ego}_{i}+\sin\phi_{i}r^{social}_{i}$, in which
$r^{ego}_{i}$ is the AV’s specific reward (egoistic) and $r^{social}_{i}$ is
the overall reward of other vehicles (social) respect to the $i^{th}$ AV [3,
2]. The SVO angle can be changed from $\phi=0$ (purely egoistic) to
$\phi=\pi/2$ (purely altruistic). Nevertheless, none of the two extremes is
optimum, and a point in between yields the most socially advantageous result,
defined as the optimal SVO angle $\phi^{*}$. SVO helps explain the behaviors
that allow the mission vehicle to merge or exit in Figure 2. Without SVO, the
mission vehicle in Figure 2 could cause traffic congestion or an unsafe
situation. AVs need to consider SVO, since HVs can not communicate that
directly, and we should not expect HVs to cooperate.

###  III-D Driving Behaviors

The challenge of simulating diverse behaviors can be framed as the problem of
obtaining the suitable range of parameters that can generate the heterogeneous
behaviors within the simulator. Many studies from social traffic psychology
establish that driving behavior falls between aggressive and conservative.
Nonetheless, the precise definitions differ between studies [13]. In general,
the term “aggressive driving” covers a range of unsafe driving behaviors like
overspeeding or running red lights. However, the causes of aggressive driving
come in various forms and are not always obvious. Some are due to undesirable
roadway situations, while others are individual traits or states of mind.
Furthermore, there is a relationship between aggressiveness and egoism, as
egoistic drivers usually do not yield and also tend to engage in speeding,
risk-taking, and similar aggressive behaviors. While there is a correlation
between these terms [14, 13, 15], for the purpose of this paper, we separate
egoism from aggressiveness by characterizing social preferences and individual
traits.

We differentiate between social preferences and individual traits of drivers,
as they lead to different behaviors. First, we characterize egoism and
altruism as social preferences, and identify an egoistic driver as a selfish
driver who accounts for his own utility independently from his aggressiveness.
Second, we characterize aggressiveness and conservativeness as individual
traits, and identify an aggressive driver as a driver whose actions cause
aggressive behaviors. Social preferences such as egoism are characterized by
their social goals and intentions, whereas individual traits such as
aggressiveness are characterized by the consequences of their actions. In that
sense, an egoistic driver is a self-centered driver characterized by a lack of
social motivation, a driver that feels like he owns the road and does not
consider the other drivers. Egoist drivers often engage in aggressive
behaviors and while ego defensiveness is not the only cause of aggressiveness,
it is still a main contributing factor of anger and aggressive driving [14,
15]. Despite the correlation between the two categories, they arise from
different natures and lead to different behaviors. For instance, a driver
could be egoistic and conservative. We could imagine a driver who drives
cautiously in order to protect himself (selfish motivation/preference) and, as
a consequence, behaves conservatively (outcome of his actions).

Formally, in our simulation, social preferences (egoism or altruism) are
characterized by the AV’s SVO angular preference $\phi$; and individual traits
(aggressiveness, conservativeness, etc) by the HV driver model parameters
($\mathcal{P}$) as described in section IV-D. Based on the values of these
parameters, a vehicle will exhibit aggressive or conservative behaviors. In
our experiments, we assume the SVO of HVs to be unknown as they can not
communicate that directly. Finally, we define a set of behaviors
$\mathcal{B}$, i.e, aggressive, moderate and conservative,
$b_{a},b_{m},b_{c}\in\mathcal{B}$ based on the parameters ($\mathcal{P}$)
obtained in section IV-D.

###  III-E Problem Formulation

We formulate the problem as the POSG defined; where the road is shared by a
set of HVs $h_{k}\in\mathcal{H}$, with an undetermined SVO $\phi_{k}$ and
heterogeneous behaviors $b_{k}\in\mathcal{B}$; a set AVs
$i_{i}\in\mathcal{I}$, that are connected together using V2V communication,
controlled by a decentralized policy and sharing the same SVO, and a _mission
vehicle_ , $M\in\mathcal{I}\cup\mathcal{H}$ that is aiming to accomplish its
mission (highway merging/exiting) and can be AV or HV. We focus on the multi-
agent maneuver-level decision-making problem for AVs in mixed-autonomy
environments, and study the following problems: how AVs can learn in a mixed-
autonomy environment cooperative optimal policies $\pi^{*}(s)$ that are robust
to different scenarios $f\in\mathcal{F}$ and behaviors $b\in\mathcal{B}$ while
ensuring safety on the decision-making, and how sensitive is the performance
of the altruistic AVs to the HVs behaviors.

As AVs are connected, we assume that they receive an accurate local
observation of the environment
$\tilde{\textbf{o}}_{i}\in\widetilde{\mathcal{O}}_{i}$, sensing all the
vehicles within their perception range, i.e, a subgroup of HVs
$\widetilde{\mathcal{H}}\subset\mathcal{H}$ and a subgroup of AVs
$\widetilde{\mathcal{I}}\subset\mathcal{I}$. Nevertheless, AVs are unable to
share their actions or rewards, and they take individual actions from a set of
high-level actions $a_{i}\in\mathcal{A}_{i}(|\mathcal{A}_{i}|=5)$. The goal of
this work is to train AVs that learn how to drive in a mixed-autonomy scenario
in a robust, efficient and safe manner while benefiting all the vehicles on
the road.

##  IV Safe and Robust Altruistic Driving

To drive in a mixed-autonomy environment in a robust and safe manner, we
propose a MARL approach with a general decentralized reward function that
optimizes for a social utility by inducing altruism in the agents; the general
reward accounts for any anticipated vehicle’s mission, allowing it to be
applied to different scenarios and tasks; and ensuring safety by adding a
safety prioritizer. We train altruistic AVs that learn from experience to
perform a task, account for the interests of all the vehicles, while being
able to adapt to other traffic situations safely. We carefully design a
decentralized general reward function, a suitable architecture, and a safety
prioritizer to promote the desired safe altruistic behavior in AVs’ decision-
making process. The overview of our approach as presented in Figure 1 and
Figure 2 helps us to create intuition on these points, by introducing driving
scenarios in which altruistic AVs lead to socially advantageous results while
adapting to different traffic circumstances.

Action Space. We define a high-level action space $\mathcal{A}$ of discrete
meta-actions for decision-making. In particular, we select a set of five high-
level actions as $a_{i}\in\mathcal{A}_{i}=[\texttt{Change to Right Lane}$,
Change to Left Lane, Accelerate, Decelerate, $\texttt{Idle}]^{T}$. These meta-
actions are then converted into trajectories and low-level control signals,
which ultimately control the vehicle’s movement.

Observation Space. We use a multi-channel VelocityMap observation ($o_{i}$)
that embeds the relative speed of the vehicle with respect to the ego vehicle
in pixel values, as in [2]. We represent the information in multiple semantic
channels that embed: 1) the AVs, 2) the HVs , 3) the mission vehicle, 4) an
attention map to highlight the position of the ego vehicle, and 5) the road
layout. To map into pixels the relative speed of the vehicles, we use a
clipped logarithmic function which improves the dynamic range and shows better
results than a straightforward linear mapping. As temporal information is
necessary for safe decision-making, we use a history of VelocityMaps
successive observations to create the input state to the Q-network $\psi_{i}$.

###  IV-A Decentralized General Reward

We train the AVs from scratch using local observations and a decentralized
reward structure and expect them to learn the driving task in different
scenarios while accounting for individual diver’s missions. Consequently, we
design a well-engineered general reward function that accounts for the social
utility, traffic metrics and desired missions. The agent’s
$I_{i}\in\mathcal{I}$ local reward is defined as

$\displaystyle R_{i}(s,a)={}$ $\displaystyle
R^{\mathrm{ego}}+R^{\mathrm{social}}$ (3) $\displaystyle R^{\mathrm{ego}}={}$
$\displaystyle\cos\phi_{i}r_{i}(s,a)$ $\displaystyle R^{\mathrm{social}}=$
$\displaystyle\sin\phi_{i}\Big{[}\sum_{j}r^{\mathrm{AV}}_{i,j}(s,a)+\sum_{j}r_{i,j}^{M}(s,a)$
$\displaystyle+\sum_{k}r^{\mathrm{HV}}_{i,k}(s,a)+\sum_{k}r_{i,k}^{M}(s,a)\Big{]}$

in which $R^{\mathrm{ego}}$, $R^{\mathrm{social}}$ represents the egoistic and
social reward, $i\in\mathcal{I}$,
$j\in(\widetilde{\mathcal{I}}\setminus\\{I_{i}\\})$,
$k\in\widetilde{\mathcal{H}}$. The term $r_{i}$ represents the ego vehicle’s
reward obtained from traffic metrics and the angle $\phi$ allows to adjust the
level of egoism or altruism. We decouple the social component in cooperation
(the altruistic behavior among AVs, i.e, AV’s altruism toward others AVs) and
sympathy (AV’s altruism toward HVs) as they differ in nature. The sympathy
term, $r^{\mathrm{HV}}_{i,k}$ considers the individual reward of the HVs,
while the cooperation term, $r^{\mathrm{AV}}_{i,j}$ considers the individual
reward of the other AVs, and are defined as

$r^{\mathrm{HV}}_{i,k}=\frac{\mathcal{W}_{k}}{d_{i,k}^{\lambda}}\sum_{m}\omega_{m}x_{m}\quad
r^{\mathrm{AV}}_{i,j}=\frac{\mathcal{W}_{j}}{d_{i,j}^{\lambda}}\sum_{m}\omega_{m}x_{m}$
(4)

in which $d_{i,k}/d_{i,j}$ represents the distance between the agent and the
corresponding HV/AV, $\lambda$ is a dimensionless coefficient,
$\mathcal{W}_{k}$ a weight value for individual vehicle’s importance, $m$ are
the traffic metrics that have been considered in the vehicle’s utilities
(speed, crashes, etc.), in which $x_{m}$ is the $m$ metric normalized value
and $w_{m}$ is the weight associated to that metric. The term $r^{\mathrm{M}}$
accounts for the reward of the vehicle’s mission. A mission is defined as any
desired specific outcome for a particular vehicle, as merging, exiting, etc.

$r^{\mathrm{M}}_{i,j}=\begin{cases}\frac{w_{j}}{(d_{i,j})^{\mu}},&\mathrm{if}f(j)\\\
0,&\mathrm{o.w.}\end{cases}\quad
r^{\mathrm{M}}_{i,k}=\begin{cases}\frac{w_{k}}{(d_{i,k})^{\mu}},&\mathrm{if}f(k)\\\
0,&\mathrm{o.w.}\end{cases}$ (5)

The function $f(v)$ is an independent function to evaluate the mission; $f(v)$
return true if the vehicle $v$ has a mission defined and the mission has been
accomplished in the recent time window. $\mu$ is a dimensionless coefficient,
$w_{j}/w_{k}$ are weights for individual vehicle’s mission (importance of the
mission). This allows to define a general reward independent of the driving
scenario and mission goals for different vehicles. In our experiments, a HV
can be assigned a merging mission or a highway exiting mission, as referred to
in Figure 2.

###  IV-B Deep MARL architecture for Cooperative Driving

We use a 3D Convolutional Neural Network (CNN) with a safety prioritizer as
presented in Figure 3. The 3D CNN acts as a feature extractor and uses a
history of VelocityMap observations to account for the temporal information.

To tackle the non-stationarity of MARL, we train the agents in a semi-
sequential approach, as in [2]. The agents are trained independently for
$N_{iterations}$ iterations while freezing the policies of the remaining AVs,
$\textbf{w}^{-}$. Subsequently, the other agents’ policies are updated with
the new policy, $\textbf{w}^{+}$. To improve sample efficiency and train the
agent safely, reducing episode resets due to imminent collisions, we use a
safety prioritizer that, in the cases where the action selected by the agent
policy is unsafe, selects a safe action and stores the unsafe action ($a_{t}$)
and the related state in the $RM$ with a suitable penalty on the reward
($r_{unsafe}$) for the unsafe state-action pair. Those pairs are not removed
so the agent can also learn from unsafe experiences. The experience
$(\psi(s_{t}),a_{t},r_{unsafe},\emptyset$) is stored in $RM$ with a terminal
next state $\emptyset$, the target for this unsafe pair $(s_{t},a_{t})$ is
$Target(s_{t},a_{t})^{DDQN}=r_{unsafe}$. The details of the safety prioritizer
are given in the next section IV-C.

Algorithm 1 summarizes the overall methodology of our safety prioritized deep
MARL architecture. Additionally, we do not initiate the learning process until
the replay buffer is filled with a minimum number of sample simulations.
Moreover, inspired by [62] and [2], we update our experience replay buffer to
compensate for the highly skewed training data. Balancing skewed data is a
common practice in machine learning and is beneficial in our MARL problem as
well.

Algorithm 1 Safety Prioritized Multi-agent DDQN

Initialize experience replay buffer $RM$.

Initialize $\tilde{Q}(.;\textbf{w}^{-})$ with random weights
$\textbf{w}^{-}=\textbf{w}_{ini}$

Initialize target network $\tilde{Q}(.;\hat{\textbf{w}})$ with weights
$\hat{\textbf{w}}=\textbf{w}^{-}$

Pre-store experience of first’s 50 episodes in $RM$

for $\mathrm{e}=50$ to $N_{\mathrm{episode}}$ do

Initialize $s_{1}=\\{\tilde{\textbf{o}}_{1}\\}$ and compute
$\psi_{1}=\psi(s_{1})$

for t = 1 to T do

for $I_{i}$ in $\mathcal{I}$ do

Freeze $\textbf{w}^{-}$ for all $I_{j}$, $j\neq i$

for $m=1$ to $N_{iterations}$ do

With probability $\epsilon$ select a random action $a_{t}$ ,

otherwise select $a_{t}=\max_{a^{\prime}\in
A}Q(\psi(s_{t}),a^{\prime};\textbf{w}^{+})$

if $a_{t}$ is unsafe (Algorithm 2) then

Store $(\psi_{t},a_{t},r_{unsafe},\emptyset$) in $RM$

$a_{t}$ = Compute a safe action (Algorithm 3)

Execute safe action $a_{t}$ , and observe $r_{t},\tilde{\textbf{o}}_{t}$

Set $s_{t+1}=\\{s_{t},\tilde{\textbf{o}}_{t+1}\\}$ and
$\psi_{t+1}=\psi(s_{t+1})$

Store experience $(\psi_{t},a_{t},r_{t},\psi_{t+1})$ in $RM$

Sample a mini-batch of size $M$ from $RM$

Compute $\mathcal{L}(\textbf{w}^{+})$

Performs gradient descent

$\textbf{w}^{+}_{k+1}\leftarrow\textbf{w}^{+}_{k}-\alpha\hat{\nabla}_{\textbf{w}}\mathcal{L}(\textbf{w}^{+})$

$\textbf{w}^{-}=\textbf{w}^{+}$ for all $I_{i}\in\mathcal{I}$

Every $\quad Target_{update}\quad$ reset
$\hat{\textbf{w}}\leftarrow\textbf{w}^{-}$

###  IV-C Safety prioritizer

As safety is an essential requirement in autonomous navigation, we add a
safety prioritizer to the MARL algorithm, to avoid and penalize imminent
collisions. This allows the agent to improve sample efficiency during training
and avoid collisions during deployment. If the agent encounters an unseen
scenario and decides to take an unsafe action, that action will be avoided.
The safety prioritizer improves the simulation results and is critical in
real-life situations. The safety prioritizer consists of Algorithm 2 and
Algorithm 3.

Algorithm 2. During action selection of the agent $I_{i}$, once an action
$a_{t}$ is chosen, the safety prioritizer checks if the action is safe by
computing a safety score for $N_{steps}$ of planning. We utilize the time-to-
collision ($ttc$) as a safety score. If $safety_{score}<safe_{th}$ the action
is unsafe and we need to select a safe action. The selection of a safe action
is presented in Algorithm 3.

Algorithm 3. The safe action selection is different in training and testing.
During training, to encourage exploration, we remove the unsafe actions and
keep the random action selection following the current exploration policy on
the remaining actions. During testing, we follow the greedy policy in the
subset of safe actions
$a_{t}=\max_{a^{\prime}\in\widetilde{\mathcal{A}}_{safe}}Q(\psi(s_{t}),a^{\prime};\textbf{w})$.
It should be noted that the algorithm does not choose the safest of all
possible actions, as that action may lead to particularly conservative
behaviors that can compromise traffic efficiency; we instead remove the
imminent unsafe actions and follow the priority given by the learned
altruistic policy. If it happens that all possible actions are unsafe, we
return the action $a_{t}\in\mathcal{A}$ with the highest safety score. In that
way during training the constrained exploration will keep the agent from
taking unsafe actions which will lead to sample efficient and more stable
learning; and during testing the decision-making is based on the prosocial
learned policy with minimum intervention from the safety prioritizer,
achieving higher traveled distance while avoiding collisions.

Figure 3: Deep MARL architecture with the safety prioritizer.

###  IV-D Modeling Driver Behaviors

We model the longitudinal movements of HVs using the Intelligent Driver Model
(IDM) [63], while the lateral actions of HVs are based on the MOBIL model
[64]. The MOBIL model considers two main criteria,

The safety criterion ensures that after the lane change, the deceleration of
the new follower $\mathrm{a}_{n}$ in the target lane does not exceed a safe
limit, i.e, $\mathrm{a}_{n}>-b_{\mathrm{safe}}$.

The incentive criterion determines the advantage of HV after the lane change,
quantified by the total acceleration gain, given by

$\mathrm{a}^{\prime}_{ego}-\mathrm{a}_{ego}+\sin\phi_{ego}\Big{(}(\mathrm{a}^{\prime}_{n}-\mathrm{a}_{n})+(\mathrm{a}^{\prime}_{o}-\mathrm{a}_{o})\Big{)}>\Delta
a_{th}$ (6)

where $\mathrm{a}_{o}$, $\mathrm{a}_{n}$ and $\mathrm{a}_{ego}$ represent the
acceleration of the original follower in the current lane, the new follower in
the target lane and the ego HV, correspondingly, and
$\mathrm{a}^{\prime}_{o}$, $\mathrm{a}^{\prime}_{n}$, and
$\mathrm{a}^{\prime}_{ego}$ are the equivalent accelerations considering that
the ego HV has changed the lane, $\sin\phi_{ego}$ is the politeness factor.
Finally, the lane change is performed if the safety and incentive criterion
are mutually satisfied.

The IDM Model determines the longitudinal acceleration of a HV
$\dot{v}_{\mathrm{k}}$ as following,

$\dot{v}_{\mathrm{k}}=\mathrm{a}_{\mathrm{max}}\Big{[}1-\Big{(}\frac{v_{k}}{v_{\mathrm{k}}^{0}}\Big{)}^{\delta}-\Big{(}\frac{d^{*}(v_{k},\Delta
v_{k})}{d_{k}}\Big{)}^{2}\Big{]}$ (7)

in which $v_{k}$, $d_{k}$, $\delta$, $\Delta v_{k}$, $v_{\mathrm{0}}^{k}$
denote the speed, the actual gap, the acceleration exponent, the approach
rate, and the desired speed of the $k^{th}$ HV, respectively.

The desired minimun gap of the $k^{th}$ HV is given by,

$d^{*}(v_{k},\Delta v_{k})=d_{k}^{0}+v_{k}T_{\mathrm{k}}^{0}+\frac{v_{k}\Delta
v_{k}}{(2\sqrt{\mathrm{a}_{\mathrm{max}}.\mathrm{a}_{\mathrm{des}}})}$ (8)

where $T_{k}^{0}$, $d_{k}^{0}$, $\mathrm{a}_{\mathrm{max}}$, and
$\mathrm{a}_{\mathrm{des}}$ are the safe time gap, the minimum distance, the
comfortable maximum acceleration, and deceleration, correspondingly.

The typical parameters for MOBIL model are $\sin\phi_{e}=0.5$, $\Delta
a_{th}=0.1\frac{m}{s^{2}}$ and $b_{\mathrm{safe}}=4\frac{m}{s^{2}}$. Table I
shows typically used parameters of the IDM model [63].

TABLE I: Typical parameters for the IDM model Parameter | $v^{0}$ | $T^{0}$ | $\mathrm{a}_{\mathrm{max}}$ | $\mathrm{a}_{\mathrm{des}}$ | $\delta$ | $d^{0}$
---|---|---|---|---|---|---
Value | 30 m/s | 1.5 s | 1 m/s2 | 1.5 m/s2 | 4 | 2 m

Heterogeneous Driver Behaviors. Though those parameters are typical used for
IDM and MOBIL models, they simulate just one behavior. In order to generate
diverse behaviors $\mathcal{B}$, we frame the task of simulating diverse
behaviors as the problem of obtaining the appropriate range of parameters
($\mathcal{P}$) that can generate those behaviors. To achieve that, we
leverage a behavior classifier and iteratively simulate the parameters and
classify the behaviors, mapping parameters to behaviors. To classify the
behaviors we represent traffic using a traffic-graph at each time step $t$,
$\mathcal{G}_{t}$, with a set of edges $\mathcal{E}(t)$ and a set of vertices
$\mathcal{V}(t)$ as functions of time, i.e, the positions of vehicles
($\widetilde{\mathcal{H}}\cup\widetilde{\mathcal{I}}$) represent the vertices.
The adjacency matrix $A_{t}$ is given by $A(k,m)=d(v_{k},v_{m}),k\neq m$ , in
which $d(v_{k},v_{m})$ is the shortest travel distance between vertices $k$ to
$m$. Then we use centrality functions [47] to classify the behavior (level of
aggressiveness) resulted from $\mathcal{P}$, and then use those simulation
parameters $\mathcal{P}$ to model behaviors within the simulator with varying
levels of aggressiveness. The centrality functions are defined as,

Closeness Centrality: the discrete closeness centrality of the
$k^{\textrm{th}}$ vehicle at time $t$ is defined as,

$\mathcal{C}^{k}_{C}[t]=\frac{{N-1}}{\sum_{v_{m}\in\mathcal{V}(t)\setminus\\{v_{k}\\}}d_{t}(v_{k},v_{m})},$
(9)

where $N=|\widetilde{\mathcal{H}}\cup\widetilde{\mathcal{I}}|$. The more
central the vehicle is located, the higher $\mathcal{C}^{k}_{C}[t]$ and the
closer it is to all other vehicles.

Degree Centrality: the discrete degree centrality of the $k^{\textrm{th}}$
vehicle at time $t$ is defined as,

$\displaystyle\mathcal{C}^{k}_{D}[t]=\bigl{|}\\{v_{m}\in\mathcal{N}_{k}(t)\\}\bigr{|}+\mathcal{C}^{k}_{D}[t-1]$
(10) $\displaystyle\textrm{such that}\
(v_{k},v_{m})\not\in\mathcal{E(\tau)},\tau=0,\ldots,t-1$

in which $\mathcal{N}_{k}(t)=\\{v_{m}\in\mathcal{V}(t),\ A_{t}(k,m)\neq
0,\nu_{m}\leq\nu_{k}\\}$ represents the set of vehicles in the proximity of
the $k^{\textrm{th}}$ vehicle, given that $\nu_{m}\leq\nu_{k}$; and
$\nu_{m},\nu_{k}$ denote the velocities of the $m^{\textrm{th}}$ and
$k^{\textrm{th}}$ vehicles. The more new vehicles seen by vehicle $k$ that
meet this condition, the higher $\mathcal{C}^{k}_{D}[t]$.

With the centrality functions we can measure the Style Likelihood Estimate
(SLE) for different driver styles [47]. We consider two SLE measures. The SLE
of overtaking and sudden lane-changes ($SLE_{l}$) and the SLE of overspeeding
($SLE_{o}$). The $SLE_{l}$ and $SLE_{o}$ can be computed by measuring the
first derivative of the centrality functions as,

$\textrm{SLE}_{l}(t)=\left\lvert\frac{\partial\mathcal{C}_{C}(t)}{\partial
t}\right\rvert\quad\textrm{SLE}_{o}(t)=\left\lvert\frac{\partial\mathcal{C}_{D}(t)}{\partial
t}\right\rvert$ (11)

The maximum likelihood $\textrm{SLE}_{\textrm{max}}$ is calculated as
$\textrm{SLE}_{\textrm{max}}=\max_{t\in\Delta t}{\textrm{SLE}}(t)$.

Using those functions, we can approximately quantify and classify driver
behaviors in our simulation. The intuition behind that is that an aggressive
driver may frequently overspeed or perform sudden lane changes; while
overspeeding the $\mathcal{C}_{D}(t)$ monotonically increases (higher
$\textrm{SLE}_{o}(t)$) and during sudden lane changes the slope and the
extrema of $\mathcal{C}_{C}(t)$ changes values. Thus higher values of
$\textrm{SLE}_{\textrm{max}}$ are related to increased levels of
aggressiveness. Conversely, conservative drivers are not inclined towards
those aggressive maneuvers, and the degree centrality will be relatively flat,
thus $\textrm{SLE}_{o}(t)\approx 0$ for conservative drivers.

We use these metrics as approximations of the driver’s level of
aggressiveness. In order to compute the suitable values for our simulation, we
iteratively simulate the parameters from IDM and MOBIL models, and for each
set of parameters, we quantify the resulting behavior in the simulation (using
those metrics). Mapping the parameters $\mathcal{P}$ to behaviors (quantified
in the simulation for those parameters). The estimated simulation parameters
that simulate conservative, moderate and aggressive behavior in our scenarios
are presented in Table II.

TABLE II: Estimated simulation parameters that simulate conservative, moderate and aggressive behavior in our scenarios. Model | Parameter | Aggressive | Moderate | Conservative
---|---|---|---|---
MOBIL | $\sin\phi_{e}$ | 0 | 0.3 | 1
| $\Delta a_{th}$ | 0 $m/s^{2}$ | 0.1 $m/s^{2}$ | 0.4 $m/s^{2}$
| $b_{\mathrm{safe}}$ | 12.0 $m/s^{2}$ | 6.0 $m/s^{2}$ | 2.0 $m/s^{2}$
IDM | $T^{0}$ | 0.5s | 1s | 3s
| $d^{0}$ | 1 $m$ | 2 $m$ | 6.0 $m$
| $\mathrm{acc}_{\mathrm{max}}$ | 7.0 $m/s^{2}$ | 3.0 $m/s^{2}$ | 1.0 $m/s^{2}$
| $\mathrm{acc}_{\mathrm{des}}$ | 12.0 $m/s^{2}$ | 7.0 $m/s^{2}$ | 2.0 $m/s^{2}$

The desired velocity $v^{0}$ is set to $30m/s$ and the acceleration exponent
$\delta=4$.

###  IV-E Computational Details and Hyperparameter

We customize the OpenAI Gym environment in [65] to suit our particular driving
scenario and MARL problem. The PyTorch implementation of our architecture on
average takes 3.1GB of memory for 4 agents and 18 HVs. Using a GPU NVIDIA
Tesla V100. The training process is repeated several times to ensure
convergence of the experiments to a similar policy. The network is trained for
$N_{episodes}=10,000$ taking on average 8 hours and a forward pass during
testing requires on average 15ms. We utilize 3,200 GPU-hours for our
simulations. Table III lists our simulation and training hyper-parameters.

TABLE III: Simulation and training hyper-parameters. Parameter | Value | Parameter | Value
---|---|---|---
$N_{\mathrm{episode}}$ | 10,000 |  $\epsilon$ decay | Linear
$RM$ buffer size | 8,000 | Initial exploration $\epsilon_{0}$ | 1.0
Batch size | 32 | Final exploration | 0.05
Learning rate $\alpha_{0}$ | 0.0005 | Optimizer | ADAM
$Target_{update}$ | 300 | Discount factor $\gamma$ | 0.95
$|\mathcal{H}|$ | $18$ | $|\mathcal{I}|$ | $4$

##  V Experimental Results

Controlled Variables. We study how the $safe_{th}$, the _level of
aggressiveness_ , the _traffic scenarios_ ($f_{j}$) and the _HVs’ behaviors_
($b_{k}$) impact the performance of AVs. We consider the case in which the
mission vehicle (merging/exiting) in Figure 2 is _human-driven_ ,
$M\in\mathcal{H}$, and define the following terms:

  * •

$AV_{S}$. Social AV ($\phi_{i}=\phi^{*}$) that act _altruistically_ in the
presence of diverse HVs behaviors $b\in\mathcal{B}$.

  * •

$AV_{E}$. Egoistic AV ($\phi_{i}=0$) that act _egoistically_ in the presence
of diverse HVs behaviors $b\in\mathcal{B}$.

with $\phi^{*}$ to be the optimal SVO angle tuned to reach the optimal level
of altruism as in [2].

Performance Metrics. We measure the performance of our system based on safety,
efficiency, altruistic performance gain ($PG$) and adaptation error
$\mathrm{A}_{\mathrm{error}}$. To measure safety, we compute the percentage of
episodes that encountered a crash ($C(\%)$). For efficiency, the average
traveled distance ($DT(m)$) of the vehicles and the number of missions
accomplished by the mission vehicle are used. The altruistic performance gain
is measured by computing the difference in the safety/efficiency performance
of $AV_{E}$ and $AV_{S}$, as

$PG_{safety}(\%)=\frac{(AV_{E})_{C(\%)}-(AV_{S})_{C(\%)}}{N_{Episodes}}$ (12)

$PG_{efficiency}(\%)=\frac{(AV_{S})_{DT(m)}-(AV_{E})_{DT(m)}}{(AV_{E})_{DT(m)}}$
(13)

Finally the adaptation error is a weighted sum function of the safety
($C(\%)$) and efficiency ($DT(m)$) performance of the $AV_{S}$ when trained
and tested in different scenarios/behaviors. Defined as,

$A_{error}(\%)=w_{s}\times(C(\%))+w_{e}\times 100(1-\frac{DT}{DT_{max}})$ (14)

such that an adaptation between different situations that result in $0\%$
crash and $DT=DT_{max}$ will have $\mathrm{A}_{\mathrm{error}}=0\%$.

###  V-A Hypotheses

In this section we examine the following hypotheses

  * •

H1. _The higher the level of aggressiveness in a mixed-autonomy scenario, the
greater the impact of cooperation. Thus, we expect a higher performance gain (
$PG$) when altruistic AVs face environments with higher level of
aggressiveness_.

  * •

H2. _Altruistic AVs agents using the decentralized framework can adapt to
different driver behaviors and traffic scenarios without compromising the
overall traffic metrics. However, the higher the similarity of testing
scenarios to the ones seen during training (
$(f_{test},b_{test})\approx(f_{train},b_{train})$), the lowest adaption error
($\mathrm{A}_{\mathrm{error}}$)_.

  * •

H3. _With the inclusion of the safety prioritizer, we anticipate improvement
in safety and efficiency. We expect that AVs will cause more crashes in the
absence of a safety prioritizer ( $safe_{th}=0$)_.

###  V-B Analysis and Results

Based on the hypotheses, we explore their correctness through the experiments
in this section.

####  V-B1 Sensitivity analyses to HVs behaviors

To study the hypothesis H1 we investigate the effect of HV behaviors on the
altruistic AV agents. We focus on scenarios with a HV mission vehicle, with
safe AVs that act _altruistically_ ($AV_{S}$) or _egoistic_ ($AV_{E}$) , in
environments with increasing levels of HVs aggressiveness. Figure 4
illustrates the altruistic performance gain for increasing levels of HVs’
aggressiveness for 2 AVs (left) and 4 AVs (right). It demonstrates that the
more aggressive the HVs are, the higher is the impact of cooperation and thus
confirms the H1. This is also observed in Figure 5 where the level of
aggressiveness is decomposed into lateral and longitudinal aggressiveness.
Lateral and longitudinal aggressiveness is varied by changing the MOBIL and
IDM parameters (Table II) from aggressive to conservative. Figure 5 shows that
the altruistic gain increases in both directions, but is more pronounced in
the longitudinal direction. That is probably due to the simulated scenarios
having more longitudinal maneuvers.

Figure 4: Sensitivity analyses measured by altruistic performance gain (PG) of
AVs, the more aggressiveness of the HVs, the higher the impact/gain of
cooperation. Figure 5: Lateral and longitudinal sensitivity analyses, the
altruistic performance gain (PG) increase in both lateral and longitudinal
directions.

####  V-B2 Domain adaptation of altruistic agents

Following the sensitivity analysis, we investigate the domain adaptation of
the AVs to validate the H2. Figure 6 shows how the altruistic AVs learn to
adapt to different scenarios and behaviors, based on an adaptation score. For
the experiments, $AV_{S}$ are trained in different scenarios
$f_{i}\in\mathcal{F}$ in the presence of HVs with different behaviors
$b_{k}\in\mathcal{B}$ and tested in other scenarios $f_{j}\in\mathcal{F}$ and
behaviors $b_{l}\in\mathcal{B}$. In our experiments, we consider two case
study scenarios $f_{m},f_{e}\in\mathcal{F}$ (merging/exiting) in environments
with three different HVs behaviors $b_{a},b_{m},b_{c}\in\mathcal{B}$
(aggressive, moderate, conservative) see Table II; and a mixed behavior
environment, in which HVs are created randomly and their behaviors are
selected based on a uniform distribution over the behaviors in $\mathcal{B}$,
given equal probability to the defined behaviors. In total, we have eight
combinations of scenarios and behaviors, namely: ($f_{m},b_{mix}$),
($f_{m},b_{a}$), ($f_{m},b_{m}$), ($f_{m},b_{c}$), ($f_{e},b_{mix}$),
($f_{e},b_{a}$), ($f_{e},b_{m}$), ($f_{e},b_{c}$).

The results are presented in Figure 6 as an adaptation matrix, showing the
$\mathrm{A}_{\mathrm{error}}$ for different domains, the
$\mathrm{A}_{\mathrm{error}}$ is in percentage ($\%$) and color-map in
logarithmic scale to increase the perceived dynamic range for visualization.
In our analyses, the weights used for $\mathrm{A}_{\mathrm{error}}(\%)$ are
$w_{s}=\frac{2}{3}$ and $w_{e}=\frac{1}{3}$, which weighs the safety
performance higher. $DT_{max}$ is computed based on the maximum distance for
each situation. Additionally, Figure 7 and Figure 8 illustrate how the AVs
adapt in terms of safety (measured by $C(\%)$) and efficiency (measured by
$DT(m)$), separately.

The matrix shows the best performances in the diagonal as agents trained and
tested in the same environment (($f_{i},b_{k}$); ($f_{j},b_{l}$) with $i=j$
and $k=l$) experience during testing similar situations to the ones seen in
training. The vehicles trained in the merging environment are able to perform
the exiting mission for different behaviors, and vice-versa. It is interesting
to notice that when trained in a conservative environment ($b_{c}$), the
performance when tested in aggressive environments ($b_{a}$) is poor. We
believe that the reason is that in conservative environments, the HVs yield
the mission vehicle, and the AVs learn to rely on HVs to guide the traffic.
This learned policy is valid in a conservative environment where one can
expect the HVs to always create a safe space for the mission vehicle. However,
the same is not valid in more aggressive environments, in which AVs have to
guide the traffic to avoid dangerous situations. As a result, the performance
of vehicles trained in a conservative environment and tested in an aggressive
one is the worse.

On the other hand, an adequate performance adaptation (lower
$\mathrm{A}_{\mathrm{error}}$) is obtained when agents are trained in the
presence of all moderate HVs ($b_{m}$) or a mixed behavior environment
($b_{mix}$), in which AVs face situations where the HVs yield, but also
situations that require learning how to guide the traffic to optimize for the
social utility. The results from the domain adaptation matrix indicate that a
moderate or mixed environment is the most suitable for training robust AVs and
show the adaptability of AVs to different situations, thereby confirming the
H2 hypothesis.

It can be concluded that the adaptation between the environments is not
reciprocal and the selection of the environment and situations should be
considered during training, based on the application needs and target
situations. The adaptation matrices serve as reference and provide insights on
domain adaptation in mixed-autonomy traffic, the matrices present the settings
in which altruistic AVs can best learn cooperative policies that are robust to
different traffic scenarios and human behaviors.

Figure 6: The domain adaptation matrix with adaptation error
($\mathrm{A}_{\mathrm{error}}$) between different traffic scenarios and
behaviors. $\mathrm{AV}_{\mathrm{S}}$ are trained (rows of the matrix) in
different scenarios $f_{i}\in\mathcal{F}$ in the presence of HVs with
different behaviors $b_{k}\in\mathcal{B}$ and tested (columns of the matrix)
in other scenarios $f_{j}\in\mathcal{F}$ and behaviors $b_{l}\in\mathcal{B}$.
Each pair ($f_{i},b_{k}$) is a combination of scenario and behavior. The lower
$\mathrm{A}_{\mathrm{error}}$ the most suitable the adaptability between those
domains. Figure 7: The domain adaptation matrix with crash percentage
($C(\%)$) between different traffic scenarios and behaviors.The lower $C(\%)$
the most suitable the adaptability in terns of safety (measured by $C(\%)$)
between those domains.  Figure 8: The domain adaptation matrix with distance
traveled ($DT(m)$). Illustrating how the AVs adapt to other situations in
terms efficiency (measured by $DT(m)$).

####  V-B3 Transfer Learning

Together with domain adaptation we exploit transfer learning to foster
generalization while efficiently learning harder tasks from trained models and
therefore accelerate the learning. We study how the policies learned during
merging can be transferred to the exiting environment. For that, we train AVs
agents from scratch for the mission/task of merging
$\mathrm{AV}_{\mathrm{merging}}$ (T1), train AVs agents to drive on a highway,
and then use that model as the starting point to learn the merging task
$\mathrm{AV}_{\mathrm{drive-to-merging}}$ (T2), train AVs agents for the
exiting task and then use that model as the starting point to learn the
merging task $\mathrm{AV}_{\mathrm{exiting-to-merging}}$ (T3); and apply the
same procedure for the exiting task, learning to exit from scratch
$\mathrm{AV}_{\mathrm{exiting}}$ (T4), after learned how to drive
$\mathrm{AV}_{\mathrm{drive-to-exiting}}$ (T5) and after learned how to merge
$\mathrm{AV}_{\mathrm{merging-to-exiting}}$ (T6). The results of the
experiments are presented in Figure 9 and show that our transfer learning
approach speeds up the learning process while archiving similar performance as
when learning the task from scratch.

Figure 9: Transfer learning performance. Showing how policies learned during
merging can be transferred to the exiting environment to speed up the learning
process while archiving similar performance as when learning the task from
scratch.

####  V-B4 Safety

Finally, we compared state of the art architectures related to our approach
[61, 2, 3, 18] in terms of safety and efficiency to validate H3. We trained
the different architectures in the same situations and examined their
performance under different levels of HVs behaviors. As noted in Table IV our
safe altruistic agents consistently outperformed the other approaches, and the
results are more notable when the level of aggressiveness is higher. We
conclude that when using the safety prioritizer, immediate collisions are
avoided reducing the overall number of crashes in the episodes.

TABLE IV: Architectures’ performance comparison. Our safe altruistic AVs
outperformed the others approaches.

| Aggressive HVs | Moderate HVs | Conservative HVs
---|---|---|---
Approaches | C (%) | MF (%) | DT (m) | C (%) | MF (%) | DT (m) | C (%) | MF (%) | DT (m)
Conv2D+DQN [61] | 31.2 | 28.9 | 316 | 25.4 | 20.3 | 302 | 14.0 | 7.9 | 274
Toghi  _et al._ [2] | 21.3 | 16.4 | 339 | 12.7 | 10.1 | 333 | 1.6 | 0.6 | 269
Conv3D+A2C [18] | 14.8 | 12.6 | 341 | 9.4 | 8.8 | 328 | 1.1 | 0.1 | 267
Conv3D+DQN [3] | 3.1 | 2.8 | 359 | 2.6 | 2.4 | 341 | 0.3 | 0 | 284
Ours | 0.2 | 0.1 | 397 | 0.1 | 0.1 | 354 | 0 | 0 |  281

C: _Crashed_ , MF: _Mission Failed_ , DT: _Distance Traveled_

##  VI Conclusion and Future Work

We study the problem of multi-agent maneuver-level decision-making in mixed-
autonomy environments and investigate how AVs can learn cooperative policies
that are robust to different scenarios and driver behaviors safely. Our
altruistic AVs learn the decision-making process from experience, considering
the interests of all vehicles while prioritizing safety and optimizing a
general decentralized social utility function. We expose the settings for our
MARL problem in which transfer learning and domain adaptation are more
feasible, and conducted a sensitivity analysis under different HVs’ behaviors.
Our safe altruistic AVs learn to coordinate and influence the behavior of HVs
with socially advantageous results in diverse situations.

Limitations and Future Work.  While we explored different aspects of social
navigation in various environments and in the presence of diverse HVs
behaviors, the HV models are not learned from real human drivers’ data and the
traffic scenarios are limited to merging and exiting. Nevertheless, we
speculate that our approach could be effective in realistic traffic situations
by utilizing and learning from real human data and traffic scenarios.
Additionally, extra emphasis is required on safety for this approach to be
utilized in the real-world scenarios.

In future work, we plan to investigate more sophisticated architecture and
state representations, as well as develop a more realistic simulation
environment that incorporates data from real-world traffic and can handle more
complex interactions between HVs and AVs and diverse traffic agents such as
bicycles or pedestrians. Despite the limitations, we are thrilled to see safe
and robust social AVs on the road that learn from experience. We also
anticipate applications of these ideas beyond driving, to general MA humans-
robot interactions in which agents influence humans and cooperate safely for a
socially advantageous outcome.

## References

  * [1] W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus, “Social behavior for autonomous vehicles,” _Proceedings of the National Academy of Sciences_ , vol. 116, no. 50, pp. 24 972–24 978, 2019.
  * [2] B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Social coordination and altruism in autonomous driving,” _arXiv preprint arXiv:2107.00200_ , 2021.
  * [3] ——, “Cooperative autonomous vehicles that sympathize with human drivers,” _arXiv preprint arXiv:2107.00898_ , 2021.
  * [4] D. Sadigh, N. Landolfi, S. S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state,” _Autonomous Robots_ , vol. 42, no. 7, pp. 1405–1426, 2018.
  * [5] G. Shah, M. Saifuddin, Y. P. Fallah, and S. D. Gupta, “Rve-cv2x: A scalable emulation framework for real-time evaluation of cv2x-based connected vehicle applications,” in _2020 IEEE Vehicular Networking Conference (VNC)_. IEEE, 2020, pp. 1–8.
  * [6] B. Toghi, M. Saifuddin, H. N. Mahjoub, M. Mughal, Y. P. Fallah, J. Rao, and S. Das, “Multiple access in cellular v2x: Performance analysis in highly congested vehicular networks,” in _2018 IEEE Vehicular Networking Conference (VNC)_. IEEE, 2018, pp. 1–8.
  * [7] B. Toghi, M. Saifuddin, Y. P. Fallah, and M. Mughal, “Analysis of distributed congestion control in cellular vehicle-to-everything networks,” in _2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall)_. IEEE, 2019, pp. 1–7.
  * [8] B. Toghi, M. Saifuddin, M. Mughal, and Y. P. Fallah, “Spatio-temporal dynamics of cellular v2x communication in dense vehicular networks,” in _2019 IEEE 2nd Connected and Automated Vehicles Symposium (CAVS)_. IEEE, 2019, pp. 1–5.
  * [9] G. Shah, R. Valiente, N. Gupta, S. O. Gani, B. Toghi, Y. P. Fallah, and S. D. Gupta, “Real-time hardware-in-the-loop emulation framework for dsrc-based connected vehicle applications,” in _2019 IEEE 2nd Connected and Automated Vehicles Symposium (CAVS)_. IEEE, 2019, pp. 1–6.
  * [10] M. Saifuddin, M. Zaman, B. Toghi, Y. P. Fallah, and J. Rao, “Performance analysis of cellular-v2x with adaptive & selective power control,” in _2020 IEEE 3rd Connected and Automated Vehicles Symposium (CAVS)_. IEEE, 2020, pp. 1–7.
  * [11] M. Razzaghpour, S. Shahram, R. Valiente, and Y. P. Fallah, “Impact of communication loss on mpc based cooperative adaptive cruise control and platooning,” _arXiv preprint arXiv:2106.09094_ , 2021.
  * [12] S. Aoki, T. Higuchi, and O. Altintas, “Cooperative perception with deep reinforcement learning for connected vehicles,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 328–334.
  * [13] F. Sagberg, Selpi, G. F. Bianchi Piccinini, and J. Engström, “A review of research on driving styles and road safety,” _Human factors_ , vol. 57, no. 7, pp. 1248–1275, 2015.
  * [14] P. B. Harris, J. M. Houston, J. A. Vazquez, J. A. Smither, A. Harms, J. A. Dahlke, and D. A. Sachau, “The prosocial and aggressive driving inventory (padi): A self-report measure of safe and unsafe driving behaviors,” _Accident Analysis & Prevention_, vol. 72, pp. 1–8, 2014.
  * [15] E. F. Vallières, R. J. Vallerand, J. Bergeron, and P. McDuff, “Intentionality, anger, coping, and ego defensiveness in reactive aggressive driving,” _Journal of Applied Social Psychology_ , vol. 44, no. 5, pp. 354–363, 2014.
  * [16] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for autonomous cars that leverage effects on human actions.” in _Robotics: Science and Systems_ , vol. 2. Ann Arbor, MI, USA, 2016\.
  * [17] C. Wu, A. Kreidieh, E. Vinitsky, and A. M. Bayen, “Emergent behaviors in mixed-autonomy traffic,” in _Conference on Robot Learning_. PMLR, 2017, pp. 398–407.
  * [18] B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Altruistic maneuver planning for cooperative autonomous vehicles using multi-agent advantage actor-critic,” _arXiv preprint arXiv:2107.05664_ , 2021.
  * [19] J. Rios-Torres and A. A. Malikopoulos, “A survey on the coordination of connected and automated vehicles at intersections and merging at highway on-ramps,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 5, pp. 1066–1077, 2016.
  * [20] H. N. Mahjoub, B. Toghi, and Y. P. Fallah, “A stochastic hybrid framework for driver behavior modeling based on hierarchical dirichlet process,” in _2018 IEEE 88th Vehicular Technology Conference (VTC-Fall)_ , 2018, pp. 1–5.
  * [21] ——, “A driver behavior modeling structure based on non-parametric bayesian stochastic hybrid architecture,” in _2018 IEEE 88th Vehicular Technology Conference (VTC-Fall)_ , 2018, pp. 1–5.
  * [22] H. N. Mahjoub, B. Toghi, S. O. Gani, and Y. P. Fallah, “V2x system architecture utilizing hybrid gaussian process-based model structures,” in _2019 IEEE International Systems Conference (SysCon)_. IEEE, 2019, pp. 1–7.
  * [23] Z. Li, U. Kalabić, and T. Chu, “Safe reinforcement learning: Learning with supervision using a constraint-admissible set,” in _2018 Annual American Control Conference (ACC)_. IEEE, 2018, pp. 6390–6395.
  * [24] A. Cosgun, L. Ma, J. Chiu, J. Huang, M. Demir, A. M. Anon, T. Lian, H. Tafish, and S. Al-Stouhi, “Towards full automated drive in urban environments: A demonstration in gomentum station, california,” in _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2017, pp. 1811–1818.
  * [25] R. Valiente, A. Raftari, M. Zaman, Y. P. Fallah, and S. Mahmud, “Dynamic object map based architecture for robust cvs systems,” SAE Technical Paper, Tech. Rep., 2020.
  * [26] R. Valiente, M. Zaman, S. Ozer, and Y. P. Fallah, “Controlling steering angle for cooperative self-driving vehicles utilizing cnn and lstm-based deep networks,” in _2019 IEEE intelligent vehicles symposium (IV)_. IEEE, 2019, pp. 2423–2428.
  * [27] R. Valiente, M. Zaman, Y. P. Fallah, and S. Ozer, “Connected and autonomous vehicles in the deep learning era: A case study on computer-guided steering,” in _Handbook Of Pattern Recognition And Computer Vision_. World Scientific, 2020, pp. 365–384.
  * [28] B. Toghi, D. Grover, M. Razzaghpour, R. Jain, R. Valiente, M. Zaman, G. Shah, and Y. P. Fallah, “A maneuver-based urban driving dataset and model for cooperative vehicle applications,” 2020.
  * [29] B. Toghi, R. Valiente, R. Pedarsani, and Y. P. Fallah, “Towards learning generalizable driving policies from restricted latent representations,” _arXiv preprint arXiv:2111.03688_ , 2021.
  * [30] P. Hernandez-Leal, M. Kaisers, T. Baarslag, and E. M. de Cote, “A survey of learning in multiagent environments: Dealing with non-stationarity,” _arXiv preprint arXiv:1707.09183_ , 2017.
  * [31] T. Chu, J. Wang, L. Codecà, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 3, pp. 1086–1095, 2019.
  * [32] I. Arel, C. Liu, T. Urbanik, and A. G. Kohls, “Reinforcement learning-based multi-agent system for network traffic signal control,” _IET Intelligent Transport Systems_ , vol. 4, no. 2, pp. 128–135, 2010.
  * [33] A. OroojlooyJadid and D. Hajinezhad, “A review of cooperative multi-agent deep reinforcement learning,” _arXiv preprint arXiv:1908.03963_ , 2019.
  * [34] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [35] J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” in _International Conference on Autonomous Agents and Multiagent Systems_. Springer, 2017, pp. 66–83.
  * [36] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _arXiv preprint arXiv:1706.02275_ , 2017.
  * [37] A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh, “Learning latent representations to influence multi-agent interaction,” in _Proceedings of the 4th Conference on Robot Learning (CoRL)_ , November 2020.
  * [38] A. Shih, A. Sawhney, J. Kondic, S. Ermon, and D. Sadigh, “On the critical role of conventions in adaptive human-ai collaboration,” in _9th International Conference on Learning Representations (ICLR)_ , 2021.
  * [39] E. Vinitsky, R. Köster, J. P. Agapiou, E. Duéñez-Guzmán, A. S. Vezhnevets, and J. Z. Leibo, “A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings,” _arXiv preprint arXiv:2106.09012_ , 2021.
  * [40] A. Pokle, R. Martín-Martín, P. Goebel, V. Chow, H. M. Ewald, J. Yang, Z. Wang, A. Sadeghian, D. Sadigh, S. Savarese _et al._ , “Deep local trajectory replanning and control for robot navigation,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 5815–5822.
  * [41] M. Lauer and M. Riedmiller, “An algorithm for distributed reinforcement learning in cooperative multi-agent systems,” in _In Proceedings of the Seventeenth International Conference on Machine Learning_. Citeseer, 2000.
  * [42] S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and J. Vian, “Deep decentralized multi-task multi-agent reinforcement learning under partial observability,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2681–2690.
  * [43] K. Brown, K. Driggs-Campbell, and M. J. Kochenderfer, “A taxonomy and review of algorithms for modeling and predicting human driver behavior. arxiv e-prints, article,” _arXiv preprint arXiv:2006.08832_ , 2020.
  * [44] B. Ivanovic, E. Schmerling, K. Leung, and M. Pavone, “Generative modeling of multimodal multi-human behavior. in 2018 ieee,” in _RSJ International Conference on Intelligent Robots and Systems_ , pp. 3088–3095.
  * [45] H. N. Mahjoub, A. Raftari, R. Valiente, Y. P. Fallah, and S. K. Mahmud, “Representing realistic human driver behaviors using a finite size gaussian process kernel bank,” in _2019 IEEE Vehicular Networking Conference (VNC)_. IEEE, 2019, pp. 1–8.
  * [46] K. H. Beck, B. Ali, and S. B. Daughters, “Distress tolerance as a predictor of risky and aggressive driving,” _Traffic injury prevention_ , vol. 15, no. 4, pp. 349–354, 2014.
  * [47] R. Chandra, U. Bhattacharya, T. Mittal, A. Bera, and D. Manocha, “Cmetric: A driving behavior measure using centrality functions,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2035–2042.
  * [48] Z. Constantinescu, C. Marinoiu, and M. Vladoiu, “Driving style analysis using data mining techniques,” _International Journal of Computers Communications & Control_, vol. 5, no. 5, pp. 654–663, 2010.
  * [49] M. Kuderer, S. Gulati, and W. Burgard, “Learning driving styles for autonomous vehicles from demonstration,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2641–2646.
  * [50] D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooperative inverse reinforcement learning,” _Advances in neural information processing systems_ , vol. 29, pp. 3909–3917, 2016.
  * [51] C. Wu, A. M. Bayen, and A. Mehta, “Stabilizing traffic with autonomous vehicles,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 6012–6018.
  * [52] E. Vinitsky, N. Lichtle, K. Parvate, and A. Bayen, “Optimizing mixed autonomy traffic flow with decentralized autonomous vehicles and multi-agent rl,” _arXiv preprint arXiv:2011.00120_ , 2020.
  * [53] D. A. Lazar, E. Bıyık, D. Sadigh, and R. Pedarsani, “Learning how to dynamically route autonomous vehicles on shared roads,” _Transportation Research Part C: Emerging Technologies_ , vol. 130, p. 103258, 2021.
  * [54] E. Bıyık, D. Lazar, R. Pedarsani, and D. Sadigh, “Altruistic autonomy: Beating congestion on shared roads,” _arXiv preprint arXiv:1810.11978_ , 2018\.
  * [55] W. Z. Wang, M. Beliaev, E. Biyik, D. A. Lazar, R. Pedarsani, and D. Sadigh, “Emergent prosociality in multi-agent games through gifting,” in _30th International Joint Conference on Artificial Intelligence (IJCAI)_ , 2021.
  * [56] J. Wang, Q. Zhang, D. Zhao, and Y. Chen, “Lane change decision-making through deep reinforcement learning with rule-based constraints,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–6.
  * [57] S. Nageshrao, H. E. Tseng, and D. Filev, “Autonomous highway driving using deep reinforcement learning,” in _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_. IEEE, 2019, pp. 2326–2331.
  * [58] C.-J. Hoel, K. Driggs-Campbell, K. Wolff, L. Laine, and M. J. Kochenderfer, “Combining planning and deep reinforcement learning in tactical decision making for autonomous driving,” _IEEE transactions on intelligent vehicles_ , vol. 5, no. 2, pp. 294–305, 2019.
  * [59] A. Mohammadhasani, H. Mehrivash, A. Lynch, and Z. Shu, “Reinforcement learning based safe decision making for highway autonomous driving,” _arXiv preprint arXiv:2105.06517_ , 2021.
  * [60] D. Chen, Z. Li, Y. Wang, L. Jiang, and Y. Wang, “Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic,” _arXiv preprint arXiv:2105.05701_ , 2021.
  * [61] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.
  * [62] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” _arXiv preprint arXiv:1511.05952_ , 2015.
  * [63] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” _Physical review E_ , vol. 62, no. 2, p. 1805, 2000.
  * [64] A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,” _Transportation Research Record_ , vol. 1999, no. 1, pp. 86–94, 2007.
  * [65] E. Leurent, Y. Blanco, D. Efimov, and O.-A. Maillard, “Approximate robust control of uncertain dynamical systems,” _arXiv preprint arXiv:1903.00220_ , 2019.

|  Rodolfo Valiente is a Ph.D. candidate in Computer Engineering at the
University of Central Florida (UCF). His research interests include connected
autonomous vehicles (CAVs), reinforcement learning, computer vision, and deep
learning with a focus on the autonomous driving problem. He received a M.Sc.
degree from the University of Sao Paulo (USP) and his B.Sc. degree from the
Technological University Jose Antonio Echeverria.
---|---
|  Behrad Toghi is a Ph.D. candidate at the University of Central Florida. He
received the B.Sc. degree in electrical engineering from Sharif University of
Technology in 2016 and has worked as a research intern at Honda Research
Institute, Mercedes-Benz R&D North America and Ford Motor Company R&D between
2018 and 2021. His work is mainly in the domains of prediction & behavior
planning for autonomous driving.
---|---
|  Ramtin Pedarsani is an Assistant Professor in the ECE Department at the
University of California, Santa Barbara. He received the B.Sc. degree in
electrical engineering from the University of Tehran in 2009, the M.Sc. degree
in communication systems from the Swiss Federal Institute of Technology (EPFL)
in 2011, and his Ph.D. from the University of California, Berkeley, in 2015.
His research interests include networks, game theory, machine learning, and
transportation systems.
---|---
|  Yaser P. Fallah is an Associate Professor in the ECE Department at the
University of Central Florida. He received the Ph.D. degree from the
University of British Columbia, Vancouver, BC, Canada, in 2007. From 2008 to
2011, he was a Research Scientist with the Institute of Transportation
Studies, University of California Berkeley, Berkeley, CA, USA. His research,
sponsored by industry, USDoT, and NSF, is focused on intelligent
transportation systems and automated and networked vehicle safety systems.
---|---