# Learning Driver Models for Automated Vehicles via Knowledge Sharing and
Personalization

Wissam Kontar § Xinzhi Zhong § Soyoung Ahn §,∗
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>§Department of Civil and
Environmental Engineering
University of Wisconsin-Madison, USA

###### Abstract

This paper describes a framework for learning Automated Vehicles (AVs) driver
models via knowledge sharing between vehicles and personalization. The innate
variability in the transportation system makes it exceptionally challenging to
expose AVs to all possible driving scenarios during empirical experimentation
or testing. Consequently, AVs could be blind to certain encounters that are
deemed detrimental to their safe and efficient operation. It is then critical
to share knowledge across AVs that increase exposure to driving scenarios
occurring in the real world. This paper explores a method to collaboratively
train a driver model by sharing knowledge and borrowing strength across
vehicles while retaining a personalized model tailored to the vehicle’s unique
conditions and properties. Our model brings a federated learning approach to
collaborate between multiple vehicles while circumventing the need to share
raw data between them. We showcase our method’s performance in experimental
simulations. Such an approach to learning finds several applications across
transportation engineering including intelligent transportation systems,
traffic management, and vehicle-to-vehicle communication.
Code and sample dataset are made available at the project page
https://github.com/wissamkontar.

##  1 Introduction

The curse of variability stands as a critical barrier in the development and
deployment of Automated Vehicles (AVs) in the open world. Variability in the
open world stems from the innate dynamic, stochastic, and unpredictable nature
of the transportation system. The driving task can change significantly
depending on the traffic state (congestion, free flow, etc.), weather
conditions (foggy, snowing, etc.), and roadway design (divided highway, one-
way, etc.). It even depends on what other agents (pedestrians, bikers, busses,
etc.) are present. Human drivers adapt and respond to these variations
instinctively, but replicating such situational knowledge for an AV to
maneuver safely in any given scenario, is extremely challenging in light of
limited data availability and difficulty of real-world experimentation.
Additionally, AVs are designed with some desired performance in mind. This
means that AV’s data can exhibit significant heterogeneity. This creates
another challenge in learning driver models, as one needs to decouple the
unique and common features from AV data to create personalized driver models
for each vehicle.

In a recent release of large-scale AV dataset from Waymo [3] we see limited
variability in the testing environment. For instance, the Waymo open dataset
shows the majority of trip logs in sunny (99.3%), daytime (80.6%), and urban
scenarios [5]. Another large dataset from nuScenes includes more diverse
driving environments and scenarios as some driving logs come from different
cities (Boston and Singapore) and locations (urban, residential, and
industrial), and does include some rainy and cloudy weather conditions.
However, most open world experiments and data collection (e.g., Cruise, Lyft,
Aurora) are being done in dedicated routes and locations with limited exposure
to variability in the transportation system. Dedicated experiments in similar
driving scenarios allow for a deeper understanding of the AV performance in
certain scenarios. However, a breadth of exposure is critically needed to
train on extensive scenarios and encounters.

The constraint on data availability is compounded by the difficulty of
conducting real-world experiments. The development and testing of AVs require
extensive resources and come with inherent safety concerns. Consequently, we
see many experiments with exposure to limited scenarios, likely leaving AVs
blind to a wide array of encounters. It is thus critical to develop methods to
share knowledge across AVs to increase exposure to a wide range of scenarios
occurring in the open world.

Recent literature has also uncovered how AVs can exhibit different behaviors
on the road based on the underlying design and control logic of these
vehicles. For instance, we show in [11, 10], how the car-following (CF)
behavior of an AV exhibits a range of behavior depending on the underlying
control parameters on spacing, desired speed, and acceleration constraints.
Thus, data from AVs –specifically multi-class AVs (i.e., those designed with
different performances in mind)– is extremely heterogeneous. This
heterogeneity is in fact a blessing rather than a curse since it gives us the
opportunity to train a tailored model for each vehicle that considers its own
unique characteristics. Such a process is referred to as personalization
hereafter.

This work is concerned with developing a framework through which different AVs
can share knowledge from their encounters and borrow strength from each other,
yet retain a personalized model tailored to their conditions and unique
properties. A driver model here refers to a model capable of safely and
effectively maneuvering the AV in various driving scenarios. This driver model
controls the vehicle’s perception, decision-making (on acceleration, braking,
and steering), and navigation. However, the main scope of this work is in
presenting how the sharing of information (through parameter transfer) while
retaining a personalized model is done, and not on designing an optimal driver
model.

###  1.1 Motivation

One can question the need to share knowledge through collaboration between
vehicles as opposed to just pooling the data and learning one model. We argue
that learning one-size-fits all can lead to misleading results, and dilutes
the distinction between (i) different experienced driving scenarios, and (ii)
heterogeneous driving behavior from subject vehicles.

For instance, consider the basic unit of a driver model - the car following
(CF) model - whereby an AV uses its sensor data to regulate its own
acceleration/speed such that it follows its leader in a safe manner. The
design of such a CF controller for an AV requires specific consideration of
desired spacing, speed preference, and comfortable acceleration. We can see
such a model in play when we consider the widely adopted linear controller in
Adaptive Cruise Control (ACC) and even full self driving systems [17, 13, 11].
In a linear controller, the desired spacing is first modeled based on the
constant time gap policy as

$d_{v}^{*}(t)=v_{v}(t)\times\tau_{v}^{*}+\delta_{v}^{*}$ (1)

where $d_{v}^{*}(t)$ is the desired equilibrium spacing of vehicle $v$ at any
time $t$; $v_{v}(t)$ is the respective speed of vehicle v; $\tau_{v}^{*}$ is
the constant time gap; and $\delta_{v}^{*}$ is the standstill distance.
Accordingly, the deviation from the equilibrium spacing can be written as
$\Delta d_{v}(t)=d_{v}(t)-d_{v}^{*}(t)$, where $d_{v}(t)$ represents the
actual spacing between vehicle $v$ and its leader ($v-1$) at time $t$, and the
speed difference between vehicle $v$ and its leader ($v-1$) is $\Delta
v_{v}(t)=v_{v-1}(t)-v_{v}(t)$.

As such, a system state for the AV controller can be described by
$\bm{x}_{v}(t)=[\Delta d_{v}(t),\Delta v_{v}(t),a_{v}(t)]^{T}$ and input state
as $u_{v}(t)$

$\displaystyle u_{v}(t)=\bm{K}_{v}^{T}\bm{x}_{v}(t)$ (2)
$\displaystyle\bm{K}_{v}^{T}=[k_{sv},k_{vv},k_{av}]$

where $k_{sv}$, $k_{vv}$, $k_{av}$ are the feedback gains for the deviation
from equilibrium spacing ($\Delta d_{v}(t)$), speed difference ($\Delta
v_{v}(t)$) and acceleration ($a_{v}(t)$), respectively. The parameter setting
of $\bm{K}_{v}^{T}$ denotes the regulation magnitude for each component in the
system state $\bm{x}_{v}(t)$, and thus regulates the AV behavior. We refer
readers to [11] for an in-depth analysis on this phenomenon.

It follows that one can design an AV with specific consideration of
$d_{v}^{*}$, and $\bm{K}_{v}$ in mind. The dichotomy of these parameters is
that they are (i) influenced by designer preference (or rider preference as
shown in time headway parameter $\tau_{v}^{*}$ in commercial ACC), and (ii)
influenced by the driving environment. The authors have a prior work that
analyzes in depth how an AV can be designed with specific performance in mind,
see [11]. However, if data from heterogeneous AVs (i.e., AVs that differ in
design and desired performance) is pooled together to train a single driver
model, heterogeneity is lost since the training aims to achieve generalization
and the underlying assumption is that data is homogeneous. One may work with
stochastic models by estimating parameter distributions to regain some
heterogeneity in the data. However, personalized models for each AV still
cannot be tracked when the data are pooled. Accordingly, personalizing desired
speed, headway, and spacing is unenforceable.

It is important to note that in some cases data pooling is not even
achievable. Commercial AV data can be protected by propriety rights and
privacy concerns. Thus accessing raw data for training purposes may not be
available.

Accordingly, this work is motivated by the need to (i) share driving knowledge
between different vehicles to increase the exposure of an AV to different
driving scenarios/environments, (ii) retain a personalized model for each
vehicle under heterogeneous behavior, and (iii) bypass the need to access raw
data for training driving models.

We follow the motivation discussed here with a simulation study (presented in
Section 3.3) that shows how pooling data is not ideal under heterogeneity and
how our approach can tackle this problem.

###  1.2 Related work

With the advent increase of computational power and sheer amount of data
collected in today’s systems, federated learning became a powerful tool with
the intent of _processing the data where it was created on the edge_. The edge
here refers to single device, vehicle, or the like. As a consequence,
traditional Internet of Things (IoT) applications have shifted to a
decentralized approach termed Internet of Federated Things (IoFT) [9]. This
brought about multiple analytical and computational tools that define how
devices are set to collaborate with each other and share information. One of
the early forefront tools in federated data analytics is the federated
averaging (FedAvg), which was tailored for deep learning applications [12].
The idea here is simple; devices in a network structure would collaborate to
learn a global deep learning model with the coordination of the central sever.
Local devices perform iterations of stochastic gradient descent (SGD) using
their data to obtain local parameters of their deep learning model, and send
those parameters to a central server. Then the central server takes an average
of those parameters to update the global model. Since then, several works have
refined federated data analytics and tailored it to different applications.
Notably, [19, 20] scales the application of federated learning into Gaussian
processes, and general linear models. [16] presents Fed-ensemble, bringing
ensemble methods to federated learning, improving generalization and
uncertainty quantification. [15] tailors a federated algorithm to learn unique
and shared features for principal component analysis. And several other models
exist, which are reviewed here [9, 18, 1].

While the application of federated learning in transportation system has seen
some momentum, it has yet to expand. Recently, [2] presented a survey review
of federated learning for connected and automated vehicles. They note several
applications of federated learning for in-vehicle human monitoring, steering
wheel prediction, object detection, motion control, and vehicle trajectory
prediction. Most relevant to this work are the ones related to vehicle
trajectory prediction. For example, [4] uses an encrypted federated network
algorithm to learn driver behavior and predict trajectories. [14] designs a
federated deep reinforcement learning for trajectory planning. However, only
few efforts have been invested with limited scope, and the application domain
of vehicle trajectory prediction via federated learning remains largely
unexplored.

The focus of this paper is substantially different from available work and
explores the potential of federated learning in a different direction.
Specifically, what we aim to do is to share knowledge across vehicles to learn
driving scenarios that might have been missed due to the variability in the
transportation system, while also retaining personalization for each vehicle.

###  1.3 Main contribution

We summarize our contributions in the following:

  * •

Modeling: In our model, we acknowledge that data coming from each vehicle in
uniquely heterogeneous given variability of driving scenarios, and personal
preferences, and thus learning one-size-fits-all model is not ideal. Instead
our model is personalized to encode unique encounter data and allows for
transferring knowledge of unseen driving scenarios from one-vehicle to
another.

  * •

Algorithm: We showcase a training algorithm based on federated learning, where
vehicles only need to share iterations of personalized model parameters thus
preserving privacy and minimizing communication cost.

  * •

Application: We present an application of such framework on learning traffic
oscillations by knowledge transfer between three vehicles, and another
application on training personalized models under heterogeneity.

The rest of the paper is organized as follows: Section 2 details our
methodology. In Section 3 we introduce and analyze two simulation studies for
knowledge sharing and personalization under heterogeneity in behavior.
Finally, Section 4 concludes.

##  2 Methodology

In this section we discuss the problem setting and the model formulation.

###  2.1 Problem Setting

Consider the problem setting illustrated in Fig. 1. Multiple vehicles are
being tested in different driving environments, and each has a different
personalized driver model. All of the driving encounters experienced by the
vehicles are of interest to us, as the ultimate goal is to design a global
driving model capable of maneuvering the vehicle in different driving
environments. In perspective, to build such a model, one can extract the data
for each vehicle alone, then train a local (i.e., using the vehicle’s own
data) driving model. This approach yields a single driving model for each
vehicle that is blind to some driving encounters observed by other vehicles
(due to different driving environments).

Figure 1: Problem Illustration

What we seek to accomplish here is to allow vehicles to collaboratively train
a global driving model, that allows for (i) a personalized model that focuses
on their own local data, and (ii) knowledge sharing by transfer of information
from one vehicle to another. The underlying assumption here is that data from
each vehicle is uniquely heterogeneous. This heterogeneity is due to the
unique encounter the vehicle is exposed to and its personalized design (e.g.,
specific desired speed, acceleration constraints, etc.). To achieve this we
structure our driving model learning process as shown in Fig. 2.

Figure 2: Learning Structure

###  2.2 Problem Formulation

Suppose there exist $v\geq 2$ vehicles. Each vehicle $v\in[V]:=\\{1,...,V\\}$
has local dataset expressed as $D_{v}=\\{\bm{X_{v}},\bm{y_{v}}\\}$ with the
cardinality of $N_{v}$. We have an output $\bm{y}_{v}=[y_{1},...,y_{N}]^{T}$,
and an input $\bm{X}_{v}=[x_{1}^{T},...,x_{N}^{T}]$. Here, $\bm{X}_{v}$
represents the state of the subject vehicle and nearby vehicles, usually a
vector of data informed from on-board sensors. For instance, the state data
can be relative speed between subject vehicle and leading vehicle, spacing,
acceleration, jerk, and others. $\bm{y}_{v}$ represents an action by the AV,
as acceleration magnitude for example. Formally, learning a driver model
$f_{v}$ given training data $\bm{X}_{v}$ and $\bm{y}_{v}$ is a function
defined as:

$y_{v}(x_{v})=f_{v}(x_{v};\theta_{v})$ (3)

where $f(*)$ can hold any functional form. Most notably, $f(*)$ can be a
general deep learning network, Gaussian process, reinforcement learning,
control model, or a linear model. The main interest here is parameter
$\theta_{v}$, which parameterize $f(*)$ based on training data. The training
data are ultimately tied to driving scenarios that vehicle $v$ was exposed to.
One can notice that predicting an accurate output $f(x_{v}^{*}:\theta_{v})$
ultimately depends on accurate estimation of $\theta_{v}$. Training $f(*)$
entails minimizing a general loss function,
$\mathcal{L}(\theta_{v};x_{v};y_{v})$. There exist several optimizers to solve
the minimization problem of $\mathcal{L}$ (e.g., Adams [8], Stochastic
Gradient Descent (SGD), etc.). However, most widely adopted is the SGD
approach as it offers generalization power and is extremely efficient [7]. In
typical fashion in SGD, the model training is performed in successive
iterations. At each iteration of training $t$, a subset of training data
(indexed by $\xi$) $\bm{X}_{v\xi}$ and $\bm{y}_{v\xi}$ is taken to update
model parameters according to the below:

$\theta_{v}^{(t+1)}\leftarrow\theta_{v}^{(t)}-\eta^{(t)}g_{v}(\theta_{v}^{(t)};\xi^{(t)})$
(4)

where $\eta^{(t)}$ is the learning rate and $g_{v}$ is the stochastic gradient
of the loss function $\mathcal{L}$. The outcome is a driving model
$f(x_{v};\theta_{v})$ parameterized by optimal $\theta_{v}$.

###  2.3 Knowledge Sharing through Federated Learning

The above is a general approach to learning a data-driven driver model, based
on local data. However, in our approach we do not want to solely rely on
vehicle $v$’s own data that is tied to the driving scenarios it was exposed
to, but want to integrate knowledge from other AVs that might have encountered
different driving scenarios. As such, we present a collaborative learning
scheme that aims at learning a global parameter $\hat{\theta}$ that minimizes
a global loss function defined as:

$\mathcal{L}(\hat{\theta}):=\sum^{V}_{v=1}{\alpha_{v}{\mathcal{L}(\theta_{v};D_{v})}}$
(5)

where $\alpha_{v}$ is the weight parameter for vehicle $v$ in the
collaborative training scheme.
$\alpha_{v}=\frac{N_{v}}{\sum_{v=1}^{V}{N_{v}}}$, such that
$\sum_{v=1}^{V}{\alpha_{v}}=1$. Consequently, at each communication round,
each vehicle runs steps of SGD to estimate its local parameters:

$\theta^{(t+1)}_{v}\leftarrow\theta^{(t)}_{v}-\eta^{(t)}g_{v}(\theta^{(t)}_{v};\xi^{(t)}_{v})$
(6)

Afterwards, the global coordinator aggregates the model parameters
$\hat{\theta}$ according to the below rule and sends back $\hat{\theta}$ to
each vehicle. Essentially, the global coordinator here averages out the local
parameters coming out from each individual vehicle’s driver model.

$\hat{\theta}=\sum_{v=1}^{V}{\alpha_{v}\theta_{v}}$ (7)

In this scheme, we have all vehicles participating during the training rounds
to send their local parameters to the global coordinator. Note that an
underlying assumption is that all vehicles have the same functional form
$f(*)$ of the driver model. Thereafter, the global coordinator sends back the
updated parameters to each vehicle. Knowledge from each vehicle’s driver model
is shifted around to all collaborating vehicles. We summarize the above in
Algorithms 1 & 2.

1 Data: number of vehicles $V$, their initial model parameters $\theta$,
number of sharing rounds $s\in S$, and weight parameters $\alpha_{v}$

2 for  _$s=1:S$_ do

3 Global coordinator broadcasts $\theta$;

4 for  _$v\in V$_ do

5 $\theta^{(0)}_{v}=\theta$ ;

6 Perform SGD (Algorithm 2) to update local vehicle parameters;

7 end for

8  _Aggregation:_ $\hat{\theta_{s}}=\sum_{v=1}^{V}{\alpha_{v}\theta_{v}}$ ;

9 Set $\theta=\hat{\theta_{s}}$;

10 end for

Return $\hat{\theta_{S}}$

Algorithm 1 Knowledge Sharing through Federation (Learning a Global Model)

1 Data: index $v$, learning rate $\eta$, number of local updates $U$, and
vehicle model parameters $\theta_{v}^{(0)}$

2 for  _$t=1:U$_ do

3 Perform random sampling to get mini-batch data $D_{v}$ and index it
$\xi_{v}^{(t)}$;

4
$\theta^{(t+1)}_{v}\leftarrow\theta^{(t)}_{v}-\eta^{(t)}g_{v}(\theta^{(t)}_{v};\xi^{(t)}_{v})$

5 end for

Return $\theta_{v}^{(U)}$

Algorithm 2 SGD for Local Vehicle Updates

###  2.4 Personalization

Consequently, knowledge sharing was achieved by learning a global model
parameterized by $\hat{\theta}$ and shared with all participating vehicles. It
follows that each vehicle needs to personalize $\hat{\theta}$ based on its own
local data. Through this process we encode heterogeneous behavior by each
vehicle that is described by its own design parameters. Such personalization
can be achieved in numerous ways. However the most commonly used is the
regularization concept, where each vehicle uses its own local data $D_{v}$ to
minimize a penalized least squares loss function defined as:

$min_{\theta_{personal}}\frac{1}{N_{v}}\sum_{n=1}^{N_{v}}\mathcal{L}(f(x_{v};\theta_{personal}),y_{v})-\omega||\hat{\theta}-\theta_{personal}||_{2}^{2}$

(8)

where $\omega$ is a positive coefficient, and $\hat{\theta}$ is the global
parameter learned from Algorithms 1 & 2. This approach personalized
$\theta_{personal}$ to each vehicle $v\in V$ while retaining global knowledge
by encouraging a solution close to $\hat{\theta}$. This regularization
encourages $\theta_{personal}$ not to drift to far away from $\hat{\theta}$,
and it shown statistically to reduce over fitting and the bias-variance trade
off. Accordingly, each vehicle re-runs Algorithm 2, with $g_{v}$ now based on
the regularized loss function defined in Eq. 8. Note that one does not
necessarily need to do as many iteration steps ($S$) as done in Algorithms 1 &
2 , but a fraction of the steps can suffice.

##  3 Simulation Analysis

We provide here a simulation experiment to demonstrate the applicability of
our approach in (i) collaborative knowledge sharing to learn un-encountered
driving scenarios, (ii) personalization of driving models under heterogeneity.

###  3.1 Simulation Setup

Consider that we have different AVs denoted by $v$. In here, we adopt a simple
driver model setup as the goal is not to analyze the prediction performance of
the underlying driver model, nor to design a driver model (several work in the
literature exist on that), but rather to see how much knowledge sharing and
personalization can be achieved through our proposed training structure.
Accordingly, we define our driver model as a car-following model that predicts
the AV speed (i.e., the output $y_{v}$) based on an input of leader speed
($x_{v}$). We use a Guassian Process Regression ($\mathcal{GP}$) to design
such driver model. Note that our framework (knowledge sharing and
personalization) works under any driver model, so users can replace the
$\mathcal{GP}$ with a model of choice. We formulate the driver model as
follows:

$\displaystyle f_{v}\sim\mathcal{GP}(m(\cdot),c(\cdot,\cdot))$ (9)
$\displaystyle
y_{v}=f(x_{v})+\epsilon,\epsilon^{i.i.d}\sim\mathcal{N}(0,\sigma_{\epsilon}^{2})$
(10)

where $x\in X$ is the input, and $m(\cdot):X\rightarrow\mathbb{R}$ is the
prior mean function, $c(\cdot,\cdot):X\times X\rightarrow\mathbb{R}$ is the
prior covariance function, and $\epsilon$ is the observational noise with
variance $\sigma_{\epsilon}$. We further consider the zero mean function, and
we assume the covariance function $c(\cdot,\cdot)=\sigma_{o}k(\cdot,\cdot)$
for a kernel function $k(\cdot,\cdot):X\times X\rightarrow\mathbb{R}$, where
$\sigma_{o}$ is the output variance. We adopt the well known Radial Basis
Function (RBF) as our kernel that is parameterized by the length scale $l$.
Accordingly, we denote
$\theta_{v}=(\sigma_{0},l,\sigma_{\epsilon})^{T}\in\mathbb{R}^{3}$ the
parameters to be estimated. To estimate $\theta_{v}$, we use SGD (Algorithm 2)
to minimize the scaled negative log marginal likelihood function defined as

$\mathcal{L}(\theta_{v},X_{v},y_{v})=\frac{-1}{n}logp(y_{v}|X_{v},\theta_{v})$
(11)

Accordingly, the collaboration here is based on
$\theta_{v}=(\sigma_{0},l,\sigma_{\epsilon})^{T}\in\mathbb{R}^{3}$. At every
sharing round $S$ (see Algorithm 1), each vehicle performs steps of SGD on its
$\mathcal{L}(\theta_{v},X_{v},y_{v})$ to output a local set of parameters
$\theta_{v}$ (see Algorithm 2). At the aggregation step, a global coordinator
then computes $\hat{\theta}$ based on the law described in Eq. 7, and sets
$\theta_{v}=\hat{\theta}$. This scheme of collaborative learning and sharing
of parameters is what encodes knowledge transfer between participating
vehicles. It is important to note that at no point during the collaboration,
raw data ($X_{v}$ and $y_{v}$) are shared between vehicles, which makes this
approach very computationally efficient and privacy aware. Additionally, the
parameters $\theta_{v}$ are dependent on the assumed functional form $f(*)$ of
the driver model. For instance, if one is using neural network predictor, then
$\theta_{v}$ would be the weights of the neural net.

Finally, each vehicle re-runs Algorithm 2 based on the regularized loss
function described in Eq. 8 to compute the final $\theta_{personalized}$ used
in the prediction of AV speed (i.e., predictions from the driver model).

###  3.2 Experiment 1: Knowledge Sharing to Learn Traffic Oscillations

Now we consider a specific experimental setup for which we want to demonstrate
how knowledge is transferred between vehicles. Consider now that we have three
automated vehicles ($V=3$). We assume each vehicle was operating under
completely different driving environment. To signify this, we consider three
different scenario-based datasets shown in Fig. 3: (i) vehicle 1 is operating
under constant speed, (ii) vehicle 2 experiences a deceleration maneuver, and
(iii) vehicle 3 experiences an acceleration maneuver. Note that these
experiments are extracted from Waymo dataset [6]. We use this dataset to give
realism to our experimental setup. Each of the three driving scenarios runs
for $19.7$-seconds long run at a $10Hz$ resolution. Consequently, each vehicle
has a local dataset of $N_{v}=197$. In Fig. 3 the top row shows the driving
scenario for each vehicle. The blue curve shows a human driver (leader)
oscillation, and the red curve shows the resulting response of AV follower
(the Waymo vehicle). Recall that our driver model is based on speed input and
output. Thus, training our model requires an input, $X_{v}$, of the leading
vehicle (blue curve) speed. And predicts an output speed $y_{v}$ of an AV (the
red curve).

(a) Vehicle 1: Constant speed scenario

(b) Vehicle 2: Deceleration scenario

(c) Vehicle 3: Acceleration scenario

Figure 3: Driving Scenarios Observed by each Vehicle

The three driving encounters in our setup (Fig. 3), represent portions of a
full traffic oscillation – constant speed followed by a deceleration and
acceleration maneuver to reach a constant speed again. Such oscillations are
ubiquitous in traffic systems and are certain to occur in the open world. The
goal here is to allow the three vehicles to train a driving model in a
collaborative fashion, in such a way that they would individually be able to
respond to a traffic oscillation knowing that none of them had seen such a
driving scenario. After collaborative training, we test each vehicle on the
full oscillation scenario, seen in Fig. 4. The full oscillation in Fig. 4,
represents an observed empirical oscillation created by a human driven vehicle
(HDV) – extracted from the Waymo dataset.

Figure 4: Testing Scenario: Full Oscillation

####  3.2.1 Simulation Results

After training the driver model with our approach that focuses on knowledge
sharing and personalization, we test against the oscillation shown in Fig. 4,
whereby the input is the leader speed, and output is the AV speed. The goal
here is to see whether each vehicle (1-2-3) is able to produce an oscillation.
Results are shown in Fig. 5. Interestingly, we see that “After Knowledge
Sharing” (blue curve) each of the vehicle was able to produce a traffic
oscillation even though none of the vehicles have full knowledge of an
oscillation (recall Fig. 3). This is not the case when we look at “Without
Knowledge Sharing” (red curve). However, it is notable that the oscillations
“After Knowledge Sharing” are not perfect. This is rather expected given the
limited training data, as the focus is not on building a driver model rather
on knowledge sharing. In practice, usually a driver model would have a complex
set of input based on speed of leader, position of leader, and multiple other
factors. But, success here is denoted as the ability of each vehicle (1-2-3)
to produce an AV response of an oscillation when subjected to one.

(a) Vehicle 1

(b) Vehicle 2

(c) Vehicle 3

Figure 5: Prediction Results of each Vehicle on the Testing Scenario (Full
Oscillation): Before and After Knowledge Sharing

The effect of knowledge sharing is specifically amplified when looking at
vehicle 1 profile in Fig. 5. Remember that vehicle 1 had only access to its
local data which is a constant speed profile. Specifically, it shows that the
driver model of vehicle 1 can extract knowledge on deceleration and
acceleration phases from vehicles 2 & 3, respectively. This gives it knowledge
to respond to an oscillation. On the contrary, when we train the driver models
of vehicles 1-2-3 without knowledge sharing, we notice that they can be blind
to some maneuvers (deceleration/acceleration) that inhibits their ability to
fully produce an oscillation, see red curves in Fig. 5.

###  3.3 Experiment 2: Knowledge Sharing and Personalization under
Heterogeneity

Now we revisit the problem of data pooling and heterogeneity in AV behavior
from the _Motivation_ (Section 1.1). We consider here two AVs, each having a
different design setup and thus driving behavior. Note that such design of AVs
is not uncommon in the real world, see [11].

Figure 6: Speed Profiles for Experiment 2 Setup. Note: Leader HDV (black) and
Aggressive AV (red) Curves Nearly Overlap each other

  * •

Aggressive AV: is an AV that is designed to be very responsive to the leader
speed and prioritizes the minimization of speed different $\Delta v_{v}(t)$.
Specifically, for the aggressive AV we set $\bm{K}_{v}=[0.01,10,-0.01]$, with
$\tau^{*}_{v}=0.5$, and $\delta_{v}^{*}=5m$.

  * •

Passive AV: is an AV that is designed to very passive to the leader speed and
prioritizes the minimization in the deviation from target spacing
$d_{v}^{*}(t)$. Specifically, we set $\bm{K}_{v}=[10,0.01,-0.01]$, with
$\tau^{*}_{v}=2.5$, and $\delta_{v}^{*}=7m$.

Further, we consider a specific leader (HDV) speed oscillation and then based
on the linear controller explained in Section 1.1 and for the settings
described above, we simulate an AV speed profile. This is shown in Fig. 6.
Note that the leader (HDV) profile is common between the two generated AV
profiles. One can directly notice the difference in behavior between the two
AVs. The aggressive AV (red) nearly masks its leader (black), while the
passive AV (green) is much less reactive. It follows then that the data (i.e.,
output $y_{v}$) generated from control design of these two AVs exhibit
heterogeneous features.

####  3.3.1 Simulation Results

When output data from these two heterogeneous vehicles is pooled together to
learn one global driving model, it masks the underlying difference in
behavior. This is further illustrated in Fig. 7. When we train a driver model
based on pooling the speed data from the aggressive and passive AVs, we get a
behavior shown in Fig. 7 (light blue color). The prediction fails to
distinguish between an aggressive and passive behavior as it tries to achieve
generalization.

Figure 7: Prediction from a Driver Model based on Pooled Data

However, in our proposed learning structure, the personalization step produces
$\theta_{personalized}$ that results in different driver models for each
vehicle rather than one-model-fits-all approach. This allows vehicles to
retain their desired behavior which still sharing knowledge. Fig. 8 shows the
predictions for both aggressive and passive AVs based on our personalized
knowledge sharing approach. Table 1 further shows the Root Mean Squared Error
(RMSE) based on speed for each model. It is evident that under heterogeneity
our proposed learning structure can better encode different driving behaviors
and thus better fit personalized driver models.

(a) Personalized Aggressive Driver Model. Note: the Three Curves Nearly
Overlap each other

(b) Personalized Passive Driver Model. Note: Blue and Green Curves Nearly
Overlap each other

Figure 8: Prediction from Driver Models based on Knowledge Sharing and Personalization RMSE(speed) | Aggressive | Passive
---|---|---
Pooled Model | 0.53 | 2.46
Knowledge Sharing & Personalization | 0.12 | 0.22
Table 1: Prediction Error

###  3.4 Limitations and Remarks

  1. 1.

On the driver model function: Here we explored a $GP$ function; however in
practice other models might be more widespread such as deep learning or
reinforcement learning methods. While our training framework remains malleable
enough to apply for different functions, it is important to extend to other
driver models.

  2. 2.

On parameter sharing: In our framework, all vehicles share the same driver
model ($GP$) with same feature input (leader speed) and output (AV speed).
However, if one wants to collaboratively share knowledge between vehicles with
different functional form of the driver model (i.e., one vehicle has neural
network and other reinforcement learning models) an alternative parameter
sharing scheme is needed. A way to approach this is to decompose
$\theta_{s}:=\theta_{functional}+\theta_{shared}$. $\theta_{functional}$ are
unique parameters that relevant to the vehicle driver model, while
$\theta_{shared}$ come from a global learned model. Additionally, one can
regularize the $\theta$ (i.e., assign weights on $\theta_{functional}$ and
$\theta_{shared}$) so that a vehicle can put more value on its own driver
model and preferences.

  3. 3.

On aggregation ($\hat{\theta}$ in Eq. 7): Our aggregation strategy (averaging)
is based on the widely used FedAvg. However, other strategies exist and can be
designed to suit the desired application.

  4. 4.

On personalization under complex driver models: We note that in our approach
here we use a very simplified driver model (only taking speed data), however
in complex driver models it can become hard to encode personalization for each
vehicle. Since in this case, you can have multiple layers of designed
parameters that are all contributing to the behavioral change of the vehicle.
This is rather a complex problem to solve, as one would first need to
decompose unique and common features between vehicles. This problem is
currently understudy by the team.

  5. 5.

On when to pool data: In some cases pooling data can be beneficial. For
instance, consider the setup in Experiment 1 (Section 3.2). If vehicles 1-2-3
have similar design (driver model with same parameter setting), one can pool
all the data from different scenarios and train one-modal-that-fits-all.
However, pooling data might not even be applicable given that access to some
data is restricted (privacy concerns of propriety rights). As such our
approach circumvents this by only sharing parameter values and never raw data.

##  4 Conclusions

In this work we present a way of learning and training driver models for AVs
in a collaborative way. Different vehicles can share knowledge between each
other through a collaborative iterative process that entails sharing and
discovering optimal parameters that minimize a desired global loss function.
We also show how vehicles can share knowledge while retaining a personalized
model tailored to their own data.

We showcase two experimental applications of the designed model. In the first
experiment, three vehicles collaborate to learn a speed oscillation, by
decomposing and transferring knowledge between each other. In the second
experiment, we show how under heterogeneous AV behavior, learning a driver
model while pooling data is not ideal and thus personalization yields better
results.

Several limitations and extensions of this work are yet to be tackled and are
highlighted above. A large scale experimentation and benchmarking of driver
models with or without knowledge sharing and personalization is important, yet
goes beyond what this paper can provide, and is left for future work by the
authors. We hope that our endeavors in this modeling direction would spur
interest and motivate further work on designing complex AV driver model’s that
can safely and efficiently maneuver in the open world.

##  5 Acknowledgements

This research was sponsored by the Unites States National Science Foundation
through Award CMMI 1932932 and the University of Wisconsin-Madison.

## References

  * [1] Kallista Bonawitz, Peter Kairouz, Brendan Mcmahan, and Daniel Ramage.  Federated learning and privacy.  Communications of the ACM, 65(4):90–97, 2022.
  * [2] Vishnu Pandi Chellapandi, Liangqi Yuan, Stanislaw H Zak, and Ziran Wang.  A survey of federated learning for connected and automated vehicles.  arXiv preprint arXiv:2303.10677, 2023.
  * [3] Kan Chen, Runzhou Ge, Hang Qiu, Rami Ai-Rfou, Charles R. Qi, Xuanyu Zhou, Zoey Yang, Scott Ettinger, Pei Sun, Zhaoqi Leng, Mustafa Mustafa, Ivan Bogun, Weiyue Wang, Mingxing Tan, and Dragomir Anguelov.  Womd-lidar: Raw sensor dataset benchmark for motion forecasting.  arXiv preprint arXiv:2304.03834, April 2023.
  * [4] Mu Han, Kai Xu, Shidian Ma, Aoxue Li, and Haobin Jiang.  Federated learning-based trajectory prediction model with privacy preserving for intelligent vehicle.  International Journal of Intelligent Systems, 37(12):10861–10879, 2022.
  * [5] Xiangwang Hu, Zuduo Zheng, Danjue Chen, and Jian Sun.  Autonomous vehicle’s impact on traffic: empirical evidence from waymo open dataset and implications from modelling.  IEEE Transactions on Intelligent Transportation Systems, 2023.
  * [6] Xiangwang Hu, Zuduo Zheng, Danjue Chen, Xi Zhang, and Jian Sun.  Processing, assessing, and enhancing the waymo autonomous vehicle open dataset for driving behavior research.  Transportation Research Part C: Emerging Technologies, 134:103490, 2022.
  * [7] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836, 2016.
  * [8] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [9] Raed Kontar, Naichen Shi, Xubo Yue, Seokhyun Chung, Eunshin Byon, Mosharaf Chowdhury, Jionghua Jin, Wissam Kontar, Neda Masoud, Maher Nouiehed, et al.  The internet of federated things (ioft).  IEEE Access, 9:156071–156113, 2021.
  * [10] Wissam Kontar and Soyoung Ahn.  Bayesian methods in automated vehicle’s car-following uncertainties: Enabling strategic decision making, 2022.
  * [11] Wissam Kontar, Tienan Li, Anupam Srivastava, Yang Zhou, Danjue Chen, and Soyoung Ahn.  On multi-class automated vehicles: Car-following behavior and its implications for traffic dynamics.  Transportation research part C: emerging technologies, 128:103166, 2021.
  * [12] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [13] Vicente Milanés and Steven E Shladover.  Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data.  Transportation Research Part C: Emerging Technologies, 48:285–300, 2014.
  * [14] Gaith Rjoub, Jamal Bentahar, and Omar Abdel Wahab.  Explainable ai-based federated deep reinforcement learning for trusted autonomous driving.  In 2022 International Wireless Communications and Mobile Computing (IWCMC), pages 318–323. IEEE, 2022.
  * [15] Naichen Shi and Raed Al Kontar.  Personalized pca: Decoupling shared and unique features.  arXiv preprint arXiv:2207.08041, 2022.
  * [16] Naichen Shi, Fan Lai, Raed Al Kontar, and Mosharaf Chowdhury.  Fed-ensemble: Ensemble models in federated learning for improved generalization and uncertainty quantification.  arXiv preprint Arxiv:2107.10663, 2021.
  * [17] Steven E Shladover, Christopher Nowakowski, Xiao-Yun Lu, and Robert Ferlis.  Cooperative adaptive cruise control: Definitions and operating concepts.  Transportation Research Record, 2489(1):145–152, 2015.
  * [18] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al.  A field guide to federated optimization.  arXiv preprint arXiv:2107.06917, 2021.
  * [19] Xubo Yue and Raed Al Kontar.  Federated gaussian process: Convergence, automatic personalization and multi-fidelity modeling.  arXiv preprint arXiv:2111.14008, 2021.
  * [20] Xubo Yue, Raed Al Kontar, and Ana María Estrada Gómez.  Federated data analytics: A study on linear models.  IISE Transactions, pages 1–13, 2022.