# CARL: Congestion-Aware Reinforcement Learning for Imitation-based
Perturbations in Mixed Traffic Control

Bibek Poudel1, Weizi Li1, Shuai Li2 1Bibek Poudel and Weizi Li are with Min H.
Kao Department of Electrical Engineering and Computer Science at University of
Tennessee, Knoxville, TN, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>Li
is with Department of Civil and Environmental Engineering at University of
Tennessee, Knoxville, TN, USA<EMAIL_ADDRESS>
###### Abstract

Human-driven vehicles (HVs) exhibit complex and diverse behaviors. Accurately
modeling such behavior is crucial for validating Robot Vehicles (RVs) in
simulation and realizing the potential of mixed traffic control. However,
existing approaches like parameterized models and data-driven techniques
struggle to capture the full complexity and diversity. To address this, in
this work, we introduce CARL, a hybrid technique combining imitation learning
for close proximity car-following and probabilistic sampling for larger
headways. We also propose two classes of RL-based RVs: a safety RV focused on
maximizing safety and an efficiency RV focused on maximizing efficiency. Our
experiments show that the safety RV increases Time-to-Collision above the
critical $4$ second threshold and reduces Deceleration Rate to Avoid a Crash
by up to $80\%$, while the efficiency RV achieves improvements in throughput
of up to $49\%$. These results demonstrate the effectiveness of CARL in
enhancing both safety and efficiency in mixed traffic.

##  I Introduction

Robot Vehicles (RVs) have the potential to revolutionize transportation by
enhancing safety, efficiency, and accessibility for all road users [1]. To
fully realize the benefits of this technology, it is crucial to validate their
performance, trustworthiness, and robustness through comprehensive testing
[2]. While real-world evaluation is essential, it involves significant risks,
costs, and time constraints, making simulation an attractive alternative,
which represents a safer, more efficient, and cost-effective approach [3].
Nevertheless, the validation of RVs in simulation faces the challenge of
accurately modeling the behaviors of Human-driven Vehicles (HVs) [4]. As more
vehicles with varying levels of autonomy are introduced into our
transportation system, the idea of mixed traffic control, which involves the
use of RVs to mitigate problems such as congestion and delays produced by HVs,
has emerged [5, 6, 7, 8, 9, 10, 11]. In mixed traffic scenarios where RVs and
HVs co-exist, accurately modeling real-world human driving behavior becomes
even more critical. However, this remains an open problem [12].

Figure 1: Instantaneous accelerations observed during car-following behaviors
at densities $[70,150]~{}veh/km$. TOP: Real-world data from the I-$24$ MOTION
dataset reveals a distribution having long tails extending to
$[-3,3]~{}m/s^{2}$. BOTTOM: IDM (in simulation) produces accelerations mostly
within $[-0.5,0.5]~{}m/s^{2}$, indicating much ‘timid’ driving behaviors than
the real world.

Among the various aspects of real-world human driving, the longitudinal
behavior, especially car-following, is the most prevalent mode [13].
Consequently, accurate modeling of car-following behavior is crucial for
reproducing realistic traffic flows and vehicle interactions in simulation,
and is a key component in addressing the sim-$2$-real gap [14]. To reproduce
the car-following behaviors of HVs in simulation, there exist two mainstream
methods: parameterized models and data-driven approaches. Popular choices of
parameterized models include Gipp’s [15], Krauss [16], and Intelligent Driver
Model (IDM) [17] which require calibration of various parameters that
represent driving behaviors and vehicle dynamics such as maximum acceleration,
minimum time gap, and desired velocity. These models rely on robust
calibration and even then may fall short in capturing traffic diversity and
lack broad applicability [18]. Enhancements to the IDM model (often a default
choice in simulation like SUMO [19]) by addition of random noise [20] and
tuning with real-world data [21, 22] have been proposed, but they struggle to
reproduce real-world variability [23] and fail to generalize outside the
reference datasets [24]. Fig. 1 illustrates a typical example, where the IDM
with random noise model fails to capture the long-tailed distribution of
accelerations observed in real-world data, exhibiting a $24\%$ discrepancy
(between real-world and simulation) in the percentage of accelerations
observed within $[-0.5,0.5]~{}m/s^{2}$, i.e., depicting primarily safe or
timid behaviors in simulation. Moreover, prior studies in traffic control and
coordination often impose artificial limits on vehicle behaviors, such as
bounded accelerations [25, 26, 27, 28, 29], which further reduce the accuracy
in reproducing HV behaviors.

Data-driven approaches such neural networks and Reinforcement Learning (RL)
have surpassed IDM in car-following simulation accuracy [30]. These techniques
utilize real-world data to extract features for supervised learning [30] or
tune the reward function in RL [31, 32]. Some RL-based approaches also equip
HVs with both leader and follower vehicle information [33]. However, these
methods still face challenges in capturing the full complexity and diversity
of human driving, as they often require hand-crafted features and extensive
training and tuning [34]. Imitation Learning (IL) offers a promising
alternative to address these shortcomings. By learning directly from expert
demonstrations, IL captures the implicit knowledge and individual preferences
of human drivers, such as safety and comfort, without the need to specify
these objectives. In addition, IL models can be more robust under
environmental uncertainties and disturbances, as the expert’s behavior may
already account for these factors [35]. While IL shows great potential,
relying solely on it may still not be sufficient to capture the full spectrum
of human driving behaviors and reactions as IL may struggle to generalize
outside the situations seen during training [36].

To address these limitations, we introduce CARL, a hybrid technique that
combines imitation learning and probabilistic sampling. The key difference in
our work is that we consider the proximity between HVs during car-following to
be a crucial factor in determining the appropriate acceleration model. When
HVs are at close proximity to a leader HV during car-following, their
accelerations are obtained from the imitation learning model, which captures
the nuances of human driving behavior in these more sensitive situations, as
the consequences of small changes in accelerations are more profound and
potentially dangerous. Conversely, when the space headway exceeds a threshold,
we employ probabilistic sampling to generate accelerations that introduce
realistic variability. This hybrid approach enables CARL to produce more
accurate representations of real-world driving behaviors, including aggressive
acceleration profiles. CARL leverages the strengths of both machine learning
methods, which offer accurate representations of real-world driving behaviors,
and probabilistic sampling, which minimizes approximations and assumptions. In
addition, we propose two classes of RL-based RVs: a safety RV focused on
maximizing safety and an efficiency RV focused on maximizing efficiency. Our
RVs are designed to optimize their actions based on the congestion conditions
predicted by a supervised classifier, whose output is incorporated into the
observation and reward of the RL algorithm.

Under the realistic-accelerations we evaluate the safety and efficiency and
compare CARL with RVs proposed in prior studies for mixed traffic control. To
evaluate safety, we use two surrogate measures, time to collision (TTC) and
deceleration rate to avoid a crash (DRAC) whereas for efficiency we measure
fuel economy and throughput. Our experiments show that our RV consistently
outperforms other methods in safety, increasing TTC above $4~{}s$ and reducing
DRAC by up to $80\%$. Whereas in efficiency our RV achieves improvements of up
to $49\%$ in throughput while also consistently maintaining the second-highest
fuel economy among all evaluated methods. These results demonstrate the
effectiveness of CARL in enhancing both safety and efficiency in mixed
traffic. The project code can be found in the repository:
https://github.com/poudel-bibek/CARL.

##  II Methodology

We introduce our data processing procedure, Intelligent Driver Model (IDM),
Model-based and Heuristic-based RVs, our RL-based RV, and the imitation of
real-world driving behaviors and perturbations.

###  II-A Data Processing and Intelligent Driver Model (IDM)

We apply a car-following-filter [24] to the I-$24$ MOTION dataset [37] with
study length $=6.75~{}km$ and study time $=4~{}h$. The dataset contains
various vehicle types such as semi-trailers, mid-sized trucks, motorbikes, and
cars under different traffic conditions such as approaching standing traffic,
lane changing, and free flow. To extract car-following trajectories, we select
data points that meet the following criteria:

  * •

Ego car is following another car, i.e., has a leader.

  * •

Leader and ego cars are in the same lane $\geq 5~{}s$.

  * •

Ego car’s speed is $>10\%$ of the speed limit, i.e., not approaching
stationary traffic.

  * •

Ego car’s space headway is $<124~{}m$, applying $4~{}s$ rule at the speed
limit to avoid free flow conditions.

By applying the filter, we extract a total of $172,000$ instantaneous
accelerations, their distribution is shown in Fig.1 TOP. In comparision, Fig.
1 BOTTOM depicts the accelerations obtained from the IDM car-following model,
which assumes that drivers strive to maintain a safe distance from the leader
vehicle while trying to achieve a target speed. In IDM, vehicles speed up when
the distance to the leading vehicle is substantial, and decelerate when the
distance decreases below a target minimum gap. The parameters of IDM are set
according to Treiber and Kesting [17] as maximum acceleration $(a)=1$, maximum
deceleration $(b)=1.5$, target time headway $(T)=1$, acceleration coefficient
$(\delta)=4$, and minimum gap $(s_{0})=2$.

###  II-B Heuristic-based Robot Vehicles

FollowerStopper (FS): FS [38] is an RV that travels at a fixed command
velocity (target) under safe conditions but when required, slightly lowers the
target velocity to open a gap to the leader vehicle. This allows the RV to
dampen oscillations and brake smoothly when needed. The command velocity is
given by

$v_{cmd}=\begin{cases}0,&\text{if }\Delta x\leq\Delta x_{1}\\\ v\frac{\Delta
x-\Delta x_{1}}{\Delta x_{2}-\Delta x_{1}},&\text{if }\Delta x_{1}<\Delta
x\leq\Delta x_{2}\\\ v+(U-v)\frac{\Delta x-\Delta x_{2}}{\Delta x_{3}-\Delta
x_{2}},&\text{if }\Delta x_{2}<\Delta x\leq\Delta x_{3}\\\ U,&\text{if }\Delta
x_{3}<\Delta x\end{cases}$

where $v=\min\left(\max\left(v_{\text{lead}},0\right),U\right)$ is the speed
of the leader vehicle, $\Delta x$ is the headway of the RV, and $U$ is the
desired velocity. The thresholds ($\Delta x_{1},~{}\Delta x_{2},~{}\Delta
x_{3}$) are defined as

$\Delta x_{k}=\Delta x_{k}^{0}+\frac{1}{2d_{k}}(\Delta v_{-})^{2},\quad
k=1,2,3.$

The model parameters $\Delta x_{k}^{0}$, $\Delta v_{-}$, and $d_{k}$ together
determine the spacing between vehicles and the RV’s responsiveness to changes
in velocity.

Proportional-integral with saturation (PIwS): PIwS [38] estimates the desired
average velocity ($U$) of the vehicles in the network using its historical
average velocity. The PIwS RV calculates the target velocity as

$v_{target}=U+v_{catch}\times\min\left(\max\left(\frac{\Delta
x-g_{l}}{g_{u}-g_{l}},0\right),1\right),$

which is used to calculate the command velocity at $t+1$

$v_{cmd}^{t+1}=\beta_{t}(\alpha_{t}v_{target}^{t}+(1-\alpha_{t})v_{lead}^{t})+(1-\beta_{t})v_{cmd}^{t},$

where $v_{catch}$ is the catch-up velocity—a velocity higher than the average
velocity allows the RV to catch up with its leader, $\Delta x$ is the
difference in position between the RV and its leader, $g_{l}$ and $g_{u}$
represent the lower and upper threshold distance, respectively; $\alpha_{t}$
and $\beta_{t}$ represent the weight factors for target velocity $v_{target}$
and command velocity $v_{cmd}$, respectively. Finally, $v_{lead}$ represents
the velocity of the leader vehicle.

###  II-C Model-based Robot Vehicles

Bilateral Control Module (BCM): BCM [39] uses information about both follower
and leader vehicles to obtain a linear model whose acceleration is given by:

$a=k_{d}\cdot\Delta_{d}+k_{v}\cdot(\Delta v_{l}-\Delta
v_{f})+k_{c}\cdot(v_{des}-v),$

where $\Delta_{d}$, $\Delta v_{l}$, $\Delta v_{f}$, $v_{des}$, and $v$,
represent the difference in distance to the leader compared to the distance to
the follower, the difference in velocity to the leader, the difference in
velocity to the follower, the set desired velocity, and the current velocity
of the vehicle, respectively. $k_{d}=1$, $k_{v}=1$, and $k_{c}=1$ are gain
parameters.

Linear Adaptive Cruise Control (LACC): LACC is an improvement over existing
cruise control systems that allows vehicles to maintain a safe distance or
speed without communication. One implementation is the constant time-headway
model by Rajamani [40], which employs a first-order differential equation for
approximation. The control acceleration at time $t$ is given by

$a_{t}=(1-\frac{\Delta t}{\tau})\cdot a_{(t-1)}+\frac{\Delta
t}{\tau}a_{cmd,(t-1)},$ $a_{cmd}=k_{1}\cdot e_{x}+k_{2}\cdot\Delta
v_{l},~{}\text{and}~{}e_{x}=s-h\cdot v,$

where $k_{1}=0.3$ and $k_{2}=0.4$ are design parameters, $e_{x}$ is the gap
error, $s$ is the space headway, $\Delta v_{l}$ is the relative velocity
difference to the leader, $h=1$ is the desired time gap, $\Delta t$ is the
control time-step, and $\tau=0.1$ is the time lag of the control system.

Figure 2: Input data labeling for the congestion classifier (sensing zone
shown in blue). The congestion classifier takes as input (position, velocity)
of all vehicles in the sensing zone and outputs the traffic condition based on
patterns in space headway.

###  II-D RL-based Robot Vehicles

RL is a $T$-step episodic task where an agent interacts with its environment
and seeks to maximize the sum of discounted rewards. At each time step, the
agent is given a state $s$, responds with an action $a$, and then the
environment returns the next state $s^{\prime}$ and reward $r$. This is
formalized as a partially observable Markov Decision Process (MDP) represented
as
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\Omega,\mathcal{O},\gamma)$,
where $\mathcal{S}$ represents the set of states, $\mathcal{A}$ is the set of
actions, $\mathcal{P}(s^{\prime},r|s,a)$ describes the environment dynamics,
$\mathcal{R}(s,a)$ is the reward function, $\Omega$ represents the set of
observations, $\mathcal{O}(o|s^{\prime},a)$ is the observation function
defining the probability of observing $o$ given the action $a$ has led to
state $s^{\prime}$, and $\gamma$ is the discount factor. RL-based methods have
gained popularity as an effective alternative to model-based or heuristic-
based methods.

In driving scenarios, safety and efficiency goals are often in conflict [31];
for example, optimizing for throughput may lead to reduced space headways and
increased velocities, which can compromise safety. To address this trade-off,
we propose two types of RVs: the _safety_ RV, which prioritizes safety, and
the _efficiency_ RV, which emphasizes efficiency. Both RV types use the
congestion classifier and operate within the same action and observation
space. We train our RVs using the PPO algorithm [41] with the following MDP
components:

  * •

Observation. The RV’s observation $o_{t}$ at time $t$ is a combination of its
own velocity ($v_{t}$), the relative position ($\Delta p_{t}$) to immediate
leader HV and relative velocity ($\Delta v_{t}$) with respect to its immediate
leader HV, and the predicted congestion stage ($c_{t}$) from the congestion
classifier ($f_{\text{CC}}$):

$\displaystyle\vspace{-5pt}o_{t}$ $\displaystyle=[v_{t},\Delta p_{t},\Delta
v_{t}]\oplus c_{t},$ $\displaystyle c_{t}$
$\displaystyle=f_{\text{CC}}(\\{r_{p,i},r_{v,i}\\}_{i\in\mathcal{Z}}),\vspace{-5pt}$

where $\mathcal{Z}$ denotes the set of $|\mathcal{Z}|=n$ vehicles within the
sensing zone ($55~{}m$), and $r_{p,i}$ and $r_{v,i}$ are the relative position
and velocity of the $i$-th vehicle, respectively.

  * •

Action.The RV’s action ($a_{t}$) is its acceleration, bounded within $[-3,3]$
m/s2.

  * •

Reward. The reward function $R(s_{t},a_{t})$ is a weighted sum of the RV’s
velocity $v_{t}$ (for the efficiency RV) or the average velocity of all
vehicles $\bar{v}_{t}$ (for the safety RV), an acceleration penalty, and a
shaping term based on the predicted congestion stage $c_{t}$.

Reward Functions

efficiency RV

$R(s_{t},a_{t})=0.75v_{t}-2|a_{t}|$

if $c_{t}=\text{Congested}\wedge a_{t}>0$ then

$R(s_{t},a_{t})\mathrel{{+}{=}}\min(-1,\lambda_{1}|a_{t}|)$

end if

if $c_{t}=\text{Leaving}\wedge a_{t}<0$ then

$R(s_{t},a_{t})\mathrel{{+}{=}}\lambda_{2}|a_{t}|$

end if

safety RV

$R(s_{t},a_{t})=0.15\bar{v}_{t}-4|a_{t}|$

if $c_{t}=\text{Forming}$ then

$R(s_{t},a_{t})\mathrel{{+}{=}}\min(-1,\lambda_{3}|a_{t}|)$

end if

where $\bar{v}_{t}=\frac{1}{n}\sum_{i=1}^{n}v_{i,t}$ is the average velocity
of all $n$ vehicles at time $t$, and $\lambda_{1}=-10$, $\lambda_{2}=-10$, and
$\lambda_{3}=-5$ are empirically determined weights.

  * •

Scaling laws. For RV penetration rates above $5\%$, our RVs form platoons with
a single leader and multiple followers. Our RV platoon at $40\%$ penetration,
for instance, consists of $9$ RVs $(22\times 0.4=8.8\approx 9)$, with a leader
RV trained at $5\%$ penetration and $8$ followers. The follower RVs observe
the platoon’s state (position and velocity of all vehicles in the platoon) and
optimize the reward:

$R_{\text{follower}}(s_{t},a_{t})=\lambda_{4}\Delta p_{t,j}+\lambda_{5}\Delta
v_{t,j}+\lambda_{6}|a_{t,j}|+\lambda_{7},$

where $\Delta p_{t,j}$ and $\Delta v_{t,j}$ are the relative position and
velocity of the $j$-th follower with respect to the leader, $a_{t,j}$ is the
follower’s acceleration, and $\lambda_{4}=-2$, $\lambda_{5}=4$,
$\lambda_{6}=-4$, and $\lambda_{7}=10$ are empirically chosen weights.

####  II-D1 Congestion Classifier

Our RL-based approach employs a congestion classifier, a neural network
trained with supervised learning to predict the congestion stage $10$ time-
steps in advance to allow for pre-emptive responses from our RV. The input to
the classifier consists of the positions and velocities of preceding cars
within the sensing zone. These input features are then mapped to one of six
labels, each representing a distinct stage of congestion based on the
asymmetric driving theory [42]. The theory suggests that human drivers tend to
underestimate the required space headway during deceleration and overestimate
it during acceleration. Consequently, when congestion is forming, the
available space headway decreases monotonically from one vehicle to the next
as we move downstream within the sensing zone. Conversely, when congestion is
dissipating, the available space headway increases monotonically. The six
labels used to capture these congestion stages are: ’Forming’, ’Leaving’,
’Congested’, ’Free flow’, ’Undefined’, and ’No Vehicle’.

To train the congestion classifier, we collect position and velocity data from
all vehicles within the RV’s sensing zone (set to $55~{}m$) at density ranges
between $70$ and $133~{}veh/km$. The collected data is then clustered using
the K-means algorithm across all six classes, as shown in Fig. 3 (RIGHT). The
dispersion of clusters in the data indicates its classifiability. Due to the
sequential nature of the data and the need for making predictions several
time-steps into the future, we choose a time offset of $10$ time-steps. This
offset strikes a balance between the usefulness and accuracy of the
predictions. A larger offset, such as $100$ time-steps, would provide more
time for the RV to react but may compromise prediction accuracy. Conversely, a
shorter offset, like $1$ time-step, would yield highly accurate predictions
but may not allow sufficient time for the RV to take effective action.

Figure 3: LEFT: Confusion Matrix of a trained congestion classifier in Ring on
the validation set with the six classes abbreviated as: L=‘Leaving’,
F=‘Forming’, FF=‘Free Flow’, C=‘Congested’, U=‘Undefined’, and N=‘No Vehicle’.
RIGHT: The results of applying K-means clustering with t-SNE on a subset of
the training data of the congestion classifier. The clusters are spread out
and distinct suggesting that the data is easily classifiable.

After windowing, the dataset includes instances where the congestion stage
changes from t to t$\thinspace\raisebox{0.0pt}[0.0pt][0.0pt]{+}\thinspace 10$,
as well as instances where the congestion stage remains the same over the time
window. To train the congestion classifier, we sample data to ensure a
balanced representation of transition/non-transition instances as well as
instances containing all six classes. Worth noting, the ‘No vehicle’ class
presents a unique challenge. The collected data may contain instances changing
from ‘No vehicle’ to another class after the $10$ time-steps. However, based
on the input corresponding to ‘No vehicle’ at t, we cannot predict the
congestion stage at t$\thinspace\raisebox{0.0pt}[0.0pt][0.0pt]{+}\thinspace
10$. Consequently, we discard data points where the ‘No Vehicle’ class
transitions to another class after $10$ time-steps. We replace this discarded
data with synthetic examples that simulate various scenarios for the RV’s
position and velocity without leader vehicles. The congestion classifier is
trained for $50$ epochs with validation accuracy of $95.5\%$ (training
parameters are provided in Table I and the confusion matrix on a validation
set is shown in Fig. 3 LEFT). Finally, We incorporate the predictions of the
congestion classifier into the observations and reward function of the RV.

####  II-D2 Benchmarking RL Policies

To benchmark with other RL techniques, we reproduce their original policies by
following the provided experiment parameters and closely matching the
performance. Specifically, to obtain RL policy with only local observations,
we follow Wu et al. [43] and refer the policy as Wu hereafter; Our reproduced
Wu achieves the performance within $1\%$ error (measured with stabilization
time and average velocity during stabilization) of the original work. We
benchmark CARL with RL-based RVs with only microscopic/local observations such
as the relative position and velocity to its leader vehicle [43].

Category | Parameter | Value
---|---|---
| Time Step ($\Delta t$) | $0.1$
Simulation Horizon ($T$) | $4500$
Simulation | Warmup Time-steps | $2500$
| Speed Limit ($m/s$) | $30$
| Initial Speed ($m/s$) | $0$
PPO | Learning Rate ($\alpha$) | $0.00005$
Discount Factor ($\gamma$) | $0.999$
GAE Estimation ($\lambda$) | $0.97$
KL Divergence Target | $0.02$
Algorithm | Entropy Coefficient Initial | $0.1$
| Entropy Coefficient Final | $0.01$
| Value Function Clip Param | $20$
| SGD Iterations | $2$
Congestion | Neural Network | $32,16,16$
Batch size | $32$
Classifier | Learning rate | $0.01$
| Epochs | $50$
Policy | Our leader RV | $64,32,16$
Our follower RV | $64,32,16$
Networks | Wu | $32,32,32$

TABLE I: Detailed experiment parameters. We show the simulation parameters as
well as the parameters of Proximal Policy Optimization (PPO) and the
congestion classifier. The hidden layer dimensions of various policy networks
are also shown.

###  II-E Perturbations Via Imitation Learning and Sampling

To ensure an accurate reproduction of real-world driving behaviors and
perturbations in the simulation, we adopt a hybrid technique combining
imitation learning and probabilistic sampling. To train an imitation learning
(behavior cloning) model, we extract relevant HV data during car-following
such as ego velocity, headway distance, and leader velocity corresponding to
the instantaneous accelerations shown in Fig. 1 TOP. Using this data, we train
an $801$-parameter neural network model that takes these variables as input
and outputs acceleration commands. To inject acceleration perturbations to HVs
in simulation, we characterize the discrete acceleration behaviors extracted
from the dataset in terms of intensity, frequency, and duration. Upon
analysis, we find a negative correlation of intensity with frequency and
durations i.e., higher intensity accelerations tend to have shorter duration
and occur less frequently. Then, we uniformly sample the frequency of the
perturbations within the observed range of $[10,~{}30]$ per HV, for every $6$
minutes of car-following. The intensity of the acceleration perturbations is
determined based on the space headway of the HV. When the space headway is
less than $5~{}m$, human driving behavior becomes more sensitive, as the
consequences of accelerations are more significant at shorter distances;
hence, the accelerations are obtained from the imitation learning model to
better capture the nuances of human driving. whereas if the we space headway
is greater than $5~{}m$, we sample acceleration intensity uniformly within
$[-3,~{}3]~{}m/s^{2}$. To determine the duration ($\tau_{a_{i}}$) of each
selected intensity ${a}_{i}$, we first find the most common duration (mode)
$\tilde{\tau}_{a_{i}}$ by linearly mapping $a_{i}$ within the minimum
($\tau{{}_{\text{min}}}$) and maximum ($\tau{{}_{\text{max}}}$) observed
durations. We then sample $\tau_{a_{i}}$ from a piecewise triangular
distribution using the following conditional probability density function:

$P(\tau_{a_{i}}|\tilde{\tau}_{a_{i}})=\begin{cases}\frac{2(\tau_{a_{i}}-\tau_{\text{min}})}{\tau_{\text{range}}(\tilde{\tau}_{a_{i}}-\tau{\text{min}})},&\tau_{\text{min}}\leq\tau_{a_{i}}<\tilde{\tau}_{a_{i}},\\\
\vspace{-10pt}\\\
\frac{2(\tau_{\text{max}}-\tau_{a_{i}})}{\tau_{\text{range}}(\tau_{\text{max}}-\tilde{\tau}_{a_{i}})},&\tilde{\tau}_{a_{i}}\leq\tau_{a_{i}}\leq\tau_{\text{max}}.\end{cases}$

where $\tau_{\text{range}}=\tau_{\text{max}}-\tau_{\text{min}}$. The sampled
acceleration perturbation is randomly assigned to HVs during experiments.

##  III Experiments

We introduce the mixed traffic environment, the evaluation metrics, the
experimental setup, and finally the results. To begin with, we test on the
mixed traffic environment, the Ring: a single-lane circular road network with
$22$ vehicles. This classical scene simulates ‘stop-and-go traffic’ where
repeated cycles of accelerations and decelerations occur in HVs even in the
absence of external disturbances. For our evaluations, we measure safety using
surrogate measures associated with near-crash events, including Time to
Collision (TTC) [44] to indicate collision risk and Deceleration Rate to Avoid
a Crash (DRAC) [45] to quantify necessary braking force to avoid a collision.
For efficiency, Fuel Economy (FE) estimates fuel consumption using the
Handbook Emission Factors for Road Transport $3$ Euro $4$ passenger car
emission model [46], while Throughput indicates network capacity utilization
(flow rate).

For experiment setup, we use FLOW [43] and SUMO [19] with RVs platooned when
penetration rate is $>5\%$ (all evaluated RVs can stabilize traffic in a
platoon configuration, BCM and LACC require minimum $20\%$ and $40\%$
penetration respectively [27]). We select penetration rates $5\%$, $20\%$,
$40\%$, and $60\%$ to align with the minimum rates required for stabilizing
traffic by different RVs [27, 11]. We adhere to stringent safety standards in
our evaluations of TTC, DRAC hence when multiple RVs are present, we report
the worst-case values among the RVs (including IDM baseline where the worst
case among all vehicles in considered). In contrast, to measure efficiency we
consider all vehicles in the network. In each experiment, we first allow the
RV enough time to stabilize traffic and then we introduce acceleration
perturbations for six-minute equivalent of real-world time.

Figure 4: Average velocity profile of RL-based approaches (Wu, Our efficiency
RV, and Our safety RV) at $5\%$ penetration, averaged over $10$ simulation
rollouts. The solid lines indicate average velocity and colored ranges
indicate standard deviation across rollouts. During application of real-world
perturbations between $800~{}s$ and $1160~{}s$, Our efficiency RV has the
highest average velocity whereas Wu has the highest standard deviation,
indicating more sensitivity.

Fig. 4 shows the velocity profiles of RL-based approaches in our study, namely
Wu, Our efficiency RV, and Our safety RV at $5\%$ penetration rate. All
vehicles in the network are controlled using IDM before during initial warmup
time of $250~{}s$ and RVs are activated after warmup. After activation, Wu
stabilizes traffic at $440~{}s$, Our efficiency RV stabilizes traffic at
faster than Wu at $395~{}s$ whereas Our safety RV stabilizes traffic gradually
at $690~{}s~{}$; traffic is considered as stable if the standard deviation of
acceleration is lower than the IDM noise ($0.2~{}m/s^{2}$). During
stabilization, Wu maintains a higher average velocity of $4.88~{}m/s$ compared
to Our efficiency RV at $4.51~{}m/s$, while our safety RV has the lowest
average velocity at $3.18~{}m/s$. However, when perturbations are applied to
HVs between $800~{}s$ and $1160~{}s$, Our efficiency RV maintains a higher
average velocity at $4.02~{}m/s$ (also a higher throughput) compared to Wu at
$3.15~{}m/s$ while Our safety RV at $3.11~{}m/s$ has comparable average
velocity to Wu. Notably, Wu has the highest standard deviation among all RVs
between $800~{}s$ and $1140~{}s$ indicating more sensitivity to real-world
perturbations.

RV% | RV Type | Safety | Efficiency
---|---|---|---
TTC | DRAC | FE | Throughput
| IDM* | $1.82$ | $1.62$ | $7.63$ | $988$
$5\%$ | FS | $3.99$ | $0.89$ | $12.34$ | $1283$
PIwS | $1.95$ | $1.89$ | $12.99$ | $1343$
BCM | $1.03$ | $2.82$ | $8.17$ | $1023$
LACC | $1.09$ | $1.42$ | $8.01$ | $1031$
Wu | $2.39$ | $1.23$ | $9.92$ | $1048$
Ours | $\bf{7.95}$ | $\bf{0.36}$ | $\bf{12.80}$ | $\bf{1242}$
$20\%$ | FS | $4.22$ | $0.74$ | $12.08$ | $1344$
PIwS | $1.71$ | $1.48$ | $11.76$ | $1328$
BCM | $2.21$ | $0.75$ | $13.19$ | $1392$
LACC | $1.08$ | $1.73$ | $9.42$ | $1166$
Wu | $2.24$ | $1.27$ | $6.41$ | $791$
Ours | $\bf{5.84}$ | $\bf{0.59}$ | $\bf{12.87}$ | $\bf{1336}$
$40\%$ | FS | $3.53$ | $1.02$ | $11.55$ | $1323$
PIwS | $1.58$ | $1.89$ | $11.09$ | $1294$
BCM | $4.35$ | $0.70$ | $11.96$ | $1400$
LACC | $3.14$ | $0.90$ | $14.39$ | $1426$
Wu | $2.18$ | $1.27$ | $3.66$ | $456$
Ours | $\bf{4.22}$ | $\bf{0.89}$ | $\bf{14.06}$ | $\bf{1430}$
$60\%$ | FS | $2.39$ | $1.00$ | $9.99$ | $1124$
PIwS | $1.59$ | $1.94$ | $10.70$ | $1264$
BCM | $2.40$ | $0.84$ | $11.79$ | $1393$
LACC | $2.55$ | $1.47$ | $13.91$ | $1448$
Wu | $2.83$ | $1.17$ | $4.41$ | $556$
Ours | $\bf{6.58}$ | $\bf{0.57}$ | $\bf{13.61}$ | $\bf{1473}$

TABLE II: Comparative evaluation of various RVs at different penetration rates
(IDM* denotes the $100\%$ HV baseline, Ours highlighted in bold with results
of the efficiency RV shown in Efficiency column and the results of safety RV
shown in Safety column). Across all penetration rates, our safety RV
consistently outperforms other methods in terms of safety, exceeding the
critical $4~{}s$ TTC threshold and reducing DRAC by up to $80\%$ compared to
IDM, with the exception of $40\%$ penetration where it achieves the second-
best performance in DRAC. Our efficiency RV improves the throughput by up to
$49\%$ (at $60\%$ penetration) compared to IDM, while consistently maintaining
the second-highest fuel economy among all evaluated RVs, with an improvement
of up to $84\%$ at $40\%$ penetration. All results are averaged values over
$10$ randomized rollouts.

Table II presents comparative evaluation of various RVs in the Ring at
$85~{}veh/km$ density. Across all other penetration rates, our safety RV
consistently outperforms other methods in terms of TTC. Importantly, only our
RV manages to exceed the critical $4~{}s$ threshold for TTC at all penetration
rates, a threshold that is often used to activate automatic collision
avoidance systems or warn drivers [47] and has alse been recommended by
earlier studies [48, 44]. Additionally, DRAC is reduced by up to $80\%$ (at
$5\%$ penetration) in comparison to IDM and Our safety RV delivers the lowest
DRAC at all penetrations except at $40\%$. Similarly, Our efficiency RV
achieves the highest throughput at higher penetration rates (improvements of
$44\%$ and $49\%$ at penetration rates $40\%$ and $60\%$ respectively,
compared to IDM). Furthermore, Our efficiency RV improves fuel economy up to
$84\%$ (at $60\%$ penetration) and is consistently the second highest among
all other RVs at all penetration rates.

##  IV Conclusion and Future Work

In this work, we propose a novel approach for optimizing safety and efficiency
in mixed traffic using reinforcement learning-based robot vehicles (RVs). We
introduce CARL with two classes of RVs: safety RV and efficiency RV, both
leveraging a congestion classifier to predict upcoming traffic states and take
preemptive actions to mitigate congestion and smoothen traffic flow. Through
extensive experiments in the Ring with injected real-world perturbations, we
demonstrate our RV is able to increase the time to collision (TTC) above the
critical $4~{}s$ threshold, reduce the deceleration rate to avoid a crash
(DRAC) by up to $80\%$, and increase throughput by up to $49\%$.

In the future, first, we plan to incorporate additional vehicle dynamics such
as lane-changing [49] and heterogeneous vehicle types as well as study the
generalizability of our RVs in more complex mixed traffic environments such as
intersections [9]. Second, we would like to integrate the insights of CARL
with existing techniques [50, 51, 52, 53, 54] to further enable the learning,
planning, and robustness of robot vehicles. Further, we plan to perform
control and coordination at city-wide scale by relying on large-scale traffic
simulation and prediction, and evaluate the robustness of such approaches [55,
56, 57, 58, 59, 60, 61, 62, 63, 64]. Similarly, applying our approach on real
hardware including micro-mobility vehicles [65] also serve as interesting
future directions.

## References

  * [1] USDOT’s automated vehicles comprehensive plan, 2024.
  * [2] U.S. Environmental Protection Agency’ green vehicle guide on self-driving vehicles, 2024.
  * [3] Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer.  Multi-agent imitation learning for driving simulation. in 2018 ieee.  In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1534–1539.
  * [4] Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer.  Imitating driver behavior with generative adversarial networks.  In 2017 IEEE intelligent vehicles symposium (IV), pages 204–211. IEEE, 2017.
  * [5] Xuan Di and Rongye Shi.  A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to ai-guided driving policy learning.  Transportation research part C: emerging technologies, 125:103008, 2021.
  * [6] Michael Villarreal, Bibek Poudel, Jia Pan, and Weizi Li.  Mixed traffic control and coordination from pixels.  In IEEE International Conference on Robotics and Automation (ICRA), 2024.
  * [7] Michael Villarreal, Dawei Wang, Jia Pan, and Weizi Li.  Analyzing emissions and energy efficiency in mixed traffic control at unsignalized intersections.  In IEEE Forum for Innovative Sustainable Transportation Systems (FISTS), 2024.
  * [8] Michael Villarreal, Bibek Poudel, and Weizi Li.  Can chatgpt enable its? the case of mixed traffic control via reinforcement learning.  In IEEE International Conference on Intelligent Transportation Systems (ITSC), pages 3749–3755, 2023.
  * [9] Dawei Wang, Weizi Li, Lei Zhu, and Jia Pan.  Learning to control and coordinate mixed traffic through robot vehicles at complex and unsignalized intersections.  arXiv preprint arXiv:2301.05294, 2023.
  * [10] Dawei Wang, Weizi Li, and Jia Pan.  Large-scale mixed traffic control using dynamic vehicle routing and privacy-preserving crowdsourcing.  IEEE Internet of Things Journal, 11(2):1981–1989, 2024.
  * [11] Bibek Poudel, Weizi Li, and Kevin Heaslip.  Endurl: Enhancing safety, stability, and efficiency of mixed traffic under real-world perturbations via reinforcement learning.  arXiv preprint, 2024.
  * [12] Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemysław Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, et al.  Urban driving with conditional imitation learning.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 251–257. IEEE, 2020.
  * [13] Duo Zhang, Xiaoyun Chen, Junhua Wang, Yinhai Wang, and Jian Sun.  A comprehensive comparison study of four classical car-following models based on the large-scale naturalistic driving experiment.  Simulation Modelling Practice and Theory, 113:102383, 2021.
  * [14] Nathan Lichtlé, Eugene Vinitsky, Matthew Nice, Benjamin Seibold, Dan Work, and Alexandre M Bayen.  Deploying traffic smoothing cruise controllers learned from trajectory data.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2884–2890. IEEE, 2022.
  * [15] Peter G Gipps.  A behavioural car-following model for computer simulation.  Transportation research part B: methodological, 15(2):105–111, 1981.
  * [16] Stefan Krauß.  Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics.  1998\.
  * [17] Martin Treiber and Arne Kesting.  Traffic flow dynamics.  Traffic Flow Dynamics: Data, Models and Simulation, Springer-Verlag Berlin Heidelberg, pages 983–1000, 2013.
  * [18] Saleh Albeaik, Alexandre Bayen, Maria Teresa Chiri, Xiaoqian Gong, Amaury Hayat, Nicolas Kardous, Alexander Keimer, Sean T McQuade, Benedetto Piccoli, and Yiling You.  Limitations and improvements of the intelligent driver model (idm).  SIAM Journal on Applied Dynamical Systems, 21(3):1862–1892, 2022.
  * [19] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner.  Microscopic traffic simulation using sumo.  In 2018 21st international conference on intelligent transportation systems (ITSC), pages 2575–2582. IEEE, 2018.
  * [20] Martin Treiber and Arne Kesting.  The intelligent driver model with stochasticity-new insights into traffic flow oscillations.  Transportation research procedia, 23:174–187, 2017.
  * [21] Arne Kesting and Martin Treiber.  Calibrating car-following models by using trajectory data: Methodological study.  Transportation Research Record, 2088(1):148–156, 2008.
  * [22] Li Li, Xiqun Micheal Chen, and Lei Zhang.  A global optimization algorithm for trajectory data based car-following model calibration.  Transportation Research Part C: Emerging Technologies, 68:311–332, 2016.
  * [23] MN Sharath and Nagendra R Velaga.  Enhanced intelligent driver model for two-dimensional motion planning in mixed traffic.  Transportation Research Part C: Emerging Technologies, 120:102780, 2020.
  * [24] Meixin Zhu, Xuesong Wang, Andrew Tarko, et al.  Modeling car-following behavior on urban expressways in shanghai: A naturalistic driving study.  Transportation research part C: emerging technologies, 93:425–445, 2018.
  * [25] Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen.  Flow: Architecture and benchmarking for reinforcement learning in traffic control.  arXiv preprint arXiv:1710.05465, 10, 2017.
  * [26] Abdul Rahman Kreidieh, Cathy Wu, and Alexandre M Bayen.  Dissipating stop-and-go waves in closed and open networks via deep reinforcement learning.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 1475–1480. IEEE, 2018.
  * [27] Fang-Chieh Chou, Alben Rome Bagabaldo, and Alexandre M Bayen.  The lord of the ring road: a review and evaluation of autonomous control policies for traffic in a ring road.  ACM Transactions on Cyber-Physical Systems (TCPS), 6(1):1–25, 2022.
  * [28] Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen.  Benchmarks for reinforcement learning in mixed-autonomy traffic.  In Conference on robot learning, pages 399–409. PMLR, 2018.
  * [29] Mayuri Sridhar and Cathy Wu.  Piecewise constant policies for human-compatible congestion mitigation.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2499–2505. IEEE, 2021.
  * [30] Xiao Wang, Rui Jiang, Li Li, Yilun Lin, Xinhu Zheng, and Fei-Yue Wang.  Capturing car-following behaviors by deep learning.  IEEE Transactions on Intelligent Transportation Systems, 19(3):910–920, 2017.
  * [31] Meixin Zhu, Yinhai Wang, Ziyuan Pu, Jingyun Hu, Xuesong Wang, and Ruimin Ke.  Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving.  Transportation Research Part C: Emerging Technologies, 117:102662, 2020.
  * [32] Meixin Zhu, Xuesong Wang, and Yinhai Wang.  Human-like autonomous car-following model with deep reinforcement learning.  Transportation research part C: emerging technologies, 97:348–368, 2018.
  * [33] Tianyu Shi, Yifei Ai, Omar ElSamadisy, and Baher Abdulhai.  Bilateral deep reinforcement learning approach for better-than-human car following model.  arXiv preprint arXiv:2203.04749, 2022.
  * [34] Raunak Bhattacharyya, Blake Wulfe, Derek J Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel J Kochenderfer.  Modeling human driving behavior through generative adversarial imitation learning.  IEEE Transactions on Intelligent Transportation Systems, 24(3):2874–2887, 2022.
  * [35] Tianya Zhang, Peter J Jin, Sean T McQuade, and Benedetto Piccoli.  Car-following models: A multidisciplinary review.  arXiv preprint arXiv:2304.07143, 2023.
  * [36] Luc Le Mero, Dewei Yi, Mehrdad Dianati, and Alexandros Mouzakitis.  A survey on imitation learning techniques for end-to-end autonomous vehicles.  IEEE Transactions on Intelligent Transportation Systems, 23(9):14128–14147, 2022.
  * [37] Derek Gloudemans, Yanbing Wang, Junyi Ji, Gergely Zachar, William Barbour, Eric Hall, Meredith Cebelak, Lee Smith, and Daniel B Work.  I-24 motion: An instrument for freeway traffic science.  Transportation Research Part C: Emerging Technologies, 155:104311, 2023.
  * [38] Raphael E Stern, Shumo Cui, Maria Laura Delle Monache, Rahul Bhadani, Matt Bunting, Miles Churchill, Nathaniel Hamilton, Hannah Pohlmann, Fangyu Wu, Benedetto Piccoli, et al.  Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments.  Transportation Research Part C: Emerging Technologies, 89:205–221, 2018.
  * [39] Berthold KP Horn.  Suppressing traffic flow instabilities.  In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), pages 13–20. IEEE, 2013.
  * [40] Rajesh Rajamani.  Vehicle dynamics and control.  Springer Science & Business Media, 2011.
  * [41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [42] Hwasoo Yeo.  Asymmetric microscopic driving behavior theory.  University of California, Berkeley, 2008.
  * [43] Cathy Wu, Abdul Rahman Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen.  Flow: A modular learning framework for mixed autonomy traffic.  IEEE Transactions on Robotics, 38(2):1270–1286, 2021.
  * [44] Katja Vogel.  A comparison of headway and time to collision as safety indicators.  Accident analysis & prevention, 35(3):427–433, 2003.
  * [45] Dale F Cooper and N Ferguson.  Traffic studies at t-junctions. 2. a conflict simulation record.  Traffic Engineering & Control, 17(Analytic), 1976.
  * [46] Peter De Haan and Mario Keller.  Modelling fuel consumption and pollutant emissions based on real-world driving patterns: the hbefa approach.  International journal of environment and pollution, 22(3):240–258, 2004.
  * [47] Michiel M Minderhoud and Piet HL Bovy.  Extended time-to-collision measures for road traffic safety assessment.  Accident Analysis & Prevention, 33(1):89–97, 2001.
  * [48] TJ Ayres, L Li, David Schleuning, and D Young.  Preferred time-headway of highway drivers.  In ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No. 01TH8585), pages 826–829. IEEE, 2001.
  * [49] Dawei Wang, Weizi Li, Lei Zhu, and Jia Pan.  Learning to change: Choreographing mixed traffic through lateral control and hierarchical reinforcement learning.  arXiv preprint arXiv:2403.14879, 2024.
  * [50] Weizi Li, David Wolinski, and Ming C. Lin.  ADAPS: Autonomous driving via principled simulations.  In IEEE International Conference on Robotics and Automation (ICRA), pages 7625–7631, 2019.
  * [51] Yu Shen, Weizi Li, and Ming C. Lin.  Inverse reinforcement learning with hybrid-weight trust-region optimization and curriculum learning for autonomous maneuvering.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7421–7428, 2022.
  * [52] Lei Lin, Weizi Li, Huikun Bi, and Lingqiao Qin.  Vehicle trajectory prediction using LSTMs with spatial-temporal attention mechanisms.  IEEE Intelligent Transportation Systems Magazine, 14(2):197–208, 2022.
  * [53] Yu Shen, Laura Zheng, Manli Shu, Weizi Li, Tom Goldstein, and Ming C. Lin.  Gradient-free adversarial training against image corruption for learning-based steering.  In Advances in Neural Information Processing Systems (NeurIPS), pages 26250–26263, 2021.
  * [54] Michael Villarreal, Bibek Poudel, Ryan Wickman, Yu Shen, and Weizi Li.  Autojoin: Efficient adversarial training for robust maneuvering via denoising autoencoder and joint learning.  2023\.
  * [55] David Wilkie, Jason Sewall, Weizi Li, and Ming C. Lin.  Virtualized traffic at metropolitan scales.  Frontiers in Robotics and AI, 2:11, 2015.
  * [56] Weizi Li, David Wolinski, and Ming C. Lin.  City-scale traffic animation using statistical learning and metamodel-based optimization.  ACM Trans. Graph., 36(6):200:1–200:12, 2017.
  * [57] Weizi Li, Dong Nie, David Wilkie, and Ming C. Lin.  Citywide estimation of traffic dynamics via sparse GPS traces.  IEEE Intelligent Transportation Systems Magazine, 9(3):100–113, 2017.
  * [58] Weizi Li, Meilei Jiang, Yaoyu Chen, and Ming C. Lin.  Estimating urban traffic states using iterative refinement and wardrop equilibria.  IET Intelligent Transport Systems, 12(8):875–883, 2018.
  * [59] Lei Lin, Weizi Li, and Srinivas Peeta.  Efficient data collection and accurate travel time estimation in a connected vehicle environment via real-time compressive sensing.  Journal of Big Data Analytics in Transportation, 1(2):95–107, 2019.
  * [60] Lei Lin, Weizi Li, and Srinivas Peeta.  Predicting station-level bike-sharing demands using graph convolutional neural network.  In Transportation Research Board 98th Annual Meeting (TRB), 2019.
  * [61] Qianwen Chao, Huikun Bi, Weizi Li, Tianlu Mao, Zhaoqi Wang, Ming C. Lin, and Zhigang Deng.  A survey on visual traffic simulation: Models, evaluations, and applications in autonomous driving.  Computer Graphics Forum, 39(1):287–308, 2020.
  * [62] Bibek Poudel and Weizi Li.  Black-box adversarial attacks on network-wide multi-step traffic state prediction models.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3652–3658. IEEE, 2021.
  * [63] Lei Lin, Weizi Li, and Lei Zhu.  Data-driven graph filter based graph convolutional neural network approach for network-level multi-step traffic prediction.  Sustainability, 14(24):16701, 2022.
  * [64] Ke Guo, Zhenwei Miao, Wei Jing, Weiwei Liu, Weizi Li, Dayang Hao, and Jia Pan.  Lasil: Learner-aware supervised imitation learning for long-term microscopic traffic simulation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  * [65] Bibek Poudel, Thomas Watson, and Weizi Li.  Learning to control dc motor for micromobility in real time with reinforcement learning.  In IEEE International Conference on Intelligent Transportation Systems (ITSC), pages 1248–1254, 2022.