# A Decentralized Reinforcement Learning Framework for Efficient Passage of
Emergency Vehicles

Haoran Su† , Yaofeng Desmond Zhong§, Biswadip Dey§, Amit Chakraborty§
† New York University | § Siemens Corporation, Technology
<EMAIL_ADDRESS>(yaofeng.zhong, biswadip.dey<EMAIL_ADDRESS>This study is conducted with his internship with Siemens. He is also sponsored
by DDETFP by U.S. Department of Transportation Federal Highway Administration.

###### Abstract

Emergency vehicles (EMVs) play a critical role in a city’s response to time-
critical events such as medical emergencies and fire outbreaks. The existing
approaches to reduce EMV travel time employ route optimization and traffic
signal pre-emption without accounting for the coupling between route these two
subproblems. As a result, the planned route often becomes suboptimal. In
addition, these approaches also do not focus on minimizing disruption to the
overall traffic flow. To address these issues, we introduce EMVLight in this
paper. This is a decentralized reinforcement learning (RL) framework for
simultaneous dynamic routing and traffic signal control. EMVLight extends
Dijkstra’s algorithm to efficiently update the optimal route for an EMV in
real-time as it travels through the traffic network. Consequently, the
decentralized RL agents learn network-level cooperative traffic signal phase
strategies that reduce EMV travel time and the average travel time of non-EMVs
in the network. We have carried out comprehensive experiments with synthetic
and real-world maps to demonstrate this benefit. Our results show that
EMVLight outperforms benchmark transportation engineering techniques as well
as existing RL-based traffic signal control methods.

##  1 Introduction

As any reduction in the travel time of an EMV increases the likelihood that it
would save someone’s life or reduce property loss, there is a significant
benefit in reducing the average EMV travel time on increasingly crowded roads.
To tackle this challenge, existing approaches rely upon – route optimization
which uses graph-theoretic tools to find the fastest path; and traffic signal
pre-emption which alters the traffic signals to prioritize EMV passage and
improves overall safety at the intersections. Please see Section S1 for
additional details about the existing approaches on route optimization and
traffic signal control. However, the constantly changing traffic conditions
can significantly affect the optimality of a route prescribed by route
optimization. In addition, traffic signal pre-emption also changes the optimal
route by causing serious disruptions to the traffic flow.

To address these issues and achieve efficient EMV passage, we introduce
EMVLight which is a decentralized multi-agent RL framework with a dynamic
routing algorithm to control traffic signal phases. We introduce a
computationally efficient dynamic routing algorithm and an RL-based traffic
signal pre-emption strategy to not only reduce _EMV travel time_ but also to
reduce the _average travel time of non-EMVs_. Our experiments demonstrate that
EMVLight outperforms traditional traffic engineering methods and existing RL
methods on both metrics under different traffic configurations.

##  2 Application Context

Emergency response time is the key indicator of a city’s emergency management
capability. Reducing response time saves lives and prevents property loss. For
instance, the survivor rate from a sudden cardiac arrest without treatment
drops 7% - 10% for every minute elapsed, and there is barely any chance to
survive after 8 minutes [1]. The time interval for an _emergency vehicle
(EMV)_ , e.g., ambulances, fire trucks, and police cars, to travel from a
rescue station to an incidence site, i.e., the EMV travel time, accounts for a
significant portion of the emergency response time. However, overpopulation
and urbanization have been exacerbating road congestion, making it more and
more challenging to reduce the average EMV travel time [2]. There is a severe
urgency and significant benefit in reducing the average EMV travel time on
increasingly crowded roads.

##  3 Preliminaries

Traffic map, Link, Lane: A traffic map can be represented by a graph
$G(\mathcal{V},\mathcal{E})$, with intersections as

Figure 1: This figure illustrates traffic movements through an intersection
and an example pressure calculation for incoming lane #2. Vehicles on Lane 1
are turning left, and vehicles on Lane 2 may either go straight or turn right.
After turning right, vehicles can enter either lane. Thus, the incoming link
from South has the following traffic movement: (1, 7), (1, 8), (2, 5), (2, 6),
(2, 3), (2, 4). Assuming each lane has the maximum capacity of 5, the lane
pressure for Lane 2 can then be calculated as:
$w(2)=|1/5-(1/2)*(1/5+2/5)-(1/2)*(3/5+0/5)|=0.4$.

nodes and road segments between intersections as edges. We refer to a one-
directional road segment between two intersections as a link. A link has a
fixed number of lanes, denoted as $h(l)$ for lane $l$. Fig. 1 shows 8 links
and each link has 2 lanes.

Figure 2: _Top_ panel of this figure shows the 8 signal phases for a 4-way
intersection. The _bottom-left_ and _bottom-right_ panels show phase 2 and 5,
respectively.

Traffic Movements: A traffic movement $(l,m)$ is defined as the traffic
traveling across an intersection from an incoming lane $l$ to an outgoing lane
$m$. The intersection shown in Fig. 1 has 24 permissible traffic movements.
The set of all permissible traffic movements of an intersection is denoted as
$\mathcal{M}$.

Traffic Signal Phase: A traffic signal phase is defined as the set of
permissible traffic movements. As shown in Fig. 2, an intersection with 4
links has 8 phases.

Pressure of an incoming lane: The pressure of an incoming lane $l$ measures
the unevenness of vehicle density between lane $l$ and corresponding outgoing
lanes in permissible traffic movements. The vehicle density of a lane is
$x(l)/x_{max}(l)$, where $x(l)$ is the number of vehicles on lane $l$ and
$x_{max}(l)$ is the vehicle capacity on lane $l$, which is related to the
length of a lane. Then the pressure

of an incoming lane $l$ is

$w(l)=\left|\frac{x(l)}{x_{max}(l)}-\sum_{\\{m|(l,m)\in\mathcal{M}\\}}\frac{1}{h(m)}\frac{x(m)}{x_{max}(m)}\right|,$

where $h(m)$ is the number of lanes of the outgoing link which contains $m$.
This concept has been explained in Fig. 1 wherein $h(m)=2$ for all the
outgoing lanes.

Pressure of an intersection: The pressure of an intersection ($P$) is the
average of the pressure of all incoming lanes. It indicates the unevenness of
vehicle density between incoming and outgoing lanes in an intersection.
Intuitively, reducing the pressure leads to more evenly distributed traffic,
which indirectly reduce congestion and average travel time of vehicles.

##  4 Dynamic Routing

A traffic network can be perceived as a weighted graph with the weights
representing the _intra-link travel time_ , i.e., the EMV travel time along
each link which has been estimated from the number of vehicles on that link.
This enables the use of Dijkstra’s algorithm for finding solutions to the EMV
routing problem. However, continuously changing traffic conditions keep
changing the intra-link travel time and thereby render the weighted graph
dynamic. Although this challenge can be addressed by running Dijkstra’s
algorithm repeatedly as the EMV travels through the network, it is not viable
from a computational perspective. In this work, we extend Dijkstra’s algorithm
to efficiently update the optimal route based on the most recent information
about the intra-link travel times (Algorithm 1). We first run Dijkstra’s
algorithm to obtain the estimated time of arrival (ETA) from each intersection
to the destination and the next intersection along the shortest path. Since a
sequence of processes, including call-taker processing, are performed before
an EMVs is dispatched, it is reasonable to assume that this process can be
done before an EMV starts to travel. Once an EMV has started its journey, we

Input :

$G=(\mathcal{V},\mathcal{E})$ | traffic map as a graph
---|---
$T^{t}=[T_{ij}^{t}]$ | intra-link travel time at time $t$
$i_{d}$ | index of the destination

Output :

$\mathsf{ETA}^{t}=[\mathsf{ETA}^{t}_{i}]$ | ETA of each intersection
---|---
$\mathsf{Next}^{t}=[\mathsf{Next}^{t}_{i}]$ | next intersection to go
| from each intersection

/* pre-population */

$\mathsf{ETA}^{0},\mathsf{Next}^{0}$ $=$ Dijkstra$(G,T^{0},i_{d})$;

/* dynamic routing */

for  _$t=0\to T$_ do

foreach  _$i\in\mathcal{V}$_ do (in parallel)

$\mathsf{ETA}_{i}^{t+1}\leftarrow\min_{(i,j)\in\mathcal{E}}(\mathsf{ETA}_{j}^{t}+T_{ji}^{t})$;

$\mathsf{Next}_{i}^{t+1}\leftarrow\arg\min_{\\{j|(i,j)\in\mathcal{E}\\}}(\mathsf{ETA}_{j}^{t}+T_{ji}^{t}$)

Algorithm 1 Dynamic Dijkstra’s for EMV routing

update $\mathsf{ETA}$ and $\mathsf{Next}$ values for each intersection. As
these updates only depend on information about neighboring intersections, they
are computationally inexpensive and can be executed in parallel. Please see
Section S5 for further details about how intra-link travel time is estimated
in real time.

##  5 Setting up the RL Problem

While dynamic routing directs the EMV to the destination, it does not take
into account the possible wait times due to red lights at the intersections.
Thus, traffic signal pre-emption is also required for the EMV to arrive at the
destination in the least amount of time. However, since traditional pre-
emption only focuses on reducing the EMV travel time, it can significantly
increase the average travel time of non-EMVs. In this work, we set up traffic
signal control for efficient EMV passage as a decentralized RL problem. In our
formulation, we employ multiple RL agents (one for each intersection) that
work cooperatively to reduce both EMV travel time and the average travel time
of non-EMVs.

###  5.1 Types of agents for EMV passage

Whenever an EMV is travelling through the traffic network, the agents that
control the traffic signal at the intersections are grouped into 3 distinct
categories based on EMV location and routing (Fig. 3). An agent is categorized
as a _primary pre-emption agent_ $i_{p}$ if an EMV is travelling on one of its
incoming links. The agent at the next intersection along the optimal route is
categorized as a _secondary pre-emption agent_ , i.e.
$i_{s}=\mathsf{Next}_{i_{p}}$. The rest of the agents are _normal agents_.
Each of these agent types has different local goals which is captured into
their reward functions.

###  5.2 Agent design

Figure 3: EMVLight employs 3 types of agents.

$\bullet$ State: The state of an agent $i$ at time $t$ includes the number of
vehicles on each of the incoming and outgoing lanes at intersection $i$, the
distance of the EMV to the intersection, the estimated time of arrival
($\mathsf{ETA}$), and which link the EMV will be routed to ($\mathsf{Next}$).
Therefore, we have

$s^{t}_{i}=\\{x^{t}(l),x^{t}(m),d^{t}_{\text{EMV}}[L_{ji}],\mathsf{ETA}^{t^{\prime}}_{i},\mathsf{Next}^{t^{\prime}}_{i}\\},$

where $L_{ji}$ denotes the links incoming to intersection $i$, and with a
slight abuse of notation $l$ and $m$ denote the set of incoming and outgoing
lanes, respectively. For primary pre-emption agents, one of the elements of
$d^{t}_{\text{EMV}}$ represents the distance of the EMV to the intersection in
the corresponding link and the rest of the elements are set to -1. For
secondary pre-emption agents and normal agents, every element of
$d^{t}_{\text{EMV}}$ is set to -1.
$\bullet$ Action: We define the action of an agent as one of the 8 signal
phases shown in Fig. 2. This enables more flexible signal patterns as compared
to the traditional cyclical patterns. Also, due to safety concerns, once a
signal phase has been initiated, it remains unchanged for certain minimum
amount of time.
$\bullet$ Reward: PressLight has shown that minimizing the pressure can
encourage efficient vehicle passage [3]; here we adopt a similar approach for
the normal agents. For secondary pre-emption agents, we additionally encourage
less vehicle on the link where the EMV is about to enter in order to encourage
efficient EMV passage. For primary pre-emption agents, we assign a unit
penalty at each time step to encourage fast EMV passage. Thus, the local
reward for agent $i$ at time $t$ is defined as

$r_{i}^{t}=\left\\{\begin{array}[]{ll}-P_{i}^{t}&\quad
i\notin\\{i_{p},i_{s}\\}\textrm{, i.e., Normal Agent},\\\ -\beta
P_{i}^{t}-\frac{1-\beta}{|L_{i_{p}i}|}\sum\limits_{l\in
L_{i_{p}i}}\frac{x(l)}{x_{max}(l)}&\quad i=i_{s}\textrm{, i.e., Secondary pre-
emption agent,}\\\ -1&\quad i=i_{p}\textrm{, i.e., Primary pre-emption
agent.}\end{array}\right.$ (1)

###  5.3 Multi-agent Advantage Actor-critic

To realize the decentalized RL framework, we adopt a multi-agent advantage
actor-critic (MA2C) approach that is similar to [4]. In this formulation, the
RL agent at intersection $i$ learns a policy $\pi_{\theta_{i}}$ (actor) and
the corresponding value function $V_{\phi_{i}}$ (critic), where ${\theta_{i}}$
and ${\phi_{i}}$ are learnable neural network parameters. In our formulation,
the local state includes dynamic routing information and the local reward
encourages efficient passage of the EMV, whereas [4] focuses on a general
traffic signal control problem with no EMV considered. Please see Section S3
for additional details.

##  6 Experiments and Results

To evaluate performance of our proposed EMVLight framework, we carry out
experiments using the _SUMO_ [5] simulation tool that can simulate traffic
dynamics in both microscopic and macroscopic settings. In our experiments, the
decentralized RL agents collect observations from SUMO and the preferred
signal phases are then fed back into the traffic simulator.

###  6.1 Datasets and Maps Descriptions

We carry out our experiments using both synthetic and real-world maps.

Synthetic Grid: We first consider a synthetic grid of size $5\times 5$, where
the intersections are connected with bi-directional links with each link
containing two lanes (Fig. 4a). For the synthetic map, we consider two
different traffic flow configurations with peak/non-peak flow values of
240/200 _veh/lane/hr_ and 320/120 _veh/lane/hr_. In both configurations,
vehicles enter the grid via north/south-bound links and leave the grid via
east/west-bound links. The traffic for this map has a time span of 1200s and
the EMV is dispatched at $t=600s$ to ensure roads are compacted before the EMV
starts travelling; also, the peak flow occurs between 400s-800s.

Manhattan Map: This is a $16\times 3$ traffic network extracted from Manhattan
Hell’s Kitchen area (Fig. 4b). In this traffic network, intersections are
connected by 16 one-way 2-lane streets and 3 one-way 4-lane avenues. The
traffic flow for this map is generated from open-source NYC taxi data
(_https://traffic-signal-control.github.io/_).

Figure 4: _Left_ : the synthetic $\text{grid}_{5\times 5}$. _Right_ : an
intersection illustration in SUMO, the teal area are inductive loop detected
area. Origin and destination for EMV are labeled. Manhattan map: a 16-by-3
traffic network in Hell’s Kitchen area. Origin and destination for the EMV
dispatching are labeled as “O" and “D", respectively.

###  6.2 Baselines

As RL-based approaches for efficient EMV passage do not yet exist, we select
traditional methods and RL methods for individual subproblems and combine them
to set up baselines. For traffic signal pre-emption, we use _Walabi_ [6], a
rule-based method that implements _Green Wave_ [7] for EMVs. We then integrate
Walabi (_W_) with combinations of routing and _traffic signal control_
strategies. For routing, we consider 2 baselines – a _Static_ routing based on
A*; and a _dynamic_ routing by running A* every $50s$. For controlling the
traffic signals, we consider the following 4 baseline approaches – a cyclical
_fixed time (FT)_ traffic phases with random offset that splits all phases
with an predefined green ratio [8]; a network-level signal control strategy
based on _max pressure (MP)_ that smoothens congestion by aggressively
selecting the phase with the maximum pressure [9]; a Q-learning based
_coordinated learner (CL)_ that directly learns joint local value functions
for adjacent intersections [10]; and finally _PressLight (PL)_ which uses RL
to optimize the pressure at each intersection [3].

###  6.3 Results

We evaluate performance of models under two metrics: _EMV travel time_ , which
reflects routing and pre-emption ability, and _average travel time_ , which
indicates the ability of traffic signal control for efficient vehicle passage.
The performance of our EMVLight and the baselines in both the synthetic and
the Manhattan map is shown in Table 1. The results of all methods are averaged
over five independent runs and RL methods are tested with random seeds. We
observe that EMVLight outperforms all baseline models under both metrics.

Method | EMV Travel Time [s] | Average Travel Time [s]
---|---|---
Config 1 | Config 2 | Mahanttan | Config 1 | Config 2 | Mahanttan
No EMV + FT | N/A | N/A | N/A | 353.43 | 371.13 | 1649.64
W + Static + FT | 257.20 | 272.00 | 487.20 | 372.19 | 389.13 | 1711.03
W + Static + MP | 255.00 | 269.00 | 461.80 | 349.38 | 352.54 | 708.13
W + Static + GRL | 281.20 | 286.20 | 492.20 | 503.35 | 524.26 | 2013.54
W + Static + PL | 276.00 | 282.20 | 476.00 | 358.18 | 369.45 | 1410.76
W + Dynamic + FT | 229.60 | 231.20 | 442.60 | 370.09 | 293.40 | 1699.30
W + Dynamic + MP | 226.20 | 234.60 | 438.80 | 345.45 | 348.43 | 721.32
W + Dynamic + GRL | 273.40 | 269.60 | 450.20 | 514.29 | 536.78 | 1987.86
W + Dynamic + PL | 251.20 | 257.80 | 436.20 | 359.31 | 342.59 | 1412.12
EMVLight | 198.60 | 192.20 | 391.80 | 322.40 | 318.76 | 681.23
Table 1: _Performance comparison of different methods evaluated on four
configurations in the synthetic traffic grid as well as Manhattan Map. For
both metrics, the lower value indicates better performance. The lowest values
are highlighted in bold. The average travel time of Manhattan map (1649.64) is
retrieved from data._

In terms of EMV travel time $T_{\textrm{EMV}}$, the dynamic routing baseline
performs better than static routing baselines. This is expected since dynamic
routing considers the time-dependent nature of traffic conditions and update
optimal route accordingly. EMVLight further reduces EMV travel time by 18% in
average as compared to dynamic routing baselines. This advantage in
performance can be attributed to the design of secondary pre-emption agents.
This type of agents learn to “reserve a link" by choosing signal phases that
help clear the vehicles in the link to encourage high speed EMV passage (Eqn.
(1)).

As for average travel time $T_{\textrm{avg}}$, we first notice that the
traditional pre-emption technique (W+Static+FT) indeed increases the average
travel time by around 10% as compared to a traditional Fix Time strategy
without EMV (denoted as “FT w/o EMV" in Table 1), thus decreasing the
efficiency of vehicle passage. Different traffic signal control strategies
have a direct impact on overall efficiency. Fixed Time is designed to handle
steady traffic flow. Max Pressure, as a SOTA traditional method, outperforms
Fix Time and, surprisingly, outperforms both RL baselines in terms of overall
efficiency. This shows that pressure is an effective indicator for reducing
congestion and this is why we incorporate pressure in our reward design.
Coordinate Learner performs the worst probably because its reward is not based
on pressure. PressLight doesn’t beat Max Pressure because it has a reward
design that focuses on smoothing vehicle densities along a major direction,
e.g. an arterial. Grid networks with the presence of EMV make PressLight less
effective. Our EMVLight improves its pressure-based reward design to encourage
smoothing vehicle densities of all directions for each intersection. This
enable us to achieve an advantage of 5% over our best baselines (Max
Pressure).

##  7 Conclusion

In this paper, we proposed a decentralized reinforcement learning framework,
EMVLight, to facilitate the efficient passage of EMVs and reduce traffic
congestion at the same time. Leveraging the multi-agent A2C framework, agents
incorporate dynamic routing and cooperatively control traffic signals to
reduce EMV travel time and average travel time of non-EMVs. Evaluated on both
synthetic and real-world map, EMVLight significantly outperforms the existing
methods. Future work will explore more realistic microscopic interaction
between EMV and non-EMVs, efficient passage of multiple EMVs and closing the
sim-to-real gap.

## References

  * [1] G. Nichol, E. Thomas, C. W. Callaway, J. Hedges, J. L. Powell, T. P. Aufderheide, T. Rea, R. Lowe, T. Brown, J. Dreyer _et al._ , “Regional variation in out-of-hospital cardiac arrest incidence and outcome,” _Jama_ , vol. 300, no. 12, pp. 1423–1431, 2008.
  * [2] N. Analytics, “End-to-end response times,” https://www1.nyc.gov/site/fdny/about/resources/data-and-analytics/end-to-end-response-times.page, 2021\.
  * [3] H. Wei, C. Chen, G. Zheng, K. Wu, V. Gayah, K. Xu, and Z. Li, “Presslight: Learning max pressure control to coordinate traffic signals in arterial network,” in _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 1290–1298.
  * [4] T. Chu, J. Wang, L. Codecà, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,” _IEEE Transactions on Intelligent Transportation Systems_ , 2019.
  * [5] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic traffic simulation using sumo,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 2575–2582.
  * [6] L. Bieker-Walz and M. Behrisch, “Modelling green waves for emergency vehicles using connected traffic data,” _EPiC Series in Computing_ , vol. 62, pp. 1–11, 2019.
  * [7] F. Corman, A. D’Ariano, D. Pacciarelli, and M. Pranzo, “Evaluation of green wave policy in real-time railway traffic management,” _Transportation Research Part C: Emerging Technologies_ , vol. 17, no. 6, pp. 607–616, 2009.
  * [8] R. P. Roess, E. S. Prassas, and W. R. McShane, _Traffic engineering_. Pearson/Prentice Hall, 2004\.
  * [9] P. Varaiya, “Max pressure control of a network of signalized intersections,” _Transportation Research Part C: Emerging Technologies_ , vol. 36, pp. 177–195, 2013.
  * [10] E. Van der Pol and F. A. Oliehoek, “Coordinated deep reinforcement learners for traffic light control,” _Proceedings of Learning, Inference and Control of Multi-Agent Systems (at NIPS 2016)_ , 2016.
  * [11] J. Wang, W. Ma, and X. Yang, “Development of degree-of-priority based control strategy for emergency vehicle preemption operation,” _Discrete dynamics in nature and society_ , vol. 2013, 2013.
  * [12] H. Mu, Y. Song, and L. Liu, “Route-based signal preemption control of emergency vehicle,” _Journal of Control Science and Engineering_ , vol. 2018, pp. 1–11, 04 2018.
  * [13] E. Kwon, S. Kim, and R. Betts, “Route-based dynamic preemption of traffic signals for emergency vehicle operations,” in _Transportation Research Board 82nd Annual MeetingTransportation Research Board_ , 2003.
  * [14] A. Jotshi, Q. Gong, and R. Batta, “Dispatching and routing of emergency vehicles in disaster mitigation using data fusion,” _Socio-Economic Planning Sciences_ , vol. 43, no. 1, pp. 1 – 24, 2009.
  * [15] N. A. M. Nordin, Z. A. Zaharudin, M. A. Maasar, and N. A. Nordin, “Finding shortest path of the ambulance routing: Interface of a-star algorithm using c programming,” in _2012 IEEE Symposium on Humanities, Science and Engineering Research_. IEEE, 2012, pp. 1569–1573.
  * [16] A. K. Ziliaskopoulos and H. S. Mahmassani, “Time-dependent, shortest-path algorithm for real-time intelligent vehicle highway system applications,” in _Transportation Research Record_ , 1993, pp. 94–100.
  * [17] G. Musolino, A. Polimeni, C. Rindone, and A. Vitetta, “Travel time forecasting and dynamic routes design for emergency vehicles,” _Procedia-Social and Behavioral Sciences_ , vol. 87, pp. 193–202, 2013.
  * [18] A. Haghani, H. Hu, and Q. Tian, “An optimization model for real-time emergency vehicle dispatching and routing,” in _82nd annual meeting of the Transportation Research Board, Washington, DC_. Citeseer, 2003.
  * [19] S. Koh, B. Zhou, H. Fang, P. Yang, Z. Yang, Q. Yang, L. Guan, and Z. Ji, “Real-time deep reinforcement learning based vehicle navigation,” _Applied Soft Computing_ , vol. 96, p. 106694, 2020.
  * [20] M. Asaduzzaman and K. Vidyasankar, “A priority algorithm to control the traffic signal for emergency vehicles,” in _2017 IEEE 86th Vehicular Technology Conference (VTC-Fall)_ , 2017, pp. 1–7.
  * [21] L. Lu and S. Wang, “Literature review of analytical models on emergency vehicle service: Location, dispatching, routing and preemption control,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , 2019, pp. 3031–3036.
  * [22] S. Humagain, R. Sinha, E. Lai, and P. Ranjitkar, “A systematic review of route optimisation and pre-emption methods for emergency vehicles,” _Transport reviews_ , vol. 40, no. 1, pp. 35–53, 2020.
  * [23] C. You, Q. Yang, L. Gjesteby, G. Li, S. Ju, Z. Zhang, Z. Zhao, Y. Zhang, W. Cong, G. Wang _et al._ , “Structurally-sensitive multi-scale deep neural network for low-dose CT denoising,” _IEEE Access_ , 2018.
  * [24] L. Cheng and C. You, “Hybrid non-linear dimensionality reduction method framework based on random projections,” in _2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA)_. IEEE, 2016, pp. 43–48.
  * [25] G. Li, S. Luo, C. You, M. Getzin, L. Zheng, G. Wang, and N. Gu, “A novel calibration method incorporating nonlinear optimization and ball-bearing markers for cone-beam ct with a parameterized trajectory,” _Medical physics_ , 2019.
  * [26] L. Cheng, C. You, and Y. Guan, “Random projections for non-linear dimensionality reduction,” _International Journal of Machine Learning and Computing_ , vol. 6, no. 4, pp. 220–225, 2016.
  * [27] L. Cheng, C. You, Y. Guan, and Y. Yu, “Body activity recognition using wearable sensors,” in _2017 Computing Conference_. IEEE, 2017, pp. 756–765.
  * [28] C. You, L. Yang, Y. Zhang, and G. Wang, “Low-Dose CT via Deep CNN with Skip Connection and Network in Network,” in _Developments in X-Ray Tomography XII_. International Society for Optics and Photonics, 2019.
  * [29] C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong _et al._ , “CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (gan-circle),” _IEEE Transactions on Medical Imaging_ , 2019.
  * [30] Q. Lyu, C. You, H. Shan, Y. Zhang, and G. Wang, “Super-resolution mri and ct through gan-circle,” in _Developments in X-ray tomography XII_. International Society for Optics and Photonics, 2019.
  * [31] C. You, J. Yang, J. Chapiro, and J. S. Duncan, “Unsupervised wasserstein distance guided domain adaptation for 3d multi-domain liver segmentation,” in _Interpretable and Annotation-Efficient Learning for Medical Image Computing_ , 2020.
  * [32] C. You, N. Chen, and Y. Zou, “Contextualized attention-based knowledge transfer for spoken conversational question answering,” in _INTERSPEECH_ , 2021.
  * [33] C. You, N. Chen, F. Liu, D. Yang, and Y. Zou, “Towards data distillation for end-to-end spoken conversational question answering,” _arXiv preprint arXiv:2010.08923_ , 2020.
  * [34] L. Yang, R. P. Ghosh, J. M. Franklin, S. Chen, C. You, R. R. Narayan, M. L. Melcher, and J. T. Liphardt, “Nuset: A deep learning tool for reliably separating and analyzing crowded cells,” _PLoS computational biology_ , 2020\.
  * [35] N. Chen, F. Liu, C. You, P. Zhou, and Y. Zou, “Adaptive bi-directional attention: Exploring multi-granularity representations for machine reading comprehension,” in _ICASSP_ , 2020.
  * [36] N. Chen, C. You, and Y. Zou, “Self-supervised dialogue learning for spoken conversational question answering,” in _INTERSPEECH_ , 2021.
  * [37] C. You, N. Chen, and Y. Zou, “Knowledge distillation for improved accuracy in spoken question answering,” in _ICASSP_ , 2021.
  * [38] C. You, Y. Zhou, R. Zhao, L. Staib, and J. S. Duncan, “Simcvd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation,” _arXiv preprint arXiv:2108.06227_ , 2021.
  * [39] C. You, N. Chen, and Y. Zou, “MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering,” in _IJCAI_ , 2021.
  * [40] ——, “Self-supervised contrastive cross-modality representation learning for spoken question answering,” in _Findings of the Association for Computational Linguistics: EMNLP_ , 2021.
  * [41] C. You, L. Han, A. Feng, R. Zhao, H. Tang, and W. Fan, “Megan: Memory enhanced graph attention network for space-time video super-resolution,” in _WACV_ , 2022.
  * [42] W. Xu, P. Zhou, C. You, and Y. Zou, “Semantic transportation prototypical network for few-shot intent detection,” in _INTERSPEECH_ , 2021.
  * [43] F. Liu, X. Wu, C. You, S. Ge, Y. Zou, and X. Sun, “Aligning source visual and target language domains for unpaired video captioning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [44] F. Liu, C. You, X. Wu, S. Ge, X. Sun _et al._ , “Auto-encoding knowledge graph for unsupervised medical report generation,” in _NeurIPS_ , 2021.
  * [45] C. You, R. Zhao, F. Liu, S. Chinchali, U. Topcu, L. Staib, and J. S. Duncan, “Class-aware generative adversarial transformers for medical image segmentation,” _arXiv preprint arXiv:2201.10737_ , 2022.
  * [46] B. Abdulhai, R. Pringle, and G. J. Karakoulas, “Reinforcement learning for true adaptive traffic signal control,” _Journal of Transportation Engineering_ , vol. 129, no. 3, pp. 278–285, 2003.
  * [47] L. Prashanth and S. Bhatnagar, “Reinforcement learning with function approximation for traffic signal control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 12, no. 2, pp. 412–421, 2010.
  * [48] H. Wei, N. Xu, H. Zhang, G. Zheng, X. Zang, C. Chen, W. Zhang, Y. Zhu, K. Xu, and Z. Li, “Colight: Learning network-level cooperation for traffic signal control,” in _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 1913–1922.
  * [49] G. Zheng, Y. Xiong, X. Zang, J. Feng, H. Wei, H. Zhang, Y. Li, K. Xu, and Z. Li, “Learning phase competition for traffic signal control,” in _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , 2019, pp. 1963–1972.
  * [50] C. Chen, H. Wei, N. Xu, G. Zheng, M. Yang, Y. Xiong, K. Xu, and Z. Li, “Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, pp. 3414–3421, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/5744
  * [51] X. Zang, H. Yao, G. Zheng, N. Xu, K. Xu, and Z. Li, “Metalight: Value-based meta-reinforcement learning for traffic signal control,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 01, pp. 1153–1160, Apr. 2020.
  * [52] S. El-Tantawy, B. Abdulhai, and H. Abdelgawad, “Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 14, no. 3, pp. 1140–1150, 2013.
  * [53] M. Aslani, M. S. Mesgari, and M. Wiering, “Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events,” _Transportation Research Part C: Emerging Technologies_ , vol. 85, pp. 732–752, 2017.
  * [54] B. Xu, Y. Wang, Z. Wang, H. Jia, and Z. Lu, “Hierarchically and cooperatively learning traffic signal control,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 2021, pp. 669–677.
  * [55] H. Wei, G. Zheng, V. Gayah, and Z. Li, “A survey on traffic signal control methods,” _arXiv preprint arXiv:1904.08117_ , 2019.
  * [56] J. Gajda, R. Sroka, M. Stencel, A. Wajda, and T. Zeglen, “A vehicle classification based on inductive loop detectors,” in _IMTC 2001. Proceedings of the 18th IEEE Instrumentation and Measurement Technology Conference. Rediscovering Measurement in the Age of Informatics_ , vol. 1. IEEE, 2001, pp. 460–464.
  * [57] A. Buchenscheit, F. Schaub, F. Kargl, and M. Weber, “A vanet-based emergency vehicle warning system,” in _2009 IEEE Vehicular Networking Conference (VNC)_. IEEE, 2009, pp. 1–8.
  * [58] Y. Wang, Z. Wu, X. Yang, and L. Huang, “Design and implementation of an emergency vehicle signal preemption system based on cooperative vehicle-infrastructure technology,” _Advances in Mechanical Engineering_ , vol. 5, p. 834976, 2013.
  * [59] H. Noori, L. Fu, and S. Shiravi, “A connected vehicle based traffic signal control strategy for emergency vehicle preemption,” in _Transportation Research Board 95th Annual Meeting, Report Np. 16-6763_ , 2016.
  * [60] H. Su, Z. Ji, K. Johansson, L. Jin _et al._ , “A hybrid queuing model for coordinated vehicle platooning on mixed-autonomy highways: Training and validation,” _arXiv preprint arXiv:2103.14202_ , 2021.
  * [61] J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. Torr, P. Kohli, and S. Whiteson, “Stabilising experience replay for deep multi-agent reinforcement learning,” in _International conference on machine learning_. PMLR, 2017, pp. 1146–1155.
  * [62] H. Su, K. Shi, J. Y. J. Chow, and L. Jin, “Dynamic queue-jump lane for emergency vehicles under partially connected settings: A multi-agent deep reinforcement learning approach,” 2021.

##  Appendix S1 Related Work:

Conventional routing optimization and traffic signal pre-emption for EMVs.
Although, in reality, routing and pre-emption are coupled, the existing
methods usually solve them separately. Many of the existing approaches
leverage Dijkstra’s shortest path algorithm to get the optimal route [11, 12,
13, 14]. An A* algorithm for ambulance routing has been proposed by [15].
However, as this line of work assumes that the routes and traffic conditions
are fixed and static, they fail to address the dynamic nature of real-world
traffic flows. Another line of work has considered the change of traffic flows
over time. [16] has proposed a shortest-path algorithm for time-dependent
traffic networks, but the travel time associated with each edge at each time
step is assumed to be known in prior. [17] proposes different routing
strategies for different times in a day (e.g., peak/non-peak hours) based on
traffic history data at those times. However, in the problem of our
consideration, routing and pre-emption strategies can significantly affect the
travel time associated with each edge during the EMV passage, and the existing
methods cannot deal with this kind of real-time changes. [18] formulated the
dynamic shortest path problem as a mixed-integer programming problem. [19] has
used RL for real-time vehicle navigation and routing. However, both of these
studies have tackled a general routing problem, and signal pre-emption and its
influence on traffic have not been modeled. Once an optimal route for the EMV
has been determined, traffic signal pre-emption is deployed. A common pre-
emption strategy is to extend the green phases of green lights to let the EMV
pass each intersection along a fixed optimal route [11, 6]. [20] has proposed
pre-emption strategies for multiple EMV requests. Please refer to [21] and
[22] for a thorough survey of conventional routing optimization and traffic
signal pre-emption methods. We would also like to point out that the
conventional methods prioritize EMV passage and have significant disturbances
on the traffic flow which increases the average non-EMV travel time.

RL-based traffic signal control. Traffic signal pre-emption only alters the
traffic phases at the intersections where an EMV travels through. However, to
reduce congestion, traffic phases at nearby intersections also need to be
changed cooperatively. Recently, deep neural network has the promise to
revolutionize the real world applications, especially in the context of
healthcare and humanitarian assistance [23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45]. The coordination of
traffic signals to mitigate traffic congestion is referred to as traffic
signal control which has been addressed by leveraging deep RL in a growing
body of work. Many of the existing approaches use Q-learning [46, 47, 3, 48,
49, 50]. [51] leverages meta-learning algorithms to speed up Q-learning for
traffic signal control. Another line of work has used actor-critic algorithms
for traffic signal control [52, 53, 4]. [54] proposes a hierarchical actor-
critic method to encourage cooperation between intersections. Please refer to
[55] for a review on traffic signal control methods. However, these RL-based
traffic control methods focus on reducing the congestion in the traffic
network and are not designed for EMV pre-emption. In contrast, our RL
framework is built upon state-of-the-art ideas such as max pressure and is
designed to reduce both EMV travel time and overall congestion.

##  Appendix S2 Justification of the Agent Design

The quantities in local agent state can be obtained at each intersection using
various technologies. Numbers of vehicles on each lane $(x^{t}(l),x^{t}(m))$
can be obtained by vehicle detection technologies, such as inductive loop [56]
based on the hardware installed underground. The distance of the EMV to the
intersection $d^{t}_{EMV}[L_{ji}]$ can be obtained by _vehicle-to-
infrastructure_ technologies such as VANET [57], which broadcasts the real-
time position of a vehicle to an intersection. Prior work [58, 59, 60] has
explored these technologies for traffic signal pre-emption.

The dynamic routing algorithm (Algorithm 1) can provide
$(\mathsf{ETA},\mathsf{Next})$ for each agent at every time step. However, due
to the stochastic nature of traffic flows, updating the route too frequently
might confuse the EMV driver, since the driver might be instructed a new
route, say, every 5 seconds. There are many ways to ensure reasonable
frequency. One option is to inform the driver only once while the EMV is
travels in a single link. We implement it by updating the state of an RL agent
$(\mathsf{ETA}^{t^{\prime}}_{i},\mathsf{Next}^{t^{\prime}}_{i})$ at the time
step when the EMV travels through half of a link. For example, if the EMV
travels through a link to agent $i$ from time step 11 to 20 in constant speed,
then dynamic routing information in $s_{i}^{16}$ to $s_{i}^{20}$ are the same,
which is $(\mathsf{ETA}_{i}^{15},\mathsf{Next}_{i}^{15})$, i.e.,
$t^{\prime}=15$.

As for the reward design, one might wonder how an agent can know its type. As
we assume an agent can observe the state of its neighbors, agent type can be
inferred from the observation. This will become clearer below.

##  Appendix S3 Multi-agent Actor Critic Method

Local Observation. In an ideal setting, agents can observe the states of every
other agent and leverage this global information to make a decision. However,
this is not practical in our problem due to communication latency and will
cause scalability issues. We assume agents can observe its own state and the
states of its neighbors, i.e.,
$s^{t}_{\mathcal{V}_{i}}=\\{s^{t}_{j}|j\in\mathcal{V}_{i}\\}$. The agents feed
this observation to its policy network $\pi_{\theta_{i}}$ and value network
$V_{\phi_{i}}$.

Fingerprint. In multi-agent training, each agent treats other agents as part
of the environment, but the policy of other agents are changing over time.
[61] introduces _fingerprints_ to inform agents about the changing policies of
neighboring agents in multi-agent Q-learning. [4] brings fingerprints into
MA2C. Here we use the probability simplex of neighboring policies
$\pi^{t-1}_{\mathcal{N}_{i}}=\\{\pi^{t-1}_{j}|j\in\mathcal{N}_{i}\\}$ as
fingerprints, and include it into the input of policy network and value
network. Thus, our policy network can be written as
$\pi_{\theta_{i}}(a_{i}^{t}|s^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$
and value network as
$V_{\phi_{i}}(\tilde{s}^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$,
where $\tilde{s}^{t}_{\mathcal{V}_{i}}$ is the local observation with spatial
discount factor, which is introduced below.

Spatial Discount Factor and Adjusted Reward. MA2C agents cooperatively
optimize a global cumulative reward. We assume the global reward is
decomposable as $r_{t}=\sum_{i\in\mathcal{V}}r^{t}_{i}$, where $r^{t}_{i}$ is
defined in Eqn. 1. Instead of optimizing the same global reward for every
agent, [4] proposes a spatial discount factor $\alpha$ to let each agent pay
less attention to rewards of agents far away. The adjusted reward for agent
$i$ is

$\tilde{r}_{i}^{t}=\sum_{d=0}^{D_{i}}\Big{(}\sum_{j\in\mathcal{V}|d(i,j)=d}(\alpha)^{d}r^{t}_{j}\Big{)},$
(2)

where $D_{i}$ is the maximum distance of agents in the graph from agent $i$.
When $\alpha>0$, the adjusted reward include global information, it seems this
is in contradiction to the local communication assumption. However, since
reward is only used for offline training, global reward information is
allowed. Once trained, the RL agents can control traffic signal without
relying on global information.

Temporal Discount Factor and Return. The local return $\tilde{R}^{t}_{i}$ is
defined as the cumulative adjusted reward
$\tilde{R}^{t}_{i}:=\sum_{\tau=t}^{T}\gamma^{\tau-t}\tilde{r}^{\tau}_{i}$,
where $\gamma$ is the temporal discount factor and $T$ is the length of an
episode. we can estimate the local return using value function,

$\tilde{R}^{t}_{i}=\tilde{r}^{t}_{i}+\gamma
V_{\phi_{i}^{-}}(\tilde{s}^{t+1}_{\mathcal{V}_{i}},\pi^{t}_{\mathcal{N}_{i}}|\pi_{\theta_{-i}^{-}}),$
(3)

where $\phi_{i}^{-}$ means parameters $\phi_{i}$ are frozen and
$\theta_{-i}^{-}$ means the parameters of policy networks of all other agents
are frozen.

Network architecture and training. As traffic flow data are spatial temporal,
we leverage a long-short term memory (LSTM) layer along with fully connected
(FC) layers for policy network (actor) and value network (critic). Our multi-
agent actor-critic training pipeline is similar to that in [4]. We provide
neural architecture details, policy loss expression, value loss expression as
well as a training pseudocode.

##  Appendix S4 Training Details

###  S4.1 Value loss function

With a batch of data
$B=\\{(s_{i}^{t},\pi_{i}^{t},a_{i}^{t},s_{i}^{t+1},r_{i}^{t})_{i\in\mathcal{V}}^{t\in\mathcal{T}}\\}$,
each agent’s value network is trained by minimizing the difference between
bootstrapped estimated value and neural network approximated value

$\mathcal{L}_{v}(\phi_{i})=\frac{1}{2|B|}\sum_{B}\Big{(}\tilde{R}^{t}_{i}-V_{\phi_{i}}(\tilde{s}^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})\Big{)}^{2}.$
(4)

###  S4.2 Policy loss function

Each agent’s policy network is trained by minimizing its policy loss

$\displaystyle\mathcal{L}_{p}(\theta_{i})=-$
$\displaystyle\frac{1}{|B|}\sum_{B}\bigg{(}\log\pi_{\theta_{i}}(a_{i}^{t}|s^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})\tilde{A}^{t}_{i}$
(5)
$\displaystyle-\lambda\sum_{a_{i}\in\mathcal{A}_{i}}\pi_{\theta_{i}}\log\pi_{\theta_{i}}(a_{i}|s^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})\bigg{)},$
(6)

where
$\tilde{A}^{t}_{i}=\tilde{R}^{t}_{i}-V_{\phi_{i}^{-}}(\tilde{s}^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$
is the estimated advantage which measures how much better the action
$a^{t}_{i}$ is as compared to the average performance of the policy
$\pi_{\theta_{i}}$ in the state $s_{i}^{t}$. The second term is a
regularization term that encourage initial exploration, where
$\mathcal{A}_{i}$ is the action set of agent $i$. For an intersection as shown
in Fig. 1, $\mathcal{A}_{i}$ contains 8 traffic signal phases.

###  S4.3 Training algorithm

Algorithm 2 shows the multi-agent A2C training process.

Input :

$T$ | maximum time step of an episode
---|---
$N_{\mathrm{bs}}$ | batch size
$\eta_{\theta}$ | learning rate for policy networks
$\eta_{\phi}$ | learning rate for value networks
$\alpha$ | spatial discount factor
$\gamma$ | (temporal) discount factor
$\lambda$ | regularizer coefficient

Output :

$\\{\phi_{i}\\}_{i\in\mathcal{V}}$ | learned parameters in value networks
---|---
$\\{\theta_{i}\\}_{i\in\mathcal{V}}$ | learned parameters in policy networks

initialize $\\{\phi_{i}\\}_{i\in\mathcal{V}}$,
$\\{\theta_{i}\\}_{i\in\mathcal{V}}$, $k\leftarrow 0$,
$B\leftarrow\varnothing$; initialize SUMO, $t\leftarrow 0$, get
$\\{s^{0}_{i}\\}_{i\in\mathcal{V}}$;

repeat

/* generate trajectories */

foreach  _$i\in\mathcal{V}$_ do (in parallel)

sample $a^{t}_{i}$ from $\pi^{t}_{i}$;

receive $\tilde{r}^{t}_{i}$ and $s^{t+1}_{i}$;

$B\leftarrow
B\cup\\{(s_{i}^{t},\pi_{i}^{t},a_{i}^{t},s_{i}^{t+1},r_{i}^{t})_{i\in\mathcal{V}}\\}$;

$t\leftarrow t+1$, $k\leftarrow k+1$;

if  _$t==T$_ then

initialize SUMO, $t\leftarrow 0$, get $\\{s^{0}_{i}\\}_{i\in\mathcal{V}}$;

/* update actors and critics */

if  _$k==N_{\mathrm{bs}}$_ then

foreach  _$i\in\mathcal{V}$_ do (in parallel)

calculate $\tilde{r}^{t}_{i}$ (Eqn. (4)), $\tilde{R}^{t}_{i}$ (Eqn. (5));

$\phi_{i}\leftarrow\phi_{i}-\eta_{\phi}\nabla\mathcal{L}_{v}(\phi_{i})$;

$\theta_{i}\leftarrow\theta_{i}-\eta_{\theta}\nabla\mathcal{L}_{p}(\theta_{i})$;

$k\leftarrow 0,B\leftarrow\varnothing$;

until  _Convergence_ ;

Algorithm 2 Multi-agent A2C Training

##  Appendix S5 Intralink EMV travel time

The intra-link traffic pattern with the presence of an EMV on duty is
complicated and is under-explored in the current literature. For simplicity,
here we demonstrate a simple intra-link traffic model for a link with 2 lanes.
The model can be easily extended for multiple lanes.

In a two-lane link, the EMV takes a lane and the non-EMVs on the other lane
usually slows down or entirely stop. Some non-EMVs ahead of the EMV find pull-
over spots in the other lane and park there. Those that cannot find a parking
spot continue to drive in front of the EMV, potentially blocking the EMV
passage [62]. In this study, we propose a meso-scopic model to estimate the
intra-link travel time of an EMV.

Figure 5: Normal traffic state.

Normally, the traffic flow of a link is modeled by a fundamental diagram, see
Fig 5. This diagram depicts a simplified relationship between the flow rate,
i.e. number of vehicles passing within the unit amount of the time, and number
of vehicles on a link. The max number of vehicles $H$ indicates the capacity
of this link. The critical number of vehicles $F$ indicates the boundary
differentiates the non-congested state and congested state. When the number of
vehicles is smaller than $F$, all vehicles are traveling at the free flow
speed, which is represented by the slope of $OB$. When number of vehicles is
larger than $F$, vehicles are slowing down and traffic flows declines since
the link is now congested. The max flow is attained when the number of
vehicles is at $F$. The travel speed of vehicles in a congested state $Q$ is
obtained by the slope of $OQ$.

Figure 6: Traffic state during EMV pre-emption.

During the EMV pre-emption, the original traffic flow relationship pictured in
black are diverted into two parts, representing two lanes respectively. The
green line represents the traffic conditions of the pre-emption lane, i.e. the
lane where the EMV is traveling, and the orange line represents the other
lane. During pre-emption, part of the vehicles originally travelling in front
of the EMV pull over onto the adjacent lane, resulting a significant decrease
in the max capacities of the pre-emption lane. Meanwhile, because vehicles can
park onto the curbs, orange line depicts a larger maximum capacity $I$ than
normal.

Regarding the max flow, the adjacent lane has a smaller max flow as the
vehicles on this lane are required to slow down when EMV is on duty. The pre-
emption lane obtains a max capacity when there is one vehicle on the lane,
i.e. the EMV itself. Under this circumstance, the flow rate is equivalent as
the free flow speed of the EMV. However, when there are vehicles remaining in
front of the EMV, the travel speed of the EMV might be slowed down, but still
higher than the free flow speeds of the non-EMVs. Furthermore, the travel
speed of EMV has a discrete value corresponding to the number of non-EMVs
blocking. For example, pre-emption lane has a traffic state represented by
$M$, and the travel speed of the EMV is obtained as the slope of $M$. We use
this simple model, especially the green plot to estimate the intra-link travel
time of EMV in a link as a function of number of vehicles in that link. We
calibrate this model with the SUMO environment. Intuitively, the travel speed
of the EMV is affected more by the number of vehicles on the pre-emption lane
than their positions. The reason behind is since the ETA is frequently updated
every $\Delta T$ seconds ($\Delta T$ in our experiment), and the estimated ETA
would eventually converge. Imagine there are vehicles very far away from the
EMV and about to leave the intersections at $t$, their presence would not slow
down the approaching EMV. Therefore, when they have left the intersection at
$t+1$, we have an updated number of non-EMVs count and updated travel speed
estimation.

##  Appendix S6 Implementation Details

Although MDP step length can be arbitrarily small enough for optimality,
traffic signal phases should maintained a minimum amount of time so that
vehicles and pedestrians can safely cross the intersections. To avoid rapid
switching between the phases, we set our MDP time step length to be 5 seconds.

###  S6.1 Implementation details for synthetic $\text{grid}_{5\times 5}$

  * •

dimension of $s^{t}_{\mathcal{V}_{i}}$: $5\times(8+8+4+2)=110$

  * •

dimension of $\tilde{s}^{t}_{\mathcal{V}_{i}}$: $5\times(8+8+4+2)=110$

  * •

dimension of $\pi^{t-1}_{\mathcal{N}_{i}}$: $4\times 8=32$

  * •

Policy network
$\pi_{\theta_{i}}(a_{i}^{t}|s^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$:
concat[$110\xrightarrow[]{\textrm{FC}}128$ReLu,
$32\xrightarrow[]{\textrm{FC}}64$ReLu] $\xrightarrow[]{}64$LSTM
$\xrightarrow[]{\textrm{FC}}8$Softmax

  * •

Value network
$V_{\phi_{i}}(\tilde{s}^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$:
concat[$110\xrightarrow[]{\textrm{FC}}128$ReLu,
$32\xrightarrow[]{\textrm{FC}}64$ReLu] $\xrightarrow[]{}64$LSTM
$\xrightarrow[]{\textrm{FC}}1$Linear

  * •

Each link is $200m$. The free flow speed of the EMV is $12m/s$ and the free
flow speed for non-EMVs is $6m/s$.

  * •

Temporal discount factor $\gamma$ is $0.99$ and spatial discount factor
$\alpha$ is $0.90$.

  * •

Initial learning rates $\eta_{\phi}$ and $\eta_{\theta}$ are both 1e-3 and
they decay linearly. Adam optimizer is used.

  * •

MDP step length $\Delta t=5s$ and for secondary pre-emption reward weight
$\beta$ is $0.5$.

  * •

Regularization coefficient is $0.01$.

###  S6.2 Implementation details for $\text{Manhattan}_{16\times 3}$

The implementation is similar to the synthetic network implementation, with
the following differences:

  * •

Initial learning rates $\eta_{\phi}$ and $\eta_{\theta}$ are both 5e-4.

  * •

Since the avenues and streets are both one-directional, the number of actions
of each agent are adjusted accordingly.

##  Appendix S7 Ablation Study

###  S7.1 Ablation study on pressure and agent types

We propose three types of agents and design their rewards (Eqn. 1) based on
our improved pressure definition and heuristics. In order to see how our
improved pressure definition and proposed special agents influence the
results, we (1) replace our pressure definition by that defined in PressLight,
(2) replace secondary pre-emption agents with normal agents and (3) replace
primary pre-emption agents with normal agents.

Ablations | (1) | (2) | (3) | EMVLight
---|---|---|---|---
$T_{\text{EMV}}$ [s] | 197 | 289 | 320 | 199
$T_{\text{avg}}$ [s] | 361.05 | 347.13 | 359.62 | 322.40
Table 2: Ablation study on pressure and agent types. Experiments are conducted
on the Config 1 synthetic $\text{grid}_{5\times 5}$.

Table 2 shows the results of these ablations: (1) PressLight-style pressure
(see Appendix) yields a slightly smaller EMV travel time but significantly
increases the average travel time; (2) Without secondary pre-emption agents,
EMV travel time increases by 45% since almost no “link reservation" happened;
(3) Without primary pre-emption agents, EMV travel time increases
significantly, which shows the importance of pre-emption.

###  S7.2 Ablation study on fingerprint

In multi-agent RL, fingerprint has been shown to stabilize training and enable
faster convergence. In order to see how fingerprint affects training in
EMVLight, we remove the fingerprint design, i.e., policy and value networks
are changed from
$\pi_{\theta_{i}}(a_{i}^{t}|s^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$
and
$V_{\phi_{i}}(\tilde{s}^{t}_{\mathcal{V}_{i}},\pi^{t-1}_{\mathcal{N}_{i}})$ to
$\pi_{\theta_{i}}(a_{i}^{t}|s^{t}_{\mathcal{V}_{i}})$ and
$V_{\phi_{i}}(\tilde{s}^{t}_{\mathcal{V}_{i}})$, respectively. Fig. 7 shows
the influence of fingerprint on training. With fingerprint, the reward
converges faster and suffers from less fluctuation, confirming the
effectiveness of fingerprint.

Figure 7: Reward convergence with and without fingerprint. Experiments are
conducted on Config 1 synthetic $\text{grid}_{5\times 5}$.