# Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement
Learning

Michael Villarreal, Bibek Poudel, Weizi Li Michael Villarreal, Bibek Poudel,
and Weizi Li are with the Department of Computer Science, University of
Memphis, Memphis, TN 38152, USA {tmvllrrl<EMAIL_ADDRESS>
###### Abstract

The surge in Reinforcement Learning (RL) applications in Intelligent
Transportation Systems (ITS) has contributed to its growth as well as
highlighted key challenges. However, defining objectives of RL agents in
traffic control and management tasks, as well as aligning policies with these
goals through an effective formulation of Markov Decision Process (MDP), can
be challenging and often require domain experts in both RL and ITS. Recent
advancements in Large Language Models (LLMs) such as GPT-4 highlight their
broad general knowledge, reasoning capabilities, and commonsense priors across
various domains. In this work, we conduct a large-scale user study involving
70 participants to investigate whether novices can leverage ChatGPT to solve
complex mixed traffic control problems. Three environments are tested,
including ring road, bottleneck, and intersection. We find ChatGPT has mixed
results. For intersection and bottleneck, ChatGPT increases number of
successful policies by 150% and 136% compared to solely beginner capabilities,
with some of them even outperforming experts. However, ChatGPT does not
provide consistent improvements across all scenarios.

##  I INTRODUCTION

Large Language Models (LLMs) represent a significant advancement in artificial
intelligence. Their usefulness as a general-purpose tool is anticipated to
have a profound societal impact. Experiments with LLMs such as GPT-4 [1, 2],
LLaMA [3], and PALM2 [4] demonstrate their strong reasoning and common sense
abilities across domains such as math, science, and coding. To achieve this
success, LLMs leverage Reinforcement Learning from Human Feedback (RLHF) [5,
6] to solve the alignment problem, i.e., follow user intent by fine-tuning
them on human feedback. While there is a strong focus on improving LLMs though
Reinforcement Learning (RL), the possibility of leveraging LLMs to assist RL
problems is in nascent stages.

The RL framework is inherently challenging because it demands an understanding
of Markov Decision Process (MDPs). Any RL problem must be formulated to a MDP
through designing the state, action, reward, and discount-factor among other
components [7]. This is usually a tedious task that requires numerous
experimentation and careful analysis, often by a domain expert. Specifically,
in research areas such as Intelligent Transportation Systems (ITS), it is
challenging for novices to understand and design effective MDPs. Moreover, no
standard technique exists for creating general-purpose MDPs that will work
across environments and satisfy various goals.

Over the last decade, RL has been adopted to address complex control problems
in ITS such as traffic management and autonomous driving [8, 9]. As autonomous
agents, including vehicles and traffic lights, become more prevalent [10] in
ITS, they introduce new challenges and opportunities. One emerging topic is
mixed traffic control that uses RL-empowered robot vehicles (RVs) to regulate
human-driven vehicles (HVs), thus improving the overall traffic flow [11, 12,
13, 14]. This surge in research interest has attracted a broader audience to
participate in the topic, resulting in a growing demand for creative decision-
making and control strategies enabled by RL. However, the initial technical
barrier, specifically formulating MDP components to align with a control
strategy, poses a challenge. LLMs, with their broad knowledge, commonsense
priors, and creative capacity, show promise in reducing these barriers and
simplifying the process.

In this project, we explore whether ChatGPT (with GPT-4 backend) can assist
non-experts in ITS research to solve mixed traffic control problems. We
conduct a large-scale user study with $70$ participants who have no prior
experience in ITS research. The participants are tasked to develop MDP
components (state and reward) for three mixed traffic scenarios: a ring road,
an intersection, and a bottleneck as shown in Fig. 1. We split participants
into a control group, where participants attempt to solve problems solely
based on common sense and prior knowledge, and a study group, where
participants can prompt ChatGPT unrestricted. Participants are provided a
manuscript with a general overview of RL, a few examples of MDP components, a
reference bank of metrics related to traffic, and images and descriptions of
the mixed traffic control environments. The formulated MDPs are then used to
train policies and evaluate performance. From the study, we find that:

  1. 1.

In the intersection environment, using ChatGPT can lead to a performance
better than an expert.

  2. 2.

In the bottleneck and intersection environments, using ChatGPT results in an
increase in number of successful policies by $150$% and $136$%, respectively.

  3. 3.

ChatGPT’s creativity enables a $363$% increase in utilization of new metrics,
although the use of these new metrics does not always result in a successful
policy.

  4. 4.

In the ring environment, ChatGPT’s help does not increase the policy success
rate.

The wide range of results observed, from no success to outperforming the
expert, across the mixed traffic control environments indicates that the
effectiveness of utilizing ChatGPT in ITS depends on the complexity of the
specific problem, as well as the extent and manner in which ChatGPT is
utilized.

Figure 1: Three mixed traffic control environments [11] (a deep reinforcement
learning framework for traffic management), Ring, Bottleneck, and
Intersection, are provided to the study participants. Robot vehicles are red
and are controlled by learnt RL policies. Human-driven vehicles are white and
are modeled by the Intelligent Driver Model [15].

##  II Related Work

Recent works have studied how Large Language Models (LLMs) can be leveraged in
a variety of RL tasks. In robotics, LLMs are utilized in planning and
navigation by representing the robotic agent’s input (state) in natural
language, additionally incorporating the input with visual or raw sensor data
for grounding. This approach demonstrates high data efficiency and
generalization to unseen environments [16, 17].

Limited research has been found in minimizing the human effort required to
generate an effective policy. Some works use LLMs to substitute components of
MDPs, such as making LLM a surrogate or proxy reward function. By using the
in-context learning capability of LLMs, hard-to-specify reward functions
(e.g., versatility, fairness) have been attempted [18]. Meanwhile, other
studies have used LLMs to replace a policy entirely [19, 20]. However, to the
best of our knowledge, no previous work has directly evaluated the
effectiveness of LLMs (with and without them) as a tool to reduce the effort
in designing MDP components by novices, comparing the obtained policies with
those from experts.

##  III PRELIMINARIES

In the following, we introduce the formulation of mixed traffic control as
reinforcement learning (RL) tasks and discuss the corresponding test
environments.

###  III-A Reinforcement Learning

We model mixed traffic control as a Partially Observable Markov Decision
Process (POMDP) represented by a tuple ($S$, $A$, $P$, $R$, $p_{0}$, $\gamma$,
$T$, $\Omega$, $O$) where $S$ is the state space; $A$ is the action space;
$P(s^{\prime}|s,a)$ is the transition probability function; $R$ is the reward
function; $p_{0}$ is the initial state distribution; $\gamma\in(0,1]$ is the
discount factor; $T$ is the episode length (horizon); $\Omega$ is the
observation space; and $O$ is the probability distribution of retrieving an
observation $\omega\in\Omega$ from a state $s\in S$. At each timestep
$t\in[1,T]$, a robot vehicle (RV) uses its policy $\pi_{\theta}(a_{t}|s_{t})$
to take an action $a_{t}$ $\in$ $A$, given the state $s_{t}$ $\in$ $S$. The
RV’s environment provides feedback from taking action $a_{t}$ by calculating a
reward $r_{t}$ and transitioning the agent into the next state $s_{t+1}$. The
RV’s goal is to learn a policy $\pi_{\theta}$ that maximizes the discounted
sum of rewards, i.e., return, $R_{t}=\sum^{T}_{i=t}\gamma^{i-t}r_{i}$.
Proximal Policy Optimization [21] is used to learn $\pi_{\theta}$.

###  III-B Mixed Traffic Control Environments

####  III-B1 Ring

The ring environment (shown in Fig. 1 top) consists of a single-lane circular
road network and $22$ vehicles ($21$ HVs and one RV). It simulates how
perturbations due to imperfections in human driving behavior can amplify and
propagate, leading to an eventual standstill for some vehicles. This
situation, known as ‘stop-and-go traffic’, acts as a wave that propagates
continually through the ring, opposite the direction of travel. The RV’s goal
is to prevent the formation of these waves. Ring is a widely used benchmark in
traffic control [22]. An expert’s [11] state space (major metrics given in
Table I) is:

$\text{s}=\left\\{\frac{v_{RV}}{v_{\text{max}}},\frac{v_{\text{lead}}-v_{RV}}{v_{\text{max}}},f\left(x_{\text{lead}}-x_{RV}\right)\right\\}.$
(1)

The difference in $x_{\text{lead}}$ and $~{}x_{RV}$ is passed through a
normalization function $f$. An expert’s reward function encourages high
average velocity and low control actions (acceleration) though a weighted
combination given by:

$\text{r}=\frac{1}{n}\sum_{i}v_{i}-\alpha*\left|a_{RV}\right|,$ (2)

where $n=22$ and $\alpha$ is chosen empirically.

####  III-B2 Bottleneck

The bottleneck environment (shown in Fig. 1 middle) simulates vehicles
experiencing a capacity drop [23] where a road’s outflow significantly
decreases after the road’s inflow surpasses a threshold. The RVs’ goal is to
improve outflow. Bottleneck represents a bridge with lanes decreasing from
$4\times l$ to $2\times l$ to $l$ (where $l$ is a scaling factor and is one
for our work). The RV penetration rate is $10\%$. An expert’s [24] state space
is:

$\omega_{\text{precise}}=\left\\{\overline{X}_{HV},\overline{V}_{HV},\overline{X}_{RV},\overline{V}_{RV},o_{20}\right\\}\text{,}$
(3)

where the mean positions and velocities of both vehicle types are considered
across user-defined segments of the road network. An expert’s reward function
rewards increasing bottleneck outflow:

$\text{r}=o_{10}\text{.}\vspace{-0.5em}$ (4)

####  III-B3 Intersection

The intersection environment (shown in Fig. 1 bottom) represents an
unsignalized intersection where east/westbound traffic flow is less than
north/southbound traffic flow. This flow discrepancy leads to east/westbound
traffic queues as crossing the intersection would be unsafe otherwise. RVs
drive in the north/south directions with a $20\%$ penetration rate. The RVs’
objective is minimizing east/west queues and increased average vehicle
velocity. An expert’s [24] state space is:

$\text{s}=\\{V_{\text{all}},I_{\text{all}},E_{\text{all}},D_{\text{edge}},\overline{V}_{\text{edge}}\\}\text{,}$
(5)

where $I_{\text{all}}$ is all vehicle’s distance to intersection,
$E_{\text{all}}$ is all vehicle’s edge number, $D_{\text{edge}}$ is density of
each edge, and $\overline{V}_{\text{edge}}$ is average vehicle velocity of
each edge. There are eight edges for each direction on both sides of the
intersection. An expert’s reward function penalizes vehicle delay and vehicle
standstills in traffic:

$\text{r}=-\frac{t*{\sum}((V_{\text{max}}-V_{\text{all}})/V_{\text{max}})}{\text{n}+\text{eps}}-(\text{gain}*ss_{n}),$
(6)

where $t$ is current timestep, $n$ is number of vehicles, $eps$ prevents zero
division, $gain$ is 0.2, and $ss_{n}$ is the number of standstill vehicles.

Type |  Metrics
---|---
Vehicle |  Position ($x$), Velocity ($v$), Acceleration ($a$), Gap to leader/follower ($g_{\text{lead}}$/$g_{\text{fol}}$), Fuel consumption ($fc$)
Road Environment |  Outflow over last $i$ seconds ($o_{i}$), Density ($d$), Speed limit ($v_{\text{max}})$, Average speed ($\overline{v}$), Average acceleration ($\overline{a})$, Minimum speed ($v_{\text{min}})$, Average time standstill ($\overline{ss}_{\tau}$)
Driving Behavior |  Distance/Time to complete stop ($st_{\text{dist}}$/$st_{\tau}$), Time to collide ($ttc$), Jerk ($j$)
TABLE I: The bank of metrics provided to participants. Capitalization of a
metric indicates its vector.

##  IV User Study

###  IV-A Participants and Cohorts

Figure 2: Example state space from novice (left) and reward function from
novice with ChatGPT (right) for ring. The reward function contains complex
terms generated by ChatGPT. ChatGPT provides sound reasoning for using the
reward function metrics (see Sec. V-A for details).

We recruit 70 graduate students as participants ($54$ male, $16$ female). The
male/female participants have a median age of $23$/$22$ and average
$4.93$/$4.26$ years of driving experience.

The $70$ participants are split into two cohorts: a control group ($38$
participants) that only use prior non-expert knowledge and the manuscript, and
a study group ($32$ participants) that additionally has access to ChatGPT [25,
1]. The control group helps assess whether ChatGPT can assist non-experts in
solving traffic problems by providing baseline capabilities. We find the
control group heavily uses collision-prevention metrics, including
$g_{lead}/g_{fol}\text{,}~{}ttc\text{,}~{}\text{or}~{}st_{dist}/st_{\tau}$,
with consistent use of $x_{RV}\text{,}~{}v_{RV}\text{,}~{}\text{and}~{}a_{RV}$
across all environments. Additionally, the control group tends to include
objective-oriented metrics such as $o~{}\text{or}~{}\overline{ss}_{\tau}$ for
bottleneck and metrics about queue length/time standstill in the east/west
directions for intersection. These metrics are straightforward and intuitive,
aligning with our expectations for non-experts in ITS research.

We poll the study group’s ChatGPT [25] use frequency with the following
descending frequencies (number of participant selections): daily ($2$),
several times a week ($13$), once a week ($6$), several times a month ($5$),
once a month or less ($2$), and never ($3$).

###  IV-B ChatGPT Setup

The study group participants use GPT-4 with $8$k context length and
temperature $=0.7$, a static model provided by OpenAI with no additional fine-
tuning. To simulate the practical use of ChatGPT and enable in-context
learning, a chat interface is employed using TypingMind [26].

###  IV-C Manuscript

We develop two versions of manuscripts111https://github.com/tmvllrrl/its-study
(for different cohorts) with the following sections. Study duration is $95$
minutes.

####  IV-C1 General Instructions and Background Questions

The general instructions outline participant conduct for the study’s duration.
Participants complete the background questions discussed in Sec. IV-A.

####  IV-C2 Reinforcement Learning Overview

The RL overview section provides a brief explanation of the state space and
reward function RL components. We provide transportation-related examples and
explanations to ensure common understanding of RL across all participants.

####  IV-C3 Answer Instructions and Bank of Metrics

The answer instructions cover how their answers should be formatted with
examples. The bank of metrics is provided in Table I. The participants can use
any new metric with explanation.

####  IV-C4 Mixed Traffic Environment Descriptions and Questions

Each of the three traffic environments (shown in Fig. 1; details in Sec.
III-B) receives a general description, problem explanation, and the mixed
traffic objective of the RVs. The general description includes details such as
the number of RVs present, general flow behavior of traffic, ratio of RVs to
human-driven vehicles. We provide a supplementary
video222https://youtu.be/qTqgfl76FAo demonstrating the mixed traffic
environments. We ask the participants three questions per environment: the
state space, the reward function, and briefly explain the rationale behind the
reward function. The state spaces participants provide are observation spaces;
however, we solely use “state space” to prevent additional complexity/avoid
confusion.

##  V RESULTS

Next, we analyze the participants’ answers. Then, we briefly discuss
experiment setup and the results in the three mixed traffic control
environments.

Figure 3: Normalized reward curves during training are shown for the three
traffic control environments. Each environment consists of three curves for
the control group (Novices), the study group (Novices + ChatGPT), and an
expert. For Novices and Novices + ChatGPT, the solid line indicates the
average for all participants, with the shaded region representing variance. In
both groups across all networks, average rewards increase over the course of
training. This validates that both groups are able to develop state spaces and
reward functions that are trainable using RL.

###  V-A Participant Answers Analysis

Responses | Ring | Bottleneck | Intersection
---|---|---|---
Control | Study | Control | Study | Control | Study
Valid | 25 | 28 | 24 | 27 | 21 | 24
Invalid | 13 | 4 | 14 | 5 | 17 | 8
Total | 38 | 32 | 38 | 32 | 38 | 32
TABLE II: Number of valid and invalid answers for each traffic environment for
the control and study groups.

We find that ChatGPT [25, 1] impacts provided state spaces/reward functions by
significantly decreasing invalid answers over using prior knowledge. We
consider an answer valid if it contains metrics provided in the metrics bank
or have well-defined explanations. Table II presents the number of
valid/invalid answers for all three environments. On average, only $61$% of
answers from the control group are valid, while $82$% are valid for the study
group, a $21$% increase. This illustrates ChatGPT’s capabilities in guiding
participants to create valid state spaces/reward functions.

Another impact of ChatGPT is that the study group uses (in ring, bottleneck,
intersection order) $35$, $63$, and $59$ new metrics compared to the control
group’s 8, $17$, and $21$ new metrics that do not exist in the metrics bank.
On average, this is a $363$% increase in new metrics. This significant
increase implies that ChatGPT can provide new perspectives to solving the
mixed traffic problems.

We provide example state space and reward function in Fig. 2. The left image
is a control group (Novice) state space, while the right image is a reward
function from the study group (Novice + ChatGPT). The novice state space
heavily uses existing metrics (a trend with the control group), while the
ChatGPT reward shows the intricate metrics ChatGPT generates. ChatGPT also
offers explanations that appear reasonable with the terms. For example, for
the “maintain safe time to collision” in Fig. 2, ChatGPT states, “Since your
state space includes the time-to-collision (TTC) for the front and rear
vehicles, you can encourage the agent to maintain a safe TTC with both
vehicles…. The reward is applied only when the actual TTC is less than the
determined $safeTTC$.”

Figure 4:  Results for the three mixed traffic environments with successful RL
policies denoted. For ring, five policies using ChatGPT’s help are successes,
while seven policies are successful using only non-expert knowledge. Using
ChatGPT sees a decrease in successful policies by two compared to only using
non-expert knowledge, illustrating ChatGPT needs to be better prompted or
further improvements to be useful in this task. For bottleneck, 14 policies
are successful without ChatGPT’s assistance, while 19 are successful without
ChatGPT, a $136$% increase. For intersection, only using non-expert knowledge
results in four successful policies, while ChatGPT increases successes to six
(a $150$% increase) with four (the rightmost green markers) of those
outperforming the expert. The bottleneck and intersection increases illustrate
how ChatGPT can enable more non-experts to solves complex mixed traffic
control tasks. However, the number of increases is lower than expected,
potentially showing ChatGPT needs better prompts or further improvement.

###  V-B Experiment Setup

We train an RL policy for each valid answer using Proximal Policy Optimization
(PPO) [21] with default RLlib hyperparameters [27]. The HVs use the
Intelligent Driver Model (IDM) [15] with the stochastic noise range [$-0.2$,
$0.2$] added to account for heterogeneous driving behaviors. Policies are
trained for $200$ episodes. A fully-connected neural network with 2 hidden
layers of size $8$ are used for the ring and intersection environments and
size $16$ are used for the bottleneck environment. Experiments are conducted
with Intel i7-12700k CPU and $32$G RAM.

###  V-C Experiment Results

####  V-C1 Training

We supply training curves for control (Novices) and study (Novices + ChatGPT)
groups in Fig. 3. The curves are normalized to [0,1] with the mean reward
values averaged across the two respective groups at each episode. We also
supply the expert’s training curve. For all three environments and both
groups, the curves show reward improving during training, validating
participants were able to develop trainable policies as a result of the RVs’
actions. Increasing rewards does not guarantee the RV achieves the
environment’s objective, as the RV may pursue actions that enhance rewards not
in line with the goal.

####  V-C2 Ring

Results are given in Fig. 4 (left). Due to the task’s complexity, we plot a
policy’s best performance over 10 tests for each trained ring policy. All
vehicle average speed (x-axis; meters/second) and minimum speed of any vehicle
in the ring (y-axis) for the last 100 seconds (total testing period is 600
seconds) are considered. We consider policies successful (outlined) if their
minimum speed is greater than zero while maintaining relatively good average
speed. An expert’s performance [11] is also provided for comparison.

When using ChatGPT’s assistance, five policies are successful, while seven
policies are successful using only non-expert knowledge. This result defies
expectations given ChatGPT’s ability to increase valid answers and inject new
metrics into the state spaces/reward functions for the ring environment. One
explanation is despite the addition of new metrics, the metrics do not improve
the robot vehicle’s ability to prevent stop-and-go traffic. Another conjecture
is while the study group is encouraged to use ChatGPT, the level of usage in
participants varies from asking ChatGPT a few questions to completely relying
on it. This impacts ChatGPT’s ability to help the participants. Additionally,
the high number of non-expert successful policies is unexpected. While none of
the given policies reach the expert’s level, the anticipated number is close
to zero given the tasks’ complexity. Non-experts have more capability than
originally hypothesized.

####  V-C3 Bottleneck

Fig. 4 (middle) shows the bottleneck results. Each trained RL policy is tested
$10$ times with the average reported. Outflow (x-axis; vehicles/hour) is
considered, and an expert’s performance is given as a pink, vertical line. We
consider a policy successful if the policy’s outflow is greater than $1400$.

While the outflow range between novices and novices with ChatGPT is similar,
we observe an increase in successful policies when using ChatGPT’s aid. The
control group has $14$ successful policies, while the study group has $19$
successful policies, a $136$% increase. Similar to ring, no participant-given
policy outperforms the expert being near $100$ shy.

####  V-C4 Intersection

Intersection results are given in Fig. 4 (right). The trained RL policy is
evaluated $10$ times with average results reported. All vehicle average speed
(x-axis; meters/second) and east/westbound queue length (y-axis; number of
waiting vehicles) are considered. We plot an expert’s performance [11] and
consider the nearest neighbors as successes (outlined). Four policies are
successful with only non-expert knowledge, but six policies are successful (a
$150$% increase) when using ChatGPT aid. Additionally, of the six with-
ChatGPT-help policies, four of them outperform the expert policy by a
significant margin. This result is significant and showcases how ChatGPT can
give non-experts the ability to compete with ITS domain experts. Two non-
expert policies outperform the expert policy, an unexpected outcome though
half as much as using ChatGPT.

For both bottleneck and intersection, we observe a 136% and 150% increase,
respectively, in successful policies. Examples of successful policies are
provided in Fig. 5. ChatGPT is new with its training set not provided meaning
ChatGPT could have not been trained on a sufficient amount of RL, ITS, or both
training data to provide even more assistance. While the level of ChatGPT
assistance is participant-determined, we observe a significant number of
policies with extensive ChatGPT are not successful.

Figure 5: Example successful policies with ChatGPT’s aid. (a) Ring. The RV
first creates a gap to the leading vehicle (blue) to slow every vehicle down
before gradually accelerating to stabilize the ring, preventing stop-and-go
traffic. (b) Bottleneck. The RV gradually slows down (before speeding up again
in last frame) to force following vehicles to slow down as well, easing
congestion closer to the bottleneck’s end. (c) Intersection. The first RV
travels safely through the intersection, while a second RV slows down so all
east/westbound vehicles can cross without causing an incident.

##  VI CONCLUSIONS

In this work, we conduct a large-scale user study involving non-experts in
intelligent transportation systems (ITS) research trying to provide quality
reinforcement learning (RL) state spaces and reward functions for three mixed
control traffic environments. Our study finds using ChatGPT can increase the
number of successful polices by $150$% and $135$% in the intersection and
bottleneck environments, respectively. However, using ChatGPT does not
increase successes in the ring environment. Additionally, the improvement rate
from using ChatGPT is less than originally theorized. This potentially means
that an insufficient amount of RL ITS problems were provided to ChatGPT during
training.

In the future, we aim to advance this study in several directions. First, we
are interested in testing whether ChatGPT can assist the design of other ITS
applications such as traffic state estimation and simulation [28, 29, 30, 31,
32, 33, 34], vehicle motion and control [35, 36, 37, 38], and robust driving
[39, 40, 41]. Second, we are also interested in studying mixed traffic control
under other mobility topics such as traffic safety [42] and micromobility
[43]. Finally, we would like to test our approach on other RL tasks, including
graph sparsification [44] and quality species selection [45].

## References

  * [1] OpenAI.  Gpt-4 technical report.  Technical report, OpenAI, 2023.
  * [2] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [4] Google.  Palm2 technical report.  Technical report, Google, 2023.
  * [5] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * [7] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [8] Ammar Haydari and Yasin Yılmaz.  Deep reinforcement learning for intelligent transportation systems: A survey.  IEEE Transactions on Intelligent Transportation Systems, 23(1):11–32, 2020.
  * [9] Matthew Veres and Medhat Moussa.  Deep learning for intelligent transportation systems: A survey of emerging trends.  IEEE Transactions on Intelligent transportation systems, 21(8):3152–3168, 2019.
  * [10] SAE International.  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles.  https://www.sae.org/standards/content/j3016_202104/, Apr 2021.
  * [11] Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen.  Flow: Architecture and benchmarking for reinforcement learning in traffic control.  arXiv preprint arXiv:1710.05465, 10, 2017.
  * [12] Zhongxia Yan and Cathy Wu.  Reinforcement learning for mixed autonomy intersections.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2089–2094. IEEE, 2021.
  * [13] Dawei Wang, Weizi Li, Lei Zhu, and Jia Pan.  Learning to control and coordinate hybrid traffic through robot vehicles at complex and unsignalized intersections.  arXiv preprint arXiv:2301.05294, 2023.
  * [14] Michael Villarreal, Bibek Poudel, Jia Pan, and Weizi Li.  Hybrid traffic control and coordination from pixels.  arXiv preprint arXiv:2302.09167, 2023.
  * [15] Martin Treiber, Ansgar Hennecke, and Dirk Helbing.  Congested traffic states in empirical observations and microscopic simulations.  Physical review E, 62(2):1805, 2000.
  * [16] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [17] Vishnu Sashank Dorbala, James F Mullen Jr, and Dinesh Manocha.  Can an embodied agent find your” cat-shaped mug”? llm-based zero-shot object navigation.  arXiv e-prints, pages arXiv–2303, 2023.
  * [18] Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.  Guiding pretraining in reinforcement learning with large language models.  arXiv preprint arXiv:2302.06692, 2023.
  * [19] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans.  Foundation models for decision making: Problems, methods, and opportunities.  arXiv preprint arXiv:2303.04129, 2023.
  * [20] Hengyuan Hu and Dorsa Sadigh.  Language instructed reinforcement learning for human-ai coordination.  arXiv preprint arXiv:2304.07297, 2023.
  * [21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [22] Fang-Chieh Chou, Alben Rome Bagabaldo, and Alexandre M Bayen.  The lord of the ring road: a review and evaluation of autonomous control policies for traffic in a ring road.  ACM Transactions on Cyber-Physical Systems (TCPS), 6(1):1–25, 2022\.
  * [23] Meead Saberi and Hani S Mahmassani.  Empirical characterization and interpretation of hysteresis and capacity drop phenomena in freeway networks.  Transportation Research Record: Journal of the Transportation Research Board, Transportation Research Board of the National Academies, Washington, DC, 2013.
  * [24] Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen.  Benchmarks for reinforcement learning in mixed-autonomy traffic.  In Conference on robot learning, pages 399–409. PMLR, 2018.
  * [25] OpenAI.  Chatgpt: Optimizing language models for dialogue., 2023.
  * [26] Typingmind.  https://www.typingmind.com.  Accessed: April 2023.
  * [27] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica.  RLlib: Abstractions for distributed reinforcement learning.  In International Conference on Machine Learning (ICML), 2018.
  * [28] David Wilkie, Jason Sewall, Weizi Li, and Ming C. Lin.  Virtualized traffic at metropolitan scales.  Frontiers in Robotics and AI, 2:11, 2015.
  * [29] Weizi Li, Dong Nie, David Wilkie, and Ming C. Lin.  Citywide estimation of traffic dynamics via sparse GPS traces.  IEEE Intelligent Transportation Systems Magazine, 9(3):100–113, 2017.
  * [30] Weizi Li, David Wolinski, and Ming C. Lin.  City-scale traffic animation using statistical learning and metamodel-based optimization.  ACM Trans. Graph., 36(6):200:1–200:12, 2017.
  * [31] Weizi Li, Meilei Jiang, Yaoyu Chen, and Ming C. Lin.  Estimating urban traffic states using iterative refinement and wardrop equilibria.  IET Intelligent Transport Systems, 12(8):875–883, 2018.
  * [32] Lei Lin, Weizi Li, and Srinivas Peeta.  Efficient data collection and accurate travel time estimation in a connected vehicle environment via real-time compressive sensing.  Journal of Big Data Analytics in Transportation, 1(2):95–107, 2019\.
  * [33] Qianwen Chao, Huikun Bi, Weizi Li, Tianlu Mao, Zhaoqi Wang, Ming C. Lin, and Zhigang Deng.  A survey on visual traffic simulation: Models, evaluations, and applications in autonomous driving.  Computer Graphics Forum, 39(1):287–308, 2020.
  * [34] Lei Lin, Weizi Li, and Lei Zhu.  Data-driven graph filter based graph convolutional neural network approach for network-level multi-step traffic prediction.  Sustainability, 14(24):16701, 2022.
  * [35] Lei Lin, Weizi Li, Huikun Bi, and Lingqiao Qin.  Vehicle trajectory prediction using LSTMs with spatial-temporal attention mechanisms.  IEEE Intelligent Transportation Systems Magazine, 14(2):197–208, 2022.
  * [36] Weizi Li, David Wolinski, and Ming C. Lin.  ADAPS: Autonomous driving via principled simulations.  In IEEE International Conference on Robotics and Automation (ICRA), pages 7625–7631, 2019.
  * [37] Yu Shen, Weizi Li, and Ming C. Lin.  Inverse reinforcement learning with hybrid-weight trust-region optimization and curriculum learning for autonomous maneuvering.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7421–7428, 2022.
  * [38] Bibek Poudel, Thomas Watson, and Weizi Li.  Learning to control dc motor for micromobility in real time with reinforcement learning.  In IEEE International Conference on Intelligent Transportation Systems (ITSC), pages 1248–1254, 2022.
  * [39] Bibek Poudel and Weizi Li.  Black-box adversarial attacks on network-wide multi-step traffic state prediction models.  In IEEE International Conference on Intelligent Transportation Systems (ITSC), pages 3652–3658, 2021.
  * [40] Yu Shen, Laura Zheng, Manli Shu, Weizi Li, Tom Goldstein, and Ming C. Lin.  Gradient-free adversarial training against image corruption for learning-based steering.  In Advances in Neural Information Processing Systems (NeurIPS), pages 26250–26263, 2021.
  * [41] Michael Villarreal, Bibek Poudel, Ryan Wickman, Yu Shen, and Weizi Li.  Autojoin: Efficient adversarial training for robust maneuvering via denoising autoencoder and joint learning.  2022\.
  * [42] Lei Lin, Feng Shi, and Weizi Li.  Assessing inequality, irregularity, and severity regarding road traffic safety during covid-19.  Scientific Reports, 11(13147):1–7, 2021.
  * [43] Lei Lin, Weizi Li, and Srinivas Peeta.  Predicting station-level bike-sharing demands using graph convolutional neural network.  In Transportation Research Board 98th Annual Meeting (TRB), 2019\.
  * [44] Ryan Wickman, Xiaofei Zhang, and Weizi Li.  A generic graph sparsification framework using deep reinforcement learning.  In IEEE International Conference on Data Mining (ICDM), pages 1221–1226, 2022.
  * [45] Ryan Wickman, Bibek Poudel, Michael Villarreal, Xiaofei Zhang, and Weizi Li.  Efficient quality-diversity optimization through diverse quality species.  In Genetic and Evolutionary Computation Conference (GECCO), 2023\.


# Infinitesimal phase response functions can be misleading

Christoph Börgers

###### Abstract

Phase response functions are the central tool in the mathematical analysis of
pulse-coupled oscillators. When an oscillator receives a brief input pulse,
the phase response function specifies how its phase shifts as a function of
the phase at which the input is received. When the pulse is weak, it is
customary to linearize around zero pulse strength. The result is called the
infinitesimal phase response function. These ideas have been used extensively
in theoretical biology, and also in some areas of engineering. I give examples
showing that the infinitesimal phase response function may predict that two
oscillators, as they exchange pulses back and fourth, will converge to
synchrony, yet this is false when the exact phase response function is used,
for all positive interaction strengths. For short, the analogue of the
Hartman-Grobman theorem that one might expect to hold at first sight is
invalid. I give a condition under which the prediction derived using the
infinitesimal phase response function does hold for the exact phase response
function when interactions are sufficiently weak but positive. However, I
argue that this condition may often fail to hold.

###### keywords:

phase oscillator , phase response function , infinitesimal phase response
function , pulse-coupled oscillators , synchrony , Hartman-Grobman theorem

††journal: Examples and Counterexamples

organization=Department of Mathematics, Tufts University,
city=Medford, MA 02421,, country=USA

##  1 Phase oscillators, phase response functions, and synchrony

In this paper, I present a greatly expanded discussion of a calculation from
[1], concerning pulse-coupled phase oscillators. Although I have in mind
periodically firing neurons, or perhaps populations of neurons that
periodically fire spike volleys, the mathematics may be applicable in other
contexts as well. In this section, I review some background.

###  1.1 Phase oscillators

Stable oscillators are usually described mathematically as points moving along
an attracting limit cycle in a phase space. For simplicity, we will say that
the oscillator (rather than the point in phase space) moves along the limit
cycle.

A phase oscillator is one that never leaves the limit cycle, not even in
response to inputs. Further, the dynamics are assumed to be autonomous,
meaning that in the absence of inputs, the velocity of motion only depends on
position in phase space. Therefore at any time, even when there have been past
inputs, the remaining time to cycle completion in the absence of future inputs
lies between $0$ and $T$, where $T$ is the period of the oscillator in the
absence of any inputs. “Cycle completion” must, of course, be defined in an
arbitrary way. We think of neurons firing periodic trains of action
potentials, or networks of neurons firing periodic spike volleys, and mean
firing by “cycle completion”.

We define the phase of a phase oscillator to be

$\varphi=\frac{T-r}{T}$

where $r$ is the remaining time to cycle completion in the absence of future
inputs. We have

$\varphi\in[0,1]$

always, and the state of the oscillator is completely characterized by
$\varphi$.

The phase description is a common idealization in the study of coupled
oscillators. It is an approximation to reality, always somewhat inaccurate,
and often inappropriate. For example, an input to a neuron can delay the next
firing by far more than $T$ [2, Figure 1], and it can even halt firing
altogether [3]. Such examples do involve leaving the limit cycle, and are
outside the scope of this article.

The main example in this paper is the Ermentrout-Kopell theta neuron model
[4], which explicitly incorporates the assumption that the oscillator never
leaves its limit cycle in response to external inputs; in fact, the model
describes the motion of a point on the unit circle.

To view an oscillator as a phase oscillator is a more plausible approximation
when inputs to the oscillator are weaker or the limit cycle underlying the
oscillation is more strongly attracting. Nonetheless the phase description is
commonly used not only for infinitesimally weak inputs, but also for inputs of
positive strength. In neuroscience, inputs of positive (sometimes
considerable) strength are certainly of interest, especially if we think of an
oscillator as representing a network of neurons.

###  1.2 Phase response functions

We assume inputs to the oscillators to be instantaneous pulses. This, too, is
a simplification, and it is commonly denoted with the phrase pulse-coupling.
The simplification is valid perhaps for instance as an idealization of fast
AMPA-receptor-mediated excitatory synaptic pulses, but not valid in many other
context.

When an oscillator at phase $\varphi$ receives an input pulse of strength
$\epsilon$, we assume that its phase changes instantaneously as a result:

$\varphi\mapsto\varphi+g(\varphi,\epsilon).$

The function $g$ is called the phase response function. Phase response
functions have been measured experimentally in many studies in the
neuroscience literature; see for instance [2].

We will not yet be specific about what we mean by the “strength” of the input
pulse. The parameter $\epsilon$ is assumed to be $\geq 0$, with

$g(\varphi,0)=0~{}~{}~{}\mbox{for all $\varphi\in[0,1]$}.$ (1)

Since phase, by our definition, always lies between $0$ and $1$, we must have

$-\varphi\leq g(\varphi,\epsilon)\leq 1-\varphi~{}~{}~{}~{}\mbox{for
$\varphi\in[0,1]$, $\epsilon\in[0,\infty)$}.$ (2)

We further assume that receipt of an input cannot instantly set a neuron to
phase $0$ or phase $1$:

$-\varphi<g(\varphi,\epsilon)<1-\varphi~{}~{}~{}~{}\mbox{for
$\varphi\in(0,1)$, $\epsilon\in[0,\infty)$}.$ (3)

“Phase $0$” and “phase $1$” denote the same thing — the time of firing.
Nonetheless we think of $g(0,\epsilon)$ and $g(1,\epsilon)$ as two different
quantities: $g(0,\epsilon)$ is the phase shift that an input immediately
following firing will cause, and $g(1,\epsilon)$ is the phase shift that an
input immediately preceding firing will cause. However, we will now argue that
in the neuroscience context, $g(0,\epsilon)$ and $g(1,\epsilon)$ should in
fact be the same, and both should be zero.

The assumption

$g(0,\epsilon)=0$ (4)

is justified by analogy with a neuron, which has a refractory period
immediately following firing, during which it is input-insensitive. On the
other hand, since input has little effect once the spike-generating currents
have been activated, it is also reasonable to assume

$g(1,\epsilon)=0.$ (5)

Finally, we assume that $g$ is smooth:

$g\in C^{1}([0,1]\times[0,\infty)).$ (6)

We have defined phase as a number in $[0,1]$. It is mathematically tempting to
view it instead as a point on the unit circle $S^{1}$, identifying $\varphi$
with the complex number $e^{2\pi i\varphi}$. In that case, the mapping

$\varphi\mapsto\varphi+g(\varphi,\epsilon)$

(for fixed $\epsilon$) becomes the mapping

$S^{1}\rightarrow S^{1}$

described by

$e^{2\pi i\varphi}\mapsto e^{2\pi i\varphi}e^{2\pi ig(\varphi,\epsilon)}.$

Assumptions (2)–(6) make this mapping continuous. However, continuity of the
mapping $S^{1}\rightarrow S^{1}$ is a weaker condition than (2)–(6); it would
only require that the periodic extension of $g(\varphi,\epsilon)$ (fixed
$\epsilon\geq 0$) from $[0,1)$ to $\mathbb{R}$ be continuous up to jumps of
integer magnitude. Our more specific assumptions about $g$ are motivated by
neuroscience, as I have explained.

###  1.3 Two interacting pulse-coupled phase oscillators

Consider two identical phase oscillators $A$ and $B$ with period $T$. When $A$
reaches phase $1$ (fires), it sends a pulse to $B$, which causes the phase
$\varphi_{B}$ of $B$ to shift from $\varphi_{B}$ to
$\varphi_{B}+g(\varphi_{B},\epsilon)$. Similarly, when $B$ reaches phase 1
(fires), it sends a pulse to $A$, causing the shift
$\varphi_{A}\mapsto\varphi_{A}+g(\varphi_{A},\epsilon)$.

We can now ask questions about the dynamics of $A$ and $B$. For instance we
can ask whether synchrony is attracting, that is, whether the two oscillators
will synchronize their firing if started out sufficiently close in phase to
each other.

Consider a time when $\varphi_{A}=0$ and $\varphi_{B}\in(0,1)$. So $A$ has
just fired. We assume that it has had its effect on $B$ already. After time
$(1-\varphi_{B})T$ elapses, $B$ fires. At that time, $A$ is at phase
$1-\varphi_{B}$ and is advanced to phase
$1-\varphi_{B}+g(1-\varphi_{B},\epsilon)$. Now the roles reverse: $B$ is at
phase 0 and $A$ is at a phase
$1-\varphi_{B}+g(1-\varphi_{B},\epsilon)\in(0,1)$. After $A$ fires next and
has its effect on $B$, $B$ will be at phase $F(\varphi_{B},\epsilon)$ with

$F(\varphi,\epsilon)=\varphi-g(1-\varphi,\epsilon)+g(\varphi-g(1-\varphi,\epsilon),\epsilon).$
(7)

We call $F$ the strobe map — it is as though a strobe light were turned on
each time $A$ completes a cycle, to check where $B$ is. This idea for
analyzing synchronization was suggested by Peskin in [5]. I first saw the word
“strobe” for it in [6].

Since $F(\varphi,\epsilon)$ is a phase, we must always have $0\leq
F(\varphi,\epsilon)\leq 1$. Indeed this follows easily from (2). The two
oscillators implement fixed point iteration for the function
$F(\cdot,\epsilon)$:

$\varphi_{k+1}=F(\varphi_{k},\epsilon),~{}~{}~{}k=0,1,\ldots$ (8)

where $\varphi_{k}$ is the phase of $B$ after $A$ has completed its $k$-th
cycle and has had its effect on $B$. Therefore the analysis of synchronization
now simply becomes an analysis of a fixed point iteration. Our assumptions
imply that $\varphi=0$ and $\varphi=1$ are fixed points:

$F(0,\epsilon)=0,~{}~{}~{}F(1,\epsilon)=1.$

Synchrony is attracting if $0$ and $1$ are attracting fixed points.

In the standard analysis of fixed point iteration, derivatives at fixed points
plays a central role. As should be expected, the derivatives of
$F(\cdot,\epsilon)$ at $\varphi=0$ and $\varphi=1$ are the same. A
straightforward calculation shows the following lemma.

###### Lemma 1

Assume (1)–(7). Then

$\frac{\partial F}{\partial\varphi}(0,\epsilon)=\frac{\partial
F}{\partial\varphi}(1,\epsilon)=\left(1+\frac{\partial
g}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial
g}{\partial\varphi}(1,\epsilon)\right).$ (9)

The idea of the strobe map is usually attributed to Winfree [7], although
Peskin’s 1975 course notes [5], where the same idea appeared in a special
context, precede [7]. Peskin’s result was later generalized by Mirollo and
Strogatz in [8]. The literature on phase response functions and their use for
the analysis of networks of oscillators has since become vast. For a review of
phase response functions in neuroscience, see [9]. Phase response functions
have also been used in engineering; see for instance [10, 11, 12].

###  1.4 Infinitesimal phase response function

The linearization of $g$ around $\epsilon=0$,

$\tilde{g}(\varphi,\epsilon)=\frac{\partial
g}{\partial\epsilon}(\varphi,0)\epsilon,$ (10)

is commonly called the infinitesimal phase response function.

###### Lemma 2

Let $g\in C^{2}([0,1]\times[0,\infty))$ satisfy assumptions (1)–(5). Then for
sufficiently small $\epsilon>0$, (1)–(6) hold with $g$ replaced by
$\tilde{g}$.

Proof. Because of the factor of $\epsilon$ in (10), $\tilde{g}(\varphi,0)=0$
for all $\varphi\in[0,1]$. Because $g\equiv 0$ for $\varphi=0$, also
$\frac{\partial g}{\partial\epsilon}\equiv 0$ for $\varphi=0$, and the
analogous holds for $\varphi=1$. Therefore
$\tilde{g}(0,\epsilon)=\tilde{g}(1,\epsilon)=0$ for all $\epsilon\geq 0$.
Since $g\in C^{2}([0,1]\times[0,\infty))$, $\tilde{g}\in
C^{1}([0,1]\times[0,\infty))$. Finally, these properties imply that (2) and
(3) with $g$ replaced by $\tilde{g}$ hold for sufficiently small $\epsilon>0$.

The infinitesimal phase response function $\tilde{g}$ gives rise to the strobe
map

$\tilde{F}(\varphi,\epsilon)=\varphi-\tilde{g}(1-\varphi,\epsilon)+\tilde{g}(\varphi-\tilde{g}(1-\varphi,\epsilon),\epsilon).$

##  2 Conditions for synchrony to be attracting

Throughout this section we consider two identical phase oscillators $A$ and
$B$, interacting via a phase response function $g\in
C^{2}([0,1]\times[0,\infty))$ that satisfies (1)–(5). We assume that $F$ is
the strobe map defined by (7). We take $\tilde{g}$ to be the associated
infinitesimal phase response function; it is a phase response function for
sufficiently small $\epsilon>0$ by Lemma 2. The phrase “for sufficiently small
$\epsilon>0$” will appear often in the subsequent discussion; we will
abbreviate it by “for small $\epsilon$” from here on.

###  2.1 Strongly and weakly attracting synchrony

###### Definition 1

We say that synchrony is $g$-attracting if $0$ and $1$ are locally attracting
fixed points of $F(\cdot,\epsilon)$. We define $g$-repelling analogously.

The standard theory of fixed point iteration, together with eq. (9), shows:

###### Lemma 3

Synchrony is $g$-attracting if

$\left|\left(1+\frac{\partial
g}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial
g}{\partial\varphi}(1,\epsilon)\right)\right|<1,$ (11)

and it is $g$-repelling if

$\left|\left(1+\frac{\partial
g}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial
g}{\partial\varphi}(1,\epsilon)\right)\right|>1.$ (12)

It should be noted though that (11) and (12) are sufficient conditions for
synchrony to be $g$-attracting and $g$-repelling, respectively, not necessary
ones.

###### Definition 2

We say that synchrony is strongly $g$-attracting if (11) holds. We say that
synchrony is weakly $g$-attracting if it is $g$-attracting but not strongly
$g$-attracting. The terms strongly $g$-repelling and weakly $g$-repelling are
defined analogously.

To say that synchrony is strongly $g$-attracting is to say that the linearized
analysis of the fixed point iteration shows that $0$ and $1$ are attracting
fixed points of $F$. To avoid confusion, we stress that this refers to
linearization of $F$ around $\varphi=0$ and $\varphi=1$; $\epsilon>0$ is fixed
here. By contrast, $\tilde{g}$ is obtained by linearizing $g$ around
$\epsilon=0$.

When synchrony is strongly $g$-attracting, two oscillators $A$ and $B$
interacting through the phase response function $g$ converge to synchrony
exponentially fast when started out sufficiently close to synchrony.

Since $\tilde{g}$ is, for small $\epsilon$, a phase response function, we can
also talk about synchrony being $\tilde{g}$-attracting, strongly
$\tilde{g}$-attracting, and so on.

###### Lemma 4

Synchrony is strongly $\tilde{g}$-attracting for small $\epsilon$ if and only
if one of the following two conditions holds.

$\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)<0,$
(13)

or

$\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)=-\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\neq
0.$ (14)

Synchrony is strongly $\tilde{g}$-repelling for small $\epsilon$ if and only
if

$\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)>0,$
(15)

Proof. By definition, synchrony is strongly $\tilde{g}$-attracting if and only
if

$\left|\left(1+\frac{\partial\tilde{g}}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial\tilde{g}}{\partial\varphi}(1,\epsilon)\right)\right|<1.$

By definition of $\tilde{g}$, this means

$\left|\left(1+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)\epsilon\right)\left(1+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\epsilon\right)\right|<1,$

that is,

$\left|1+\left(\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\right)\epsilon+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\epsilon^{2}\right|<1.$

This clearly holds for sufficiently small $\epsilon>0$ if and only if either
(13) or (14) holds.

Similarly, synchrony is strongly $\tilde{g}$-repelling if and only if

$\left|1+\left(\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\right)\epsilon+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\epsilon^{2}\right|>1.$

This holds for sufficiently small $\epsilon>0$ if and only if (15) holds.

###  2.2 Why weakly attracting synchrony may not be unusual

If inputs arriving around the time of firing have very little effect — as is
typical for a neuron — then it seems not unreasonable that (4) and (5) could
be strengthened like this:

$g(0,\epsilon)=\frac{\partial g}{\partial\varphi}(0,\epsilon)=0,$ (16)

and

$g(1,\epsilon)=\frac{\partial g}{\partial\varphi}(1,\epsilon)=0.$ (17)

For example (16) clearly holds if there is an absolute refractory period, a
brief time following firing during which the neuron is entirely input-
insensitive. Both (16) and (17) hold for an Ermentrout-Kopell theta neuron
responding to instantaneous charge injections; see Section 4. Note that (16)
and (17) imply that neither (11) nor (12) hold.

###  2.3 Very strongly $\tilde{g}$-attracting synchrony

###### Definition 3

We say that synchrony is very strongly $\tilde{g}$-attracting for small
$\epsilon$ if (13) holds.

We don’t define very strongly $\tilde{g}$-repelling, nor do we define very
strongly $g$-attracting or very strongly $g$-repelling.

##  3 Results

###### Theorem 1

It is possible for synchrony to be weakly $\tilde{g}$-attracting for small
$\epsilon$, yet not $g$-attracting for any $\epsilon$. Two identical
Ermentrout-Kopell theta neurons interacting via instantaneous charge injection
provide an example. (See Section 4 for details.)

Theorem 2 strengthens Theorem 1.

###### Theorem 2

It is possible for synchrony to be weakly, or even strongly
$\tilde{g}$-attracting, yet strongly $g$-repelling for all $\epsilon$.

Theorem 3 provides conditions under which the infinitesimal phase response
function does make the correct prediction.

###### Theorem 3

(a) If synchrony is very strongly $\tilde{g}$-attracting for small $\epsilon$,
it is strongly $g$-attracting for small $\epsilon$. (b) If synchrony is
strongly $\tilde{g}$-repelling for small $\epsilon$, it is strongly
$g$-repelling for small $\epsilon$.

##  4 Proof of Theorem 1

###  4.1 The theta neuron

Ermentrout and Kopell [4] proposed to model a neuron as a point moving along
the unit circle, with its position $(\cos\theta(t),\sin\theta(t))$ governed by

$\frac{d\theta}{dt}=1-\cos\theta+I(1+\cos\theta).$ (18)

The right-hand side of (18) is positive for all $\theta$ if $I>0$, which we
will assume here. The period, that is, the time it takes for $\theta$ to
increase by $2\pi$ and therefore for the moving point to move once around the
unit circle, is then

$T=\int_{-\pi}^{\pi}\frac{dt}{d\theta}~{}\\!d\theta=\int_{-\pi}^{\pi}\frac{1}{1-\cos\theta+I(1+\cos\theta)}~{}\\!d\theta=\frac{\pi}{\sqrt{I}}.$
(19)

See [13], [14], or [1] for discussions of what (18) has to do with a neuron,
and also for an explanation why

$v=\frac{1}{2}+\frac{1}{2}\tan\frac{\theta}{2}$ (20)

should be considered the analogue of the “membrane potential” (interior
voltage) of the neuron — notwithstanding the fact that it becomes infinite
when $\theta$ is an odd multiple of $\pi$. We say that the theta neuron
“fires” when $\theta$ reaches an odd multiple of $\pi$.

###  4.2 Instantaneous charge injections

An instantaneous injection of a positive amount of charge into a neuron would
make the membrane potential jump. We will therefore assume that the response
of a theta neuron to a brief input pulse is the jump

$v\mapsto v+\Delta v.$ (21)

We’ll assume $\Delta v>0$ here (positive charge is injected, the input is
excitatory). Using eq. (20), eq. (21) translates into

$\theta\mapsto 2\arctan\left(\tan\frac{\theta}{2}+2\Delta v\right).$ (22)

###  4.3 The phase response function

Equation (22) does not yet describe the phase response, since $\theta$ is not
the same as the phase $\varphi$. The relation between $\theta$ and $\varphi$
is

$\varphi=\frac{1-\int_{\theta}^{\pi}\frac{1}{1-\cos s+I(1+\cos
s)}ds}{T}=\frac{1}{2}+\frac{1}{\pi}\arctan\left(\frac{\tan\frac{\theta}{2}}{\sqrt{I}}\right).$
(23)

We use (23) to express the right-hand side of (22) in terms of $\varphi$, and
conclude that (22) translates into

$\varphi\mapsto\varphi+\frac{1}{\pi}\arctan\left(\tan\left(\left(\varphi-\frac{1}{2}\right)\pi\right)+\frac{2\Delta
v}{\sqrt{I}}\right)-\left(\varphi-\frac{1}{2}\right).$

To simplify the notation, we write

$\epsilon=\frac{2\Delta v}{\sqrt{I}}.$

So

$g(\varphi,{\epsilon})=\frac{1}{\pi}\arctan\left(\tan\left(\left(\varphi-\frac{1}{2}\right)\pi\right)+{\epsilon}\right)-\left(\varphi-\frac{1}{2}\right).$
(24)

A straightforward computation verifies (16) and (17) now.

###  4.4 Synchrony is neutrally $g$-stable for all $\epsilon\geq 0$

We insert (24) into (7) and find a surprise:

$F(\varphi,{\epsilon})=\varphi~{}~{}~{}\mbox{for all $\varphi\in[0,1]$,
~{}$\epsilon\geq 0$}.$ (25)

The exchange of one pulse from $B$ to $A$ and one from $A$ to $B$ brings the
phase difference between the two theta neurons back to where it started. Any
initial phase difference persists. This is true for all $\epsilon>0$.

###  4.5 Synchrony is weakly $\tilde{g}$-attracting for small $\epsilon$

We now replace (24) by its local linear approximation near $\epsilon=0$:

$\tilde{g}(\varphi,{\epsilon})=\frac{1}{\pi}~{}\\!\frac{1}{1+\tan^{2}\left(\left(\varphi-\frac{1}{2}\right)\pi\right)}~{}\\!\epsilon.$
(26)

For sufficiently small $\epsilon>0$, for example for $\epsilon\leq 1$,
$\tilde{g}<1-\varphi$ for $\varphi\in(0,1)$, so $\tilde{g}$ is indeed a valid
phase response function. We insert (26) into (7) to find the strobe map

$\tilde{F}(\varphi,{\epsilon})=\varphi+\frac{\epsilon}{2\pi}\cos(2\pi\varphi)-\frac{\epsilon}{2\pi}\cos\left(2\pi\varphi+\epsilon\cos(2\pi\varphi)-\epsilon\right).$
(27)

Let $\epsilon>0$ be fixed. A straightforward calculation shows that in the
limit as $\varphi\rightarrow 0$,

$\tilde{F}(\varphi,\epsilon)=\varphi-2\epsilon^{2}\pi^{2}\varphi^{3}+O(\varphi^{4})<\varphi.$
(28)

This implies that for sufficiently small $\varphi$,
$\tilde{F}(\varphi,\epsilon)<\varphi$. Therefore the sequence
$\varphi_{0},\varphi_{1},\ldots$ generated by fixed point iteration is
decreasing if $\varphi_{0}$ is close enough to $0$, and since it is bounded
below by $0$ it must converge. The limit must be a fixed point of
$\tilde{F}(\cdot,\epsilon)$, and for sufficiently small $\varphi_{0}>0$, that
fixed point must be $0$. So $\varphi=0$ is an attracting fixed point.

Arguments like those given in the preceding paragraph show that $\varphi=1$ is
an attracting fixed point as well. We conclude that synchrony is
$\tilde{g}$-attracting. It is weakly $\tilde{g}$-attracting because
$\frac{\partial\tilde{F}}{\partial\varphi}(0,\epsilon)=\frac{\partial\tilde{F}}{\partial\varphi}(1,\epsilon)=1$.
On the other hand, when the phase response function is $g$, synchrony is just
neutrally stable, not attracting.

This proves Theorem 1.

##  5 Proof of Theorem 2

###  5.1 Example 1: Weakly $\tilde{g}$-attracting, yet strongly $g$-repelling

This example is a modification of the phase response function of the theta
neuron:

$g(\varphi,\epsilon)=\frac{1}{\pi}\arctan\left(\tan\left(\left(\varphi-\frac{1}{2}\right)\pi\right)+{\epsilon}\right)-\left(\varphi-\frac{1}{2}\right)+\varphi(1-\varphi)^{2}\epsilon^{2}.$
(29)

Compare this with eq. (24): It is the phase response function for the theta
neuron, plus $\varphi(1-\varphi)^{2}\epsilon^{2}$. Unfortunately I am not
aware of any neuroscience motivation for this formula. I added the term
$\varphi(1-\varphi)^{2}\epsilon^{2}$ simply to construct an interesting
mathematical example.

Using that (16) and (17) hold for the phase response function of the theta
neuron, we have

$\left(1+\frac{\partial{g}}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial{g}}{\partial\varphi}(1,\epsilon)\right)=1+\epsilon^{2}>1.$

So synchrony is strongly ${g}$-repelling for all $\epsilon>0$. However, the
infinitesimal phase response function, namely the linearization around
$\epsilon=0$, is the same as for the theta neuron — we just added a term of
size $O(\epsilon^{2})$. Synchrony is weakly $\tilde{g}$-attracting for small
$\epsilon$, but strongly $g$-repelling for all $\epsilon>0$.

###  5.2 Example 2: Strongly $\tilde{g}$-attracting yet strongly $g$-repelling

Let

$g(\varphi,\epsilon)=\varphi(1-\varphi)\epsilon-2\varphi(\varphi-1)^{2}(2\varphi-1)\epsilon^{2}.$
(30)

Here

$\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)=-\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)=1.$

Therefore synchrony is strongly $\tilde{g}$-attracting for small $\epsilon$ by
Lemma 4. However,

$\left(1+\frac{\partial
g}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial
g}{\partial\varphi}(1,\epsilon)\right)=\left(1+\epsilon+2\epsilon^{2}\right)\left(1-\epsilon\right)=1+\epsilon^{2}-2\epsilon^{3}>1$

for small $\epsilon$, so synchrony is strongly $g$-repelling for small
$\epsilon$.

This concludes the proof of Theorem 2.

##  6 Proof of Theorem 3

Now suppose that synchrony is very strongly $\tilde{g}$-attracting. Recall
that by definition, this means that (13) holds. Then, expanding around
$\epsilon=0$, we find

$\displaystyle\left(1+\frac{\partial
g}{\partial\varphi}(0,\epsilon)\right)\left(1+\frac{\partial
g}{\partial\varphi}(1,\epsilon)\right)$ $\displaystyle=$
$\displaystyle\left(1+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)\epsilon+o(\epsilon)\right)\left(1+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\epsilon+o(\epsilon)\right)$
$\displaystyle=$ $\displaystyle
1+\left(\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(0,0)+\frac{\partial^{2}g}{\partial\varphi\partial\epsilon}(1,0)\right)\epsilon+o(\epsilon)<1$

for small $\epsilon$. This proves part (a) of Theorem 3. Part (b) is proved
analogously.

##  7 Summary and a concluding comment

###  7.1 Summary

The infinitesimal phase response function first approximates the behavior of
$g$ near $\epsilon=0$, then analyzes synchrony by studying the behavior near
$\varphi=0$ and $\varphi=1$. What one really wants to know is the behavior
near $\varphi=0$ and $\varphi=1$ for fixed small $\epsilon>0$. There is no
reason why these should be the same, and the examples given here show that
they aren’t always the same.

###  7.2 Comparison with the Hartman-Grobman theorem

The Hartman-Grobman theorem states that if linearization near an equilibrium
of a dynamical system yields a definite prediction about the dynamics near the
equilibrium, that prediction holds (qualitatively) for the non-linear problem.
See [15] for a particularly nice version of the theorem. The examples I have
given here show that if linearization around $\epsilon=0$ yields a definite
prediction about whether synchrony is attracting, that prediction may still be
false for the non-linear problem, for arbitrarily small but possible
interaction strengths. The analogue of the Hartman-Grobman theorem that one
might expect at first sight does not hold here.

Theorem 2 shows that even if linearization of $g$ around $\epsilon=0$, then
linearization of $F$ around $\varphi=0$ and $\varphi=1$, yields a definite
prediction for small $\epsilon>0$, that prediction may still be false. In
other words, the analogue of the Hartman-Grobman theorem that one might expect
at second sight is also false.

Theorem 3 gives a condition under which the prediction of the analysis based
on $\tilde{F}$ does hold for $F$, but Section 2.2 explains why situations in
which that condition is satisfied may not be typical.

Acknowledgments. I thank an anonymous reviewer for an exceptionally thorough
and thoughtful report, which led to very substantial clarifications,
improvements, and extensions. I also thank Nancy Kopell, whose comments and
questions resulted in Theorems 2 and 3.

## References

  * [1] C. Börgers, An Introduction to Modeling Neuronal Dynamics, Texts in Applied Mathematics 66, Springer, 2017.
  * [2] E. Phoka, H. Cuntz, A. Roth, and M. Häusser, A new approach for determining phase response curves reveals that Purkinje cells can act as perfect integrators, PLOS Comp. Biol. 6 (2010) e1000768.
  * [3] C. Börgers, M. Krupa, and S. Gielen, The response of a population of classical Hodgkin-Huxley neurons to an inhibitory input pulse, J. Comp. Neurosic. 28 (2010) 509–526.
  * [4] G. B. Ermentrout, N. Kopell, Parabolic bursting in an excitable system coupled with a slow oscillation, SIAM J. Appl. Math 46 (1986) 233–253.
  * [5] C. S. Peskin, Mathematical Aspects of Heart Physiology, Courant Institute of Mathematical Sciences, New York University, New York, 1975.
  * [6] S. H. Strogatz, Sync, Hachette Books, 2003.
  * [7] A. T. Winfree, The Geometry of Biologial Time, Springer-Verlag, 1980.
  * [8] R. E. Mirollo, S. H. Strogatz, Synchronization of pulse-coupled biological oscillators, SIAM J. Appl. Math. 50 (1990) 1645–1662.
  * [9] R. M. Smeal, G. B. Ermentrout, J. A. White, Phase-response curves and synchronized neural networks, Philos. Trans. R. Soc. Lond. B 365 (2010) 2407–2422.
  * [10] Y. Kato, N. Yamamoto, H. Nakao, Semiclassical phase reduction theory for quantum synchronization, Phs. Rev. Research 1, 033012 (2019).
  * [11] I. A. Loe, H. Nakao, Y. Jimbo, K. Kotani, Phase-reduction for synchronization of oscillating flow by perturbation on surrounding structure, J. Fluid Mech. 911 (R2) (https://doi.org/10.1017/jfm.2020.1110b) (2021).
  * [12] Y. Wang, F. Núnez, F. J. Doyle, Increasing sync rate of pulse-coupled oscillators via phase response function design: Theory and application to wireless networks, IEEE Trans. Control Syst. Technol. 21 (4) (2013) 1455–1462.
  * [13] B. B. Ermentrout, Type I membranes, phase resetting curves, and synchrony, Neural Compl 8 (1996) 979–1001.
  * [14] C. Börgers, N. Kopell, Effects of noisy drive on rhythms in networks of excitatory and inhibitory neurons, Neural Comp. 17 (3) (2005) 557–608.
  * [15] M. Guysinsky, B. Hasselblatt, V. Rayskin, Differentiability of the Hartman-Grobman linearization, Discrete Contin. Dyn. Syst. 9 (4) (2003) 979–984.