# Policy Optimization for Continuous Reinforcement Learning

Hanyang Zhao Wenpin Tang David D. Yao Department of Industrial Engineering and
Operations Research, Columbia University, New York, New York 10027, USA,
<EMAIL_ADDRESS>of Industrial Engineering and Operations
Research, Columbia University, New York, New York 10027, USA,
<EMAIL_ADDRESS>of Industrial Engineering and Operations
Research, Columbia University, New York, New York 10027, USA<EMAIL_ADDRESS>
###### Abstract

We study reinforcement learning (RL) in the setting of continuous time and
space, for an infinite horizon with a discounted objective and the underlying
dynamics driven by a stochastic differential equation. Built upon recent
advances in the continuous approach to RL, we develop a notion of occupation
time (specifically for a discounted objective), and show how it can be
effectively used to derive performance-difference and local-approximation
formulas. We further extend these results to illustrate their applications in
the PG (policy gradient) and TRPO/PPO (trust region policy optimization/
proximal policy optimization) methods, which have been familiar and powerful
tools in the discrete RL setting but under-developed in continuous RL. Through
numerical experiments, we demonstrate the effectiveness and advantages of our
approach.

Key Words. exploratory stochastic control, occupation time, performance
difference, policy optimization

##  1 Introduction

Reinforcement Learning (RL, [52]) has been successfully applied to wide-
ranging domains in the past decade, including achieving superhuman performance
in games like Atari and Go [34, 48, 49], enhancing Large Language Models using
human feedback [8, 10], and showing potentials in improving traditional model-
based decisions in healthcare, inventory management, and finance [9, 30, 33].
Most existing works, including all references cited above, are formulated and
solved as discrete-time sequential optimization problems such as Markov
decision processes (MDPs, [42]). Yet in many applications, agents may need to
monitor and interact with the random environment at an ultra-high frequency
(e.g., autonomous driving, robot navigation, and high-frequency stock
trading), which calls for a continuous-time/space approach. Recent years have
witnessed a fast growing body of research that has extended the frontiers of
continuous RL in several important directions including, for instance,
modeling the noise or randomness in the environment dynamics as following a
stochastic differential equation (SDE), and incorporating an entropy-based
regularizer into the objective function [59] to facilitate the exploration-
exploitation tradeoff; designing model-free methods and algorithms, along with
applications to portfolio optimization [18, 20, 21, 22]; studying regret
bounds [5, 54], and so forth.

In this paper, we continue the above trend in continuous RL, focusing on an
infinite horizon formulation with a discounted objective and the underlying
dynamics driven by an SDE [24, 39]. We are specifically motivated by the
following two questions. (Q1) The visitation frequency in MDP (with a
discounted objective) is defined as:
$\rho(s)=\sum_{t=0}^{\infty}\gamma^{t}\cdot\mathbb{P}(Y_{t}=s)$, where
$\\{Y_{t}\\}$ is a Markov chain with state space $\mathcal{S}:=\\{s\\}$, and
$\gamma\in(0,1)$ is a discount factor. It plays an important role in many RL
algorithms for MDP. So, a natural question is, what is the continuous
counterpart of $\rho(s)$? (Q2) For continuous RL, how can we characterize the
difference in performance between two policies? In particular, can we derive
performance-difference formulas similar to those in the MDP case [23, 45]? Can
we adapt and apply the ideas and tools of the efficient policy optimization
methods (e.g., [45, 47]) to the continuous RL setting? Main contributions. We
provide a unified theory/framework for policy optimization in continuous time
and space. Specifically, we have addressed the above two questions (Q1) and
(Q2) by developing the notion of occupation time/measure, specifically for a
discounted objective, and focusing on its associated $q$-value. Based on these
two quantities, we derive the performance-difference formula for continuous RL
by means of perturbation analysis. Leveraging the performance-difference
formula, we develop the continuous counterparts of the policy gradient (PG,
[53]) and also propose the local approximation for the performance metric, for
which we derive a bound on it and allow the development of a minorization-
majorization (MM) algorithm. We further develop the continuous counterparts of
trust region policy optimization/ proximal policy optimization (TRPO/PPO)
methods in [45, 47], which have been familiar and powerful tools in the
discrete RL setting but under-developed in continuous RL, as approximations to
the previous algorithms. (What is worth mentioning is that these policy
optimization algorithms do not require any a priori discretization of time and
space.) Through numerical examples we show the convergence of these algorithms
when applied to certain stochastic control tasks in continuous time and space.
Related works. One line of research on continuous RL focuses on modeling the
underlying dynamics as a deterministic system, typically following a
deterministic ordinary differential equation. Several papers [9, 37, 44] solve
the problems via a priori discretization in either time or space; [13]
develops a framework to apply the temporal difference to the continuous
setting, and proposes algorithms that combine value iteration or advantage
update as in [3, 4, 6] to avoid explicit discretization; [37] further
investigates policy gradient methods, followed by more recent studies on
model-free continuous RL methods [25, 29, 57] or model-based ones [14]; [55]
studies the sensitivity of existing off-policy algorithms along with advantage
updating to propose continuous RL algorithms that are robust to time
discretization. The formulation of continuous RL in a stochastic setting
(i.e., with the state process driven by an SDE), can be traced back to [38],
which however provides no data-driven solution. Recently, [59] develops an
exploratory control model for the continuous RL. Built upon this approach and
for a finite-horizon objective, [20] studies policy evaluation, and [21]
policy gradient. Furthermore, [22] brings forth the notion of $q$-value, which
leads to a continuous analogue of $Q$-learning. Also worth noting is [2, 28],
which studies RL in the mean-field regime where continuous-time processes
occur in the limit, and [16] extends the study to jump-diffusion processes. In
discrete-time MDPs, the body of research on bounding the performance
difference between two policies also relates to our work: [1, 45] develop a
policy improvement bound for the discounted total reward; [12, 64] studies the
long-run average reward, and [11] proposes a bound that is continuous with
respect to the discount factor.

Organization of the paper. In Section 2 we present the continuous RL
formulation and develop necessary tools. The main results, the performance-
difference formula (Theorem 2) and the bound (Theorem 5) are provided in
Section 3. In Section 4 we propose two algorithms, policy gradient with random
rollout and PPO with adaptive penalty, based on our analyses and theoretical
results; and illustrate their performance via numerical experiments.
Concluding remarks are summarized in Section 5.

Notation. For a measurable set $\mathcal{A}$, denote
$\mathcal{P}(\mathcal{A})$ for the set of probability distributions over
$\mathcal{A}$. For a vector $x$, denote by $||x||_{2}$ the Euclidean norm of
$x$. For a matrix $A$, denote by $||A||_{F}$ the Frobenius norm of $A$, and
$A^{2}:=AA^{\top}$ where $A^{\top}$ is the transpose of $A$. For a positive-
definite matrix $A$, denote by $A^{\frac{1}{2}}$ the square root matrix of
$A$. For $A,B$ two matrices of the same size, denote by $A\circ B$ the inner
product of $A$ and $B$. For a function $f$ on an Euclidean space, $\nabla f$
(resp. $\nabla^{2}f$) denotes the gradient (resp. the Hessian) of $f$. For
$\pi,\pi^{\prime}\in\mathcal{P}(A)$, denote by $W_{2}(\pi,\pi^{\prime})$ the
Wasserstein-2 distance between $\pi$ and $\pi^{\prime}$, and
$D_{\mathrm{KL}}(\pi||\pi^{\prime})$ the KL-divergence between $\pi$ and
$\pi^{\prime}$.

##  2 Formulation and Preliminaries

Continuous RL. We start with a quick formulation of the continuous RL, based
on the same modeling framework as in [59]. Assume that the state space is
$\mathbb{R}^{n}$, and denote by $\mathcal{A}$ the action space. Let
$\pi(\cdot\mid x)\in\mathcal{P}(\mathcal{A})$ be a (state) feedback policy
given the state $x\in\mathbb{R}^{n}$. A continuous RL problem is formulated by
a distributional (or relaxed) control approach [62], which is motivated by the
trial and error process in RL. The state dynamics $(X^{a}_{s},\,s\geq 0)$ is
governed by the Itô process:

$\mathrm{d}X_{s}^{a}=b\left(X_{s}^{a},a_{s}\right)\mathrm{d}s+\sigma\left(X_{s}^{a},a_{s}\right)\mathrm{d}B_{s},\quad
X^{a}_{0}\sim\mu\in\mathcal{P}(\mathbb{R}^{n}),$ (1)

where $(B_{t},\,t\geq 0)$ is the $m$-dimensional Brownian motion,
$b:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}^{n}$,
$\sigma:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}^{n\times m}$, and the
action $a_{s}$ is generated from the distribution $\pi\left(\cdot\mid
X^{a}_{s}\right)$ by external randomization. To avoid technical difficulties,
we assume that the stochastic processes (1) (and (3), (9) below) are well-
defined, see [24, Section 5.3] or [50, Chapter 6] for background.

From now on, write $(X^{\pi}_{s},a^{\pi}_{s})$ for the state and action at
time $s$ given by the process (1) under the policy $\pi=\\{\pi(\cdot\mid
x)\in\mathcal{P}(\mathcal{A}):x\in\mathbb{R}^{n}\\}$. The goal here is to find
the optimal feedback policy $\pi^{*}$ that maximizes the expected discounted
reward over an infinite time horizon:

${V^{*}:=}\max_{\pi}\mathbb{E}\left[\int_{0}^{+\infty}e^{-\beta
s}\left[r\left(X_{s}^{\pi},a_{s}^{\pi}\right)+\gamma
p\left(X_{s}^{\pi},a_{s}^{\pi},\pi\left(\cdot\mid
X_{s}^{\pi}\right)\right)\right]\mathrm{d}s\mid X_{0}^{\pi}\sim\mu\right],$
(2)

where $r:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}^{+}$ is the running
reward of the current state and action $(X^{\pi}_{s},a^{\pi}_{s})$;
$p:\mathbb{R}^{n}\times\mathcal{A}\times\mathcal{P}(\mathcal{A})\mapsto\mathbb{R}$
is a regularizer which facilitates exploration (e.g., in [59], $p$ is taken as
the differential entropy defined by $p(x,a,\pi(\cdot))=-\log\pi(a)$);
$\gamma\geq 0$ is a weight parameter on exploration (also known as the
“temperature” parameter); and $\beta>0$ is a discount factor that measures the
time-depreciation of the objective value (or the impatience level of the
agent). Performance metric. A standard approach to solving the problem in (2)
is to find a sequence of policies $\pi_{k}=\\{\pi_{k}(\cdot\mid
x):x\in\mathbb{R}^{n}\\}$, $k=1,2,\ldots$ such that the value functions
following the policies $\pi_{k}$ will converge to $V^{*}$, or be at least
increasing in $k$, i.e., demonstrating policy improvement.

Given a policy $\pi(\cdot)$, let
$\tilde{b}(x,\pi(\cdot)):=\int_{\mathcal{A}}b(x,a)\pi(a)\mathrm{d}a$ and
$\tilde{\sigma}(x,\pi(\cdot)):=\left(\int_{\mathcal{A}}\sigma^{2}(x,a)\pi(a)\mathrm{d}a\right)^{\frac{1}{2}}$.
Assume (for technical purpose) that $\tilde{\sigma}(x,\pi(\cdot))$ is positive
definite for every $x\in\mathbb{R}^{n}$. It is sometimes more convenient to
consider the following equivalent SDE representation of (1):

$\mathrm{d}\tilde{X}_{s}=\tilde{b}\left(\tilde{X}_{s},\pi(\cdot\mid\tilde{X}_{s})\right)\mathrm{d}s+\tilde{\sigma}\left(\tilde{X}_{s},\pi(\cdot\mid\tilde{X}_{s})\right)\mathrm{d}\tilde{B}_{s},\quad\tilde{X}_{0}\sim\mu,$
(3)

in the sense that there exists a probability measure $\tilde{\mathbb{P}}$
which supports the $m$-dimensional Brownian motion $(\tilde{B}_{s},\,s\geq
0)$, and for each $s\geq 0$, the distribution of $\tilde{X}_{s}$ under
$\tilde{\mathbb{P}}$ agrees with that of $X_{s}$ under $\mathbb{P}$ defined by
(1), see Appendix A. Note that the dynamics in (3) does not require external
randomization. We also set
$\tilde{r}(x,\pi):=\int_{\mathcal{A}}r(x,a)\pi(a)\mathrm{d}a$ and
$\tilde{p}(x,\pi):=\int_{\mathcal{A}}p(x,a,\pi)\pi(a)\mathrm{d}a$.

We formally define the (state) value function given the feedback policy
$\\{\pi(\cdot\mid x):x\in\mathbb{R}^{n}$} by

$\displaystyle V(x;\pi)$
$\displaystyle:=\mathbb{E}\left[\int_{0}^{+\infty}e^{-\beta
s}\left[r\left(X_{s}^{\pi},a_{s}^{\pi}\right)+\gamma
p\left(X_{s}^{\pi},a_{s}^{\pi},\pi\left(\cdot\mid
X_{s}^{\pi}\right)\right)\right]\mathrm{d}s\mid X_{0}^{\pi}=x\right]$ (4)
$\displaystyle\,=\mathbb{E}\left[\int_{0}^{\infty}e^{-\beta
s}\left[\tilde{r}\left(\tilde{X}_{s}^{\pi},\pi(\cdot\mid\tilde{X}_{s}^{\pi})\right)+\gamma\tilde{p}\left(\tilde{X}_{s}^{\pi},\pi(\cdot\mid\tilde{X}_{s}^{\pi})\right)\right]\mathrm{d}s\mid\tilde{X}_{0}^{\pi}=x\right],$

which, under suitable conditions on model parameters $(b,\sigma,r,p)$ and the
policy $\pi$, is characterized by the Hamilton-Jacobi equation (see [21, 56]):

$\beta V(x;\pi)-\tilde{b}(x,\pi)\cdot\nabla
V(x;\pi)-\frac{1}{2}\tilde{\sigma}^{2}(x,\pi)\circ\nabla^{2}V(x;\pi)-\tilde{r}(x,\pi)-\gamma\tilde{p}(x,\pi)=0.$
(5)

More technical details regarding the above formulation are spelled out in the
Appendix. We can now define the performance metric as follows:

$\eta(\pi):=\int_{\mathbb{R}^{n}}V(x;\pi)\mu(dx),$ (6)

so $V^{*}=\max_{\pi}\eta(\pi)$. The main task of the continuous RL is to
approximate $\max_{\pi}\eta(\pi)$ by constructing a sequence of policies
$\pi_{k}$, $k=1,2,\ldots$ recursively such that $\eta(\pi_{k})$ is non-
decreasing.

Policy evaluation. Let’s first recall a general approach in [20], which can be
used to learn the state value function in (4) or the performance metric in (6)
for a given policy $\pi$. The idea is that for any $T>0$ and a suitable test
process $(\xi_{t},\,t\geq 0)$,

$\mathbb{E}\int_{0}^{T}\xi_{t}\,\left[\mathrm{d}V\left(X_{t}^{\pi};\pi\right)+r\left(X_{t}^{\pi},a_{t}^{\pi}\right)\mathrm{d}t+\gamma
p\left(X_{t}^{\pi},a_{t}^{\pi},\pi\left(\cdot\mid
X_{t}^{\pi}\right)\right)\mathrm{d}t-\beta
V\left(X_{t}^{\pi};\pi\right)\mathrm{d}t\right]=0.$ (7)

If we parameterize $V(x;\pi)=V^{\phi}(x)$ and choose the special test function
$\xi_{t}=\frac{\partial V^{\phi}\left(X_{t}^{\pi}\right)}{\partial\phi}$,
stochastic approximation leads to the online update:

$\phi\leftarrow\phi+\alpha\frac{\partial
V^{\phi}\left(X_{t}^{\pi}\right)}{\partial\phi}[\mathrm{d}V^{\phi}\left(X_{t}^{\pi}\right)+r\left(X_{t}^{\pi},a_{t}^{\pi}\right)\mathrm{d}t+\gamma
p\left(X_{t}^{\pi},a_{t}^{\pi},\pi\left(\cdot\mid
X_{t}^{\pi}\right)\right)\mathrm{d}t-\beta
V^{\phi}\left(X_{t}^{\pi}\right)\mathrm{d}t],$ (8)

where $\alpha>0$ is the learning rate. This recovers the mean-squared TD error
(MSTDE) method for policy evaluation in the discrete RL [51].

$q$-value. The Q-value function [61] and the advantage function [3, 35] in
discrete-time MDPs play a critical role in reinforcement learning theory and
algorithms. However, as pointed out in [55], these concepts will not apply
when the time interval shrinks to $0$ (as in the continuous setting). To
derive algorithms that fit the need of a continuous stochastic environment,
[55, 22] proposed the advantage rate function. Namely, given a policy $\pi$
and $(t,x,a)\in[0,\infty)\times\mathbb{R}^{n}\times\mathcal{A}$, consider a
“perturbed" policy $\hat{\pi}$ as follows. It takes the action
$a\in\mathcal{A}$ on $[t,t+\Delta t)$ where $\Delta t>0$, and then follows
$\pi$ on $[t+\Delta t,\infty)$. The corresponding state process
$X^{\hat{\pi}}$ given $X_{t}^{\hat{\pi}}=x$ is broken into two pieces. On
$[t,t+\Delta t)$, it is $X^{a}$ which is the solution to

$\mathrm{d}X_{s}^{a}=b\left(X_{s}^{a},a\right)\mathrm{d}s+\sigma\left(X_{s}^{a},a\right)\mathrm{d}B_{s},s\in[t,t+\Delta
t),\quad X_{t}^{a}=x,$ (9)

while on $[t+\Delta t,\infty)$, it is $X^{\pi}$ following (3) with the initial
time-state pair $\left(t+\Delta t,X_{t+\Delta t}^{a}\right)$. With $\Delta
t>0$ as time discretization, the generalization of the conventional Q-function
can be expressed as

$\displaystyle Q_{\Delta
t}(x,a;\pi)=V(x;\pi)+\left[\mathcal{H}^{a}\left(x,\frac{\partial V}{\partial
x}\left(x;\pi\right),\frac{\partial^{2}V}{\partial
x^{2}}\left(x;\pi\right)\right)-\beta V\left(x;\pi\right)\right]\Delta
t+o(\Delta t).$ (10)

where $\mathcal{H}^{a}(x,y,A):=b(x,a)\cdot y+\frac{1}{2}\sigma^{2}(x,a)\circ
A+r(x,a)$ is the (generalized) Hamilton function in stochastic control theory
[62]. This motivates the following definition.

######  Definition 1.

[22] For a given policy $\pi\in\Pi$ and
$(x,a)\in\mathbb{R}^{n}\times\mathcal{A}$, define the $q$-value as

$q(x,a;\pi):=\lim_{\Delta t\rightarrow 0}\frac{Q_{\Delta
t}(x,a;\pi)-V(x;\pi)}{\Delta t}=\mathcal{H}^{a}\left(x,\frac{\partial
V}{\partial x}\left(x;\pi\right),\frac{\partial^{2}V}{\partial
x^{2}}\left(x;\pi\right)\right)-\beta V\left(x;\pi\right),$ (11)

which represents the instantaneous advantage rate of an action in a given
state under a given policy.

##  3 Main Results

This section is concerned with theoretical developments. In Section 3.1, we
define the (discounted) occupation time/measure which is the continuous analog
of visitation frequency in MDPs. It is crucial in deriving the performance-
difference formula in Section 3.2, which spins off two different algorithms –
policy gradient and TRPO/PPO. In Section 3.3, we propose a local approximation
for the performance metric, and derive a bound from which an MM algorithm is
constructed.

###  3.1 Discounted Occupation Time

Here we first provide an answer to (Q1) by defining the notion of discounted
occupation time for the continuous RL.

######  Definition 2.

Let $X=(X_{t},\,t\geq 0)$ be governed by the SDE (3), and assume that it has a
probability density function $p^{\pi}(\cdot,t)$ at each time $t$. For each
$x\in\mathbb{R}^{n}$ and $t\geq 0$, define the $\beta$-discounted occupation
time of $X$ at the state $x$ by

$d^{\pi}_{\mu}(x):=\int_{0}^{\infty}e^{-\beta s}p^{\pi}(x,s)ds.$ (12)

So $d^{\pi}_{\mu}(\cdot)$ induces a finite measure on $\mathbb{R}^{n}$ with a
total mass of $\beta^{-1}$, which we call the discounted occupation measure.

In probability theory, the definition in (12) is referred to as the
$\beta$-potential of $X$, which gives discounted visitation frequencies of the
state process. We record the following result, which will be useful in the
derivation of the performance-difference formula. It is a consequence of the
occupation time formula [41, 43].

######  Lemma 1.

Under the conditions in 2, we have $\mathbb{E}\int_{0}^{\infty}e^{-\beta
s}\varphi\left(X_{s}\right)\mathrm{d}s=\int_{\mathbb{R}^{n}}d_{\mu}^{\pi}(x)\varphi(x)\mathrm{d}x$,
for any measurable function $\varphi:\mathbb{R}^{n}\mapsto\mathbb{R}_{+}$ for
which the expectation exists.

###  3.2 Performance-Difference Formula

We are now ready to answer (Q2) by deriving the performance-difference formula
between two policies in terms of the discounted occupation time in (12) and
the $q$-values in (11).

######  Theorem 2.

: Given two feedback policies $\hat{\pi}=\\{\hat{\pi}(\cdot\mid
x):x\in\mathbb{R}^{n}\\}$ and $\pi=\\{\pi(\cdot\mid x):x\in\mathbb{R}^{n}\\}$,
we have

$\eta(\hat{\pi})-\eta(\pi)=\int_{\mathbb{R}^{n}}d^{\hat{\pi}}_{\mu}(x)\left[\int_{\mathcal{A}}\hat{\pi}(a\mid
x)\left(q(x,a;\pi)+\gamma
p(x,a,\hat{\pi})\right)\mathrm{d}a\right]\mathrm{d}x$ (13)

Proof sketch. The full proof is detailed in Appendix B.1. The essence of the
proof is to use the perturbation theory and properties of the discounted
occupation time. Define an operator
$\mathcal{L}^{\pi}:C^{2}(\mathbb{R}^{n})\mapsto C(\mathbb{R}^{n})$ associated
with the diffusion process as:

$\displaystyle\left(\mathcal{L}^{\pi}\varphi\right)(x)$
$\displaystyle:=-\beta\varphi(x)+\tilde{b}(x,\pi)\cdot\nabla\varphi(x)+\frac{1}{2}\tilde{\sigma}(x,\pi)^{2}\circ\nabla^{2}\varphi(x).$
(14)

Then the Hamilton-Jacobi equation that characterizes the state value function
can be expressed as:

$-\mathcal{L}^{\pi}V(x;\pi)=\tilde{r}(x,\pi)+\gamma\tilde{p}(x,\pi)$ (15)

Note that for any $\varphi\in C^{2}(\mathbb{R}^{n})$, we have
$\int_{\mathbb{R}^{n}}d^{\pi}_{\mu}(y)(-\mathcal{L}^{\pi}\varphi)(y)\mathrm{d}y=\int_{\mathbb{R}^{n}}\varphi(y)\mu(\mathrm{d}y)$.
This allows us to express the performance difference in model-dynamics related
terms:

$\eta(\hat{\pi})-\eta(\pi)=\int_{\mathbb{R}}d^{\hat{\pi}}_{\mu}(y)\left[(\mathcal{L}^{\hat{\pi}}-\mathcal{L}^{\pi})V(y;\pi)+\tilde{r}(y,\hat{\pi})+\gamma\tilde{p}(y,\hat{\pi})-\tilde{r}(y,\pi)-\gamma\tilde{p}(y,\pi)\right]\mathrm{d}y.$
(16)

What remains is to reduce the above to the desired result in (13). $\square$
As discussed in Section 2, our main task is to construct (algorithmically) a
sequence of policies $\pi_{k}$ along which the performance improves. Here we
illustrate how some well known approaches of policy improvement (from $\pi$ to
$\hat{\pi}$) are instances of the performance difference formula (13). (a)
$q$-learning and soft $q$-learning. Since $d^{\hat{\pi}}_{\mu}\geq 0$, we only
need to ensure that for all $x\in\mathbb{R}^{n}$,
$\int_{\mathcal{A}}\hat{\pi}(a\mid x)\left(q(x,a;\pi)+\gamma
p(x,a,\hat{\pi})\right)\mathrm{d}a\geq 0$. This boils down to the problem that
for any $x$, find $v\equiv\hat{\pi}(\cdot|x)$ to maximize

$\int_{\mathcal{A}}v(a)\left(q(x,a;\pi)+\gamma p(x,a,v)\right)\mathrm{d}a.$
(17)

There are two special cases:

(i) If $p(x,a,v)\equiv 0$, then $v=\delta(a^{*})$ where
$a^{*}=\arg\max_{a}q(x,a,\pi)$. This is essentially the counterpart of
$Q$-Learning [61] in the discrete time, which we call $q$-learning.

(ii) If $p(x,a,v)=-\log(v(a))$, this is known as the entropy regularizer [17,
58]. Concretely, we need to solve

$\max_{v\in\mathcal{P}(\mathcal{A})}\int_{\mathcal{A}}v(a)\left(q(x,a;\pi)-\gamma\log
v(a)\right)\mathrm{d}a.$ (18)

which has a closed form solution with
$v^{*}(a)\propto\exp(\frac{q(x,a,\pi)}{\gamma})$, i.e. $v^{*}$ is the
Boltzmann policy for $q$-functions. This is a “soft” (à la [17]) version of
the $q$-learning mentioned above. (b) Policy gradient. We use function
approximations to $\pi$ by a parametric family $\pi^{\theta}$, with
$\theta\in\Theta\subseteq\mathbb{R}^{L}$. For simplicity, write
$d^{\theta_{0}}_{\mu}$ (resp. $\eta(\theta)$) for $d^{\pi^{\theta_{0}}}_{\mu}$
(resp. $\eta(\pi^{\theta})$). Setting $\hat{\pi}=\pi^{\theta}$ and
$\pi=\pi^{\theta_{0}}$ in (13) and taking derivative with respect to $\theta$
on both sides, we get the following result.

######  Theorem 3 (Policy Gradient).

The policy gradient at $\pi^{\theta_{0}}$ is:

$\nabla_{\theta}\eta(\theta)\mid_{\theta=\theta_{0}}=\frac{1}{\beta}{\mathbb{E}}_{(x,a)}\left[\nabla_{\theta}\log\left(\pi^{\theta}(a\mid
x)\right)\left(q(x,a;\pi^{\theta_{0}})+\gamma
p(x,a,\pi^{\theta_{0}})\right)+\gamma\nabla_{\theta}p(x,a,\pi^{\theta})\right],$
(19)

where the expectation is w.r.t. $(x,a)\sim(\beta
d^{\theta_{0}}_{\mu},\pi^{\theta_{0}})$, meaning $x\sim\beta
d^{\theta_{0}}_{\mu}(\cdot)$ and then $a\sim\pi^{\theta_{0}}(\cdot\mid x)$.

The above formula is indeed the continuous analogue to the well-known PG
formula (without regularization) in the MDP setting, where
$\nabla_{\theta}\eta(\theta)\mid_{\theta=\theta_{0}}=\frac{1}{\beta}{\mathbb{E}}_{(x,a)}\left[\nabla_{\theta}\log\left(\pi^{\theta}(a\mid
x)\right)A(x,a;\pi^{\theta_{0}})\right]$ ([53]), with $A$ denoting the
advantage function. Specifically, as a comparison, the formula in (19)
replaces the visitation frequency by the occupation time, and the advantage
function by the $q$-function, while keeping the same score function
$\nabla_{\theta}\log\left(\pi^{\theta}(a\mid x)\right)$.

###  3.3 Continuous TRPO/PPO

Leveraging the performance-difference formula derived above, we can now move
on to spell out the continuous counterpart of TRPO and PPO originally
developed in [45, 47] for the discrete RL.

Local Approximation Function. Given a feedback policy $\hat{\pi}$, we define
the local approximation function to $\eta(\hat{\pi})$ by

$\displaystyle
L^{\pi}(\hat{\pi})=\eta(\pi)+\int_{\mathbb{R}^{n}}d^{\pi}_{\mu}(x)\left[\int_{\mathcal{A}}\hat{\pi}(a\mid
x)\left(q(x,a;\pi)+p(x,a,\hat{\pi})\right)\mathrm{d}a\right]\mathrm{d}x.$ (20)

Comparing (20) to the formula (13), we see that the difference is to replace
$d_{\mu}^{\hat{\pi}}(s)$ with $d_{\mu}^{\pi}(s)$. Observe that $\text{(i)
}L^{\pi}(\pi)=\eta(\pi)$, $\text{(ii)
}\nabla_{\theta}\left(\eta(\pi^{\theta})\right)\mid_{\theta=\theta_{0}}=\nabla_{\theta}\left(L^{\pi^{\theta_{0}}}(\pi^{\theta})\right)\mid_{\theta=\theta_{0}}$,
i.e. the local approximation function and the true performance objective share
the same value and the same gradient with respect to the policy parameters.
Thus, the local approximation function can be regarded as the first order
approximation to the performance metric. Furthermore, similar to [23, 45], we
can apply simulation methods to evaluate the local approximation function only
using the data generated from the current policy $\pi$:

$\displaystyle
L^{\pi}(\hat{\pi})=\eta(\pi)+\frac{1}{\beta}\underset{(x,a)\sim(\beta
d^{\pi}_{\mu},\pi(\cdot|x))}{\mathbb{E}}\left[\frac{\hat{\pi}(a\mid
x)}{\pi(a\mid x)}\left(q(x,a;\pi)+\gamma p(x,a,\hat{\pi})\right)\right].$ (21)

Next, we provide analysis and bounds on the gap
$\eta(\hat{\pi})-L^{\pi}(\hat{\pi})$, which can then be used to ensure policy
improvement (similar to approaches in [45, 64] for discounted/average reward
MDP). First, we need some technical conditions on the model dynamics.

######  Assumption 1.

Assume the following conditions for the state dynamics hold true:
(i) Global boundedness: There exists $0<\sigma_{0}\leq\bar{\sigma}_{0}$ such
that $\sigma_{0}^{2}\cdot
I\leq\tilde{\sigma}^{2}(x,a)\leq\bar{\sigma}_{0}^{2}\cdot I$ for all $x,a$;
(ii)Uniformly Lipschitz: There exists $C_{\tilde{\sigma}}>0$ such that
$\left\|\tilde{\sigma}(x,\pi)-\tilde{\sigma}(x^{\prime},\pi)\right\|_{F}\leq
C_{\tilde{\sigma}}\left\|x-x^{\prime}\right\|_{2}$ for all $\pi$ and
$x,x^{\prime}$;
(iii) Monotonicity (for drift) or growth condition:
There exists $C_{\tilde{b}}>0$ such that
$(x-x^{\prime})^{\top}\left(\tilde{b}(x,\pi)-\tilde{b}(x^{\prime},\pi)\right)\leq
C_{\tilde{b}}\left\|x-x^{\prime}\right\|_{2}^{2}$ for all $\pi$ and
$x,x^{\prime}$.

The following lemma provides a Wasserstein-2 bound between the discounted
occupation measures $d_{\mu}^{\pi}(\cdot)$ and $d_{\mu}^{\hat{\pi}}(\cdot)$
for two policies $\pi$ and $\pi^{\prime}$.

######  Lemma 4.

Let $\pi,\pi^{\prime}$ be two feedback policies, and suppose the conditions in
Assumption 1 hold. Define
$C_{\tilde{b},\tilde{\sigma}}:=2C_{\tilde{b}}+1+2C_{\tilde{\sigma}}^{2}$ and
$C=\sup_{x,a}|b(x,a)|^{2}+\frac{n\bar{\sigma}_{0}^{2}}{2\sigma_{0}^{2}}$.
(Recall $n$ is the dimension of the state.) Assume further that
$\beta>C_{\tilde{b},\tilde{\sigma}}$ and $C<\infty$. Then there is the bound

$W_{2}\left(\beta d_{\mu}^{\hat{\pi}},\beta
d_{\mu}^{\pi}\right)\leq\frac{C}{\beta C_{\tilde{b},\tilde{\sigma}}(\beta-
C_{\tilde{b},\tilde{\sigma}})}\cdot\max\left(\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|_{1},\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|^{\frac{1}{2}}_{1}\right).$
(22)

The proof is deferred to Appendix B.3. To derive a performance difference
bound, we define the Sobolev semi-norm:
$K:=\|f\|_{\dot{H}^{1}}:=\left(\int_{\mathbb{R}^{n}}|\nabla
f(x)|^{2}\mathrm{d}x\right)^{\frac{1}{2}}$ and its dual norm defined as
$\|\mu\|_{\dot{H}^{-1}}=\sup\left\\{|\langle
g,\mu\rangle|\mid\|g\|_{\dot{H}^{1}}\leq 1\right\\}$. [32, 40] show the
equivalence of this dual norm to the Wasserstein-2 distance. Combining this
fact with Lemma 4 yields the following result.

######  Theorem 5.

Suppose the conditions in Lemma 4 hold, and assume
$d_{\mu}^{\hat{\pi}}(x),d_{\mu}^{\pi}(x)$ are upper bounded by $M$ for all
$x\in\mathbb{R}^{n}$. Define $K:=\|f\|_{\dot{H}^{1}}$ with
$f(x;\pi,\hat{\pi}):=\int_{\mathcal{A}}\hat{\pi}(a\mid
x)\left(q(x,a;\pi)+p(x,a,\hat{\pi})\right)\mathrm{d}a$ and
$C(\mu,\pi,\hat{\pi}):=\frac{\sqrt{M}K}{2\beta^{2}C_{\tilde{b},\tilde{\sigma}}(\beta-
C_{\tilde{b},\tilde{\sigma}})}\left(\sup_{x,a}\|b(x,a)\|^{2}+\frac{n\bar{\sigma}_{0}^{2}}{2\sigma_{0}^{2}}\right)$.
Assuming $K,\,C(\mu,\pi,\hat{\pi})<\infty$, we have
$\eta({\hat{\pi}})\geq\underline{L}^{\pi}(\hat{\pi})$, where

$\underline{L}^{\pi}(\hat{\pi}):=L^{\pi}(\hat{\pi})-C(\mu,\pi,\hat{\pi})\cdot\max\left(\sup_{x}D_{\mathrm{KL}}(\hat{\pi}(\cdot|x)\|\pi(\cdot|x)),\sup_{x}\sqrt{D_{\mathrm{KL}}(\hat{\pi}(\cdot|x)\|\pi(\cdot|x))}\right).$
(23)

The proof is given in Appendix B. By Theorem 5, we can use the minorization-
maximization (MM) algorithm in [19, 23, 26], where
$\underline{L}^{\pi}(\hat{\pi})$ is taken as the surrogate function for
$\eta(\pi)$. Specifically, given the policy $\pi_{k}$, if we can indeed solve
the optimization problem $\max_{\hat{\pi}}\underline{L}^{\pi_{k}}(\hat{\pi})$,
and designate its solution as $\pi_{k+1}$. Then, we have

$\eta(\pi_{k+1})\geq\underline{L}^{\pi_{k}}(\pi_{k+1})\geq\underline{L}^{\pi_{k}}(\pi_{k})=\eta(\pi_{k})$
(24)

i.e., a guaranteed performance improvement. See also [26, Chapter 7] and [27]
for the (global) convergence analysis of the MM algorithm (which exceeds the
scope of this work). However, in general this optimization problem is not easy
to solve directly since $C(\mu,\pi,\hat{\pi})$ is unknown, and we may have to
work with sample based estimates of the approximation functions. In the spirit
of [45, 47], we provide algorithms in the next section that can be practically
implemented by incorporating an adaptive penalty constant $C_{\text{penalty}}$
as an alternative to $C(\mu,\pi,\hat{\pi})$. Consequently, the resulting
algorithms may no longer preserve the increasing performance of $\eta$ at each
iteration, but overall increasing trend will clearly be there (as demonstrated
in Figure 3).

##  4 Algorithms and Experiments

###  4.1 Sample-based Algorithms

Based on the analysis and results developed above, we provide sample-based
estimates of the objective functions that lead to practical algorithms. Here
we highlight several hyper-parameters: the learning rate $\alpha$; the
trajectory truncation parameter (time horizon) $T$ (needs to be sufficiently
large); the total sample size $N$ or the sampling interval $\delta_{t}$, with
$N\cdot\delta_{t}=T$. Also denote $t_{i}:=i\cdot\delta_{t}$, $i=0,\ldots,N-1$,
for the time points that we observe data from the environment. Continuous
Policy Gradient (CPG). To estimate the policy gradient (19) from data, we
first sample an independent exponential variable $\tau\sim\exp(\beta)$ to get
$(X^{\pi}_{\tau},a^{\pi}_{\tau})\sim(d^{\theta_{0}}_{\mu},\pi^{\theta_{0}}(\cdot|x))$.
If there is a $q$-function oracle, then we can obtain an unbiased estimate of
the policy gradient (of which the convergence analysis follows [63]). Lack of
such an oracle, we employ the generalized advantage estimation (GAE) technique
[46] to get $q(X_{t},a_{t})\approx\left(Q_{\Delta
t}(X_{t},a_{t};\pi)-V(X_{t};\pi)\right)/\delta_{t}\approx\left(r_{t}\delta_{t}+e^{-\beta\delta_{t}}V(X_{t+\delta_{t}})-V(X_{t})\right)/\delta_{t}$.
This yields the policy gradient Algorithm 1 below.

Algorithm 1 CPG: Policy Gradient with $\exp(\beta)$ random rollout

Input: Policy parameters $\theta_{0}$, critic net parameters $\phi_{0}$,
batch/sample size $J$

1:for $k=0,1,2,\cdots$ until $\theta_{k}$ converges do

2: Collect a truncated trajectory
$\left\\{X_{t_{i}},a_{t_{i}},r_{t_{i}},p_{t_{i}}\right\\},i=1,\ldots,N$ from
the environment using $\pi_{\theta_{k}}$.

3: for $i=0,\ldots,N-1$ do: Update the critic parameters as in (8)

4: for $j=1,,\ldots,J$ do: Draw i.i.d. $\tau_{j}$ from $\exp(\beta)$, round
$\tau_{j}$ to the largest multiple of $\delta_{t}$ no larger than it, and
compute the GAE estimator of $q(X_{\tau_{j}},a_{\tau_{j}})$
$\tilde{q}(X_{\tau_{j}},a_{\tau_{j}}):=\left(r_{\tau_{j}}\delta_{t}+e^{-\beta\delta_{t}}V(X_{\tau_{j}+\delta_{t}})-V(X_{\tau_{j}})\right)/\delta_{t}$
(25)

5: Get an estimator of $\nabla_{j}\eta(\theta_{k})$ as
$\frac{1}{\beta}\left[\nabla_{\theta}\log\left(\pi^{\theta_{k}}(a_{\tau_{j}}\mid
X_{\tau_{j}})\right)\left(\tilde{q}(X_{\tau_{j}},a_{\tau_{j}})+\gamma
p(X_{\tau_{j}},a_{\tau_{j}},\pi^{\theta_{k}})\right)+\gamma\nabla_{\theta}p(X_{\tau_{j}},a_{\tau_{j}},\pi^{\theta_{k}})\right]$
(26)

6: Let
$\tilde{\nabla}\eta(\theta_{k})=\frac{1}{J}\sum_{j=1}^{J}\nabla_{j}\eta(\theta_{k})$
and perform PG update:
$\theta_{k+1}=\theta_{k}-\alpha\tilde{\nabla}\eta(\theta_{k})$

Continuous PPO (CPPO). We now present Algorithm 2, a continuous version of the
PPO, also as an approximation to the MM algorithm in Section 3.3. To do so, we
need more hyper-parameters: the tolerance level $\epsilon$, and the KL-
divergence radius $\delta$. Moreover, we set
$\bar{D}_{\mathrm{KL}}(\theta\|\theta_{k}):=\mathbb{E}_{x\sim
d^{\theta_{k}}_{\mu}}\sqrt{D_{\mathrm{KL}}(\pi_{\theta}(\cdot|x)\|\pi_{\theta_{k}}(\cdot|x))}$.
(Empirically we find that taking average, instead of supremum, over $x$ does
not affect the performance while reducing computational burden, similar to
what’s observed in the discrete-time TRPO in [45].)

Algorithm 2 CPPO: PPO with adaptive penalty constant

Input: Policy parameters $\theta_{0}$, critic net parameters $\phi_{0}$

1:for $k=0,1,2,\cdots$ until $\theta_{k}$ converge do

2: Follow the same as Steps 2-6 in Algorithm 1.

3: Compute policy update (by taking a fixed $s$ steps of gradient descent)
$\theta_{k+1}=\arg\max_{\theta}L^{\theta_{k}}(\theta)-C^{k}_{\text{penalty}}\bar{D}_{\mathrm{KL}}\left(\theta\|\theta_{k}\right)$
(27)

4: if
$\bar{D}_{\mathrm{KL}}\left(\theta_{k+1}\|\theta_{k}\right)\geq(1+\epsilon)\delta$
then $\quad C^{k+1}_{\text{penalty}}=2C^{k}_{\text{penalty}}$

5: else if
$\bar{D}_{\mathrm{KL}}\left(\theta_{k+1}\|\theta_{k}\right)\leq\delta/(1+\epsilon)$
then $\quad C^{k+1}_{\text{penalty}}=C^{k}_{\text{penalty}}/2$

Algorithm 2 is essentially a continuous analogue of the TRPO/PPO methods. Note
that in the penalty term we use the mean square-root of the KL-divergence,
since we choose the radius $\delta<1$; hence, the square-root distance will
dominate in the bound in (23). Interestingly, throughout our primary
experiments, using the square-root KL-divergence outperforms (using the KL-
divergence itself). Refer to Appendix C,D for more details.

###  4.2 Experiments

LQ stochastic control. Consider an environment driven by an SDE with linear
state dynamics and quadratic rewards, with $b(x,a)=Ax+Ba$,
$\sigma(x,a)=Cx+Da$, where $A,B,C,D\in\mathbb{R}$,
$p(x,a,\pi)=-\log(\pi(a|x))$, and
$r(x,a)=-\left(\frac{M}{2}x^{2}+Rxa+\frac{N}{2}a^{2}+Px+Qa\right),$ where
$M\geq 0,N>0,R,P,Q\in\mathbb{R}$. Linear-quadratic (LQ) control problems play
an important role in the control literature, not only because it has elegant
and simple solutions but also because more complex, nonlinear problems can be
approximated by LQ problems. In general, we do not know the model parameters
(e.g., $A,B,\ldots$), and the idea is to use continuous RL methods to find the
optimal policy. Here we adopt a Gaussian exploration: $\pi_{\theta}(\cdot\mid
x)=\mathcal{N}(\theta_{1}x+\theta_{2},\exp(\theta_{3}))$, and parameterize the
value function as $V_{\phi}(x)=\frac{1}{2}\phi_{2}x^{2}+\phi_{1}x+\phi_{0}$.
(In fact, as shown in [59, Theorem 4], the optimal exploration and value
functions are of this form, and constants such as $\theta$ and $\phi$ can be
computed explicitly given the model dynamics.) We randomly choose a set of
initial constants, and compute the optimal $\theta$ and $\phi$ with respect to
these parameters; refer to Appendix D.1 for more details. Figure 2 and Figure
2 show the convergence of the algorithms.

Figure 1: Convergence of $\theta$ in $l_{2}$ distance

Figure 2: Convergence of $\pi_{\theta}$ in KL-divegence

In Figure 2, we compute the $l_{2}$ distance between the current policy
parameters and the optimal ones, i.e. $\|\theta_{k}-\theta^{*}\|_{2}$, which
tracks the convergence of the policy parameters. In Figure 2, we plot the
sample estimated KL divergence between the current policy $\pi_{k}$ (specified
by $\theta_{k}$) and $\pi^{*}$ (specified by $\theta^{*}$), i.e.
$\mathbb{E}_{x\sim
d^{\theta_{k}}_{\mu}}D_{\mathrm{KL}}(\pi_{\theta}(\cdot|x)\|\pi_{\theta_{k}}(\cdot|x))$.
The reason to consider the KL-divergence between $\pi_{k}$ and $\pi^{*}$ is
that minimizing the KL-divergence to the optimal solution is (in some sense)
equivalent to minimizing the distance between the current policy objective and
the optimal objective (see Appendix D.1)). The experiments illustrate that our
proposed algorithms do converge to the (local) optimum. 2-dimensional optimal
pair trading. Consider the 2-dimensional optimal pair trading problem in [36].
The state space is $X=(S,W)\in\mathbb{R}^{2}$ with $X(0)=(s_{0},w_{0})$, where
$S$ represents the spread between two stocks, and $W$ denotes the
corresponding wealth process. The trader intends to maximize the total
discounted reward, with the reward function $r(X,a)=\log(1+W)$. The state
dynamics are:

$\mathrm{d}S_{t}=k(\theta-
S_{t})\mathrm{d}t+\eta\mathrm{d}B_{t},\quad\mathrm{d}W_{t}=a_{t}W_{t}(k(\theta-
S_{t})+\frac{1}{2}\eta^{2}+\rho\sigma\eta+r_{f})\mathrm{d}t+\eta
W_{t}\mathrm{d}B_{t},$ (28)

We set $p(x,a,\pi)\equiv 0$, and add a constraint on the action:
$a_{t}\in[-\ell,\ell]$. (The action $a_{t}$ is the position taken on the first
stock, which can be long/positive or short/negative.) Since the action space
is bounded and continuous, we consider a beta distribution for policy
parameterization: $\pi_{\theta}(a\mid
X):=f\left(\frac{a+\ell}{2\ell},\alpha_{\theta}(X),\beta_{\theta}(X)\right)$
with
$f(x,\alpha,\beta):=\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}\left(1-x^{\beta-1}\right)$.
For $\alpha_{\theta}$ and $\beta_{\theta}$, we use a 3-layer neural network
(NN) parameterized by $\theta$ for function approximation; and use another
3-layer NN for value function approximation. (More details are provided in
Appendix D.2.)

Figure 3: Performance of both algorithms to the task

Figure 3 shows that both algorithms, CPG and CPPO, converge to a local optimum
(different between the two), and with an overall increasing trend over
iterations. (Averaging is taken over $100$ Monte Carlo estimates for each
policy evaluation.)

##  5 Conclusion and Further Works

We have developed in this paper the basic theoretical framework for policy
optimization in continuous RL, and illustrated its potential applications
using numerical experiments. For further research, two topics are high on our
agenda. First, we plan to study the convergence (rate) of the continuous
policy gradient and TRPO/PPO, vis-a-vis the error due to the time increment
$\delta_{t}$. Our conjecture is that it is likely to be polynomial-bounded
under mild assumptions, similar to the analysis in [20, 16]), thus extending
beyond the condition required by [63] and [60, 31]. Second, for the bounds on
the statistical distance and the performance difference, we want to further
develop a consistent bound like the one in [11] (for the discrete setting),
i.e., one that remains meaningful when the discount factor $\beta\rightarrow
0$.

## Ackbowledgements

W. Tang gratefully acknowledges financial support through NSF grants
DMS-2113779 and DMS-2206038, and through a start-up grant at Columbia
University. The works of Hanyang Zhao and David Yao are part of a Columbia-
CityU/HK collaborative project that is supported by InnotHK Initiative, The
Government of the HKSAR and the AIFT Lab.

## References

  * Achiam et al. [2017] Achiam, J., Held, D., Tamar, A., and Abbeel, P.  Constrained policy optimization.  In _International conference on machine learning_ , pages 22–31. PMLR, 2017.
  * Agazzi and Lu [2020] Agazzi, A. and Lu, J.  Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime.  _arXiv:2010.11858_ , 2020.
  * Baird [1994] Baird, L. C.  Reinforcement learning in continuous time: Advantage updating.  In _Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94)_ , volume 4, pages 2448–2453. IEEE, 1994.
  * Baird III [1993] Baird III, L. C.  Advantage updating.  Technical report, WRIGHT LAB WRIGHT-PATTERSON AFB OH, 1993.
  * Basei et al. [2022] Basei, M., Guo, X., Hu, A., and Zhang, Y.  Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon.  _Journal of Machine Learning Research_ , 23(178):1–34, 2022.
  * Bradtke [1992] Bradtke, S.  Reinforcement learning applied to linear quadratic regulation.  _Advances in neural information processing systems_ , 5, 1992.
  * Brunick and Shreve [2013] Brunick, G. and Shreve, S.  Mimicking an itô process by a solution of a stochastic differential equation.  _Ann. Appl. Probab._ , 23(4), 2013.
  * Bubeck et al. [2023] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Buehler et al. [2019] Buehler, H., Gonon, L., Teichmann, J., and Wood, B.  Deep hedging.  _Quantitative Finance_ , 19(8):1271–1291, 2019\.
  * Christiano et al. [2017] Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Dai and Gluzman [2021] Dai, J. G. and Gluzman, M.  Refined policy improvement bounds for mdps.  _arXiv preprint arXiv:2107.08068_ , 2021.
  * Dai and Gluzman [2022] Dai, J. G. and Gluzman, M.  Queueing network controls via deep reinforcement learning.  _Stochastic Systems_ , 12(1):30–67, 2022.
  * Doya [2000] Doya, K.  Reinforcement learning in continuous time and space.  _Neural computation_ , 12(1):219–245, 2000.
  * Du et al. [2020] Du, J., Futoma, J., and Doshi-Velez, F.  Model-based reinforcement learning for semi-markov decision processes with neural odes.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 19805–19816. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/e562cd9c0768d5464b64cf61da7fc6bb-Paper.pdf.
  * Fleming and Soner [2006] Fleming, W. H. and Soner, H. M.  _Controlled Markov processes and viscosity solutions_ , volume 25.  Springer Science & Business Media, 2006.
  * Guo et al. [2021] Guo, X., Hu, A., and Zhang, Y.  Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls.  _arXiv preprint arXiv:2104.09311_ , 2021.
  * Haarnoja et al. [2017] Haarnoja, T., Tang, H., Abbeel, P., and Levine, S.  Reinforcement learning with deep energy-based policies.  In _International conference on machine learning_ , pages 1352–1361. PMLR, 2017.
  * Huang et al. [2022] Huang, Y., Jia, Y., and Zhou, X.  Achieving mean–variance efficiency by continuous-time reinforcement learning.  In _Proceedings of the Third ACM International Conference on AI in Finance_ , pages 377–385, 2022.
  * Hunter and Lange [2004] Hunter, D. R. and Lange, K.  A tutorial on MM algorithms.  _The American Statistician_ , 58(1):30–37, 2004\.
  * Jia and Zhou [2022a] Jia, Y. and Zhou, X. Y.  Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach.  _Journal of Machine Learning Research_ , 23(154):1–55, 2022a.
  * Jia and Zhou [2022b] Jia, Y. and Zhou, X. Y.  Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms.  _Journal of Machine Learning Research_ , 23(154):1–55, 2022b.
  * Jia and Zhou [2022c] Jia, Y. and Zhou, X. Y.  q-learning in continuous time.  _arXiv preprint arXiv:2207.00713_ , 2022c.
  * Kakade and Langford [2002] Kakade, S. and Langford, J.  Approximately optimal approximate reinforcement learning.  In _Proceedings of the Nineteenth International Conference on Machine Learning_ , pages 267–274, 2002.
  * Karatzas and Shreve [1991] Karatzas, I. and Shreve, S. E.  _Brownian motion and stochastic calculus_ , volume 113 of _Graduate Texts in Mathematics_.  Springer-Verlag, New York, second edition, 1991.
  * Kim et al. [2021] Kim, J., Shin, J., and Yang, I.  Hamilton-jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls.  _The Journal of Machine Learning Research_ , 22(1):9363–9396, 2021.
  * Lange [2016] Lange, K.  _MM optimization algorithms_.  SIAM, Philadelphia, PA, 2016.
  * Lange et al. [2021] Lange, K., Won, J.-H., Landeros, A., and Zhou, H.  Nonconvex optimization via MM algorithms: Convergence theory.  _arXiv:2106.02805_ , 2021.
  * Leahy et al. [2022] Leahy, J.-M., Kerimkulov, B., Siska, D., and Szpruch, L.  Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime.  In _International Conference on Machine Learning_ , pages 12222–12252, 2022.
  * Lee and Sutton [2021] Lee, J. and Sutton, R. S.  Policy iterations for reinforcement learning problems in continuous time and space — fundamental theory and methods.  _Automatica_ , 126:109421, 2021.  ISSN 0005-1098.  doi: https://doi.org/10.1016/j.automatica.2020.109421.  URL https://www.sciencedirect.com/science/article/pii/S0005109820306233.
  * Ling et al. [2017] Ling, Y., Hasan, S. A., Datla, V., Qadir, A., Lee, K., Liu, J., and Farri, O.  Diagnostic inferencing via improving clinical concept extraction with deep reinforcement learning: A preliminary study.  In _Machine Learning for Healthcare Conference_ , pages 271–285. PMLR, 2017.
  * Liu et al. [2023] Liu, B., Cai, Q., Yang, Z., and Wang, Z.  Neural proximal/trust region policy optimization attains globally optimal policy, 2023.
  * Loeper [2006] Loeper, G.  Uniqueness of the solution to the vlasov–poisson system with bounded density.  _Journal de mathématiques pures et appliquées_ , 86(1):68–79, 2006.
  * Madeka et al. [2022] Madeka, D., Torkkola, K., Eisenach, C., Foster, D., and Luo, A.  Deep inventory management.  _arXiv preprint arXiv:2210.03137_ , 2022.
  * Mnih et al. [2013] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. [2016] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Mudchanatongsuk et al. [2008] Mudchanatongsuk, S., Primbs, J. A., and Wong, W.  Optimal pairs trading: A stochastic control approach.  In _2008 American Control Conference_ , pages 1035–1039, 2008.  doi: 10.1109/ACC.2008.4586628.
  * Munos [2006] Munos, R.  Policy gradient in continuous time.  _Journal of Machine Learning Research_ , 7:771–791, 2006\.
  * Munos and Bourgine [1997] Munos, R. and Bourgine, P.  Reinforcement learning for continuous stochastic control problems.  In Jordan, M., Kearns, M., and Solla, S., editors, _Advances in Neural Information Processing Systems_ , volume 10. MIT Press, 1997.  URL https://proceedings.neurips.cc/paper_files/paper/1997/file/186a157b2992e7daed3677ce8e9fe40f-Paper.pdf.
  * Øksendal [2003] Øksendal, B.  _Stochastic differential equations_.  Springer, 2003.
  * Peyre [2018] Peyre, R.  Comparison between w2 distance and h- 1 norm, and localization of wasserstein distance.  _ESAIM: Control, Optimisation and Calculus of Variations_ , 24(4):1489–1501, 2018.
  * Pitman and Yor [2003] Pitman, J. and Yor, M.  Hitting, occupation and inverse local times of one-dimensional diffusions: martingale and excursion approaches.  _Bernoulli_ , 9(1):1–24, 2003.
  * Puterman [2014] Puterman, M. L.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Revuz and Yor [2013] Revuz, D. and Yor, M.  _Continuous martingales and Brownian motion_ , volume 293.  Springer Science & Business Media, 2013.
  * Sallab et al. [2017] Sallab, A. E., Abdou, M., Perot, E., and Yogamani, S.  Deep reinforcement learning framework for autonomous driving.  _arXiv preprint arXiv:1704.02532_ , 2017.
  * Schulman et al. [2015a] Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR, 2015a.
  * Schulman et al. [2015b] Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015b.
  * Schulman et al. [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Silver et al. [2016] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Silver et al. [2017] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Stroock and Varadhan [1979] Stroock, D. W. and Varadhan, S. R. S.  _Multidimensional diffusion processes_ , volume 233 of _Grundlehren der Mathematischen Wissenschaften_.  Springer-Verlag, Berlin-New York, 1979.
  * Sutton [1988] Sutton, R. S.  Learning to predict by the methods of temporal differences.  _Machine learning_ , 3:9–44, 1988.
  * Sutton and Barto [2018] Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. [1999] Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y.  Policy gradient methods for reinforcement learning with function approximation.  _Advances in neural information processing systems_ , 12, 1999.
  * Szpruch et al. [2021] Szpruch, L., Treetanthiploet, T., and Zhang, Y.  Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models.  _arXiv:2112.10264_ , 2021.
  * Tallec et al. [2019] Tallec, C., Blier, L., and Ollivier, Y.  Making deep q-learning methods robust to time discretization.  In _International Conference on Machine Learning_ , pages 6096–6104. PMLR, 2019.
  * Tang et al. [2022] Tang, W., Zhang, Y. P., and Zhou, X. Y.  Exploratory hjb equations and their convergence.  _SIAM Journal on Control and Optimization_ , 60(6):3191–3216, 2022.
  * Vamvoudakis [2017] Vamvoudakis, K. G.  Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach.  _Systems & Control Letters_, 100:14–20, 2017.
  * Wang and Zhou [2020] Wang, H. and Zhou, X. Y.  Continuous-time mean–variance portfolio selection: A reinforcement learning framework.  _Mathematical Finance_ , 30(4):1273–1308, 2020\.
  * Wang et al. [2020] Wang, H., Zariphopoulou, T., and Zhou, X. Y.  Reinforcement learning in continuous time and space: A stochastic control approach.  _J. Mach. Learn. Res._ , 21(198):1–34, 2020.
  * Wang et al. [2019] Wang, L., Cai, Q., Yang, Z., and Wang, Z.  Neural policy gradient methods: Global optimality and rates of convergence, 2019.
  * Watkins and Dayan [1992] Watkins, C. J. and Dayan, P.  Q-learning.  _Machine learning_ , 8:279–292, 1992.
  * Yong and Zhou [1999] Yong, J. and Zhou, X. Y.  _Stochastic controls: Hamiltonian systems and HJB equations_ , volume 43.  Springer Science & Business Media, 1999.
  * Zhang et al. [2020] Zhang, K., Koppel, A., Zhu, H., and Basar, T.  Global convergence of policy gradient methods to (almost) locally optimal policies.  _SIAM Journal on Control and Optimization_ , 58(6):3586–3612, 2020.
  * Zhang and Ross [2021] Zhang, Y. and Ross, K. W.  On-policy deep reinforcement learning for the average-reward criterion.  In _International Conference on Machine Learning_ , pages 12535–12545. PMLR, 2021.

##  Appendix A  Continuous RL: Formulation and Well-Posedness

###  A.1 Exploratory Stochastic-Control

For $n,m$ positive integers, let
$b:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}^{n}$ and
$\sigma:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}^{n\times m}$ be given
functions, where $\mathcal{A}$ is a compact action space. A classical
stochastic control problem [15, 62] is to control the state (or feature)
dynamics governed by an Itô process, defined on a filtered probability space
$\left(\Omega,\mathcal{F},\mathbb{P};\left\\{\mathcal{F}_{s}^{B}\right\\}_{s\geq
0}\right)$, along with an $\\{\mathcal{F}^{B}_{s}\\}$-Brownian motion
$B=\left\\{B_{s},s\geq 0\right\\}$:

$\mathrm{d}X_{s}^{a}=b\left(X_{s}^{a},a_{s}\right)\mathrm{d}s+\sigma\left(X_{s}^{a},a_{s}\right)\mathrm{d}B_{s},\,s\geq
t,\quad X_{t}=x,$ (29)

where $a_{s}$ is the agent’s action (control) at time $s$. The goal of the
stochastic control (discounted objective over an infinite time horizon) is for
any time-state pair $(t,x)$ in (29), to find the optimal
$\left\\{\mathcal{F}_{s}^{B}\right\\}_{s\geq 0}$-progressively measurable
sequence of actions $a=\\{a_{s},s\geq t\\}$ (called the optimal policy) that
maximizes the expected total $\beta$-discounted reward:

$\mathbb{E}\left[\int_{t}^{+\infty}e^{-\beta(s-t)}r\left(X_{s}^{a},a_{s}\right)\mathrm{d}s\mid
X_{t}^{a}=x\right],$ (30)

where $r:\mathbb{R}^{n}\times\mathcal{A}\mapsto\mathbb{R}$ is the running
reward of the current state and action $(X^{a}_{s},a_{s})$, and $\beta>0$ is a
discount factor that measures the time-depreciation of the objective value (or
the impatience level of the agent). Note that the state process
$X^{a}=\left\\{X_{s}^{a},s\geq t\right\\}$ depends on the starting (initial)
time-state pair $(t,x)$. For ease of notation, we denote by $X^{a}$ instead of
$X^{t,x,a}=\left\\{X_{s}^{t,x,a},s\geq t\right\\}$ the solution to the SDE in
(29) when there is no ambiguity.

Listed below are the standard assumptions to ensure the well-posedness of the
stochastic control problem in (29)-(30).

######  Assumption 2.

The following conditions are assumed throughout:
(i) $b,\sigma,r$ are all continuous functions in their respective arguments;
(ii) b, $\sigma$ are uniformly Lipschitz continuous in $x$, i.e., there exists
a constant $C>0$ such that for $\varphi\in\\{b,\sigma\\}$,

$\left\|\varphi(x,a)-\varphi\left(x^{\prime},a\right)\right\|_{2}\leq
C\left\|x-x^{\prime}\right\|_{2},\quad\mbox{for all
}a\in\mathcal{A},\,\,x,x^{\prime}\in\mathbb{R}^{n};$ (31)

(iii) $b,\sigma$ have linear growth in $x$ and $a$, i.e., there exists a
constant $C>0$ such that for $\varphi\in\\{b,\sigma\\}$,

$\|\varphi(x,a)\|_{2}\leq C(1+\|x\|_{2}+\left\|a\right\|_{2}),\quad\mbox{for
all }(x,a)\in\mathbb{R}^{n}\times\mathcal{A};$ (32)

(iv) $r$ has polynomial growth in $x$ and $a$, i.e., there exists a constant
$C>0$ and $\mu\geq 1$ such that

$|r(x,a)|\leq C\left(1+\|x\|_{2}^{\mu}+\|a\|_{2}^{\mu}\right)\quad\mbox{for
all }(x,a)\in\mathbb{R}^{n}\times\mathcal{A}.$ (33)

The key idea underlying exploratory stochastic control is to use a randomized
policy (or relaxed control), i.e., apply a probability distribution to the
admissible action space. To do so, let’s assume the probability space is rich
enough to support a uniform random variable $Z$ that is independent of the
Brownian motion $B=\left\\{B_{t}\right\\}$. We then expand the original
filtered probability space to
$\left(\Omega,\mathcal{F},\mathbb{P};\left\\{\mathcal{F}_{s}\right\\}_{s\geq
0}\right)$, where $\mathcal{F}_{s}=\mathcal{F}_{s}^{B}\vee\sigma(Z)$ (i.e.,
augment $\mathcal{F}_{s}^{B}$ with the sigma field generated by $Z$). Let
$\pi:\mathbb{R}^{n}\ni x\mapsto\pi(\cdot\mid x)\in\mathcal{P}(\mathcal{A})$ be
a stationary feedback policy given the state at $x$, where
$\mathcal{P}(\mathcal{A})$ is a suitable collection of probability
distributions (with density functions). At each time $s$, an action $a_{s}$ is
generated from the distribution $\pi\left(\cdot\mid X^{a}_{s}\right)$, i.e.
the policy only depends on the current state. In other words, we only consider
stationary, or time-independent feedback control policies for the stochastic
control problem (29)-(30).

Given a stationary policy $\pi\in\mathcal{P}(\mathcal{A})$, an initial state
$x$, and an $\left\\{\mathcal{F}_{s}\right\\}$-progressively measurable action
process $a^{\pi}=\left\\{a_{s}^{\pi},s\geq 0\right\\}$ generated from $\pi$,
the state process $X^{\pi}=\left\\{X_{s}^{\pi},s\geq 0\right\\}$ follows:

$\mathrm{d}X_{s}^{\pi}=b\left(X_{s}^{\pi},a_{s}^{\pi}\right)\mathrm{d}s+\sigma\left(X_{s}^{\pi},a_{s}^{\pi}\right)\mathrm{d}B_{s},\,s\geq
t,\quad X_{0}^{\pi}=x,$ (34)

defined on
$\left(\Omega,\mathcal{F},\mathbb{P};\left\\{\mathcal{F}_{s}\right\\}_{s\geq
0}\right)$. It is easy to see that the dynamics in (34) define a time-
homogeneous Markov process, such that for each $t\geq 0$ and $x$:

$\left(X_{s}^{\pi}\mid X_{0}^{\pi}=x\right)\stackrel{{\scriptstyle
d}}{{=}}\left(X_{s+t}^{\pi}\mid X_{t}^{\pi}=x\right),\,s\geq 0.$

Consequently, the objective in (30) is independent of time $t$, and is equal
to:

$\mathbb{E}\left[\int_{0}^{+\infty}e^{-\beta
s}r\left(X_{s}^{\pi},a^{\pi}_{s}\right)\mathrm{d}s\mid X_{0}^{\pi}=x\right].$
(35)

Furthermore, following [59], we can add a regularizer to the objective
function to encourage exploration (represented by the randomized policy),
leading to

$\displaystyle V(t,x;\pi):=$
$\displaystyle\mathbb{E}\left[\int_{t}^{\infty}e^{-\beta(s-t)}\left[r\left(X_{s}^{\pi},a_{s}^{\pi}\right)+\gamma
p\left(X_{s}^{\pi},a_{s}^{\pi},\pi\left(\cdot\mid
X_{s}^{\pi}\right),\right)\right]\mathrm{d}s\mid X_{t}^{\pi}=x\right],$ (36)

where
$p:\mathbb{R}^{n}\times\mathcal{A}\times\mathcal{P}(\mathcal{A})\mapsto\mathbb{R}$
is the regularizer, and $\gamma\geq 0$ is a weight parameter on exploration
(also known as the “temperature" parameter). For instance, in [59], $p$ is
taken as the differential entropy,

$p(x,a,\pi(\cdot)):=-\log\pi(a),$

and hence, the “entropy” regularizer. The same argument as before justifies
that $V(t,x;\pi)$ is independent of time $t$. That is, for all $t\geq 0$,

$\displaystyle V(t,x;\pi)\equiv
V(x;\pi):=\mathbb{E}^{\mathbb{P}}\left[\int_{0}^{\infty}e^{-\beta
s}\left[r\left(X_{s}^{\pi},a_{s}^{\pi}\right)+\gamma
p\left(X_{s}^{\pi},a_{s}^{\pi},\pi\left(\cdot\mid
X_{s}^{\pi}\right)\right)\right]\mathrm{d}s\mid X_{0}^{\pi}=x\right];$ (37)

which is the state-value function under the policy $\pi$, $V(x;\pi)$, in (4),
and which, in turn, leads to the performance function $\eta(\pi)$ in (6).
Moreover, recall the main task of the continuous RL is to find (or
approximate) $\eta^{*}=\max_{\pi}\,\,\eta(\pi),$ where $\max$ is over all
admissible policies.

###  A.2 Controlled SDE and the HJ Equation

Note that the exploratory state dynamics in (34) is governed by a general Itô
process. It is sometimes more convenient to consider an equivalent SDE
representation— in the sense that its (weak) solution has the same
distribution as the Itô process in (34) at each fixed time $t$. It is known
([59]) that when $n=m=1$, the marginal distribution of
$\left\\{X_{s}^{\pi},s\geq 0\right\\}$ agrees with that of the solution to the
SDE, denoted by $\\{\tilde{X}_{s},s\geq 0\\}$:

$\mathrm{d}\tilde{X}_{s}=\tilde{b}\left(\tilde{X}_{s},\pi\left(\cdot\mid\tilde{X}_{s}\right)\right)\mathrm{d}s+\tilde{\sigma}\left(\tilde{X}_{s},\pi\left(\cdot\mid\tilde{X}_{s}\right)\right)\mathrm{d}\tilde{B}_{s},\quad\tilde{X}_{0}=x,$

where $\tilde{b}(x,\pi(\cdot))=\int_{\mathcal{A}}b(x,a)\pi(a)\mathrm{d}a$ and
$\tilde{\sigma}(x,\pi(\cdot))=\sqrt{\int_{\mathcal{A}}\sigma^{2}(x,a)\pi(a)\mathrm{d}a}$.
This result is easily extended to arbitrary $n,m$, thanks to [7, Corollary
3.7], with the precise statement presented below (assuming $n=m$ for ease of
exposition).

######  Theorem 6.

Assume that for a policy $\pi$ and for every $x$,

$\int_{\mathcal{A}}\sigma^{2}(x,a)\pi(a)\mathrm{d}a\in\mathbb{R}^{n\times n},$

is positive definite. Then there exists a filtered probability space
$\left(\tilde{\Omega},\tilde{\mathcal{F}},\left\\{\tilde{\mathcal{F}}_{t}\right\\}_{t\geq
0},\tilde{\mathbb{P}}\right)$ that supports a continuous
$\mathbb{R}^{n}$-valued adapted process $\tilde{X}$ and an $n$-dimensional
Brownian motion $\tilde{B}$ satisfying

$\mathrm{d}\tilde{X}_{s}=\tilde{b}\left(\tilde{X}_{s},\pi\left(\cdot\mid\tilde{X}_{s}\right)\right)\mathrm{d}s+\tilde{\sigma}\left(\tilde{X}_{s},\pi\left(\cdot\mid\tilde{X}_{s}\right)\right)\mathrm{d}\tilde{B}_{s},\quad\tilde{X}_{0}=x,$
(38)

where

$\tilde{b}(x,\pi(\cdot))=\int_{\mathcal{A}}b(x,a)\pi(a)\mathrm{d}a,\quad\tilde{\sigma}(x,\pi(\cdot))=\left(\int_{\mathcal{A}}\sigma^{2}(x,a)\pi(a)\mathrm{d}a\right)^{\frac{1}{2}}.$

For each $s\geq 0$, the distribution of $\tilde{X}_{s}$ under
$\tilde{\mathbb{P}}$ agrees with that of $X^{\pi}_{s}$ under $\mathbb{P}$
defined in (34).

As a consequence, the state value function in (37) is identical to

$\displaystyle V(x;\pi)=\mathbb{E}\left[\int_{0}^{\infty}e^{-\beta
s}\int_{\mathcal{A}}\left[r(\tilde{X}_{s},a)+\gamma
p\left(\tilde{X}_{s},a,\pi(\cdot\mid\tilde{X}_{s})\right)\right]\pi(a\mid\tilde{X}_{s})\mathrm{d}a\mathrm{~{}d}s\mid\tilde{X}_{0}=x\right].$

Also define

$\tilde{r}(x,\pi)=\int_{\mathcal{A}}r(x,a)\pi(a|s)\mathrm{d}a,\quad\tilde{p}(x,\pi)=\int_{\mathcal{A}}p(x,a,\pi)\pi(a|x)\mathrm{d}a,$

so we can simplify the value function to

$\displaystyle V(x;\pi)=\mathbb{E}\left[\int_{0}^{\infty}e^{-\beta
s}\left[\tilde{r}(\tilde{X}_{s},\pi)+\gamma\tilde{p}\left(\tilde{X}_{s}^{\pi},\pi(\cdot\mid\tilde{X}_{s})\right)\right]\mathrm{~{}d}s\mid\tilde{X}_{0}=x\right].$
(39)

Following the principle of optimality, $V$ then satisfies the HJ equation:

$\beta V(x;\pi)-\tilde{b}(x,\pi)\cdot\nabla
V(x;\pi)-\frac{1}{2}\tilde{\sigma}^{2}(x,\pi)\circ\nabla^{2}V(x;\pi)-\tilde{r}(x,\pi)-\gamma\tilde{p}(x,\pi)=0.$
(40)

To guarantee that the HJ equation in (40) characterizes the state-value
function in (39), we need

######  Assumption 3.

Assume the following conditions hold:
(i) $b,\sigma,r,p$ are all continuous functions in their respective arguments.
(ii) $b,r,p$ are uniformly Lipschitz continuous in $x$, i.e., there exists a
constant $C>0$ such that for $\varphi\in\\{b,r\\}$,

$\left\|\varphi(x,a)-\varphi\left(x^{\prime},a\right)\right\|_{2}\leq
C\left\|x-x^{\prime}\right\|_{2},\quad\mbox{for all
}a\in\mathcal{A},\,\,x,x^{\prime}\in\mathbb{R}^{n},$

and

$|p(x,a,\pi)-p(x^{\prime},a,\pi)|\leq
C\left\|x-x^{\prime}\right\|_{2},\quad\mbox{for all
}a\in\mathcal{A},\,\pi\in\mathcal{P}(\mathcal{A}),\,x,x^{\prime}\in\mathbb{R}^{n}.$

(iii) $\tilde{\sigma}$ is globally bounded, i.e., there exist
$0<\sigma_{0}<\bar{\sigma}_{0}$ such that

$\sigma_{0}^{2}\cdot I\leq\tilde{\sigma}^{2}(x,a)\leq\bar{\sigma}_{0}^{2}\cdot
I,\quad\mbox{for all }a\in\mathcal{A},\,\,x\in\mathbb{R}^{n}.$

(iv) the SDE (38) has a weak solution which is unique in distribution.
(v) $\pi(a|x)$ is measurable in $(x,a)$ and is uniformly Lipschitz continuous
in $x$, i.e., there exists a constant $C>0$ such that

$\int_{\mathcal{A}}|\pi(a|x)-\pi(a|x^{\prime})|\,da\leq
C\|x-x^{\prime}\|_{2},\quad\mbox{for all }x,x^{\prime}\in\mathbb{R}^{n}.$

######  Theorem 7.

Under Assumption 3, the state-value function in (39) is the unique
(subquadratic) viscosity solution to the HJ equation in (40).

###### Proof.

By [56, Section 3.1], the HJ equation in (40) has a unique (subquadratic)
viscosity solution under the conditions (i)-(iii). Further by [21, Lemma 2],
the viscosity solution is the state-value function. ∎

##  Appendix B Proofs of Main Results (in §3)

###  B.1 Proof of Theorem 2

Recall in the proof sketch of the Theorem in §3, we have defined the operator
$\mathcal{L}^{\pi}:C^{2}(\mathbb{R}^{n})\mapsto C(\mathbb{R}^{n})$ as

$\displaystyle\left(\mathcal{L}^{\pi}\varphi\right)(x)$
$\displaystyle:=-\beta\varphi(x)+\tilde{b}(x,\pi)\cdot\nabla\varphi(x)+\frac{1}{2}\tilde{\sigma}(x,\pi)^{2}\circ\nabla^{2}\varphi(x),$

which leads to the following characterization of the HJ equation:

$-\mathcal{L}^{\pi}V(x;\pi)=\tilde{r}(x,\pi)+\gamma\tilde{p}(x,\pi).$ (41)

We need the following two lemmas concerning the operator $\mathcal{L}^{\pi}$.

######  Lemma 8.

For any $\varphi\in C^{2}(\mathbb{R}^{n})$, we have

$\int_{\mathbb{R}^{n}}d^{\pi}_{x}(y)(-\mathcal{L}^{\pi}\varphi)(y)\mathrm{d}y=\varphi(x).$

Proof. The left hand side of the above equation is

$\displaystyle=$ $\displaystyle\mathbb{E}\int_{0}^{\infty}e^{-\beta
s}\left(\beta\varphi(\tilde{X}^{\pi}_{s})-\tilde{b}(\tilde{X}^{\pi}_{s},\pi)\frac{\partial\varphi}{\partial
x}(\tilde{X}^{\pi}_{s})-\frac{1}{2}\tilde{\sigma}(\tilde{X}^{\pi}_{s},\pi)^{2}\frac{\partial^{2}\varphi}{\partial
x^{2}}(\tilde{X}^{\pi}_{s})\right)\mathrm{d}s$ $\displaystyle=$
$\displaystyle\mathbb{E}\int_{0}^{\infty}e^{-\beta
s}\left[\left(\beta\varphi(\tilde{X}^{\pi}_{s})-\tilde{b}(\tilde{X}^{\pi}_{s},\pi)\frac{\partial\varphi}{\partial
x}(\tilde{X}^{\pi}_{s})-\frac{1}{2}\tilde{\sigma}(\tilde{X}^{\pi}_{s},\pi)^{2}\frac{\partial^{2}\varphi}{\partial
x^{2}}(\tilde{X}^{\pi}_{s})\right)\mathrm{d}s-\tilde{\sigma}(\tilde{X}^{\pi}_{s},\pi)\frac{\partial\varphi}{\partial
x}(\tilde{X}^{\pi}_{s})\mathrm{d}B_{s}\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}\int_{0}^{\infty}\mathrm{d}\left(-e^{-\beta
s}\varphi(\tilde{X}^{\pi}_{s})\right)$ $\displaystyle=$
$\displaystyle\lim_{s\rightarrow\infty}\left(-e^{-\beta
s}\varphi(\tilde{X}^{\pi}_{s})\right)+\varphi(\tilde{X}^{\pi}_{0})$
$\displaystyle=$ $\displaystyle\varphi(x),$

where the first equality follows from the definition of the occupation time
and the third equality from Itô’s formula. $\square$

######  Lemma 9.

Let $\pi,\hat{\pi}$ be two feedback policies. We have

$(\mathcal{L}^{\hat{\pi}}-\mathcal{L}^{\pi})V(x;\pi)+\tilde{r}(x,\hat{\pi})-\tilde{r}(x,\pi)-\gamma\tilde{p}(x,\pi)=\int_{\mathcal{A}(x)}\hat{\pi}(a\mid
x)q(x,a;\pi)\mathrm{d}a.$ (42)

Proof. By definition of $q(x,a;\pi)$ in (11), we have

RHS $\displaystyle=$ $\displaystyle\int_{\mathcal{A}(x)}\hat{\pi}(a\mid
x)\left(\mathcal{H}^{a}\left(x,\frac{\partial V}{\partial
x}\left(x;\pi\right),\frac{\partial^{2}V}{\partial
x^{2}}\left(x;\pi\right)\right)-\beta V\left(x;\pi\right)\right)\mathrm{d}a$
$\displaystyle=$ $\displaystyle\int_{\mathcal{A}(x)}\hat{\pi}(a\mid
x)\left(b(x,a)\cdot\frac{\partial V}{\partial
x}\left(x;\pi\right)+\frac{1}{2}\sigma^{2}(x,a)\circ\frac{\partial^{2}V}{\partial
x^{2}}\left(x;\pi\right)+r(x,a)-\beta V\left(x;\pi\right)\right)\mathrm{d}a$
$\displaystyle=$
$\displaystyle\tilde{r}(x,\hat{\pi})+\mathcal{L}^{\hat{\pi}}V^{\pi}(x)$
$\displaystyle=$
$\displaystyle\tilde{r}(x,\hat{\pi})-\tilde{r}(x,\pi)-\gamma\tilde{p}(x,\pi)+\mathcal{L}^{\hat{\pi}}V^{\pi}(x)-\mathcal{L}^{\pi}V^{\pi}(x)$
$\displaystyle=$ $\displaystyle\text{LHS}.$

$\square$

Proof of Theorem 2. Note that in (13), the equation to be proven, the right
hand side can be written as
$\int_{\mathbb{R}}d^{\hat{\pi}}_{\mu}(y)f(x;\pi,\hat{\pi})\mathrm{d}y$, with

$f(x;\pi,\hat{\pi}):=\int_{\mathcal{A}}\hat{\pi}(a\mid
x)\left(q(x,a;\pi)+\gamma p(x,a,\hat{\pi})\right)\mathrm{d}a.$

From Lemma 9, we have

$f(x;\pi,\hat{\pi})=(\mathcal{L}^{\hat{\pi}}-\mathcal{L}^{\pi})V(x;\pi)+\tilde{r}(x,\hat{\pi})+\gamma\tilde{p}(x,\hat{\pi})-\tilde{r}(x,\pi)-\gamma\tilde{p}(x,\pi).$
(43)

On the other hand, for the left hand side of (13), we have

$\eta(\pi)=\int_{\mathbb{R}^{n}}V(y;\pi)\mu(\mathrm{d}y)=\int_{\mathbb{R}^{n}}d^{\hat{\pi}}_{\mu}(y)(-\mathcal{L}^{\hat{\pi}})V(y;\pi)\mathrm{d}y,$
(44)

with the second equality following from Lemma 8; and

$\eta(\hat{\pi})=\int_{\mathbb{R}}d^{\hat{\pi}}_{\mu}(y)\left[\tilde{r}(y,\hat{\pi})+\gamma\tilde{p}(y,\hat{\pi})\right]\mathrm{d}y,$
(45)

following the definition of the discounted expected occupation time; moreover,
from (41), we have

$0=\int_{\mathbb{R}}d^{\hat{\pi}}_{\mu}(y)\left[(-\mathcal{L}^{\pi})V(y;\pi)-\tilde{r}(y,\pi)-\gamma\tilde{p}(y,\pi)\right]\mathrm{d}y.$
(46)

Hence, combining the last three equations (44,45,46), we have

$\eta(\hat{\pi})-\eta(\pi)=\int_{\mathbb{R}}d^{\hat{\pi}}_{\mu}(y)\left[(\mathcal{L}^{\hat{\pi}}-\mathcal{L}^{\pi})V(y;\pi)+\tilde{r}(y,\hat{\pi})+\gamma\tilde{p}(y,\hat{\pi})-\tilde{r}(y,\pi)-\gamma\tilde{p}(y,\pi)\right]\mathrm{d}y.$
(47)

Thus, we have shown LHS=RHS in (13). $\square$

###  B.2 Proof of Theorem 3

Proof. It suffices to show the integral version of the theorem:

$\displaystyle\nabla_{\theta}\left(\eta(\pi^{\theta})\right)\mid_{\theta=\theta}=\int_{\mathbb{R}^{n}}d^{\pi^{\theta}}_{\mu}(x)\left[\int_{\mathcal{A}}\nabla_{\theta}\pi^{\theta}(a\mid
x)\left(q(x,a;\pi^{\theta})+\gamma p(x,a,\pi^{\theta})\right)+\right.$ (48)
$\displaystyle\left.\gamma\cdot\pi^{\theta}(a\mid
x)\nabla_{\theta}p(x,a,\pi^{\theta})\mathrm{d}a\right]\mathrm{d}x.$

As before, we simplify notation by denoting $\eta(\pi^{\theta})$ as
$\eta(\theta)$ and $d^{\pi^{\theta}}$ as $d^{\theta}$. Then, by Theorem 2), we
have

$\eta(\theta+\delta\theta)-\eta(\theta)=\int_{\mathbb{R}^{n}}d^{\theta+\delta\theta}_{\mu}(x)\left[\int_{\mathcal{A}}\pi^{\theta+\delta\theta}(a\mid
x)\left(q(x,a;\theta)+\gamma
p(x,a,\theta+\delta\theta)\right)\mathrm{d}a\right]\mathrm{d}x.$ (49)

Denote

$f(\delta\theta)=\int_{\mathcal{A}}\pi^{\theta+\delta\theta}(a\mid
x)\left(q(x,a;\theta)+\gamma p(x,a,\theta+\delta\theta)\right)\mathrm{d}a.$

Note that $f(0)=0$, which follows from

$\displaystyle f(0)$ $\displaystyle=\int_{\mathcal{A}}\pi^{\theta}(a\mid
x)\left(q(x,a;\theta)+\gamma p(x,a,\theta)\right)\mathrm{d}a$
$\displaystyle=\int_{\mathcal{A}}\pi^{\theta}(a\mid
x)\left(\mathcal{H}^{a}(x,\frac{\partial V}{\partial
x}\left(x;\pi\right),\frac{\partial^{2}V}{\partial
x^{2}}\left(x;\pi\right))-\beta V\left(x;\pi\right)+\gamma
p(x,a,\theta)\right)\mathrm{d}a$ $\displaystyle=-\beta
V(x;\pi)+\tilde{b}(x,\pi)\cdot\nabla
V(x;\pi)+\frac{1}{2}\tilde{\sigma}^{2}(x,\pi)\circ\nabla^{2}V(x;\pi)+\tilde{r}(x,\pi)+\gamma\tilde{p}(x,\pi)$
$\displaystyle=0.$

Thus,

$\displaystyle\eta(\theta+\delta\theta)-\eta(\theta)$ $\displaystyle=\langle
d^{\theta+\delta\theta}_{\mu},f(\delta\theta)\rangle$ $\displaystyle=\langle
d^{\theta+\delta\theta}_{\mu},f(\delta\theta)\rangle-\langle
d^{\theta+\delta\theta}_{\mu},f(0)\rangle$ $\displaystyle=\langle
d^{\theta+\delta\theta}_{\mu},f(\delta\theta)-f(0)\rangle$
$\displaystyle=\langle
d^{\theta+\delta\theta}_{\mu}-d^{\theta}_{\mu},f(\delta\theta)-f(0)\rangle+\langle
d^{\theta}_{\mu},f(\delta\theta)-f(0)\rangle.$

Dividing both sides by $\delta\theta$ completes the proof, as the first term
on the last line above is of higher order than $\delta\theta$. $\square$

###  B.3 Proofs of Lemma 4 and Theorem 5

We need a lemma for the perturbation bounds.

######  Lemma 10.

Assume that both $\tilde{\sigma}^{2}(x,\hat{\pi}(\cdot))$ and
$\tilde{\sigma}^{2}(x,\pi(\cdot))$ are positive definite and

$\tilde{\sigma}^{2}(x,\pi(\cdot)),\tilde{\sigma}^{2}(x,\hat{\pi}(\cdot))\geq\sigma_{0}^{2}\cdot
I.$

where $\sigma_{0}>0$, then we have that the difference between the square root
matrix is bounded by

$\|\tilde{\sigma}(x,\hat{\pi})-\tilde{\sigma}(x,\pi)\|_{2}\leq\frac{1}{2\sigma_{0}}\|\tilde{\sigma}^{2}(x,\hat{\pi})-\tilde{\sigma}^{2}(x,\pi)\|_{2}.$

If we also assume that the upper bounds, i.e.

$\tilde{\sigma}^{2}(x,\pi(\cdot)),\tilde{\sigma}^{2}(x,\hat{\pi}(\cdot))\leq\bar{\sigma}_{0}^{2}\cdot
I.$

by some $\bar{\sigma}_{0}>\sigma_{0}>0$, then we have

$\|\tilde{\sigma}(x,\hat{\pi})-\tilde{\sigma}(x,\pi)\|_{2}\leq\frac{\bar{\sigma}_{0}}{2\sigma_{0}}\|\hat{\pi}-\pi\|^{\frac{1}{2}}_{1}.$

Proof. Consider a normalized vector $x$ with $\|x\|_{2}=1$ is an eigenvector
of $A^{\frac{1}{2}}-B^{\frac{1}{2}}$ with eigenvalue $\mu$ then

$\displaystyle x^{T}(A-B)x$
$\displaystyle=x^{T}(A^{\frac{1}{2}}-B^{\frac{1}{2}})A^{\frac{1}{2}}x+x^{T}B^{\frac{1}{2}}(A^{\frac{1}{2}}-B^{\frac{1}{2}})x$
$\displaystyle=\mu x^{T}(A^{\frac{1}{2}}+B^{\frac{1}{2}})x.$

thus, if $A,B\geq\sigma_{0}^{2}I$, this implies

$\mu\leq\frac{|x^{T}(A-B)x|}{x^{T}(A^{\frac{1}{2}}+B^{\frac{1}{2}})x}\leq\|A-B\|_{2}\cdot\lambda_{\min}(A^{\frac{1}{2}}+B^{\frac{1}{2}})^{-1}\leq\|A-B\|_{2}/(2\sigma_{0}).$

Furthermore, note that

$\tilde{\sigma}^{2}(x,\hat{\pi})-\tilde{\sigma}^{2}(x,\pi)=\int_{\mathcal{A}}\sigma^{2}(x,a)(\tilde{\pi}(a|x)-\pi(a|x))\mathrm{d}a.$

so

$\|\tilde{\sigma}^{2}(x,\hat{\pi})-\tilde{\sigma}^{2}(x,\pi)\|_{2}\leq\bar{\sigma}_{0}^{2}\int_{\mathcal{A}}|\tilde{\pi}(a|x)-\pi(a|x)|\mathrm{d}a=\bar{\sigma}_{0}^{2}\cdot\|\tilde{\pi}(a|x)-\pi(a|x)\|_{1}.$

$\square$

Proof (of Lemma 4). Consider the Wasserstein-2 distance $W_{2}(\mu,v)$ between
distribution $\mu$ and $v$ as

$W_{2}(\mu,\nu)=\left(\inf_{\gamma\in\Gamma(\mu,\nu)}\mathbf{E}_{(x,y)\sim\gamma}\|x-y\|_{2}^{2}\right)^{1/2},$

where $\Gamma(\mu,\nu)$ is the set all probability measures on the product
space $\mathbb{R}^{n}\times\mathbb{R}^{n}$ with the marginal distributions
being $\mu$ and $v$, and $\|\cdot\|_{2}$ is the standard Euclidean distance.
Denote

$\bar{d}^{\pi}_{\mu}:=\beta d^{\pi}_{\mu}.$

We want to get an upper bound on
$W_{2}(\bar{d}^{\pi}_{\mu},\bar{d}^{\hat{\pi}}_{\mu})$ in terms of the
distance between two policies $\pi$ and $\hat{\pi}$. Consider a specific
coupling $(X_{t},Y_{t})$ below:

$\left\\{\begin{aligned}
\mathrm{d}X_{s}&=\tilde{b}\left(X_{s},\pi\left(\cdot\mid
X_{s}\right)\right)\mathrm{d}s+\tilde{\sigma}\left(X_{s},\pi\left(\cdot\mid
X_{s}\right)\right)\mathrm{d}B_{s},\\\
\mathrm{d}Y_{s}&=\tilde{b}\left(Y_{s},\hat{\pi}\left(\cdot\mid
Y_{s}\right)\right)\mathrm{d}s+\tilde{\sigma}\left(Y_{s},\hat{\pi}\left(\cdot\mid
Y_{s}\right)\right)\mathrm{d}B_{s}.\end{aligned}\right.$ (50)

with $X_{0}=Y_{0}$, which leads to a joint distribution over
$\mathbb{R}^{n}\times\mathbb{R}^{n}$:

$\tilde{\gamma}:=\left\\{\tilde{p}(x,y)=\int_{0}^{\infty}\frac{1}{\beta}e^{-\beta
t}f_{(X_{t},Y_{t})}(x,y)\mathrm{d}t\right\\}.$

Hence,

$W_{2}^{2}(\bar{d}^{\pi}_{\mu},\bar{d}^{\hat{\pi}}_{\mu})\leq\mathbb{E}_{(x,y)\sim\tilde{\gamma}}\|x-y\|_{2}^{2}=\int_{0}^{\infty}\frac{1}{\beta}e^{-\beta
s}\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}\mathrm{d}s.$ (51)

It then boils down to estimating $\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}$. By Itô’s
formula,

$\displaystyle\mathrm{d}\|X_{s}-Y_{s}\|_{2}^{2}=$ $\displaystyle
2(X_{s}-Y_{s})^{\top}\left[(\tilde{b}\left(X_{s},\pi\right)-\tilde{b}\left(Y_{s},\hat{\pi}\right))\mathrm{d}s+(\tilde{\sigma}\left(X_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\hat{\pi}\right))\mathrm{d}B_{s}\right]$
$\displaystyle+\operatorname{Tr}\left[(\tilde{\sigma}\left(X_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\hat{\pi}\right))^{2}\right]\mathrm{d}s.$

Taking expectation on both sides yields

$\frac{\mathrm{d}}{\mathrm{d}s}\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}=2\underset{(A)}{\underbrace{\mathbb{E}\left[(X_{s}-Y_{s})^{\top}(\tilde{b}\left(X_{s},\pi\right)-\tilde{b}\left(Y_{s},\hat{\pi}\right))\mathrm{d}s\right]}}+\underset{(B)}{\underbrace{\operatorname{Tr}\left[\mathbb{E}(\tilde{\sigma}\left(X_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\hat{\pi}\right))^{2}\right]}},$
(52)

with

(A)
$\displaystyle=\mathbb{E}\left[(X_{s}-Y_{s})^{\top}(\tilde{b}\left(X_{s},\pi\right)-\tilde{b}\left(Y_{s},\pi\right))\mathrm{d}s\right]+\mathbb{E}\left[(X_{s}-Y_{s})^{\top}(\tilde{b}\left(Y_{s},\pi\right)-\tilde{b}\left(Y_{s},\hat{\pi}\right))\mathrm{d}s\right]$
$\displaystyle\leq
C_{\tilde{b}}\cdot\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}+\frac{1}{2}\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}+\frac{1}{2}\mathbb{E}\|\tilde{b}\left(Y_{s},\pi\right)-\tilde{b}\left(Y_{s},\hat{\pi}\right)\|_{2}^{2}$
$\displaystyle\leq(C_{\tilde{b}}+\frac{1}{2})\cdot\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}+\frac{1}{2}\|\tilde{b}(\cdot,\pi)-\tilde{b}(\cdot,\hat{\pi})\|_{2,\infty}^{2};$

and

(B)
$\displaystyle=\mathbb{E}\|\tilde{\sigma}\left(X_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\hat{\pi}\right)\|^{2}_{F}$
$\displaystyle\leq
2\mathbb{E}\|\tilde{\sigma}\left(X_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\pi\right)\|^{2}_{F}+2\mathbb{E}\|\tilde{\sigma}\left(Y_{s},\pi\right)-\tilde{\sigma}\left(Y_{s},\hat{\pi}\right)\|^{2}_{F}$
$\displaystyle\leq
2C_{\tilde{\sigma}}^{2}\cdot\mathbb{E}\left\|X_{s}-Y_{s}\right\|^{2}_{2}+2\sup_{x}\|\tilde{\sigma}\left(x,\pi\right)-\tilde{\sigma}\left(x,\hat{\pi}\right)\|^{2}_{F}$
$\displaystyle:=2C_{\tilde{\sigma}}^{2}\cdot\mathbb{E}\left\|X_{s}-Y_{s}\right\|^{2}_{2}+2\|\tilde{\sigma}\left(\cdot,\pi\right)-\tilde{\sigma}\left(\cdot,\hat{\pi}\right)\|^{2}_{F,\infty}.$

Combining the above, we get

$\frac{\mathrm{d}}{\mathrm{d}s}\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}\leq\underset{C_{\tilde{b},\tilde{\sigma}}}{\underbrace{(2C_{\tilde{b}}+1+2C_{\tilde{\sigma}}^{2})}}\mathbb{E}\|X_{s}-Y_{s}\|_{2}^{2}+\underset{C(\pi,\hat{\pi})}{\underbrace{\|\tilde{b}(\cdot,\pi)-\tilde{b}(\cdot,\hat{\pi})\|_{2,\infty}^{2}+2\|\tilde{\sigma}\left(\cdot,\pi\right)-\tilde{\sigma}\left(\cdot,\hat{\pi}\right)\|^{2}_{F,\infty}}}.$

By Grönwall’s inequality, we have

$\mathbb{E}\|X_{t}-Y_{t}\|_{2}^{2}\leq\frac{C(\pi,\hat{\pi})}{C_{\tilde{b},\tilde{\sigma}}}\left(e^{C_{\tilde{b},\tilde{\sigma}}t}-1\right).$
(53)

Substituting back into (51), we obtain

$W_{2}^{2}(\bar{d}^{\pi}_{\mu},\bar{d}^{\hat{\pi}}_{\mu})\leq\frac{C(\pi,\hat{\pi})}{C_{\tilde{b},\tilde{\sigma}}}\int_{0}^{\infty}\frac{1}{\beta}e^{-\beta
s}\left(e^{C_{\tilde{b},\tilde{\sigma}}s}-1\right)\mathrm{d}s.$

Thus, if $\beta>C_{\tilde{b},\tilde{\sigma}}$, we have

$W_{2}(\bar{d}^{\pi}_{\mu},\bar{d}^{\hat{\pi}}_{\mu})\leq\frac{C(\pi,\hat{\pi})}{C_{\tilde{b},\tilde{\sigma}}(\beta-
C_{\tilde{b},\tilde{\sigma}})\beta}.$

Concerning the term $C(\pi,\hat{\pi})$, we have

$\|\tilde{b}(\cdot,\pi)-\tilde{b}(\cdot,\hat{\pi})\|_{2,\infty}=\sup_{x}\|\tilde{b}\left(x,\pi\right)-\tilde{b}\left(x,\hat{\pi}\right)\|_{2}\leq\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|_{1}\cdot\sup_{x,a}|b(x,a)|,$

and

$\|\tilde{\sigma}(\cdot,\pi)-\tilde{\sigma}(\cdot,\hat{\pi})\|_{F,\infty}=\sup_{x}\|\tilde{\sigma}\left(x,\pi\right)-\tilde{\sigma}\left(x,\hat{\pi}\right)\|_{F}\leq\sqrt{n}\frac{\bar{\sigma}_{0}}{2\sigma_{0}}\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|_{1}^{\frac{1}{2}}.$

Thus we have:

$\displaystyle C(\pi,\hat{\pi})$
$\displaystyle=\|\tilde{b}(\cdot,\pi)-\tilde{b}(\cdot,\hat{\pi})\|_{2,\infty}^{2}+2\|\tilde{\sigma}\left(\cdot,\pi\right)-\tilde{\sigma}\left(\cdot,\hat{\pi}\right)\|^{2}_{F,\infty}$
$\displaystyle\leq\left(\sup_{x,a}|b(x,a)|^{2}+\frac{d\cdot\bar{\sigma}_{0}^{2}}{2\sigma_{0}^{2}}\right)\max\left(\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|_{1},\sup_{x}\|\hat{\pi}(\cdot|x)-\pi(\cdot|x)\|^{\frac{1}{2}}_{1}\right)$

which proves our upper bound. $\square$

Proof (of Theorem 5). We have that

$\displaystyle|\eta^{\hat{\pi}}-L^{\pi}(\hat{\pi})|$ $\displaystyle=|\langle
d_{\mu}^{\hat{\pi}}-d_{\mu}^{\pi},f\rangle|=\frac{\|f\|_{\dot{H}^{1}}}{\beta}\left|\left\langle\bar{d}_{\mu}^{\hat{\pi}}-\bar{d}_{\mu}^{\pi},\frac{f}{\|f\|_{\dot{H}^{1}}}\right\rangle\right|$
(54)
$\displaystyle\leq\frac{K}{\beta}\|\bar{d}_{\mu}^{\hat{\pi}}-\bar{d}_{\mu}^{\pi}\|_{\dot{H}^{-1}}\leq\frac{K\sqrt{M}}{\beta}W_{2}\left(\bar{d}_{\mu}^{\hat{\pi}},\bar{d}_{\mu}^{\pi}\right).$

where $K:=\sup_{\hat{\pi}}\|f\|_{\dot{H}^{1}}<\infty$ (more about $K$ in the
remarks below). Combining (54) with the estimate in (22) (of Lemma 4) yields
the desired result in (23). $\square$

Remarks (on $K$). In the performance-difference bound developed above, we
assume $K$ is finite:

$K:=\|f\|_{\dot{H}^{1}}:=\left(\int_{\mathbb{R}^{n}}|\nabla
f(x)|^{2}\mathrm{d}x\right)^{\frac{1}{2}}<\infty,$

where $f(x;\pi,\hat{\pi}):=\int_{\mathcal{A}}\hat{\pi}(a\mid
x)\left(q(x,a;\pi)+p(x,a,\hat{\pi})\right)\mathrm{d}a$. The famous Poincaré
inequality can provide a lower bound on this quantity; but we need an upper
bound as well, i.e.,

$K=\left(\int_{\mathbb{R}^{n}}|\nabla
f(x)|^{2}\mathrm{d}x\right)^{\frac{1}{2}}\leq
C\left(\int_{\mathbb{R}^{n}}|f(x)|^{2}\mathrm{d}x\right)^{\frac{1}{2}}.$

This above is essentially a reverse Poincaré Inequality, which is not likely
to hold (in particular, the existence of the constant $C$).

Should we indeed have a reverse Poincaré Inequality, then we can further bound
$f$ by

$\displaystyle|f(x)|$ $\displaystyle=|\int_{\mathcal{A}}\left(\hat{\pi}(a\mid
x)-\pi(a\mid x)\right)\left(q(x,a;\pi)+p(x,a,\hat{\pi})\right)\mathrm{d}a|$
$\displaystyle\leq\int_{\mathcal{A}}\left|\hat{\pi}(a\mid x)-\pi(a\mid
x)\right|\cdot\left|q(x,a;\pi)+p(x,a,\hat{\pi})\right|\mathrm{d}a$
$\displaystyle\leq
2\sup_{a}\left|q(x,a;\pi)+p(x,a,\hat{\pi})\right|D_{\operatorname{TV}}(\pi(\cdot\mid
x),\hat{\pi}(\cdot\mid x)),$

and

$\displaystyle\left(\int_{\mathbb{R}^{n}}|f(x)|^{2}\mathrm{d}x\right)^{\frac{1}{2}}$
$\displaystyle\leq\left(\int_{\mathbb{R}^{n}}4\sup_{a}\left|q(x,a;\pi)+p(x,a,\hat{\pi})\right|^{2}D^{2}_{\operatorname{TV}}(\pi(\cdot\mid
x),\hat{\pi}(\cdot\mid x))\mathrm{d}x\right)^{\frac{1}{2}}$
$\displaystyle\leq\left(\int_{\mathbb{R}^{n}}2\sup_{a}\left|q(x,a;\pi)+p(x,a,\hat{\pi})\right|^{2}\mathrm{d}x\right)^{\frac{1}{2}}\sqrt{\sup_{x}D_{\operatorname{KL}}(\pi(\cdot\mid
x),\hat{\pi}(\cdot\mid x))},$

where the second inequality is from Pinsker’s inequality. This way, we would
have recovered a similar bound as in the discrete RL. Since we do not have the
reverse Poincaré inequality, however, we have to assume that $K$ is finite.

##  Appendix C Algorithms

###  C.1 Performance of CPPO with Square-root KL and Linear KL

Here we present a detailed version of the CPPO algorithm. For two probability
distributions $P$ and $Q$ over the action space with density functions $p$ and
$q$ correspondingly, the KL-divergence between these two is defined as:

$D_{\mathrm{KL}}(P\|Q)=\int_{\mathcal{A}}\log(\frac{q(a)}{p(a)})q(a)\mathrm{d}a,$

Denote $D_{\mathrm{KL}}(\theta,\theta_{k}):=\mathbb{E}_{x\sim
d^{\theta_{k}}_{\mu}}D_{\mathrm{KL}}(\pi_{\theta}(\cdot|x)\|\pi_{\theta_{k}}(\cdot|x))$,
to distinguish it from

$\bar{D}_{\mathrm{KL}}(\theta\|\theta_{k}):=\mathbb{E}_{x\sim
d^{\theta_{k}}_{\mu}}\sqrt{D_{\mathrm{KL}}(\pi_{\theta}(\cdot|x)\|\pi_{\theta_{k}}(\cdot|x))}$

which was used in CPPO Algorithm in 2. Note that bounding the performance
difference by the linear KL-divergence $D_{\mathrm{KL}}(\theta,\theta_{k})$,
instead of its square-root counterpart
$\bar{D}_{\mathrm{KL}}(\theta\|\theta_{k})$, will generally require stronger
conditions (which may be difficult to satisfy). For completeness, we present
the following algorithm, the CPPO with linear KL-divergence:

Algorithm 3 CPPO: PPO with adaptive penalty constant (linear KL-divergence)

Input: Policy parameters $\theta_{0}$, critic net parameters $\phi_{0}$

1:for $k=0,1,2,\cdots$ until $\theta_{k}$ converge do

2: Collect a truncated trajectory
$\left\\{X_{t_{i}},a_{t_{i}},r_{t_{i}},p_{t_{i}}\right\\},i=1,\ldots,N$ from
the environment using $\pi_{\theta_{k}}$.

3: for $i=0,\ldots,N-1$ do: Update the critic parameters as in (8)

4: for $j=1,,\ldots,J$ do: Draw i.i.d. $\tau_{j}$ from $\exp(\beta)$, round
$\tau_{j}$ to the largest multiple of $\delta_{t}$ no larger than it, and
compute the GAE estimator of $q(X_{\tau_{j}},a_{\tau_{j}})$
$\tilde{q}(X_{\tau_{j}},a_{\tau_{j}}):=\left(r_{\tau_{j}}\delta_{t}+e^{-\beta\delta_{t}}V(X_{\tau_{j}+\delta_{t}})-V(X_{\tau_{j}})\right)/\delta_{t}.$

5: Compute policy update (by taking a fixed $s$ steps of gradient descent)
$\theta_{k+1}=\arg\max_{\theta}L^{\theta_{k}}(\theta)-C^{k}_{\text{penalty}}D_{\mathrm{KL}}\left(\theta,\theta_{k}\right).$

6: if
$D_{\mathrm{KL}}\left(\theta_{k+1},\theta_{k}\right)\geq(1+\epsilon)\delta$,
then $\quad C^{k+1}_{\text{penalty}}=2C^{k}_{\text{penalty}}.$

7: else if
$D_{\mathrm{KL}}\left(\theta_{k+1},\theta_{k}\right)\leq\delta/(1+\epsilon)$,
then $\quad C^{k+1}_{\text{penalty}}=C^{k}_{\text{penalty}}/2.$

A comparison between the above and Algorithm 2 (using square-root KL
divergence) is presented in §D.3 below, which clearly illustrates the
advantage of square-root KL divergence.

###  C.2 KL-divergence

We elaborate here on the KL-divergence between the current policy and the
optimal policy, along with the entropy regularizer. By the performance
difference formula, we have

$\eta(\pi)-\eta(\pi^{*})=\int_{\mathbb{R}^{n}}d^{\pi}_{\mu}(x)\left[\int_{\mathcal{A}}\pi(a\mid
x)\left(q(x,a;\pi^{*})-\gamma\log(\pi(a))\right)\mathrm{d}a\right]\mathrm{d}x.$

Notice that by the definition of KL-divergence we defined before, we have

$D_{\mathrm{KL}}(\pi^{*}(\cdot|x)\|\pi(\cdot|x))=\int_{\mathcal{A}}\log(\frac{\pi(a|x)}{\pi^{*}(a|x)})\pi(a|x)\mathrm{d}a.$

Similar as the previous discussion of soft $q$-learning, $\pi^{*}$ is optimal
implies that

$\pi^{*}(a\mid x)\propto\exp(\frac{q(x,a,\pi^{*})}{\gamma}),$

and the normalization constant is 1 can be proved through considering the
exploratory HJB equation, see [22, 56]. Thus

$D_{\mathrm{KL}}(\pi^{*}(\cdot|x)\|\pi(\cdot|x))=\int_{\mathcal{A}}\log(\pi(a|x))\pi(a|x)\mathrm{d}a-\int_{\mathcal{A}}\frac{q(x,a,\pi^{*})}{\gamma}\pi(a|x)\mathrm{d}a,$

which leads to

$\eta(\pi)-\eta(\pi^{*})=-\gamma\cdot\mathbb{E}_{x\sim
d^{\pi}_{\mu}}D_{\mathrm{KL}}(\pi^{*}(\cdot|x)\|\pi(\cdot|x)).$

This justifies our claim that the KL-divergence is essentially equivalent to
the distance to the optimal performance.

##  Appendix D Experiments

###  D.1 Example 1

Recall, in the LQ control problem, the reward function is

$r(x,a)=-\left(\frac{M}{2}x^{2}+Rxa+\frac{N}{2}a^{2}+Px+Qa\right),$

with $M\geq 0,N>0,R,Q,P\in\mathbb{R}$ and $R^{2}<MN$, and we adopt the entropy
regularizor as

$p(x,a,\pi)=-\log(\pi(a)).$

Furthermore, suppose that the discount rate satisfies
$\beta>2A+C^{2}+\max\left(\frac{D^{2}R^{2}-2NR(B+CD)}{N},0\right)$.

The following results are readily derived from Theorem 4 of [59]. The value
function of the optimal policy $\pi^{*}$ is

$V(x)=\frac{1}{2}k_{2}x^{2}+k_{1}x+k_{0},\quad x\in\mathbb{R},$

where

$\begin{gathered}k_{2}:=\frac{1}{2}\frac{\left(\rho-\left(2A+C^{2}\right)\right)N+2(B+CD)R-D^{2}M}{(B+CD)^{2}+\left(\rho-\left(2A+C^{2}\right)\right)D^{2}}\\\
-\frac{1}{2}\frac{\sqrt{\left(\left(\rho-\left(2A+C^{2}\right)\right)N+2(B+CD)R-D^{2}M\right)^{2}-4\left((B+CD)^{2}+\left(\rho-\left(2A+C^{2}\right)\right)D^{2}\right)\left(R^{2}-MN\right)}}{(B+CD)^{2}+\left(\rho-\left(2A+C^{2}\right)\right)D^{2}},\\\
k_{1}:=\frac{P\left(N-k_{2}D^{2}\right)-QR}{k_{2}B(B+CD)+(A-\rho)\left(N-k_{2}D^{2}\right)-BR},\end{gathered}$

and

$k_{0}:=\frac{\left(k_{1}B-Q\right)^{2}}{2\rho\left(N-k_{2}D^{2}\right)}+\frac{\gamma}{2\rho}\left(\ln\left(\frac{2\pi
e\gamma}{N-k_{2}D^{2}}\right)-1\right)$

respectively. Moreover, the optimal feedback control is Gaussian, with density
function

$\pi^{*}(a;x)=\mathcal{N}\left(a\mid\frac{\left(k_{2}(B+CD)-R\right)x+k_{1}B-Q}{N-k_{2}D^{2}},\frac{\gamma}{N-k_{2}D^{2}}\right).$

For a set of model parameters:
$A=-1,B=C=0,D=1,M=N=Q=2,R=P=1,\beta=1,\gamma=0.1$, following the formulas and
the parameterized policy $\pi_{\theta}(\cdot\mid
x)=\mathcal{N}(\theta_{1}x+\theta_{2},\exp(\theta_{3}))$, and the
corresponding value function
$V_{\phi}(x)=\frac{1}{2}\phi_{2}x^{2}+\phi_{1}x+\phi_{0}$, we can derive the
optimal parameters:

$\phi^{*}=[0.71914874,-0.10555128,-0.53518376],$

and

$\theta^{*}=[-0.39444872,-0.78889745,-1.40400944].$

Table 1: Hyper-parameter values for Example 1 Alphabet | Description | Value
---|---|---
$T$ | Trajectory Truncation Length | 25
$\beta$ | discount factor | 1
$\delta_{t}$ | time interval | 0.005
$J$ | batch size for sampling $\exp(\beta)$ | 100
$\alpha_{1}$ | learning rate for policy iteration $k$ | 0.02 when $k\leq 50$ and $0.02\times\log(\frac{50}{k})$ when $k>50$
$\alpha_{2}$ | learning rate for value iteration $k$ | 0.01 when $k\leq 50$ and $0.01\times\log(\frac{50}{k})$ when $k>50$
$K$ | iteration threshold | 2000
$s$ | steps of gradient descent | 10
$\delta$ | radius | 0.0002
$\epsilon$ | tolerance level | 0.5

###  D.2 Example 2

The model parameters are
$k=0.01,\theta=7,\eta=0.1,\rho=0.3,\sigma=1,r_{f}=0.01,\ell=5$. For both the
value function and the policy parameterization, we use a 3-layer neural
network, and with the initial parameters sampled form the uniform distribution
over [-0.5,0.5]. We use the tanh activation function for the hidden layer.

Table 2: Hyperparameter values for Example 2 Alphabet | Description | Value
---|---|---
$T$ | Trajectory Truncation Length | 25
$\beta$ | discount factor | 1
$\delta_{t}$ | time interval | 0.005
$J$ | batch size for sampling $\exp(\beta)$ | 100
$\alpha_{1}$ | learning rate for policy iteration $k$ | 0.005 when $k\leq 50$ and $0.005\times\log(\frac{50}{k})$ when $k>50$
$\alpha_{2}$ | learning rate for value iteration $k$ | 0.01 when $k\leq 50$ and $0.01\times\log(\frac{50}{k})$ when $k>50$
$K$ | iteration threshold | 200
$s$ | steps of gradient descent | 10
$\delta$ | radius | 0.025
$\epsilon$ | tolerance level | 0.5

###  D.3 Performance of CPPO with Square-root KL and Linear KL

We compare the performance of CPPO with square-root KL-divergence (denote as
CPPO), and linear KL-divergence (denoted as CPPO (nst) — non square-root)
applied to the experiments in Example 1 and Example 2.

Figure 4: Performance of CPPO and CPPO (nst) to the Example 1 Figure 5:
Performance of CPPO and CPPO (nst) to the Example 2

Figure 4 compares the distance between the current policy parameters and the
optimal parameters, with $x$-axis denoting the iteration times and $y$-axis
denoting the $L_{2}$ distance. Figure 5 compares the current expected return,
with $x$-axis denoting the iteration times and $y$-axis denoting the current
performance by taking the average of 100 times of Monte Carlo evaluation. In
both figures, the blue curve represents the algorithm with square-root KL-
divergence as opposed to the orange one corresponding to the linear version.
Both figures clearly demonstrate the advantage of the former. In particular,
the linear version can suffer from getting stuck at the local optimum as
demonstrated in Example 1.