Mixed ℋ_2/ℋ_∞ LQ Games for Robust Policy Optimization Under Unknown Dynamics

Leilei Cui
and Lekan Molu, Member, IEEE.

Leilei Cui is with the Control and Networks Lab, Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, Brooklyn, NY 11201. Lekan Molu is with Microsoft Research, 300 Lafayette Street, New York, NY 10012, USA.(email<EMAIL_ADDRESS>lekanmolu@microsoft.com).

L^2(S; F)
L^2(S; F̆)

§ INTRODUCTION

We are poised with robustly stabilizing optimal policies for stochastic dynamical systems

* with unknown state transition and control matrix parameters;

* exhibiting non-parametric uncertainties; or

* exhibiting parametric uncertainties.

For parametric uncertainties, our policy optimization (PO) scheme learns stabilizing and optimal policies for systems with imperfect information.
Techniques for systems possessing non-parametric uncertainties in literature typically assume an idealization of the noise as an additive stochastic process with zero correlation time (white noise) – unrealistic for most biological and cyberphysical systems. Here, the noise is an additive stochastic Brownian process with a nontrivial correlation structure. For non-parametric uncertainties that are additive in nature, it learns a robust policy for the control problem. When the system dynamics is altogether unknown, in an iterative fashion it learns the associated system model.

Control design with $\mathcal{H}_2$ or LQG lend many applicability to real-world stochastic control processes. These controllers construct linear systems' feedback compensators by minimizing a quadratic cost in the presence of a fixed noise (covariance) intensity (usually an additive Gaussian noise) that is subject to the system dynamics [1]. In this form, $\mathcal{H}_2$ controllers have found  applications in various problem domains since their introduction [2] such as robotics [3], autonomous vehicle [4], and recently in motor control [5] inter alia.

$\mathcal{H}_2$ control systems provide a few interesting properties. Firstly, the optimal feedback controller is a linear time-varying function of the state variable. As is well-known, linear time-varying systems tend to possess parametric and dynamic uncertainties that must be carefully managed throughout the life-cycle of a control process. Time-varying controllers deployed on systems with parametric or dynamic uncertainties (especially when unknown aforetime) are difficult to stabilize and they notoriously have no formal guarantees (e.g. see the counterexamples of [6]).

Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ policy optimization on a 3-link pendulum subject to an additive stochastic Brownian motion. The policy fails to regulate the system around equilibrium.

In light of these drawbacks, various authors have proposed robust time-domain schemes for $\mathcal{H}_2$ controllers. Of importance is Jacobson's pioneering work on linear exponential quadratic Gaussian control design [7]. Here, by taking the exponent of the LQ cost, a designer obtains stabilizing control laws that provide a measure of risk aversion or risk propensity. This formulation is particularly well-suited to certain economic decision processes. While Jacobson obtained a smooth solution to the associated Hamilton-Jacobi-Bellman equation [7], Duncan [8] generalized an elementary solution using a squares completion and Radon-Nikodym derivative scheme.

Khargonekar et al. [9] and Bernstein et al. [10] proposed an algebraic Riccati equation (ARE) solution under an $\mathcal{H}_\infty$ attenuation constraint of the closed-loop transfer function of a mixed  $\mathcal{H}_2/\mathcal{H}_\infty$ system.
Mustafa [11], via a minimum entropy approach, showed that a maximum entropy/$\mathcal{H}_\infty$ control is equivalent to a mixed $\mathcal{H}_2/\mathcal{H}_\infty$ control problem. Basar et al. [12] solved mixed $\mathcal{H}_2/\mathcal{H}_\infty$ control problem in a linear quadratic zero-sum differential game setting. It should be noted that most works require an accurate system model in order to solve the associated nonlinear indefinite ARE in an iterative fashion [13, 14, 15, 16]. Such models are typically difficult to obtain for complex systems.

Notably, reinforcement learning algorithms solve these problems under unknown system models. However, it is not clear what convergence guarantees thay do possess. With policy and value iterations in an adaptive dynamic programming framework, reinforcement learning has found applications in linear, nonlinear, and periodic continuous-time  systems particularly when handling optimal stabilization and output regulation problems [17, 18, 19, 20, 21, 22, 23, 24]. Pang et al. [25] studied the robustness of policy iteration under process noise in an input-to-state stability framework and showed that policy iteration finds an approximate solution to the optimal control problem. Utilizing the gradient of the performance index with respect to the parameters of the control policy, policy gradient algorithms were proposed in [26, 27, 28].  Along these lines of work, a learned controller optimizes the given performance index in an $\mathcal{H}_2$ sense only. As a result, it is not clear what robustness properties the resulting controller possesses.

Robust reinforcement learning based on mixed $\mathcal{H}_2/\mathcal{H}_\infty$ minimizes the performance index with guaranteed policy robustness to worst-case disturbance. For example, [29, 30, 31, 32] proposed adaptive dynamic programming approaches for zero-sum differential games.  The additional  $\mathcal{H}_\infty$ norm constraint imposed on the performance index in fact ensures robustness as the learning algorithm approaches infinity.  By estimating the gradient with zero-order methods, derivative-free algorithms were used to directly search for optimal policy parameters in [33, 34]. In [35, 14], the authors proposed an on-policy reinforcement learning scheme in a zero-sum LQ differential game setting based on gradients estimated by zeroth-order schemes.  Fazel et al. [36] estimated gradients (in a zeroth-order sense) of the cost difference between nominal and perturbed polices. For stochastic systems, zeroth-order methods are a special case of Monte Carlo methods – essentially high variance methods that produce slow learning [37]. Robust policy optimization in LQ zero-sum two-player game settings have also found applications in general simulated robotics and RL video game problems [35, 38, 39, 40]. Iteratively updating a controller over a performance index, these policy optimization schemes optimize the system's $\mathcal{H}_2$ norm. In the presence of additive noise to the system, however, it is not clear that these frameworks provide robustness, especially under unknown system dynamics. To buttress this point, consider the three-link pendulum under a classical mixed $\mathcal{H}_2/\mathcal{H}_\infty$ policy optimization scheme, but with a dynamic uncertainty present in the form of a Brownian disturbance.
As seen in <ref>, the PO scheme fails to stabilize the pendulum along the equilibrium position  (here $(0,0,0)$) for all three joint angles.

Classical LQ two-player games require an accurate measurement of the control inputs of the two players – which are rarely a given for many physical, chemical, and biological systems. In addition, disturbance and uncertainties are the norm rather than the exception for many feedback systems. It seems reasonable to place the convergence and robustness analyses of mixed $\mathcal{H}_2$/$\mathcal{H}_\infty$ LQ two-player zero sum game systems under imperfect information on a rigorous mathematical footing.  This is the essence of this article.

In this article, we are concerned with robust stabilization of optimal control problems in the presence of incorrect model assumptions, model parameters, or when  there is an unknown model altogether. Revisiting mixed  $\mathcal{H}_2$/$\mathcal{H}_\infty$ control [41], we introduce an iterative solution to the cost matrix of $\mc{H}_2$ control problems for two-player zero-sum differential games; we learn robustly stabilizing optimal policies in the presence of a worst-case disturbance in an iterative optimization scheme. Our scheme imbues policy optimization schemes with a robustness-preserving metric in a two-player zero-sum linear exponential quadratic Gaussian (LEQG) framework [7]. Our inquiry is motivated by the lack of robustness guarantees of time-domain linear quadratic  Gaussian (LQG) frameworks [6, 8] and the well-posedness of $\mathcal{H}_\infty$ control objectives in the presence of a worst-case disturbance for multivariable robust control [42, 43].

Contributions: Here are our contributions.

* Leveraging the model-based Gauss-Newton iterative algorithm of [14], we  propose a model-based two-stage iterative algorithm in an indefinite algebraic Ricatti equation (ARE) setting;

* The two-stage process is made up of an outer- and an inner-stage iteration – essentially two separate nonlinear discrete-time systems. We show that the systems globally exponentially  converge to the best control policy under the worst disturbance linearly;

* For small noise intensities at each algorithm iteration, we show that the computed control policy is  stabilizing and simultaneously satisfies the  $\mathcal{H}_\infty$ norm constraint;

* As the iteration approaches infinity, the obtained control policy  enters a small neighborhood of the exact solution of mixed $\mathcal{H}_2/\mathcal{H}_\infty$ control;

* In light of the above proposals, we propose a learning model-based  $\mathcal{H}_2/\mathcal{H}_\infty$ control algorithm;

* We provide an open-source implementation of our algorithm.

Applications of the methods prescribed herein could find use when optimizing a class of policies in modern control and reinforcement learning systems that

* are open-loop unstable or possess badly damped poles such as robot manipulators and space flexures;

* are minimum variance self-tuning in nature – these lack robustness to unmodeled certainties because they are constructed on the certainty equivalence principle <cit.>;

* are nonlinear, with stochastic  time-varying closed-loop feedback control schemes; these exhibit unknown convergence properties but for least-squares cost criterion <cit.>;

* have a misspecified model order causing poor performance – especially in linear quadratic regulators and LQG designs used in linearizing dynamical systems that arise in reinforcement learning settings.

The possible applications mentioned in the foregoing is not exhaustive.

The rest of this article is structured as follows: in Section <ref>, we set up notation, and introduce the problem. In Section <ref>, we present an iterative optimization scheme for solving a model-based mixed $\htwo$/$\hinf$ policy optimization problem, and a learning-based mixed $\mathcal{H}_2/\mathcal{H}_\infty$ control scheme is presented in Section <ref>. In Section <ref>, we analyze the convergence properties of our proposed algorithm. The robustness of the iterative algorithm is analyzed in Section <ref>. Finally, we  demonstrate the efficacy of our proposed algorithm with numerical results in Section <ref>. We discuss our findings and draw conclusions in Section <ref>. All theoretical machinery needed for proving our main results are given in the appendices.

§ BACKGROUND

In this section, we first set up notations that are commonly used throughout this article, give a few preliminary results for some of the machinery needed for proving our main results, then formally introduce the problem formulation.

§.§ Notations

We adopt vector-matrix notations throughout. Conventions: capital Roman letters are matrices; in lower-case they are vectors. Exceptions: $p, q, n, m$ are matrix or vector indices or dimensions.  Unless otherwise stated, optimization iteration indices are denoted by $i$ or $j$. $A:=B$ means that $A$ is defined by $B$, and $A=:B$ implies that $A$ defines $B$. We let $\mathbb{R}$ denote the set of real numbers, $\mathbb{N}$ the set of natural numbers, and $\mathbb{N}_+$ the set of positive integers. We let $\norm{\cdot}$ denote the 2-norm of a vector or the induced matrix norm.  We let $\norm{\cdot}_F$ denote the Frobenius norm of a matrix.

The Kronecker product is denoted by $\otimes$. The sub-matrix of the matrix $A$ that is comprised of the rows between the $i$th and $j$th rows is denoted by $[A]_{i:j}$. The maximum and minimum singular values of a matrix $T$ are respectively denoted by $\eigmax(T)$ and $\eigmin(T)$. For $X \in \mathbb{R}^{m \times n}$ and $\delta > 0$, $\mathcal{B}(X,\delta) := \{Y \in \mathbb{R}^{m \times n}\,|\, \norm{Y-X}_F \leq \delta\}$. The identity matrix with dimension $n$ is $I_n$.  For the transfer function $G(s)$, its $\mathcal{H}_\infty$ norm is a bounded linear operator defined as $\norm{G}_{\mathcal{H}_\infty} = \sup_{\omega \in \mathbb{R}} \eigmax(G(j\omega))$.

A symmetric matrix with dimension $n$ is denoted  $\mathbb{S}^n$. For a  $P \in \mathbb{S}^n$, $\vecs(P) := [p_{11}, 2p_{12}, \cdots, 2p_{1n}, p_{22}, \cdots, p_{nn}]^T$, where $p_{ij}$ is the $i$th row and $j$th column entry of $P$. Let the operator $\vect(A) := [a_1^T, \cdots, a_n^T]^T$, where $a_i$ is the $i$th column of the matrix $A$; and let $\vecv(x) := [x_1^2, x_1x_2, \cdots, x_1x_n, x_2^2, x_2x_3,\cdots,x_n^2]^T$. We let $T_v^{vs}$ and $T_{vs}^{v}$ respectfully denote inverse transformations that represent the matricization of $\vecs(P)$ and $\vect(P)$ for a $P \in \mathbb{S}^n$. $\vecs(P) = T_v^{vs}\vect(P), \quad  \vect(P) = T_{vs}^{v}\vecs(P)$. In addition, we let $T_{vt}$ denote the matricization of the vectorized $X \in \mathbb{R}^{m \times n}$, so that $\vect(X^T) =T_{vt}\vect(X)$.

§.§ Mixed Design as the LEQG Problem

Consider the following stochastic autonomous system

\begin{align}
dx(t) &= Ax(t)\,dt + Bu(t)\,dt + Ddw(t), \, x(0) = x_0, \\
z(t) &= Cx(t) + Eu(t),
\end{align}

where $x(t) \in \mathbb{R}^n$ is the state, $x(0)$ is the initial state, $u(t) \in \mathbb{R}^{m}$ is the control input, and $w(t) \in \mathbb{R}^q$ is the standard Wiener process defined over the probability space $(\Omega, \mathcal{F}, \mathcal{P})$[Here, $\Omega$ is the sample space, $\mathcal{F}$ is the $\sigma$-algebra i.e. the natural filtration for the Brownian motion, $\mathcal{P}$ is the probability measure for $t \in [0, T]$ where $T>0$ is fixed.], and the system output is $z(t) \in \mathbb{R}^{p}$. Matrices $A\in \mathbb{R}^{n \times n}$ and $B \in \mathbb{R}^{n \times m}$ while matrix $D \in \mathbb{R}^{n \times q}$. By design, $C  \in \mathbb{R}^{p \times n}$ and $E \in\mathbb{R}^{p \times m}$ are known. The transfer function from the disturbance $w(t)$ to $z(t)$ is

\begin{align}
\mathcal{T}(K)(s) = (C-EK)(sI_n - A + BK)^{-1}D.
\label{eq:HinfGainconstraint}
\end{align}

The linear exponential quadratic Gaussian (LEQG) stochastic optimal control problem (SOCP) [7, 44] is an adaptation of the standard LQG problem to processes that are risk-sensitive. Taking an exponent of the standard LQG cost and penalizing it with a scalar term $\gamma^{-2}$, the LEQG cost is

\begin{align}
\begin{split}
\mc{J}(K) &= \limsup_{\tau \rightarrow \infty} \frac{2\gamma^2}{\tau}\log\mathbb{E}\left[\exp\left(\frac{1}{2\gamma^2}\int_{0}^{\tau}z^T(t)z(t) dt\right)\right], %\\
%&\text{such that } \quad \eqref{eq: LTI with disturbance}.
\end{split}
\label{eq:LEQG}
\end{align}

where $\text{sign}(\gamma)$ for a $\gamma \in \reline$ dictates a risk-avoidance (i.e. $\gamma>0)$, risk-seeking (i.e. $\gamma <0$), or risk-neutral (i.e. $\gamma = 0$, whereupon we have the standard LQP problem) behavior for an ensuing controller to be sought. In  [14], the authors pointed out that the standard LEQG problem admits the form of the mixed $\htwo/\hinf$ suboptimal optimal control problem if written in the form (<ref>) for the linear time-invariant problem (<ref>).

The mixed design problem consists in minimizing an upper bound on the $\htwo$ cost (<ref>) subject to the $\hinf$ robustness constraint, $\|\mathcal{T}(K)\|_{\mathcal{H}_\infty}\leq \gamma$ so that all stabilizing feedback gains, $K$, for system (<ref>) can be realized within the set

\begin{align}
\mathcal{K} = \{K| (A-BK) \text{  Hurwitz, } ||\mathcal{T}(K)||_{\mathcal{H}_\infty}\leq \gamma\}.
\label{eq:feasible_set}
\end{align}

Formally, the control optimization problem is to find the set of stabilizing gains $K$ via

$\min_K \mc{J}(K) \text{ such that } K \in \mc{K}$.

In (<ref>) the noise intensity term must be well-conditioned to guarantee the existence of a solution to the SOCP. Therefore, we make the following standard assumptions.

Because we desire a stabilizing feedback controller, we require $(A,B)$ to be stabilizable. Furthermore, we require that unstable modes of $A$ be observable in order to guarantee a solution to the minimization of $\mc{J}_{K\in \mc{K}}$. Whence, we require that $(C,A)$ be observable.

$E^TE = R \succ 0$ and  $E^TC=0$.

Assumption <ref> implies that the indefinite ARE associated with (<ref>) has a unique positive definite solution and hence guarantees the existence of a stabilizing control law $u$.

Assumption <ref> simplifies the derivation of the solution to the indefinite ARE.

Given a constant $\gamma >0$, the solution to the LEQG problem (<ref>) is

\begin{align}
u^\star(x(t)) = -\underbrace{ R^{-1} B^T P^\star}_{K^\star} x(t),
\label{eq:opt_fdbk}
\end{align}

where $P^\star$ is the unique, stabilizing symmetric positive definite solution to the ARE

\begin{align}
A^T P^\star + P^\star A + C^TC - P^\star(B R^{-1} B^T - \gamma^{-2}DD^T)P^\star = 0.
\label{eq:ARE}
\end{align}

Our Riccati equation is an extension of the Riccati equation introduced by Duncan <cit.> for the finite-horizon LEQG problem. The controller

\begin{align}
u(x(t)) = -K_{\tau}(t)x(t):= -R^{-1}B^TP_{\tau}(t)x(t)
\end{align}

minimizes the  finite-horizon cost (by a slight abuse of notation)

\begin{align}
\mc{J}(K) = 	\mathbb{E}\left[\exp\left(\frac{1}{2\gamma^{2}}\int_{0}^{\tau}z^T \, z \, dt\right)\right],
\end{align}

where $P_{\tau}$ is the solution to the following continuous-time closed-loop (generalized) algebraic Riccati equation (GARE)

\begin{align}
A^T P_{\tau} + P_{\tau} A - P_{\tau} (B R^{-1} B^T - \gamma^{-2} D D^T)P_{\tau}  = 0
\end{align}

A fortiori, given assumptions <ref> and <ref>, we have $\lim_{\tau \rightarrow \infty}P_{\tau}(t) = P^\star$, and $\lim_{\tau \rightarrow \infty}K_{\tau}(t) = K^\star$ (This conclusion is a special case of <cit.>.

An LQG controller applied to a 3-link pendulum whose dynamics is corrupted by the worst-case disturbance. The controller fails to drive the system into steady-state.

Figure <ref> illustrates the ineffectiveness of a designed LQG controller in maintaining a 3-link pendulum system's trajectories at steady state as the system dynamics evolves.

§.§ Mixed Design as an LQ Zero-Sum Differential

Let us now consider the following two-player, zero-sum differential game with the quadratic linear cost,

\begin{align}
\min_{u \in \mathbb{R}^m} \max_{w \in \mathbb{R}^q}J(\cdot) = \int_{t=0}^{\infty} z^T(t) z(t) - \gamma^2 w^T(t) w(t) \, dt
\label{eq:LQZSDG}
\end{align}

for system (<ref>). The controller $u(t)$ is  minimizing while the disturbance $w(t)$ is maximizing. The solution to this differential game is given by Theorem 4.8 and 9.7 of [12], and it is summarized in Proposition <ref>. Henceforth, for conciseness we abuse notations, dropping the time arguments in $x(t)$ and $u(x(t))$ when the meaning is not diminished in our notations.

Suppose that there exists a nonempty set $\gamma_\infty = \inf\{\,\gamma>0 \, \mid \, \min_{u}\max_{w}J(0,u,w) \leq 0\}$, then the associated Riccati differential equation to  (<ref>) has a solution for a sufficiently large $\gamma$ $\gamma > \gamma_\infty$ on $[0, \tau]$.

The respective optimal controllers for the two players at time step $t$ are

\begin{align}
u^\star(\cdot) = -R^{-1} B^T P^\star x(t), \, w^\star (t) = \gamma^{-2}D^T P^\star x(t).
\label{eq:LQZSDGGain}
\end{align}

In addition, $P^\star \succ 0$ is the stabilizing solution of (<ref>).

This is just a statement of Th. 4.8 and 9.7 in [12].

Enforcing the gains $K$ over the set (<ref>), which is in the frequency domain, in general requires a difficult transformation. However, with the bounded real Lemma <ref>, we can express a relationship between a Riccati equation solution and a Riccati inequality. The Lemma is given in Appendix <ref>.
Now, given the Riccati equation (<ref>) and the equivalence of the $\hinf$ norm bound on the system transfer function to the Riccati equation and inequality in Lemma <ref>, we conclude that the optimal $K^\star \in \mathcal{K}$. Observe: minimizing the performance index (<ref>) under the worst-case disturbance, the optimal controller $u^*$ can robustly improve the system performance w.r.t the $\hinf$ norm penalty (this is demonstrated in our experiments).

Observing Propositions <ref> and <ref>, we see that both the LEQG and zero-sum differential game generate the same robust and optimal controller for the system, c.f. (<ref>) and (<ref>).

While Duncan [8]  proposed the Riccati equation (<ref>) as a means to realizing the optimal controller in (<ref>), no closed-form solution to the ARE exists to our knowledge. Conventional LQG control cannot guarantee the robustness of system (<ref>) or (<ref>) as found by Doyle [6]. In what follows, we first introduce a model-based algorithm followed by a model-free proposal for realizing the policies for $u$ and $w$ respectively.

§ MODEL-BASED ITERATIVE ALGORITHM

The ARE (<ref>) is a nonlinear matrix equation that does not have a closed-form solution. We establish the model-based iterative solution to (<ref>) in this section in what follows.

§.§ Algorithm Description

The procedure for obtaining the optimal $P^\star$ is now described. Let $i \in \bar{i}$ and $j \in \bar{j}$ denote the iteration indices for the outer and inner loop stages (i.e. update loops for the controller and disturbance respectively) of the algorithm respectively, where $\{\bar{i}, \bar{j}\} \in \mathcal{N}_+$. To aid our derivations, let us first define the following matrices:

Model-Based Iterative Algorithm

Initialize $K_1 \in \mathcal{K}$ pole placement
$i \leq \bar{i}$

Set $L_{K_i}^1 = 0$
$Q_{K_i} = C^TC + K_i^T R K_i$
$j \leq \bar{j}$

$A_{K_i}^{j} = A - BK_i + DL_{K_i}^j$  Update $A_{K_i}^{j}$
$(A_{K_i}^{j})^TP_{K_i}^j + P_{K_i}^j A_{K_i}^{j} + Q_{K_i} - \gamma^2(L_{K_i}^{j})^TL_{K_i}^{j} = 0$, Get $P_{K_i}^j$
$L_{K_i}^{j+1} = \gamma^{-2}D^T P_{K_i}^j$ Update disturbance gain $L_{K_i}^{j+1}$

$K_{i+1} = R^{-1}B^T P_{K_i}^{\bar{j}}$; Update control gain $K_{i+1}$

\begin{align}
A_{K_i}&= A-BK_i, \,\,A_{i}= A_{K_i}+\gamma^{-2}DD^TP_{K_i}, \\
A_{K_i}^{j}&= A_{K_i}+DL_{K_i}^j,\,\,A^\star= A-BK^\star + DL^{\star}, \\
Q_{K_i}&= C^TC + K_i^T R K_i,
\end{align}

where $A_{K_i}$ is the first player's closed-loop system transition matrix under an arbitrary feedback gain $K_i$ while $A_{i}$ is the second player's closed-loop system transition matrix. $A_{K_i}^{j}$ is the closed-loop system's transition matrix with arbitrary gains $K_i$ and $L_{K_i}$; and $A^\star$ is the closed-loop system's transition matrix with the optimal gains $K^\star$ and $L^\star$. The algorithm, described in Algorithm <ref>, is explained as follows:

* Starting at iteration $i=1$, set a $K_1 \in \mc{K}$ and an $L_{K_1}=0$. Iterate for $K_i$ until convergence:

* For $j=\{1,2,\cdots, \bar{j}\}$, iteratively solve the Riccati equation (<ref>). Call this solution  $P_{K_i}^j$;

* Increment $i$ by $1$; then, update the maximizing player's gain $L_{K_i}^{j+1}$ given $P_{K_i}^j$;

* Update the minimizing player's gain $K_{i+1}$ given $P_{K_i}^{\bar{j}}$.

Note that the system matrices $(A, B)$ are required to successfully run Algorithm <ref>. We defer treatment of when matrices $(A, B)$ are unknown to section <ref>.
Ours is similar to best-response alternating minimax iterative dynamic games of [45] between the two players in (<ref>).

§ MODEL-FREE ITERATIVE ALGORITHM

We are concerned with learning an optimal controller $K^\star$ when the matrices $(A, B)$ are unknown.

For this algorithm, only the trajectories of state $x$ and control input $u$ collected along system (<ref>) are required for computing the policies (<ref>). During the data collection phase, suppose the respective control policy is

\begin{align}
&u = -\hat{K}_1 x + \sigma_u \xi, \ d \xi = - \xi dt + dv
\label{eq: exploration input}
\end{align}

where $\xi \in \mathbb{R}^m$ is the exploration noise, $\sigma_u >0$ is a constant, and $v$ is a standard Brownian motion independent of $w$.

The Itô stochastic differential equation (<ref>) admits equal upper and lower values because the generalized ARE for $P$ admits a positive definite solution <cit.> given by $\mc{J}^\star = x^T P x$. Therefore, along the state trajectories of (<ref>), by Itô's formula <cit.>, the derivative of $x^T{P}x$, where $P \in \mathbb{S}^n$, can be verified to be

\begin{align}
\begin{split}
&d(x^TPx) = (dx)^TPx + x^TPdx + (dx)^TPdx \\
& \quad = x^T(A^TP + PA)xdt + 2x^T PBu dt + 2x^T P Ddw \\
& \qquad +  \underbrace{(Ax+Bu)^TP(Ax+Bu)(dt)^2}_{=0} \\
& \qquad + \underbrace{2(dw)^TD^TP(Ax+Bu)(dt)}_{=0} + \underbrace{(dw)^T D^TPDdw}_{=\Tr( D^TPDdwdw^T)} \\
& \quad = x^T(A^TP + PA)xdt + 2x^T PBu dt \\
&\qquad + 2x^T P Ddw + \Tr(D^T P D) dt.
\label{eq: derivative xPx}
\end{split}
\end{align}

Vectorization (<ref>), we have

\begin{align}
\begin{split}
&d(\vecv^T(x))\vecs(P) = \vecv^T(x)\vecs(A^TP + PA)dt+ \\
&2(x^T \otimes u^T)dt\vect(B^TP) + \Tr(D^T P D) dt + 2x^T P Ddw.
\end{split}
\label{eq: vectorization}
\end{align}

Suppose that $\phi(t) = \left[\vecv^T(x),2(x^T \otimes u^T),1\right]^T$. Integrating both sides of (<ref>) on the interval $[0, t_f]$ yields

\begin{align}
\begin{split}
&\dfrac{1}{t_f}\int_{0}^{t_f} \phi d(\vecv^T(x))\vecs(P)
= \begin{bmatrix}\vecs({A^T P + PA}) \\  \vect(B^TP) \\ \Tr(D^T {P} D)\end{bmatrix} \times \\
& \qquad\qquad \dfrac{1}{t_f}\int_{0}^{t_f} \phi \phi^T dt +  \dfrac{1}{t_f}\int_{0}^{t_f}2\phi x^T P Ddw.
\end{split}
\label{eq: intergration}
\end{align}

Furthermore, let

\begin{align}
\hat{\Phi}(t_f)&= \dfrac{1}{t_f}\int_{0}^{t_f}\phi(t) \phi^{T}(t) dt, \,\,%\nonumber \\
\hat{\Xi}(t_f)=\dfrac{1}{t_f}\int_{0}^{t_f} \phi d(\vecv^T(x)).
\label{eq:PhiAndXi}
\end{align}

Then, by Lemmas <ref> and <ref>, the following hold almost surely

\begin{align}
&\lim_{t_f \to \infty}\frac{1}{t_f}\int_{0}^{t_f} 2 \phi  x^T {P} D dw = 0, \, \lim_{t_f \to \infty} \hat{\Phi}(t_f) = \Phi := \mathbb{E}(\phi \phi^{T}).
\label{eq: ergodic definition}
\end{align}

Combining (<ref>) and (<ref>), we see that there exists a constant matrix $\Xi$ such that the following  holds almost surely

\begin{align}
\lim_{t_f \to \infty}\hat{\Xi}(t_f) = \Xi.
\label{eq: ergodic dvecv}
\end{align}

It follows from (<ref>) that

\begin{align}
\begin{bmatrix}\vecs({A^T P + PA}) \\  \vect(B^TP) \\ \Tr(D^T {P} D)\end{bmatrix} = \Phi^{-1} \Xi \vecs(P).
\end{align}

Suppose we let $n_1=n(n+1)/2$ and $n_2= n_1 + mn$, we find

\begin{align}
\begin{split}
&\vecs({A^T P + PA}) = [\Phi^{-1}]_{1:n_1} \Xi \vecs(P) \\
&\vect(B^TP) = [\Phi^{-1}]_{n_1+1:n_2} \Xi \vecs(P).
\end{split}
\label{eq: data represent model}
\end{align}

Let the $(i,j)$'th iterate of the GARE be

\begin{align}
\left(A_{K_i}^{j}\right)^TP_{K_i}^j + P_{K_i}^j A_{K_i}^{j} + Q_{K_i} - \gamma^2(L_{K_i}^{j})^TL_{K_i}^{j} = 0.
\label{eq:inner-loop evaluation}
\end{align}

Suppose that $P_{K_i}^j$ is the positive definite solution at iteration $j$ for the maximizing player corresponding to the solution at iteration $i$ for the minimizing player.

Equation (<ref>) is recursively solved for $L_{K_i}^{j+1} = \gamma^{-2}D^T P_{K_i}^j$ using the identities in equation (<ref>). At the $i$'th iteration, in the vectorization style of (<ref>),  becomes

\begin{align}
\begin{split}
&\vecs(A^TP_{K_i}^j + P_{K_i}^jA) - T_{v}^{vs}(I_n \otimes K_i^T)\vect(B^T P_{K_i}^j) \\
&- T_{v}^{vs}(K_i^T \otimes I_n)T_{vt}\vect(B^T P_{K_i}^j) \\
&+T_v^{vs}(I_n \otimes L_{K_i}^{jT}D^T + L_{K_i}^{jT}D^T \otimes I_n)T_{vs}^{v}\vecs(P_{K_i}^j) \\
&+ \vecs(Q_{K_i} - \gamma^{2} L_{K_i}^{jT} L_{K_i}^{j})= 0.
\end{split}
\label{eq: vector of inner-loop}
\end{align}

Putting the terms in (<ref>) in (<ref>), we find that

\begin{align}
\begin{split}
&[\Phi^{-1}]_{1:n_1} \Xi \vecs(P_{K_i}^j) \\
&- T_{v}^{vs}[(I_n \otimes K_i^T) + (K_i^T \otimes I_n)T_{vt}] [\Phi^{-1}]_{n_1+1:n_2} \Xi \vecs(P_{K_i}^j) \\
&+T_v^{vs}(I_n \otimes L_{K_i}^{jT}D^T + L_{K_i}^{jT}D^T \otimes I_n)T_{vs}^{v}\vecs(P_{K_i}^j) \\
&+ \vecs(Q_{K_i} - \gamma^{2} L_{K_i}^{jT} L_{K_i}^{j})= 0.
\end{split}
\label{eq: inner_loop by data}
\end{align}

Now, let

\begin{align}
\begin{split}
&\Lambda_{i}^{j} :=  [\Phi^{-1}]_{1:n_1} \Xi \\
&- T_{v}^{vs}[(I_n \otimes K_i^T) + (K_i^T \otimes I_n)T_{vt}][\Phi^{-1}]_{n_1+1:n_2} \Xi \\
&+ T_v^{vs}(I_n \otimes L_{K_i}^{jT}D^T + L_{K_i}^{jT}D^T \otimes I_n)T_{vs}^{v}.
\end{split}
\label{eq:alpha}
\end{align}

Then, the equation to be iterated upon becomes

\begin{align}
\vecs(P_{K_i}^j) = -(\Lambda_{i}^{j})^{-1}\vecs(Q_{K_i} - \gamma^{2} L_{K_i}^{jT} L_{K_i}^{j}).
\label{eq: solve P by data}
\end{align}

Observe: equation (<ref>) consists of unknown variables in the cost ($P_{K_i}^j$) and gain matrices ($K_i, \, L_{K_i}^{j}$), which can be iteratively computed using Algorithm <ref>. The other unknown variables from (<ref>) are trajectory variables. Hence, we have essentially transformed the model-based algorithm of Section <ref> into a model-free algorithm that requires only the state $x$ and control signal $u$ at time $t$ in solving for a robustly stabilizing optimal control law.
The learning-based algorithm for mixed $\mathcal{H}_2/\mathcal{H}_\infty$ is shown in Algorithm <ref>. Compared against Algorithm <ref> that requires the accurate model to update control policy, Algorithm <ref> only requires the input-state data to construct the necessary matrices $\hat{\Phi}(t_f)$ and $\hat{\Xi}(t_f)$. The gain $K_1$ can be determined via sum of squares means [47] for example, whereupon a control Lyapunov function (CLF) candidate can be found to guarantee global or local asymptotic stability. This is only done once before the algorithm is run. Hence, it does not greatly hamper the time-efficiency of the proposed scheme.

Learning-based $\mathcal{H}_2/\mathcal{H}_\infty$ Control
Initialize $\hat{K}_{1} \in \mathcal{K}$ searching for a valid CLF [47];

Collect data from (<ref>) with exploratory input (<ref>); and

Construct matrices $\hat{\Phi}(t_f)$ and $\hat{\Xi}(t_f)$;

$i \leq \bar{i}$

$j \leq \bar{j}$

Construct the matrices $\hat{\Lambda}_i^j(t_f)$ using (<ref>);

Calculate $\hat{P}_{K_i}^j$ using (<ref>);

Update $\hat{L}_{K_i}^j = \gamma^{-2}D^T\hat{P}_{K_i}^j$;

Form $\vect(\widehat{B^T{P}_{K_i}^{\bar{j}}})$ as $[\hat{\Phi}^{-1}(t_f)]_{n_1+1:n_2} \hat{\Xi}(t_f) \vecs(\hat{P}_{K_i}^{\bar{j}})$;

Calculate $\hat{K}_{i+1} = R^{-1} \widehat{B^T{P}_{K_i}^{\bar{j}}}$;

Next, we analyze the convergence of both loops.

§ CONVERGENCE ANALYSES

As seen in Algorithms <ref> and <ref>, the solution $P_{K_i}$ to the Riccati equation must converge to the unique optimal positive-definite solution $P^\star$ so that the gains $L_{K_i}^{j}$ and $K_{i+1}$ are optimal. In what follows, we provide a rigorous analysis of the convergence of the Riccati equation via a successive substitution scheme that is inspired by Kleinman's iterative Riccati computational scheme [48].

§.§ Control Update (Outer) Loop

The control law in the outer-loop of the algorithms
seeks to decrease the cost (<ref>) by iterating the following equations until convergence

\begin{align}
&A_{K_i}^T P_{K_i} + P_{K_i}A_{K_i} + Q_{K_i} + \gamma^{-2}P_{K_i}DD^TP_{K_i} = 0 	\label{eq:AREOuter_Riccati} \\
& K_{i+1} = R^{-1} B^T P_{K_i}, \quad i=1, 2, \cdots.
\end{align}

The control sequence (policy) $K_i$ guarantees the system's safety via the stabilizing robust controller (as we show in Theorem <ref>).

Previous works have shown that the controller update phase i.e. the outer-loop iteration has a global sub-linear convergence rate and local quadratic convergence rate <cit.>. We improve upon existing results in literature and demonstrate that the outer-loop iteration has a global linear convergence rate –  which improves the sub-linear convergence rate.

For any $K_1 \in \mathcal{K}$, the outer-loop iteration has a global linear convergence rate, i.e. there exists $\alpha \in [0,1)$, such that

\begin{align}
\Tr(P_{K_{i+1}} - P^\star) \leq \alpha \Tr(P_{K_{i}} - P^\star)
\label{eq:global_conv_control}
\end{align}

The proof to this theorem is provided in Appendix <ref>.

With Theorem <ref>, we have $\norm{P_{K_i} - P^\star}_F \leq \Tr(P_{K_i} - P^\star) \leq \alpha^i \Tr(P_{K_1} - P^\star)$.

§.§ Disturbance Update (Inner) Loop

In this part, via a successive substitution scheme inspired by Kleiman's iterative Riccati computation scheme [48], we will analyze the monotonic convergence of the inner loop of Algorithm <ref> and  <ref>.

Given $K \in \mathcal{K}$, the inner-loop iteration has a global linear convergence rate, i.e. for any $j \in \mathbb{N}_+$, there exists $\beta(K) \in [0,1)$, such that

\begin{align}
\Tr(P_K - P_K^{j+1}) \leq \beta(K) \Tr(P_K - P_K^{j}).
\label{eq: linear conv}
\end{align}

The proof is established in Appendix <ref>.

As seen from Theorem <ref>, $P_K - P_K^j \succeq 0$. From Lemma <ref> and the result of Theorem <ref>, we have $\norm{P_K - P_K^j}_F \leq \Tr(P_K - P_K^j) \leq \beta^{j-1}(K)\Tr(P_K)$, i.e. $P_K^j$ exponentially converges to $P_K$ in the sense of Frobenius norm.

§.§ Iterative Uniform Convergence

Given our construction so far,
for each $K_i$ the inner-loop iteration generates sequences $\{P_{K_i}^j\}_{i=1,j=1}^{i=\bar{i}, j=\bar{j}}$ and $\{L_{K_i}^j\}_{i=1,j=1}^{i=\bar{i}, j=\bar{j}}$ which converge to the worst-case cost matrix and  disturbance $P_{K_i}$ and $L_{K_i}$ respectively. We require that $\{P_{K_i}^j\}_{i=1,j=1}^{i=\bar{i}, j=\bar{j}}$ and $\{L_{K_i}^j\}_{i=1,j=1}^{i=\bar{i}, j=\bar{j}}$ enter the given neighborhood of $P_{K_i}$ and $L_{K_i}$ in a constant number of iterations. The following theorem guarantees uniform convergence after an equal number of inner-loop iterations i.e. the sequences generated by  $\{P_{K_i}^j\}$ and $\{L_{K_i}^j\}$ enter the vicinity of $P_{K_i}$ and $L_{K_i}$, irrespective of the different values of $K_i$.

For any $i \in \mathbb{N}_+$, and $\epsilon>0$, there exists $\bar{j} \in \mathbb{N}_+$ independent of $i$, such that if $j \geq \bar{j}$,

\begin{align}
\norm{P_{K_i}^j - P_{K_i}}_F \leq \epsilon.
\end{align}

The proof of this theorem is given in Appendix <ref>. That is, the iterations converge uniformly to an $\epsilon > 0$.

§ ROBUSTNESS ANALYSES

In the last section, we assumed that the accurate model of the system is known and the iterative algorithm can be implemented exactly. In practice, due to model mismatch and various noise induced by measurements and external disturbance, the proposed two-loop iterative algorithm can hardly be executed precisely. Hence, the robustness of the algorithm to these aforementioned noise and disturbance is critical. The question of the iterative algorithm finding an approximate optimal solution to (<ref>) and (<ref>) under noise uncertainties needs to be answered. In this section, by considering the outer loop and inner loop as two separate discrete nonlinear systems, we will analyze the robustness of inner-loop and outer-loop iterations separately in the sense of input-to-state stability (ISS) [49, 50].

§.§ Control (Outer) Loop

The exact outer-loop iteration is as (<ref>). At each iteration, $K_{i+1}$ can be updated precisely without the influence from noise and disturbance. Let

\begin{align}
&\hat{A}_{K_i} := A - B\hat{K}_i, \,\hat{A}_i = A - B\hat{K}_i + \gamma^{-2} DD^T \hat{P}_{K_i}, \\ &\hat{Q}_{K_i} = C^TC + \hat{K}_i^T R \hat{K}_i.
\end{align}

When noise exists and the policy is updated inaccurately, the inexact outer-loop iteration is

\begin{align}
& (\hat{A}_{K_i})^T\hat{P}_{K_i} + \hat{P}_{K_i}\hat{A}_{K_i} + \hat{Q}_{K_i} + \gamma^{-2}\hat{P}_{K_i}DD^T\hat{P}_{K_i} = 0, \label{eq:inexactOuter_evaluate}\\
& \hat{K}_{i+1} = R^{-1} B^T \hat{P}_{K_i} + \Delta K_i.
\label{eq:inexactOuter_update}
\end{align}

Henceforth, we set $\tilde{K}_i =  R^{-1} B^T \hat{P}_{K_i}$. Let us now give a statement of the theorem of the outer loop's robustness to perturbations.

There exists an $ \underline{l} > 0$, $\hat{\alpha} \in [0,1)$, and $\kappa(\cdot) \in \mathcal{K}_\infty$, such that $\norm{\hat{P}_{K_{i}} - P^\star}_F \leq \hat{\alpha}^{i-1} \Tr(\hat{P}_{K_1}-P^\star) + \kappa(\norm{\Delta K}_\infty)$, as long as $\norm{\Delta K}_\infty \leq \underline{l}$.

The proof is provided in Appendix <ref>.

That is, as iteration goes to infinity, $\hat{P}_{K_i}$ approaches the optimal value $P^\star$, entering its neighborhood. The radius of the neighbor is proportional to $\norm{\Delta K}_\infty^2$. Therefore we conclude that the outer loop of the iteration is robust to noise and uncertainties.

§.§ Disturbance (Inner) Loop

The exact inner-loop Iteration is (<ref>), and the control policy $L_{K_i}^j$ can be updated precisely. In reality, due to the influence of disturbance and noise, $L_{K_i}^j$ may be updated inaccurately. Therefore, the inexact inner-loop iteration is

\begin{align}
&(\hat{A}_{K_i}^{j})^T \hat{P}_{K_i}^j + \hat{P}_{K_i}^j \hat{A}_{K_i}^{j} + Q_{K_i} - \gamma^2(\hat{L}_{K_i}^{j})^T\hat{L}_{K_i}^{j} = 0, \label{eq:inexactInner_evaluate}\\
&\hat{L}_{K_i}^{j+1} = \gamma^{-2}D^T \hat{P}_{K_i}^j + \Delta L_{K_i}^j.\label{eq:inexactInner_update}
\end{align}

where $\{\hat{L}_{K_i}^{j}\}_{j=1}^{\infty}$ and $\{\hat{P}_{K_i}^{j}\}_{j=1}^{\infty}$ are sequences generated by the inexact inner-loop iteration (<ref>).

Assume $\norm{\Delta L_K^j} < e$ for all $j\in \mathbb{N}_+$. There exists $\hat{\beta}(K) \in [0,1)$, and $\lambda(\cdot) \in \mathcal{K}_\infty$, such that

\begin{align}
\norm{\hat{P}_{K}^j - P_{K}}_F \leq \hat{\beta}^{j-1}(K) \Tr(P_{K}) + \lambda(\norm{\Delta L}_\infty).
\end{align}

The proof of this Theorem can be found in Appendix <ref>. From Theorem <ref>, as $j \to \infty$, $\hat{P}_K^j$ approaches the solution $P_K$ and enters the ball centered by $P_K$. The radius of the ball is proportional to $\norm{\Delta L}_\infty$. Hence, the proposed inner-loop iterative algorithm finds the approximate value of $P_K$ in the presence of unmodeled noise dynamics.

§ NUMERICAL SIMULATIONS

In this section, we will demonstrate our theoretical results on double and three-link inverted pendulums.
The triple inverted pendulum is the base model for humanoid robots[51, 52] with the two upper hinge joints (hip and knee) being actuated while the lowest hinge (ankle) joint is passive. There are several challenges for a stabilizing controller design: the system is inherently unstable being non-minimum phase; it is underactuated system since the degrees of freedom is larger than the number of actuators; and the physical parameters of the humanoid robots are hard to accurately measure. In this section, the triple inverted pendulum is adopted as the numerical setups for our proposed algorithms. Specifically, we will design a learning-based balance PO scheme for this three-link robot with  inaccurate system model.

The state of the triple inverted pendulum is $x=[\theta_1, \theta_2, \theta_3, \dot{\theta}_1, \dot{\theta}_2, \dot{\theta}_3]^T$, where $\theta_1$, $\theta_2$, and $\theta_3$ are the angles of the ankle, hip, and knee. With actuator noise and possible installation error of the mechanism (the base is not securely attached to the ground), the linearized model of the triple inverted pendulum can be depicted by (<ref>), where $A \in \mathbb{R}^{6 \times 6}$ and $B \in \mathbb{R}^{6 \times 2}$ are as given in <cit.>, and $D = \begin{bmatrix} 0_{3 \times 3}, I_{3}\end{bmatrix}^T$.

Furthermore, we bound the system's $\hinf$ norm by $\gamma = 5$ from above. The initial state is set as

\begin{align}
x(0) = [0,-5,10,10,-10,10]^T.
\end{align}

The matrices related to the controlled output $z(t) = Cx(t) + Eu(t)$ are set as

\begin{align}
C = \begin{bmatrix} I_6 , 0_{2 \times 6} \end{bmatrix}^T, \quad E = \begin{bmatrix} 0_{6 \times 2} , I_2 \end{bmatrix}^T.
\end{align}

§.§ Comparison with LQG Control

Here, we assume the model of the systems are known, and Algorithm <ref> is applied to solve for the optimal controller of $u^\star(t) = -K^\star x(t)$ and the worst-case disturbance $w^\star(t) = L^\star x(t)$. We choose the LQG cost function as,

\begin{align}
J_{LQG} = \int_{0}^{\infty} x^TC^TCx + u^TE^TEu dt,
\end{align}

and we find the LQG feedback gain as

\begin{align}
K_{LQG} = \begin{bmatrix} -26.77 &-8.755 &-4.20	&-9.033	&-3.05	&-2.30 \\
-65.10	&-23.79	&-8.24	&-21.60	&-9.27	&-3.93 \end{bmatrix}.
\end{align}

While executing Algorithm <ref>, the numbers of iterations for outer and inner loops are heuristically chosen as $\bar{i} = 20$ and $\bar{j} = 20$ and the initial controller, $K_1$, was chosen via linear matrix inequality approach.

The result is shown in Fig. <ref>. We see that after around $5$ iterations, the controller and the cost matrix converge to the optimal solution.  Moreover, the ${\mathcal{H}_\infty}$ norms of closed-loop system with LQG and mixed ${\mathcal{H}_2/\mathcal{H}_\infty}$ controllers are respectively $8.72$ and  $4.99$. Thus, this algorithm generates an optimal controller and guarantees the ${\mathcal{H}_\infty}$ norm system constraint, while the LQG controller does violate the constraint.

Carrying along with $x(0)$, we compare the mixed $\mathcal{H}_2/\mathcal{H}_\infty$ controller with the LQG controller when $w$ is a

* Brownian motion; and

* worst-case disturbance respectively.

When the disturbance exhibits Brownian motion in nature, mixed $\mathcal{H}_2/\mathcal{H}_\infty$ controller and LQG controller results are illustrated in Figures <ref> and <ref> respectively. Notice the chattering in joint angles and angular velocities around equilibrium after $5s$. In addition, the magnitude of the joint angle under a mixed $\mathcal{H}_2/\mathcal{H}_\infty$ controller is within $[-5, +5]$ while that of the LQG controller is within $[-10, +10]$. Thus, for the Brownian motion, the mixed $\mathcal{H}_2/\mathcal{H}_\infty$ controller does suppress the chattering.

Under a worst-case disturbance, the results LQG and for mixed $\mathcal{H}_2/\mathcal{H}_\infty$ controller are shown in Figures  <ref> and <ref> respectively. The LQG controller fails to satisfy the $\mathcal{H}_\infty$ norm constraint as seen in Fig. <ref> i.e. the states becomes unstable after $1s$. As a comparison, consider Fig. <ref>, the state converges to equilibrium under the influence of the worst-case disturbance.

Using Algorithm <ref>, the gain matrix and cost matrix converge to the optimal values, and the $\mathcal{H}_\infty$ norm satisfies the constraint.

With LQG controller, the evolution of the joint angles and velocities under Brownian motion disturbance.

With our mixed $\mathcal{H}_2/\mathcal{H}_\infty$ policy, the evolution of the joint angles and velocities under a worst-case disturbance.

§.§ Comparison with Natural Policy Gradient Algorithm

We further compare Algorithm <ref> with the natural policy gradient (NPG) algorithm of Zhang et al [14] to test the veracity of the convergence rate and the robustness to process noise, $\Delta K$. At each iteration of the algorithm, a $\Delta K_i$ sampled from a standard Gaussian distribution with Frobenius norm normalized to $0.15$ is introduced into the algorithm following our derivations in Section <ref>. The results are shown in Figures <ref> and <ref>. As seen in Fig. <ref>,  the proposed iterative algorithm does approach the optimal solution after the $5$th iteration despite the disturbance. At the last iteration, the deviation from the optimal cost matrix, $\frac{\norm{\hat{P}_{K_{20}}- P^\star}}{\norm{P^\star}_F}$, is $2.9\%$, while the gain error, $\frac{\norm{\hat{K}_{20} - K^\star}_F}{\norm{K^\star}_F}$, is $2.6\%$. In contrary, the natural policy gradient has a cost matrix and controller gain errors that are unbounded as the iteration proceeds. Therefore, our algorithm is more robust than the natural policy gradient algorithm to process noise.

The computational time of Algorithm <ref> is compared with that of NPG, and the result is shown in Table <ref>. It is seen that for the double and triple inverted pendulums, the computational time of our algorithm is much less than that of NPG by around $90\%$. This is in fact a validation of our superior convergence rate (i.e. a global linear and local quadratic rate) compared to NPG's sublinear convergence rate.

Comparison of Alg. <ref> and NPG.

6|c|Computational time (sec)

3|c||Double Inverted Pendulum     3|c|Triple Inverted Pendulum

Alg. <ref>     Alg. <ref>     NPG     Alg. <ref>    Alg. <ref>    NPG

$0.0901$     $0.3061$       $2.1649$       $0.1455$      $0.7829$        $2.3209$

Robustness of Algorithm <ref> under a process noise whose norm is $\norm{\Delta K}_\infty = 0.15$.

Robustness of natural policy gradient under a noise norm $\norm{\Delta K}_\infty = 0.1$.

Algorithm <ref> generates an approximation to the robustly optimal controller based on noisy system data.

§.§ Results on Learning-based Control

For the parameters of Algorithm <ref>, we set $\bar{i} = 20$ and $\bar{j} = 30$, and collected data for  $t_f = 1500s$. The parameters of the $A$ and $B$ matrices are unknown but the initial controller $\hat{K}_1 \in \mathcal{K}$ is known.

We run Algorithm <ref> to find a near optimal solution of (<ref>) using the input and state data collected from system (<ref>). As seen in Fig. <ref>, the obtained controller $\hat{K}_i$ at each iteration converges after $5$ iterations. The corresponding evaluative matrix $\hat{P}_{K_i}$ also converges. At $20$th iteration, the relative error of ${||\hat{K}_{20}-K_{*}||}/{||K_{*}||} = 31.5\%$ and ${||\hat{P}_{K_{20}}-P_{*}||}/{||P_{*}||} = 31.6\%$. These demonstrate that the proposed algorithm can find an approximate optimal solution using the noisy data.

§ CONCLUSIONS

In the research effort presented in this article, we have proposed a two-loop iterative algorithm for the robust and optimal control of linear time-invariant systems. Rigorous convergence has been given that demonstrate that the proposed two-loop iterative algorithm has a global linear convergence alongside uniform convergence. Furthermore, by considering the iterative algorithm as a nonlinear system, we have presented novel robustness analyses and evaluation of the iterative algorithm in the sense of small-disturbance input-to-state stability. Based on these premises, a learning-based iterative algorithm has been developed to generate an approximate robust and optimal controller using noisy data collected from the system. The proposed algorithms are evaluated on two- and three-link inverted pendulum testbeds and our results confirm our various hypotheses.

§ ACKNOWLEDGMENTS

A vote of thanks to Professor Zhong-Ping Jiang of NYU Tandon School of Engineering for bringing this line of  research inquisition to our notice and for pointing out helpful reading materials.

§ A CALATOG OF PRELIMINARY LEMMAS

In this appendix, we provide Lemmas necessary for the construction of our main results. For further reading, readers can consult $H_\infty$ control theory texts such as [12, 54, 55].

For any symmetric and positive semi-definite matrix $P \in \mathbb{S}^n$, $\norm{P}_F \leq \Tr(P)$, $\norm{P} \leq \Tr(P)$, and $\Tr(P) \leq n \norm{P}$. For $x \in \mathbb{R}^n$, $x^T P x \geq \eigmin(P) \norm{x}^{2}$.

Let $\sigma_1 \geq \cdots \geq \sigma_n$ be ordered singular values of $P$. Since $P$ is symmetric and positive semi-definite, its singular values are equal to its eigenvalues. Then, $\norm{P}_F = \sqrt{\sum_{i=1}^n \sigma_i^2}$, $\Tr(P) = \sum_{i=1}^n \sigma_i$, and $\norm{P} = \sigma_1(P)$. Since $\sum_{i=1}^n \sigma_i^2 \leq (\sum_{i=1}^n \sigma_i)^2$, we have $\norm{P}_F \leq \Tr(P)$, $\norm{P} \leq \Tr(P)$, and $\Tr(P) \leq n \norm{P}$. Using Rayleigh's theorem <cit.>, we have $x^T P x \geq \eigmin(P) \norm{x}^{2}$.

For $X \in \mathbb{R}^{m \times n}$ and $Y \in \mathbb{R}^{n \times p}$, $\norm{XY}_F \leq \norm{X}\norm{Y}_F$.

Let $Y = [y_1,\cdots,y_p]$, then it follows that $XY = [Xy_1,\cdots,Xy_p]$. This implies that $\norm{XY}_F^2 = \sum_{i=1}^{p}\norm{Xy_i}^2$. Furthermore, as the spectral norm is defined by $\norm{X} = \max_{x \neq 0}\frac{\norm{Xx}}{\norm{x}}$, we have $\norm{Xy_i}^2 \leq \norm{X}^2 \norm{y_i}^2$. Hence, $\norm{XY}_F^2 \leq \norm{X}^2 \sum_{i=1}^{p}\norm{y_i}^2 = \norm{X}\norm{Y}_F $, which is in fact a proof of the theorem.

Assume $A \in \mathbb{R}^{n\times n}$ is Hurwitz and satisfies $A^TP + PA +Q = 0$. Then, the following properties hold

* $P = \int_{t=0}^{\infty} e^{(A^Tt)}Qe^{(At)} dt$;

* $P \succ 0$ if $Q\succ 0$, and $P \succeq 0$ if $Q\succeq 0$;

* If $Q\succeq 0$, then $(Q,A)$ is observable iff $P \succ 0$;

* If $P^\prime$ satisfies $A^TP^\prime + P^\prime A +Q^\prime = 0$, and $Q^\prime \preceq Q$, then $P^\prime \preceq P$.

The first three statements are proven in <cit.>. Consequently, $P^\prime$ can be expressed as

\begin{align}
&P^\prime = \int_{0}^\infty e^{A^Tt}Q^\prime e^{At} dt.
\end{align}

Since $Q^\prime \preceq Q$, $P^\prime \preceq P$.

Suppose $P$ satisfies $A^TP + PA +Q = 0$, then

* $A$ is Hurwitz if $P \succ 0$ and $Q \succ 0$.

* $A$ is Hurwitz if $P \succeq 0$, $Q \succeq 0$ and $(Q,A)$ is detectable.

This Lemma is proven in <cit.>.

Let $\left(X,\,Y\right) \in \mathbb{R}^{n \times n}$ and let $Y = Y^T \geq 0$. Then,

\begin{align*}
-\mu_2(-X)\Tr(Y) \leq \Tr(XY) \leq \mu_2(X) \Tr(Y)
\end{align*}

where $\mu_2(X)$ is the matrix measure, as a function of the spectral norm of the matrix $X$, i.e. $\frac{1}{2}\lambda_{max}(X+X^T)$.

This Lemma is proven in [57].

Let $U:S \to \mathbb{R}^{m \times r}$ and $V: S \to \mathbb{R}^{r \times p}$ be two matrix functions defined and differentiable on an open set $S$ in $\mathbb{R}^{n \times q}$. Then the simple product $UV$ is differentiable on $S$ and the Jacobian matrix is the $mp \times nq$ matrix
\begin{align}
\frac{\partial \vect(UV)}{\partial\vect(X)} = (V^T \otimes I_m)\frac{\partial \vect(U)}{\partial\vect(X)} + (I_p \otimes U)\frac{\partial \vect(V)}{\partial\vect(X)}.
\end{align}

This Lemma is proven in <cit.>.

Let $(\Omega, \mathcal{F},P)$ be a probability space, and $\{T_t \mid t \ge 0\}$ be a measurable semi-group of transformations preserving $(\Omega, \mathcal{F},P)$. Suppose that the time average operator $A_t$ is so defined

\begin{align}
A_tf(w) = \frac{1}{t}\int_{0}^{t}f(T_s(w)ds.
\end{align}

Then, for every $f \in L^{1}(P)$, there exists $\bar{f} \in L^{1}(P)$ such that $A_tf \to \bar{f}$ both P-almost surely and in $L^{1}(P)$ as $t \to \infty$, where $\bar{f} = \mathbb{E}[f]$.

This Lemma is proven in <cit.> and <cit.>.

Let $a(x(t))$ be a vector function such that $\mathbb{E}[a^T(x(t))a(x(t))] < \infty$ where $\{x(t)| t\geq 0\}$ is the solution of process $dx(t) = \mu(x(t))dt + dw$, and $w$ is a standard independent Brownian motion. It follows that

\begin{align}
\lim_{t_f \to \infty}\frac{1}{t_f}\int_{0}^{t_f} a(x(t))dw = 0.
\end{align}

This lemma is proven in <cit.>.

For the stabilizing gain matrix $K$, the following conditions are equivalent:

* $||\mathcal{T}(K)||_{\mathcal{H}_\infty} \leq \gamma$;

* The Riccati equation

\begin{align}
(A-B{K})^T & P_{K} + P_{K}(A-B{K}) + C^TC\nonumber \\
&\,  + K^TRK + \gamma^{-2}P_{K}DD^TP_{K} = 0
\end{align}

admits a unique stabilizing solution with a Hurwitz $A-B{K} + \gamma^{-2}DD^TP_K$;

* There exists some $P_K \succ  0$ such that

\begin{align}
(A-B{K})^TP_{K} + P_{K}(A-B{K}) + C^TC \nonumber \\
+ K^TRK + \gamma^{-2}P_{K}DD^TP_{K} \preceq 0.
\end{align}

This Lemma is a restatement of [54, 12].

§ PROOF OF CONVERGENCE OF ITERATIONS

In this appendix, we set up the proofs for the convergence of the inner and outer loops of the two algorithms.

§.§ Outer Loop Convergence

Let us first introduce the following results.

Consider Assumptions <ref> and <ref>. If $K_1 \in \mathcal{K}$, then for any $i\geq 1$

* $K_i \in \mathcal{K}$;

* $P_{K_1} \succeq \cdots P_{K_i} \cdots \succeq P^{\star}$, for $K_i \succeq K_1$;

* $\lim_{i \rightarrow \infty}||K_i - K^*||_F = 0$, $\lim_{i \rightarrow \infty}||P_{K_i} - P^*||_F = 0$.

We here establish the proof to convergence of $K_i$ per iteration $i$ to the optimal gain $K^\star$.

Statements 1) and 3) and the corresponding proof are given in  <cit.>. To make the paper self-contained, the method in [48] is adopted to prove the statements.  The first statement will be proven by induction. When $i = 1$, $K_1 \in \mathcal{K}$, and it satisfies 1). When $i>1$, assume $K_i \in \mathcal{K}$. Therefore, by the second condition of Lemma <ref>, $P_{K_i} \succ 0$ is the unique stabilizing solution to (<ref>).

Now consider the identities,

\begin{align}
(A-B K_i)^T = (A-B K_{i+1})^T P_{K_i} + (K_{i+1} - K_i)^T B^T P_{K_i} \nonumber \\
P_{K_i}(A-BK_i)=P_{K_i}(A - B K_{i+1}) + P_{K_i} B ( K_{i+1}-K_i).
\label{eq:riccati_identities}
\end{align}

It follows that equation (<ref>) can be rewritten as

\begin{align}
&=A_{K_{i+1}}^T P_{K_i} + P_{K_i}A_{K_{i+1}} + C^TC + \gamma^{-2}P_{K_i}DD^TP_{K_i} \nonumber \\
&\,\, + K_{i+1}^TRK_{i+1} + \left(K_{i+1}-K_i\right)^T R \left(K_{i+1}-K_i\right) = 0.
\label{eq:RiccatiOLEq}
\end{align}

As $\left(K_{i+1}-K_i\right)^T R \left(K_{i+1}-K_i\right) \succeq 0$, following the third condition of Lemma <ref>, we find that
$K_{i+1} \in \mathcal{K}$. A fortiori, we establish that $K_i \in \mc{K}$.   We proceed as follows for statement 2). Writing out (<ref>) for the $(i+1)$'th iteration, and subtracting the resulting equation from (<ref>), we find that

\begin{align}
\begin{split}
&A_{i+1}^T (P_{K_{i}} - P_{K_{i+1}}) + (P_{K_{i}} - P_{K_{i+1}})A_{i+1} \\
&\,\, + (K_{i} - K_{i+1})^TR(K_{i} - K_{i+1}) \\
&\,\, + \gamma^{-2} (P_{K_{i}} - P_{K_{i+1}})DD^T(P_{K_{i}} - P_{K_{i+1}}) = 0.
\end{split}
\label{eq: P_K_i - P_K_i-1}
\end{align}

Given statement 1), it follows that $K_{i+1} \in \mathcal{K}$. Furthermore, by the second condition of Lemma <ref>,  $A_{i+1}$ is Hurwitz. As $(K_{i} - K_{i+1})^TR(K_{i} - K_{i+1}) \succeq 0$ and $\gamma^{-2} (P_{K_{i}} - P_{K_{i+1}})DD^T(P_{K_{i}} - P_{K_{i+1}}) \succ 0$, by Lemma <ref>, we find that $P_{K_i} - P_{K_{i+1}} \succeq 0$. Because $K_{i+1} \in \mathcal{K}$, $A_{i+1}$ is Hurwitz by Lemma <ref>. Hence, $P_{K_i} \succeq P_{K_{i+1}}$ i.e., the sequence $\{P_{K_i}\}_{i=1}^{\infty}$ is decreasing, and lower bounded by $0$. A fortiori, $\{P_{K_i}\}_{i=1}^{\infty}$ convergences to $P_{K_\infty}$, which also satisfies (<ref>). Due to the uniqueness of the solution to (<ref>), $P_{K_\infty} = P^\star$.

§.§ Global Linear Convergence of the Outer (Control) Loop

Before proving Theorem <ref>, we first establish the following cost matrix' quadratic convergence result.

For any $i \in \mathbb{N}_+$, there exists an $a>0$, such that $\Tr(P_{K_{i+1}} - P^\star) \leq a \left[\Tr(P_{K_{i}} - P^\star)\right]^2$.

For the $(i+1)'th$ iteration, (<ref>) can be rewritten as

\begin{align}
&A_{i+1}^T P_{K_{i+1}} + P_{K_{i+1}}A_{i+1} + C^TC + K_{i+1}^TRK_{i+1} \nonumber \\
&\,\, - \gamma^{-2} P_{K_{i+1}}DD^T P_{K_{i+1}} = 0.
\label{eq:iteration_i+1}
\end{align}

Also, (<ref>) can be rewritten as

\begin{align}
& A_{i+1}^T P^\star + P^\star A_{i+1} + C^TC - K^{\star T}RK^{\star} \nonumber \\
&\,\, + K_{i+1}^TRK^\star + K^{\star T} R K_{i+1} - \gamma^{-2} P_{K_{i+1}}DD^TP^{\star} \nonumber\\
&\,\, - \gamma^{-2} P^{\star}DD^TP_{K_{i+1}} + \gamma^{-2} P^{\star}DD^TP^{\star} = 0.
\label{eq:ARERewritten}
\end{align}

Subtracting (<ref>) from (<ref>), and completing squares, we have

\begin{align}
\begin{split}
&A_{i+1}^T (P_{K_{i+1}} - P^\star) + (P_{K_{i+1}} - P^\star)A_{i+1} \\
&\,\, + (K_{i+1} - K^\star)^TR(K_{i+1} - K^\star) \\
&\,\, - \gamma^{-2} (P_{K_{i+1}} - P^\star)DD^T(P_{K_{i+1}} - P^\star)  = 0.
\end{split}
\label{eq: P_K_i+1 - P_*}
\end{align}

From Theorem <ref> and Lemma <ref>, we see that $A_{i+1}$ is Hurwitz. From Lemma <ref>, it follows that

\begin{align}
\begin{split}
&P_{K_{i+1}} - P^\star \\
&\preceq  \int_{0}^{\infty}e^{A_{i+1}^Tt} (P_{K_{i}} - P^\star)BR^{-1}B^T(P_{K_{i}} - P^\star) e^{A_{i+1}t}  dt.
\end{split}
\end{align}

Using the cyclic property of matrix trace,

\begin{align}
\begin{split}
&\Tr(P_{K_{i+1}} - P^\star) \leq \\
&\Tr\left[(P_{K_{i}} - P^\star)BR^{-1}B^T(P_{K_{i}} - P^\star)\int_{0}^{\infty}e^{A_{i+1}t} e^{A_{i+1}^Tt}  dt \right].
\end{split}
\end{align}

Let us define $M_{i+1} :=  \int_{0}^{\infty}e^{A_{i+1}t} e^{A_{i+1}^Tt}dt$. Because $A_{i+1}$ is Hurwitz, $M_{i+1}$ satisfies

\begin{align}
A_{i+1} M_{i+1} + M_{i+1}A_{i+1}^T + I_n = 0.
\end{align}

From Theorem <ref>, it follows that as $i \to \infty$, $P_{K_i}$ converges $P^\star$ and ${A_i}$ converges to $A^\star$. Consequently, $M_i$ converges to $M^\star \in \mathbb{S}^n$, which is the solution to

\begin{align}
A^\star M^\star + M^\star (A^\star)^T + I_n = 0.
\label{eq:Mopt}
\end{align}

Thus, $\lim_{i \to \infty}\norm{M_i} = \norm{M^\star}$ and $\bar{m}:=\sup_{i \in \mathbb{N}_+}\norm{M_i} < \infty$. As a consequence,

\begin{align}
\begin{split}
\Tr(P_{K_{i+1}} - P^\star) &\leq \Tr\left[(P_{K_{i}} - P^\star)BR^{-1}B^T(P_{K_{i}} - P^\star){M_i} \right] \\
%\Tr(P_{K_{i+1}} - P^\star) &\leq \bar{m} \norm{BR^{-1}B^T} \Tr\left[(P_{K_{i}} - P^\star)^2 \right]\\
\Tr(P_{K_{i+1}} - P^\star) &\leq  \bar{m}\norm{BR^{-1}B^T} \left[\Tr(P_{K_{i}} - P^\star)\right]^2.
\end{split}
\end{align}

Setting $a:=\bar{m}\norm{BR^{-1}B^T}$, we see that the outer-loop's cost matrix convergences in a quadratic manner in the vicinity of $P^\star$.

For any $i \in \mathbb{N}_+$, there exists a scalar $b>0$, such that $\Tr(P_{K_{i}} - P^\star) \leq b \Tr(P_{K_{i-1}} - P_{K_{i}})$. In addition, there exists $b'>0$, such that $\norm{K_{i+1} - K^\star} \leq b' \norm{K_{i} - K_{i+1}}$.

From Proposition <ref>,
for any $\epsilon > 0$, there exists $\hat{i} \in \mathbb{N}_+$, such that if $i \geq \hat{i}$,

\begin{align}
\Tr(P_{K_{i-1}} - P^\star) \leq \frac{1}{a(1+\epsilon)},
\end{align}

where $a>0$ is as given in Proposition <ref>. We have by Proposition <ref> for any $i \geq \hat{i}$ that

\begin{align}
&\frac{\Tr(P_{K_{i-1}} - P_{K_{i}})}{\Tr(P_{K_{i}} - P^\star)} = \frac{\Tr(P_{K_{i-1}} - P^\star)}{\Tr(P_{K_{i}} - P^\star)} - 1 \\
&\geq \frac{1}{a\Tr(P_{K_{i-1}} - P^\star)} -1 \geq \epsilon.
\end{align}

Now, for $i < \hat{i}$, from Theorem <ref> we have

\begin{align}
0 < \Tr(P_{K_{i}} - P^\star) \leq \Tr(P_{K_{1}} - P^\star).
\end{align}

Suppose that we let $b_1 = \min_{i<\hat{i}} \Tr(P_{K_{i-1}} - P_{K_{i}})$. Then, by Theorem <ref>, $b_1 > 0$ so that

\begin{align}
\frac{\Tr(P_{K_{i-1}} - P_{K_{i}})}{\Tr(P_{K_{i}} - P^\star)} \geq \frac{b_1}{\Tr(P_{K_{1}} - P^\star)}.
\end{align}

\begin{align}
b := \max \left[\frac{1}{\epsilon}, \frac{\Tr(P_{K_{1}} - P^\star)}{b_1}\right].
\end{align}

We see that $\Tr(P_{K_{i}} - P^\star) \leq b \Tr(P_{K_{i-1}} - P_{K_{i}})$ for all $i \in \mathbb{N}_+$. From Lemma <ref>, we have

\begin{align}
\norm{K_{i+1} - K^\star} &\leq \norm{R^{-1}B^T} \norm{P_{K_{i}} - P^\star} \nonumber \\
&\leq \norm{R^{-1}B^T}\Tr(P_{K_{i}} - P^\star) \nonumber \\
&\leq  b\norm{R^{-1}B^T} \Tr(P_{K_{i-1}} - P_{K_{i}}) \nonumber \\
&\leq bn\norm{R^{-1}B^T} \norm{P_{K_{i-1}} - P_{K_{i}}}.
\end{align}

\begin{align}
\begin{split}
&\norm{K_i - K_{i+1}} = \norm{R^{-1}B^T(P_{K_i} - P_{K_{i-1}})} \\
&\geq {\eigmin}(R^{-1}B^T) \norm{P_{K_i} - P_{K_{i-1}}}.
\end{split}
\end{align}

As $B$ is full column rank, $\eigmin(R^{-1}B^T) > 0$.  Setting $b' = {bn\norm{R^{-1}B^T}}/{\eigmin(R^{-1}B^T)}$, we establish the second statement.

For any $i \in \mathbb{N}_+$, we have

\begin{align}
(K_i - K^\star)^T &R (K_i - K^\star) \succeq \nonumber \\
& \qquad \gamma^{-2}(P_{K_i} - P^\star)DD^T(P_{K_i} - P^\star).
\label{eq:prop_gain_bound}
\end{align}

Similar to (<ref>), we have

\begin{align}
&A_{i}^T (P_{K_{i}} - P^\star) + (P_{K_{i}} - P^\star)A_{i}  + (K_{i} - K^\star)^TR(K_{i} - K^\star) \nonumber\\
&\qquad - \gamma^{-2} (P_{K_{i}} - P^\star)DD^T(P_{K_{i}} - P^\star)  = 0.
\label{eq:P_K_i-P_*}
\end{align}

Using Theorem <ref>, $A_i$ is Hurwitz, and $P_{K_i} - P^\star \succeq 0$. Therefore, by Lemma <ref>, we arrive at the required inequality.

Let $E_{K_i} = (K_i - K_{i+1})^T R (K_i - K_{i+1})$. For the sequences $\{P_{K_i} \}_{i=1}^{\infty}$ and $\{K_i \}_{i=1}^{\infty}$ obtained by the control loop update, the following inequality holds

\begin{align}
&\Tr(P_{K_i} - P_*) \leq c\norm{E_{K_i}}, \textbf{ where, }\nonumber \\
c &= \left[\eigmin(R)\right]^{-1}(2+2b'){\norm{R}}\Tr\left(\int_{0}^{\infty} e^{A^\star t} e^{A^{\star T}t} dt\right).
\label{eq:bound_on_E_K}
\end{align}

For the $i$th iteration, (<ref>) can be rewritten as

\begin{align}
\begin{split}
&A^{\star T}P_{K_i} + P_{K_i}A^\star + (K^\star - K_i)^T R K_{i+1}  \\
&+ K_{i+1}^TR(K^\star - K_i) +\gamma^{-2}(P_{K_i} - P^\star)DD^TP_{K_i}  \\
&+ \gamma^{-2}P_{K_i}DD^T(P_{K_i} - P^\star) + C^TC + K_i^T R K_i \\
&- \gamma^{-2}P_{K_i}DD^T P_{K_i} = 0,
\end{split}
\label{eq:AOptPi}
\end{align}

and (<ref>) can be rewritten as

\begin{align}
&A^{\star T} P^\star + P^\star A^\star + C^TC + K^{\star T}RK^\star \nonumber \\
&\qquad - \gamma^{-2}P^\star DD^T P^\star = 0.
\label{eq:AoptPopt}
\end{align}

Subtracting (<ref>) from (<ref>) and completing squares, we have

\begin{align}
\begin{split}
&A^{\star T} (P_{K_i} - P^\star) + (P_{K_i} - P^\star)A^\star + E_{K_{i}} \\
&+ \gamma^{-2}(P_{K_i} - P^\star)DD^T(P_{K_i} - P^\star) \\
& - (K_{i+1} - K^\star)^TR(K_{i+1} - K^\star) = 0.
\end{split}
\label{eq:AoptPDiff}
\end{align}

Using Proposition <ref>, and completing the squares, (<ref>) becomes

\begin{align}
\begin{split}
&A^{\star T} (P_{K_i} - P^\star) + (P_{K_i} - P^\star)A^\star + 2E_{K_i} \\
&\qquad + (K_{i} - K_{i+1})^T R (K_{i+1} - K^\star) \\
&\qquad + (K_{i+1} - K^\star)^T R (K_{i} - K_{i+1})  \succeq 0.
\end{split}
\end{align}

Now, using Lemma <ref>, we have

\begin{align}
&\Tr(P_{K_i} - P^\star) \leq \Tr\left\{\int_{0}^{\infty} e^{A^{\star T}t} \left[ 2E_{K_i} \right. \right. \nonumber\\
&  + (K_{i} - K_{i+1})^T R (K_{i+1} - K^\star )  \nonumber\\
&  \left. + (K_{i+1} - K^\star)^T \left.  R (K_{i} - K_{i+1})  \right] e^{A^\star t}  dt \right\}  \\
&\leq \Tr \left\{\left[ 2E_{K_i} + (K_{i} - K_{i+1})^T R (K_{i+1} - K^\star) \right. \right. \nonumber\\
& \, \left. \left.+ (K_{i+1} - K^\star )^T R (K_{i} - K_{i+1})\right]  \int_{0}^{\infty} e^{A^\star t}e^{A^{\star T}t}dt \right\} \\
&\leq \left[2\norm{E_{K_i}} + 2\norm{K_{i} - K_{i+1}} \norm{R} \norm{K_{i+1} - K_*} \right] \nonumber \\
&\qquad \Tr(\int_{0}^{\infty} e^{A^*t}e^{(A^{*T}) t}dt)  \\
&\leq \left[2\norm{E_{K_i}} + 2b'\norm{K_{i} - K_{i+1}} \norm{R} \norm{K_{i} - K_{i+1}}  \right] \nonumber \\
&\qquad \Tr(\int_{0}^{\infty} e^{A^\star t}e^{A^{\star T} t}dt),
\end{align}

where the last expression is as a result of Proposition <ref>. And by Lemma <ref>, we arrive at the required inequality (<ref>), i.e.

\begin{align}
\begin{split}
&\Tr(P_{K_i} - P^\star) \leq \\
&\underbrace{(2+2b')\frac{\norm{R}}{{\eigmin}(R)}}_{:=c}\Tr(\int_{0}^{\infty} e^{A^\star t}e^{A^{\star T}t}dt)\norm{E_{K_i}}.
\end{split}
\end{align}

Given that $E \in \mathbb{S}^n$ is positive semi-definite, and $W \in \mathbb{R}^{n \times n}$ is Hurwitz. Let $F := \int_{0}^{\infty} e^{W^Tt}Ee^{Wt} dt$, and  $d(W) = {\log(5/4)}/{\norm{W}}$. Then, $\norm{F} \geq \frac{1}{2} d(W) \norm{E}.$

A Taylor expansion of $e^{Wt}$ yields

\begin{align}
e^{Wt} = I_n + \underbrace{\left[{\sum_{k=1}^{\infty} (W t)^k}/{k!}\right]}_{:=S(t)},
\end{align}

so that $\norm{S(t)} \leq e^{\norm{W}t} - 1$. For an $x_0 \neq 0$ satisfying $x_0^T E x_0 = \norm{E}\norm{x_0}^2$, we have,

\begin{align}
\begin{split}
&x_0^T F x_0 \geq \int_{0}^{d(W)}x_0^T e^{W^Tt}Ee^{Wt}x_0 dt \\
&= \int_{0}^{d(W)}x_0^T (I_n + S(t))E(I_n + S(t))x_0 dt \\
&\geq \int_{0}^{d(W)}\norm{E}\norm{x_0}^2 - 2 \norm{S(t)}\norm{E}\norm{x_0}^2 dt \\
% &\geq \int_{0}^{d(W)}\norm{E}\norm{x_0}^2 - 2 (e^{(\norm{W}d(W))}-1)\norm{E}\norm{x_0}^2 dt \\
% &= \int_{0}^{d(W)} (3-2e^{(\norm{W}d(W))})\norm{E}\norm{x_0}^2 dt \\
% &= \int_{0}^{d(W)} \frac{1}{2}\norm{E}\norm{x_0}^2 dt \\
&\geq \frac{1}{2}d(W)\norm{E}\norm{x_0}^2.
\end{split}
\label{eq:bound_on_E}
\end{align}

From (<ref>), we see that $ \norm{F} \geq \frac{1}{2} d(W) \norm{E}$. This proves the Lemma.

The proof of Theorem <ref> is now straightforward. Let us write

\begin{align*}
\begin{split}
(P_{K_i} - P_{K_{i+1}} ) \succeq \int_{0}^{\infty} e^{A^T_{i+1}t}  E_{K_i} e^{A_{i+1}t}dt =: F_{K_{i}},
\end{split}
\end{align*}

following (<ref>). Recall that $P_{K_{i}} \succeq P_{K_{i+1}}$ and $A_{i+1}$ is Hurwitz. Thus, because $P_{K_1} \succeq P_{K_i}$, we have

\begin{align}
\norm{A_{i+1}} \leq \norm{A} + (\norm{BR^{-1}B^T}+\gamma^{-2}\norm{DD^T})\norm{P_{K_1}}.
\end{align}

Let us set

\begin{align}
d = \frac{\log(5/4)}{\norm{A} + (\norm{BR^{-1}B^T}+\gamma^{-2}\norm{DD^T})\norm{P_{K_1}}}.
\end{align}

It follows that (Lemma <ref>)

$\norm{ F_{K_{i}}} \geq \frac{1}{2} d \norm{E_{K_i}}$,

so that by Lemma <ref>, we can write

\begin{align}
\Tr(P_{K_{i+1}} &- P^*) \leq \Tr(P_{K_{i}} - P^*) - \frac{1}{2}d\norm{E_{K_i}} \\
&\leq \Tr(P_{K_{i}} - P^*) - \frac{d}{2c}\Tr(P_{K_i} - P^*) \\
&\leq \underbrace{\left(1-\frac{d}{2c}\right)}_{:=\alpha}\Tr(P_{K_{i}} - P^*)
\label{eq:trace_bound}
\end{align}

Equation (<ref>) is in fact the required inequality for (<ref>), that is, $\Tr(P_{K_{i+1}} - P^*) \le \alpha \Tr(P_{K_{i}} - P^*)$.

Next, we establish the proof of the convergence of the inner loop of the iteration.

§.§ Proof of Inner Loop Convergence (Theorem <ref>)

To establish this theorem, we first introduce a few results.

Assume $L_{K_i}^1 = 0$, then for any $i,j \in \mathbb{N}_+$, the following holds

* $A_{K_i}^j$ is Hurwitz,

* $P_{K_i} \succeq \cdots \succeq P_{K_i}^{j+1} \succeq P_{K_i}^{j} \succeq \cdots \succeq P_{K_i}^{1}$,

* $\lim_{j \rightarrow \infty} \norm{P_{K_i}^j - P_{K_i}}_F = 0$, where $P_{K_i}$ is the solution to (<ref>).

The first statement will be proven by induction. By Theorem <ref>, $K_i \in \mathcal{K}$. According to Lemma <ref>, there exists a unique stabilizing solution $P_{K_i} \succ 0$ to (<ref>) and $A_i$ is Hurwitz. When $j=1$, $L_{K_i}^1 = 0$, then, (<ref>) can be rewritten as

\begin{align}
A_{K_i}^TP_{K_i}^1 + P_{K_i}^1A_{K_i} + C^TC + K_i^TRK_i = 0.
\end{align}

Since $A_{K_i}$ is Hurwitz and $(C,A)$ is observable, by Lemma <ref>, $A_i^1 = A_{K_i}$ is Hurwitz, and $P_{K_i}^1 \succ 0$.
Subtracting (<ref>) from (<ref>) yields

\begin{align}
\begin{split}
&(A_{K_i}^j)^T(P_{K_i} - P_{K_i}^j) +(P_{K_i} - P_{K_i}^j)A_{K_i}^j \\
& \qquad + \gamma^{2} (L_{K_i}^j - L_{K_i})^T(L_{K_i}^j - L_{K_i})= 0.
\end{split}
\label{eq: P_K_i - P_K_i^j}
\end{align}

When $j=1$, since $A_{K_i}^1$ is Hurwitz and $\gamma^{2} (L_{K_i}^j - L_{K_i})^T(L_{K_i}^j - L_{K_i}) \succeq 0$, according to Lemma <ref>, (<ref>) results in $P_{K_i} \succeq P_{K_i}^1 \succ 0$. We can rewrite (<ref>) as

\begin{align}
A_i^T P_{K_i} + P_{K_i} A_i + Q_{K_i} - \gamma^{-2}P_{K_i}DD^TP_{K_i} = 0.
\end{align}

Since $A_i$ is Hurwitz and $P_{K_i} \succ 0$, by Lemma <ref>, the following inequality holds

\begin{align}
Q_{K_i} - \gamma^{-2}P_{K_i}DD^TP_{K_i} \succ 0.
\label{eq: Q>PDDP}
\end{align}

Assume $A_i^j$ is Hurwitz, and from (<ref>), we have $P_{K_i} \succeq P_{K_i}^j$. Following (<ref>), we have

\begin{align}
\begin{split}
&(A_i^{j+1})^T P_{K_i} + P_{K_i} A_i^{j+1} + Q_{K_i} - \gamma^{2}P_{K_i}^{j}DD^TP_{K_i}^{j} \\
&\qquad + \gamma^2(L_{K_i}^{j+1} - L_{K_i})^T(L_{K_i}^{j+1} - L_{K_i}) = 0.
\end{split}
\label{eq:Aj+1P}
\end{align}

By (<ref>) and $P_{K_i} \succeq P_{K_i}^j$,

\begin{align}
Q_{K_i} - \gamma^{2}P_{K_i}^{j}DD^TP_{K_i}^{j} \succ 0.
\end{align}

Hence, by Lemma <ref> and (<ref>),  $A_i^{j+1}$ is Hurwitz. As a consequence, the proof of statement 1) is completed.

Rewriting (<ref>) for the $(j+1)'th$ iteration and subtracting the resulting equation from (<ref>) results in

\begin{align}
\begin{split}
&(A_{K_i}^{j+1})^T (P_{K_i}^{j+1} - P_{K_i}^{j}) + (P_{K_i}^{j+1} - P_{K_i}^{j})A_{K_i}^{j+1} \\
& \qquad + \gamma^2(L_{K_i}^{j+1} - L_{K_i}^{j})^T(L_{K_i}^{j+1}-L_{K_i}^{j}) = 0.
\end{split}
\label{eq: P_K^j+1 - P_K^j}
\end{align}

As $ \gamma^2(L_{K_i}^{j+1} - L_{K_i}^{j})^T(L_{K_i}^{j+1}-L_{K_i}^{j}) \succeq 0$ and $A_{K_i}^{j+1}$ is Hurwitz, by (<ref>) and Lemma <ref>, we have $P_{K_i}^{j+1} \succeq P_{K_i}^j$. As a result, the proof of statement 2) is completed.

Statement 2) implies that the sequence $\{ P_{K_i}^{j} \}_{j=1}^{\infty}$ is monotonically increasing and upper-bounded by $P_{K_i}$. Hence, $P_{K_i}^{\infty}$ exists, and is the solution to (<ref>). Due to the uniqueness of the solution, we have $P_{K_i}^{\infty} = P_{K_i}$, which proves the statement 3).

As shown in [48], policy iteration has a quadratic convergence rate in the vicinity of the solution $P_{K_i}$. Next, we will show that the inner loop iteration has a global linear convergence rate.

Suppose that

\begin{align}
E_K^j := \gamma^{-2} (D^T P_K^{j}-\gamma^2L_K^j)^T (D^T P_K^{j}-\gamma^2L_K^j),
\end{align}

and $K \in \mathcal{K}$. Then, for the sequences $\{P_K^j \}_{j=0}^{\infty}$ and  $\{L_K^j \}_{j=1}^{\infty}$ obtained by the inner-loop iteration (<ref>), the following inequality holds

\begin{align}
&\Tr(P_K - P_K^j) \leq \norm{E_K^j}c(K),
\label{eq: trace inner}
\end{align}

\begin{align}
c(K) = \Tr\left(\int_{0}^{\infty} e^{(A_K + DL_K)t} e^{(A_K + DL_K)^Tt} dt\right).
\end{align}

Subtracting (<ref>) from (<ref>) and completing squares, we have

\begin{align}
&(A_K + DL_K)^T(P_K - P_K^j) + (P_K - P_K^j) (A_K + DL_K) \nonumber\\
&\qquad - \gamma^{-2}(D^TP_K - D^T P_K^j)^T(D^TP_K - D^T P_K^j) \\
&\qquad + \gamma^{-2}(D^T P_K^j - \gamma^{2}L_K^j)^T(D^T P_K^j - \gamma^{2}L_K^j) = 0. \nonumber
\label{eq: P_K - P_K^j}
\end{align}

Because $K \in \mathcal{K}$, $A_K + DL_K$ is Hurwitz by Lemma <ref>. Using Lemma <ref>, we have

\begin{align*}
P_K - P_K^j \preceq \int_{0}^{\infty} e^{(A_K + DL_K)^Tt} E_K^j e^{(A_K + DL_K)t} dt.
\end{align*}

By Lemma <ref> and the cyclic property of the trace, we have

\begin{align}
\begin{split}
\Tr(P_K &- P_K^j) \leq \\
&\norm{E_K^j} \underbrace{\Tr\left(\int_{0}^{\infty} e^{(A_K + DL_K)t} e^{(A_K + DL_K)^Tt} dt\right)}_{:=c(K)}.
\end{split}
\end{align}

Therefore, $\Tr(P_K - P_K^j) \leq \norm{E_K^j}c(K)$ holds.

The proof of the theorem given in Theorem <ref> is now given.

By Theorem <ref>, $A_{K}^{j+1}$ is Hurwitz. By Lemma <ref> and (<ref>), we have

\begin{align}
P_{K}^{j+1} - P_{K}^{j} = \underbrace{\int_{0}^{\infty} e^{(A_K^{j+1})^Tt}E_K^j e^{A_K^{j+1}t} dt}_{:=F_K^j}.
\end{align}

\begin{align}
\begin{split}
P_K - P_{K}^{j+1} %&=  P_K - P_{K}^{j} -  \int_{0}^{\infty} e^{(A_K^{j+1})^Tt}E_K^je^{A_K^{j+1}t} dt \\
&= P_K - P_{K}^{j} -  F_K^j.
\end{split}
\label{eq:P_k_inner_loop}
\end{align}

By Theorem <ref>, $P_K \succeq P_K^j$ so that

\begin{align}
\norm{A_K^{j+1}} &= \norm{A-BK + \gamma^{-2}DD^TP_{K}^{j}}  \nonumber\\
&\leq \norm{A-BK} + \gamma^{-2}\norm{DD^T}\norm{P_{K}}.
\end{align}

\begin{align}
d(K) := \frac{\log(5/4)}{\norm{A-BK} + \gamma^{-2}\norm{DD^T}\norm{P_{K}}}.
\label{eq: d(K)}
\end{align}

Taking the trace of both sides of (<ref>), we find that

\begin{align}
\Tr(P_K - P_{K}^{j+1}) &=  \Tr(P_K - P_{K}^{j}) - \Tr(F_K^j) \\
&\leq \Tr(P_K - P_{K}^{j}) - \norm{F_K^j}
\label{eq:normF}\\
&\leq \Tr(P_K - P_{K}^{j}) - \frac{1}{2}d(K)\norm{E_K^j} \label{eq:normFnormE},
\end{align}

where we have used Lemma <ref> to arrive at the inequality in (<ref>), and we have used Lemma <ref> to arrive at the inequality in (<ref>).
Furthermore, using Lemma <ref>, we find that

\begin{align}
\Tr(P_K - P_{K}^{j+1}) &\leq \Tr(P_K - P_{K}^{j})-  \dfrac{1}{2}\dfrac{d(K)}{c(K)}\Tr(P_K - P_{K}^{j}) \nonumber \\
&\leq    \underbrace{\left(1 - \dfrac{1}{2}\dfrac{d(K)}{c(K)}\right)}_{:=\beta(K)}\Tr(P_K - P_{K}^{j}).
\label{eq: linear conv proof}
\end{align}

Equation (<ref>)
is equivalent to (<ref>), and gives us the required result.

§.§ Proof: Uniform Convergence

We now establish the proof of Theorem <ref>.

Let $M_i := \int_{0}^{\infty} e^{A_{i}^Tt} e^{A_{i}t} dt$. Following Theorem <ref> and Lemma <ref>, we see that $A_{i}$ is Hurwitz. Hence, by Lemma <ref> for any $i \in \mathbb{N}_+$,

\begin{align}
\begin{split}
&A_i^TM_i + M_i A_i + I_n = 0.
\end{split}
\label{eq:Mi}
\end{align}

From Theorem <ref>, we get that as $i \to \infty$, $P_{K_i}$ converges $P^\star$ and ${A_i}$ converges to $A^\star$. Consequently, $M_i$ converges to $M^\star$, which is the solution of (<ref>). Consequently, $\bar{c} := \sup_{i \in \mathbb{R}_+}\Tr(M_i) < \infty$.

Hence, we have

\begin{align}
c(K_i) = \Tr(M_i) \leq \bar{c},
\end{align}

where $c(K_i)$ is defined in (<ref>). Recall from Theorem <ref> that

\begin{align}
d(K_i) \geq \frac{\log(5/4)}{\norm{A} + ( \norm{BR^{-1}B^T} + \gamma^{-2}\norm{DD^T})\norm{P_{K_1}}} =: \underline{d}.
\end{align}

\begin{align}
\beta(K_i) = 1 - \frac{d(K_i)}{2c(K_i)} \leq 1 - \frac{ \underline{d}}{2\bar{c}} =:\bar{\beta}.
\end{align}

Therefore, by Theorem <ref>, for any $i\in \mathbb{N}_+$, we have

\begin{align}
\norm{P_{K_{i}}^j - P_{K_i}}_F \leq \bar{\beta}^{j-1} \Tr(P_{K_i}) \leq \bar{\beta}^{j-1} \Tr(P_{K_1}).
\end{align}

We see that for any $i \in \mathbb{N}_+$ and $\epsilon >0$, there exists $\bar{j}>0$, such that if $j \geq \bar{j}$, $\norm{P_{K_i}^j - P_{K_i}}_F \leq \epsilon$. A fortiori,  the iteration in fact converges uniformly to an $\epsilon > 0$.

[1]
G. Stein and M. Athans, “The LQG/LTR procedure for multivariable feedback
control design,” IEEE Transactions on Automatic Control, vol. 32,
no. 2, pp. 105–114, 1987.

[2]
M. Athans, “The Role and Use of The Stochastic Linear-Quadratic-Gaussian
Problem in Control System Design,” IEEE Transactions on Automatic
Control, vol. 16, no. 6, pp. 529–552, 1971.

[3]
N. Papanikolopoulos, P. Khosla, and T. Kanade, “Visual tracking of a moving
target by a camera mounted on a robot: a combination of control and vision,”
IEEE Transactions on Robotics and Automation, vol. 9, no. 1,
pp. 14–35, 1993.

[4]
W. Xu, J. Pan, J. Wei, and J. M. Dolan, “Motion planning under uncertainty for
on-road autonomous driving,” in 2014 IEEE International Conference on
Robotics and Automation (ICRA), pp. 2507–2512, 2014.

[5]
E. Todorov and M. I. Jordan, “Optimal feedback control as a theory of motor
coordination,” Nature Neuroscience, vol. 5, no. 11, p. 1226–1235,

[6]
J. Doyle, “Guaranteed margins for LQG regulators,” IEEE Transactions
on Automatic Control, vol. 23, no. 4, pp. 756–757, 1978.

[7]
D. Jacobson, “Optimal stochastic linear systems with exponential performance
criteria and their relation to deterministic differential games,” IEEE
Transactions on Automatic Control, vol. 18, no. 2, pp. 124–131, 1973.

[8]
T. E. Duncan, “Linear-Exponential-Quadratic Gaussian control,” IEEE
Transactions on Automatic Control, vol. 58, no. 11, pp. 2910–2911, 2013.

[9]
P. Khargonekar, I. Petersen, and M. Rotea, “$\mathcal{H}_\infty$ optimal
control with state-feedback,” IEEE Transactions on Automatic Control,
vol. 33, no. 8, pp. 786–788, 1988.

[10]
D. Bernstein and W. Haddad, “LQG control with an $\mathcal{H}_\infty$
performance bound: a Riccati equation approach,” IEEE Transactions on
Automatic Control, vol. 34, no. 3, pp. 293–305, 1989.

[11]
D. Mustafa, “Relations between maximum-entropy/$\mathcal{H}_\infty$ control
and combined $\mathcal{H}_\infty$/LQG control,” Systems and Control
Letters, vol. 12, no. 3, pp. 193–203, 1989.

[12]
T. Başar, $H_\infty$-Optimal Control and Related Minimax Design Problems:
A Dynamic Game Approach.
Springer, 2008.

[13]
A. Lanzon, Y. Feng, B. D. O. Anderson, and M. Rotkowitz, “Computing the
positive stabilizing solution to algebraic riccati equations with an
indefinite quadratic term via a recursive method,” IEEE Transactions on
Automatic Control, vol. 53, no. 10, pp. 2280–2291, 2008.

[14]
K. Zhang, B. Hu, and T. Başar, “Policy Optimization for
$\mathcal{H}_2$ Linear Control with $\mathcal{H}_\infty$ Robustness
Guarantee: Implicit Regularization and Global Convergence,” arXiv
e-prints, p. arXiv:1910.09496, Oct. 2019.

[15]
H.-N. Wu and B. Luo, “Simultaneous policy update algorithm for learning the
solution of linear continuous-time $\mathcal{H}_\infty$ state feedback
control,” Information Sciences, vol. 222, p. 472–485, 02 2013.

[16]
J. Bu, L. J. Ratliff, and M. Mesbahi, “Global Convergence of Policy
Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games,” arXiv e-prints, Nov. 2019.

[17]
Z.-P. Jiang, T. Bian, and W. Gao, “Learning-based control: A tutorial and some
recent results,” Foundations and Trends® in Systems and Control,
vol. 8, no. 3, pp. 176–284, 2020.

[18]
Y. Jiang and Z. P. Jiang, Robust Adaptive Dynamic Programming.
NJ, USA: Wiley-IEEE Press, 2017.

[19]
Y. Jiang and Z. P. Jiang, “Computational adaptive optimal control for
continuous-time linear systems with completely unknown dynamics,” Automatica, vol. 48, no. 10, pp. 2699–2704, 2012.

[20]
T. Bian and Z. P. Jiang, “Value iteration and adaptive dynamic programming for
data-driven adaptive optimal control design,” Automatica, vol. 71,
pp. 348–360, 2016.

[21]
T. Bian and Z. P. Jiang, “Reinforcement learning and adaptive optimal
control for continuous-time nonlinear systems: a value iteration approach,”
IEEE Transactions on Neural Networks and Learning Systems, in press,

[22]
B. Pang and Z. P. Jiang, “Adaptive optimal control of linear periodic
systems: an off-policy value iteration approach,” IEEE Transactions on
Automatic Control, vol. 66, no. 2, pp. 888–894, 2021.

[23]
W. Gao and Z. Jiang, “Adaptive dynamic programming and adaptive optimal
output regulation of linear systems,” IEEE Transactions on Automatic
Control, vol. 61, no. 12, pp. 4164–4169, 2016.

[24]
F. L. Lewis and D. Liu, Reinforcement Learning and Approximate Dynamic
Programming for Feedback Control.
NJ, USA: Wiley-IEEE Press, 2013.

[25]
B. Pang, T. Bian, and Z.-P. Jiang, “Robust policy iteration for
continuous-time linear quadratic regulation,” IEEE Transactions on
Automatic Control, vol. 67, no. 1, pp. 504–511, 2022.

[26]
J. Bu, A. Mesbahi, and M. Mesbahi, “Policy Gradient-based Algorithms
for Continuous-time Linear Quadratic Control,” arXiv e-prints,
p. arXiv:2006.09178, June 2020.

[27]
H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanović, “Convergence
and sample complexity of gradient methods for the model-free
linear–quadratic regulator problem,” IEEE Transactions on Automatic
Control, vol. 67, no. 5, pp. 2435–2450, 2022.

[28]
G. Qu, C. Yu, S. Low, and A. Wierman, “Combining Model-Based and
Model-Free Methods for Nonlinear Control: A Provably Convergent Policy
Gradient Approach,” arXiv e-prints, June 2020.

[29]
B. Luo, H.-N. Wu, and T. Huang, “Off-policy reinforcement learning for $
h_\infty $ control design,” IEEE Transactions on Cybernetics, vol. 45,
no. 1, pp. 65–76, 2015.

[30]
H. Zhang, Q. Wei, and D. Liu, “An iterative adaptive dynamic programming
method for solving a class of nonlinear zero-sum differential games,” Automatica, vol. 47, no. 1, pp. 207–214, 2011.

[31]
H. Li, D. Liu, and D. Wang, “Integral reinforcement learning for linear
continuous-time zero-sum games with completely unknown dynamics,” IEEE
Transactions on Automation Science and Engineering, vol. 11, no. 3,
pp. 706–714, 2014.

[32]
D. Vrabie and F. Lewis, “Adaptive dynamic programming for online solution of a
zero-sum differential game,” J. Contr. Theory Appl., vol. 9,
pp. 353–360, 08 2011.

[33]
Y. Li, Y. Tang, R. Zhang, and N. Li, “Distributed reinforcement learning for
decentralized linear quadratic control: A derivative-free policy optimization
approach,” IEEE Transactions on Automatic Control, pp. 1–1, 2021.

[34]
H. Mania, A. Guy, and B. Recht, “Simple random search of static linear
policies is competitive for reinforcement learning,” in Advances in
Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle,
K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), vol. 31, Curran
Associates, Inc., 2018.

[35]
K. Zhang, Z. Yang, and T. Basar, “Policy optimization provably converges to
nash equilibria in zero-sum linear quadratic games,” in Advances in
Neural Information Processing Systems (H. Wallach, H. Larochelle,
A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett,
eds.), vol. 32, Curran Associates, Inc., 2019.

[36]
M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy
gradient methods for the linear quadratic regulator,” in Proceedings of
the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, pp. 1467–1476, PMLR, 10–15 Jul

[37]
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.
MIT press, 2018.

[38]
B. Luo, Y. Yang, and D. Liu, “Policy iteration q-learning for data-based
two-player zero-sum game of linear discrete-time systems,” IEEE
Transactions on Cybernetics, vol. 51, no. 7, pp. 3630–3640, 2021.

[39]
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial
reinforcement learning,” in Proceedings of the 34th International
Conference on Machine Learning (D. Precup and Y. W. Teh, eds.), vol. 70 of
Proceedings of Machine Learning Research, pp. 2817–2826, PMLR, 06–11
Aug 2017.

[40]
J. Morimoto and K. Doya, “Robust reinforcement learning,” Neural
Computation, vol. 17, no. 2, pp. 335–359, 2005.

[41]
M. Rotea and P. Khargonekar, “Mixed $H2/H_\infty$ Control: A Convex
Optimization Approach,” IEEE Trans. Automat. Control, vol. 36,
no. 5, pp. 824–837, 1991.

[42]
G. Zames, “Feedback and optimal sensitivity: Model reference transformations,
multiplicative seminorms, and approximate inverses,” IEEE Transactions
on Automatic Control, vol. 26, no. 2, pp. 301–320, 1981.

[43]
J. Doyle, K. Glover, P. Khargonekar, and B. Francis, “State-space solutions to
standard $\mathcal{H}_2$ and $\mathcal{H}_\infty$ control problems,” IEEE Transactions on Automatic Control, vol. 34, no. 8, pp. 831–847, 1989.

[44]
P. Whittle, “Risk-sensitive linear/quadratic/gaussian control,” Advances
in Applied Probability, vol. 13, no. 4, pp. 764–777, 1981.

[45]
O. Ogunmolu, N. Gans, and T. Summers, “Minimax iterative dynamic game:
Application to nonlinear robot control tasks,” in 2018 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS),
pp. 6919–6925, IEEE, 2018.

[46]
G. A. Pavliotis, Stochastic Processes and Applications.
Springer, 2014.

[47]
W. Tan, Nonlinear Control Analysis and Synthesis using Sum-of-Squares
PhD thesis, Dept. of Engineering-Mechanical Engineering, University
of Califonia, Berkeley, Berkeley, CA, USA, 2006.

[48]
D. Z. Kleinman, “On an iterative technique for riccati equation
computations,” IEEE Transactions on Automatic Control, vol. 13,
pp. 114–115, 1968.

[49]
E. D. Sontag, Input to State Stability: Basic Concepts and Results,
pp. 163–220.
Berlin, Heidelberg: Springer Berlin Heidelberg, 2008.

[50]
Z.-P. Jiang, Y. Lin, and Y. Wang, “Nonlinear Small-gain Theorems for
Discrete-time Feedback Systems and Applications,” Automatica,
vol. 40, no. 12, pp. 2129–2136, 2004.

[51]
M. González-Fierro, C. Balaguer, N. Swann, and T. Nanayakkara, “A humanoid
robot standing up through learning from demonstration using a multimodal
reward function,” in 2013 13th IEEE-RAS International Conference on
Humanoid Robots (Humanoids), pp. 74–79, 2013.

[52]
R. D. Pristovani, D. R. Sanggar, and P. Dadet, “Implementation of push
recovery strategy using triple linear inverted pendulum model in
“t-FloW” humanoid robot,” Journal of Physics: Conference Series, vol. 1007, p. 012068, apr 2018.

[53]
K. Furut, T. Ochiai, and N. Ono, “Attitude control of a triple inverted
pendulum,” International Journal of Control, vol. 39, no. 6,
pp. 1351–1365, 1984.

[54]
K. Zhou and J. C. Doyle, Essentials of robust control, vol. 104.
Prentice hall Upper Saddle River, NJ, 1998.

[55]
K. Zhou, J. C. Doyle, and K. Glover, Robust and Optimal Control.
Prentice hall Upper Saddle River, NJ, 1996.

[56]
R. A. Horn and C. R. Johnson, Matrix Analysis, second edition.
Cambridge University Press, 2013.

[57]
T. Mori, “Comments on "a matrix inequality associated with bounds on solutions
of algebraic Riccati and Lyapunov equation" by J. M. Saniuk and I.B.
Rhodes,” IEEE Transactions on Automatic Control, vol. 33, no. 11,
pp. 1088–, 1988.

[58]
J. R. Magnus and H. Neudecker, “Matrix differential calculus with applications
to simple, hadamard, and kronecker products,” Journal of Mathematical
Psychology, vol. 29, no. 4, pp. 474–492, 1985.

[59]
L. Koralov and Y. G. Sinai, Theory of Probability and Random Processes.
Springer Berlin, Heidelberg, 2nd ed. ed., 2007.

[60]
A. Arapostathis, V. S. Borkar, and M. K. Ghosh, Ergodic Control of
Diffusion Processes.
Cambridge University Press, 2011.

[61]
T. S. Lee and F. Kozin, “Almost sure asymptotic likelihood theory for
diffusion processes,” Journal of Applied Probability, vol. 14, no. 3,
p. 527–537, 1977.

[]Leilei Cui
received a B.Sc. degree in Automation from Northwestern Polytechnical University, Xian, China, in 2016, and an M.Sc. degree in Control Science and Engineering at Shanghai Jiao Tong University, Shanghai, China, in 2019. He is currently a Ph.D. candidate in the Control and Networks Lab, Tandon School of Engineering, New York University. He was a Microsoft Research Intern during Summer of 2022. His research interests include robot control, reinforcement learning, adaptive dynamic programming, and optimal control.

0pt plus -1fil
[]Lekan Molu is a Senior Researcher at Microsoft Research, New York City. He got his Master of Science in Engineering in Control Systems from the University of Sheffield, South Yorkshire, United Kingdom in 2013. He was subsequently a PhD student at the University of Texas at Dallas from 2014 through 2019 where he was concurrently a visiting student researcher in the Medical Physics and Engineering Division of the University of Texas Southwestern Medical Center. He was recently a postdoctoral scholar at the University of Pennsylvania in Philadelphia, PA, USA. His research interests span machine learning, optimal, adaptive, and nonlinear control with emphasis on their applications to robotics and complex systems. He has regularly been an associate editor for the flagship IEEE International Conference on Robotics and Automation; and he was a member of the New York Academy of Sciences, and American Association of Physicists in Medicine previously.