# On Building Myopic MPC Policies using Supervised Learning

Christopher A. Orrico  Bokan Yang  Dinesh Krishnamoorthy∗ Department of
Mechanical Engineering, Eindhoven University of Technology, 5600 MB,
Eindhoven, The Netherlands.
(email<EMAIL_ADDRESS>
###### Abstract

The application of supervised learning techniques in combination with model
predictive control (MPC) has recently generated significant interest,
particularly in the area of approximate explicit MPC, where function
approximators like deep neural networks are used to learn the MPC policy via
optimal state-action pairs generated offline. While the aim of approximate
explicit MPC is to closely replicate the MPC policy, substituting online
optimization with a trained neural network, the performance guarantees that
come with solving the online optimization problem are typically lost. This
paper considers an alternative strategy, where supervised learning is used to
learn the optimal value function offline instead of learning the optimal
policy. This can then be used as the cost-to-go function in a myopic MPC with
a very short prediction horizon, such that the online computation burden
reduces significantly without affecting the controller performance. This
approach differs from existing work on value function approximations in the
sense that it learns the cost-to-go function by using offline-collected state-
value pairs, rather than closed-loop performance data. The cost of generating
the state-value pairs used for training is addressed using a sensitivity-based
data augmentation scheme.

###### keywords:

model predictive control, value function approximation, parametric sensitivity

##  1 Introduction and Motivation

Model predictive control (MPC) is a popular approach where an optimization
problem is solved iteratively online at each sampling instant, taking into
account the current state of the system. The resulting optimal trajectory
provides the first control input, which is implemented in the system, leading
to an implicit control policy. However, the need for online computation can be
a limiting factor for embedded systems, since computational resources are
often limited and decisions have to be made with fast sampling times. One
popular approach to address this issue is to pre-compute optimal actions
offline that can be stored and accessed online without the need for online
optimization. This approach was first explored under the context of explicit
MPC by Bemporad et al. (2002), which was limited to linear time-invariant
systems with very few states (typically less than 10).

More recently, supervised learning and reinforcement learning techniques have
been used to directly learn a parameterized control policy $\pi(x,\vartheta)$.
The parameters $\vartheta$ can be learned by reinforcement learning to
optimize some measure of performance w.r.t. $\vartheta$. That is, $\vartheta$
is updated to optimize a suitable closed-loop cost under the policy
$\pi(\cdot,\vartheta)$, e.g., using policy gradient or cross-entropy methods.
See for example (Bertsekas, 2019, Section 5.7.1) and the references therein.

Alternatively, the parameterized policy can also be trained offline using
supervised learning, under the context of expert-assisted supervised learning
(Bertsekas, 2019, Section 5.7.2), also referred to as imitation learning or
behavioral cloning (Esmaili et al., 1995). The key idea behind direct MPC
policy approximation using supervised learning is as follows: The MPC problem
is solved offline for different state realizations $\\{x_{i}\\}$ to obtain the
corresponding optimal action $\pi_{\text{mpc}}(x_{i})$ (i.e., the software
“expert”). The MPC actions are then learned using supervised learning with the
labeled training data set $\\{x_{i},\pi_{\text{mpc}}(x_{i})\\}$ to obtain an
approximate policy $\pi(\cdot,\vartheta)$. By learning the optimal control
actions, one can then replace the online optimization with the trained
function approximator $\pi(\cdot,\vartheta)$, which would tell us what action
to take given the current state of the system.

The idea of using neural networks to approximate the MPC policy was first
proposed in the mid 90s by Parisini and Zoppoli (1995). With the promises of
deep learning techniques, and the need to build computationally cheap MPC
policies for embedded applications, the idea of approximating the MPC policy
using deep learning gained renewed interest in the last few years (Chen et
al., 2018; Hertneck et al., 2018; Zhang et al., 2019; Drgoňa et al., 2018;
Karg and Lucia, 2020; Paulson and Mesbah, 2020; Kumar et al., 2021; Åkesson
and Toivonen, 2006; Cao and Gopaluni, 2020), with several applications ranging
from energy (Drgoňa et al., 2018), automotive (Zhang et al., 2020; Quan and
Chung, 2019), chemical processing (Kumar et al., 2021), robotics (Nubert et
al., 2020), aerospace (Sánchez-Sánchez et al., 2016), and healthcare
(Bonzanini et al., 2020).

Despite the enormous popularity, one of the fundamental challenges with this
approach pertains to performance guarantees of the closed-loop system (or lack
thereof). Although many machine learning models in theory can efficiently
represent any function (in this case a policy mapping $\pi(x)$), in practice
additional approximation errors may exist beyond the model uncertainties and
disturbances. For example, these may be caused by the choice of the
hyperparameters in the parametric function architecture
$\pi(\cdot;\vartheta)$, poor generalization, and insufficient training data in
some regions of the feasible state space. Such approximation errors not only
lead to asymptotic performance losses, but can severely degrade the closed-
loop performance and/or violate safety critical constraints. Consequently, by
replacing the online optimization with a parametric function approximator, we
lose guarantees on constraint feasibility and stability of the closed-loop
system that normally come with the standard MPC (i.e. online optimization) by
design. Although there have been developments to address this issue, such as
the use of a safety filter (Wabersich and Zeilinger, 2021), projection on to a
safe set (Chen et al., 2018; Paulson and Mesbah, 2020), probabilistic
verification (Hertneck et al., 2018) etc., these are typically performed a
posteriori (i.e. after training the approximate policy) as a fix-up to account
for constraint satisfaction.

Figure 1: Contribution of this paper in context to existing literature.

The high computation cost of MPC predominantly stems from the long prediction
horizon used by the MPC. Ideally, we want a long prediction horizon length in
order to consider the effect of the current control action far into the
future, such that the cost is decreasing, thus resulting in good performance.
A myopic controller (with very small prediction horizon) can significantly
reduce the online computation cost, but leads to performance degradation
(Maciejowski, 2002, Ch. 6).

This paper argues that in want for very short prediction horizons, all is not
lost. If we have a representation of the “cost-to-go” starting from any point
within the state-space, then the celebrated Bellman’s principle of optimality
enables us to reduce the online optimization problem to even a greedy problem
(i.e., a one-step-look-ahead controller with prediction horizon as small as
$N=1$) without jeopardizing the control performance. Simply put, the
prediction horizon of the original MPC problem is divided into two parts,
where the optimal value function in the tail of the prediction horizon is
replaced by a cost-to-go function that represents the future costs associated
with the control actions. However, getting an exact representation of the
optimal cost-to-go function (using backward induction) is an intractable
problem. To address this, the cost-to-go function can be approximated using a
suitable parametric function $\mathcal{V}(x;\theta)$.

Learning the optimal cost-to-go function has been predominantly studied under
the context of approximate dynamic programming (ADP) and reinforcement
learning (RL) (Bertsekas, 2019), using closed-loop performance data. That is,
the parameters $\theta$ in $\mathcal{V}(x;\theta)$ are updated using
systematic methods such as Q-learning or temporal difference using closed-loop
data obtained from different episodes (Mesbah et al., 2022, Section V).

On the other hand, the use of supervised learning to learn the cost-to-go
function offline (akin to approximate explicit MPC) has received very little
attention, which is the main focus of this paper. The key idea behind learning
the value function offline is similar to MPC policy approximation. That is,
the MPC problem with long prediction horizon (1) is solved offline for
different state realizations $\\{x_{i}\\}$, and the corresponding optimal
value function $V^{*}(x_{i})$ is recorded. The value function is then learned
via supervised learning with the labeled training dataset consisting of state-
value pairs $\\{x_{i},V^{*}(x_{i})\\}$ to fit a function approximator
$\mathcal{V}(x;\theta)$ using supervised learning. The online optimization
problem is then solved with a very short prediction horizon with the learned
value function $\mathcal{V}(x;\theta)$ as the cost-to-go function.

To this end, this paper mainly differs from the vast literature on using RL-
based methods to learn the optimal cost-to-go function, in the fact that we
learn the optimal cost-to-go from “open-loop” data generated offline using
supervised learning, as opposed to closed-loop performance data Mesbah et al.
(2022). An advantage of using supervised learning with open-loop data is that
we do not need a simulator or the actual system in-the-loop to learn the cost-
to-go function, as opposed to RL-based methods. As such, this paper falls
broadly under the category of learning from demonstrations, where we build a
myopic policy for online implementation, using data generated from a
computationally intensive expert MPC. Fig. 1 schematically places this work in
the context of existing literature.

#### Main Contribution

The main contribution of this paper is a myopic MPC policy, where the cost-to-
go function is learned offline using supervised learning. The optimal state-
value pairs used for training is generated by solving long horizon MPC
problems offline for several state realizations. Although this trivially seems
to follow from Bellman’s principle of optimality, generating the labeled
training data set consisting of optimal state-value pairs covering the
feasible state-space can be expensive, albeit performed offline. To address
the cost of generating the training data samples used to learn the optimal
value function, this paper proposes a sensitivity-based data augmentation
scheme to cheaply augment several state-value data pairs using only a single
offline MPC computation, and provides error bounds on the learned value
function when using inexact augmented data.

##  2 Proposed method

###  2.1 Myopic MPC policy

Consider the following MPC problem $\Pi_{N}(x(t))$ with a long prediction
horizon $N$,

$\displaystyle V^{*}(x(t))=\min_{x_{k},u_{k}}$
$\displaystyle\;\sum_{k=0}^{N-1}\ell(x_{k},u_{k})$ (1a) s.t. $\displaystyle
x_{k+1}=f(x_{k},u_{k})\quad\forall k\in\mathbb{I}_{0:N-1}$ (1b) $\displaystyle
u_{k}\in\mathcal{U},\quad x_{k}\in\mathcal{X}$ (1c) $\displaystyle x_{0}=x(t)$
(1d)

where $x\in\mathcal{X}\subseteq\mathbb{R}^{n}$ are the states,
$u\in\mathcal{U}\subseteq\mathbb{R}^{m}$ denotes the inputs, $x(t)$ is the
current state, $\ell:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}$
denotes the stage cost,
$f:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$ denotes the
system model, and $N$ is the length of the prediction horizon.

###### Assumption 1

Functions $f:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$ and
$\ell:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}$ are continuous,
and without loss of generality $\ell(x,u)\geq 0$, $f(0,0)=0$ and
$\ell(0,0)=0$.

###### Assumption 2

$N$ is chosen sufficiently large, such that $\ell(x_{k},u_{k})=0$ for all
$k\geq N$.

Assumption 2 in practice can be satisfied by choosing $N$ arbitrarily large
(albeit at the expense of computation cost). Consequently, the terminal cost
is zero, and hence not shown in (1).

MPC with a long prediction horizon can be computationally expensive for online
implementation. To address this, we would like to leverage Bellman’s principle
of optimality and build a myopic MPC policy that can be used online, given by

$\displaystyle\min_{u\in\mathcal{U}}$
$\displaystyle\;\ell\left(x(t),u\right)+V_{f}(x^{+})$ (2a) s.t. $\displaystyle
x^{+}=f(x(t),u)\in\mathcal{X}$ (2b)

where $V_{f}(x^{+}):\mathbb{R}^{n}\rightarrow\mathbb{R}$ is the optimal cost-
to-go from any point $x^{+}\in\mathcal{X}$. Instead of a one-step look-ahead
controller (2) with a prediction horizon of size one, we consider a more
general myopic controller with a short prediction horizon $\eta<<N$, denoted
by $\mathcal{P}(x(t),V_{f}(\cdot))$

$\displaystyle V_{\text{myopic}}(x(t))=\min_{x_{k},u_{k}}$
$\displaystyle\;\sum_{k=0}^{\eta-1}\ell(x_{k},u_{k})+V_{f}(x_{\eta})$ (3a)
s.t. $\displaystyle x_{k+1}=f(x_{k},u_{k})\quad\forall
k\in\mathbb{I}_{0:\eta-1}$ (3b) $\displaystyle u_{k}\in\mathcal{U},\quad
x_{k}\in\mathcal{X}$ (3c) $\displaystyle x_{0}=x(t)$ (3d)

Note that (2) is a special case of (3) with $\eta=1$. However, since the
optimal cost-to-go $V_{f}(x)$ is not known, we aim to replace it with some
suitable function approximator $\mathcal{V}(x;\theta)$ trained using optimal
state-value pairs generated offline. This closely resembles approximate
dynamic programming (ADP) (Bertsekas, 2019), with the key difference that the
cost-to-go function is learned offline using supervised learning, without the
need for a simulator or the actual system in-the-loop.

###  2.2 Learning optimal cost-to-go using supervised learning

Much like approximate explicit MPC schemes, the training dataset is collected
by solving offline the original MPC problem with long prediction horizon (1)
for several state realizations $x_{i}\in\mathcal{X}$ using any suitable
sampling strategy. The data set consisting of $n_{\text{tr}}$ optimal state-
value pairs is denoted by
$\mathcal{D}:=\\{x_{i},V^{*}(x_{i})\\}_{i=1}^{n_{\text{tr}}}$. Using
$\mathcal{D}:=\\{(x_{i},V^{*}(x_{i}))\\}_{i=1}^{n_{\text{tr}}}$ as our labeled
training data set, a suitable function approximator $\mathcal{V}(x;\theta)$,
parameterized by the parameters $\theta$, is trained in order to minimize the
mean squared error

$\hat{\theta}=\arg\min_{\theta}\frac{1}{n_{\text{tr}}}\sum_{i=1}^{n_{\text{tr}}}\|\mathcal{V}(x_{i};\theta)-V^{*}(x_{i})\|^{2}$
(4)

The optimal cost-to-go $V_{f}(x)$ in (3) or (2) is then replaced with the
learned function $\mathcal{V}(x_{i};\theta)$ in the online implementation. If
we can learn an exact representation of the optimal cost-to-go, then by
Bellman’s principal of optimality, the myopic policy (3) will be equivalent to
the expert MPC problem (1), but with significantly lower computation cost.

###  2.3 Data augmentation for generating optimal state-value pairs

Learning the cost-to-go function offline requires sufficiently rich training
data covering the state-space. To this end, both learning the optimal action,
as well as learning the value function requires solving the full MPC problem
(1) offline for several state realizations $x\in\mathcal{X}$, and the
challenge of generating the training data samples still remain. This section
extends the recently proposed “data-augmentation” scheme (Krishnamoorthy,
2022, 2023) to efficiently generate several optimal state-value pairs using
only a few offline MPC computations.

Noting that the MPC problem is parametric in the initial state $x_{i}$, the
key idea behind the data augmentation scheme is as follows: Once the
optimization problem (1) is solved for a given initial state $x_{i}$, the
parametric sensitivities tells us how the optimum changes when the initial
state changes.

For the sake of notational simplicity, the offline MPC problem (1) is
expressed in a standard parametric NLP problem of the form,

$\displaystyle V^{*}(x)=$ $\displaystyle\min_{\mathbf{w}}\;\;J(\mathbf{w},x)$
(5a) s.t. $\displaystyle{c}(\mathbf{w},x)=0$ (5b) $\displaystyle
g(\mathbf{w},x)\leq 0$ (5c)

where $\mathbf{w}:=[u(0|t),\dots,u(N-1|t),x(1|t),\dots,x(N|t)]^{\mathsf{T}}$
denotes all the decision variables, the parameter $x$ denotes the initial
state. The Lagrangian of (5) is given by

$\mathcal{L}(\mathbf{w},x,\lambda,\mu):=J(\mathbf{w},x)+\lambda^{\mathsf{T}}{c}(\mathbf{w},x)+\mu^{\mathsf{T}}g(\mathbf{w},x)$
(6)

where $\lambda$ and $\mu$ are the Lagrangian multipliers of (5b) and (5c)
respectively. A primal-dual solution vector
$\mathbf{s}^{*}(x)=[\mathbf{w},\lambda,\mu]^{\mathsf{T}}$ satisfies the KKT
condition

$\varphi(\mathbf{s}^{*}(x),x)=\begin{bmatrix}\nabla_{\mathbf{w}}\mathcal{L}(\mathbf{w},x,\lambda,\mu)\\\
c(\mathbf{w},x)\\\ g_{\mathbb{A}}(\mathbf{w},x)\\\ \end{bmatrix}=0$ (7)

where $g_{\mathbb{A}}$ denotes the set of active constraints.

###### Assumption 3

The cost and the constraints of (1) are twice continuously differentiable in a
neighborhood of the KKT point $\mathbf{s}^{*}(x_{i})$. Linear independence
constraint qualification (LICQ), second order sufficient conditions (SOSC) and
strict complementarity hold for the solution vector $\mathbf{s}^{*}(x)$.

As explained by Fiacco (1976), given Assumption 3, for parametric
perturbations $\Delta x$ in the neighborhood of $x_{i}$, there exists a
unique, continuous vector function $\mathbf{s}^{*}(x_{i}+\Delta x)$ which is a
KKT point for the problem with parameter $x_{i}+\Delta x$, with
$V^{*}(x_{i}+\Delta x)$ being its corresponding optimal value function.
Furthermore, there exists a positive Lipschitz constant $L_{v}$ such that the
optimal cost satisfies $\|V^{*}(x_{i}+\Delta x)-V^{*}(x_{i})\|\leq
L_{v}\|\Delta x\|$.

Algorithm 1 Myopic MPC policy design using supervised learning.

1:${\Pi}_{N}(x)$, $\mathcal{X}$, $\mathcal{D}=\emptyset$

2:for $i=1,\dots,N_{s}$ do

3: Sample $x_{i}\in\mathcal{X}$

4: $\mathbf{s}^{*}(x_{i}),V^{*}(x_{i})\leftarrow$ Solve $\Pi_{N}(x_{i})$

5: $\mathcal{D}\leftarrow\mathcal{D}\cup\\{(x_{i},V^{*}(x_{i}))\\}$

6:
$\mathcal{H}_{i}\leftarrow-\left[\frac{\partial\varphi}{\partial\mathbf{s}}\right]^{-1}\frac{\partial\varphi}{\partial
x}$

7: for $j=1,\dots,N_{p}$ do

8: Sample $\Delta x_{j}\in\Delta\mathcal{X}_{i}$ in the neighborhood of
$x_{i}$

9: $\hat{\mathbf{s}}^{*}(x_{i}+\Delta
x_{j})=\mathbf{s}^{*}(x_{i})-\mathcal{H}_{i}\Delta{x}_{j}$

10: Evaluate cost $\hat{{V}}(x_{i}+\Delta x_{j})$ at the primal-dual solution
vector $\mathbf{\hat{s}}^{*}(x_{i}+\Delta{x}_{j})$

11: if Active constraint set is same then

12: $\mathcal{D}\leftarrow\mathcal{D}\cup\\{(x_{i}+\Delta
x_{j},\hat{{V}}(x_{i}+\Delta x_{j}))\\}$

13: end if

14: end for

15:end for

16:${\theta}_{1}\leftarrow\arg\min_{\theta}\frac{1}{N_{s}N_{p}}\sum_{i=1}^{N_{s}N_{p}}\|\mathcal{V}(x_{i};\theta)-V(x_{i})\|^{2}$

17:Append $\mathcal{V}(x_{i};\hat{\theta})$ as the cost-to-go function in (3)

18:Myopic MPC $\mathcal{P}(x(t),\mathcal{V}(\cdot;\theta_{1}))$

###### Assumption 4

For $i=1,\dots,N_{s}$ expert samples, the neighborhood $\Delta\mathcal{X}_{i}$
is chosen for data augmentations such that
$x_{i}\in\langle\Delta\mathcal{X}_{i}\rangle$,
$\cup_{i=1}^{N_{s}}\Delta\mathcal{X}_{i}=\mathcal{X}_{feas}$, and the active
constraint set remains the same within each $\Delta\mathcal{X}_{i}$.

######  Theorem 1 (Sensitivity-based Data Augmentation)

Under Assumptions 3 and 4, let $V^{*}(x_{i})$ be the optimal value obtained by
solving (1) for any given state $x_{i}$. Then additional samples
$\\{(x_{i}+\Delta x_{j},\hat{V}_{N}(x_{i}+\Delta x_{j}))\\}_{j}$ for $\Delta
x_{j}\in\Delta\mathcal{X}_{i}$ can be augmented to the data set without
solving (1), that satisfies

$\|V^{*}(x_{i}+\Delta x_{j})-\hat{V}(x_{i}+\Delta x_{j})\|\leq L_{V}\|\Delta
x_{j}\|^{2}$ (8)

###### Proof.

Using Taylor series expansion of the optimal solution vector
$\mathbf{s}^{*}(x)$ around $x_{i}$, we get

$\displaystyle\mathbf{s}^{*}(x_{i}+\Delta
x)=\mathbf{s}^{*}(x_{i})+\frac{\partial\mathbf{s}^{*}}{\partial x}\Delta
x+\mathcal{O}(\|\Delta x\|^{2})$ (9)

Since $\mathbf{s}^{*}(x)$ satisfies (7) for any $x$ in the neighborhood of
$x_{i}$, using implicit function theorem, we have

$\displaystyle\frac{\partial}{\partial
x}\varphi(\mathbf{s}^{*}(x),x)\bigg{|}_{x=x_{i}}=\frac{\partial\varphi}{\partial\mathbf{s}}\frac{\partial\mathbf{s}^{*}}{\partial
x}+\frac{\partial\varphi}{\partial x}=0$ $\displaystyle\Rightarrow$
$\displaystyle\frac{\partial\mathbf{s}^{*}}{\partial
x}=-\left[\frac{\partial\varphi}{\partial\mathbf{s}}\right]^{-1}\frac{\partial\varphi}{\partial
x}=:\mathcal{H}_{i}$

Ignoring the higher order terms and substituting this in (9), we get

${\mathbf{s}}^{*}(x_{i}+\Delta x)\approx\hat{\mathbf{s}}(x_{i}+\Delta
x)=\mathbf{s}^{*}(x_{i})+\underbrace{\mathcal{H}_{i}\Delta
x}_{:=\Delta\mathbf{s}}$ (10)

Noting that $V^{*}(x_{i})$ corresponds to the optimal value function at the
optimal solution $\mathbf{s}^{*}(x_{i})$, the new augmented data pair
$(x_{i}+\Delta x_{j},\hat{V}_{N}(x_{i}+\Delta x_{j}))$ is then obtained by
evaluating the cost using the approximate solution
$\hat{\mathbf{s}}(x_{i}+\Delta x)$. From continuity and differentiability of
the optimal solution vector, there exists a positive Lipschitz constant
$L_{V}$ such that (8) holds. ∎

The proposed framework is summarized in Algorithm 1.

###### Remark 1

Note that augmenting any number of optimal state-value pairs using a single
data sample $(x_{i},V^{*}(x_{i}))$ requires computing $\mathcal{H}_{i}$ only
once! In other words, using only a single NLP solve at $x_{i}$, and a single
linear solve (to compute $\mathcal{H}_{i}$), one can cheaply augment several
data samples using (10).

###### Assumption 5

The functional form $\mathcal{V}(\cdot;\theta)$ is sufficiently richly
parameterized and $\exists\;\hat{\theta}$ s.t.
$\mathcal{V}(x;\hat{\theta})=V^{*}(x)$.

Under Assumption 5 Bellman’s principle of optimality dictates that a myopic
policy (3) will yield a solution akin to that obtained from a long horizon
“expert” MPC (1).

###### Theorem 2

Consider a problem with the same setup as Theorem 1, where $N_{p}$ data
samples are augmented using (10). Under Assumptions 4 and 5

$\|\mathcal{V}(x;\theta_{1})-V^{*}(x)\|\leq\max\left(\\{L_{V_{i}}R_{i}^{2}\\}\right)$
(11)

holds in probability as $N_{p}\rightarrow\infty$ if $\theta_{1}$ is a
consistent estimator of (4), where $R_{i}:=\sup_{\Delta
x\in\Delta\mathcal{X}_{i}}\|\Delta x\|$.

###### Proof.

Aggregating (8) from Theorem 1 over all the neighborhoods used for data
augmentation, we have

$\|V^{*}(x)-\hat{V}(x)\|\leq\max\left(\\{L_{V_{i}}R_{i}^{2}\\}\right)\forall
x\in\mathcal{X}_{feas}$ (12)

If $\theta_{1}$ is a consistent estimator of (4), then (11) follows under
Assumption 5. ∎

The accuracy of the augmented samples can be further improved using additional
corrector steps in the sensitivity-update, as described in (Krishnamoorthy,
2023). In any case, the idea with value function approximation is to capture
the flavor of what are the good states and what are the bad states. As such,
the myopic MPC policy can perform well even when approximating the value
function using inexact augmented samples, unlike direct policy approximation.
This is formally stated below.

###### Proposition 1

Let the optimal action at a state $x_{i}\in\mathcal{X}_{feas}$ be denoted as
$u_{i}^{*}=\pi(x_{i})$, such that $V^{*}(f(x_{i},u_{i}^{*}))\leq
V^{*}(f(x_{i},u))$ for all $u\in\mathcal{U}\setminus u_{i}^{*}$. By using an
approximate function $\mathcal{V}(\cdot;\theta)$ in a myopic MPC (2), if
$\mathcal{V}(f(x_{i},u_{i}^{*});\theta)\leq\mathcal{V}(f(x_{i},u);\theta)$ for
all $u\in\mathcal{U}\setminus u_{i}^{*}$, then the myopic policy
$\pi_{\text{myopic}}(x_{i})=\pi(x_{i})$ even if
$\|V^{*}(\cdot)-\mathcal{V}(\cdot;\theta)\|>0$.

###### Proof.

If $\mathcal{V}(f(x,u_{i}^{*});\theta)\leq\mathcal{V}(f(x,u_{i});\theta)$ for
all $u\in\mathcal{U}$, the myopic MPC (2) using $\mathcal{V}(\cdot;\theta)$ as
the cost-to-go function will naturally choose $u_{i}^{*}$ as its optimal
solution, since this would give the lowest cost-to-go among all the possible
actions. Hence $\pi(x_{i})=\pi_{\text{myopic}}(x_{i})$ irrespective of
approximation errors between $\mathcal{V}(\cdot;\theta)$ and $V^{*}(\cdot)$. ∎

This is of course a restrictive case that yields the optimal policy exactly.
However, some deviation from the optimal policy may be tolerated if the myopic
policy is stable, which is stated below.

###### Proposition 2

By using an approximate function $\mathcal{V}(\cdot;\theta)$ in a myopic MPC
(2), if for any $x\in\mathcal{X}_{feas}$, $\exists\;u\in\mathcal{U}$ such that

$\mathcal{V}(f(x,u);\theta)+\ell(x,u)\leq\mathcal{V}(x;\theta)$ (13)

then $\mathcal{V}(\cdot;\theta)$ yields a stable policy even if
$\|V^{*}(\cdot)-\mathcal{V}(\cdot;\theta)\|>0$.

###### Proof.

The proof follows from the fact that under (13), we have

$V_{\text{myopic}}(x^{+})-V_{\text{myopic}}(x)\leq-\ell(x,\pi_{\text{myopic}}(x))$

where $x^{+}=f(x,u)$ is the successor state. This implies that the the myopic
MPC policy is closed-loop stable. ∎

The condition (13) can be achieved by suitably choosing the parametric
function approximator, e.g., input convex neural networks.

###### Remark 2

The choice of sampling strategies for generating the training dataset is
pivotal. Some common sampling strategies include random sampling or uniform
gridding to more purpose-built sampling strategies such as control-oriented
sampling based on closed-loop simulation (Krishnamoorthy et al., 2021), or
sampling based on geometric random walks (Chen et al., 2022). The data
augmentation framework presented in the paper complements the different
sampling strategies in the sense that when a sampling strategy is used to
decide where to sample, data augmentation can be used to generate the data
point without querying the expert at that sample.

##  3 Illustrative Example

The proposed approach is demonstrated on a benchmark inverted pendulum problem
with the nonlinear dynamics
$ml^{2}\ddot{\omega}=u-b\dot{\omega}-mgl\sin\omega$ described by the two
states, angle $\omega$, and angular velocity $\dot{\omega}$. The stage cost is
defined as $\ell(x,u):=\|x-x_{sp}\|^{2}$, where
$x_{sp}=[3.14,0]^{\mathsf{T}}$. In this example, we have
$x=[\omega,\dot{\omega}]^{\mathsf{T}}$, $\mathcal{X}=[0,2\pi]\times[-5,5]$ and
$\mathcal{U}\in[-50,50]$. The long horizon MPC problem used to generate the
offline training data set uses a prediction horizon of $N=30$ samples, with a
sampling frequency of 30Hz. Note that we deliberately choose this simple
example to facilitate visualization of the exact and inexact cost-to-go
manifolds to illustrate the proposed approach.

#### Case 0 - No data augmentation:

For the offline data generation, we use the long horizon MPC problem (1) with
$N=30$. We first use a uniform grid-based sampling strategy with
100$\times$100 samples across $x_{1}$ and $x_{2}$ resulting in a total of
10000 samples (i.e., the long horizon MPC is solved 10000 times to generate
the training data set).

Using the state-value pairs $\mathcal{D}=\\{x_{i},V^{*}(x_{i})\\}$, we now fit
a radial basis function network (RBFN) with 100 neurons. The unit center for
each neuron is determined using k-means clustering algorithm, and the width of
the kernel function is set to 0.1. The trained cost-to-go function
$\hat{\mathcal{V}}(x)$ given by the radial basis function network is then used
in the myopic MPC policy (3) with a single step prediction horizon ($\eta=1$).

The top subplot in Fig. 2 shows the closed-loop trajectories using the long
horizon MPC (black solid lines) and the trained myopic MPC (red dashed lines)
starting from different initial conditions. The corresponding CPU times are
shown in the bottom subplot, where it can be seen that by learning the cost-
to-go function offline, we can get a good closed-loop performance at a lower
online computation cost. However, generating the training data set offline can
be expensive. For instance, generating 10000 samples required a total CPU time
of 555.9511s.

Figure 2: Closed-loop trajectories of the myopic MPC policy (red dashed lines)
trained using exact dataset (Case 0), compared with the expert long horizon
MPC (black solid lines). The bottom subplot shows the CPU time, where the
solid lines denote the mean computation time over the different trajectories,
and the shaded area denotes the maximum and minimum CPU times over all the
closed-loop simulation runs.  Figure 3: Augmented data set consisting of
optimal state-value pairs, where the exact samples are denoted by a red
circle, and the augmented samples are denoted by a black dot (left subplots).
The right subplots show the corresponding closed-loop trajectories of the
myopic MPC policy trained using augmented datasets (red dashed), compared with
the expert long horizon MPC (black solid).

The cost of generating the training data samples can be addressed using data
augmentation as described in Section 2.3.

#### Case 1 - Data augmentation using 100 offline MPC solves:

To demonstrate this, we now use a uniform grid-based sampling strategy with
$10\times 10$ samples across $x_{1}$ and $x_{2}$ dimension, resulting in only
100 offline MPC solves. We further augment this data set with a uniform grid-
based sampling strategy with $10\times 10$ samples around each MPC solve using
the sensitivity-based data augmentation strategy described in Section 2.3.
This results in a total of $10100$ samples used for supervised learning, which
took a total of 6.6539s. The generated samples are shown in Fig. 3(a), where
the MPC samples are shown in red circles, and the augmented samples are shown
in black dots. The closed-loop trajectories using the myopic MPC policy
trained using the augmented samples are shown in Fig. 3(b).

#### Case 2 - Data augmentation using 9 offline MPC solves:

We then use a uniform grid of $3\times 3$ samples, resulting in 9 offline MPC
solves, using which a further 9810 samples are augmented using the
sensitivity-based data augmentation scheme. The generated samples are shown in
Fig. 3(c), where the MPC samples are shown in red circles, and the augmented
samples are shown in black dots. The closed-loop trajectories using the myopic
MPC policy trained using the augmented samples are shown in Fig. 3(d).

#### Case 3 - Data augmentation using 1 offline MPC solve:

We then solve only 1 MPC offline, using which a further 10000 samples are
augmented. The generated samples are shown in Fig. 3(e), where the MPC samples
are shown in red circles, and the augmented samples are shown in black dots.
The closed-loop trajectories using the myopic MPC policy trained using the
augmented samples are shown in Fig. 3(f).

Table 1: Total CPU time required to generate the different augmented data sets, and the maximum error compared to generating all the data samples exactly.  | CPU time [s] | Max Error
---|---|---
Case 0 | 555.95 | –
Case 1 | 6.6539 | 0.1442
Case 2 | 0.7429 | 1.0193
Case 3 | 0.1649 | 2.7362

As discussed in Section 2, a myopic MPC controller with online optimization
can also, to some extent, handle perturbations to the controller parameters
(such as cost function or constraints) online, unlike approximate explicit MPC
schemes, where any change in the controller renders the trained policy
function useless. To demonstrate this, we now keep the the same cost-to-go
function $\hat{\mathcal{V}}(x;\theta)$ trained earlier in Case 0, but tighten
the constraint set $\mathcal{U}\in[-20,20]$ in the online myopic MPC. The
closed-loop trajectories of the modified myopic MPC with a tighter constraint
set, benchmarked against the long horizon MPC with the new constraint set, is
shown in Fig. 4(a), where it can be seen that despite changes in the
constraint set online, the myopic MPC policy is able to respect the
constraints and results in a similar performance as the long horizon MPC. This
is because the constraints are enforced explicitly in the myopic policy (3).

Similarly, we now use the same cost-to-go function
$\hat{\mathcal{V}}(x;\theta)$ trained earlier in case 0, but add an input
rate-of change penalty term $0.01\|\Delta u\|^{2}$ to the stage cost. The
closed-loop trajectories of the myopic MPC with a modified stage-cost,
benchmarked against the long horizon MPC, is shown in Fig. 4(b), where it can
be seen that despite changes in the stage cost, the myopic MPC policy is able
to provide a similar performance as the long horizon MPC. However, since the
cost-to-go function is trained without additional input penalty terms, the
closed-loop performance is rather aggressive, compared to the full horizon MPC
with the modified cost function. Nevertheless, the myopic MPC is able to
stabilize the inverted pendulum starting from various initial conditions.

Figure 4: Closed-loop trajectories of the Myopic MPC (red dashed) with
modified controller parameters during online implementation compared to the
corresponding full horizon MPC (black solid). (a) Tighter constraint set
$\mathcal{U}$. (b) Additional penalty term in the stage cost.

##  4 Conclusion

This paper presented a supervised learning approach to building a myopic MPC
with approximate cost-to-go function trained from expert demonstrations
collected offline. This paper also presented a data augmentation scheme to
efficiently generate the state-value labeled training data samples to learn
the cost-to-go function, where it was shown that several data samples can be
augmented using a single MPC solve, and a single linear solve. The error bound
due to augmenting inexact samples in the offline learning problem is
quantified. The paper also shows that despite approximation errors in the
learned value function, the myopic policy using the cost-to-go approximation
may be capable of recovering the optimal policy under suitable conditions. The
proposed approach was demonstrated on a benchmark inverted pendulum example.

When querying the expert MPC (1) to generate the state-value data pairs
$\\{(x_{i},V_{I}^{*})\\}$, we can also store the corresponding optimal actions
$u^{*}_{i}$. This can be used to simultaneously train a function approximator
$\pi_{\text{approx}}(x;\vartheta)$, which can be used to warm start the myopic
controller (3), further accelerating the online computation of the myopic
policy.

Such myopic policies may also be built using expert demonstrations in the form
of state-action pairs $\\{(x_{i},u_{i}^{*})\\}$, where the parameters of the
cost-to-go function $\theta$ are computed using inverse optimization. In other
words, the KKT residuals of the myopic controller (3) evaluated at the state-
action pairs $\\{(x_{i},u_{i}^{*})\\}$ obtained from (1) are minimized
(Keshavarz et al., 2011). The expert demonstrations in the form of state-
action pairs can be obtained by querying (1) with additional data samples
augmented using the sensitivity-based data augmentation scheme as presented by
Krishnamoorthy (2022, 2023).

## References

  * Åkesson and Toivonen (2006) Åkesson, B.M. and Toivonen, H.T. (2006).  A neural network model predictive controller.  _Journal of Process Control_ , 16(9), 937–946.
  * Bemporad et al. (2002) Bemporad, A., Morari, M., Dua, V., and Pistikopoulos, E.N. (2002).  The explicit linear quadratic regulator for constrained systems.  _Automatica_ , 38(1), 3–20.
  * Bertsekas (2019) Bertsekas, D.P. (2019).  _Reinforcement learning and optimal control_.  Athena Scientific Belmont, MA.
  * Bonzanini et al. (2020) Bonzanini, A.D., Paulson, J.A., Graves, D.B., and Mesbah, A. (2020).  Toward safe dose delivery in plasma medicine using projected neural network-based fast approximate NMPC.  _IFAC-PapersOnLine_ , 53(2), 5279–5285.
  * Cao and Gopaluni (2020) Cao, Y. and Gopaluni, R.B. (2020).  Deep neural network approximation of nonlinear model predictive control.  _IFAC-PapersOnLine_ , 53(2), 11319–11324.
  * Chen et al. (2018) Chen, S., Saulnier, K., Atanasov, N., Lee, D.D., Kumar, V., Pappas, G.J., and Morari, M. (2018).  Approximating explicit model predictive control using constrained neural networks.  In _2018 Annual American control conference (ACC)_ , 1520–1527. IEEE.
  * Chen et al. (2022) Chen, S.W., Wang, T., Atanasov, N., Kumar, V., and Morari, M. (2022).  Large scale model predictive control with neural networks and primal active sets.  _Automatica_ , 135, 109947.
  * Drgoňa et al. (2018) Drgoňa, J., Picard, D., Kvasnica, M., and Helsen, L. (2018).  Approximate model predictive building control via machine learning.  _Applied Energy_ , 218, 199–216.
  * Esmaili et al. (1995) Esmaili, N., Sammut, C., and Shirazi, G. (1995).  Behavioural cloning in control of a dynamic system.  In _1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century_ , volume 3, 2904–2909. IEEE.
  * Fiacco (1976) Fiacco, A.V. (1976).  Sensitivity analysis for nonlinear programming using penalty methods.  _Mathematical programming_ , 10(1), 287–311.
  * Hertneck et al. (2018) Hertneck, M., Köhler, J., Trimpe, S., and Allgöwer, F. (2018).  Learning an approximate model predictive controller with guarantees.  _IEEE Control Systems Letters_ , 2(3), 543–548.
  * Karg and Lucia (2020) Karg, B. and Lucia, S. (2020).  Efficient representation and approximation of model predictive control laws via deep learning.  _IEEE Transactions on Cybernetics_.
  * Keshavarz et al. (2011) Keshavarz, A., Wang, Y., and Boyd, S. (2011).  Imputing a convex objective function.  In _2011 IEEE international symposium on intelligent control_ , 613–619. IEEE.
  * Krishnamoorthy (2022) Krishnamoorthy, D. (2022).  A sensitivity-based data augmentation framework for model predictive control policy approximation.  _IEEE Transactions on Automatic Control_ , 67, 6090 – 6097.
  * Krishnamoorthy (2023) Krishnamoorthy, D. (2023).  An improved data augmentation scheme for model predictive control policy approximation.  _IEEE Control Systems Letters_ , 7, 1867 – 1872.
  * Krishnamoorthy et al. (2021) Krishnamoorthy, D., Mesbah, A., and Paulson, J.A. (2021).  An adaptive correction scheme for offset-free asymptotic performance in deep learning-based economic MPC.  _IFAC-PapersOnLine_ , 54(3), 584–589.
  * Kumar et al. (2021) Kumar, P., Rawlings, J.B., and Wright, S.J. (2021).  Industrial, large-scale model predictive control with structured neural networks.  _Computers & Chemical Engineering_, 150, 107291.
  * Maciejowski (2002) Maciejowski, J.M. (2002).  _Predictive control: with constraints_.  Pearson education.
  * Mesbah et al. (2022) Mesbah, A., Wabersich, K.P., Schoellig, A.P., Zeilinger, M.N., Lucia, S., Badgwell, T.A., and Paulson, J.A. (2022).  Fusion of machine learning and mpc under uncertainty: What advances are on the horizon?  In _2022 American Control Conference (ACC)_ , 342–357. IEEE.
  * Nubert et al. (2020) Nubert, J., Köhler, J., Berenz, V., Allgöwer, F., and Trimpe, S. (2020).  Safe and fast tracking on a robot manipulator: Robust mpc and neural network control.  _IEEE Robotics and Automation Letters_ , 5(2), 3050–3057.
  * Parisini and Zoppoli (1995) Parisini, T. and Zoppoli, R. (1995).  A receding-horizon regulator for nonlinear systems and a neural approximation.  _Automatica_ , 31(10), 1443–1451.
  * Paulson and Mesbah (2020) Paulson, J.A. and Mesbah, A. (2020).  Approximate closed-loop robust model predictive control with guaranteed stability and constraint satisfaction.  _IEEE Control Systems Letters_.
  * Quan and Chung (2019) Quan, Y.S. and Chung, C.C. (2019).  Approximate model predictive control with recurrent neural network for autonomous driving vehicles.  In _2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE)_ , 1076–1081. IEEE.
  * Sánchez-Sánchez et al. (2016) Sánchez-Sánchez, C., Izzo, D., and Hennes, D. (2016).  Learning the optimal state-feedback using deep networks.  In _2016 IEEE Symposium Series on Computational Intelligence (SSCI)_ , 1–8. IEEE.
  * Wabersich and Zeilinger (2021) Wabersich, K.P. and Zeilinger, M.N. (2021).  A predictive safety filter for learning-based control of constrained nonlinear dynamical systems.  _Automatica_ , 129, 109597.
  * Zhang et al. (2019) Zhang, X., Bujarbaruah, M., and Borrelli, F. (2019).  Safe and near-optimal policy learning for model predictive control using primal-dual neural networks.  In _2019 American Control Conference (ACC)_ , 354–359. IEEE.
  * Zhang et al. (2020) Zhang, X., Bujarbaruah, M., and Borrelli, F. (2020).  Near-optimal rapid mpc using neural networks: A primal-dual policy learning framework.  _IEEE Transactions on Control Systems Technology_ , 29(5), 2102–2114.