# Passivizing learned policies and learning passive policies with virtual
energy tanks in robotics

Riccardo Zanella1, Gianluca Palli1, Stefano Stramigioli2, Federico Califano2 1
Department of Electrical, Electronic and Information Engineering, University
of Bologna, Viale Risorgimento 2, 40136 Bologna, Italy. Corresponding author
email<EMAIL_ADDRESS>Robotics & Mechatronics (RaM) group,
University of Twente, The Netherlands.This work was supported by the European
Commission’s Horizon Europe Programme with the project IntelliMan - AI-Powered
Manipulation System for Advanced Robotic Service, Manufacturing and
Prosthetics - under grant agreement No 101070136.

###### Abstract

Within a robotic context, we merge the techniques of passivity-based control
(PBC) and reinforcement learning (RL) with the goal of eliminating some of
their reciprocal weaknesses, as well as inducing novel promising features in
the resulting framework. We frame our contribution in a scenario where PBC is
implemented by means of virtual energy tanks, a control technique developed to
achieve closed-loop passivity for any arbitrary control input. Albeit the
latter result is heavily used, we discuss why its practical application at its
current stage remains rather limited, which makes contact with the highly
debated claim that passivity-based techniques are associated to a loss of
performance. The use of RL allows to learn a control policy which can be
passivized using the energy tank architecture, combining the versatility of
learning approaches and the system theoretic properties which can be inferred
due to the energy tanks. Simulations show the validity of the approach, as
well as novel interesting research directions in energy-aware robotics.

##  I Introduction

Robotics is increasingly focusing on the development of control frameworks
allowing the transition from industrial cages to unstructured environments.
This transition carries the ambitious objective of stable and safe execution
of complex tasks where robots coexist with other robots or possibly humans.
The practical impossibility of a careful dynamic modeling of the environment
along those tasks makes the stability objective even more challenging, and
model-based approaches poorly suitable.

Passive controllers have been presented as a feasible solution to tackle this
problem as the stability of the closed-loop system is in principle independent
on the external environmental interaction [1, 2]. Passive systems route
physical energy rather than producing it, which is the ultimate reason why
stability proofs (using energy functions as Lyapunov candidates) are
conveniently assessed in this framework [3, 4]. A powerful technique allowing
the passivisation of any control action is represented by virtual energy tanks
[5, 6, 7]. These are able to store information about physical energy flows
undergoing the system, and introduce at a control level the scalar information
representing the energy budget for the controlled robot. A conditional check
on this budget with proper passive reaction strategies in case of depletion of
the latter, is what guarantees closed-loop passivity. The main drawback of
passivity-based control methods (comprising those involving energy tanks) is
the lack of optimisation over a task performance metric along the design of
the controller [8]. This fact led to the claim that passivity-based control
methods are associated to a loss of performance, which becomes more severe as
the complexity of tasks increases.

On the other side of the control theoretic spectrum, recent advancements in
the machine learning community are leading to robots with an outstanding
awareness of complex environments and tasks. The intelligence encoded in the
control policies, normally optimised by means of performance-based metrics,
reflects complex high-level decision making strategies which are learned
thanks to availability of huge datasets [9, 10, 11]. The drawback of these
families of approaches is the difficulty in guaranteeing system theoretic
properties as passivity and stability for the controlled system [12].

In this work we merge the passivity-based control (PBC) design involving
energy tanks and reinforcement learning framework (RL), combining system
theoretic properties of the closed-loop system induced by the ultimately
passive design, and high performance achievement peculiar of the RL framework.
This procedure, presented both in inference and in training, allows to
meaningful scale PBC to tasks requiring learning of complex control policies.

#### Related work

We recognise related work casting the energy tank algorithm (seen as a task-
agnostic passivisation tool) into a framework using task-based information for
performance augmentation of the underlying passive system. In [13] the authors
introduce the so-called valve-based energy tanks (also used in e.g., [14]), in
which extra parameters are introduced on the energy tanks in order to embed
control objectives in the design, beyond achieving passivity. Task
specifications are translated into tank design rules by controlling the power
flows undergoing the system. The idea of using a reference power trajectory in
combination with energy tanks is present also in [15, 16], where task-based
specifications are used to tune the tank parameters. These works present very
stiff task specifications, and badly scale to complex tasks in which high
level policies need to be learned. A family of methods using energy tanks and
sharing a similar motivation of this work are are [17, 18], where an explicit
optimisation problem is introduced to find the closest passive approximation
of a given control action. The idea is to exploit the versatility of energy
tank architecture to perform an optimisation which can be generalised and
scaled for different tasks and whose outcome is a passively controlled system.
An important difference between this work and the cited ones (beyond the
specific technique to solve the optimisation) is that in [17, 18] the desired
control action is already given, and the optimisation aims at finding the
closest passive approximation to it, imposed by a non-depletion constraint of
the tank. We will comment on the difference between the approaches, which
involves the degree of freedom represented by the initial state in the tank.

The paper is organised as follows. In Sec. II a throughout explanation of PBC
problem in robotics and its achievement using energy tanks is reviewed,
followed by a discussion regarding limitations of the methodology. In Sec. III
the proposed scheme is presented, and validated in Sec. IV by means of
different simulations. Conclusions and future works are sketched in Sec. V.

##  II Background and Motivation

We review, restricted to robotics, the motivation underlying PBC, and how
energy tanks formally solve the problem of passivizing an arbitrary control
input. We then discuss the limits of the approach, which are tackled with the
subsequent RL integration.

###  II-A Passivity based control and energy tanks

Consider the dynamic model of an $n$-DoF robot in Lagrangian joint-space
coordinates:

$M(q)\ddot{q}+C(q,\dot{q})\dot{q}+D(q)\dot{q}+\frac{\partial V(q)}{\partial
q}=\tau$ (1)

where $q\in\mathbb{R}^{n}$ are the joint coordinates, $M(q)=M^{\top}(q)>0$ is
the inertia tensor, $C(q,\dot{q})$ is the matrix collecting centrifugal and
Coriolis terms, $D(q)=D^{\top}(q)\geq 0$ is the matrix collecting friction
coefficients, $V:\mathbb{R}^{n}\to\mathbb{R}$ maps the joint coordinates into
the total conservative potentials (e.g., gravity and elastic springs), and
$\tau\in\mathbb{R}^{n}$ are the generalized forces at the joints, collocated
with the joint coordinates $q$. The time derivative of total mechanical energy
$E(q,\dot{q})=\frac{1}{2}\dot{q}^{\top}M(q)\dot{q}+V(q)$ verifies the
inequality

$\dot{E}=\dot{q}^{\top}\tau-\dot{q}^{\top}D(q)\dot{q}\leq\dot{q}^{\top}\tau$
(2)

which is a statement of passivity for system (1) with the energy function
$E(q,\dot{q})$ as storage function. Inequality (2) states that system (1) is
passive with respect to its energy function $E(q,\dot{q})$, and its so called
power port $(\tau,\dot{q})$, an input-output pair whose pairing
$\dot{q}^{\top}\tau$ produces the scalar power flow undergoing the system.
Once (2) is integrated in time, the left hand side of the equation $E(t)-E(0)$
represents the stored energy in the system, which is always less or equal than
the supplied energy through the power port, represented by the right term
$\int_{0}^{t}\dot{q}(s)^{\top}\tau(s)ds$. The positive dissipated energy
$\int_{0}^{t}\dot{q}(s)^{\top}D(q(s))\dot{q}(s)ds$ determines the convergence
rate to the equilibrium, and acts as a passivity margin, i.e., the higher it
is, the bigger is the margin for which the passivity inequality (2) results
satisfied111If $D=0$, (2) holds with equal sign and the system is denoted as
lossless.. If applied to a (open-loop) physical system like (1), passivity
just represents the thermodynamic statement of energy conservation, while if
applied to a controlled system, passivity implies stability for the minimum of
the storage function under weak conditions qualifying the latter as a Lyapunov
function for the equilibrium [2, 4]. Next we illustrate the full motivation
behind the necessity of achieving a passive closed-loop system.

#### Passivity as must

To understand the motivation behind a passive design consider system (1) as an
open one, i.e., a system that, independently on the way it is controlled, can
interact with a new, dynamically unknown system, that we call the
s-environment222We avoid calling this system simply ”environment”, since the
latter will refer to the concept of environment in the RL framework.. This is
shown by considering system (1) in which we distinguish the torque
contributions in two terms, i.e.,

$\tau=\tau_{c}+\tau_{e},$ (3)

where $\tau_{c}$ are the control torques that can be directly applied at the
joints by means of the actuators, while $\tau_{e}=J^{\top}(q)F$ corresponds to
the torques at the joints produced by an external interaction at the end-
effector of the robot. Here $F\in\mathbb{R}^{6}$ is a vector collecting forces
and torques at the end-effector in the work-space and $J(q)$ is the Jacobian
matrix of the robot. The latter relates joint-space and work-space velocities
and forces as the dual relations $\dot{x}=J(q)\dot{q}$ and
$\tau_{e}=J^{\top}(q)F$, where $\dot{x}\in\mathbb{R}^{6}$ is the rate of work-
space coordinates $x$, collocated with the forces $F$, resulting in
$\tau_{e}^{\top}\dot{q}=F^{\top}\dot{x}$. Specializing the power balance (2)
to the uncontrolled system (1) with the split (3) one obtains

$\dot{E}=\dot{q}^{\top}\tau_{c}+\dot{x}^{\top}F-\dot{q}^{\top}D(q)\dot{q}\leq\dot{q}^{\top}\tau_{c}+\dot{x}^{\top}F,$
(4)

i.e., the robotic system (1) is passive with respect to two different power
ports: $(\tau_{c},\dot{q})$, whose pairing is the power flow undergoing the
actuators, and $(F,\dot{x})$, whose pairing is the power flow undergoing the
s-environment, an external dynamical system with its own (unknown) dynamics.
In this context the PBC objective can be described as follows

######  PBC objective.

Given the system (1) with the split (3) inducing power balance (4), design a
control law on the port $(\tau_{c},\dot{q})$, possibly by designing a
controller as a dynamical system in its own right, with state $x_{c}$ and
energy $V_{c}(x_{c})$, such that for the closed-loop storage function
$\mathcal{E}(q,\dot{q},x_{c})=E(q,\dot{q})+V_{c}(x_{c})$, the passivity
condition

$\dot{\mathcal{E}}\leq\dot{x}^{\top}F$ (5)

is verified, i.e., that the controlled system is passive with respect to the
s-environment port $(F,\dot{x})$.

This objective is considered of utmost importance (from which the passivity as
must claim [1]) since, when an interaction takes place ($F\neq 0$) the
dynamics of the closed-loop system deeply depends on the dynamics of the
s-environment, which cannot be modelled in an oversimplified way, e.g.,
assuming linearity. This claim is particularly valid if the task to be
executed takes place in an unstructured, hazardous scenario where sources of
disturbances like unforeseen interactions with humans are possible.

The passive design (5) aims at excluding the possibility that in the moment
the robot interacts with a passive s-environment, an unbounded amount of
energy can be produced during the interaction.333We refer to [1] for a
converse proof: if PBC is not achieved, there exists a passive s-environment
destabilising the controlled system. This property is referred to as contact
stability. For completeness, we report that passivity induces also a stronger
property than contact stability, which is sometimes referred to as robust
stability [17]: even if the s-environment is not a passive system, contact
stability can be assured if the passivity margin of the controlled system
covers the energy generated by the active s-environment. All these properties
follow from the fact that passive systems are closed under power-preserving
interconnection [2, 3], and we refer to [2] for further relations between
passivity and different system theoretic stability properties. To conclude, in
this work we are aligned with the necessity of achieving the PBC objective
when a robotic task needs to be performed.

#### Energy Tanks

An extremely effective method to achieve the PBC objective relies on the
energy tank control algorithm, which we describe next and whose basic
schematics is represented in Fig. 1.

Figure 1: The essential of the energy tank algorithm.

Mathematically an energy tank is a dynamical system which constitutes an
atomic energy storing element. It can be represented as the (lossless) $1$D
system:

$\displaystyle\dot{x}_{c}$ $\displaystyle=u_{\textrm{t}}$ (6) $\displaystyle
y_{\textrm{t}}$ $\displaystyle=\frac{\partial V_{c}(x_{c})}{\partial x_{c}},$
(7)

where the energy function is simply $V_{c}(x_{c})=\frac{1}{2}x_{c}^{2}$, and
as a consequence it presents a power port $(u_{\textrm{t}},y_{\textrm{t}})$
since $\dot{V}_{c}=y_{\textrm{t}}u_{\textrm{t}}$. The tank (6-7) is
interconnected to (1) in a way that allows the implementation of some control
action which fulfills the execution of some task, and at the same time meeting
the desired passivity constraint. This is possible thanks to a suitable power-
preserving interconnection between the two systems, defined as follows:

$\begin{pmatrix}\tau_{c}\\\
u_{\textrm{t}}\end{pmatrix}=\begin{bmatrix}0&\frac{w}{x_{c}}\\\
-\frac{w^{\top}}{x_{c}}&0\end{bmatrix}\begin{pmatrix}\dot{q}\\\
y_{\textrm{t}}\end{pmatrix},$ (8)

where $w\in\mathbb{R}^{n}$ is the desired task-dependent control action to be
passively implemented. This interconnection, which is the core of energy tank
algorithm, produces two effects: i) the desired action $w$ is correctly
implemented, i.e., from the side of the robot (1), one obtains

$M(q)\ddot{q}+C(q,\dot{q})\dot{q}+D(q)\dot{q}+\frac{\partial V(q)}{\partial
q}=w+J^{\top}(q)F;$ (9)

and ii) at every time the mechanical power exiting the robot produced by the
actuators $\dot{q}^{\top}\tau_{c}$ leaves the energy tank since

$\dot{V}_{c}=y_{\textrm{t}}u_{\textrm{t}}=-w^{\top}\dot{q}=-\dot{q}^{\top}w=-\dot{q}^{\top}\tau_{c}.$
(10)

Evaluating the variation of the closed-loop storage function
$\mathcal{E}=E+V_{c}$ one achieves the PBC objective (5), which shows that the
interconnection achieves indeed a passive closed-loop system with respect to
the s-environmental port. The scalar $V_{c}(x_{c})$ indicates the amount
energy that is still at disposal of the control mechanism implementing the
action $w$ before loosing the described formal passivity. In fact the
interconnection (8), which is modulated by the tank state, becomes singular
when $x_{c}$, and thus $V_{c}(x_{c})$ are zero, representing the moment in
which it becomes impossible to passively perform the desired action $w$. To
meaningfully implement the energy tank algorithm, it is thus needed to
constantly observe the energy in the tank in order to implement a control
action $\bar{w}$, rather than $w$, where

$\bar{w}=\begin{cases}w,&\text{if }V_{c}(x_{c})\geq\epsilon\geq 0\\\
0,&\text{if }V_{c}(x_{c})\leq\epsilon,\end{cases}$ (11)

for some small positive energy level $\epsilon$. In this way formal passivity
of the closed-loop system is recovered completely since the system is just
detached from the controller in the moment in which no energy budget is left.

Before discussing the peculiarities and limitations of the described
algorithm, we introduce the concept of task energy, that we will denote by
$e^{*}$, firstly defined in [19], which is the minimum energy in the tank
necessary to fulfill passively some task. In other words, if
$V_{c}(x_{c}(t))|_{t=0}>e^{*}$, the tank never depletes along the whole task
horizon. It is important to notice that $e^{*}$ does not depend on the task
only, but also on the specific tank dynamics which is chosen: the choice
$u_{\textrm{t}}$ in (8) is not unique, and many variations have been proposed
[5] (e.g., recirculation in the tank of dissipation) which would change the
value of task energy for the same task.

#### Comments and limitations

i) The energy tank algorithm constitutes a clever way to disembody from any
physical dynamics the implementation of a passive control action. In fact
there is no need to design the controller in a way that it mimics a physical
system (e.g., in the PD case the controller can be seen as a control spring
and damper): as long as a task is achieved though a control input $w$ and
$e^{*}$ is finite, a passive implementation is possible by means of the energy
tank algorithm, just setting $V_{c}(x_{c}(t))|_{t=0}>e^{*}$; ii) The
aforementioned initial energy assignment is a degree of freedom in the
algorithm, whose implications are often naively addressed. In fact a very high
(yet bounded) energy initialisation in the tank would technically still
fulfill the PBC objective (5), yet creating so called ”practically unstable
behaviours” [17, 5]. As a consequence, a naive tank design de facto makes
robust stability a property which is and not connected to any safety
guarantee. In fact notice that till the energy in the tank is not depleted,
the control input $w$ is completely transparent to the tank algorithm, which
reduces to a trick to formally proof passivity, with limited significance in
the context in which tasks need to be performed in unstructured scenarios. The
fact that the mechanical power flow is undefined in sign (i.e., when
$\dot{q}^{\top}\tau$ the tank gets refilled) worsens this criticality, which
is sometimes addressed with empirical tank saturation arguments.

To conclude the discussion, both the task energy $e^{*}$ and the control
action $w$ are often difficult to be determined a priori, and these are not
independent variables. For complex task executions it is reasonable to take
advantage of simulations to optimise for both variables in a combined way. If
the PBC objective can be naively achieved just by initialising the energy in
the tank to a sufficiently high value, what we claim being an useful system
theoretic property in the energy tank context is the achievement of the PBC
objective combined with an estimation of the task energy. In fact the
knowledge of the latter would lead to a meaningful energy tank inisialisation,
so that a depletion of the tank can be used as a diagnostic tool to detect an
important divergence from nominal task execution, beyond formally achieving a
passive closed-loop system.

###  II-B Reinforcement learning

Reinforcement learning (RL) [20] is a model-free framework, consisting of an
agent interacting with an environment, to solve optimal control problems
stated as Markov decision processes (MDPs). MDPs are mathematical formulation
of decision making in situations where outcomes are partly stochastic and
partly under the control of a decision maker. A MDP is defined by the tuple
$(\mathcal{S},\mathcal{A},p,r)$: at any given state $s_{k}\in\mathcal{S}$ at
time step $k$, the agent chooses and executes an action $a_{k}\in\mathcal{A}$
according to a learnt policy $\pi(a_{k}|s_{k})$, then the environment
transitions to a new state $s_{k+1}\in\mathcal{S}$ with the unknown state
transition probability $p(s_{k+1}|s_{k},a_{k})$ and produces a reward
$r_{k}=r(s_{k},a_{k})$. An important condition that characterize the MDPs is
that the state transition probability satisfies the Markov property
$p(s_{k+1}|s_{k},a_{k})=p(s_{k+1}|s_{k},a_{k},\dots,s_{1},a_{1})$ for any
trajectory $s_{1},a_{1},\dots,s_{k},a_{k}$, which means that the environment
is memoryless because the transition to the next state depends only on the
current state and action. The Markov property is usually fundamental to
guarantee the convergence to the optimal policy in RL algorithms [20]. The
goal of a RL algorithm is to find an optimal policy $\pi^{*}$ that maximizes
the expected $\gamma$-discounted sum of future rewards (that we define as
return).

While the proposed framework could be combined with any RL algorithm, a
benchmarking is out of the scope of this work, for this reason in the rest of
the paper we only consider the off-policy soft actor-critic (SAC) algorithm
[21] with continuous action spaces.

##  III Energy tanks meet RL

The arbitrariness of the control input $w$ and its disembodiment from any
physical dynamics, are a tempting motivation to choose it as decision variable
in an optimisation framework. Indeed, the output of a RL control policy can be
directly mapped to the torques $w$ commanded at an actuator level, as shown in
Fig. 2. This map can be trivially the identity (in case the generalised forces
are directly learned in the RL scheme) or be represented by a transducer
(e.g., an internal PID controller that maps a position or velocity reference
to the commanded torques $w$), which is often shown to critically improve
sample efficiency in the RL framework [22]. No specific transducer between the
action space and $w$ is required to implement the proposed scheme, as long as
the commanded torques $w$ are accessible. Since MDPs are discrete-time
processes, we start by exploiting a simple but powerful result that allows to
reproduce the energy tank algorithm at the discrete time level.

#### The energy sampling approach

Let us denote $e_{k}$ the level of energy in the tank at the discrete time
step $k$, i.e., $e_{k}=V_{c}(kT)$. The energy in the tank at the next time
step, initialized by $e_{0}$, is updated according to the rule

$e_{k+1}=e_{k}-\Delta e_{k+1}$ (12)

where $\Delta e_{k+1}$ is the amount of energy exiting the tank at time $k+1$,
which approximates, at a discrete-time level, the energy leaving the tank in a
sampling interval with duration $T$, i.e.,
$\int_{Tk}^{T(k+1)}-\dot{V}_{c}(s)\,\,ds$. With a look at the continuous
energy balance (10), this energy equals the sum of energies used by each
actuator of the robot in the time interval $[Tk,T(k+1))$:

$\int_{Tk}^{T(k+1)}-\dot{V}_{c}(s)ds=\int_{Tk}^{T(k+1)}w(s)^{\top}\dot{q}(s)ds.$

Now, assuming the torque signal is constant with value $w_{k}$ along the
$k$-th sampling interval, and indicating with $q_{k}:=q(Tk)$, we can further
massage the previous expression to define $\Delta e_{k+1}$ as:

$w_{k}^{\top}\int_{Tk}^{T(k+1)}\dot{q}(s)ds=w_{k}^{\top}(q_{k+1}-q_{k})=:\Delta
e_{k+1}.$ (13)

It is worth remarking that, in the common conditions of constant torque along
the sampling time interval and position sensor collocated to the motor, such
defined quantity produces the exact amount of injected energy by the actuators
independently of the value of $T$, and not an estimate. This property makes
the connection between the discrete time RL framework and the energy tank
framework particularly appealing, since the tank algorithm can exactly be
reproduced at a discrete time level without the need of integrating a tank
dynamics (6) with the input defined in (8).

The definition of $\Delta e_{k+1}$ in (13) exactly reproduces at a discrete
time level the tank algorithm as reviewed in Sec. II, but this choice is not
unique, and variations are possible on the basis of the passivity margin that
one wants to achieve for the closed-loop system. In the sequel of this work we
want to prevent a refilling of the tank, and thus use the following update
rule

$\Delta e_{k+1}:=\begin{cases}w_{k}^{\top}(q_{k+1}-q_{k})&\text{ if
}w_{k}^{\top}(q_{k+1}-q_{k})\geq 0\\\ 0&\text{ otherwise }\end{cases}$ (14)

Such an update rule physically represents the impossibility of recovering
negative mechanical energy flowing into the system, which is a reasonable
assumption for many mechanical systems. Furthermore this choice serves best to
our purpose of interpreting the task energy as a metabolic metric to assess an
initial energy budget: the energy in the tank can only decrease along the task
execution.

In this discrete time implementation of energy tanks we aim to achieve the
following finite difference version of the PBC objective (5). We indicate
$E_{k}=E(kt)$ the sampled mechanical energy of the robot and
$\mathcal{E}_{k}=E_{k}+e_{k}$ the sampled closed-loop storage function.

######  Sampled PBC objective.

Given the system (1) with the split (3) inducing power balance (4), design a
control policy inducing desired control action $\tau_{c}=w_{k}$ such that the
condition

$\mathcal{E}_{k+1}-\mathcal{E}_{k}\leq\int_{Tk}^{T(k+1)}\dot{x}(s)^{\top}F(s)ds$
(15)

is verified $\forall k$, i.e., that the sampled difference of closed-loop
storage function is bounded by the energy injected in a sampling interval by
the s-environment through the port $(F,\dot{x})$.

######  Theorem.

Assuming constant generalised forces $w_{k}$ in the sampling interval
$[Tk,T(k+1)]$, the PBC objective (15) is achieved using the tank update rule
(14).

###### Proof.

Define $R_{k+1}:=\int_{Tk}^{T(k+1)}\dot{q}(s)^{\top}D(q(s))\dot{q}(s)ds$ the
dissipated power by friction in a sampling interval. Using the fact that
$w_{k}$ is constant in a sampling interval, (12) and (13), it holds
$\mathcal{E}_{k+1}-\mathcal{E}_{k}=E_{k+1}-E_{k}+e_{k+1}-e_{k}=-R_{k+1}+w_{k}^{\top}(q_{k+1}-q_{k})+\int_{Tk}^{T(k+1)}\dot{x}(s)^{\top}F(s)ds-\Delta
e_{k+1}$. Condition (15) follows from (14) since $R_{k+1}\geq 0$. ∎

Note that the update rule (14) induces a stronger passivity margin with
respect to (13) since when $\Delta e_{k+1}=0$ the term
$w_{k}^{\top}(q_{k+1}-q_{k})$ is negative, and as such acts as dissipation.

We define the energy spent at step $k$ as the sum of the energies exiting the
tank up to that step:

$\hat{e}_{k}:=\sum_{l=0}^{k-1}\Delta e_{l+1}.$ (16)

Since with the update rule (14) the sequence $\hat{e}_{k}$ is monotonically
non decreasing, the task energy $e^{*}$ over a task with $N$ time steps can be
simply calculated as

$e^{*}=\hat{e}_{N}.$ (17)

We describe now two procedures combining the tank and the RL algorithm. The
first one aims at passivizing a control policy which was previously learned
agnostically to any passive design. The second exploits the tank architecture
also in training phase, and produces by construction a passive learned policy.

Figure 2: RL with energy tank passivization

###  III-A Passivization in Inference (Passivizing learned policies)

We indicate with the term inference the phase where the training is finished
and the agent implements the learnt policy, without exploring anymore. The
combination of the energy tank architecture and the reinforcement learning
technique in inference is quite straightforward, as an RL agent represents in
this context an arbitrary controller that the tank algorithm is able to
passivize. In fact, we can use any previously trained agent in a generic
environment and passivize the controlled system in inference by wrapping the
control action with (11), where the energy in the tank is updated according to
(12) and (14). Furthermore, an agent able to fulfill a given task in a generic
environment with a specific task energy $e^{*}$, is also able to passively
fulfill that task in an environment where the control action is wrapped with
(11) and the energy tank is initialized with a value greater than $e^{*}$.
This represents indeed the only design requirement to preserve the agent’s
performance in executing the task with the learnt policy while achieving
formal passivity (15). In order to equip the controlled system with a
meaningful passivity property, it is important to initialize the tank with an
amount of energy which slightly exceeds $e^{*}$, which can be estimated by
running the agent without (11) for a number of episodes and measuring the
maximum consumption of energy along all the episodes. The proposed procedure
provides a constructive way to achieve the PBC objective, and brings the
additional benefit of the task energy estimation, which produces a meaningful
closed-loop passivisation. This step provides a diagnostic tool indicating a
non regular task execution in case of tank depletion, which is an extra
information that a naive tank design lacks.

######  Note.

It is meaningful to compare the described procedure with the works [17, 18],
which aim at passivizing an already given control input, and as such can also
be seen as a passivisation in inference algorithm. In [17, 18] the task energy
is not explicitly estimated and as a consequence the initialisation of the
tank (whose non depletion constraint in an optimisation problem is what
produces the passive approximation) is arbitrary. By solving the optimisation
with different tank initialisation, in [17] different passive approximations
are achieved. Here instead, motivated by the fact that passivity can be
achieved anyhow for any finite task energy and that the learnt policy
represents the way the task is optimally executed, we first estimate the task
energy to constraint the degree of freedom of tank initialisation.

###  III-B Passivization in Training (Learning passive policies)

In many circumstances it might be desirable to continuously achieve a passive
closed-loop system also in training phase, i.e., when the task-dependent
control policy is being learned. This is desirable for instance when the
training takes place in real life, and not in simulations, such that initial
exploration phases are guaranteed not to undergo hazardous unbounded energy
generation. Furthermore, the passivized training phase endows the agent with
awareness of the metabolic spending encoded in the tank architecture, induced
by the initial energy budget $e_{0}$. As a consequence, the learned policy
will be influenced by the tank initialisation $e_{0}$, and in particular the
task energy $e^{*}$ will be directly learned together with the control policy
in a combined way, strengthening the significance of the resulting passivity
property. This mechanism provides to the scheme a clear biomimetic
perspective, since the energy budget and the task-dependent policy are not
independent variables.

The proposed procedure is achieved by implementing the learning in the
extended RL framework depicted in Fig. 2., i.e., the standard RL scenario with
the tank algorithm embedded. We recognise the following technical issue which
needs to be addressed before implementing the training.

#### Loss of Markovian property

Notice that the current value of the tank state $e_{k}$ depends on the entire
state-action trajectory of the system from the beginning to step $k$. This is
true due to the update rule (12), where $\Delta e_{k+1}$ depends both on the
joint position $q_{k}$ (which in most robotics applications is part of the RL
state $s_{k}$) and the action $a_{k}$. Furthermore the value $e_{k}$ is able,
due to (11), to influence the future dynamics of the environment in case of
tank depletion. For these reasons the new environment that the agent perceives
by including also the energy tank architecture as in Fig. 2 does not preserve
the memorylessness of the original environment, or in other words it looses
the Markov property. If Markov property is not satisfied, most of RL algorithm
loose formal proof of convergence to the optimal policy, and long-term
prediction performance can degrade when the one-step predictions defining the
Markov property become inaccurate [23]. We identify two solutions to restore
the Markov property.

#### Extended State

A common practice when the environment is non-Markovian because a relevant
time-dependent part of information is missing from the agent’s state but
accessible, is simply to include it in the state [23]. However, under this
formulation the agent might require a dedicated tuning process to maintain
similar performance.

#### Extended Termination

As an alternative, the episode’s termination condition can be extend to the
situation in which the energy in the tank depletes. In case a robotic platform
is involved, terminating the episode would mean breaking the joints, instead
of giving zero torque as done in the original formulation. After the
termination, the environment is reset and a new episode can start with a
restored level of energy in the tank. In this formulation, the Markov property
is restored without changing the state and the reward. In fact the termination
of the episode removes the switch behavior (11), and the agent never meets a
state influenced by the free dynamics. The reward function has to be non
negative, since the length of an episode is not fixed. In fact, using a
negative reward, the agent might attempt to maximize the cumulative return by
learning actions leading to tank depletion.

##  IV Simulations

Figure 3: DoorOpening environment

We consider two environments implemented in MuJoCo physics simulator [24]:

####  IV-1 DoorOpening

Here, we adopted an implementation from Robotsuite simulation framework [11]
where a 7-DoF robotic arm must learn to open a door by turning the handle as
shown in Fig. 3. The door location is randomized at the beginning of each
episode. The robot is a Panda Franka Emika which mounts a parallel-jaw gripper
equipped with two small finger pads. The torques provided to the 7 joints of
the robot are generated using a proportional control law
$\tau=k_{p}(\dot{q}_{d}-\dot{q})$, with $k_{p}=0.03$, that follow a target
joint velocity $\dot{q}_{d}$ provided by the agent at $50Hz$.

####  IV-2 Pendulum

In this environment we created a pendulum composed of a rod suspended by one
extremity from a pivot actuated by a motor controlled in torque with control
frequency of $50Hz$. The rod is subject to gravity while friction loss is
present in the joint. Let us denote the joint angle as $\beta$. In the
reference configuration $\beta=0$ the pendulum heads horizontally right. At
the beginning of each episode the angle is randomly initialized
$\beta\sim\mathcal{N}(-\pi/2,0.05\pi)$. The environment state is defined as
the vector
$s_{k}=\begin{bmatrix}\sin\beta_{k}&\cos\beta_{k}&\tanh\dot{\beta}_{k}\end{bmatrix}^{\top}$,
while the agent action corresponds to the torque applied to the joint. Since,
the inverted pendulum problem consists on bringing the rod the inverted
position ($\sin\beta^{*}=1$) and holding it there as long as possible, we
design a positive reward function
$r_{k}=(1+\left|\sin\beta_{k}-1\right|+0.1\left|\tanh\dot{\beta}_{k}\right|+0.01\left|\tau_{k}\right|)^{-1}$
which is inversely proportional to a weighted sum of the absolute values of
position error ($\sin\beta_{k}-1$), velocity error ($\tanh\dot{\beta}_{k}$)
and torque ($\tau_{k}$). These last two components help to remove oscillatory
behaviours in the transition phase and at the equilibrium point.

TABLE I: Hyperparameters | DoorOpening | Pendulum
---|---|---
Actor Learning Rate | 0.01 | 0.005
Critic Learning Rate | 0.0005 | 0.005
Soft Update Coefficient | 0.005 | 0.003
Batch Size | 128 | 256
Number of Epochs | 2000 | 200
Steps per Epoch | 2500 | 2500
Steps per Trajectory | 500 | 500
Steps Before Training | 3300 | 2500
Target Update Period | 5 | 5
Gradient Steps | 1000 | 500
Neurons per Layer | [256,256] | [256,256]

In all the simulations presented in this work the agents are trained using SAC
algorithm implemented in PyTorch and trained with an NVIDIA GeForce GTX 1080
Ti on an Intel Core i7-7700K CPU clocked at 4.20GHz. In Table I the training
configurations for the both the environments are reported. Additionally,
generalized State Dependent Exploration (gSDE) is employed, where a new noise
matrix is sampled every episode. The entropy regularization coefficient is
learned automatically as done in [25].

In the rest of this section we expose the results obtained from the
simulations run on the two environment introduced above. In particular, the
passivization of a policy learned on the DoorOpening and the learning of a
passive policy on Pendulum are discussed respectively in IV-A and IV-B. In
Table II the notation adopted for each agent in training and inference is
briefly reported.

TABLE II:  $\phi_{\infty}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent trained without passivization
---|---|---
$\phi_{e_{0}}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent trained with passivization and tank initialized at $e_{0}$
$\phi_{\infty|\infty}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent $\phi_{\infty}$ in inference without passivization
$\phi_{\infty|e^{*}}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent $\phi_{\infty}$ in inference with passivization and tank initialized at $e^{*}$
$\phi_{e_{0}|e^{*}}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent $\phi_{e_{0}}$ in inference with passivization and tank initialized at $e^{*}$
$\phi_{\infty|\infty,\delta}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent $\phi_{\infty}$ in inference with external force field and without passivization
$\phi_{\infty|e^{*},\delta}$ | $\overset{\underset{\mathrm{\Delta}}{}}{=}$ |  agent $\phi_{\infty}$ in inference with external force field, passivization and tank initialized at $e^{*}$

###  IV-A Passivization in Inference

Figure 4: Average energy spent in DoorOpening in inference.

To study the effects of the passivization in inference, an external field of
force is applied to the system as a way to perturb the nominal task. The
magnitude of the force is chosen in order to let the tank deplete, when it is
initialized with an amount of energy corresponding to the task energy measured
in the nominal case. For this experiment we consider the DoorOpening
environment and we employ a pretrained model available
online444https://github.com/ARISE-Initiative/robosuite-benchmark. As visible
in Fig. 4, under the effect of the external force the agent without the
passivization in inference $\phi_{\infty|\infty,\delta}$ can spend an
unbounded amount of energy, while the consumption is limited when the same
agent is wrapped with passivization $\phi_{\infty|e^{*},\delta}$. The energy
tank is initialized with the task energy $e^{*}$ measured as the maximum
$\hat{e}_{k}$ over the $100$ episodes in the agent without the external force
and without passivization $\phi_{\infty|\infty}$. From this simulation we can
better comprehend and appreciate the versatility of the proposed framework. In
fact, an agent trained by third party to optimize the process in a non-passive
way, is readily passivized by wrapping the decision variable as detained in
III-A and without need of retrain/modify the agent.

###  IV-B Passivization in Training

((a))

((b))

((c))

((d))

Figure 5: In (a) and (b) respectively the average returns obtained and the
energy spent in training. In (c) and (d) respectively the average position
error and energy spent in inference.

Let us consider two agents $\phi_{\infty}$ and $\phi_{e_{0}}$ trained in the
Pendulum environment sharing the same hyperparameter configuration, but
$\phi_{e_{0}}$ is passivized with the framework of Sec. III-B and the Extended
Termination method. For each simulation on the Pendulum, 5 instances with
different random seeds for the pseudo random generators are
trained.Implementing the training with a sufficiently low $e_{0}$ provides
some level of meaningful robust stability in training phase. Furthermore we
aim to analyse how the energy budget $e_{0}$ imposed in training influences
the learned policy and the task energy $e^{*}$, which are correlated
variables.

We choose as $e_{0}$ the task energy estimated from $\phi_{\infty}$, measured
in inference as the maximum $\hat{e}_{k}$ over $100$ episodes. As visible in
Fig. 5(d), the task energy $e^{*}$ estimated by $\phi_{e_{0}}$ is lower than
the energy that the system spends to perform the task when using the policy
$\phi_{\infty}$. Indeed we appreciate how the agent $\phi_{e_{0}}$ learns to
perform the task with lower energy consumption. Furthermore, the average
return of the two agents during the training visible in Fig. 5(a) converges to
the same value (almost $2500$), while $\phi_{\infty}$ can arrive to spend a
level of energy that is almost $50$ times greater than the one spent in
$\phi_{e_{0}}$, see Fig. 5(b). Clearly, the episode terminations caused by the
energy constraints (i.e., when the tank depletes), disturb the exploration
during the training, which results in a slower convergence to the final
policy. Fig. 5(c) shows a comparison in inference of the two agents where a
measure of the position error $d_{k}=1-\sin(\beta_{k})$ is adopted as a
success metric of the task. As visible both the agents converges to the same
error, with a smoother trajectory for $\phi_{e_{0}}$.

##  V Conclusions and Future Work

We introduced a framework merging the energy tank algorithm, used as a tool to
passivize arbitrary control schemes, and reinforcement learning, representing
the most versatile method to learn control policies along complex tasks. The
presented procedures allow to passivize any control policy, and to learn
constructively passive policies, tackling the problem represented by a lack of
optimisation in the design phase of passive controllers. As future work, we
are we are processing a throughout study of energy-aware robotics in the
proposed neural framework, exploiting the energy tank architecture to embed
metabolic and safety metrics, which depend on physical energy and power flows
undergoing the robot. This goes beyond passive designs, which is necessary
step for many important tasks which require continuous energy injection (e.g.,
periodic locomotion), and because safety and passivity, although sometimes
naively considered equivalent, are distinct concepts.

## References

  * [1] Stefano Stramigioli  “Energy-Aware robotics”  In _Lecture Notes in Control and Information Sciences_ 461, 2015, pp. 37–50
  * [2] Arjan Schaft  “L2-Gain and Passivity Techniques in Nonlinear Control”  Springer Publishing Company, Incorporated, 2016
  * [3] Vincent Duindam, Alessandro Macchelli, Stefano Stramigioli and Herman Bruyninckx  “Modeling and Control of Complex Physical Systems”  In _Modeling and Control of Complex Physical Systems_ , 2009
  * [4] Romeo Ortega and Eloísa García-Canseco  “Interconnection and damping assignment passivity-based control: A survey”  In _European Journal of Control_ 10.5, 2004, pp. 432–450
  * [5] Federico Califano, Ramy Rashad, Cristian Secchi and Stefano Stramigioli  “On the Use of Energy Tanks for Robotic Systems”  In _Human-Friendly Robotics 2022_ Cham: Springer International Publishing, 2023, pp. 174–188
  * [6] Cristian Secchi, Stefano Stramigioli and Cesare Fantuzzi  “Position drift compensation in port-Hamiltonian based telemanipulation”  In _IEEE International Conference on Intelligent Robots and Systems_ IEEE, 2006, pp. 4211–4216
  * [7] Vincent Duindam and Stefano Stramigioli  “Port-based asymptotic curve tracking for mechanical systems”  In _European Journal of Control_ 10.5 Elsevier, 2004, pp. 411–420
  * [8] Fotios Dimeas and Nikos Aspragathos  “Online Stability in Human-Robot Cooperation with Admittance Control”  In _IEEE Transactions on Haptics_ 9.2, 2016, pp. 267–278
  * [9] Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez and Thomas Funkhouser  “Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning”, 2018
  * [10] Avi Singh, Larry Yang, Chelsea Finn and Sergey Levine  “End-To-End Robotic Reinforcement Learning without Reward Engineering”, 2019
  * [11] Yuke Zhu, Josiah Wong, Ajay Mandlekar and Roberto Martín-Martín  “robosuite: A Modular Simulation Framework and Benchmark for Robot Learning”, 2020
  * [12] Lucian Buşoniu, Tim Bruin, Domagoj Tolić, Jens Kober and Ivana Palunko  “Reinforcement learning for control: Performance, stability, and deep approximators”  In _Annual Reviews in Control_ 46, 2018, pp. 8–28
  * [13] Erfan Shahriari, Lars Johannsmeier and Sami Haddadin  “Valve-based Virtual Energy Tanks: A Framework to Simultaneously Passify Controls and Embed Control Objectives”  In _Proc. of the American Control Conference_ 2018-June, 2018, pp. 3634–3641
  * [14] Federico Benzi, Maximilian Brunner, Marco Tognon, Cristian Secchi and Roland Siegwart  “Adaptive Tank-based Control for Aerial Physical Interaction with Uncertain Dynamic Environments Using Energy-Task Estimation”  In _IEEE Robotics and Automation Letters_ 7.4 IEEE, 2022, pp. 9129–9136
  * [15] Erfan Shahriari, Lars Johannsmeier, Elisabeth Jensen and Sami Haddadin  “Power Flow Regulation, Adaptation, and Learning for Intrinsically Robust Virtual Energy Tanks”  In _IEEE Robotics and Automation Letters_ 5.1 IEEE, 2020, pp. 211–218
  * [16] Federico Califano, Daniel van Dijk and Wesley Roozing  “A Task-Based Post-Impact Safety Protocol Based on Energy Tanks”  In _IEEE Robotics and Automation Letters_ 7.4, 2022, pp. 8791–8798
  * [17] Federico Benzi, Federica Ferraguti, Giuseppe Riggio and Cristian Secchi  “An Energy-Based Control Architecture for Shared Autonomy”  In _IEEE Transactions on Robotics_ , 2022, pp. 1–19
  * [18] Beatrice Capelli, Cristian Secchi and Lorenzo Sabattini  “Passivity and Control Barrier Functions: Optimizing the Use of Energy”  In _IEEE Robotics and Automation Letters_ 7.2 IEEE, 2022, pp. 1356–1363
  * [19] Christopher Schindlbeck and Sami Haddadin  “Unified passivity-based Cartesian force/impedance control for rigid and flexible joint robots via task-energy tanks”  In _Proc. - IEEE International Conference on Robotics and Automation_ 2015-June.June IEEE, 2015, pp. 440–447
  * [20] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”, 2018
  * [21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “[SAC] Soft actor-critic”  In _35th International Conference on Machine Learning, ICML 2018_ 5, 2018, pp. 2976–2989
  * [22] Laura Smith, Ilya Kostrikov and Sergey Levine  “A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning”, 2022
  * [23] Steven D. Whitehead and Long Ji Lin  “Reinforcement learning of non-Markov decision processes”  In _Artificial Intelligence_ 73.1-2, 1995, pp. 271–306
  * [24] Emanuel Todorov, Tom Erez and Yuval Tassa  “MuJoCo: A physics engine for model-based control”  In _IEEE International Conference on Intelligent Robots and Systems_ IEEE, 2012, pp. 5026–5033
  * [25] Tuomas Haarnoja et al.  “Soft Actor-Critic Algorithms and Applications”, 2018