# Intrinsic Motivation in Dynamical Control Systems

Stas Tiomkin  Computer Engineering Department, Charles W. Davidson College of
Engineering San Jose State University, CA, 95192  Ilya Nemenman  Department of
Physics  Department of Biology  Initiative in Theory and Modeling of Living
Systems, Emory University, Atlanta, GA 30322, USA  Daniel Polani  Adaptive
Systems Research Group, University of Hertfordshire, Hatfield, UK  Naftali
Tishby Prof. Naftali Tishby passed away when this work was in development.
This project began under his leadership when Stas Tiomkin was a PhD student in
his group. The rest of the authors agree that he should be a senior author on
this manuscript, but his consent for this was not obtained. The Rachel and
Selim Benin School of Computer Science and Engineering  Edmond and Lilly Safra
Center for Brain Sciences (ELSC), Hebrew University of Jerusalem, 96906 Israel

###### Abstract

Biological systems often choose actions without an explicit reward signal, a
phenomenon known as intrinsic motivation. The computational principles
underlying this behavior remain poorly understood. In this study, we
investigate an information-theoretic approach to intrinsic motivation, based
on maximizing an agent’s empowerment (the mutual information between its past
actions and future states). We show that this approach generalizes previous
attempts to formalize intrinsic motivation, and we provide a computationally
efficient algorithm for computing the necessary quantities. We test our
approach on several benchmark control problems, and we explain its success in
guiding intrinsically motivated behaviors by relating our information-
theoretic control function to fundamental properties of the dynamical system
representing the combined agent-environment system. This opens the door for
designing practical artificial, intrinsically motivated controllers and for
linking animal behaviors to their dynamical properties.

Keywords— information capacity $|$ sensitivity gain $|$ stabilization $|$
predictive information

## Introduction

Living organisms are able to generate behaviors that solve novel challenges
without prior experience. Can this ability be explained by a single, generic
mechanism? One proposal is that novel, useful behaviors can be generated
through intrinsic motivation [1], which is defined informally as a set of
computational algorithms that are derived directly from the intrinsic
properties of the organism-environment dynamics and not specifically learned.

Increasingly, there is a move away from reinforcement learning and its
extrinsically specified reward structure [2, 3] in the theory and practice of
artificial agents, robots, and machine learning more generally [4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. A specific class of such
intrinsic motivation algorithms for artificial systems is known as empowerment
maximization. It proposes that agents should maximize the mutual information
[21] between their potential actions and a subsequent future state of the
world [22]. This corresponds to maximizing the diversity of future world
states achievable as a result of the chosen actions, potentiating a broader
set of behavior options in the future. Intrinsically motivated synthetic
agents develop behaviors that are atypical for inanimate engineered systems
and often resemble those of simple living systems. Interestingly, potentiating
future actions is also a key part of the success of modern reward-based
training algorithms [8, 23, 24].

Despite the successes of empowerment maximization, it remains unclear how well
it can be used as a general intrinsic motivation principle. There are many
different versions of intrinsic motivation related to empowerment, and their
relation to each other is unknown [20, 23, 25]. Additionally, most work on
empowerment maximization has relied on simulational case studies and ad hoc
approximations, and analytical results are scarce. In order to gain insight,
it is important to link empowerment to other, better-understood
characterizations of the systems in question. Finally, calculating the mutual
information between two interlinked processes in the general case is a
challenging task [26, 27], which has so far limited the use of empowerment
maximization to simple cases.

In this work, we unify different versions of intrinsic motivation related to
the empowerment maximization paradigm. Here our main contribution is in
showing analytically that empowerment-like quantities are linked to the
sensitivity of the agent-environment dynamics to the agent’s actions. This
connects empowerment maximization to well-understood properties of dynamical
systems. Since highly sensitive regions of the dynamics potentiate many
diverse future behaviors, the connection to dynamical systems also explains
why empowerment-based intrinsic motivations succeed in generating behaviors
that resemble those of living systems.

The analytical results allow us to develop a practical computational algorithm
for calculating empowerment for complex scenarios in the continuous time
limit, which is the second major contribution of the paper. We apply the
algorithm to standard benchmarks used in intrinsic motivation research [28,
14, 29]. Specifically, a controller based on the efficient calculation of
empowerment manages to balance an inverted pendula without extrinsic rewards.
This opens the door for designing complex robotic intrinsically motivated
agents with systematically computed — rather than heuristically estimated —
empowerment.

$\displaystyle\underset{}{\mathcal{C}^{{T_{e}},{T_{a}},{\Delta
T}}\bigl{(}x_{0}\equiv x(t)\bigr{)}}=\underset{p(\vec{a}\mid
x_{0})}{\max}\;\mathcal{I}[\underbrace{\\{X_{{T_{e}}},X_{{T_{e}}-1},\dots,X_{{T_{a}}+{\Delta
T}}\\}}_{\vec{X}-\mbox{future
states}};\underbrace{\\{A_{{T_{a}}-1},A_{{T_{a}}-2},\dots,A_{0}\\}}_{\vec{A}-\mbox{possible
actions}}\mid x_{0}\equiv x(t)]$ Empowerment${T_{a}}={T_{e}},{\Delta
T}=0$$\cdots$controlled Lyapunov expt.${T_{a}}=1,{\Delta
T}={T_{e}}-1$$\cdots$kicked CEF${T_{a}}=1,{\Delta T}=0$ Figure 1: Unified view
on information theoretical intrinsic motivation, for a discretized process
sequence. Starting at time $x_{0}$ (i.e. $x(t)$), potential actions are
applied for ${T_{a}}$ times, following that, after waiting for ${\Delta T}$
time steps, the future system trajectory is considered until ${T_{e}}$. A
controlled Lyapunov exponent is a Lyapunov exponent, but only in directions
controlled by the agent, cf., (11). “Kicked CEF” refers to a variant of Causal
Entropic Forcing [30], with the addition that an action kicks the system at
the beginning of a trajectory. For more details see Generalized Empowerment.

## Results

### A Preliminaries

#### Notation

We consider an agent that takes on states
$x(t)\in\mathcal{X}:=\mathbb{R}^{d_{x}}$, evolving in time under the dynamics
$f$ with (small) stochastic perturbations $\eta(t)\in\mathbb{R}^{d_{x}}$. Via
its (small) actions, $a(t)\in\mathcal{A}:=\mathbb{R}^{d_{a}}$ filtered through
the control gain $g$, the agent can affect the dynamics of the system:

$\displaystyle dx(t)=f(x(t))dt+g(x(t))da(t)+d\eta\;.$ (1)

Here $d\eta$ denotes the system noise, modeled as a Wiener process. The
agent’s actions $a(t)$ are modeled by a stochastic control process with
variance $\sigma^{2}_{t}$ controlled by the agent and with a mean of zero.
This models potential effect of actions centered around the null action.

To compute various quantities of interest, we will consider a discretized
versions of this system, for which we adopt a modified notation. To
distinguish it from the continuous version, we replace the continuous time in
parentheses by an integer index, $x_{k}:=x(t+k\cdot\Delta t)$. Here $\Delta t$
denotes the physical time step, and we adopted the convention that
$x_{0}=x(t)$, so that the index corresponding to the current physical time,
$t$, is chosen as 0. We will consider trajectories of a fixed duration, and
the agent will apply actions over a part of that trajectory. We denote by
$T_{e}$ the time index of the very last state of the trajectory, which we also
refer to as the time horizon. We further use $T_{a}$ to denote the
(discretized) duration of the action sequence. Then state, control and
perturbation trajectories at finite equidistant times, $\\{t+k\cdot\Delta
t\\}_{k=0}^{T}$, are denoted by
$x_{0}^{T_{e}}\equiv\\{x_{k}\\}_{k=0}^{T_{e}}$,
$a_{0}^{T_{a}}\equiv\\{a_{k}\\}_{k=0}^{T_{a}}$, and
$\eta_{0}^{T_{e}}\equiv\\{\eta_{k}\\}_{k=0}^{T_{e}}$, respectively. For
consistency with the control theory literature, we write a trajectory in the
reverse order, e.g., $x_{0}^{T_{e}}=(x_{T_{e}},\dots x_{0})$. When we wish to
emphasize the continuous nature of the underlying process, we will write
$t_{e}\equiv t+{T_{e}}\cdot\Delta t$ and $t_{a}\equiv t+{T_{a}}\cdot\Delta t$
for explicitly continuous times.

#### Reinforcement Learning vs. Intrinsic Motivation

To elicit a desired behavior in an agent, one typically uses reinforcement
learning (RL). RL is task-specific, and an agent needs an extrinsic feedback
about its performance from a reward function to learn the behavior. The
precise construction of this reward function is critical to achieve a desired
performance in a short training time [2]. Some of the complications include a
significant degree of arbitrariness when choosing amongst reward functions
with equivalent performance [31] and the difficulty of translating an often
vague desired behavior into a concrete reward function. Furthermore, complex
behaviors consist of combinations of shorter sequences. Designing a reward
function capable of partitioning the solution into such parts and hence
learning it in a realistic time is hard [32].

In contrast to this, in living systems, acquisition of skills often starts
with task-unspecific learning. This endows organisms with potentiating skills,
which are not rewarding on their own. This is then followed by task-oriented
specialization, which combines task-unspecific behaviors into complex and
explicitly rewarding tasks [1, 33]. While specific tasks are often refined
with the help of an extrinsic reinforcement, the potentiating tasks usually
are intrinsically motivated [9].

#### Empowerment

The type of intrinsic motivation we focus on is empowerment. Empowerment is
based on information-theoretic quantities [34, 35, 23, 36, 4, 37, 38, 39, 30,
40]. It defines a pseudo-utility function on the state space, based on the
system dynamics only, without resorting to a reward. Formally, we express the
dynamics of the system by the conditional probability distribution
$p(x_{{T_{e}}}\mid a_{0}^{{T_{e}}-1},x_{0})$ of the resulting state when one
starts in a state $x_{0}$ and subsequently carries out an action sequence
$a_{0}^{{T_{e}}-1}$. Then the empowerment $\mathcal{C}(x_{0})$ is a function
of the starting state, $x_{0}$. It is given by the maximally achievable mutual
information (the channel capacity [21]) between the control action sequence of
length ${T_{e}}$ and the final state when starting in the state $x_{0}$:

$\displaystyle\mathcal{C}(x_{0})$
$\displaystyle:=\underset{p(a_{0}^{{T_{e}}-1}\mid
x_{0})}{\max}\;\;I(X_{{T_{e}}};A_{0}^{{T_{e}}-1}|x_{0}).$ (2)

Here $p(\cdot)$ denotes a probability density or a probability distribution
function, and $I$ is the mutual information [21]

$I(X_{{T_{e}}};A_{0}^{{T_{e}}-1}|x_{0})=H(X_{T_{e}}|x_{0})-H(X_{T_{e}}\mid
A_{0}^{{T_{e}}-1}x_{0}).$ (3)

$H$ is the entropy, and conditioning an entropy on a random variable means the
entropy of the conditional distribution, averaged over the conditioning
variable. The empowerment $\mathcal{C}(x_{0})$ depends on both the state,
$x_{0}$, and the time horizon, ${T_{e}}$. However, for notational convenience,
we omit all parameters from the notation except for the dependency on $x_{0}$.

Locally maximizing empowerment (e.g., by following its gradient over $x_{0}$)
guides an agent to perform actions atypical within the natural dynamics of the
system. Indeed, since empowerment measures the diversity of achievable future
states, maximizing it increases this diversity (“empowers” the agent – hence
the name). Thus it is expected to be particularly useful for learning
potentiating tasks [9]. Crucially, empowerment quantifies the relation between
the final state and the _intentional_ control, rather than the diversity of
states due to the stochasticity of the system. In particular, it is not just
the entropy of a passive diffusion process in the state variables, but of the
subprocess that the agent can actively generate. Furthermore, it quantifies
diversity due to _potential_ future action sequences, which are not then
necessarily carried out.

Empowerment is typically used in the form of the empowerment maximization
principle [17], treats $\mathcal{C}(x_{0})$ as a pseudo-utility function. At
each time step, an agent chooses an action to greedily optimize its
empowerment at the next time step. That is, the agent climbs up in its
empowerment landscape, eventually achieving a local maximum of $\mathcal{C}$:

$\displaystyle a^{*}\bigl{(}x(t)\bigr{)}$
$\displaystyle=\underset{a\in\mathcal{A}}{\mbox{argmax}}\;\mathbb{E}_{\eta}\bigl{[}\mathcal{C}\bigl{(}f(x(t))+g(x(t))a\Delta{t}^{\prime}+d\eta\bigr{)}\bigr{]}\;.$
(4)

Here $\mathcal{A}$ is the set of permitted actions, $\Delta{t}^{\prime}$ is a
small time step used to simulate the actual behavior of the system (and which
is selected independently from the time step $\Delta t$ used to discretize
(1)). An empowerment-maximizing agent generates its behavior by repeating this
action selection procedure for each decision step it takes.

Crucially, no general analytical solutions or efficient algorithms for
numerical estimation of empowerment for arbitrary dynamical systems are known,
limiting adoption of the empowerment maximization principle. Our goal is to
provide a method to calculate it under specific approximations.

### B Empowerment in dynamical systems

#### The linear response approximation

To relate empowerment to traditional quantities used to describe dynamical
systems, we assume that the control signal $a$ in (1) is small. This is true
in some of the most interesting cases, where the challenge is to solve a
problem with only weak controls that cannot easily “force” a solution. Under
this assumption, (1) is approximated by a linear time-variant dynamics around
the trajectories of the autonomous dynamics (i.e., for $a=0$). To proceed, we
now introduce the following notation. We define $\bar{x}_{s}$ as the $s$-th
step of the trajectory in the discretized deterministic approximation of the
dynamics (1), given by

$\displaystyle\bar{x}_{s}=f(\bar{x}_{s-1})+g(\bar{x}_{s-1})\Delta a_{s-1}$ (5)

with $\bar{x}_{0}=x_{0}\equiv x(t)$. For example,
$\bar{x}_{3}=f(f(f(\bar{x}_{0})+g(\bar{x}_{0})\Delta
a_{0})+g(\bar{x}_{1})\Delta a_{1})+g(\bar{x}_{2})\Delta a_{2}$. We denote this
recursive mapping from $\bar{x}_{0}$ to $\bar{x}_{s}$ by $F$,
$\bar{x}_{s}=F(\bar{x}_{0};\Delta a_{0}^{s-1})$. Then the sensitivity of the
state at the time step $s$ to the action at the time step $r$ can be
calculated via the iterated differentiation chain rule applied to the state
derivative of the dynamics $F$:

$\displaystyle\frac{\partial\bar{x}_{s}}{\partial a_{r}}=$
$\displaystyle\prod_{\tau=r+2}^{s}\\!\nabla_{\bar{x}}f(\bar{x}_{\tau-1})\;g(\bar{x}_{r}),$
(6)

where $\nabla_{\bar{x}}f(\bar{x}_{\tau})$ is the $d_{x}\times d_{x}$ Jacobian
matrix, which approximates $f$ up to the linear order in the state and the
control. Specifically, the $(i,j)$-th entry of
$\nabla_{\bar{x}}f(\bar{x}_{\tau})$ is $\frac{\partial
f_{i}(\bar{x}_{\tau})}{\partial\bar{x}_{\tau,j}}$, where indices $i,j$ stand
for components of the vectors $x$ and $f$. For $s=r+1$, the expression in (6)
evaluates to $\frac{\partial\bar{x}_{r+1}}{\partial a_{r}}=g(x_{r})$.

Now we define the linear response of the sequence of the system’s states
$x_{s_{1}}^{s_{2}}$ to the sequence of the agent’s actions $\Delta
a_{r_{1}}^{r_{2}}$

$\mathcal{F}_{r_{1},r_{2}}^{s_{1},s_{2}}({x}_{0})=\leavevmode\hbox
to102.56pt{\vbox to86.3pt{\pgfpicture\makeatletter\hbox{\hskip
51.28185pt\lower-43.84564pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } {}{{}}{{}}{{}}{
{}}{{}}\hbox{\hbox{\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{\offinterlineskip{}{}{{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}}{{{}}}{{}{}{{
{}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-43.08214pt}{-37.00694pt}\pgfsys@invoke{
}\hbox{\vbox{\halign{\pgf@matrix@init@row\pgf@matrix@step@column{\pgf@matrix@startcell#\pgf@matrix@endcell}&#\pgf@matrix@padding&&\pgf@matrix@step@column{\pgf@matrix@startcell#\pgf@matrix@endcell}&#\pgf@matrix@padding\cr\hfil\hskip
10.23581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-7.2361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}}}{\partial a_{r_{2}}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}}}&\hskip
10.23581pt\hfil&\hfil\hskip 11.66919pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.66948pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}}}{\partial a_{r_{2}-1}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}}}&\hskip
11.66919pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\dots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}}}&\hskip
7.87471pt\hfil&\hfil\hskip 10.23581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-7.2361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}}}{\partial a_{r_{1}}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
10.23581pt\hfil\cr\vskip 0.0pt\cr\hfil\hskip
11.73581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.7361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}-1}}{\partial a_{r_{2}}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
11.73581pt\hfil&\hfil\hskip 11.73581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.7361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}-1}}{\partial a_{r_{2}-1}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
11.73581pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\dots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil&\hfil\hskip 11.73581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.7361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{2}-1}}{\partial a_{r_{1}}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
11.73581pt\hfil\cr\vskip 0.0pt\cr\hfil\hskip
7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\vdots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\vdots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\dots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\vdots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil\cr\vskip 0.0pt\cr\hfil\hskip
10.23581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-7.2361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{1}}}{\partial a_{r_{2}}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
10.23581pt\hfil&\hfil\hskip 11.66919pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.66948pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{1}}}{\partial a_{r_{2}-1}}$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
11.66919pt\hfil&\hfil\hskip 7.87471pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.875pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\dots$ }}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
7.87471pt\hfil&\hfil\hskip 8.73581pt\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.7361pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\frac{\partial\bar{x}_{s_{1}}}{\partial a_{r_{1}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}&\hskip
8.73581pt\hfil\cr}}}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}}{{{{}}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}}\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ }}{ }
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-51.28185pt}{-37.95665pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{{\nullfont{{{
{}{}{}}}{{ {}{}{}}}}}$\left[\vbox{\hrule
height=40.20665pt,depth=40.20665pt,width=0.0pt}\right.$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}}\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{46.28185pt}{-37.95665pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{{\nullfont{{{
{}{}{}}}{{ {}{}{}}}}}$\left.\vbox{\hrule
height=40.20665pt,depth=40.20665pt,width=0.0pt}\right]$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}}{}{{}}{}{
{}}{}{}{}{ {}}{}{}{}\pgfsys@beginscope\pgfsys@invoke{
}\color[rgb]{0.485,0.94,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.485,0.94,0.5}\pgfsys@color@cmyk@stroke{0.455}{0}{0.44}{0.06}\pgfsys@invoke{
}\pgfsys@color@cmyk@fill{0.455}{0}{0.44}{0.06}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.485,0.94,0.5}\pgfsys@setlinewidth{1.2pt}\pgfsys@invoke{
}{}\pgfsys@moveto{-41.78214pt}{30.78325pt}\pgfsys@lineto{-20.91052pt}{30.78325pt}\pgfsys@lineto{-20.91052pt}{-43.24565pt}\pgfsys@lineto{-41.78214pt}{-43.24565pt}\pgfsys@lineto{-41.78214pt}{30.78325pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{{}}{}{ {}}{}{}{}{
{}}{}{}{}\pgfsys@beginscope\pgfsys@invoke{
}\color[rgb]{0.342,0.622,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.342,0.622,1}\pgfsys@color@cmyk@stroke{0.658}{0.378}{0}{0}\pgfsys@invoke{
}\pgfsys@color@cmyk@fill{0.658}{0.378}{0}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0.342,0.622,1}\pgfsys@setlinewidth{1.2pt}\pgfsys@invoke{
}{}\pgfsys@moveto{20.91052pt}{30.78325pt}\pgfsys@lineto{41.78214pt}{30.78325pt}\pgfsys@lineto{41.78214pt}{10.92506pt}\pgfsys@lineto{20.91052pt}{10.92506pt}\pgfsys@lineto{20.91052pt}{30.78325pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}}{}{{}}{}{ {}}{}{}{}{
{}}{}{}{}\pgfsys@beginscope\pgfsys@invoke{
}\color[rgb]{1,0.2,0.2}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.2,0.2}\pgfsys@color@cmyk@stroke{0}{0.8}{0.8}{0}\pgfsys@invoke{
}\pgfsys@color@cmyk@fill{0}{0.8}{0.8}{0}\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{1,0.2,0.2}\pgfsys@setlinewidth{1.2pt}\pgfsys@invoke{
}\pgfsys@setdash{3.0pt,3.0pt}{0.0pt}\pgfsys@invoke{
}{}\pgfsys@moveto{-41.78214pt}{30.78325pt}\pgfsys@lineto{41.78214pt}{30.78325pt}\pgfsys@lineto{41.78214pt}{10.92506pt}\pgfsys@lineto{-41.78214pt}{10.92506pt}\pgfsys@lineto{-41.78214pt}{30.78325pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}_{d_{x}\cdot s\times
d_{a}\cdot r},$ (7)

where $s=s_{2}-s_{1}+1$, $r=r_{2}-r_{1}+1$, $s+{\Delta T}+r-1={T_{e}}$, and
the entries are computed via (6). Usually we consider situations where the
agent applies its controls for $r$ time steps, and then after a gap observes
the state for $s$ steps. That is, $s_{1}=r_{2}+1+{\Delta T}$, where ${\Delta
T}\geq 0$ is the gap between the end of the control sequence and the start of
the observations.

Notice that traditional definitions of sensitivity of a dynamical system to
its controls are blocks
$\mathcal{F}_{r^{\prime}_{1},r^{\prime}_{2}}^{s^{\prime}_{1},s^{\prime}_{2}}$
in this overall sensitivity matrix, $\mathcal{F}_{r_{1},r_{2}}^{s_{1},s_{2}}$.
For example, if $r^{\prime}_{1}=r^{\prime}_{2}=0$, ${\Delta
T}^{\prime}={T_{e}}-1$, and $s^{\prime}_{1}={T_{e}}$, then
$s^{\prime}_{2}={T_{e}}$, and the sensitivity matrix collapses to just the
entries that measure the sensitivity of the current state to the controls
during the immediately preceding time step,
$\mathcal{F}_{0,0}^{{T_{e}},{T_{e}}}(x_{0})=\frac{\partial\bar{x}_{{T_{e}}}}{\partial
a_{0}}$. This is also the blue block of the overall sensitivity matrix, (7).

With the definitions above, in the linear response regime, the effect of a
sequence of (small) actions on a sequence of states, (1), becomes

$\displaystyle\Delta
x_{s_{1}}^{s_{2}}=\mathcal{F}_{r_{1},r_{2}}^{s_{1},s_{2}}(x_{0})\Delta
a_{r_{1}}^{r_{2}}+\tilde{\eta},$ (8)

where $\Delta a$ and $\Delta x$ are the reverse-time-ordered vectors of small
actions and the induced deviations of states (which themselves can be
vectors).Here $\tilde{\eta}$ models both the total noise resulting from the
integration of the process noise $d\eta$ from (1) and the noise of the
subsequent observation of the state perturbation $\Delta x_{s_{1}}^{s_{2}}$,
which we assume as Gaussian.

#### Generalized Empowerment

Since the entire dynamics is now linear, cf. (8), we can consider formally
effects of arbitrary length sequences of actions on arbitrary length sequences
of future states. In other words, we can define the generalized empowerment,

$\displaystyle\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}(x_{0})$
$\displaystyle:=\underset{p(\vec{a}\mid x_{0})}{\max}\;\;I(X_{{T_{a}}+{\Delta
T}}^{{T_{e}}};A_{0}^{{T_{a}}-1}|x_{0})\;.$ (9)

Here, ${T_{a}}$ denotes the number of time steps at which actions are
performed, ${\Delta T}$ is the time gap between the action sequence and the
beginning of the observation of the resulting states, and ${T_{e}}$ is the
last step in that observed sequence. That is,
$\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}$ measures the maximum mutual
information between a sequence of actions and a later sequence of states,
rather than just one final state, like empowerment does.

Plugging in (8) into (4), we observe that computing the generalized
empowerment in discretized time with an arbitrary discretization step and an
arbitrary time horizon $T_{e}$ reduces to a traditional calculation of the
channel capacity of a linear Gaussian channel, though with a large number of
dimensions reflecting both the duration of the signal and the duration of the
response. Specifically,

$\displaystyle\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}(x_{0})=$
$\displaystyle\underset{\begin{subarray}{c}\sigma_{i}\geq 0\\\
{\textstyle\sum\limits_{i}}\sigma_{i}=P\end{subarray}}{\max}\;\frac{1}{2}\sum_{i=1}^{d_{x}}\ln(1+\rho_{i}(x_{0})\sigma_{i}).$
(10)

Here $\rho_{i}(x_{0})$ are the singular values of the appropriate submatrix
$\mathcal{F}_{r^{\prime}_{1},r^{\prime}_{2}}^{s^{\prime}_{1},s^{\prime}_{2}}(x_{0})$;
for example, the traditional empowerment corresponds to the red-dashed
submatrix in (7). Further, $P$ is the power of the control signal $\Delta a$
over the whole control period, and $\sigma_{i}\geq 0$ is that part of the
overall power of the control signal which is associated with the $i$-th
singular value (called channel power). The channel power can be computed by
the usual water-filling procedure [21]. Note that here we denote $P$ as power,
as per control-theoretic convention, but since we fix the time interval over
which it is applied, the units of $P$ are those of energy. As per our weak
control assumption, we assume $P$ to be suitably small.

With (10), calculation of any generalized empowerment becomes tractable, at
least in principle. This also shows explicitly that the (generalized)
empowerment is a function of the sensitivity matrix ${\mathcal{F}}$, and with
it of quantities used to characterize dynamics, such as the Lyapunov
exponents.

To compute $\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}(x_{0})$ efficiently for
an arbitrary dynamical system (1) and arbitrary long time horizons and
arbitrary small discretization steps, we start by discretizing the time and
calculating the linear response matrix ${\cal F}$. While in this paper we do
this by analytical differentiation, numerical differentiation can be used
whenever $f$ is unknown. We then calculate the singular values of ${\cal F}$;
this is straightforward on modern computers for dimensionalities of up to a
few hundred. Finally, we apply the “water filling” procedure to find the set
of channel powers $\sigma_{i}$ to match the available total power $P$ in (10),
and from there we calculate the (generalized) empowerment value. We will
employ this approach for all examples in this paper.

#### Connecting Generalized Empowerment to Related Quantities

Generalized empowerment with different durations of action and observation
sequences is related to various quantities describing dynamical systems,
including those defining intrinsic motivation [41, 20, 23, 8]. For example,
Causal Entropic Forcing (CEF) [20] is defined as actions that maximize the
entropy of future trajectories of a system. With ${T_{a}}=1$ and ${\Delta
T}=0$, $\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}$ in (9) measures the
immediate consequences of a single action on a trajectory with a fixed time
horizon $T_{e}$. Maximizing $\mathcal{C}^{{T_{e}},{T_{a}},{\Delta T}}$ is then
equivalent to choosing actions that maximize susceptibility, and not the
entropy of trajectories with a given time horizon. In other words, one can
interpret $\mathcal{C}^{{T_{e}},1,0}$ as a “kicked”, or agent-controllable,
version of CEF, where just the first action can be selected by the agent at
any time, and uncontrolled future variability is discarded in action planning
(see Fig. 1 for an illustration). Such kicked CEF corresponds to the green
submatrix in (7).

Now consider the top right corner (blue) of (7) with $T_{e}=T_{a}=1$, or,
equivalently, $s^{\prime}_{2}=s_{2}$ and $s^{\prime}_{1}=s^{\prime}_{2}-1$. In
the limit of a very long horizon, $s_{2}\to\infty$, the appropriate submatrix
of ${\mathcal{F}}$ is

$\displaystyle\Lambda\equiv\underset{s_{2}\rightarrow\infty}{\lim}\Biggl{(}\Bigl{(}\frac{\partial\bar{x}_{s_{2}}}{\partial
a_{r_{1}}}\Bigr{)}\Bigl{(}\frac{\partial\bar{x}_{s_{2}}}{\partial
a_{r_{1}}}\Bigr{)}^{\dagger}\Biggr{)}^{\frac{1}{s_{2}}},$ (11)

where ${\dagger}$ is the transpose, and
$\frac{\partial\bar{x}_{s_{2}}}{\partial a_{r_{1}}}$ is given by (6). In the
special case that the control gain is the identity, $g(x)=x$, the logarithm of
the eigenvalues of $\Lambda$ reduces to the usual characteristic Lyapunov
exponents of the dynamical system [42]. However, once a more general control
gain is applied, the action-controlled perturbation, $a_{r_{1}}$ may be able
to affect only a part of the state space. This means that $\Lambda$ not only
is a generalized empowerment with specific indices, but it is also a
specialization of the concept of Lyapunov exponents to the controllable
subspace. Thus we refer to the log-spectrum of $\Lambda$ as the control
Lyapunov exponents, cf. Fig. 1.

In summary, (9) and the linearization, (7), provide a unified view of various
sensitivties of the dynamics to the controls, and hence on various versions of
intrinsic motivation.

Figure 2: Intrinsic motivation based control in the power-constrained regime.
Top row: generalized empowerment landscapes in the linear approximation for
empowerment (left), controlled Lyapunov exponent (middle), and kicked CEF
(right) versions of the problem, plotted against $\theta$ (horizontal axis)
and $\dot{\theta}$ (vertical axis), measured in $\mathrm{rad}$ and
$\mathrm{rad}\mathrm{/}\mathrm{s}$, respectively. Black dots in each panel are
the final state, and white lines are the trajectories of the pendulum,
starting at the bottom denoted by the red dots. Bottom row: the control
signals chosen from the generalized empowerment maximization as a function of
time. Here the time horizon is $t_{e}=0.5$\mathrm{s}$$.

### C Intrinsic motivation in power-constrained agents

An agent controlling a system with unconstrained actions can trivially reach
any state in a controllable dynamical system [43] by simply forcing their
desired outcome without sophisticated control. Thus to render the setup
interesting, we consider only power-constrained, or weak agents. To show that
empowerment maximization, in the linearized regime, is an efficient control
principle, we use it to stabilize a family of inverted pendula (single pole,
double pole, and cart-pole), which are simple, paradigmatic models of
important phenomena, such as human walking [44].

Solutions for the stabilization problem are known. They require to accumulate
energy by swinging the pendulum back and forth into resonance without
overshooting and then to keep the pendulum upright. When details of the system
are not specified a priori, this solution needs to be learned by the agent.
Finding such an indirect control policy by traditional reinforcement learning
is nontrivial [3], since the increasing oscillations require a long time for
the balancing to take place, and the acquisition of informative rewards
indicating success is significantly delayed. As we will show, it is precisely
in such situations that intrinsic motivation based on empowerment is
especially useful, since it is determined from only comparatively local
properties of the dynamics along the present trajectory and its potential
future variations.

#### Inverted pendulum

We start with a relatively simple task of swinging up and stabilizing an
inverted pendulum without an external reward. With an angle of $\theta$ (in
radians) from the upright vertical, the equations of motion of the pendulum
are

$\displaystyle\begin{pmatrix}d\theta(t)\\\ d\dot{\theta}(t)\end{pmatrix}$
$\displaystyle=\begin{pmatrix}\dot{\theta}(t)dt\\\
\frac{g}{l}\sin(\theta(t))\,dt+\frac{da(t)}{ml^{2}}+\frac{dW(t)}{ml^{2}}\end{pmatrix},$
(12)

where $\dot{\theta}$ is the angular velocity of the pendulum, $m$ is its mass,
$l$ is the length, $a$ is the torque applied by the agent, $g$ is the free
fall acceleration, and $dW(t)$ is a Wiener process.

We apply a (stochastically chosen) control signal $a(t)$ for the duration
${T_{e}}$ and observe the final state
$\tilde{\theta}=\theta+\tilde{\eta}_{\text{obs}}$, where
$\tilde{\eta}_{\text{obs}}$ is the standard Gaussian observation noise at the
final state. Empowerment is then given by the maximally achievable mutual
information between $a(t)$ and $\tilde{\theta}$ at a given power level for
$a(t)$, i.e., the channel capacity between the two.

The observation noise effectively determines the resolution, at which the end
state is considered. Note that in our linear approximation the process noise
$dW(t)$ undergoes the same gain sequence as the control signal, and thus it
rescales the empowerment landscape and changes the behavior of the system.
Thus to compare empowerment values in different states, it is essential to
include the observation noise.

We now apply our empowerment-based control protocol, (4), to the inverted
pendulum. We calculate the empowerment landscape by using the time-discretized
version of Eqs. (1, 12). For this, we map the deterministic part of the
dynamics ($f,g$ in (1)) onto discrete time as per (5). We then compute the
channel capacity by applying (10) using the singular values from (8), where
states are given by $(\theta,\dot{\theta})\in\mathbb{R}^{d_{x}}$, and actions
consist of applying a torque $a$. The landscapes for the original empowerment,
the controlled Lyapunov exponent, and the kicked CEF versions of the problem,
all with the time horizons of $t_{e}=$0.5\text{\,}\mathrm{s}$$ and the
discretization $\Delta t=10^{-3}$ are shown in Fig. 2.

Figure 3: Convergence of the method for $\Delta t\to 0$ and
$t_{e}=0.5$\mathrm{s}$$. As time resolution is refined fourfold at every
stage, one arrives at a well-defined value for the empowerment estimation as
$\Delta t\to 0$. The numerical stability of this limit approximation is
consistent throughout the landscape.

Then, from each state, we choose the control action to greedily optimize the
generalized empowerment. The panels in the upper row in this Figure also show
trajectories obtained this way. The lower row shows time traces of the control
signal derived from the generalized empowerment maximization. In all cases,
initially, the agent drives the pendulum at the maximum allowable torque,
which we set to be power-constrained to $\pm
1\text{\,}\mathrm{N}\text{\,}\mathrm{m}$. Around 13, 10, and 10 seconds after
the start (for the three versions of the empowerment, respectively), the
pendulum accumulates enough energy to reach the vertical, and the agents
reduce the torques to very small values,
$a\ll$1\text{\,}\mathrm{N}\text{\,}\mathrm{m}$$, which are now sufficient to
keep the pendulum in the upright position and prevent it from falling. It is
striking that the generalized empowerment landscapes and their induced
trajectories are qualitatively similar to those that would be generated by an
optimal value function, derived by standard optimal control techniques based
on a reward specifically designed to achieve the top position [3].

In our analysis, we chose a particular discretization $\Delta t=10^{-3}$ s,
and we need to show that our results depend only weekly on this choice. For
this, we repeat our analysis at different $\Delta t$. Figure 3 shows the
dependence of the maximum value of the original empowerment (black dot in left
panel of Fig. 2) on $\Delta t$. To the extent that the estimate converges to a
well-defined number linearly as $\Delta t\to 0$, the discrete time dynamics
provides a consistent approximation to the continuous time dynamics.

#### Double Pendulum

Figure 4: Top left: Double pendulum with control torque on the joint between
the links with dynamics given by (13) Top right: Slices through the
empowerment landscape of a double pendulum. Each subplot shows a particular
slice in the 4D landscape, when two other coordinates are zero. For example,
the plot with axes $\dot{\theta}_{2}$, $\dot{\theta}_{1}$ is shown for
$\theta_{2}=$0\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$ and
$\theta_{1}=$0\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$. Bottom: Traversing
the state space of the double pendulum according to (4). The first and the
second 15s are shown with different scale for the instantaneous empowerment.
The initial and the final positions are both links down and both links up,
respectively. Torque is applied to the middle joint only.

Now we show that the empowerment maximization formalism is capable of dealing
with more challenging problems, such as a power-constrained control of a
(potentially chaotic) double pendulum [16], Fig. 4, with equations of motion:

$\displaystyle d\ddot{\theta}_{1}(t)=$
$\displaystyle-\frac{1}{d_{1}(t)}\biggl{(}d_{2}(t)\ddot{\theta}_{2}(t)+\phi_{1}(t)\biggr{)},$
(13) $\displaystyle d\ddot{\theta}_{2}(t)=$
$\displaystyle\frac{1}{m_{2}\ell_{c_{2}}^{2}+I_{2}-\frac{d^{2}_{2}(t)}{d_{1}(t)}}\biggl{(}da(t)+dW(t)+\frac{d^{2}_{2}(t)}{d_{1}(t)}\phi_{1}(t)$
$\displaystyle\qquad\qquad\qquad\quad-
m_{2}\ell_{1}\ell_{c_{2}}\dot{\theta}_{1}(t)^{2}\sin\theta_{2}(t)-\phi_{2}(t)\biggr{)},$

with

$\displaystyle d_{1}(t)=$ $\displaystyle
m_{1}\ell_{c_{1}}^{2}+m_{2}(\ell_{1}^{2}+\ell_{c_{2}}^{2}+2\ell_{1}\ell_{c_{2}}\cos\theta_{2}(t))+I_{1}+I_{2},$
$\displaystyle d_{2}(t)=$ $\displaystyle
m_{2}(\ell_{c_{2}}^{2}+\ell_{1}\ell_{c_{2}}\cos\theta_{2}(t))+I_{2},$
$\displaystyle\phi_{1}(t)=$ $\displaystyle-
m_{2}\ell_{1}\ell_{c_{2}}\dot{\theta}(t)^{2}\sin\theta_{2}(t)\\!-\\!2m_{2}\ell_{1}\ell_{c_{2}}\dot{\theta}_{2}(t)\dot{\theta}_{1}(t)\sin\theta_{2}(t)$
$\displaystyle\qquad\qquad\qquad\quad+(m_{1}\ell_{c_{1}}+m_{2}\ell_{1})g\cos\theta_{1}(t)+\phi_{2}(t),$
$\displaystyle\phi_{2}(t)=$ $\displaystyle
m_{2}\ell_{c_{2}}g\cos(\theta_{1}(t)+\theta_{2}(t)).$

We add Wiener noise, $dW(t)$, and permit the controller to apply a scalar
control signal $|a(t)|\leq 1$, at the joint between the two links. In the
equations of motion, $m_{i}$, $\ell_{i}$, $\ell_{c_{i}}$, and $I_{i}$ stand
for the mass, the length, the length to center of mass, and the moment of
inertia of the $i$-th link, $i\in[1,2]$, respectively. Figure 4 shows the
landscape for the original empowerment for selected slices of the phase space.
This landscape is more complex than for the single-pendulum. Nonetheless it
retains the property that, following the local gradient in the state space
directly, one ultimately reaches the state of the maximum empowerment, which
is precisely where both links of the pendulum are balanced upright. The
vertical position, however, is a priori not sufficient to guarantee the
balancing since the control only applies torque at the joint linking the
pendulum halves. That is, the controller cannot move the pendulum in arbitrary
directions through the state space. Surprisingly, this concern
notwithstanding, the algorithm still balances the pendulum, cf. Fig. 4.

#### Cart-Pole

Figure 5: Left: Cart-Pole system with control force, $\vec{a}(t)$, applied to
the cart only, which moves on the rail (or on the edge of a table), allowing
the pole to rotate in the x-y plane. Its dynamics is given by (14). Right:
Traversing the state space of the pendulum on a cart according to empowerment
maximization. The initial and the final state of the pole are down and up,
respectively. The horizontal axis is time in seconds $t\in[0,20]s$.

We have additionally verified that the empowerment maximization also balances
an inverted pendulum on a moving cart, cf. Fig.5. Here the control signal
(force) is applied to the cart. Thus the pendulum is now affected only
indirectly. The dynamics of this system is:

$\displaystyle d\ddot{x}(t)=$
$\displaystyle\frac{m\sin\theta(t)(\ell\dot{\theta}^{2}(t)+g\cos\theta(t))+da(t)+dW(t)}{M+m\sin^{2}\theta(t)},$
(14) $\displaystyle d\ddot{\theta}(t)=$ $\displaystyle-
da(t)\cos\theta(t)-m\ell\dot{\theta}^{2}(t)\cos\theta(t)\sin\theta(t)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\quad-(M+m)g\sin\theta(t),$

where $x(t)$, $\theta(t)$, $m$, $M$, $\ell$, $g$, $|a(t)|\leq 1$ are the $x$
coordinate of the center of mass of the cart, the angle of the pole, the pole
mass, the cart mass, the pole length, the free fall acceleration, and the
force applied to the cart.

## Discussion

In this study, we focused on a class of intrinsic motivation models that mimic
decision-making abilities of biological organisms in various situations
without explicit reward signals. We used an information-theoretic formulation
in which the controller starts with knowledge of the (stochastic) dynamical
equations describing the agent and the environment, and then selects actions
that “empower” the agent. That is, the controller improves its ability to
affect the system in the future, as measured by the mutual information between
the action sequence and the subsequent responses. This leads the system to the
most sensitive points in the state space, which we showed solves a problem
known to be difficult for simple reinforcement learning algorithms: balancing
inverted pendula. Depending on which subsets of the past actions and future
responses are used to drive the intrinsic motivation, our approach
interpolates between the original formulation of empowerment maximization,
maximization of the “kicked” version of Causal Entropic Forcing, and
maximization of the “controlled” subset of the Lyapunov exponents of the
agent-environment pair. This provides insight into which properties of the
dynamical system are responsible for the behaviors produced by these different
motivation functions.

One big challenge in using information-theoretic quantities is computing them,
which can be difficult to do either analytically or from data. Our paper makes
a significant contribution to solving this problem in the context of
empowerment by providing an explicit algorithm for computing various versions
of empowerment, for arbitrary lengths of pasts and futures, using the small
noise/small control approximation to the dynamics, while still treating the
dynamics as nonlinear. This is often the most interesting regime, modeling
weak, power-constrained controllers. Crucially, our algorithm is local, so
that climbing up the empowerment gradient only requires estimation of the
dynamics in the vicinity of the current state of the system. This should be
possible in real control applications by using the data directly, possibly
with the help of deep neural networks to approximate the relevant dynamical
landscapes [45, 46, 47]. Therefore, knowing the exact form of the dynamical
system, which could be a potential limitation of our approach, is not strictly
required. This opens up opportunities for scaling our method to more complex
scenarios.

Our work suggests that, in addition to the Lyapunov spectrum, defined via the
trajectory divergence in time due to a small arbitrary perturbation, one may
want to consider the optimal Lyapunov spectrum, where the initial perturbation
is optimally aligned with the controllable directions in the dynamics. We
defer a systematic study of optimal Lyapunov spectra to future work.

A potential extension of our analysis relates to social interactions.
Interacting agents have their own intrinsic motivations and affect each
other’s ability to achieve their goals. Understanding how multiple agents
interact, each trying to empower itself in the presence of others, and whether
and when this leads to cooperation or conflict is a promising area for future
research. Crucially, the ability to affect someone else’s empowerment may
provide insight into what distinguishes social interactions from purely
physical interactions among nearby individuals.

### Acknowledgements

ST was supported in part by California State University, and the College of
Engineering at SJSU. IN was supported in part by the Simons Foundation
Investigator award, the Simons-Emory Consortium on Motor Control, and NIH
grant 2R01NS084844. DP acknowledges partial support by the EC H2020-641321
socSMCs FET Proactive project and the Pazy Foundation.

## References

  * [1] P.-Y. Oudeyer and F. Kaplan.  What is intrinsic motivation? a typology of computational approaches.  Frontiers in neurorobotics, 1:6, 2009.
  * [2] R. S. Sutton and A. G. Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [3] K. Doya.  Reinforcement learning in continuous time and space.  Neural computation, 12(1):219–245, 2000.
  * [4] S. Mohamed and D. J. Rezende.  Variational information maximisation for intrinsically motivated reinforcement learning.  In Advances in neural information processing systems, pages 2125–2133, 2015.
  * [5] K. Gregor, D J. Rezende, and D. Wierstra.  Variational intrinsic control.  arXiv preprint arXiv:1611.07507, 2016.
  * [6] K. Baumli, D. Warde-Farley, S. Hansen, and V. Mnih.  Relative variational intrinsic control.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(8):6732–6740, 2021.
  * [7] T. Kwon.  Variational intrinsic control revisited.  International Conference on Learning Representations, (ICLR), 2021, 2021.
  * [8] A. Sharma, S. Gu, S Levine, V. Kumar, and K. Hausman.  Dynamics-aware unsupervised discovery of skills.  arXiv preprint arXiv:1907.01657, 2019.
  * [9] A. Sharma, M Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu.  Emergent real-world robotic skills via unsupervised off-policy reinforcement learning.  RSS, Robotics: Science and Systems 2020, 2020.
  * [10] B. Eysenbach, A Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  International Conference on Learning Representations, (ICLR), 2019, 2018.
  * [11] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel.  Vime: Variational information maximizing exploration.  Advances in neural information processing systems, 29, 2016.
  * [12] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel.  Variational option discovery algorithms.  arXiv preprint arXiv:1807.10299, 2018.
  * [13] J. Choi, A. Sharma, H. Lee, S. Levine, and S. S. Gu.  Variational empowerment as representation learning for goal-conditioned reinforcement learning.  International Conference on Machine Learning, pages 1953–1963, 2021\.
  * [14] C. Salge, C. Glackin, and D. Polani.  Empowerment–an introduction.  In Guided Self-Organization: Inception, pages 67–114. Springer, 2014.
  * [15] A. S. Klyubin, D Polani, and C. L. Nehaniv.  Empowerment: A universal agent-centric measure of control.  IEEE Congress on Evolutionary Computation, 2005, 1:128–135, 2005\.
  * [16] T. Jung, D. Polani, and P. Stone.  Empowerment for continuous agent—environment systems.  Adaptive Behavior, 19(1):16–39, 2011.
  * [17] A. S. Klyubin, D. Polani, and C. L. Nehaniv.  Keep your options open: An information-based driving principle for sensorimotor systems.  PLoS ONE, 3(12):e4018, Dec 2008.
  * [18] R. Zhao, P. Abbeel, and S. Tiomkin.  Efficient empowerment estimation for unsupervised stabilization.  International Conference Learning Representations, 2020.
  * [19] R. Zhao, S. Tiomkin, and P. Abbeel.  Dynamical system embedding for efficient intrinsically motivated artificial agents.  Advances in Neural Information Processing Systems, (NeurIPS), DeepRL, 2019.
  * [20] A. D. Wissner-Gross and C. E. Freer.  Causal entropic forces.  Physical review letters, 110(16):168702, 2013.
  * [21] T. M. Cover and J. A. Thomas.  Elements of information theory.  John Wiley & Sons, 2012.
  * [22] A. S. Klyubin, D. Polani, and C. L. Nehaniv.  Empowerment: A universal agent-centric measure of control.  In 2005 IEEE Congress on Evolutionary Computation, volume 1, pages 128–135. IEEE, 2005.
  * [23] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  In ICLR, 2019.
  * [24] Y. Du, S. Tiomkin, E. Kiciman, D. Polani, P. Abbeel, and A. D. Dragan.  Ave: Assistance via empowerment.  Neural Information Processing Systems, NeurIPS 2020, 2020.
  * [25] C. Salge and D. Polani.  Empowerment as replacement for the three laws of robotics.  Frontiers in Robotics and AI, 4:25, 2017.
  * [26] W. Bialek, I. Nemenman, and N. Tishby.  Predictability, complexity, and learning.  Neural computation, 13(11):2409–2463, 2001.
  * [27] C. Holmes and I. Nemenman.  Estimation of mutual information for real-valued data with error bars and controlled bias.  Phys Rev E, 100:022404, 2019.
  * [28] T. Jung, D. Polani, and P. Stone.  Empowerment for continuous agent—environment systems.  Adaptive Behavior, 19(1):16–39, 2011.
  * [29] R. Zhao, K. Lu, P. Abbeel, and S. Tiomkin.  Efficient empowerment estimation for unsupervised stabilization.  In International Conference on Learning Representations, ICLR, 2021\.
  * [30] A. D. Wissner-Gross and C. E. Freer.  Causal entropic forces.  Physical review letters, 110(16):168702, 2013.
  * [31] A. Y. Ng, D. Harada, and S. Russell.  Policy invariance under reward transformations: Theory and application to reward shaping.
  * [32] P. Dayan and G. E. Hinton.  Feudal reinforcement learning.  In Advances in neural information processing systems, pages 271–278, 1993.
  * [33] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum.  Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.  In Advances in Neural Information Processing Systems, NeurIPS, 2016\.
  * [34] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros.  Large-scale study of curiosity-driven learning.  In ICLR, 2019.
  * [35] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In ICML, 2017.
  * [36] M. Karl, M. Soelch, P. Becker-Ehmck, D. Benbouzid, P. van der Smagt, and J. Bayer.  Unsupervised real-time control through variational empowerment.  2018\.
  * [37] C. Salge, C. Glackin, and D. Polani.  Approximation of empowerment in the continuous domain.  Advances in Complex Systems, 16(02n03):1250079, 2013.
  * [38] C. Salge, C. Glackin, and D. Polani.  Empowerment–an introduction.  In Guided Self-Organization: Inception, pages 67–114. Springer, 2014.
  * [39] T. Anthony, D. Polani, and C. L. Nehaniv.  General self-motivation and strategy identification: Case studies based on sokoban and pac-man.  IEEE Transactions on Computational Intelligence and AI in Games, 6(1):1–17, 2014.
  * [40] H. J. Charlesworth and M. S. Turner.  Intrinsically motivated collective motion.  Proceedings of the National Academy of Sciences, 116(31):15362–15367, 2019.
  * [41] J. Schmidhuber.  Formal theory of creativity, fun, and intrinsic motivation (1990–2010).  IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010.
  * [42] H. D. Abarbanel, R. Brown, and M. B. Kennel.  Local lyapunov exponents computed from observed data.  Journal of Nonlinear Science, 2(3):343–365, 1992.
  * [43] L. C. Evans.  An introduction to mathematical optimal control theory version 0.2.  1983\.
  * [44] A. D. Kuo.  The six determinants of gait and the inverted pendulum analogy: A dynamic walking perspective.  Human movement science, 26(4):617–656, 2007.
  * [45] B. C. Daniels and I. Nemenman.  Automated adaptive inference of phenomenological dynamical models.  Nature communications, 6(1):1–8, 2015.
  * [46] S. L. Brunton, J. L. Proctor, and J. N. Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  * [47] B. Chen, K. Huang, S. Raghupathi, I. Chandratreya, Q. Du, and H. Lipson.  Automated discovery of fundamental variables hidden in experimental data.  Nature Computational Science, 2(7):433–442, 2022.