# PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor
Features and Inverse Temporal Difference Learning

Angelos Filos  Clare Lyle  Yarin Gal  Sergey Levine  Natasha Jaques  Gregory
Farquhar

###### Abstract

We study reinforcement learning (RL) with no-reward demonstrations, a setting
in which an RL agent has access to additional data from the interaction of
other agents with the same environment. However, it has no access to the
rewards or goals of these agents, and their objectives and levels of expertise
may vary widely. These assumptions are common in multi-agent settings, such as
autonomous driving. To effectively use this data, we turn to the framework of
successor features. This allows us to disentangle shared features and dynamics
of the environment from agent-specific rewards and policies. We propose a
multi-task inverse reinforcement learning (IRL) algorithm, called _inverse
temporal difference learning_ (ITD), that learns shared state features,
alongside per-agent successor features and preference vectors, purely from
demonstrations without reward labels. We further show how to seamlessly
integrate ITD with learning from online environment interactions, arriving at
a novel algorithm for reinforcement learning with demonstrations, called
$\Psi\Phi$-learning (pronounced ‘Sci-Fi’). We provide empirical evidence for
the effectiveness of $\Psi\Phi$-learning as a method for improving RL, IRL,
imitation, and few-shot transfer, and derive worst-case bounds for its
performance in zero-shot transfer to new tasks.

Machine Learning, Reinforcement Learning, Imitation Learning, Successor
Features, Transfer Learning, Multi-Agent, Autonomous Driving,

##  1 Introduction

If artificial agents are to be effective in the real world, they will need to
thrive in environments populated by other agents. Agents are typically goal-
directed, sometimes by definition (Franklin & Graesser, 1996). While their
goals can be different, they often depend on shared salient features of the
environment, and may be able to interact with and affect the environment in
similar ways. Humans and other animals make ready use of these similarities to
other agents while learning (Henrich, 2017; Laland, 2018). We can observe the
goal-directed behaviours of other humans, and combine these observations with
our own experiences, to quickly learn how to achieve our own goals. If
reinforcement learning (RL, Sutton & Barto, 2018) agents could similarly
interpret the behaviour of others, they could learn more efficiently, relying
less on solitary trial and error.

Figure 1:  The $\Psi\Phi$-learning algorithm for RL with no-reward
demonstrations. Demonstrations $\mathcal{D}$ contain data from other agents
for _unknown_ tasks. We employ _inverse temporal difference learning_ (ITD,
cf. Section 3.1) to recover other agents’ successor features (SFs) $\Psi^{k}$
and preferences $\mathbf{w}^{k}$. The ego-agent combines the estimated SFs of
others along with its own preferences $\mathbf{w}^{\text{ego}}$ and SFs
$\Psi^{\text{ego}}$ with generalised policy improvement (GPI, cf. Section
2.2), generating experience. Both the demonstrations and the ego-experience
are used to learn the shared cumulants $\Phi$.

To this end, we formalise and address a problem setting in which an agent (the
‘ego-agent’) is given access to observations and actions drawn from the
experiences of other goal-directed agents interacting with the same
environment, but pursuing distinct goals. These observed trajectories are
unlabelled in the sense that they lack the goals or rewards of the other
agents. This type of data is readily available in many real-world settings,
either from (i) observing other agents acting simultaneously with the ego-
agent in the same (multi-agent) environment, or (ii) multi-task demonstrations
collected independently from the ego-agent’s experiences. Consider autonomous
driving as a motivating example: the car can observe the decisions of many
nearby human drivers with various preferences and destinations, or may have
access to a large offline dataset of such demonstrations. Because the other
agents are pursuing their own varied goals, it can be difficult to directly
use this information with conventional imitation learning methods (Widrow &
Smith, 1964) or inverse RL (IRL) (Ng et al., 2000; Ziebart et al., 2008).

While the ego-agent should not copy other agents directly, it is likely that
the behaviour of all agents depends on shared features of the environment. To
disentangle such shared features from agent-specific goals, we turn to the
framework of successor features (Dayan, 1993; Kulkarni et al., 2016; Barreto
et al., 2017). Successor features are a representation that captures the sum
of state features an agent’s policy will reach in the future. An agent’s goal
is represented separately as a preference vector.

In this paper, we demonstrate how a reinforcement learner can benefit from
multi-task demonstrations using the framework of successor features. The key
contributions are:

  1. 1.

Offline multi-task IRL: We propose an inverse RL algorithm, called _inverse
temporal difference_ (ITD) _learning_. Using only demonstrations, we learn
shared state features, alongside per-agent successor features and inferred
preferences. The reward functions can be trivially computed from these learned
quantities. We show empirically that ITD achieves superior or comparable
performance to prior methods.

  2. 2.

RL with no-reward demonstrations: By combining ITD with learning from
environment interactions, we arrive at a novel algorithm for RL with
unlabelled demonstrations, called $\Psi\Phi$-learning (pronounced ‘Sci-Fi’).
$\Psi\Phi$-learning is compatible with sub-optimal demonstrations. It treats
the demonstrated trajectories as being soft-optimal under _some_ task and
employs ITD to recover successor features for the demonstrators’ policies.
$\Psi\Phi$-learning inherits the unbiased, asymptotic performance of RL
methods while leveraging the provided demonstrations with ITD. When the goals
of any of the demonstrators are even partially aligned with the
$\Psi\Phi$-learner, this enables much faster learning than solitary RL.
Otherwise, when the demonstrations are not useful or even misleading, it
gracefully falls back to standard RL, unlike naïve behaviour cloning or IRL.

  3. 3.

Few-shot adaptation with task inference: Taking full-advantage of the
successor features framework, our $\Psi\Phi$-learner can even adapt zero-shot
to new goals it has never seen or experienced during training, but which are
partially aligned with the demonstrated goals. This is possible due to the
disentanglement of representations into task-specific features (i.e.,
preferences) and shared state features. We can efficiently update the task-
specific preferences and rely on generalised policy improvement for safe
policy updates. We derive worst-case bounds for the performance of
$\Psi\Phi$-learning in zero-shot transfer to new tasks.

We evaluate $\Psi\Phi$-learning in a set of grid-world environments, a
traffic-flow simulator (Leurent, 2018), and a task from the ProcGen suite
(Cobbe et al., 2020), observing advantages over vanilla RL, imitation learning
(Reddy et al., 2019; Ho & Ermon, 2016), and auxiliary-task baselines
(Hernandez-Leal et al., 2017). Thanks to the shared state features between the
ITD and RL components, we find empirically that the $\Psi\Phi$-learner not
only improves its ego-learning with demonstrations, but also enhances its
ability to model others agents using its own experience.

##  2 Background and Problem Setting

We consider a world that can be represented as an infinite horizon controlled
Markov process (CMP) given by the tuple:
$\mathcal{C}\triangleq\langle\mathcal{S},\mathcal{A},P,\gamma\rangle$.
$\mathcal{S}$ and $\mathcal{A}$ represent the continuous state and discrete
action spaces, respectively, $\mathbf{s}^{\prime}\sim
P(\cdot|\mathbf{s},\mathbf{a})$ describes the transition dynamics and $\gamma$
is the discount factor. A task is formulated as a Markov decision process
(MDP, Puterman, 2014), characterised by a reward function,
$R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, i.e.,
$M\triangleq\langle\mathcal{C},R\rangle$.

The goal of an agent is to find a policy which maps from states to a
probability distribution over actions,
$\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$, maximising the expected
discounted sum of rewards
$G^{R}\triangleq\sum_{t=0}^{\infty}\gamma^{t}R(\mathbf{s}_{t},\mathbf{a}_{t})$.
The action-value function of the policy $\pi$ is given by
$Q^{\pi,R}(\mathbf{s},\mathbf{a})\triangleq\operatorname*{\mathbb{E}}^{\mathcal{C},\pi}\left[\left.G^{R}\right|\mathbf{s}_{0}=\mathbf{s},\mathbf{a}_{0}=\mathbf{a}\right]$,
where $\operatorname*{\mathbb{E}}^{\mathcal{C},\pi}\left[\cdot\right]$ denotes
expected value when following policy $\pi$ in environment $\mathcal{C}$.

###  2.1 RL with No-Reward Demonstrations

We are interested in settings in which, in addition to an environment
$\mathcal{C}$, the agent has also access to demonstrations without rewards,
i.e., behavioural data of mixed and unknown quality. The demonstrations are
generated by other agents, whose goals and levels of expertise are unknown,
and who have no incentive to educate the controlled agent. We will refer to
the controlled agent, i.e., reinforcement learner, as the ‘ego-agent’ and to
the agents that generated the demonstrations as ‘other-agents’. We denote the
demonstrations with $\mathcal{D}=\\{\tau_{1},\tau_{2},\ldots,\tau_{N}\\}$,
where the trajectory
$\tau\triangleq\left(\mathbf{s}_{0},\mathbf{a}_{0},\ldots,\mathbf{s}_{T},\mathbf{a}_{T};k\right)$
is generated by the $k$-th agent. Note that each trajectory does include an
identifier of the agent that generated it. The ego-agent also gathers its own
experience by interacting with the environment, collecting data
$\mathcal{B}=\\{\left(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},r^{\text{ego}}\right)\\}$.
Due to the lack of reward annotations in $\mathcal{D}$, and the fact that the
data may be irrelevant to the ego-agent’s task, it is not trivial to combine
demonstrations from $\mathcal{D}$ with the ego-agent’s experience
$\mathcal{B}$.

###  2.2 Successor Features and Cumulants

To make use of the demonstrations $\mathcal{D}$, we wish to capture the notion
that while the agents’ rewards may differ, they share the same environment. To
do so we turn to the framework of successor features (SFs) (Barreto et al.,
2017), in which rewards are decomposed into cumulants and preferences:

######  Definition 1 (Cumulants and Preferences).

The (one-step) rewards are decomposed into task-agnostic _cumulants_
$\Phi(\mathbf{s},\mathbf{a})\in\mathbb{R}^{d}$, and task-specific
_preferences_ $\mathbf{w}\in\mathbb{R}^{d}$:

$R^{\mathbf{w}}(\mathbf{s},\mathbf{a})\triangleq\Phi(\mathbf{s},\mathbf{a})^{\top}\mathbf{w}\,.$
(1)

The preferences $\mathbf{w}$ are a representation of a possible goal in the
world $\mathcal{C}$, in the sense that each $\mathbf{w}$ gives rise to a task
$M^{\mathbf{w}}=\langle\mathcal{C},R^{\mathbf{w}}\rangle$. We use ‘task’,
‘goal’, and ‘preferences’ interchangeably when context makes it clear whether
we are referring to $\mathbf{w}$ itself, or the corresponding $M^{\mathbf{w}}$
or $R^{\mathbf{w}}$. The action-value function for a policy $\pi$ in
$M^{\mathbf{w}}$ is then a function of the preferences $\mathbf{w}$ and the
$\pi$’s successor features.

######  Definition 2 (Successor Features).

For a given discount factor $\gamma\in[0,1)$, policy $\pi$ and cumulants
$\Phi(\mathbf{s},\mathbf{a})\in\mathbb{R}^{d}$, the successor features (SFs)
for a state $\mathbf{s}$ and action $\mathbf{a}$ are:

$\Psi^{\pi}(\mathbf{s},\mathbf{a})\triangleq\mathbb{E}^{\mathcal{C},\pi}\left[\left.\sum_{t=0}^{\infty}\gamma^{t}\Phi(\mathbf{s}_{t},\mathbf{a}_{t})\right|\mathbf{s}_{0}=\mathbf{s},\mathbf{a}_{0}=\mathbf{a}\right]\,.$
(2)

The $i$-th component of $\Psi^{\pi}(\mathbf{s},\mathbf{a})$ gives the expected
discounted sum of $\Phi(\mathbf{s},\mathbf{a})$’s $i$-th component, when
starting from state $\mathbf{s}$, taking action $\mathbf{a}$ and then
following policy $\pi$. Intuitively, cumulants $\Phi$ can be seen as a vector-
valued reward function and SFs $\Psi^{\pi}$ the corresponding vector-valued
state-action value function for policy $\pi$.

An action-value function is then given by the dot product of the preferences
$\mathbf{w}$ and $\pi$’s SFs:

$Q^{\pi,\mathbf{w}}(\mathbf{s},\mathbf{a})=\Psi^{\pi}(\mathbf{s},\mathbf{a})^{\top}\mathbf{w}\,.$
(3)

###### Proof.

See (Barreto et al., 2017). ∎

Note that if we have $\Psi^{\pi}$, the value of $\pi$ for a new preference
$\mathbf{w}^{\prime}$ can be easily computed. This property allows the
successor features of a set of policies to be repurposed for accelerating
policy updates, as follows.

######  Definition 3 (Generalised Policy Improvement).

Given a set of policies $\Pi=\\{\pi_{1},\ldots,\pi_{K}\\}$ and a task with
reward function $R$, generalised policy improvement (GPI) is the definition of
a policy $\pi^{\prime}$ s.t.

$Q^{\pi^{\prime},R}(\mathbf{s},\mathbf{a})\geq\sup_{\pi\in\Pi}Q^{\pi,R}(\mathbf{s},\mathbf{a})\,,\forall\mathbf{s}\in\mathcal{S},\mathbf{a}\in\mathcal{A}\,.$
(4)

Provided the SFs of a set of policies, i.e., $\\{\Psi^{\pi^{k}}\\}_{k=1}^{K}$,
we can apply GPI to derive a new policy $\pi^{\prime}$ whose performance on a
task $\mathbf{w}$ is no worse that the performance of any of $\pi\in\Pi$ on
the same task, given by

$\pi^{\prime}(\mathbf{s})=\operatorname*{arg\,max}_{a}\max_{\pi\in\Pi}\Psi^{\pi}(\mathbf{s},a)^{\top}\mathbf{w}\,.$
(5)

Eqn. (5) suggests that if we could estimate the SFs of other agents, we could
utilise them for improving the ego-agent’s policy with GPI. However, to do so
with conventional methods we would require access to their rewards, cumulants
and/or preferences. In our setting, we can only observe their sequence of
states and actions (Section 2.1). Next, we introduce our method that only
requires no-reward demonstrations to estimate SFs and can be integrated
seamlessly with GPI for accelerating reinforcement learning.

##  3 Accelerating RL with Demonstrations

We now present a novel method, $\Psi\Phi$-learning, that leverages reward-free
demonstrations to accelerate RL, shown in Figure 1. Our approach consists of
two components: (i) a novel inverse reinforcement learning algorithm, called
_inverse temporal difference_ (ITD) _learning_ , for learning cumulants, per-
agent successor features and corresponding agent preferences from
demonstration without reward labels, and (ii) a novel RL algorithm that
combines ITD with generalised policy improvement (GPI, Section 2.2).

###  3.1 Inverse Temporal Difference Learning

Given demonstrations without rewards, $\mathcal{D}$, we model the agents that
generated the data (i.e., blue nodes in Figure 1) as soft-optimal for an
_unknown_ task. In particular, the $k$-th agent’s policy is soft-optimal under
task $\mathbf{w}^{k}$ and is given by

$\pi^{k}(\mathbf{a}|\mathbf{s})=\frac{\exp(\Psi^{\pi^{k}}(\mathbf{s},\mathbf{a})^{\top}\mathbf{w}^{k})}{\sum_{a}\exp(\Psi^{\pi^{k}}(\mathbf{s},a)^{\top}\mathbf{w}^{k})}\,,\forall\mathbf{s}\in\mathcal{S},\mathbf{a}\in\mathcal{A}\,.$
(6)

We choose to represent the action-value functions of the other agents with
their SFs and preferences to enable GPI, and to expose task- and policy-
agnostic structure in the form of shared cumulants $\Phi$. The $k$-th agent’s
successor features are temporally consistent with these cumulants $\Phi$

$\Psi^{\pi^{k}}(\mathbf{s},\mathbf{a})=\Phi(\mathbf{s},\mathbf{a})+\gamma\mathbb{E}^{\mathcal{C},\pi^{k}}\left[\Psi^{\pi^{k}}(\mathbf{s}^{\prime},\pi^{k}(\mathbf{s}))\right]\,.$
(7)

To learn these quantities from $K$ demonstrators, we parameterise the SFs with
$\boldsymbol{\theta}^{\Psi^{k}}$, preferences with $\mathbf{w}^{k}$, and
shared cumulants with $\boldsymbol{\theta}_{\Phi}$. A schematic of the model
architecture and further details are provided in Appendix B. The parameters
are learned by minimising a behavioural cloning and SFs TD loss based on
equations (6) and (7).

Behavioural cloning loss. Given demonstrations generated only by the $k$-th
agent, i.e., $\mathcal{D}^{k}\subset\mathcal{D}$, we train its successor
features $\boldsymbol{\theta}_{\Psi^{k}}$ and the preferences $\mathbf{w}^{k}$
by minimising the negative log-likelihood of the demonstrations

$\tiny\\!\\!\\!\\!\\!\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})\triangleq-\\!\\!\\!\\!\\!\operatorname*{\mathbb{E}}_{\tau\sim\mathcal{D}^{k}}\\!\\!\\!\log\frac{\exp(\Psi(\mathbf{s}_{t},\mathbf{a}_{t};\boldsymbol{\theta}_{\Psi^{k}})^{\top}\mathbf{w}^{k})}{\sum_{a}\exp(\Psi(\mathbf{s}_{t},a;\boldsymbol{\theta}_{\Psi^{k}})^{\top}\mathbf{w}^{k})}\,.$
(8)

Importantly, Eqn. (8) reflects the fact that along a trajectory $\tau$, the
successor features are a function of the state and action at each time-step,
$\mathbf{s}_{t}$ and $\mathbf{a}_{t}$, while the preferences $\mathbf{w}^{k}$
are learnable but consistent across time and trajectories. The direction of
the preference $\mathbf{w}^{k}$ indicates the goal of the agent $k$ by showing
how relatively rewarding it finds the different dimensions of the cumulant
features $\Phi$. The learned magnitude of $\mathbf{w}^{k}$ can further capture
how greedily the agent $k$ pursues this goal.

A sparsity prior, i.e., $\mathcal{L}_{1}$ loss, on preferences
$\mathbf{w}^{k}$ is also used to promote disentangled cumulant dimensions (see
Figure 4). We found the $\mathcal{L}_{1}$ loss made the algorithm more robust
to the choice of dimension of $\Phi$ (see Figure 13 in the Appendix), but did
not substantially affect overall performance.

Inverse temporal difference loss. The cumulant parameters
$\boldsymbol{\theta}_{\Phi}$ are trained to be TD-consistent with all agents’
successor features. This procedure inverts111Hence the name _inverse TD
learning_. the standard TD-learning framework for SFs (Dayan, 1993; Barreto et
al., 2017) where they are trained to be consistent with a fixed cumulant
$\Phi$. Instead, we first train the SFs and preference vectors to ‘explain’
the other agents’ behaviour with the behavioural cloning loss Eqn. (8), and
then train $\boldsymbol{\theta}_{\Phi}$, $\boldsymbol{\theta}_{\Psi^{k}}$ to
be (self-)consistent with these SFs by minimising

$\displaystyle\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Phi},$
$\displaystyle\boldsymbol{\theta}_{\Psi^{k}})\triangleq\operatorname*{\mathbb{E}}_{(\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{s}_{t+1},\mathbf{a}_{t+1},k)\sim\mathcal{D}}\|\Psi(\mathbf{s}_{t},\mathbf{a}_{t};\boldsymbol{\theta}_{\Psi^{k}})\hookleftarrow$
$\displaystyle-\Phi(\mathbf{s}_{t},\mathbf{a}_{t};\boldsymbol{\theta}_{\Phi})-\hskip
1.19995pt\underbracket{\gamma\Psi(\mathbf{s}_{t+1},\mathbf{a}_{t+1};\tilde{\boldsymbol{\theta}}_{\Psi^{k}})}_{\texttt{stop\\_gradient}}\|\,.$
(9)

In practice, our ITD-learning algorithm alternates between minimising
$\mathcal{L}_{\text{BC-}Q}$ for training only the successor features and
preferences, and $\mathcal{L}_{\text{ITD}}$ for training both the shared
cumulants and successor features, provided only with no-reward demonstrations,
as illustrated in blue in Figure 1.

From the definition of cumulants and preferences, we can recover the $k$-th
demonstrator’s reward, by applying Eqn. (1), i.e.,
$R^{k}(\mathbf{s},\mathbf{a})\approx\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi})^{\top}\mathbf{w}^{k}$.
Our ITD algorithm returns both Q-functions that can be used for imitating a
demonstrator and an explicit reward function for each agent, requiring only
access to demonstrations without any online interaction with the simulator.
Hence ITD-learning is an offline multi-task IRL algorithm. ITD is summarised
in Algorithm 1.

######  Theorem 1 (Validity of the ITD Minimiser).

The minimisers of $\mathcal{L}_{\text{BC-}Q}$ and $\mathcal{L}_{\text{ITD}}$
are potentially-shaped cumulants that explain the observed reward-free
demonstrations.

###### Proof.

See Appendix C. ∎

Single-task setting. To gain more intuition about the ITD algorithm, consider
the simpler case of performing IRL with demonstrations from a single policy.
This obviates the need for a representation of preferences, so we can use
$\mathbf{w}=1$. In this case $\Psi$ is the action-value function $Q$ and
$\Phi$ is simply the reward $R$. Minimising (8) reduces to finding a
$Q$-function whose softmax gives the observed policy, and minimising (9) finds
a scalar reward that explains the $Q$-function. Our more general formulation,
with cumulants $\Phi$ in place of a scalar reward, allows us to perform ITD-
learning on demonstrations from many policies, and to efficiently transfer to
new tasks, as we show next.

###  3.2 $\Psi\Phi$-Learning with No-Reward Demonstrations

Now we present our main contribution, $\Psi\Phi$-learning, which combines our
ITD inverse RL algorithm with RL and GPI, using no-reward demonstrations from
other agents to accelerating the ego-agent’s learning. $\Psi\Phi$-learning is
depicted in Figure 1 and summarised in Algorithm 1.

$\Psi\Phi$-learning is an off-policy algorithm based on Q-learning (Watkins &
Dayan, 1992; Mnih et al., 2013). The action-value function is represented with
successor features, $\Psi^{\text{ego}}$, and preferences,
$\mathbf{w}^{\text{ego}}$, as in Eqn. (3). The ego-agent interacts with the
environment, storing its experience in a replay buffer,
$\mathcal{B}\leftarrow\mathcal{B}\cup\\{\left(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},r^{\text{ego}}\right)\\}$.
The $\Psi\Phi$-learner also has estimates for the cumulants, per-agent SFs and
preferences obtained with ITD from the demonstrations $\mathcal{D}$.

Reward loss. The ego-rewards, $r^{\text{ego}}$, are used to ground the
cumulants $\Phi$ and the preferences $\mathbf{w}^{\text{ego}}$, via the loss

$\displaystyle\mathcal{L}_{\text{R}}(\boldsymbol{\theta}_{\Phi},\mathbf{w}^{\text{ego}})\triangleq\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a},r^{\text{ego}})\sim\mathcal{B}}$
$\displaystyle\|\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi})^{\top}\mathbf{w}^{\text{ego}}-r^{\text{ego}}\|\,.$
(10)

Importantly, we share the _same_ cumulants between the ITD-learning from other
agents and the ego-learning, so that they span the joint space of reward
functions. This can be also seen as a representation learning method, where by
enforcing all agents, including the ego-agent, to share the same $\Phi$, we
transfer information about salient features of the environment from learning
about one agent to benefit learning about all agents.

Temporal difference learning. The ego-agent’s successor features are learned
using two losses. First, we train the SFs to fit the Q-values using the
Bellman error

$\displaystyle\mathcal{L}_{Q}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})\triangleq$
$\displaystyle\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},r^{\text{ego}})\sim\mathcal{B}}\|\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{\text{ego}}})^{\top}\mathbf{w}^{\text{ego}}\hookleftarrow$
$\displaystyle-r^{\text{ego}}-\hskip
1.19995pt\underbracket{\gamma\max_{a^{\prime}}\Psi(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\tilde{\boldsymbol{\theta}}_{\Psi^{\text{ego}}})^{\top}\mathbf{w}^{\text{ego}}}_{\texttt{stop-
gradient}}\|\,.$ (11)

We additionally train the successor features to be self-consistent (i.e. to
satisfy Equation 2) using a TD loss $\mathcal{L}_{\text{TD-}\Psi}$.

$\displaystyle\mathcal{L}_{\text{TD-}\Psi}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})\triangleq$
$\displaystyle\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},\mathbf{a}^{\prime})\sim\mathcal{B}}\|\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{\text{ego}}})\hookleftarrow$
$\displaystyle-\Phi(\mathbf{s},\mathbf{a};\tilde{\boldsymbol{\theta}}_{\Phi})-\gamma\hskip
1.19995pt\underbracket{\Psi(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{\Psi^{\text{ego}}})}_{\texttt{stop-
gradient}}\|\,.$ (12)

GPI behavioural policy. Provided SFs estimates for the other agents,
$\\{\boldsymbol{\theta}_{\Psi^{k}}\\}_{k=1}^{K}$, and the ego-agent
$\boldsymbol{\theta}_{\Psi^{\text{ego}}}$, and inferred ego preferences,
$\mathbf{w}^{\text{ego}}$, we adopt an action selection mechanism according to
the GPI rule in Eqn. (5)

$\displaystyle\pi^{\text{ego}}(\mathbf{s})=\operatorname*{arg\,max}_{a}\max_{\boldsymbol{\theta}_{\Psi}}\Psi(\mathbf{s},a;\boldsymbol{\theta}_{\Psi})^{\top}\mathbf{w}^{\text{ego}}\,.$
(13)

The GPI step lets the agent estimate the value of the demonstration policies
on its current task, and then copy the policy that it predicts will be most
useful. We combat model overestimation by acting pessimistically with regard
to an ensemble of two successor features approximators. If the agent’s
estimated values are accurate and the demonstration policies are useful for
the ego task, the GPI policy can obtain good performance faster than policy
iteration with only the ego value function. The next section quantifies this
claim.

Highway

Roundabout

CoinGrid

Fruit Bot

Figure 2:  Environments studied in this paper. Environments (a-b) are multi-
agent environments in which the ego-agent must learn online from other agents,
and learn to navigate around other agents in the environment (see Section
4.2). Environments (c-d) are single-agent. In environment (c) we test whether
the ego-agent can learn offline from a set of demonstrations previously
collected by other agents (see Section 4.5. Environment (d) is used to test
whether our method can scale to more complex, high-dimensional tasks (see
Section 4.2).

Performance Bound. Given a set of demonstration task vectors
$\\{w_{k}\\}_{k=1}^{K}$ and successor features for the corresponding optimal
policies $\\{\Psi^{\pi^{k}}\\}_{k=1}^{K}$, Barreto et al. (2017) show that it
is possible to bound the performance of the GPI policy on the ego-agent task
$w^{\prime}$ as a function of the distance of $w^{\prime}$ from the closest
demonstration task $w_{j}$, and the value approximation error of the predicted
value functions $\widetilde{Q}^{\pi_{i}}=\Psi^{\pi_{i}}w^{\prime}$. We extend
this result to explicitly account for the reward approximation error
$\delta_{r}$ obtained by the the learned cumulants and the SF approximation
error $\delta_{\Psi}$.

######  Theorem 2 (Generalisation Bound of $\Psi\Phi$-Learning).

Let $\pi^{*}$ be the optimal policy for the ego task $w^{\prime}$ and let
$\pi$ be the GPI policy obtained from $\\{\tilde{Q}^{\pi_{i}}\\}$, with
$\delta_{r},\delta_{\Psi}$ the reward and successor feature approximation
errors. Then $\forall s,a$

$\displaystyle Q^{*}(s,a)-Q^{\pi}(s,a)\leq$
$\displaystyle\frac{2}{1-\gamma}\bigg{[}\phi_{\max}\min_{j}\|w^{\prime}-w_{j}\|\hookleftarrow$
$\displaystyle+2\delta_{r}+\|w^{\prime}\|\delta_{\Psi}+\frac{\delta_{r}}{1-\gamma}\bigg{]}\;.$
(14)

###### Proof.

See Appendix C for a formal statement. ∎

In settings where the agent has a good reward and SFs approximation, and the
ego task vector $w^{\prime}$ is close to the demonstration tasks, Theorem 2
says that the ego-agent will attain near-optimal performance from the start of
training.

##  4 Experiments

We conduct a series of experiments to determine how well $\Psi\Phi$-learning
functions as an RL, IRL, imitation learning, and transfer learning algorithm.

Baselines. We benchmark against the following methods: (i) DQN: Deep
Q-learning (Mnih et al., 2013), (ii) BC: Behaviour Cloning, a simple imitation
learning method in which we learn $p(\mathbf{a}|\mathbf{s})$ via supervised
learning on the demonstration data, (iii) DQN+BC-AUX: DQN with an additional
behavior-cloning auxiliary loss (Hernandez-Leal et al., 2019), (iv) GAIL:
Generative Adversarial Imitation Learning (Ho & Ermon, 2016), which uses a
GAN-like approach to approximate the expert policy, and (v) SQILv2: Soft Q
Imitation Learning (Reddy et al., 2019), a recently proposed imitation
technique that combines imitation and RL, and works in the absence of rewards.
For high-dimensional environments, we replace DQN with PPO, Proximal Policy
Optimization (Schulman et al., 2017). Both DQN and PPO are trained to optimize
environment reward through experience, and do not have access to other agents’
experiences.

###  4.1 Environments

Experiments are conducted using four environments, shown in Figure 2. We cover
a broad range of problem setting, including both multi-agent and single-agent
environments, as well as learning online during RL training, or offline from
previously collected demonstrations.

Highway (Leurent, 2018) is a multi-agent autonomous driving environment in
which the ego-agent must safely navigate around other cars and reach its goal.
The other agents follow near-optimal scripted policies for various goals,
depending on the scenario. In the single-task scenario, other agents have the
same objective as the ego-agent, so their experience is directly relevant. In
the adversarial task, the other agents do not move, and the ego-agent has to
accelerate and go to a particular lane while avoiding other vehicles. Finally,
in the multi-task scenario, the other agents and ego-agent have different
preferences over target speed, preferred lane, and following distance. We
consider the multi-task scenario to be the most realistic and representative
of real highway driving with human drivers. In addition to highway driving, we
also study the more complex Roundabout task. Roundabout is inherently multi-
task, in that other agents randomly exit either the first or second exit,
while the ego-agent must learn to take the third exit.

CoinGrid is a single-agent grid-world, environment containing goals of
different colours. We collect offline trajectories of pre-trained agents with
preferences for different goals. red. During training, the ego-agent is only
rewarded for collecting a subset of the possible goals. We can then test how
well the ego-agent is able to transfer to a goal that was never experienced
during training (Section 4.5). This environment also enables learning easily
interpretable preference vectors, allowing us to visualize how well our method
works as an IRL method for inferring rewards (Section 4.3).

FruitBot is a high-dimensional, procedurally generated, single-agent
environment from the OpenAI ProcGen (Cobbe et al., 2020) suite. We use
FruitBot to test whether $\Psi\Phi$-learning can scale up to more complex RL
environments, requiring larger deep neural network architectures that learn
directly from pixels. The agent must navigate around randomly generated
obstacles while collecting fruit, and avoiding other objects and walls. To
create a multi-task version of FruitBot, we define additional tasks which vary
agents’ preferences over collecting objects in the environment, and train PPO
baselines on these task variants. The ego-agent observes the states and
actions of these trained agents playing the game in parallel with its own
interactions.

Highway: Single-task

Highway: Adversarial

Highway: Multi-task

FruitBot

Figure 3:  Learning curves for $\Psi\Phi$-learning and baselines in three
tasks in the multi-agent Highway environment (a-c), and in single-agent
FruitBot (d). Tasks (a) and (b) represent extreme cases where either RL or
imitation learning is irrelevant. In (a) other agents have the same task as
the ego-agent, so imitation learning excels. In the adversarial task (b),
other agents exhibit degenerative behaviour, so imitation learning performs
extremely poorly and traditional RL (DQN) excels. In both of these extreme
cases, $\Psi\Phi$-learning achieves good performance, showing it can flexibly
reap the benefits of either imitation or RL as appropriate. Task (c) is most
realistic; here, other agents have varied preferences and goals that may or
may not relate to the ego-agent’s task. $\Psi\Phi$-learning clearly
outperforms baseline techniques. Similar results are shown in Fruitbot (d),
showing that $\Psi\Phi$-learning scales well to high-dimensional environments,
consistently outperforming baselines like PPO and SQIL. We plot mean
performance over 3 runs and individual runs with alpha.

###  4.2 Accelerating RL with No-Reward Demonstrations

This section addresses two hypotheses: H1: When the unlabelled demonstrations
are relevant, $\Psi\Phi$-learning can accelerate or improve performance of the
ego-agent when learning with online RL; and H2: If the demonstrations are
irrelevant, biased, or are generated by sub-optimal demonstrators,
$\Psi\Phi$-learning can perform at least as well as standard RL.

Figure 3 shows the results of $\Psi\Phi$-learning and the baselines in the
Highway and FruitBot environments. In the single-task scenario (3), when other
agents’ experience is entirely relevant to the ego-agent’s task, imitation
learning methods like BC and SQILv2 learn fastest. DQN learns slowly because
it does not use the other agents’ experience. However, $\Psi\Phi$-learning
achieves competitive results, out-performing DQN+BC-Aux (Hernandez-Leal et
al., 2019; Ndousse et al., 2020). In the adversarial task (3), the other
agents’ behaviors are irrelevant for the ego-agent’s task, so imitation
learning (BC and SQILv2) performs poorly, while traditional RL techniques (DQN
and DQN+BC-Aux) perform best. The performance of $\Psi\Phi$-learning does not
suffer like other imitation learning methods; instead, it retains the
performance of standard RL (H2). $\Psi\Phi$-learning can flexibly reap the
benefits of either imitation learning or RL, depending on what is most
beneficial for the task.

The multi-task scenario (3) is the most realistic autonomous driving task, in
which other agents navigate the highway with varying driving styles. Here,
$\Psi\Phi$-learning clearly out-performs all other methods, suggesting it can
leverage information about other agents’ preferences in order to learn the
underlying task structure of the environment, acclerating performance on the
ego-agent’s RL task (H1). FruitBot (3) gives consistent results, showing that
$\Psi\Phi$-learning scales well to high-dimensional, single-agent tasks while
still outperforming BC, SQIL, PPO, and PPO+BC-Aux.

###  4.3 Inverse Reinforcement Learning

We now test hypothesis H3: ITD is an effective IRL method, and can accurately
infer other agents’ rewards. We present a quantitative and qualitative study
of the rewards for other agents that are inferred by ITD, as well as the
learned cumulants and preferences. Here, we focus solely on offline IRL and
use only ITD to learn from offline reward-free demonstrations, without any
ego-agent experience.

Table 1:  We evaluate how well $\Psi\Phi$-learning is able to infer the
correct reward function by training an RL agent on the inferred rewards, and
comparing this to alternative imitation learning methods in three
environments. All methods are trained on expert demonstrations. A
“$\diamondsuit$” indicates methods that infer an _explicit_ reward function
and then use one of DQN or PPO to train an RL agent, depending on the
environment. A “$\clubsuit$” indicates methods that directly learn a policy
from demonstrations. A “$\dagger$” indicates methods that use privileged task
id information for handling multi-task demonstrations. We report mean and
standard error of _normalised returns_ over 3 runs, where higher-is-better and
the performance is upper bounded by $1.0$, reached by the same RL agent,
trained with the ground truth reward function.

Methods | Roundabout${}^{\texttt{DQN}}$ | CoinGrid${}^{\texttt{DQN}}$ | FruitBot${}^{\texttt{PPO}}$
---|---|---|---
BC†♣ (Pomerleau, 1989) | $0.81{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.02}$ | $0.69{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.06}$ | 0.37${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.02}$
SQIL†♣ (Reddy et al., 2019) | $0.85{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.02}$ | $0.64{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.05}$ | 0.35${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.03}$
GAIL†♢ (Ho & Ermon, 2016) | $0.77{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.07}$ | $0.73{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.02}$ | $0.31{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.02}$
ITD♢ (ours, cf. Section 3.1) | 0.92${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.01}$ | 0.77${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.03}$ | 0.35${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.04}$

To quantitatively evaluate how well ITD can infer rewards, we train an RL
agent on the inferred reward function, and compare the performance to other
imitation learning and IRL methods. Table 1 gives the performance in terms of
normalised returns on all three environments. Using ITD to infer rewards
results in significantly higher performance than BC and SQIL, in two
environments, and competitive performance in FruitBot. We note that unlike
$\Psi\Phi$-learning, BC and SQIL directly learn a policy from demonstrations,
and do not actually infer an explicit reward function. In contrast, GAIL does
infer an explicit reward function, and ITD gives consistently higher
performance than GAIL in all three environments. These results demonstrate
that ITD is an effective IRL technique (H3).

Qualitatively, we can evaluate how well the cumulants inferred by ITD in the
CoinGrid environment span the space of possible goals. We compute the learned
cumulants $\hat{\phi}(s)$ for each square $s$ in the grid. Figure 4 shows the
original CoinGrid game, and Figure 4-4 shows the first three dimensions of the
learned cumulant vector, $\hat{\phi}_{1}$-$\hat{\phi}_{3}$ (the rest are given
in the Appendix). We find that $\hat{\phi}_{1}$ is most active for red coins,
$\hat{\phi}_{2}$ for green, and $\hat{\phi}_{3}$ for yellow. Clearly, ITD has
learned cumulant features that span the space of goals for this game. See
Appendix E for more details and visualisations of the learned rewards and
preferences.

CoinGrid

$\phi_{1}$

$\phi_{2}$

$\phi_{3}$

Figure 4:  Qualitative evaluation of the learned cumulants in the CoinGrid
task. Cumulants $\phi_{1}$, $\phi_{2}$, and $\phi_{3}$ seem to capture the
red, green, and yellow blocks, respectively. Therefore, linear combinations of
the learned cumulants can represent arbitrary rewards in the environment,
which involve stepping on the coloured blocks.

###  4.4 Imitation learning

Here, we investigate hypothesis H4: $\Psi\Phi$-learning works as an effective
imitation learning method, allowing for accurate prediction of other agents’
actions. To test this hypothesis, we train other agents in the Roundabout
environment, then split no-reward demonstrations from these agents into a
train dataset (80%) and a held-out test dataset. We use the train dataset to
run ITD, which means that we use the data to learn both $\Phi$ and the $\Psi$
and $\mathbf{w}$ for other agents. Because it is specifically designed to
accurately predict other agents’ actions, we use BC as the baseline. We
compare this to using only ITD, and using the full $\Psi\Phi$-learning
algorithm including ITD and learning from RL and experience to update the
shared cumulants $\Phi$.

Accuracy in predicting other agents’ actions on the held-out test set is used
to measure imitation learning performance. Figure 5 shows accuracy over the
course of training. At each phase change marked in the figure, the ego-agent
is given a new task, to test how the representation learning benefits from
diverse ego-experience. We see that although BC obtains accurate train
performance, it generalises poorly to the test set, reaching little over 80%
accuracy. Without RL, $\Psi\Phi$-learning achieves similar performance.
However, when using RL to improve imitation, $\Psi\Phi$-learning performs well
on both the train and test set, achieving markedly higher accuracy ($\approx
95$%) in predicting other agents’ behaviour. This suggests that when
$\Psi\Phi$-learning uses RL and interaction with the world to improve the
estimation of the shared cumulants $\Phi$, this in turn improves its ability
to model the $Q$ function of other agents and predict their behaviour.
Further, $\Psi\Phi$-learning adapts well when the agent’s goal changes, since
it uses SFs to disentangle the representation of an agent’s goal from
environment dynamics. Taken together, these results demonstrate that
$\Psi\Phi$-learning also works as a competitive imitation learning method
(H4).

Table 2:  We evaluate how well $\Psi\Phi$-learning is able to transfer to new
tasks in a few-shot fashion. We construct a multi-task variant of the CoinGrid
environment: The ego-agent is provided demonstrations for either capturing
only red coins R or only green coins G. Then it is evaluated on 4 different
tasks: collecting (i) both red and green coins R+G, (ii) collecting red and
avoiding green coins R-G, (iii) avoiding red and collecting green coins -R+G
and (iv) avoiding both red and green coins -R-G. A “$\diamondsuit$” indicates
methods that use a single model for all tasks, while “$\clubsuit$” indicates
methods that require one model per task, i.e., they comprise of 4 models.
Because it disentangles preferences from task representation,
$\Psi\Phi$-learning is able to adapt to reach optimal performance on the new
tasks after a single episode or improve intra-episode from the first episode
after experiencing the first rewards. In contrast, SQIL takes $100$ episodes
to adapt.

| 0-shot | 1-shot | 100-shot
---|---|---|---
Methods | R+G | R-G | -R+G | -R-G | R+G | R-G | -R+G | -R-G | R+G | R-G | -R+G | -R-G
SQILv2♣ (Reddy et al., 2019) | $1.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $0.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $0.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $-1.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $0.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $0.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $-1.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$
$\Psi\Phi$-learning♢ (ours, cf. Section 3.2) | $1.0{\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 0.2${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | 0.2${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ |  $-$0.4${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | 1.0${\color[rgb]{0.5,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0.5}\pgfsys@color@gray@stroke{0.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$

Figure 5:  Test accuracy in predicting other agents’ actions. The shared
cumulants $\Phi$ for modelling others- and ego- reward functions allow our
$\Psi\Phi$-learner to improve its ability to predict others’ actions by
experiencing new ego-tasks. Pure imitation learning and our ITD inverse RL
methods achieve high train accuracy but they do not have a mechanism for
utilising RL experience to improve their generalisation to the test set as new
tasks are provided.

###  4.5 Transfer and Few-Shot Generalisation

Since SFs have been shown to improve generalization and transfer in RL, here
we test hypothesis H5: $\Psi\Phi$-learning will be able to generalize
effectively to new tasks in a few-shot transfer setting. Using the CoinGrid
environment, it is possible to precisely test whether the ego-agent can
generalise to a task it has never experienced during training. Specifically,
we would like to determine whether: (i) the ego-agent can generalise to tasks
it was never rewarded for during training (but which it may have seen other
agents demonstrate), and (ii) the ego-agent can generalise to tasks not
experienced by _any_ agents during training.

Table 2 shows the results of transfer experiments in which agents are given 0,
1, or 100 additional training episodes to adapt to a new task. Unlike SQIL,
$\Psi\Phi$-learning is able to adapt 0-shot to obtain some reward on the new
tasks, and fully adapt after a single episode to achieve the maximum reward on
all transfer tasks. This is because $\Psi\Phi$-learning uses SFs to
disentangle preferences (goals) in the task representation, and learn about
the space of possible preferences from observing other agents. To adapt to a
new task, it need only infer the correct preference vector. Task inference is
trivially implemented as a least squares regression problem, see Eqn. (10):
Having experienced $\mathcal{B}^{\text{new}}$ in the new task, the
$\Psi\Phi$-learner identifies the preference vector for the new task by
solving
$\min_{\mathbf{w}}\sum_{\mathcal{B}^{\text{new}}}\mathcal{L}_{\text{R}}(\boldsymbol{\theta}_{\Phi},\mathbf{w})$.
In contrast, SQIL requires $100$ episodes to reach the same performance.

##  5 Related Work

Learning from demonstrations. Learning from demonstrations, also referred to
as imitation learning (IL, Widrow & Smith, 1964; Pomerleau, 1989; Atkeson &
Schaal, 1997), is an attractive framework for sequential decision making when
reliable, expert demonstrations are available. Early work on IL assumed access
to high-quality demonstrations and aimed to match the expert policy
(Pomerleau, 1991; Heskes, 1998; Ng et al., 2000; Abbeel & Ng, 2004; Billard et
al., 2008; Argall et al., 2009; Ziebart et al., 2008). Building on this
assumption, many recent works have studied various aspects of both _single-
task_ (Ratliff et al., 2006; Wulfmeier et al., 2015; Choi & Kim, 2011; Finn et
al., 2016b; Ho & Ermon, 2016; Finn et al., 2016a; Fu et al., 2017; Zhang et
al., 2018; Rahmatizadeh et al., 2018) and _multi-task_ (Dimitrakakis &
Rothkopf, 2011; Mülling et al., 2013; Stulp et al., 2013; Deisenroth et al.,
2014; Sharma et al., 2018; Codevilla et al., 2018; Fu et al., 2019; Rhinehart
et al., 2020; Filos et al., 2020) IL. However, these methods are all limited
by the performance of the demonstrator that they try to imitate. Learning from
suboptimal demonstrations has been studied by Coates et al. (2008); Grollman &
Billard (2011); Zheng et al. (2014); Choi et al. (2019); Shiarlis et al.
(2016); Brown et al. (2019), enabling, under certain assumptions, imitation
learners to surpass their demonstrators’ performance. In contrast, our method
integrates demonstrations into an online reinforcement learning pipeline and
can use the demonstrations to improve learning on a new task. Our inverse TD
(ITD) learning offline multi-task inverse reinforcement learning algorithm is
similar to the Cascaded Supervised IRL (CSI) approach (Klein et al., 2013).
However, CSI assumes a single-task, deterministic expert while ITD does not.

Reinforcement learning with demonstrations. Demonstration trajectories have
been used to accelerate the learning of RL agents (Taylor et al., 2011;
Vecerik et al., 2017; Rajeswaran et al., 2017; Hester et al., 2018; Gao et
al., 2018; Nair et al., 2018; Paine et al., 2018, 2019), as well as
demonstrations where actions and/or rewards are unknown (Borsa et al., 2017;
Torabi et al., 2018; Sermanet et al., 2018; Liu et al., 2018; Aytar et al.,
2018; Brown et al., 2019). In contrast to the standard imitation learning
setup, these methods allow improving over the expert performance as the policy
can be further fine-tuned via reinforcement learning. Offline reinforcement
learning with online fine-tuning (Kalashnikov et al., 2018; Levine et al.,
2020) can be framed under this settings too. Our method builds on the same
principles, however, unlike these works, we do not assume that the
demonstration data either come with reward annotations, or that they relate to
the same task the RL agent is learning (i.e., we learn from multi-task
demonstrations which may include irrelevant tasks).

Successor features. Successor features (SFs) are a generalisation of the
successor representation (Dayan, 1993) for continuous state and action spaces
(Barreto et al., 2017). Prior work has used SFs for (i) zero-shot transfer
(Barreto et al., 2017; Borsa et al., 2018; Barreto et al., 2020); (ii)
exploration (Janz et al., 2019; Machado et al., 2020); (iii) skills discovery
(Machado et al., 2017; Hansen et al., 2019); (iv) hierarchical RL (Barreto et
al., 2019) and theory of mind (Rabinowitz et al., 2018). Nonetheless, in all
the aforementioned settings, direct access to the rewards or cumulants was
provided. Our method, instead, uses demonstrations without reward labels for
inferring the cumulants and learning the corresponding SFs. More closely to
this work, Lee et al. (2019) propose learning cumulants and successor features
for a _single-task_ IRL setting. Their approach differs from ours in two key
respects: first, they use a learned dynamics model to learn the cumulants.
Second, the learned SFs are used for representing the action-value function,
not to inform the behaviour policy with GPI.

Model of others in multi-agent learning. Our method draws inspiration and
builds on the multi-agent learning setting, where multiple agents participate
in the same environment and the states, actions of others are observed
(Davidson, 1999; Lockett et al., 2007; He et al., 2016; Jaques et al., 2019).
However, we do not explore _strategic_ settings, where recursive reasoning
(Stahl, 1993; Yoshida et al., 2008) is necessary for optimal behaviour.

##  6 Discussion

We have presented two major algorithmic contributions. The first, ITD, is a
novel and flexible offline IRL algorithm that discovers salient task-agnostic
environment features in the form of cumulants, as well as learning successor
features and preference vectors for each agent which provides demonstrations.
The second, $\Psi\Phi$-learning, combines ITD with RL from online experience.
This makes efficient use of unlabelled demonstrations to accelerate RL, and
comes with theoretical worst-case performance guarantees. We showed
empirically the advantages of these algorithms over various baselines: how
imitation with ITD can improve RL and enable zero-shot transfer to new tasks,
and how experience from online RL can help to improve imitation in turn.

Future Work. We want to explore ways to: (i) adapt $\Psi\Phi$-learning to
multi-agent _strategic_ settings, where coordination and opponent modelling
(Albrecht & Stone, 2018) are essential and (ii) use a universal successor
features approximator (Borsa et al., 2018) for ITD, overcoming its current,
linear scaling with the number of distinct demonstrators.

Acknowledgements. We thank Pablo Samuel Castro, Anna Harutyunyan, RAIL, OATML
and IRIS lab members for their helpful feedback. We also thank the anonymous
reviewers for useful comments during the review process. A.F. is funded by a
J.P.Morgan PhD Fellowship and C.L. is funded by an Open Phil AI Fellowship.

## References

  * Abbeel & Ng (2004) Abbeel, P. and Ng, A. Y.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , pp. 1, 2004.
  * Albrecht & Stone (2018) Albrecht, S. V. and Stone, P.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  _Artificial Intelligence_ , 258:66–95, 2018.
  * Argall et al. (2009) Argall, B. D., Chernova, S., Veloso, M., and Browning, B.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ , 57(5):469–483, 2009.
  * Atkeson & Schaal (1997) Atkeson, C. G. and Schaal, S.  Robot learning from demonstration.  In _ICML_ , volume 97, pp. 12–20. Citeseer, 1997.
  * Aytar et al. (2018) Aytar, Y., Pfaff, T., Budden, D., Paine, T. L., Wang, Z., and de Freitas, N.  Playing hard exploration games by watching youtube.  _arXiv preprint arXiv:1805.11592_ , 2018.
  * Babuschkin et al. (2020) Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Fantacci, C., Godwin, J., Jones, C., Hennigan, T., Hessel, M., Kapturowski, S., Keck, T., Kemaev, I., King, M., Martens, L., Mikulik, V., Norman, T., Quan, J., Papamakarios, G., Ring, R., Ruiz, F., Sanchez, A., Schneider, R., Sezener, E., Spencer, S., Srinivasan, S., Stokowiec, W., and Viola, F.  The DeepMind JAX Ecosystem, 2020.  URL http://github.com/deepmind.
  * Barreto et al. (2017) Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D.  Successor features for transfer in reinforcement learning.  In _Advances in neural information processing systems_ , pp. 4055–4065, 2017.
  * Barreto et al. (2019) Barreto, A., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Mourad, S., Silver, D., Precup, D., et al.  The option keyboard: Combining skills in reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pp. 13052–13062, 2019.
  * Barreto et al. (2020) Barreto, A., Hou, S., Borsa, D., Silver, D., and Precup, D.  Fast reinforcement learning with generalized policy updates.  _Proceedings of the National Academy of Sciences_ , 117(48):30079–30087, 2020.
  * Biewald (2020) Biewald, L.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Billard et al. (2008) Billard, A., Calinon, S., Dillmann, R., and Schaal, S.  Survey: Robot programming by demonstration.  _Handbook of robotics_ , 59(BOOK_CHAP), 2008.
  * Borsa et al. (2017) Borsa, D., Piot, B., Munos, R., and Pietquin, O.  Observational learning by reinforcement learning.  _arXiv preprint arXiv:1706.06617_ , 2017.
  * Borsa et al. (2018) Borsa, D., Barreto, A., Quan, J., Mankowitz, D., Munos, R., van Hasselt, H., Silver, D., and Schaul, T.  Universal successor features approximators.  _arXiv preprint arXiv:1812.07626_ , 2018.
  * Boyd et al. (2004) Boyd, S., Boyd, S. P., and Vandenberghe, L.  _Convex optimization_.  Cambridge university press, 2004.
  * Bradbury et al. (2018) Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brown et al. (2019) Brown, D. S., Goo, W., Nagarajan, P., and Niekum, S.  Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.  _arXiv preprint arXiv:1904.06387_ , 2019.
  * Chevalier-Boisvert et al. (2018) Chevalier-Boisvert, M., Willems, L., and Pal, S.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Chevalier-Boisvert et al. (2019) Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y.  BabyAI: First steps towards grounded language learning with a human in the loop.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rJeXCo0cYX.
  * Choi & Kim (2011) Choi, J. and Kim, K.-E.  Inverse reinforcement learning in partially observable environments.  _Journal of Machine Learning Research_ , 12:691–730, 2011\.
  * Choi et al. (2019) Choi, S., Lee, K., and Oh, S.  Robust learning from demonstrations with mixed qualities using leveraged gaussian processes.  _IEEE Transactions on Robotics_ , 35(3):564–576, 2019.
  * Coates et al. (2008) Coates, A., Abbeel, P., and Ng, A. Y.  Learning for control from multiple demonstrations.  In _Proceedings of the 25th international conference on Machine learning_ , pp. 144–151, 2008.
  * Cobbe et al. (2020) Cobbe, K., Hesse, C., Hilton, J., and Schulman, J.  Leveraging procedural generation to benchmark reinforcement learning.  In _International conference on machine learning_ , pp. 2048–2056. PMLR, 2020.
  * Codevilla et al. (2018) Codevilla, F., Miiller, M., López, A., Koltun, V., and Dosovitskiy, A.  End-to-end driving via conditional imitation learning.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1–9. IEEE, 2018.
  * Davidson (1999) Davidson, A.  Using artificial neural networks to model opponents in texas hold’em.  _Unpublished manuscript_ , 1999.
  * Dayan (1993) Dayan, P.  Improving generalization for temporal difference learning: The successor representation.  _Neural Computation_ , 5(4):613–624, 1993.
  * Deisenroth et al. (2014) Deisenroth, M. P., Englert, P., Peters, J., and Fox, D.  Multi-task policy search for robotics.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3876–3881. IEEE, 2014.
  * Dimitrakakis & Rothkopf (2011) Dimitrakakis, C. and Rothkopf, C. A.  Bayesian multitask inverse reinforcement learning.  In _European workshop on reinforcement learning_ , pp. 273–284. Springer, 2011.
  * Espeholt et al. (2018) Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International Conference on Machine Learning_ , pp. 1407–1416. PMLR, 2018.
  * Filos et al. (2020) Filos, A., Tigkas, P., McAllister, R., Rhinehart, N., Levine, S., and Gal, Y.  Can autonomous vehicles identify, recover from, and adapt to distribution shifts?  In _International Conference on Machine Learning_ , pp. 3145–3153. PMLR, 2020.
  * Finn et al. (2016a) Finn, C., Christiano, P., Abbeel, P., and Levine, S.  A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.  _arXiv preprint arXiv:1611.03852_ , 2016a.
  * Finn et al. (2016b) Finn, C., Levine, S., and Abbeel, P.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International conference on machine learning_ , pp. 49–58, 2016b.
  * Franklin & Graesser (1996) Franklin, S. and Graesser, A.  Is it an agent, or just a program?: A taxonomy for autonomous agents.  In _International Workshop on Agent Theories, Architectures, and Languages_ , pp. 21–35. Springer, 1996.
  * Fu et al. (2017) Fu, J., Luo, K., and Levine, S.  Learning robust rewards with adversarial inverse reinforcement learning.  _arXiv preprint arXiv:1710.11248_ , 2017.
  * Fu et al. (2019) Fu, J., Korattikara, A., Levine, S., and Guadarrama, S.  From language to goals: Inverse reinforcement learning for vision-based instruction following.  _arXiv preprint arXiv:1902.07742_ , 2019.
  * Gao et al. (2018) Gao, Y., Xu, H., Lin, J., Yu, F., Levine, S., and Darrell, T.  Reinforcement learning from imperfect demonstrations.  _arXiv preprint arXiv:1802.05313_ , 2018.
  * Grollman & Billard (2011) Grollman, D. H. and Billard, A.  Donut as i do: Learning from failed demonstrations.  In _2011 IEEE International Conference on Robotics and Automation_ , pp. 3804–3809. IEEE, 2011.
  * Hansen et al. (2019) Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., and Mnih, V.  Fast task inference with variational intrinsic successor features.  _arXiv preprint arXiv:1906.05030_ , 2019.
  * He et al. (2016) He, H., Boyd-Graber, J., Kwok, K., and Daumé III, H.  Opponent modeling in deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1804–1813, 2016.
  * Hennigan et al. (2020) Hennigan, T., Cai, T., Norman, T., and Babuschkin, I.  Haiku: Sonnet for JAX, 2020.  URL http://github.com/deepmind/dm-haiku.
  * Henrich (2017) Henrich, J.  _The secret of our success: How culture is driving human evolution, domesticating our species, and making us smarter_.  Princeton University Press, 2017.
  * Hernandez-Leal et al. (2017) Hernandez-Leal, P., Kaisers, M., Baarslag, T., and de Cote, E. M.  A survey of learning in multiagent environments: Dealing with non-stationarity.  _arXiv preprint arXiv:1707.09183_ , 2017.
  * Hernandez-Leal et al. (2019) Hernandez-Leal, P., Kartal, B., and Taylor, M. E.  Agent modeling as auxiliary task for deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , volume 15, pp. 31–37, 2019\.
  * Heskes (1998) Heskes, T.  Solving a huge number of simular tasks: a combination of multi-task learning and a hierarchical bayesian approach.  1998\.
  * Hester et al. (2018) Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al.  Deep q-learning from demonstrations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Ho & Ermon (2016) Ho, J. and Ermon, S.  Generative adversarial imitation learning.  _arXiv preprint arXiv:1606.03476_ , 2016.
  * Hoffman et al. (2020) Hoffman, M., Shahriari, B., Aslanides, J., Barth-Maron, G., Behbahani, F., Norman, T., Abdolmaleki, A., Cassirer, A., Yang, F., Baumli, K., et al.  Acme: A research framework for distributed reinforcement learning.  _arXiv preprint arXiv:2006.00979_ , 2020.
  * Hunter (2007) Hunter, J. D.  Matplotlib: A 2d graphics environment.  _IEEE Annals of the History of Computing_ , 9(03):90–95, 2007.
  * Jaderberg et al. (2017) Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al.  Population based training of neural networks.  _arXiv preprint arXiv:1711.09846_ , 2017.
  * Janz et al. (2019) Janz, D., Hron, J., Mazur, P., Hofmann, K., Hernández-Lobato, J. M., and Tschiatschek, S.  Successor uncertainties: exploration and uncertainty in temporal difference learning.  In _Advances in Neural Information Processing Systems_ , pp. 4507–4516, 2019.
  * Jaques et al. (2019) Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and De Freitas, N.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 3040–3049. PMLR, 2019.
  * Kalashnikov et al. (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning_ , pp. 651–673. PMLR, 2018.
  * Kesting et al. (2010) Kesting, A., Treiber, M., and Helbing, D.  Enhanced intelligent driver model to access the impact of driving strategies on traffic capacity.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , 368(1928):4585–4605, 2010.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Klein et al. (2013) Klein, E., Piot, B., Geist, M., and Pietquin, O.  A cascaded supervised learning approach to inverse reinforcement learning.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pp. 1–16. Springer, 2013.
  * Kulkarni et al. (2016) Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J.  Deep successor reinforcement learning.  _arXiv preprint arXiv:1606.02396_ , 2016.
  * Laland (2018) Laland, K. N.  _Darwin’s unfinished symphony: How culture made the human mind_.  Princeton University Press, 2018.
  * Lee et al. (2019) Lee, D., Srinivasan, S., and Doshi-Velez, F.  Truly batch apprenticeship learning with deep successor features.  _arXiv preprint arXiv:1903.10077_ , 2019.
  * Leurent (2018) Leurent, E.  An environment for autonomous driving decision-making.  https://github.com/eleurent/highway-env, 2018.
  * Levine et al. (2020) Levine, S., Kumar, A., Tucker, G., and Fu, J.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Liaw et al. (2018) Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I.  Tune: A research platform for distributed model selection and training.  _arXiv preprint arXiv:1807.05118_ , 2018.
  * Liu et al. (2018) Liu, Y., Gupta, A., Abbeel, P., and Levine, S.  Imitation from observation: Learning to imitate behaviors from raw video via context translation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1118–1125. IEEE, 2018.
  * Lockett et al. (2007) Lockett, A. J., Chen, C. L., and Miikkulainen, R.  Evolving explicit opponent models in game playing.  In _Proceedings of the 9th annual conference on Genetic and evolutionary computation_ , pp. 2106–2113, 2007.
  * Machado et al. (2017) Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., and Campbell, M.  Eigenoption discovery through the deep successor representation.  _arXiv preprint arXiv:1710.11089_ , 2017.
  * Machado et al. (2020) Machado, M. C., Bellemare, M. G., and Bowling, M.  Count-based exploration with the successor representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 5125–5133, 2020.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mülling et al. (2013) Mülling, K., Kober, J., Kroemer, O., and Peters, J.  Learning to select and generalize striking movements in robot table tennis.  _The International Journal of Robotics Research_ , 32(3):263–279, 2013.
  * Nair et al. (2018) Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P.  Overcoming exploration in reinforcement learning with demonstrations.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 6292–6299. IEEE, 2018.
  * Ndousse et al. (2020) Ndousse, K., Eck, D., Levine, S., and Jaques, N.  Multi-agent social reinforcement learning improves generalization.  _arXiv preprint arXiv:2010.00581_ , 2020.
  * Ng et al. (1999) Ng, A. Y., Harada, D., and Russell, S.  Policy invariance under reward transformations: Theory and application to reward shaping.  In _Icml_ , volume 99, pp. 278–287, 1999.
  * Ng et al. (2000) Ng, A. Y., Russell, S. J., et al.  Algorithms for inverse reinforcement learning.  In _Icml_ , volume 1, pp. 663–670, 2000.
  * Paine et al. (2018) Paine, T. L., Colmenarejo, S. G., Wang, Z., Reed, S., Aytar, Y., Pfaff, T., Hoffman, M. W., Barth-Maron, G., Cabi, S., Budden, D., et al.  One-shot high-fidelity imitation: Training large-scale deep nets with rl.  _arXiv preprint arXiv:1810.05017_ , 2018.
  * Paine et al. (2019) Paine, T. L., Gulcehre, C., Shahriari, B., Denil, M., Hoffman, M., Soyer, H., Tanburn, R., Kapturowski, S., Rabinowitz, N., Williams, D., et al.  Making efficient use of demonstrations to solve hard exploration problems.  _arXiv preprint arXiv:1909.01387_ , 2019.
  * Pomerleau (1989) Pomerleau, D. A.  Alvinn: An autonomous land vehicle in a neural network.  In _Neural Information Processing Systems (NeurIPS)_ , pp. 305–313, 1989.
  * Pomerleau (1991) Pomerleau, D. A.  Efficient training of artificial neural networks for autonomous navigation.  _Neural computation_ , 3(1):88–97, 1991.
  * Puterman (2014) Puterman, M. L.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Rabinowitz et al. (2018) Rabinowitz, N. C., Perbet, F., Song, H. F., Zhang, C., Eslami, S., and Botvinick, M.  Machine theory of mind.  _arXiv preprint arXiv:1802.07740_ , 2018.
  * Rahmatizadeh et al. (2018) Rahmatizadeh, R., Abolghasemi, P., Bölöni, L., and Levine, S.  Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3758–3765. IEEE, 2018.
  * Rajeswaran et al. (2017) Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Ratliff et al. (2006) Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. A.  Maximum margin planning.  In _Proceedings of the 23rd international conference on Machine learning_ , pp. 729–736, 2006.
  * Reddy et al. (2019) Reddy, S., Dragan, A. D., and Levine, S.  Sqil: Imitation learning via reinforcement learning with sparse rewards.  _arXiv preprint arXiv:1905.11108_ , 2019.
  * Rhinehart et al. (2020) Rhinehart, N., McAllister, R., and Levine, S.  Deep imitative models for flexible inference, planning, and control.  In _International Conference on Learning Representations (ICLR)_ , April 2020.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sermanet et al. (2018) Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1134–1141. IEEE, 2018.
  * Sharma et al. (2018) Sharma, P., Mohan, L., Pinto, L., and Gupta, A.  Multiple interactions made easy (mime): Large scale demonstrations data for imitation.  _arXiv preprint arXiv:1810.07121_ , 2018.
  * Shiarlis et al. (2016) Shiarlis, K., Messias, J., and Whiteson, S.  Inverse reinforcement learning from failure.  2016\.
  * Stahl (1993) Stahl, D. O.  Evolution of smart-n players.  _Games and Economic Behavior_ , 5(4):604–617, 1993.
  * Stulp et al. (2013) Stulp, F., Raiola, G., Hoarau, A., Ivaldi, S., and Sigaud, O.  Learning compact parameterized skills with a single regression.  In _2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids)_ , pp. 417–422. IEEE, 2013.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Taylor et al. (2011) Taylor, M. E., Suay, H. B., and Chernova, S.  Integrating reinforcement learning with human demonstrations of varying ability.  In _The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2_ , pp. 617–624, 2011.
  * Torabi et al. (2018) Torabi, F., Warnell, G., and Stone, P.  Behavioral cloning from observation.  _arXiv preprint arXiv:1805.01954_ , 2018.
  * Van Rossum & Drake Jr (1995) Van Rossum, G. and Drake Jr, F. L.  _Python reference manual_.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * Vecerik et al. (2017) Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _arXiv preprint arXiv:1707.08817_ , 2017.
  * Watkins & Dayan (1992) Watkins, C. J. and Dayan, P.  Q-learning.  _Machine learning_ , 8(3-4):279–292, 1992.
  * Widrow & Smith (1964) Widrow, B. and Smith, F. W.  Pattern-recognizing control systems, 1964.
  * Wulfmeier et al. (2015) Wulfmeier, M., Ondruska, P., and Posner, I.  Maximum entropy deep inverse reinforcement learning.  _arXiv preprint arXiv:1507.04888_ , 2015.
  * Yoshida et al. (2008) Yoshida, W., Dolan, R. J., and Friston, K. J.  Game theory of mind.  _PLoS Comput Biol_ , 4(12):e1000254, 2008.
  * Zhang et al. (2018) Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1–8. IEEE, 2018.
  * Zheng et al. (2014) Zheng, J., Liu, S., and Ni, L. M.  Robust bayesian inverse reinforcement learning with sparse behavior noise.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 28, 2014.
  * Ziebart et al. (2008) Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

##  Appendix A Experimental Details

In this section we describe the environments used in our experiments (see
Section 4) and the experiment design.

###  A.1 Highway

We build on the highway-v0 task from the highway-env traffic simulator
(Leurent, 2018). The task is specified by:

  1. 1.

State space, $\mathcal{S}$: The kinematic information of the ego vehicle and
the five closest vehicles (ordered from closest to the furthest) is used as
the Markov state, i.e.,
$\mathbf{s}_{t}=\\{\left[x_{t},y_{t},\dot{x}_{t},\dot{y}_{t}\right]\\}_{\text{ego},\
\text{other}_{1},\ \ldots,\ \text{other}_{5}}\in\mathbb{R}^{6\times 4}$. The
ego-car is illustrated in green and the other cars in blue.

  2. 2.

Action space, $\mathcal{A}$: We use a discrete action space, constructed by
$K$-means clustering of the continuous actions of the intelligent driving
model (Kesting et al., 2010). We found out that keeping 9 actions was
sufficient, i.e., $\mathbf{a}_{t}\in\\{0,\ldots,8\\}$.

  3. 3.

Demonstrations, $\mathcal{D}$: At each time-step, the ego-car observes online
the state-action pairs for the 5 closest cars.

Figure 6: Highway

###  A.2 Roundabout

We build on the roundabot-v0 task from the highway-env traffic simulator
(Leurent, 2018). The task is specified by:

  1. 1.

State space, $\mathcal{S}$: The kinematic information of the ego vehicle and
the five closest vehicles (ordered from closest to the furthest) is used as
the Markov state, i.e.,
$\mathbf{s}_{t}=\\{\left[x_{t},y_{t},\dot{x}_{t},\dot{y}_{t}\right]\\}_{\text{ego},\
\text{other}_{1},\ \ldots,\ \text{other}_{3}}\in\mathbb{R}^{4\times 4}$. The
ego-car is illustrated in green and the other cars in blue.

  2. 2.

Action space, $\mathcal{A}$: We use a discrete action space, constructed by
$K$-means clustering of the continuous actions of the intelligent driving
model (Kesting et al., 2010). We found out that keeping 6 actions was
sufficient, i.e., $\mathbf{a}_{t}\in\\{0,\ldots,5\\}$.

  3. 3.

Demonstrations, $\mathcal{D}$: At each time-step, the ego-car observes online
the state-action pairs for the 3 closest cars.

Figure 7: Roundabout

###  A.3 CoinGrid

We build a simple multi-task grid-world. The task is specified by:

  1. 1.

State space, $\mathcal{S}$: We use a symbolic, multi-channel representation of
the $7\times 7$ gridworld (Chevalier-Boisvert et al., 2019): the first three
channels specify the presence or absence of the three different coloured
boxes, the forth channel was the walls mask and the fifth and last channel was
the position and orientation of the agent. We represent the orientation of the
agent by ‘painting’ the cell in front of the agent. Therefore
$\mathbf{s}_{t}\in\\{0,1\\}^{7\times 7\times 5}$.

  2. 2.

Action space, $\mathcal{A}$: We use the {LEFT, RIGHT, FORWARD} actions from
Minigrid (Chevalier-Boisvert et al., 2018) to navigate the maze, i.e.,
$\mathbf{a}_{t}\in\\{0,1,2\\}$.

  3. 3.

Demonstrations, $\mathcal{D}$: At the beginning of training, the agent is
given state-action pairs of other agents collecting either red or green coins.

Figure 8: CoinGrid

###  A.4 Fruitbot

We build on the Fruitbot environment from OpenAI’s ProcGen benchmark (Cobbe et
al., 2020). The task is specified by:

  1. 1.

State space, $\mathcal{S}$: We use the original high-dimensional $64\times 64$
RGB observations, i.e., $\mathbf{s}_{t}\in\left[0,1\right]^{64\times 64\times
3}$.

  2. 2.

Action space, $\mathcal{A}$: We use the original 15 discrete actions, i.e.,
$\mathbf{a}_{t}\in\\{0,\ldots,14\\}$.

  3. 3.

Demonstrations, $\mathcal{D}$: At each time-step, the agent observes online
the states and actions of 3 trained agents playing the game in parallel: One
agent collects both fruits and other objects, one collects other objects and
avoids fruits and the last one randomly selects actions.

Figure 9: Fruitbot

##  Appendix B Implementation Details

For our experiments we used Python (Van Rossum & Drake Jr, 1995). We used JAX
(Bradbury et al., 2018; Babuschkin et al., 2020) as the core computational
library, Haiku (Hennigan et al., 2020) and Acme (Hoffman et al., 2020) for
implementing $\Psi\Phi$-learning and the baselines, see Section 4. We also
used Matplotlib (Hunter, 2007) for the visualisations and Weightd & Biases
(Biewald, 2020) for managing the experiments.

###  B.1 Computation Graph

$\Phi$$\Psi^{k}$$\mathcal{L}_{\text{ITD}}$$\mathbf{w}^{k}$$Q^{k}$$\mathcal{D}$$\mathcal{L}_{\text{BC-}Q}$$k=1\ldots
K$$\Psi^{\text{ego}}$$\mathcal{L}_{\text{TD-}\Psi}$$\mathbf{w}^{\text{ego}}$$Q^{\text{ego}}$GPI$\pi^{\text{ego}}$$\mathcal{B}$$\mathcal{L}_{\text{TD-}Q}$$r^{\text{ego}}$$\mathcal{L}_{R}$

Figure 10:  Computational graph of the $\Psi\Phi$-learning algorithm.
Demonstrations $\mathcal{D}$ contain data from other agents for _unknown_
tasks. We employ _inverse temporal difference learning_ (ITD, see Section 3.1)
to recover other agents’ successor features (SFs) and preferences. The ego-
agent combines the estimated SFs of others along with its own preferences and
successor features with generalised policy improvement (GPI, see Section 2.2),
generating experience. Both the demonstrations and the ego-experience are used
to learn the shared cumulants. Losses $\mathcal{L}_{*}$ are represented with
double arrows and gradients flow according to the pointed direction(s).

###  B.2 Neural Network Architecture

$\mathbf{s}_{t}$$\mathcal{E}$$\Phi(\mathbf{s}_{t},\cdot)$$\Psi^{1}(\mathbf{s}_{t},\cdot)$$\Psi^{1}(\mathbf{s}_{t},\cdot)$$\mathbf{w}^{1}$$\cdots$$\Psi^{K}(\mathbf{s}_{t},\cdot)$$\Psi^{K}(\mathbf{s}_{t},\cdot)$$\mathbf{w}^{K}$$\Psi^{\text{ego}}(\mathbf{s}_{t},\cdot)$$\Psi^{\text{ego}}(\mathbf{s}_{t},\cdot)$$\mathbf{w}^{\text{ego}}$

Figure 11:  Neural network architecture of the $\Psi\Phi$-learner. The
rectangular nodes are tensors parametrised by MLPs and the circles are
learnable vectors. We share an observation network/torso, $\mathcal{E}$,
across all the network heads. The network heads that related to the other
agents are in blue and trained from demonstrations $\mathcal{D}$. The ego-
agent’s experience $\mathcal{B}$ is used for training the green heads. The
shared cumulants and torso are trained with both $\mathcal{D}$ and
$\mathcal{B}$. An ensemble of two successor features approximators is used for
the ego- and other- agents for combatting model overestimation, see Section 3.

###  B.3 Hyperparameters

Table 3:  $\Psi\Phi$-learner’s hyperparameters per environment. The tuning was
performed on a DQN (Mnih et al., 2013) baseline with population based training
(Jaderberg et al., 2017) using Weights & Biases (Biewald, 2020) integration
with Ray Tune (Liaw et al., 2018). We selected the best hyperparameters
configuration out of 32 trials per environment and used this for our
$\Psi\Phi$-learner.

| Highway | CoinGrid | FruitBot
---|---|---|---
Torso network, $\mathcal{E}$ | MLP([512, 256]) | IMPALA (Espeholt et al., 2018), shallow (no LSTM) | IMPALA (Espeholt et al., 2018), deep (no LSTM)
Cumulants approximator, $\Phi$ | MLP([128, 128]) | MLP([256, 128]) | MLP([256, 128])
Successor features approximator, $\Psi$ | MLP([256, 128]) | MLP([512, 256]) | MLP([512, 256])
Ensemble size, $\Psi$ | 2 | 2 | 2
$\mathcal{L}_{1}$ coefficient | 0.05 | 0.05 | 0.05
Number of dimensions in $\Phi$ | 8 | 4 | 64
Minibatch size | 512 | 64 | 32
$n$-step | 4 | 8 | 128
Discount factor, $\gamma$ | 1.0 | 0.9 | 0.999
Target network update period | 100 | 1000 | 2500
Optimiser | ADAM (Kingma & Ba, 2014), lr=1e-3 | ADAM (Kingma & Ba, 2014), lr=1e-4 | ADAM (Kingma & Ba, 2014), lr=5e-5

###  B.4 Compute Resources

All the experiments were run on Microsoft Azure Standard_NC6s_v3 machines,
i.e., with a 6-core vCPU, 112GB RAM and a single NVIDIA Tesla V100 GPU. The
iteration cycle for (i) Highway experiments was 3 hours; (ii) CoinGrid
experiments was 5.5 hours and (iii) Fruitbot experiments was 19 hours.

##  Appendix C Proofs

First, we formalise the statement of Theorem 1.

######  Theorem 1 (Validity of the ITD Minimiser).

The minimisers of $\mathcal{L}_{\text{BC-}Q}$ and $\mathcal{L}_{\text{ITD}}$
are potentially-shaped cumulants that explain the observed reward-free
demonstrations.

###### Proof.

First, we prove the validity of the minimiser of the inverse temporal
difference learning for the single-task setting. Next, we show that the result
holds true in the vector (i.e., cumulants) case.

Single task. We assume that our demonstrations are generated by an expert, who
samples actions from a Boltzmann policy, according to the optimal (for its
task) action-value function $Q^{\pi_{\text{expert}}}$ and temperature $\nu>0$,
i.e., $\mathcal{D}=\\{(\mathbf{s},\mathbf{a})\\}\sim\pi_{\text{expert}}$ s.t.

$\displaystyle\pi_{\text{expert}}(\mathbf{a}|\mathbf{s})\triangleq
p(A=\mathbf{a}|\mathbf{s})=\frac{\exp(\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s},\mathbf{a}))}{\sum_{a}\exp(\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s},a))},\
\forall\mathbf{s}\in\mathcal{S},\mathbf{a}\in\mathcal{A}\,.$ (15)

The minimiser of the behavioural cloning loss
$\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{Q})$, i.e. Eqn. (8), for a
single expert is s.t.

$\displaystyle\boldsymbol{\theta}_{Q}^{*}$
$\displaystyle\in\operatorname*{arg\,min}_{\boldsymbol{\theta}_{Q}}\
-\\!\\!\\!\\!\\!\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}}\log\frac{\exp(Q(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{Q}))}{\sum_{a}\exp(Q(\mathbf{s},a;\boldsymbol{\theta}_{Q}))}$
(16) $\displaystyle\Rightarrow
Q(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{Q}^{*})=\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s},\mathbf{a})+F(\mathbf{s}),\
\forall\mathbf{s}\in\mathcal{S},\mathbf{a}\in\mathcal{A}\,,$ (17)

where $F:\mathcal{S}\rightarrow\mathbb{R}$ is a state-dependent (bounded
potential) function. We arrive at Eqn. (17) by (1) testing
$\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s},\mathbf{a})$ as a solution
and noting that the “softmax“ function is convex in the exponent (Boyd et al.,
2004) and (2) using the translation invariance property of the assumed
Boltzmann policy parametrisation, i.e., for any
$f:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ and
$g:\mathcal{S}\rightarrow\mathbb{R}$

$\displaystyle\frac{\exp(f(\mathbf{s},\mathbf{a})+g(\mathbf{s}))}{\sum_{a}\exp(f(\mathbf{s},a)+g(\mathbf{s}))}=\frac{\exp(g(\mathbf{s}))\exp(f(\mathbf{s},\mathbf{a}))}{\exp(g(\mathbf{s}))\sum_{a}\exp(f(\mathbf{s},a))}=\frac{\exp(f(\mathbf{s},\mathbf{a}))}{\sum_{a}\exp(f(\mathbf{s},a))}\,.$
(18)

The minimiser of the inverse temporal difference learning loss
$\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{Q},\boldsymbol{\theta}_{r})$,
Eqn. (9), for a single expert is s.t.

$\displaystyle\boldsymbol{\theta}_{Q}^{*},\boldsymbol{\theta}_{r}^{*}\in\operatorname*{arg\,min}_{\boldsymbol{\theta}_{Q},\boldsymbol{\theta}_{r}}\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},\mathbf{a}^{\prime})\sim\mathcal{D}}\|Q(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{Q})-r(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{r})-\gamma
Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{Q})\|$ (19)

where $\boldsymbol{\theta}_{Q}^{*}$ is minimising
$\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{Q})$ simultaneously, as in
Eqn. (17). Therefore, it holds that
$\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{Q}^{*},\boldsymbol{\theta}_{r}^{*})=0$

$\displaystyle r(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{r}^{*})$
$\displaystyle=Q(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{Q}^{*})-\gamma
Q(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{Q}^{*})$ (20)
$\displaystyle\\!\\!\overset{(\ref{eq:btheta-q-
star})}{=}\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s},\mathbf{a})+F(\mathbf{s})-\gamma\frac{1}{\nu}Q^{\pi_{\text{expert}}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})-\gamma
F(\mathbf{s}^{\prime})$ (21) $\displaystyle=\,\,\frac{1}{\nu}\left[\hskip
1.19995pt\underbracket{Q^{\pi_{\text{expert}}}(\mathbf{s},\mathbf{a})-\gamma
Q^{\pi_{\text{expert}}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})}_{r^{\text{expert}}(\mathbf{s},\mathbf{a})}\right]+F(\mathbf{s})-\gamma
F(\mathbf{s}^{\prime})$ (22)
$\displaystyle=\frac{1}{\nu}r^{\text{expert}}(\mathbf{s},\mathbf{a})+\hskip
1.19995pt\underbracket{F(\mathbf{s})-\gamma
F(\mathbf{s}^{\prime})}_{\begin{subarray}{c}\text{potential-based reward}\\\
\text{shaping function}\end{subarray}}\,,$ (23)

where $r^{\text{expert}}$ is the (unobserved) expert’s reward function. We
have shown that the minimiser of $\mathcal{L}_{\text{BC-}Q}$ and
$\mathcal{L}_{\text{ITD}}$ leads to a reward function
$r(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{r}^{*})$ which is a potential-
based shaped and scaled reward function of the expert reward function and
hence the optimal policy for
$r(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{r}^{*})$ is also optimal for
$r^{\text{expert}}(\mathbf{s},\mathbf{a})$ for all $\mathbf{s},\mathbf{a}$ (Ng
et al., 1999).

Multiple tasks. The minimiser of the behavioural cloning loss
$\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})$,
i.e., Eqn. (8), for the $k$-th expert is s.t.

$\displaystyle\boldsymbol{\theta}_{\Psi^{k}}^{*},\mathbf{w}^{k*}$
$\displaystyle\in\operatorname*{arg\,min}_{\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k}}\
-\\!\\!\\!\\!\\!\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}^{k}}\log\frac{\exp(\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{k}})^{\top}\mathbf{w}^{k})}{\sum_{a}\exp(\Psi(\mathbf{s},a;\boldsymbol{\theta}_{\Psi^{k}})^{\top}\mathbf{w}^{k})}$
(24)
$\displaystyle\Rightarrow\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{k}})^{\top}\mathbf{w}^{k}=\frac{1}{\nu}Q^{\pi_{k\text{-expert}}}(\mathbf{s},\mathbf{a})+F^{k}(\mathbf{s}),\
\forall\mathbf{s}\in\mathcal{S},\mathbf{a}\in\mathcal{A}\,,$ (25)

where $Q^{\pi_{k\text{-expert}}}$ is the $k$-agent’s action-value function and
$H^{k}:\mathcal{S}\rightarrow\mathbb{R}$ a state-dependent (bounded potential)
function. Next, the minimiser of the inverse temporal difference learning loss
$\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Psi^{k}},\boldsymbol{\theta}_{\Phi})$,
Eqn. (9), for the $k$-th expert is s.t.

$\displaystyle\boldsymbol{\theta}_{\Psi^{k}}^{*},\boldsymbol{\theta}_{\Phi}^{*}\in\operatorname*{arg\,min}_{\boldsymbol{\theta}_{\Psi^{k}},\boldsymbol{\theta}_{\Phi}}\operatorname*{\mathbb{E}}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},\mathbf{a}^{\prime})\sim\mathcal{D}^{k}}\|\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{k}})-\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi})-\gamma\Psi(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{\Psi^{k}})\|$
(26)

where $\boldsymbol{\theta}_{\Psi^{k}}^{*}$ is minimising
$\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})$
simultaneously, as in Eqn. (25). Therefore, it holds that for
$\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Psi^{k}}^{*},\boldsymbol{\theta}_{\Phi}^{*})=0$

$\displaystyle\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi}^{*})$
$\displaystyle=\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{k}}^{*})-\gamma\Psi(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{\Psi^{k}}^{*})$
(27)
$\displaystyle\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi}^{*})^{\top}\mathbf{w}^{k*}$
$\displaystyle=\Psi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Psi^{k}}^{*})^{\top}\mathbf{w}^{k*}-\gamma\Psi(\mathbf{s}^{\prime},\mathbf{a}^{\prime};\boldsymbol{\theta}_{\Psi^{k}}^{*})^{\top}\mathbf{w}^{k*}$
(28) $\displaystyle\\!\\!\overset{(\ref{eq:btheta-q-star-
multi})}{=}\frac{1}{\nu}Q^{\pi_{k\text{-expert}}}(\mathbf{s},\mathbf{a})+F^{k}(\mathbf{s})-\gamma\frac{1}{\nu}Q^{\pi_{k\text{-expert}}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})-\gamma
F^{k}(\mathbf{s}^{\prime})$ (29) $\displaystyle=\frac{1}{\nu}\left[\hskip
1.19995pt\underbracket{Q^{\pi_{k\text{-expert}}}(\mathbf{s},\mathbf{a})-\gamma
Q^{\pi_{k\text{-expert}}}(\mathbf{s}^{\prime},\mathbf{a}^{\prime})}_{r^{k\text{-expert}}(\mathbf{s},\mathbf{a})}\right]+F^{k}(\mathbf{s})-\gamma
F^{k}(\mathbf{s}^{\prime})$ (30)
$\displaystyle=\frac{1}{\nu}r^{k\text{-expert}}(\mathbf{s},\mathbf{a})+\hskip
1.19995pt\underbracket{F^{k}(\mathbf{s})-\gamma
F^{k}(\mathbf{s}^{\prime})}_{\begin{subarray}{c}\text{potential-based
reward}\\\ \text{shaping function}\end{subarray}}\,,$ (31)

We have shown that the minimiser of $\mathcal{L}_{\text{BC-}Q}$ and
$\mathcal{L}_{\text{ITD}}$ leads to agent-agnostic cumulants
$\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi}^{*})$ and agent-
specific preference vector $\mathbf{w}^{k*}$, which when dot-producted, form a
potential-based shaped and scaled reward function of the $k$-th expert reward
function and hence the optimal policy for
$\Phi(\mathbf{s},\mathbf{a};\boldsymbol{\theta}_{\Phi}^{*})^{\top}\mathbf{w}^{k*}$
is also optimal for $r^{k\text{-expert}}(\mathbf{s},\mathbf{a})$ for all
$\mathbf{s},\mathbf{a}$ (Ng et al., 1999). The result holds for all
$k\in\\{1\ldots K\\}$ since no assumptions were made for the proof about $k$.
∎

Next, we formalise the statement of Theorem 2. When not specified the norm
$\|\cdot\|$ refers to the 2-norm. Given a function
$F:\mathcal{X}\rightarrow\mathbb{R}^{d}$ for some finite set $\mathcal{X}$, we
will write $F(x)$ to denote the value of the function on input $x$ and $F$ to
denote the matrix representation of this function in
$\mathbb{R}^{|\mathcal{X}|\times d}$.

######  Theorem 2 (Generalisation Bound of $\Psi\Phi$-Learning).

Let $\mathcal{C}=(\mathcal{S},\mathcal{A},P,\gamma)$ be a CMP with a finite
state space. Let $\phi:\mathcal{S}\rightarrow\mathbb{R}^{d}$, and let
$\Phi=\phi(\mathcal{S})\in\mathbb{R}^{|\mathcal{S}|\times d}$. Let
$(r_{i})_{i=1}^{k}$ denote a set of reward functions on $\mathcal{C}$,
$\tilde{\Psi}^{i}$ be a collection of successor features approximations for
policies $(\pi^{i})_{i=1}^{k}$ ($\pi_{i}$ optimal for $r_{i}$) with true
successor feature values $\Psi^{i}$, and $w_{i}$ the best least-squares linear
approximator of $r_{i}$ given $\Phi$, with errors

$\|\Phi w_{i}-r_{i}\|_{\infty}<\delta_{r}\text{ and
}\|\tilde{\Psi}^{i}-\Psi^{i}\|<\delta_{\Psi}\quad\forall i.$

Let $w^{\prime}$ be a new preference vector for a reward function
$r^{\prime}$, with maximal error $\delta_{r}$ as well. Let
$\tilde{Q}^{i}=\tilde{\Psi}^{i}w^{\prime}$. Let $\pi^{*}$ be the optimal
policy for the ego task $w^{\prime}$ and let $\pi$ be the GPI policy obtained
from $\\{\tilde{Q}^{\pi_{i}}\\}$, with $\delta_{r},\delta_{\Psi}$ the reward
and successor feature approximation errors. Then for all $s,a$

$Q^{*}(s,a)-Q^{\pi}(s,a)\leq\frac{2}{1-\gamma}\bigg{[}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\|w^{\prime}\|\delta_{\Psi}+\frac{1}{(1-\gamma)}\delta_{r}\bigg{]}$
(32)

Barreto et al. (2017) construct their bound on the sub-optimality of the GPI
policy as a function of the error of the value approximations
$\widetilde{Q}^{i}$. Because we bound the reward approximation error, rather
than the value approximation error, we require an additional step to obtain a
bound on the errors of the value funciton approximations. To prove Theorem 1,
we must therefore first use the following lemma to bound the effect of the
reward approximation error on the value approximation error. While this result
is straightforward, we include a short proof for completeness.

######  Lemma 1.

Fix some policy $\pi$. Let $r$ be reward vector and let $w$ be the least-
squares solution to $\min\|\Phi w-r\|$. Let $\Psi^{\pi}$ be the true successor
features for $\Phi$ under policy $\pi$, and let $Q^{\pi}$ be the value. Let
$\delta_{r}=R(\mathcal{S})-\Phi w$, $\delta_{max}=\|\delta_{r}\|_{\infty}$.
Then letting $\tilde{Q}=\Psi w$, we have

$\|Q^{\pi}-\tilde{Q}\|_{\infty}\leq\frac{1}{1-\gamma}\delta_{r}$ (33)

###### Proof.

$\displaystyle\|Q^{\pi}-\tilde{Q}\|_{\infty}$
$\displaystyle\leq\sum\gamma^{t}\|P^{\pi t}(\Phi w-r)\|_{\infty}$ (34)
$\displaystyle\leq\sum\gamma^{t}\|P^{\pi
t}\delta_{r}\|_{\infty}=\sum_{t}\gamma^{t}\max_{s^{\prime}}|\sum_{s\in\mathcal{S}}(P^{\pi})^{t}(s^{\prime},s)\delta_{r}(s)|$
(35) Since $P^{\pi}$ is a stochastic matrix, so are all of its powers, and so
the rows of $(P^{\pi})^{t}$ sum to 1.
$\displaystyle\leq\sum_{t}\gamma^{t}\max_{s^{\prime}}|\sum{P^{\pi}}^{t}(s,s^{\prime})\delta_{\max}|=\sum\gamma^{t}\delta_{\max}$
(36) $\displaystyle=\frac{1}{1-\gamma}\delta_{\max}$ (37)

∎

We now prove the main result.

###### Proof.

We follow the proof of Barreto et al. (2017, Theorem 2), with additional error
terms to account for the reward and successor feature approximation errors.

$\displaystyle Q^{*}(s,a)-Q^{\pi}(s,a)$ $\displaystyle\leq
Q^{*}(s,a)-Q^{\pi_{j}}(s,a)+\frac{2}{1-\gamma}\epsilon$ (Barreto et al., 2017,
Theorem 1)
$\displaystyle\leq\frac{2}{1-\gamma}\|r_{j}-r^{\prime}\|_{\infty}+\frac{2}{1-\gamma}\epsilon$
(Barreto et al., 2017, Lemma 1) $\displaystyle\leq\frac{2}{1-\gamma}\|\phi
w_{j}+\delta_{j}-\phi
w^{\prime}-\delta^{\prime}\|_{\infty}+\frac{2}{1-\gamma}\epsilon$
$\displaystyle\leq\frac{2}{1-\gamma}(\phi_{\max}\|w_{j}-w^{\prime}\|+\delta_{r}+\delta_{r})+\frac{2}{1-\gamma}\epsilon$
$\displaystyle\leq\frac{2}{1-\gamma}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\frac{2}{1-\gamma}\|\tilde{\Psi}^{j}w^{\prime}-\Psi_{j}w^{\prime}+\Psi_{j}w^{\prime}-Q_{j}\|$
$\displaystyle\leq\frac{2}{1-\gamma}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\frac{2}{1-\gamma}\|\tilde{\Psi}^{j}w^{\prime}-\Psi_{j}w^{\prime}\|+\|\Psi_{j}w^{\prime}-Q_{j}\|$
$\displaystyle\leq\frac{2}{1-\gamma}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\frac{2}{1-\gamma}\|w^{\prime}\|\delta_{\Psi}+\frac{2}{1-\gamma}\|\Psi_{j}w^{\prime}-Q_{j}\|$
$\displaystyle\leq\frac{2}{1-\gamma}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\frac{2}{1-\gamma}\|w^{\prime}\|\delta_{\Psi}+\frac{2}{1-\gamma}(\frac{1}{1-\gamma}\delta_{r})$
(Lemma 1)
$\displaystyle=\frac{2}{1-\gamma}\bigg{[}(\phi_{\max}\|w_{j}-w^{\prime}\|+2\delta_{r})+\|w^{\prime}\|\delta_{\Psi}+\frac{1}{(1-\gamma)}\delta_{r}\bigg{]}$

∎

##  Appendix D Algorithms

Input :

$\mathcal{D}=\\{(\mathbf{s}_{1},\mathbf{a}_{1},\ldots,\mathbf{a}_{T};k)_{k=1}^{K}\\}$ | No-reward demonstrations
---|---
$\lambda_{\mathbf{w}}$ |  $\mathcal{L}_{1}$ loss coefficient

Output :

$\boldsymbol{\theta}_{\Phi}$ | Parameters of cumulants network
---|---
$\\{\boldsymbol{\theta}_{\Psi^{k}}\\}_{k=1}^{K}$ | Parameters of successor features approximators
$\\{\mathbf{w}^{k}\\}_{k=1}^{K}$ | Preferences vectors for the $K$ agents

;

1 // initialisations  Initialise parameters
$\boldsymbol{\theta}_{\Phi},\\{\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k}\\}_{k=1}^{K}$
while  _budget_ do

Sample trajectories
$\\{\tau_{i}=(s_{1}^{(i)},a_{1}^{(i)},\ldots,s_{T}^{(i)},a_{T}^{(i)};k^{(i)})\\}_{i=1}^{N}\sim\mathcal{D}$
Calculate behavioural cloning loss
$\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})$ on
samples $\\{\tau_{i}\\}_{i=1}^{N}$ ;

$\triangleright$ see Eqn. (8)
$\boldsymbol{\theta}_{\Psi^{k}}\overset{\alpha}{\leftarrow}\nabla_{\boldsymbol{\theta}_{\Psi^{k}}}\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})$
;

$\triangleright$ update $\Psi$s
$\mathbf{w}^{k}\overset{\alpha}{\leftarrow}\nabla_{\mathbf{w}^{k}}\left(\mathcal{L}_{\text{BC-}Q}(\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k})+\lambda_{\mathbf{w}}\|\mathbf{w}^{k}\|_{1}\right)$
;

$\triangleright$ update $\mathbf{w}$s Calculate inverse temporal difference
loss
$\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Phi},\boldsymbol{\theta}_{\Psi^{k}})$
on samples $\\{\tau_{i}\\}_{i=1}^{N}$ ;

$\triangleright$ see Eqn. (9)
$\boldsymbol{\theta}_{\Phi}\overset{\alpha}{\leftarrow}\nabla_{\boldsymbol{\theta}_{\Phi}}\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Psi_{k}},\boldsymbol{\theta}_{\Phi})$
;

$\triangleright$ update $\Phi$
$\boldsymbol{\theta}_{\Psi_{k}}\overset{\alpha}{\leftarrow}\nabla_{\boldsymbol{\theta}_{\Psi_{k}}}\mathcal{L}_{\text{ITD}}(\boldsymbol{\theta}_{\Psi_{k}},\boldsymbol{\theta}_{\Phi})$
;

2 $\triangleright$ update $\Psi$s

Algorithm 1 Inverse Temporal Difference Learning

Input :

$\mathcal{D}=\\{(\mathbf{s}_{1},\mathbf{a}_{1},\ldots,\mathbf{a}_{T};k)_{k=1}^{K}\\}$ | No-reward demonstrations
---|---
$\lambda_{\mathbf{w}}$ |  $\mathcal{L}_{1}$ loss coefficient

Output :

$\boldsymbol{\theta}_{\Psi^{\text{ego}}}$ | Ego successor features approximator
---|---
$\boldsymbol{\theta}_{\Phi}$ | Parameters of cumulants network
$\\{\boldsymbol{\theta}_{\Psi^{k}}\\}_{k=1}^{K}$ | Parameters of successor features approximators
$\\{\mathbf{w}^{k}\\}_{k=1}^{K}$ | Preferences vectors for the $K$ agents

;

3 // initialisations  Empty replay buffer for ego-experience
$\mathcal{B}=\\{\\}$ Initialise parameters
$\boldsymbol{\theta}_{\Psi^{\text{ego}}},\mathbf{w}^{\text{ego}},\boldsymbol{\theta}_{\Phi},\\{\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k}\\}_{k=1}^{K}$
while  _budget_ do

;

4 // agent-environment interaction  Reset episode,
$\mathbf{s}\leftarrow\texttt{env.reset()}$, $t\leftarrow 0$ while  _not done_
do

$\mathbf{w}^{\text{ego}}\leftarrow\operatorname*{arg\,min}_{w}\mathcal{L}_{\text{R}}(\boldsymbol{\theta}_{\Phi},w;\mathcal{B})$
;

$\triangleright$ ego-task inference, see Eqn. (10)
$\mathbf{a}\leftarrow\pi^{\text{ego}}_{\text{GPI}}\left(\mathbf{s};\Psi^{\text{ego}},\mathbf{w}^{\text{ego}},\\{\boldsymbol{\theta}_{\Psi^{k}}\\}_{k=1}^{K}\right)$
;

$\triangleright$ GPI, see Eqn. (13) Step in the environment,
$\mathbf{s}^{\prime},r^{\text{ego}},\texttt{done}\leftarrow\texttt{env.step(}\mathbf{a}\texttt{)}$
Append transition in the replay buffer,
$\mathcal{B}\leftarrow\mathcal{B}\cup\left(\mathbf{s},\mathbf{a},r^{\text{ego}},\mathbf{s}^{\prime}\right)$
$\mathbf{s}^{\prime}\leftarrow\mathbf{s}$, $t\leftarrow t+1$ (online
demonstrations) Append demonstrations in $\mathcal{D}$ ;

5 $\triangleright$ optional

;

// parameter updates/learning
$\boldsymbol{\theta}_{\Phi},\\{\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k}\\}_{k=1}^{K}\leftarrow\texttt{ITD}\left(\mathcal{D},\lambda_{\mathbf{w}},\boldsymbol{\theta}_{\Phi},\\{\boldsymbol{\theta}_{\Psi^{k}},\mathbf{w}^{k}\\}_{k=1}^{K}\right)$
;

$\triangleright$ see Algorithm. (1) Sample transitions
$\\{(\mathbf{s}^{(i)},\mathbf{a}^{(i)},r^{\text{ego},(i)},\mathbf{s}^{\prime(i)})\\}\sim\mathcal{B}$
Calculate the reward loss
$\mathcal{L}_{\text{R}}(\boldsymbol{\theta}_{\Phi},\mathbf{w}^{\text{ego}})$ ;

$\triangleright$ see Eqn. (10)
$\boldsymbol{\theta}_{\Phi}\overset{\alpha}{\leftarrow}\nabla_{\boldsymbol{\theta}_{\Phi}}\mathcal{L}_{\text{R}}(\boldsymbol{\theta}_{\Phi},\mathbf{w}^{\text{ego}})$
;

$\triangleright$ update $\Phi$ Calculate TD losses
$\mathcal{L}_{Q}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})$ and
$\mathcal{L}_{\text{TD-}\Psi}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})$ ;

$\triangleright$ see Eqn. (11,12)
$\boldsymbol{\theta}_{\Psi^{\text{ego}}}\overset{\alpha}{\leftarrow}\nabla_{\boldsymbol{\theta}_{\Psi^{\text{ego}}}}\left(\mathcal{L}_{Q}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})+\frac{1}{|\Psi|}\mathcal{L}_{\text{TD-}\Psi}(\boldsymbol{\theta}_{\Psi^{\text{ego}}})\right)$
;

6 $\triangleright$ update $\Psi^{\text{ego}}$

Algorithm 2 $\Psi\Phi$-Learning

##  Appendix E Visualisations

CoinGrid

$\phi_{1}$

$\phi_{2}$

$\phi_{3}$

$\phi_{4}$

Figure 12:  Qualitative evaluation of the learned cumulants in the CoinGrid
task. Cumulants $\phi_{1}$, $\phi_{2}$, and $\phi_{3}$ seem to capture the
red, green, and yellow blocks, respectively. The yellow blocks are captured by
both and $\phi_{4}$. Therefore, linear combinations of the learned cumulants
can represent arbitrary rewards in the environment, which involve stepping on
the coloured blocks.

ITD for Roundabout

ITD for CoinGrid

$\Psi\Phi$-learning for Highway Multi-Task

Figure 13:  Sensitivity of our ITD (see Section 3.1) and $\Psi\Phi$-learning
(see Section 3.2) algorithms to the dimensionality of the learned cumulants.
We consistently observe across all three experiments (a)-(c) that for a small
number of $\Phi$ dimensions the cumulants are not expressive enough to capture
the axis of variation of the different agents’ reward functions (including the
ego-agent in (c)). We also note that the performance of both ITD and
$\Psi\Phi$-learning is relative robust for a medium and large number of $\Phi$
dimensions. We attribute this to the used sparsity prior, i.e.,
$\mathcal{L}_{1}$ loss, to the preferences $\mathbf{w}$. In our experiments we
selected the smallest number of $\Phi$ dimensions that demonstrated good
performance to keep the number of model parameters as small as possible (in
bold in the figures and reported in Table 3).