# SAFARI: Safe and Active Robot Imitation Learning with Imagination

Norman Di Palo and Edward Johns The Robot Learning Lab at Imperial College
London, London, UK<EMAIL_ADDRESS>
###### Abstract

One of the main issues in Imitation Learning is the erroneous behavior of an
agent when facing out-of-distribution situations, not covered by the set of
demonstrations given by the expert. In this work, we tackle this problem by
introducing a novel active learning and control algorithm, SAFARI. During
training, it allows an agent to request further human demonstrations when
these out-of-distribution situations are met. At deployment, it combines
model-free acting using behavioural cloning with model-based planning to
reduce state-distribution shift, using future state reconstruction as a test
for state familiarity. We empirically demonstrate how this method increases
the performance on a set of manipulation tasks with respect to passive
Imitation Learning, by gathering more informative demonstrations and by
minimizing state-distribution shift at test time. We also show how this method
enables the agent to autonomously predict failure rapidly and safely.

##  I Introduction

To teach a robot a novel task, one promising avenue of research is Imitation
Learning (IL) [37, 38, 45, 2, 3], and in particular, Behaviour Cloning (BC).
In BC, an expert collects a series of demonstrations of how to solve a task,
and an agent then uses supervised learning to generalise a policy to new
instances of the task. Showing how to solve a task can often be easier and
faster than directly designing a controller manually, or hand-crafting a
potentially complex reward function and learning a policy through
reinforcement learning. However, once deployed, the agent state visitation
distribution may be different from its training distribution, causing
erroneous and dangerous behaviour [36]. This may have two main causes: a poor
state space visitation during the expert’s demonstrations, and compounding
errors from the policy at test time that cause the state visitation
distribution to drift outside of the training distribution.

Therefore, we argue that a robot should both maximize the information obtained
from demonstrations, and at test time should act in such ways as to solve the
task while staying close to the states explored during training, by actively
minimizing state-distribution shift through a feedback action. Furthermore,
when out-of-distribution (OOD) states are encountered, this should be detected
quickly and execution should be autonomously stopped.

Based on these principles, in this work we propose a novel active imitation
learning and control framework that tackles the aforementioned problems:
SAFARI (SAFe and Active Robot Imitation Learning with Imagination). SAFARI
(Figure 1) is a framework for imitation learning and control which allows the
robot to (1) actively ask for more informative demonstrations at training
time, (2) actively minimize the distributional shift at test time, and (3)
predict test time failures to improve safety. These methods all contribute to
tackle the aforementioned issues from different perspectives, resulting in a
substantial improvement in performance and safety with respect to baseline
methods.

Specifically, SAFARI uses an interplay between a policy network, a learned
dynamics network (also referred as world model in this work), and an epistemic
uncertainty network. The policy network is trained to emulate the expert’s
behavior with BC [27, 32, 2], while the dynamics network [21] is trained to
predict future states, enabling planning alongside model-free policy acting,
and the uncertainty network [12, 13] is trained to estimate epistemic
uncertainty and out-of-distribution states, similarly to an energy model [49].

Figure 1: An overview of our training and control framework, SAFARI. We start
with an active imitation learning phase, which trains the policy, uncertainty
and dynamics networks. These networks are then used in deployment to minimize
state-distribution shift and predict dangerous states, improving performance
and safety.

We tested our methods on both simulated and real robotics environments. For
the simulated experiments, we used three robot manipulation tasks: bringing a
cube to a desired position, stacking two cubes at a desired position, both
with the Fetch robot, and a nut-and-peg insertion task with a Sawyer robot,
all using the MuJoCo physics simulator [8]. As real-world experiments, we used
a Sawyer robot, with a task to move an object on a table to a goal position,
using the end-effector position and an RGB stream from a camera as inputs.

To summarise, the main contributions of this work are the following:

  * •

We demonstrate how our Active Learning method is able to select the most
uncertain, and hence informative, instances of the task, in order to minimize
the number of required demonstrations.

  * •

We propose a new hybrid policy, which combines the model-free output of a
policy network, trained to emulate the expert’s behavior, and an online plan
that aims at bringing the state closer to the distribution of states visited
by the demonstrations. This allows to minimize state-distribution drifting
caused by compounding errors at test time.

  * •

We demonstrate the ability of our uncertainty estimation algorithm to rapidly
and safely predict failures at test time, avoiding possibly dangerous
situations that could harm the robot and its environment.

We bring these approaches together into a unified framework called SAFARI,
that uses a common approach based on an interplay of a policy, a learned world
model, and an epistemic uncertainty network to tackle real-world Imitation
Learning from three different perspectives.

##  II Related Work

Imitation Learning with deep neural networks has shown impressive results in
recent years in robotics. ([30, 2, 9, 34, 5]).

In order to obtain more informative demonstrations that properly explore the
state space, one effective approach is using Active Learning algorithms [30].
Active Learning [40] deals with the design of methods able to actively ask an
expert for labels on unseen data, improving data-efficiency with respect to
passive learners. [15] proposed DAgger, a popular active imitation learning
algorithm, to tackle state visitation shift. Several variants have been
proposed over the years [16, 42, 40], but these methods require the expert to
label states with single, optimal actions, a method that cannot be applied
efficiently to real-world systems. [17] extends these methods by allowing a
human expert to take control at training time, without querying them on single
states. Our method extends these approaches by more safely and efficiently
locating OOD state areas using future predictions, querying the expert at
train time or stopping execution at test time without human supervision. We
also compare our method with [35], which injects noise during the expert’s
demonstrations to simulate the errors of a policy in order to learn how to
compensate for it, artificially expanding the visited states during
demonstrations.

To minimize state distribution shift at test time, Adversarial Imitation
Learning methods such as [47, 48] have been shown to be superior to
Behavioural Cloning [45]. These methods, similarly to our approach, learn a
function that can distinguish between expert’s and non expert’s state
visitation distributions. However, a substantial difference is that these
methods use model-free reinforcement learning to learn a policy that can
minimize distribution shifts. This is both dangerous and impractical in real
robotics scenarios, needing autonomous exploration, environment resets, and
many time-steps [45]. In our work, we designed a method that tackles the same
problem, but designed to be efficiently trained and safely deployed on real
robots, using only supervised data from the expert’s demonstration: instead of
a discriminator that learns from expert’s and agent’s data, we use a Denoising
Autoencoder that can learn to discriminate by only receiving expert data.
Then, instead of learning a policy with model-free reinforcement learning, we
learn a dynamics model that allows the agent to plan online for a series of
actions that can minimize state distribution shift. A more detailed
mathematical comparison between these approaches can be found in section
III.F.

Computing epistemic uncertainty is a crucial aspect of our work. We use a
Denoising Autoencoder to learn to approximate the data distribution. [12, 13,
14, 19] demonstrated how (Denoising) Autoencoders [7] can be used for
detecting out-of-distribution inputs and regularizing their effects on neural
networks, with [13] comparing them to ensemble methods. [13] also showcases
the utility of using Autoencoders to regularize plans under a learned dynamics
model, by minimizing the distribution shift with the experience collected
during training, which in our case is the expert’s demonstrations.

[39] is the closest work to our method, but applied to a very different
scenario: simulated autonomous vehicles. While proposing a similar approach,
they don’t use a policy network and don’t learn a dynamics function, but
instead compute a series of desired future states. They assume the existence
of an inverse dynamical model which can then deduce actions, which is
unattainable in robotic manipulation with objects and visual inputs, and hence
we designed a dynamics model to learn that from data.

##  III Method

In Imitation Learning [2, 9, 1], a human expert gathers a set of
demonstrations of how to successfully solve a task in the form of trajectories
of states and actions $(s_{0},a_{0},...,s_{T},a_{T})$. A policy network
$f_{\theta}$ is then trained on this data, with the hope that it will
generalize to novel instances of the task. As described in the Introduction,
once deployed the learned policy can encounter states that are different from
the dataset gathered by the expert. In these situations, the output of the
parametrised policy can be erroneous and even dangerous for the robot and its
surroundings [36, 45]. To tackle these problems, we aim at collecting the most
informative demonstrations at training time through an Active Learning phase.
In addition, at test time, once the policy has been trained, we aim at
minimizing state visitation shift through a feedback action that leads the
agent closer to the distribution explored by the expert, where the policy’s
behaviour is more likely to be correct. [36]

We tackle these problems using a common approach, based on an interplay
between a policy network, a dynamics network, and an uncertainty network. In
the following section, we describe more in depth the aforementioned networks
(III.A), then describe our approach for computing epistemic uncertainty, which
is then used in three different ways, in each of the three components of our
framework (III.B). Then, we introduce the Active Learning method we
implemented and use at training time (III.C) and the state distribution shift
minimization method we use at test time (III.D). We then describe our method
for failure prediction at test time (III.E). Finally, we analyse more in depth
the mathematical aspects of our methods, describing the similarities and
differences with current state-of-the-art approaches [45] (III.F).

Figure 2: This diagram depicts the computation graph to estimate the agent’s
uncertainty described in Section III.C. At each time step, the agent predicts
the future visited states by running its policy network using its internal
world model. It then computes the epistemic uncertainty of these imagined
state using its uncertainty model, averaging the results over all $T$ steps.

###  III-A Networks Models and Architectures

Inspired by previous works on Imitation Learning [1, 27, 32, 2], we use a
feed-forward neural network $f_{\theta}$ to parametrise our policy, taking as
input the current observation, and computing as output the action. The policy
network is trained with Behavioural Cloning.

To model the uncertainty of the agent, we use Denoising Autoencoders,
$g_{\phi}$. The Denoising Autoencoder [7] is a feed-forward network that takes
as input the current state, that during training time is corrupted with
Gaussian noise, $\tilde{x}$, and is trained to give as output the denoised
input, $e=\sum_{i}(x_{i}-g_{\theta}(\tilde{x})_{i})^{2}$, where $e$ is the
denoising error that we wish to minimize with gradient descent. The denoising
error is akin to the output of an energy model [49], hence representing a
proxy of the distance of the input from the training distribution: if those
were part of the previous training samples, the errors will be lower, while
being higher on novel states or goals [12, 13]. In this work we will also
refer to the Denoising Autoencoder as the uncertainty network.

As a learned dynamics model, we use a feed-forward network $d_{\gamma}$ that
takes as input the current state and action, and predicts the one-step
difference $\Delta{}\hat{x}_{t+1}$ such that
$\hat{x}_{t+1}=x_{t}+\Delta{}\hat{x}_{t+1}$ [21].

All the networks are trained together on the same data (although used in
different ways).

###  III-B Uncertainty Estimation with Autoencoders and Imagination

The backbone of these approaches is the accurate computation of the epistemic
uncertainty of the policy. In this work, we propose a novel approach at
computing the familiarity of the agent with its current and also future
states. When we refer to ”uncertainty” in this work we will also refer to
epistemic uncertainty. Dealing with aleatoric uncertainty is outside the scope
of this work.

To compute the uncertainty on a particular state, we use an interplay of a
policy network, a dynamics network, and an uncertainty network, as we show in
Figure 2.

From a state $s_{t}$, we use the policy network and the dynamics network to
predict the future states that the agent will encounter following its current
policy, $s_{t+1},...,s_{t+T}$. We use the Autoencoder $g_{\phi}$ to compute
the aforementioned error on each of those states, and then average the result.
For brevity, we will refer to the uncertainty of a state as
$u_{t}=u(s_{t},f_{\theta},d_{\gamma},g_{\phi})$, where $s_{t}$ is the current
state and $f_{\theta},d_{\gamma},g_{\phi}$ are the policy, dynamics and
uncertainty networks, that are used as showed in Figure 2. This value is used
as a proxy for the epistemic uncertainty of the policy on the current state.
In section IV, we empirically show how this technique can be used to gather
the most informative demonstrations at training time (IV.A), and to minimize
state distribution shift (IV.B) and eventually predicting a failure (IV.C) at
test time.

Figure 3: In this figure we visually describe the hybrid behavior of the robot
that we propose. After computing the output of the policy network, the agent
plans for a series of actions that will minimize state distribution shift as
described in Section III.D, and then combines the two contributions.

###  III-C Active Imitation Learning with Uncertainty Estimation

In many Imitation Learning works [2, 9, 1], the demonstrations collection
phase is separated from the robot interaction with its environment. We propose
instead to have an iterative, interactive approach to gather demonstrations.
Instead of receiving all the demonstrations beforehand, the agent receives
only a small set of demonstrations as an initialization phase. The agent can
then request a demonstration by interacting with its environment, trying to
solve the task following its policy network and stopping when its uncertainty
surpasses a certain threshold. Each new demonstration is then added to the
training data and the networks are retrained. The uncertainty estimation
method is described in section III.B. Our experiments, detailed in Section IV,
demonstrate how we can train a better policy network using active
demonstrations than receiving the same number of demonstrations passively
beforehand.

###  III-D Online Planning for State Shift Minimization

Here we demonstrate how an interplay of the networks we introduced in this
work can be used to tackle state distribution shift at test time, one of the
most important causes of failures of Imitation Learning [15, 36, 35, 45].
While our method allows to quickly anticipate these scenarios and hence
predict failure (as we empirically show in Section IV.C), we want to prevent
this from happening altogether when possible and maximize the number of
successfully completed tasks. At test time we combine model-free actions given
by the policy to solve the task with actions planned by predicting the future
to minimize the robot’s epistemic uncertainty (Figure 3). More specifically,
at test time the agent plans for a series of actions $(a_{0},a_{1},...,a_{T})$
that minimize the uncertainty of the predicted future state visited by
applying these actions, $\hat{s}_{t+T+1}$. The uncertainty of this state is
computed as described in Section III.B. Hence, at each time step the robot
computes its action as

$a_{t}=f_{\phi}(s_{t})+\beta\cdot a_{p,0}\text{ where
}a_{p,0:T}=argmin_{a}\text{ }u(\hat{s}_{t+T+1})\\\ \text{ subject to
}\hat{s}_{t+1}=d_{\gamma}(s_{t},a_{t})$ (1)

where $a_{p}$ is a trajectory of actions that minimizes the uncertainty of the
final predicted state, $T$ is the length of the plan, and $a_{p,0}$ is the
first action of such trajectory, following the Model Predictive Control
framework. Intuitively, we are combining a policy function with an online
planning phase that outputs the action that is predicted to bring the robot to
a more familiar state, defined as being closer to the state’s visitation
distribution encountered in the expert’s demonstrations, hence increasing the
probability of a successful task completion. The hyperparameter $\beta$ is
used to tune the influence of the planned actions. High values of $\beta$
result in more conservative behaviour. We found $\beta=0.2$ to give the best
performance in our tests. Our experiments, described in Section IV,
demonstrate how this approach improves performance on robot manipulation tasks
with respect to only using the policy.

###  III-E Failure Prediction at Test Time

At test time the agent computes, at each time step, its epistemic uncertainty
on the current and future predicted states (Figure 2) as described in III.B.
This allows the robot to predict if it is headed towards an unfamiliar area of
the state space, where the robot is likely to fail the task, and even damage
itself or its environment. While we actively try to minimize this value at
test time (III.D), if the estimated uncertainty becomes larger than a
threshold, the agent stops execution. In our experiments section, we show how
this combination of an uncertainty network and future prediction through
policy and dynamics networks allows not only to accurately predict failures,
but also to do so rapidly.

###  III-F A Mathematical Analysis of Imitation Learning as Divergence
Minimization

As described in [45], Imitation Learning can be framed as a divergence
minimization problem between the expert’s state-action visitation
distribution, $p_{exp}(s,a)$, and the agent’s. In Behavioral Cloning, as in
our work, a policy $\pi(a|s)$ is trained with supervised learning to minimize
$\text{KL}_{p_{exp}(s)}(\pi_{exp}(a|s)||\pi(a|s))$, that becomes a regression
problem to minimize $-\operatorname{\mathbb{E}}_{p_{exp}(s)}[\log(\pi(a|s))]$.
While being a sample-efficient model, by only imitating the expert’s action, a
policy may fail when facing OOD states [45]. Other methods, like GAIL [47],
use model-free reinforcement learning to emulate not only the action, but the
full state-action visitation distribution of the expert, by minimizing
$\text{D}_{JS}(p_{\pi}(s,a)||p_{exp}(s,a))$. In this case, the agent
autonomously explores the environment: a discriminator, $D(s,a)$, is trained
on the expert’s and agent’s trajectories to discriminate between expert and
policy state-action pairs [47]. The agent is then rewarded if the
discriminator classifies the state-actions it visits as belonging to the
expert’s distribution. While obtaining strong performance [45], this and
similar methods [48] are based on model-free RL and require possibly unsafe
autonomous exploration from the agent. In our work, we train an uncertainty
model on the expert’s state distribution, which can learn in an unsupervised
way to discriminate between the expert’s and agent’s state-visitation
distribution (IV.B). We then use online planning with this learned model to
minimize the divergence between the agent’s current state and expert’s state-
visitation distribution. Hence, we obtain the same effect as [47, 48, 45] but
leveraging only supervised data, hence being substantially safer, in the form
of the expert’s actions and trajectories of states used to learn a dynamics
model.

Figure 4: The simulated and real environments used in this work.

##  IV Experiments

In this section, we describe the experiments we designed to benchmark the
performance of SAFARI on several manipulation tasks, both simulated and on a
real robot (Figure 4). These experiments have been designed to answer the
following questions 111Detailed hyperparameters, code and videos can be found
at the following link: https://www.robot-learning.uk/safari:

  * •

Does our proposed active imitation learning method perform better at test time
than the baselines given the same number of demonstrations? (IV.A)

  * •

Does online planning to reduce state-distribution shift improve performance at
test time with respect to only using the policy network? (IV.B)

  * •

Can our method reliably predict failure during execution before it happens?
Does imagination reduce the number of steps needed to predict failure? (IV.C)

###  IV-A Active Learning Performance at Test Time

We designed the following experiments to measure the ability of our method to
gather informative demonstrations, efficiently exploring the state space at
training time. We compare our proposed Active Learning method against several
baselines: Passive Learning (PL), randomly stopping during execution to ask
for a demonstration (Rand On Policy), and DART [35], an effective algorithm
for robot imitation learning that tackles distribution shift.

$N$ demonstrations are used to train a policy network, consisting of our
Passive Learning baseline. For Active Learning, we select the initial
$(1-\gamma)N$ (where $0<\gamma<1$) demonstrations, used as the initialization
demonstrations, and then collect an additional $\gamma N$ demonstrations using
our iterative method. We enforce that both $(1-\gamma)N$ and $\gamma N$ are
integers.

We collected the Active Learning demonstrations following the method described
in III.C. The uncertainty threshold is computed as
$u_{thr}\cdot\text{err}_{train}$, where $u_{thr}>1$ is a hyperparameter, and
$\text{err}_{train}$ is the average uncertainty on the training set computed
after training the uncertainty network $g_{\phi}$. We found the method to be
robust to changes to this hyperparameter. Each demonstration trajectory was
then added to the training data and the networks were retrained. We iterated
until we collected $\gamma N$ additional demonstrations, for a total of $N$
demonstrations. We then tested the performance of the two policy networks,
trained using the passive and active demonstrations, on the $M$ test
configurations of the task.

For the Random On-Policy benchmark, we collected $N$ demonstrations by letting
the robot follow its policy, then stopping execution at a random step and
asking the expert to complete. This benchmark is designed to measure if our
method is actually able to recognize useful demonstrations to obtain, or if
the only advantage comes from on-policy stopping.

For DART, we collected $N$ demonstrations following [35], by injecting noise
during the expert’s demonstrations. We followed the original authors’ code to
implement this baseline.

An experiment is defined as a complete round of data collection, training, and
testing, all with one random seed, which is different for each experiment.
Each experiment involved testing over a different test set, composed of a
number of different initial conditions, and recording the number of successes.
Then, each entry in Table I shows on what percentage of experiments each of
the two compared methods outperformed the other in terms of the number of
successes in that experiment. The total, normalized number of successfully
completed test instances can be found in Figure 5, B. The reason why we show
these results both as in Table I and in the raw success rates (Figure 5, B),
is that we believe this is a clearer and more statistically significant way of
directly comparing two methods with respect to robustness over random seeds.

####  IV-A1 Simulated Environments - Cube Manipulation and Nut-and-Peg
Insertion

We start by describing our results on three simulated tasks. The three
manipulation tasks are 1) 1 Cube pick-and-place, 2) 2 Cubes stacking, and 3) a
Nut-and-Peg insertion task (Figure 4). The observation vector, in the
simulated experiments, is composed of the pose of the end-effector, the
objects, and the goal. In the real-world setting, we show how this method also
scales to having vision as input. Result of Table I and Figure 5, B, are
gathered over more than 7 random seeds (details can be found on our website,
including the source code). In Figure 5, A, we also show how increasing the
ratio of AL demonstrations increases performance.

Figure 5: A (Left): How test-time performance varies as we modify the ratio of
Active Learning examples over Passive Learning ones, $\gamma$. The plot
represents the average number of successes over 13 different independent runs.
B (Center): We show F1 score at predicting failures (red) and steps needed to
predict failures (blue) on two tasks (full and dashed lines). We increase the
number of imagined states from left to right. C (Right): Normalized number of
test instances solved by each method, averaged over 7 seeds. TABLE I:

Task name / Method |  AL-PL |  AL-Rand On Policy |  AL-DART
---|---|---|---
Fetch 1-Cube Pick and Place |  88%-12% |  80%-20% |  80%-20%
Fetch 2-Cubes Stack |  83%-17% |  73%-27% |  67%-33%
Sawyer Nut and Peg |  75%-25% |  63%-37% |  71%-29%

  * •

Experimental results of our Active Learning approach (AL) against several
baselines. We show the percentage of experiments, defined as a round of data
collection, training and testing with a different random seed (Section IV.A),
on which one method successfully completes more test instances of the task
than the other.

TABLE II:

Active / Total Demos |  0/50 (PL) |  25/50 |  40/50
---|---|---|---
Sawyer - Object Manipulation w/ Vision |  15/25 |  18/25 |  20/25

  * •

Experimental results of different ratios of Active Learning demonstrations in
a real robotics environment, described in Section IV.A.2 (PL being Passive
Learning). We show the number of successfully completed tasks over the same
test set of 25 tasks.

####  IV-A2 Real Sawyer Environment

To test the ability of our method to scale to real-world scenarios, we
designed a manipulation task using the Sawyer robot. In this experiment, the
goal was to slide a blue object on a table to a goal position, while having
vision as an input to localize both the object and the goal, as well as the
end-effector position and velocity given by internal sensors readings. The
images, provided by a camera, are resized to 32x32. We use a Convolutional
Autoencoder to reduce the dimensionality of the input. We gathered around 10
minutes of random movements and interaction with the environment to quickly
collect an images dataset. We then trained the Autoencoder on this dataset and
use it, with frozen weights, in all our experiments. The encoder transforms
the input images to vectors of size 16. This vector is then concatenated to
the internal sensor readings of the robot, providing the pose of the end-
effector.

In this experiment, we provided a total of 50 demonstrations to the robot.
Demonstration are given through kinesthetic teaching. We sampled the pose of
the end-effector at 10Hz and use the one-step difference in position to
compute the velocity at each time step. We then trained the policy to output
these velocities given, as input, the encoded state as described before. For
the active learning approach, we tested two different ratios of active
demonstrations, $\gamma=0.5$ and $\gamma=0.8$. As shown in Table 2, the active
learning approach surpasses the passive learning approach in this environment
as well. We also notice how a larger ratio of active demonstrations achieves
the best results, as also happened in the experiments in Figure 5, A. This
validates our hypothesis that demonstrations collected using our method are
generally more informative and useful for the policy network to generalize.

###  IV-B Test Performance of Hybrid Model-Free Policy and Online Planning

In this section, we describe the results of our experiments designed to
measure the influence of the planning method proposed in section III.D (Figure
3) on the performance of the robot at test time. We use the same environments
and tasks described in section IV.A.1.

TABLE III:

Task / Method |  SAFARI (ours) |  Behavior Cloning
---|---|---
1-Cube Pick and Place |  8.8 $\pm$ 2.35 |  7.6 $\pm$ 2.35
2-Cubes Stack |  5.9 $\pm$ 4.0 |  3.4 $\pm$ 2.6

  * •

Comparison of our proposed control method (Section III.D) and Behavioural
Cloning (BC). Average number and standard deviation of solved test instances
out of 50 test cases, averaged on 10 different seeds. In both cases, adding
imagination-based state shift minimization improves performance.

We collected a series of expert demonstrations of how to solve the tasks and
sampled a test set. With this data, we trained the models described in this
work, $f_{\theta},g_{\phi},d_{\gamma}$. We measured the performance of model-
free acting [1, 2] on the test set by letting the robot compute actions at
each time step using the policy network, $a_{t}=f_{\theta}(s_{t})$. Then, we
tested the performance of our hybrid acting approach on the same test set by
adding, at each time step, the action computed by the policy to the first
action of the uncertainty minimizing plan described in detail in section
III.D. In order to average the effect of any source of randomness (networks
initialization, objects position, etc.) we repeat our experiments on 10
different random seeds, each having a different test set of 50 task instances.
As shown in Table III, our proposed hybrid approach outperforms the model-free
method.

###  IV-C Unsupervised Failure Prediction Performance

In this section, we experimentally demonstrate the ability of our method to
predict failures, while also minimizing the steps needed to do so.

We trained our networks on a set of expert demonstrations, and we used the
technique described in Section III.E to anticipate possible failures. We
measured the F1 score at predicting failures of our method, along with the
number of steps needed, when changing the number of future states imagined. We
empirically demonstrate how the future prediction allows our method to predict
failure in fewer steps without degrading performance. As shown in Figure 5, C,
the F1 score is roughly unchanged from 0 to 10 steps of future states
prediction, hence imagining future states with the dynamics model doesn’t
hinder failure prediction performance in that range of future steps, while the
steps needed to actually predict failures are much fewer, hence strongly
improving safety.

##  V Conclusion

In this work we introduced SAFARI, a framework that tackles some of the
principal issues of Imitation Learning in robotic scenarios: the need for the
most informative demonstrations at training time, minimization of state
distribution shifting at test time, and accurate and timely prediction of
failures when the robot is deployed. We demonstrated how to tackle these
problems using a common, versatile approach, obtained with a combination of
policy, dynamics, and uncertainty networks, both in simulated and real
robotics environments.

## References

  * [1] Lynch, Corey and Khansari, Mohi and Xiao, Ted and Kumar, Vikash and Tompson, Jonathan and Levine, Sergey and Sermanet, Pierre, Learning Latent Plans from Play, arXiv preprint arXiv:1903.01973, 2019
  * [2] Zhang, Tianhao and McCarthy, Zoe and Jow, Owen and Lee, Dennis and Chen, Xi and Goldberg, Ken and Abbeel, Pieter, Deep imitation learning for complex manipulation tasks from virtual reality teleoperation, 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8, 2018, IEEE
  * [3] Bojarski, Mariusz and Del Testa, Davide and Dworakowski, Daniel and Firner, Bernhard and Flepp, Beat and Goyal, Prasoon and Jackel, Lawrence D and Monfort, Mathew and Muller, Urs and Zhang, Jiakai and others, End to end learning for self-driving cars, arXiv preprint arXiv:1604.07316, 2016
  * [4] Duan, Yan and Andrychowicz, Marcin and Stadie, Bradly and Ho, OpenAI Jonathan and Schneider, Jonas and Sutskever, Ilya and Abbeel, Pieter and Zaremba, Wojciech, One-shot imitation learning, Advances in neural information processing systems, 1087–1098, 2017
  * [5] James, Stephen and Davison, Andrew J and Johns, Edward. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task, CoRL 2017
  * [6] Burda, Yuri and Edwards, Harrison and Storkey, Amos and Klimov, Oleg, Exploration by random network distillation, arXiv preprint arXiv:1810.12894, 2018
  * [7] Vincent, Pascal and Larochelle, Hugo and Lajoie, Isabelle and Bengio, Yoshua and Manzagol, Pierre-Antoine, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of machine learning research, volume 11, pages 3371-3408, 2010 .
  * [8] Todorov, Emanuel and Erez, Tom and Tassa, Yuval, Mujoco: A physics engine for model-based control, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, IEEE, 2012
  * [9] James, Stephen and Bloesch, Michael and Davison, Andrew J, Task-embedded control networks for few-shot imitation learning, arXiv preprint arXiv:1810.03237, 2018
  * [10] Yu, Tianhe and Finn, Chelsea and Xie, Annie and Dasari, Sudeep and Zhang, Tianhao and Abbeel, Pieter and Levine, Sergey, One-shot imitation from observing humans via domain-adaptive meta-learning, arXiv preprint arXiv:1802.01557, 2018
  * [11] Finn, Chelsea and Yu, Tianhe and Zhang, Tianhao and Abbeel, Pieter and Levine, Sergey, One-shot visual imitation learning via meta-learning, arXiv preprint arXiv:1709.04905, 2017
  * [12] Di Palo, Norman and Valpola, Harri, Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization, arXiv preprint arXiv:1812.03955, 2018
  * [13] Boney, Rinu and Di Palo, Norman and Berglund, Mathias and Ilin, Alexander and Kannala, Juho and Rasmus, Antti and Valpola, Harri, Regularizing trajectory optimization with denoising autoencoders, Advances in Neural Information Processing Systems, 2859–2869, 2019
  * [14] Zhu, Lingxue and Laptev, Nikolay, Deep and confident prediction for time series at Uber, 2017 IEEE International Conference on Data Mining Workshops (ICDMW), 103–110, IEEE, 2017
  * [15] Ross, Stephane and Gordon, Geoffrey and Bagnell, Drew, A reduction of imitation learning and structured prediction to no-regret online learning, Proceedings of the fourteenth international conference on artificial intelligence and statistics, 627–635, 2011
  * [16] Menda, Kunal and Driggs-Campbell, Katherine and Kochenderfer, Mykel J, Dropoutdagger: A bayesian approach to safe imitation learning, arXiv preprint arXiv:1709.06166, 2017
  * [17] Kelly, Michael and Sidrane, Chelsea and Driggs-Campbell, Katherine and Kochenderfer, Mykel J, HG-DAgger: Interactive Imitation Learning with Human Experts, 2019 International Conference on Robotics and Automation (ICRA), 8077–8083, IEEE, 2019
  * [18] Fan, Linxi and Zhu, Yuke and Zhu, Jiren and Liu, Zihua and Zeng, Orien and Gupta, Anchit and Creus-Costa, Joan and Savarese, Silvio and Fei-Fei, Li, SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark, Conference on Robot Learning, 2018
  * [19] Arponen, Heikki and Herranen, Matti and Valpola, Harri, On the exact relationship between the denoising function and the data distribution, arXiv preprint arXiv:1709.02797, 2017
  * [20] Sermanet, Pierre and Lynch, Corey and Chebotar, Yevgen and Hsu, Jasmine and Jang, Eric and Schaal, Stefan and Levine, Sergey and Brain, Google, Time-contrastive networks: Self-supervised learning from video, 2018 IEEE International Conference on Robotics and Automation (ICRA), 1134–1141, IEEE, 2018
  * [21] Nagabandi, Anusha and Kahn, Gregory and Fearing, Ronald S and Levine, Sergey, Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, 2018 IEEE International Conference on Robotics and Automation (ICRA), 7559–7566, IEEE, 2018
  * [22] Brockman, Greg and Cheung, Vicki and Pettersson, Ludwig and Schneider, Jonas and Schulman, John and Tang, Jie and Zaremba, Wojciech, OpenAI Gym, arXiv preprint arXiv:1606.01540, 2016
  * [23] Gal, Yarin and Ghahramani, Zoubin, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, International Conference on Machine Learning, 1050–1059, 2016
  * [24] Burda, Yuri and Edwards, Harrison and Storkey, Amos and Klimov, Oleg, ÒNarrow-band analyzer (Thesis or Dissertation style),Ó Ph.D. dissertation, Dept. Elect. Eng., Harvard Univ., Cambridge, MA, 1993.
  * [25] Saunders, William and Sastry, Girish and Stuhlmueller, Andreas and Evans, Owain, Trial without error: Towards safe reinforcement learning via human intervention, arXiv preprint arXiv:1707.05173, 2017
  * [26] Chernova, Sonia and Veloso, Manuela, Interactive policy learning through confidence-based autonomy, Journal of Artificial Intelligence Research, volume 34, 1–25, 2009
  * [27] Levine, Sergey and Finn, Chelsea and Darrell, Trevor and Abbeel, Pieter, End-to-end training of deep visuomotor policies, The Journal of Machine Learning Research, volume 17, number 1, 1334-1373, 2016
  * [28] Rahmatizadeh, Rouhollah and Abolghasemi, Pooya and Bölöni, Ladislau and Levine, Sergey, Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration, 2018 IEEE International Conference on Robotics and Automation (ICRA), 3758–3765, IEEE, 2018
  * [29] Kumar, Vikash and Todorov, Emanuel, Mujoco haptix: A virtual reality system for hand manipulation, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), 657–663, IEEE, 2015
  * [30] Argall, Brenna D and Chernova, Sonia and Veloso, Manuela and Browning, Brett, A survey of robot learning from demonstration, Robotics and autonomous systems, volume 57, number 5, 469–483, 2009, Elsevier
  * [31] Srivastava, Nitish and Hinton, Geoffrey and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, volume 15, number 1, 1929–1958, 2014
  * [32] Finn, Chelsea and Tan, Xin Yu and Duan, Yan and Darrell, Trevor and Levine, Sergey and Abbeel, Pieter, Deep spatial autoencoders for visuomotor learning, 2016 IEEE International Conference on Robotics and Automation (ICRA), 512–519, IEEE, 2016
  * [33] Rhinehart, Nicholas and McAllister, Rowan and Levine, Sergey, Deep imitative models for flexible inference, planning, and control, arXiv preprint arXiv:1810.06544, 2018
  * [34] Gupta, Abhishek and Kumar, Vikash and Lynch, Corey and Levine, Sergey and Hausman, Karol, Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, arXiv preprint arXiv:1910.11956, 2019
  * [35] Laskey, Michael and Lee, Jonathan and Fox, Roy and Dragan, Anca and Goldberg, Ken, Dart: Noise injection for robust imitation learning, arXiv preprint arXiv:1703.09327, 2017
  * [36] Pomerleau, Dean A, Alvinn: An autonomous land vehicle in a neural network, Advances in neural information processing systems, 305–313, 1989
  * [37] Osa, Takayuki and Pajarinen, Joni and Neumann, Gerhard and Bagnell, J Andrew and Abbeel, Pieter and Peters, Jan, An algorithmic perspective on imitation learning, arXiv preprint arXiv:1811.06711, 2018
  * [38] Hussein, Ahmed and Gaber, Mohamed Medhat and Elyan, Eyad and Jayne, Chrisina, Imitation learning: A survey of learning methods, ACM Computing Surveys (CSUR), volume 50, number 2, 1–35, 2017, ACM New York, NY, USA
  * [39] Filos, Angelos and Tigas, Panagiotis and McAllister, Rowan and Rhinehart, Nicholas and Levine, Sergey and Gal, Yarin, Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?, arXiv preprint arXiv:2006.14911, 2020
  * [40] Hanczor, Matthew, Improving Imitation Learning through Efficient Expert Querying, 2018, Carnegie Mellon University Pittsburgh
  * [41] Nguyen, Khanh and Daumé III, Hal, Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms, arXiv preprint arXiv:2006.07777, 2020
  * [42] Cronrath, Constantin and Jorge, Emilio and Moberg, John and Jirstrand, Mats and Lennartson, Bengt, BAgger: A Bayesian algorithm for safe and query-efficient imitation learning
  * [43] Henaff, Mikael and Canziani, Alfredo and LeCun, Yann, Model-predictive policy learning with uncertainty regularization for driving in dense traffic, arXiv preprint arXiv:1901.02705, 2019
  * [44] Wu, Alan and Piergiovanni, AJ and Ryoo, Michael S, Model-based Behavioral Cloning with Future Image Similarity Learning, Conference on Robot Learning, 1062–1077, 2020
  * [45] Ghasemipour, Seyed Kamyar Seyed and Zemel, Richard and Gu, Shixiang, A divergence minimization perspective on imitation learning methods, Conference on Robot Learning, 1259–1277, 2019
  * [46] Tianhe Yu and Deirdre Quillen and Zhanpeng He and Ryan Julian and Karol Hausman and Chelsea Finn and Sergey Levine, Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, Conference on Robot Learning (CoRL), 2019
  * [47] J. Ho and S. Ermon. Generative adversarial imitation learning. InAdvances in Neural InformationProcessing Systems, pages 4565–4573, 2016.
  * [48] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning.InInternational Conference on Learning Representations, 2018. URLhttps://openreview.net/forum?id=rkHywl-A-.
  * [49] LeCun, Yann and Chopra, Sumit and Hadsell, Raia and Ranzato, M and Huang, F., A tutorial on energy-based learning, 2006.