#  On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation

Norman Di Palo1 and Edward Johns1 Manuscript received: August 21, 2023;
Revised November 9, 2023; Accepted December 8, 2023 .This paper was
recommended for publication by Editor Aleksandra Faust upon evaluation of the
Associate Editor and Reviewers’ comments. This work was supported by the Royal
Academy of Engineering under the Research Fellowship Scheme.1Norman Di Palo
and Edward Johns are with the Robot Learning Lab at Imperial College London
<EMAIL_ADDRESS>Object Identifier (DOI): see top of this
page.

###### Abstract

Imitation learning with visual observations is notoriously inefficient when
addressed with end-to-end behavioural cloning methods. In this paper, we
explore an alternative paradigm which decomposes reasoning into three phases.
First, a retrieval phase, which informs the robot what it can do with an
object. Second, an alignment phase, which informs the robot where to interact
with the object. And third, a replay phase, which informs the robot how to
interact with the object. Through a series of real-world experiments on
everyday tasks, such as grasping, pouring, and inserting objects, we show that
this decomposition brings unprecedented learning efficiency, and effective
inter- and intra-class generalisation. Videos are available at
https://www.robot-learning.uk/retrieval-alignment-replay.

###### Index Terms:

Deep Learning in Grasping and Manipulation, Imitation Learning, Learning from
Demonstration

Figure 1: An overview of the framework we study, showing the retrieval,
alignment, and replay phases. Together, these enable one-shot imitation
learning without prior object knowledge, as well as generalisation to novel
objects and novel classes.

##  I Introduction

In this paper, we study the problem of teaching a robot how to interact with a
set of training objects, and then generalising these learned behaviours to
novel objects and novel classes of objects. Today’s dominant paradigm in
recent literature is to address this with end-to-end behavioural cloning [4].
However, to generalise everyday manipulation skills to many different objects,
such techniques require a very large number of human demonstrations, which is
slow and expensive.

But as an alternative to monolithic, end-to-end control, we can decompose
reasoning into three distinct, specialist modes of reasoning. Firstly, what
can a robot do with an object? Secondly, where should a robot interact with an
object? And thirdly, how should a robot interact with an object? Our
hypothesis is that this decomposition might be more optimal than expecting a
single control policy to be able to reason simultaneously about all three
modes.

To achieve these three modes of reasoning, we propose a new framework based on
three principles: retrieval, alignment, and replay. When provided with an
object during testing, these are then used as follows. First, to determine
what can be done with the object, the most visually similar training object to
the observed object is retrieved from a memory buffer of the training objects.
Second, to determine where the robot should interact with the object, a goal
image representing the retrieved training object is used in a goal-conditioned
alignment policy, to align the end-effector appropriately with the object.
Third, to determine how the robot should interact with object, the robot
replays the demonstration velocities for that training object.

As such, our hypothesis is that, in the absence of an explicit demonstration
for a novel test object, interacting in the same way as with the most visually
similar training object, might be sufficient to interact with the test object.
And through a series of real-world experiments, we demonstrate this to be true
for a range of everyday tasks such as grasping, pouring, and inserting.
Furthermore, we study whether our proposed decomposition into individual modes
of reasoning, is optimal compared to methods which do decomposition but not
retrieval (e.g. [10]), to methods which do retrieval but not decomposition
(e.g. [12]), and to methods which do neither decomposition nor retrieval (e.g.
behavioural cloning [4]). Our conclusion, and the key takeaway message, is
clear: decomposing into retrieval, alignment, and replay leads to an order of
magnitude better learning efficiency compared to all these other alternative
paradigms. Furthermore, our framework only requires a wrist-mounted camera,
without the need for either extrinsic external or intrinsic calibration, and
begins learning tabula rasa, with no previous knowledge of the objects
required, such as CAD models:  our only assumption is that the relative pose
between wrist-camera and end-effector remains fixed between train and test
time, i.e. the camera is rigidly mounted to the robot.

##  II Related Work

Many recent methods that learn object manipulation from demonstrations, learn
end-to-end control policies using behavioural cloning, which can be
conditioned on a task identity or language command to enable generalisation to
novel objects. However, such policies tend to require extensive datasets of
human demonstrations [4]. In our work, we show that an explicit decomposition
into retrieval, alignment, and replay, results in a substantial improvement in
data and time efficiency over end-to-end control methods.

Our work builds on top of Coarse-to-Fine Imitation Learning (C2F-IL) [9],
which also proposes alignment and replay. However, C2F-IL learns tasks in
isolation: it learns a single control policy for a single object, which does
not generalise to novel objects. As such, we study a framework based on
retrieval, which can then scale up and learn from many different
demonstrations across many different objects, generalising to both novel
objects and novel object classes, and becoming more and more capable with each
new demonstration.

Other works have proposed a decomposition of the robot trajectory into a
coarse, approaching stage and a fine, last-inch interaction [10, 18, 2, 17,
15]. However, most have stronger constraints in their design compared to our
proposed framework: [18] needs synthetic visual data, and therefore models of
the objects, to train a perception system in simulation. Our method starts
learning tabula rasa, and needs no prior knowledge of the tasks or objects,
therefore making it substantially more general and deployable in everyday
environments. [2] also performs trajectory decomposition, but needs external,
calibrated RGB-D cameras to identify affordances. [16] frames robot
manipulation entirely as a series of objects alignments, it however relies on
point clouds and calibrated external cameras. Our method only needs a single
uncalibrated wrist-camera, which is significantly easier to deploy in
unstructured environments.

The use of retrieval for few-shot learning, especially in robotics,
reinforcement learning and control, has also been investigated elsewhere [7,
12]. [12] directly retrieves the actions to execute, whilst [7] retrieves
task-related data from an unlabelled buffer to train an end-to-end policy. We
combine retrieval and decomposition in a novel way, through an interplay of
retrieving goal observations and trajectories to execute and learning of an
alignment policy through self-supervised learning.

##  III Method

We aim to provide robots with the capability to both efficiently learn to
interact with an object, and transfer the learned behaviour to novel objects.
To achieve this, we propose a novel imitation learning framework based on an
interplay between retrieval from a memory buffer of visual observations, a
learned goal-conditioned visual alignment policy, and replay of demonstrated
trajectories. Our method only needs an RGB-D wrist-camera, rigidly mounted to
the robot’s end-effector, without the need for any calibration. No prior
knowledge of objects or tasks is needed, and no external camera is needed.

###  III-A Preliminaries

Unless explicitly stated otherwise, we use these terms with the following
meanings: a trajectory executed by the robot is composed of 3D linear and 3D
angular velocities, expressed in the end-effector frame. A pose refers to a 4D
vector $x,y,z,\theta_{z}$ expressed in the world frame: it can describe either
the pose of an object on a table, or the pose of the end-effector in the
environment. A policy is a function mapping visual inputs to action outputs.
An action is an end-effector velocity executed at each time-step. A task, in
this work, can be intuitively described as a verb, like grasp. The same task
can be performed on many objects (e.g. grasping a mug, grasping a teapot). The
environment is a tabletop setting with an object the robot should interact
with. All the tasks we consider involve executing full 6-DoF velocities during
object interaction.

Figure 2: We collect a single demonstration per object following the steps in
Sec. III-C. Through autonomously collected data, our method trains a goal-
conditioned alignment policy, that will align the robot to the bottleneck pose
before executing the trajectory. All collected data is stored in the memory
buffer.

###  III-B An Overview of the Retrieval, Alignment and Replay Phases

In this section, we provide a high-level overview of our framework and its
workflow, as illustrated in Figure 1.

When deployed during testing, the robot starts by obtaining a visual
observation of the environment with its wrist-camera. To understand what to do
with the object it has observed, it queries an external memory buffer with the
visual observation. The memory buffer retrieves where to interact with the
object in the form of a goal visual observation, and how to interact with the
object as a trajectory, expressed in the end-effector’s frame. This retrieved
data is used to guide the robot’s behaviour as follows.

The goal of the robot, after retrieving this information from the buffer, is
simple: to move its end-effector so that the live observation is aligned with
the retrieved goal observation. This is done via visual servoing through a
learned, goal-conditioned visual alignment policy. Once the alignment is
completed, the robot replays the retrieved trajectory by executing the
trajectory of end-effector velocities. This allows for successful interaction
with new objects, under the hypothesis that objects which are visually similar
should be interacted with in the same way, which we later validate
experimentally.

The following subsections describe in more detail how each part is
implemented.

###  III-C Understanding the Retrieval, Alignment, and Replay Phases

The order in which the phases are executed during deployment is retireval,
followed by alignment, followed by replay. However, for clarity, we will now
adopt the reverse order to describe them. This will allow us to first start
with simple concepts, and to later describe the overall framework, providing
the reader with a more intuitive sequence of ideas. The alignment and replay
phases are inspired by Coarse-to-Fine Imitation Learning (C2F-IL) [9], which
we now build upon using retrieval to allow for generalisation to novel
objects, and scalability by allowing for data from multiple demonstrations and
across multiple tasks to be absorbed by the robot.

####  III-C1 The Replay Phase: Recording and replaying demonstrated
trajectories

To teach the robot how to interact with an object, the human operator
manoeuvres the end-effector to provide a demonstration, e.g. with kinesthetic
teaching. The end-effector starts from a pose $b^{O}$, chosen by the operator
and expressed in the imaginary object frame $O$, that we call the bottleneck
pose. The only requirement is that the object must be visible by the wrist-
camera from the bottleneck pose. The demonstration is recorded as a series of
3D linear and 3D angular velocities of the end-effector expressed in the end-
effector frame $E$, which we call a trajectory
$s=[\mathcal{V}^{E}_{1},\dots,\mathcal{V}^{E}_{T}]$, where
$\mathcal{V}=[v_{x},v_{y},v_{z},w_{x},w_{y},w_{z}]$.

We can imagine the bottleneck pose being rigidly attached to the imaginary
object frame. With $W$ being the world frame, when we move the object, $b^{W}$
moves rigidly with it, while $b^{O}$ stays constant. If the end-effector is
re-aligned to $b^{W}$ when the object is moved, re-executing the end-effector
velocities of the recorded trajectory would suffice in solving the task again
[9]. Therefore, the learning process can focus on reaching $b^{W}$ again for
novel poses of the object. The replay phase is hence the execution of a
recorded trajectory. We store all the trajectories demonstrated during
training in our memory buffer as $[s_{1},\dots,s_{D}]$, where $D$ is the total
number of demonstrations.

####  III-C2 The Alignment Phase: Learning to align to the bottleneck again

Before the demonstration starts, the operator aligns the end-effector to
$b^{O}$, which is wherever the operator plans to start recording the
trajectory from. In this configuration, the robot can compute the exact pose
of $b^{W}$ through forward kinematics. The goal of the alignment phase is
reaching $b^{W}$ again at test time, when its location cannot be simply
computed from forward kinematics (because the object has now moved), and thus
must be predicted from visual observations.

Our approach is to train a goal-conditioned visual alignment network that,
guided by the wrist-camera observations, aligns the end-effector to $b^{W}$.
The network receives, as input, the live observation of the wrist-camera, and
an observation recorded at $b^{W}$: its goal is to align the former to the
latter by moving the end-effector, and hence, the wrist-camera. To train this
neural network, we need a dataset of visual observations of the object from
many different poses, and the relative pose displacement that would align the
end-effector to $b^{W}$ from each pose.

We gather this dataset as follows. We store the bottleneck visual observation
$o_{b}$ taken from $b^{W}$ and later use it as the goal observation when
training the network. The robot then moves in the volume above $b^{W}$,
reaching new random poses $p_{i}^{W}$ and collecting visual observations
$o_{i}$ from its wrist-camera. Knowing both $p_{i}^{W}$ and $b^{W}$ through
forward kinematics, it can compute the pose displacement $d_{i}^{E}$ in end-
effector frame $E$ that would bring the end-effector back to $b^{W}$, aligning
$o_{i}$ to $o_{b}$. All this data is added to the memory buffer. Once this
autonomous process is complete, the operator can manually manoeuvre the end-
effector to provide the demonstration as in Sec. III-C1. As shown in [9], this
enables a task to be learned from just a single demonstration. This process is
illustrated in Figure 2, left side.

Once all the desired $D$ demonstrations have been collected, across a number
of objects and tasks, our method can train the goal-conditioned visual
alignment network through regression, using the data stored in the memory
buffer. As illustrated in Figure 2 right side, we sample pairs of
$o_{b},o_{i}$ collected from the same object, along with the pose displacement
$d_{i}^{E}$. We use the former as inputs to the network, and the latter as the
label, optimising the weights through backpropagation and gradient descent
using an MSE loss.

Figure 3: Overall illustration of our framework. Upon observing a new object,
the agent visually compares it to other objects it observed during training.
After finding the most similar object, the agent retrieves a goal observation
of that object to guide the visual alignment phase (where to interact), and
also a trajectory to execute once the alignment is complete (how to interact).

At test time, when the object is in a different configuration and $b^{W}$ is
unknown, we feed the visual input from the wrist-camera to the goal-
conditioned visual alignment model along with the goal observation to align
to, which will then predict the pose displacement $d_{t}^{E}$ between
$p_{t}^{W}$, the current pose of the end-effector, and $b^{W}$: moving the
end-effector by that pose will align it to $b^{W}$, so that the replay phase
can then be executed. This process constitutes the alignment phase.

But, when at test time, and the robot observes a novel object, how does the
robot obtain a goal observation to guide the goal-conditioned visual alignment
network for the alignment phase? Similarly, how does the robot then decide
which trajectory to execute during the replay phase? The robot must
autonomously obtain this information guided only by the visual inputs it
receives. This constitutes the retrieval phase, as now described.

####  III-C3 The Retrieval Phase: Goal image and trajectory retrieval from the
memory buffer

After the training stage, where the robot receives a demonstration per each
object in the dataset, our robot is equipped with an external memory buffer
which stores the following: 1) the $D$ trajectories recorded by the operator,
2) the $D$ bottleneck visual observations $o_{b,d}$ used as goal observations,
and 3) the observations $o_{i,d}$ collected for each object at random poses by
the wrist-camera, with $0\leq d<D,d\in\mathbb{N}$, $0\leq i<I,i\in\mathbb{N}$,
$I$ being the total number of observations collected per object, and $o_{i,d}$
therefore being the $i$-th observation of the $d$-th object.

After receiving the visual observation of an object the robot should interact
with, the goal of the retrieval phase is twofold: 1) to retrieve one of the
$D$ bottleneck visual observations to use as the goal for the goal-conditioned
visual alignment network, guiding the alignment phase, and 2) to retrieve one
of the $D$ trajectories that, if executed during the replay phase, will lead
to a correct interaction with the object. The visual aspect of the novel
object should therefore guide the choice of where to align with the object,
and how to interact with it.

The robot obtains this information by querying the memory buffer with the live
observation. Our framework uses a similarity function
$g(o_{live},o_{i,d})\rightarrow[-1,1]$ computed between the current live
observation and all observations stored in the buffer. It firstly computes
$o_{j,n}=\text{argmax}_{o_{i,d}}g(o_{live},o_{i,d})$, and then, it retrieves
both the corresponding bottleneck visual observation $o_{b,n}$, and the
corresponding trajectory $s_{n}$. Intuitively, replicating the alignment and
interaction of the most visually similar object in the buffer studies our
hypothesis that visually similar objects should be interacted with in the same
way.

The recent computer vision literature proposes a variety of techniques to
extract useful representations from visual inputs to map them to smaller
dimensional embedding vectors, which capture semantic information of the
image, where it is more efficient and effective to compute distances using
well known mathematical functions. In our case, the similarity function is
computed as the cosine similarity between the visual features of the RGB
channels of the two observations extracted from a Vision Transformer (ViT) [6]
trained using DINO [3]. The literature has demonstrated how these features
capture both geometric and semantic information on both a local and global
scale, generalising between similar objects of the same or different classes
[1]. We extract these features for all the observations in the buffer offline,
after the training stage. During deployment, we extract features of the live
observation and compute the cosine similarity with all the stored features in
parallel. We justify this choice and compare it to several alternatives in
Section IV-E. This process is illustrated in Figure 3.

As we will show in our experiments, retrieval from the buffer unlocks
effective generalisation, while decomposition of object manipulation into
alignment and replay brings a substantial improvement in learning efficiency.
While the latter has been explored in [9, 14, 5], the combination of retrieval
and decomposition is distinctly novel, and leads to a general and scalable
framework that can continue to improve with further data. Furthermore, with
this framework, teaching a task on an object takes no more than a minute of
human time, and this is an important property when considering practical,
real-world imitation learning. The human operator simply provides just a
single demonstration per object; the remainder of the data collection (the
visual observations) is autonomous, and training is therefore self-supervised.

##  IV Experiments

In this section, we empirically measure the ability of our retrieval,
alignment, replay framework to learn behaviours efficiently and transfer them
effectively to new objects. We use four everyday tasks: grasping an object,
pouring from a cup into a container, unscrewing a bottle’s cap, inserting a
bottle’s cap, and inserting toy bread in a toy toaster (we combine these two
inserting tasks in one group). These tasks require non-trivial 6-DoF
interactions with objects, and some require considerable precision (inserting
a bottle’s cap allows for a maximum of 4mm of error). We invite the reader to
watch the videos on our website at https://www.robot-learning.uk/retrieval-
alignment-replay.

Setup: We run our experiments on a Sawyer robot with a Robotiq gripper. We use
a wrist-mounted RGBD Intel RealSense camera only, which does not need any
calibration in our method.  Our only assumption is that the end-effector to
camera extrinsics are constant between training and testing, i.e. the camera
remains fixed, a common assumption when mounting a wrist-camera to a robot [2,
5, 9]. The camera receives RGBD images which we rescale to $128\times
128\times 4$. We use a set of everyday objects (Figure 5) to illustrate the
method’s ability to tackle common tasks. The agent starts learning tabula
rasa: no previous knowledge of tasks or objects, such as CAD models, is used.
Each method we study is trained with demonstrations using the objects depicted
in Fig. 5, top bar plot. Each is then tested using the objects depicted in all
three bar plots: training objects (top), intra-class unseen objects (middle),
inter-class unseen objects (bottom).  The wrist camera, rigidly mounted to the
end-effector, starts the episode at 70cm above the table, and its field-of-
view can observe the whole table in front of the robot, as demonstrated in a
video on our website.

###  IV-A Baselines

Different methods in the recent literature have proposed augmenting robot
manipulation with retrieval or decomposition of the trajectory, similar to the
approach we adopt [10, 18, 2, 12]. We investigate the contribution to
manipulation performance brought by these building blocks, to motivate our
design choices and better understand the effect of each.

In particular, we identify 4 different families of approaches, grouped by
whether or not they perform retrieval from a buffer of data gathered during
demonstrations, and whether or not they decompose object manipulation into an
alignment part followed by an interaction part, or treat it as a single
learned trajectory. A visual taxonomy of these families is provided in Figure
4. For each of those families, we implement and evaluate a method from the
recent literature. To have a fair comparison, we chose methods that require
similar assumptions to our method: as we discuss in Section II, some methods
require either numerous, precisely calibrated cameras, and/or 3D models of the
objects, while ours requires neither.

Our method is composed of both a retrieval phase and a decomposition into
alignment and replay. GUAPO [10] also decomposes object manipulation into a
coarse alignment phase, approaching the object by feeding visual observations
into a neural network, and a last-inch learned policy is used to perform the
interaction. This method lacks an explicit retrieval phase, as the affordances
are recognised implicitly by a neural network. To train this method, we use
the same steps described in III-C2 to record data and train an alignment
visual servoing network, which in this case is not goal conditioned, but only
receives the live observation as an input. Differently from our method, it
uses a last-inch policy network that we train through Behaviour Cloning to
interact with the object. As the original paper [10] proposes the use of
Reinforcement Learning for the policy, while we use Behaviour Cloning for time
efficiency, we denote this method as BC-GUAPO.

Figure 4: A taxonomy of the different robot manipulation paradigms we study
and examples from recent literature, based on the concepts of retrieval and/or
decomposition.

Visual Imitation through Nearest Neighbours (VINN) [12], frames Learning from
Demonstration entirely as a retrieval problem. Demonstrations are stored frame
by frame in a memory buffer, together with the action taken by the expert at
that timestep. At each timestep during deployment, the agent queries the
buffer with the current observation to find the most similar $k$ observations.
The action to execute is computed as a weighted average of the actions
corresponding to these $k$ retrieved observations. While it is retrieval
based, this method does not decompose object manipulation trajectories, and
instead models manipulation as a single trajectory.

Finally, Behaviour Cloning (BC) [4] is implemented as a baseline that does not
use retrieval nor decomposition, but trains a single policy network with the
data collected during the demonstrations, and executes its actions at
deployment guided by visual observations. In this case, we train an end-to-end
policy network through Behaviour Cloning using the collected demonstrations.

We optimised neural network architectures and hyperparameters independently
for each method, in an attempt to maximise the performance of each and obtain
a fair comparison across these paradigms. To improve generalisation to
visually similar objects, all the alignment or policy network we train for
each method extracts visual features from the RGB channels through a
pretrained, frozen ResNet-50 backbone [8], and then concatenates the depth
channel, scaled to be in the same range as the extracted features.

Figure 5: Performance of our method and baselines on a series of manipulation
tasks involving tens of different objects, with the average performances as a
dashed line.

###  IV-B General Everyday Objects Interaction

Can our framework learn everyday-like manipulation behaviour efficiently, and
transfer those trajectories to novel objects? What contribution do retrieval
and decomposition bring, and how does this paradigm compare against recent
baselines from the literature? To answer this, we train all baselines
providing demonstrations of how to interact with the objects in the training
set (Fig. 5, top). As our method requires a single demonstration per training
object, in our first experiments we provide a single demonstration per
training object to all the baselines as well.

We cannot measure inter-class performance on the bottle cap insertion,
unscrewing a bottle cap and inserting toy bread in a toaster tasks, as these
tasks require a particular class of target object: we therefore only test for
inter-class generalisation on the grasping and pouring tasks, that can instead
be performed on a larger variety of unseen object classes (e.g. pouring can
generalise from mugs to bowls). At test time, we present each object from both
the training set and the test set to the robot. For each testing episode, the
object is randomly positioned on the table in front of the robot. Through
retrieval, the robot must understand how to interact with it based on the
demonstrations it received (i.e. what task to perform: grasp it, open it,
etc.). For example, if the robot observes a toaster and the bread slice in its
end-effector, it should infer that the task is to insert the bread. It must
then correctly execute the task, hence measuring its ability to generalise to
novel poses of the objects, and also to novel objects. We run 10 trials per
object, sampling a new  $(x,y,z,\theta_{z})$ pose each time. We train on 10
objects and test on 35 objects, for a total of 350 test-time episodes for each
baseline. Of these 35 objects, 25 were novel objects, and 10 belonged to
completely novel classes, demonstrating strong generalisation and adaptation
abilities.

Results for our method’s performance on each single object can be seen in Fig.
5. Here we show the success rate, and group those results into training set
objects (for which the robot received demonstrations), intra-class objects
(unseen objects from the same categories of the training objects, e.g. mugs),
and finally inter-class objects (unseen objects from novel categories).  All
these results are obtained without the need for either intrinsic nor extrinsic
camera calibration, and without any form of prior object knowledge.

The results indicate not only that our method can obtain remarkable
performance with just a single demonstration per object (an average of 80%
success on training objects after receiving a single demonstration for each),
but that it can generalise to intra and inter-class objects: the average
performance stays considerably close to the training set performance. (73%
success percentage on the intra-class set with respect training set, and 67%
on the inter-class set).  The strong performance of our method on unseen,
inter-class objects demonstrates that the proposed pipeline is robust even
when there is not a clear object to retrieve from the buffer. When asked to
grasp a banana (see Fig. 5), our method understands that the most similar
object is an horizontally-placed bottle by matching geometrical features, as
both share a locally cylindrical-like shape, as we show in the videos on our
website. While performance on unseen, inter-class objects is lower than on
training or inter-class testing objects, we can observe a much less steep
decrease compared to baselines that do not use retrieval and/or alignment
(Fig. 5).

###  IV-C Investigating the Benefits of Retrieval and Alignment

TABLE I: Success rate for trajectory replay and closed-loop learned policies on 6 tasks from our test set. Method / Task | Grasp Can | Grasp Mug | Unscrew Bottle | Pour | Close Bottle | Insert Bread | Mean ± Standard Deviation
---|---|---|---|---|---|---|---
Trajectory Replay (Ours) | 0.8 | $0.7$ | 0.5 | 0.9 | 0.8 | 0.7 | 0.73 ± 0.12
BC-GUAPO 1 | $0.3$ | $0.3$ | $0.0$ | $0.5$ | $0.2$ | $0.1$ | $0.23\text{ \textpm }0.16$
BC-GUAPO 10 | $0.7$ | 0.8 | $0.1$ | $0.8$ | $0.6$ | 0.7 | $0.62\text{ \textpm }0.24$

Can we measure the performance benefits brought by retrieval and alignment
both in isolation and then combined? We compare the overall performance of all
methods, and how each scales with more data, in order to more closely study
the individual contributions of the retrieval component, and the decomposition
component (into alignment and replay). Figure 6 shows the results of these
experiments, where each method is provided with either 1 demonstration or 10
demonstrations per training object, and where we measure the average success
rate over all the objects and tasks of both the training set and the intra and
inter-class test sets, running 10 episodes per object.

Comparing these results with the taxonomy proposed in Figure 4 allows us to
clearly identify the contribution brought by the align/interact decomposition
and/or retrieval. Behaviour Cloning uses neither of those, performs the worst
of all the baselines whether 1 demonstration or 10 demonstrations per object
are provided. VINN is based exclusively on retrieval, and when receiving a
single demonstration per object performs poorly as well. This suggests that
retrieval alone is not sufficient to enable efficiency and effective transfer.
On the other hand, BC-GUAPO, which decomposes trajectories into two phases
similar to our method, performs much better than VINN with just a single
demonstration per object. This clearly indicates the benefits of decomposing
manipulation in this way, because the coarse alignment phase can be trained in
a self-supervised manner from just a single demonstration.

When comparing BC-GUAPO to our method, however, we see a substantial gap in
performance: as our method is based on both retrieval and decomposition, this
suggests that retrieval brings a noticeable performance boost when paired with
decomposition. In particular, from our experiments it emerges that retrieval
benefits from a large memory buffer of observations: when receiving 10
demonstrations per object, VINN clearly surpasses BC as well due to VINN
benefitting from the use of retrieval. Our framework combines the best of both
worlds: the decomposition into alignment and replay enables the robot to
autonomously gather a large buffer of demonstrations without the need for
human supervision. Given this buffer, the use of retrieval then enables the
robot to generalise efficiently to novel objects.

###  IV-D A Comparison of Trajectory Replay and Learned Policies

Is replaying a recorded trajectory as effective as learning a policy and
executing it in closed-loop guided by visual inputs? At a first glance,
replaying a trajectory may sound suboptimal. But as demonstrated by recent
works [9, 14, 5], if the alignment is precise, replaying such trajectories can
obtain equal if not superior performance, in addition to being substantially
more efficient than training a policy function in the form of a neural
network.

To highlight this effect, we conduct an experiment that compares our replay
method to a baseline that uses a policy network, where both begin after
aligning the end-effector with the object. We ignore the retrieval phase here,
and train and test each task in isolation. First, we collected results for our
method without the retrieval phase, using only the alignment and replay phases
after manually selecting the task to execute (as a goal image and trajectory),
which is thus very similar to C2F-IL [9]. Then, we collected results for our
BC-GUAPO baseline. Both methods share the same alignment policy, trained as in
Section III, and the different interaction phases are hence the only source of
difference in performance: once the alignment is complete, our method executes
the recorded trajectory, while BC-GUAPO uses a learned policy network, feeding
it visual inputs from the wrist-camera.

We train the two methods on a subset of the tasks: two grasping tasks,
unscrewing a bottle, closing a bottle, pouring from a cup into a container,
and inserting toy bread into a toy toaster (Table I). We provide a single
demonstration per task to our method and to BC-GUAPO 1, while we provide 10
demonstrations to BC-GUAPO 10. We then test the performance on an unseen
object of the same class per each task and class. We show results in Table I.
This clearly proves the effectiveness of trajectory replay compared to
explicitly learning an end-to-end policy, for a range of everyday tasks,
beyond just grasping. As demonstrated in [4], an end-to-end policy network may
need hundreds of demonstrations per task to correctly learn skilful
manipulation of everyday objects, whereas our experiments show that trajectory
replay is a simple but effective approach when data efficiency is important.

###  IV-E A Comparison of Retrieval Techniques

What technique performs best in retrieving the most similar observation from
the memory buffer? We compare the of use pre-trained neural networks trained
on large datasets of observations, including both those which do and those
which do not have a robotics focus. The process can be formulated as follows:
given an observation $o_{i}$, we compute a representation by feeding it into a
neural network, and extracting either an output embedding vector, or an
intermediate representation. We use ResNet-50 [8], CLIP [13], R3M [11], and
ViT DINO [3].

Figure 6: Average success rate of our method and the baselines, where each is
provided with either (1) or (10) demos per training object. Figure 7: Average
retrieval accuracy of different feature extraction methods. We test each
method on the intra-class test set (Fig. 5, middle), measuring the ability to
retrieve objects of the same class.

We measure retrieval accuracy by providing to each method an observation of an
unseen object from the intra-class test set (e.g. unseen mugs, pans, teapots).
We then measure how many times the method retrieves from the buffer an
observation of the object belonging to the same class, indicating that the
representation computed from that observation maximises the similarity of the
representation computed on the test object observation. We show results in
Figure 7. The features extracted via a Vision Transformer trained with the
DINO framework achieve the best accuracy, confirming recent findings from the
literature [3, 1] that such features encode semantic, geometric, global and
local information of the observation effectively.

##  V Discussion

###  V-A Further Questions and Limitations

What if there are more objects at test time? Currently, our method relies on
visual inputs to predict what to do in an environment: multiple objects may
therefore lead to ambiguity. However, if we use an off-the-shelf object
segmentation method, we can in principle interact with each object in
succession. Future work will address these limitations. Why limit the
alignment phase to 4-DoF? This is not a fundamental limitation of our method;
if required, the alignment phase could be extended to 6-DoF by simply
collecting images over 6-DoF end-effector poses. Also, in our tabletop
implementation, although the alignment phase is 4-DoF, the replay phase is in
fact 6-DoF, enabling complex interactions. What about tasks that require
feedback loops, pushing objects to a target? Our method cannot be applied to
these methods in its current form, but in this work we show that a large set
of everyday-tasks can be tackled with our framework, and that it is
surprisingly competitive and often better in terms of efficiency and
performance compared to learned closed-loop policies. What happens if you want
to interact with the same object in different ways? If an object can be
interacted with in multiple ways, the robot needs an external input, provided
by e.g. a planner or the user themselves, to select an interaction. Given
this, our proposed framework could then still be used for executing an object
manipulation task.

###  V-B Conclusions

In this work, we propose a novel framework to enable efficient and effective
learning from demonstration in robotic manipulation. In particular, we
introduce a combined use of retrieval, and trajectory decomposition into
alignment and replay. We analysed the best way of implementing retrieval and
decomposition by comparing baselines from the literature, and showed that our
combination of retrieval, alignment, and replay, performs better than
alternative paradigms. Furthermore, we demonstrated how retrieving and
replaying demonstration trajectories is a simple yet powerful alternative to
training end-to-end policies, with significant gains in data efficiency.

## References

  * Amir et al. [2021] Shir Amir et al.  Deep vit features as dense visual descriptors.  _arXiv:2112.05814_ , 2021.
  * Borja-Diaz et al. [2022] Jessica Borja-Diaz et al.  Affordance learning from play for sample-efficient policy learning.  _arXiv:2203.00352_ , 2022.
  * Caron et al. [2021] Mathilde Caron et al.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Collaboration [2023] Open X-Embodiment Collaboration.  Open X-Embodiment: Robotic learning datasets and RT-X models.  _arXiv:2310.08864_ , 2023.
  * Di Palo and Johns [2021] Norman Di Palo and Edward Johns.  Learning multi-stage tasks with one demonstration via self-replay.  In _CoRL_. PMLR, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Du et al. [2023] Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn.  Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets.  _arXiv preprint arXiv:2304.08742_ , 2023.
  * He et al. [2016] Kaiming He et al.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Johns [2021] Edward Johns.  Coarse-to-fine imitation learning: Robot manipulation from a single demonstration.  In _ICRA_ , 2021.
  * Lee et al. [2020] Michelle A Lee et al.  Guided uncertainty-aware policy optimization: Combining learning and model-based strategies for sample-efficient policy learning.  In _ICRA_. IEEE, 2020.
  * Nair et al. [2022] Suraj Nair et al.  R3m: A universal visual representation for robot manipulation.  _arXiv:2203.12601_ , 2022.
  * Pari et al. [2021] Jyothish Pari et al.  The surprising effectiveness of representation learning for visual imitation.  _arXiv:2112.01511_ , 2021.
  * Radford et al. [2021] Alec Radford et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Valassakis et al. [2022] Eugene Valassakis et al.  Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning.  _arXiv:2204.02863_ , 2022.
  * Vitiello et al. [2023] Pietro Vitiello, Kamil Dreczkowski, and Edward Johns.  One-shot imitation learning: A pose estimation perspective.  In _CoRL_ , 2023.
  * Vosylius and Johns [2023a] Vitalis Vosylius and Edward Johns.  Few-shot in-context imitation learning via implicit graph alignment.  In _CoRL_ , 2023a.
  * Vosylius and Johns [2023b] Vitalis Vosylius and Edward Johns.  Where to start? transferring simple skills to complex environments.  In _CoRL_ , pages 471–481. PMLR, 2023b.
  * Wen et al. [2022] Bowen Wen et al.  You only demonstrate once: Category-level manipulation from single visual demonstration.  _arXiv:2201.12716_ , 2022.

##  VI Supplementary Material

###  VI-A Algorithms

Here we describe our method in algorithmic form, starting from demo and data
collection in Alg. 1, then describing the deployment phase in Alg. 2.

Algorithm 1 Pseudo code for data collection and demo recording.

for n in objects do

Operator brings robot to $b^{W}$

$o_{b}\leftarrow$ Record observation

demo_buffer $\leftarrow$ Store $o_{b}$

for $t$ in $\text{range}(T)$ do

Go to random pose $p^{W}_{t}$

$o_{n,t}\leftarrow$ Record observation

$d^{W}_{t}\leftarrow b^{W}_{t}-p^{W}_{t}$

$d^{E}_{t}\leftarrow{{}^{E}}T_{W}d^{W}_{t}$

demo_buffer $\leftarrow$ Store $o_{n,t},\text{ }d^{E}_{t}$

end for

Go to $b^{W}$

trajectory $\leftarrow$ Operator records demo from $b^{W}$

demo_buffer $\leftarrow$ Store demo

full_buffer $\leftarrow$ demo_buffer

end for

Algorithm 2 Pseudo code for object interaction at test time.

$\gamma=5\times 10^{-3}$m

$d^{E}_{t}=\infty$

while $d^{E}_{t}>\gamma$ do

$o_{live}\leftarrow$ Record observation

$o_{j,n}\leftarrow\text{argmax}_{o_{i,d}}g(o_{live},o_{i,d})$ Extract most
similar obs from buffer

$o_{b,n}\leftarrow$ Retrieve corresponding bottleneck obs from $o_{j,n}$

$d^{E}_{t}\leftarrow f_{\theta}(o_{live},o_{b,n})$ Compute displacement to
align $o_{live}$ with $o_{b,n}$

Execute $d^{E}_{t}$

end while

trajectoryn $\leftarrow$ Retrieve corresponding demo from $o_{j,n}$

Execute trajectoryn

###  VI-B Test Time Examples

We provide here a series of frames taken from one of the videos on our
website, showing the robot tackling a grasping, pouring, inserting and
unscrewing. More videos are available at https://www.robot-
learning.uk/retrieval-alignment-replay.

Figure 8: A series of frames from one of our videos.