# Safely Learning Visuo-Tactile Feedback Policies in Real
For Industrial Insertion

Letian Fu1,†, Huang Huang1, Lars Berscheid3, Hui Li2, Ken Goldberg1, Sachin
Chitta2 1The AUTOLab at UC Berkeley, 2Autodesk Research, 3Karlsruhe Institute
of Technology
†Work done as an intern at Autodesk

###### Abstract

Industrial insertion tasks are often performed repetitively with parts that
are subject to tight tolerances and prone to breakage. In this paper, we
present a safe method to learn a visuo-tactile insertion policy that is robust
against grasp pose variations while minimizing human inputs and collision
between the robot and the environment. We achieve this by dividing the
insertion task into two phases. In the first align phase, we learn a tactile-
based grasp pose estimation model to align the insertion part with the
receptacle. In the second insert phase, we learn a vision-based policy to
guide the part into the receptacle. Using force-torque sensing, we also
develop a safe self-supervised data collection pipeline that limits collision
between the part and the surrounding environment. Physical experiments on the
USB insertion task from the NIST Assembly Taskboard suggest that our approach
can achieve 45/45 insertion successes on 45 different initial grasp poses,
improving on two baselines: (1) a behavior cloning agent trained on 50 human
insertion demonstrations (1/45) and (2) an online RL policy (TD3) trained in
real (0/45).

##  I Introduction

Industrial assembly [21] is a precise manipulation task requiring contact
interaction between parts. Part feeding, peg insertion and object
reorientation (three sub-tasks of industrial assembly) have been extensively
studied by the control community [19, 18, 7, 24, 21]. Many of the early work
approaches the problem from the mechanical design aspect [24, 18] and motion
planning aspect [19, 7, 26]. Through Computer Aided Design (CAD), the order of
part assembly can be predetermined in simulation with precise pose information
[3], allowing robots to plan the necessary actions to assemble the design
[14]. Learning-based approaches recently have shown promise in addressing
industrial insertion tasks [34] on the NIST taskboard [12], a standard
benchmark that represents common industrial insertion tasks with parts that
have complex geometries [17]. However, applying learning-based methods for
industrial insertion remains challenging due to the requirement for frequent
human inputs [20] or high-precision sensors for collecting training data [34].
In addition, there is a need for a safe training and data collection procedure
for learning insertion tasks since parts are prone to breakage.

Tactile Reading$\pi_{tac}$$\\{x,z,\beta\\}$AInitial GraspBTactile Guided
AlignmentCamera ObservationCamera Observation$\pi_{vis}$$\\{x,y,z\\}$CVision
Guided InsertionDInserted Figure 1: Overview of the policy with the red arrows
indicating the robot actions given by the policy. (A) The robot grasps the
part at an initial pose. (B) The tactile guided policy $\pi_{tac}$ estimates
the grasp pose using the tactile image and aligns the z-axis of the part with
the insertion axis. (C) A vision guided policy $\pi_{vis}$ is used to insert
the part. (D) The part is inserted successfully into the receptacle.

Another challenge in an industrial insertion task is that the precise grasp
pose is often unknown due to variations in kitting and feeding of parts as
they arrive for assembly. As the grasped part is often occluded by the gripper
visually, grasp pose estimation is better achieved when using tactile sensing
[25]. While recent work has shown improvement in simulation accuracy for
industrial insertion [23] and successes in Sim2Real transfer for tactile-based
insertion tasks [11, 32], the simulation of soft contacts between tactile
sensors and objects with complex geometries remains an open problem [33]. In
this work, we present a novel approach to safely learn visuo-tactile feedback
policies in real for industrial insertion tasks under grasp pose
uncertainties, with inexpensive off-the-shelf sensors. Our approach draws on
tactile and visual feedback to deal with the grasp pose uncertainty and force-
torque sensing for a self-supervised training procedure that is safe,
minimizing damage during the training phase. We divide the insertion task into
two phases (as shown in Fig. 1):

  * •

An initial Align phase where a tactile-based policy $\pi_{tac}$ estimates the
grasp pose. The robot reorients and aligns the part with the insertion axis of
the receptacle.

  * •

A second Insert phase where a RGB image-based policy $\pi_{vis}$ guides the
robot to insert the part.

By recognizing that the insertion operation is reversible only from certain
target insertion poses – i.e. starting from such poses, the part can be
repeatedly unplugged from and inserted into the receptacle – we develop a
self-supervised data collection pipeline that avoids collision between the
part and environment. Prior to data collection, a human free-drives the robot
to provide one approximate target pose where the part is inserted. The robot
refines this target pose to find such a reversible pose by minimizing the
grasping force-torque, which helps minimize collisions during data collection,
resulting in a safer training process that is less likely to damage the
insertion part and receptacle.

This paper contributes:

  1. 1.

A safe self-supervised data collection pipeline with force-torque sensing in
real for insertion;

  2. 2.

A tactile-based alignment policy for orienting the part and an RGB image-based
insertion policy;

  3. 3.

Experimental results suggest that the policy achieves 45/45 successes on USB
connector insertion, outperforming two baseline methods (1/45 and 0/45).

##  II Related Work

Industrial insertion has been central in robotics for 50 years. It is
challenging due to occlusions brought by the robot gripper, grasp uncertainty
from the process of acquiring the part and its collision with the environment,
the fragility of the parts, and the precision required in controlling the
robot for insertion. Early work approached this problem using CAD information
to infer desired assembly sequences [3] and generating designs of part feeders
based on object geometry [24]. Other work approached the problem from an
algorithmic design perspective, with a focus on developing motion planning
strategies for peg insertion [18, 26].

Recently, learning-based methods have shown success on this task, often making
certain assumptions. This includes learning insertion policies with a physical
robot via Sim2Real transfer [10], online adaptation with meta-learning [29,
36], reinforcement learning [28, 20], self-supervised data collection with
impedance control [30], accurate state estimation [34], or decomposing the
insertion algorithm into a residual policy that relies on conventional
feedback control [10]. However, these approaches assume that the parts are
grasped with a fixed pose. To overcome this assumption, [34] perform accurate
pose estimation and motion tracking with a high-precision depth camera and use
a behavioral cloning algorithm to insert the part. [30, 31] require contact
between the part and the environment to occur during data collection, a
process that is expensive and often impractical for fragile parts. In
comparison, we use inexpensive off-the-shelf sensors and a safe self-
supervised data collection procedure that does not require such contact.

Grasped parts are often visually occluded by the gripper. Tactile feedback can
be an alternative sensing modality for grasp pose estimation. Recent work uses
tactile images from vision-based tactile sensors such as GelSight [35] and
DIGIT [15] to estimate the pose of grasped objects. [16] use Gelsight sensors,
BRISK features and RANSAC to estimate grasp pose. Gelsight produces high-
quality 3D tactile images and can determine depth imprint, which improves
feature detection by isolating the object from the background. DIGIT, a more
affordable tactile sensor, provides a 2D RGB image but not the light incident
direction (to generate the depth image). [11] generates tactile image data in
simulation for pose estimation of bottles caps but simulating contact and
physical interaction between tactile sensors and objects with more intricate
geometry is still challenging [33]. In this work, we collect a dataset of
tactile images in real for the USB connector with different grasp poses to
train a model for grasp pose estimation at test time.

Most prior work on tight tolerance insertion tasks [34, 5, 16, 4] leverages a
single modality, such as vision, tactile, or force-torque, limiting the
accuracy of the system due to occlusion, perspective effect, and sensory
inaccuracy. Multi-modal systems have been explored to improve the robustness
of automated insertion. InsertionNet 1.0 [30] and InsertionNet 2.0 [31] use
RGB cameras and a force-torque sensor for learning contact and impedance
control. [2] couple vision and tactile data to perform localization and pose
estimation, and demonstrate that vision helps with disambiguating tactile
signals for objects without distinctive features. [9] leverage tactile and
vision for deformable bag manipulation by performing auto-regressive
prediction. [8] use a contact-gated tactile, vision and proprioceptive
observation to train reinforcement learning policies. [25] also tackle the
problem of grasp pose uncertainty for insertion by using Newtonian Variational
Autoencoders to combine camera observations and tactile images. They
demonstrate results for USB insertion accounting for grasp pose uncertainty in
one translation direction. In this work, we separate the insertion problem
into an alignment phase and an insertion phase, decoupling vision and tactile
inputs and also present a novel safe self-supervised approach to data
collection. We are able to handle both grasp pose rotation and translation
uncertainty for the USB insertion task.

Figure 2:  Experiment setup and coordinate system. The $x$, $y$, $z$ axes are
labeled by red, green, blue respectively. We label the gripper frame, part
frame, human-provided target pose frame, and robot frame as $F_{G}$, $F_{p}$,
$F_{h}$, $F_{R}$ respectively. The insertion direction is defined as the
$z$-axis of $F_{h}$. When the part is inserted, $F_{h}=F_{p}$.

##  III Problem Statement

Overview: We consider a part insertion task, performed by a 7-DoF robot. The
robot is equipped with a parallel-jaw gripper with a tactile sensor mounted on
one jaw. The end-effector has a wrist-mounted RGB camera, and the robot
provides reliable force-torque readings at the gripper. The objective is to
learn a policy that can robustly insert the part into the receptacle while the
part’s pose within the gripper is unknown. The policy must also minimize human
inputs and part-receptacle collisions during training. The experiment setup
and the coordinate frames are shown in Fig. 2.

Assumptions: We make the following assumptions:

  1. 1.

A human provides one top-down grasp pose of the part inserted in the
receptacle.

  2. 2.

An experiment begins with the part pre-grasped by the robot gripper with the
part grasp pose within a range relative to the human-provided pose.

  3. 3.

During data collection, the robot operates in a rectangular collision-free
configuration space above the receptacle.

Problem Setup: We are given a tight-fitting receptacle for the part to be
inserted into as shown in Fig. 2. We find the target insertion pose $T_{R,h}$
of the part (grasped at an unknown pose) from one human-provided imperfect
demonstration. At any time step $t$, we have access to the wrist mounted
camera providing RGB observation $o_{\rho}(t)$ and a DIGIT tactile sensor
providing RGB tactile images $o_{\psi}(t)$. Additionally, we can measure force
$\vec{f}(t)$ and torque $\tau(t)$ at the gripper. Since we know the insertion
axis, we parameterize the action space with 4 degrees of freedom: gripper
translation in robot frame and gripper y-axis rotation.

##  IV Method

###  IV-A Self-Supervised Data Collection

Figure 3: Data collection for alignment (Top) and insertion policies (Bottom).
Data collection for the Alignment policy starts with the part inserted into
the receptacle (Top (A)). The robot then samples and records different grasp
poses and the corresponding tactile images (Top (B)). Data collection for the
Insertion policy starts with a sampled refined grasp (Bottom (A)) and unplugs
the part to apply sampled transformations (Bottom (B)). Then the robot inserts
the part (Bottom (C)) and starts the next round of data collection with a
different grasp pose.

####  IV-A1 One Human Provided Imperfect Target Pose

The target pose $T_{R,h}$ is provided by a human free-driving the robot with a
pre-grasped part to insert it in the receptacle from top down. Since this
target pose may not have a perfect axis alignment with the receptacle, the
system performs a $z$-axis alignment of the target pose. To account for the
change in grasp center after axis alignment, we refine $T_{R,h}$ by finding a
target pose that minimizes gripper force-torque using a grid search through a
set of translations and rotations $\\{T_{\Delta}\\}$. Formally, we find

$\tilde{T}_{R,h}=\operatorname*{arg\,min}_{\tilde{T}_{R,h}\in\\{T_{\Delta}\cdot{T}_{R,h}\\}}\left\lVert\vec{f}(\tilde{T}_{R,h})\right\rVert+\left\lVert\tau(\tilde{T}_{R,h})\right\rVert.$
(IV.1)

Here $\vec{f}(\tilde{T}_{R,h})$ and $\tau(\tilde{T}_{R,h})$ denote the 3-DoF
force and torque vectors respectively when the gripper is at
$\tilde{T}_{R,h}$. Intuitively, this objective minimizes the external force
applied on the part when being unplugged, increasing the likelihood of the
insertion process being reversible. Practically, we perform grid sampling over
5 values of $x\in[\text{-}1,1]\text{mm}$, 5 values of
$y\in[\text{-}1,1]\text{mm}$ and 4 values of
$\gamma\in[\text{-}\frac{\pi}{180},\frac{\pi}{180}]\text{rad}$ ($x,y,\gamma$
are in $F_{h}$, refer to Fig. 2). The pose with minimum gripper external
force-torque is recorded as the refined demonstration pose $\tilde{T}_{R,h}$,
and we have $F_{h}=F_{G}$ at $\tilde{T}_{R,h}$.

A cascaded impedance controller, implemented within the robot’s real-time
control loop, allows fine-grained force control. In case of a force violation,
our system calculates a trajectory to a safe state within a single control
cycle. After refinement of the target pose, we search for the minimum offset
$z_{\text{min}}$ for the part to be unplugged from the receptacle. Finding the
minimum height helps to determine the boundary for data collection and allows
the pipeline to collect more data closer to the receptacle while reducing
collisions. Iteratively, the robot moves the gripper by $-\Delta_{z}$ in
$F_{G}$. We then move the gripper by $\Delta_{x}$ (in the $F_{G}$ frame) and
measure $\vec{f}_{x}$ (x-component of the gripper force in the $F_{G}$ frame).
If $\vec{f}_{x}\leq\eta$, we register the total upward distance traveled as
the minimum height $z_{\text{min}}$ for removal of the part. Empirically, we
find setting $\Delta_{x}=\Delta_{z}=$$1\text{\,}\mathrm{m}\mathrm{m}$, and
$\eta=$3.5\text{\,}\mathrm{N}$$ works well.

####  IV-A2 Data Collection for Alignment

The part remains inserted in the receptacle throughout this phase of data
collection. We explore grasp pose variations in 3-DoF ($x$, $z$ translation
and $y$-axis rotation $\beta$) in $F_{G}$ (refer to Fig. 2). We perform
uniform random sampling over the range $[-3,3]$mm, $[-8,-2]$mm,
[$-\frac{\pi}{15}$, $\frac{\pi}{15}$]rad for $x,z,\beta$, with 5, 10 and 20
samples respectively. The robot closes the gripper with a force of 70$N$ at
each of the sampled poses and records the pair of tactile image readings and
$x,z,\beta$. To account for the noise in the DIGIT tactile sensor, we take a
median filter over 5 consecutively captured tactile images. We collect 2000
data points in 120 minutes.

####  IV-A3 Data Collection for Insertion

Upon completing the tactile image collection phase, we collect robot poses and
RGB images for training the insertion policy for different grasps. We perform
grid sampling with 5 samples each of $x,z,\beta$ in $F_{G}$ within the same
range as in the previous stage, resulting in a total of 125 grasps. To account
for the difference between the sampled grasp $g$ and $\tilde{T}_{R,h}$, we
perform minimum force-torque refinement on the sampled grasp to calculate the
grasp-specific target pose $\tilde{T}_{R,h}(g)$. $\tilde{T}_{R,h}(g)$
translated by an offset of $z_{\text{min}}$ gives us the unplugged part pose
$T_{\text{unplug}}(g)$.

For each grasp $g$, we collect image data for the visuoservo policy by moving
the gripper to sampled points on a grid above the target pose. In particular,
we uniformly sample 5 values each of $x\in[-5,5]$mm, $y\in[-5,5]$mm and
$z\in[-5,0]$mm in $F_{R}$ with respect to $T_{\text{unplug}}(g)$, resulting in
125 different translations for the gripper. For each translation, we collect
one data point that does not contain additional rotation and sample two
gripper $y$-axis rotation conditioned on $z$ to avoid collision. Specifically,
given a height $z$, the two rotations are sampled from the uniform
distribution $\frac{z}{5}\cdot\mathcal{U}[-\frac{\pi}{15},0]$rad and
$\frac{z}{5}\cdot\mathcal{U}[0,\frac{\pi}{15}]$rad. These rotated data points
are added to provide the system with additional data for camera pose variation
with respect to the target pose. This choice leads to a balanced dataset with
375 distinct gripper poses. Each data point is composed of the gripper pose
(translated and rotated away from the target pose) and the corresponding RGB
image observation at that pose. Upon visiting all 375 gripper poses for a
given grasp, the robot moves to $T_{\text{unplug}}(g)$, performs a vertical
movement to $\tilde{T}_{R,h}(g)$, and opens the gripper jaws, thereby
resetting the part in the receptacle. The system repeats this data collection
process for all 125 grasps.

###  IV-B Learning to Insert

While human demonstrations usually serve as “expert policies” for industrial
insertion tasks, the self-supervised data collection pipeline allows us to
collect ground truth actions at scale. This allows us to formulate two
supervised-learning problems based on Sec. IV-A2 and Sec. IV-A3.

####  IV-B1 Alignment Policy

We use the data collected from Sec. IV-A2 to train an alignment policy
$\pi_{tac}$ that, given the tactile image, outputs the desired displacement of
the gripper $T_{p,G}$ to align the part with the receptacle. We augment the
tactile images by randomly jittering the brightness and contrast of the
tactile image over the range $\mathcal{U}[0.7,1.3]$.

####  IV-B2 Insertion Policy

We use data collected from Sec. IV-A3 to train a visuoservo insertion policy
$\pi_{vis}$ taking normalized camera observation, gripper $y$-axis rotation
$\beta$ and $x,y$ translations in $F_{h}$ as inputs, and predict the action:
desired translation $\Delta_{x},\Delta_{y}$ and rotation $\Delta_{\beta}$ in
$F_{R}$. We augment camera observations by randomly jittering the brightness
and contrast of the camera image, over the range $\mathcal{U}[0.7,1.3]$.

We use RegNet 32GF [27] as the backbone for both policies. For the alignment
policy, we replace the last layer of RegNet with a linear layer with 3
outputs. For the insertion policy, we concatenate the robot’s pose with the
latent vector of the image and replace the last layer with a linear layer with
3 outputs. For both of the networks, we use a batch size of 64, set the
learning rate to 1e-3, and use a learning rate decay of 0.99 for every 100
gradient steps. We pick the mean squared error as the loss function and use
the Adam optimizer [13].

###  IV-C Execution of Insertion

1 Input: Tactile Image $o_{\rho}(t)$, Camera Image $o_{\psi}(t)$, tactile
based grasp pose estimation network $\pi_{tac}$, and visuoservo insertion
policy $\pi_{vis}$, Target Pose $\tilde{T}_{R,h}$, Minimum Wrench Height
$z_{\text{min}}$, action norm threshold $\epsilon$, z direction step size
$d_{z}$

2 $T_{p,G}=\pi_{tac}(o_{\rho}(t))$

3 Move gripper to $\tilde{T}_{R,h}T_{p,G}+2z_{\text{min}}$

4 attempts = 0

5 $(\Delta_{x},\Delta_{y},\Delta_{\beta})=\pi_{vis}(o_{\psi}(t),T_{R,G}(t))$

6 while  _True_ do

7 while  _$\left\lVert[\Delta_{x},\Delta_{y}]\right\rVert >\epsilon$ and
attempts $<$ H_ do

8 Move gripper by $(\Delta_{x},\Delta_{y})$

9 attempts = attempts + 1

10 $(\Delta_{x},\Delta_{y},\Delta_{\beta})=\pi_{vis}(o_{\psi}(t),T_{R,G}(t))$

11

12 Translate gripper in insertion direction by $d_{z}$

13 Measure gripper force-torque in z-axis as $F_{z}$

14 if  _$F_{z} >20$N or attempts $\geq$ H_ then

15 Terminate

16

Algorithm 1 Policy Execution Procedure

To avoid catastrophic failure (i.e. collision between the part and the
surrounding environment or moving out of the training set distribution), we
deliberately formulate the visuoservo policy to only control $x,y$-axis
translation but not $z$-axis translation since the insertion direction is
already aligned with the $z$-axis by the Align policy. The formulation is
detailed in Algorithm 1. The execution procedure starts by inferring the grasp
pose from the tactile image via $\pi_{tac}$ (line 2). The system then performs
the insertion axis alignment of the part with the receptacle (line 3). We
measure the z-direction force based on the gripper force-torque sensor (line
12). We then calculate rotation and translation based on the camera
observation via $\pi_{vis}$ (line 5, 10) and execute the corresponding actions
(line 8) until the action has a norm smaller than $\epsilon$ (line 7). Note
that the rotation prediction from the Insertion Policy is not used, because
the part is aligned with the insertion axis by the Alignment Policy. When the
action converges, we lower the gripper in the $z$ direction by a step size of
$d_{z}$ (line 11) and continue to query $\pi_{vis}$ until the force constraint
is satisfied or the number of attempts exceeds the horizon $H$ (line 13-14).
We empirically set $d_{z}=1.5$mm. Empirically, we find setting the action norm
$\epsilon=0.0005$ works well.

##  V Experiments

(a) Parallel Jaw Gripper (b) Camera Mount Figure 4: CAD models for the
parallel jaw grippers and camera mount.

###  V-A Novel Hardware

We design a novel parallel jaw gripper mount to accommodate the DIGIT tactile
sensor [15] and camera mount (Fig. 2, Fig. 4). Given different parts, the
elastomer gel on the DIGIT sensor deforms, causing additional torque applied
to the part. This torque, along with the force applied from the receptacle to
the part during insertion, will lead to undesired slippage and rotation.
Therefore, we develop an asymmetric mounting setup where we mount the DIGIT
sensor on one jaw with reinforcement to prevent outward bending, while keeping
the other jaw flat. We apply sandpaper on the surface of the non-tactile
gripper, increasing the friction to reduce slippage. In practice, we find that
we can predict the part’s grasp pose of a USB connector using a single DIGIT
sensor.

###  V-B Experiment Setup

We focus on the USB insertion task on the NIST taskboard. We use a 7-DoF
Franka Emika robot with a parallel gripper, where one DIGIT tactile sensor is
mounted on the inside of one of the fingers. An Intel RealSense camera is
mounted offset from the gripper (Fig. 4). To control the Cartesian position of
the Franka robot, a time-optimal trajectory respecting velocity, acceleration,
and jerk constraints is applied to the policy’s positional output [1]. We use
grid sampling to obtain 5 values of $\beta$ ranging from
$[-\frac{\pi}{20},\frac{\pi}{20}]$rad, 3 translations in x and z from the
range $[-3,3]$mm and $[-6,-2]$mm in $F_{h}$, resulting in a test set of 45
different grasp configurations that lie in the training distribution of the
algorithms.

###  V-C Experimental Procedure

At the beginning of each test experiment, the USB connector (part) is pre-
grasped by the robot with a grasp pose selected from the test set and located
at a position with a $z$-axis translation of $2z_{\text{min}}$ relative to
$F_{h}$. At this starting pose, the gripper is aligned vertically down while
the USB connector is misaligned with the receptacle in both translation and
rotation as in Fig. 1(A). The robot first executes the Align policy to
estimate the part grasp pose and aligns it with the insertion direction of the
receptacle as in Fig. 1(B). It then uses the Insert policy to visuoservo and
inserts the part into the receptacle as in Fig. 1(C-D). The robot then resets
to the next grasp by releasing the part, re-grasping it, raising it to a start
pose as outlined above, and executing the Align and Insert policies for this
new grasp. It steps through all the test grasp poses using the same procedure.

An experiment terminates if the gripper frame ($F_{G}$) force in the $z$
direction $\vec{f}_{z}(t)$ exceeds $15\text{\,}\mathrm{N}$. An experiment
trial is considered successful if the gripper is within
$5\text{\,}\mathrm{m}\mathrm{m}$ of the target pose in $H=200$ iterations,
upon which we also visually inspect whether the insertion is successful. In
this set of experiments, the refined human-provided target pose
$\tilde{T}_{R,h}$ is provided as an input to the policies. In Sec. V-F2, we
perform ablation studies on noisy target poses.

Algorithms |  IL |  TD3 |  Proposed Approach
---|---|---|---
Success/Total |  1/45 |  0/45 |  45/45
TABLE I: Results suggest that the IL trained on 50 human demonstrations can
barely insert the part and rotations of the USB caused by collisions lead to
failure in training TD3. Our approach outperforms both baseline policies.

###  V-D Comparison

Our training procedure is designed with two objectives: (1) minimizing human
intervention so that ideally no human needs to be involved in data collection
or training of the policy, and (2) minimizing collision among the robot, the
part, and the environment. We compare our approach with two baseline learning
methods described below.

####  V-D1 Twin Delayed Deep Deterministic Policy Gradient(TD3)

An off-policy, online reinforcement learning policy [6] that learns the end-
to-end part insertion. This baseline satisfies objective (1) but violates
objective (2) — i.e. it is incapable of avoiding collisions among the robot,
the part, and the environment. We simplify the problem to a fixed, axis-
aligned grasp pose and therefore restrict the action space to translations
only. The learned policy runs with a frequency of
$10\text{\,}\mathrm{H}\mathrm{z}$ and controls the Cartesian velocity of the
robot. Due to the part’s axis-alignment with the receptacle, the policy’s
input can be restricted to the (low-dimensional) robot pose, velocity, and the
force-torque at frame $F_{G}$. We use the default TD3 implementation of Ray
RLLib [22]. If the force applied on the gripper exceeds
$10\text{\,}\mathrm{N}$, the episode terminates and the robot resets to a safe
starting pose.

####  V-D2 Imitation learning (IL)

Imitation learning from 50 human demonstrations of insertion trained with
behavior cloning. Each human demonstration starts with a randomly sampled
grasp pose as in Sec. IV-A2. This baseline algorithm violates objective (1)
but satisfies objective (2), where the human demonstrator selects actions that
minimize collision between the part and the environment. It takes about 30
minutes to provide all these human demonstrations.

###  V-E Results

The results are summarized in Table. I. The imitation learning agent (IL) is
only able to perform a single successful insertion out of the 45 grasp poses.
Intuitively, 50 different grasp configurations from human demonstrations are
not sufficient for training an accurate part pose estimation model;
additionally, for different grasp poses, at the same gripper pose, two
different human demonstrations may exist. The multi-modality in the
distribution of target insertion poses contributes to the failure of the IL
policy. Training the TD3 policy in the physical environment led to divergent
results in all 5 training trials we attempted. In all cases, the part collides
with the receptacle, leading to a drastic change in the grasp pose. This
cannot be corrected directly since there is no reset procedure that can
systematically recover the gripper to the original state without human
supervision. Our approach succeeds for every single grasp pose tested.
Empirically, we find that the part rotation and translation predicted from
tactile images are fairly accurate (refer to Table. II).

|  $x$ (mm) |  $z$ (mm) |  $\beta$ (rad)
---|---|---|---
Mean Error |  $8.97$e-$2$ |  $1.46$e-$1$ |  $5.59$e-$3$
Standard Deviation |  $4.89$e-$3$ |  $6.62$e-$2$ |  $4.89$e-$3$
TABLE II: Mean and standard deviation of the error in predicting part pose
($x,z,\beta$) by the tactile-based alignment policy on the test set of 45
grasps.

###  V-F Ablation Studies

####  V-F1 Effects of Leveraging Force-Torque Sensing in Data Collection

We compare the completion rate of the data collection process for insertion
with or without grasp pose refinement. We consider the following three
different methods for refining the target pose gained from the human
demonstration $T_{R,h}$: 1) ZA: apply $z$-axis alignment on $T_{R,h}$, 2)
ZAWF: perform minimum force-torque refinement only once for $T_{R,h}$ after
$z$-axis alignment (this step is only performed for the first grasp and the
results reused for all grasps) and 3) ZAWFG: perform $z$-axis alignment for
$T_{R,h}$ and apply minimum force-torque refinement separately for each grasp.

At the beginning of each experiment, a human provides $T_{R,h}$ by free-
driving the robot with one pre-grasped part to insert the part. A total of 125
different grasp poses are sampled. For each grasp pose $T_{h,G}$, we calculate
the grasp pose in robot frame by $T_{R,G}=\tilde{T}_{R,h}T_{h,G}$ with
$\tilde{T}_{R,h}$ determined by one of the three methods ZA, ZAWF or ZAWFG.
The robot grasps the part with the pose $T_{R,G}$, lifts the part, and tries
to re-insert the part. If insertion is successful, the robot executes the next
grasp otherwise the experiment terminates. We report total number of
successful insertions before termination. We repeat the experiment three times
for each method with different human demonstrations. The results are reported
in Table. III.

After applying $z$-axis alignment for the human-provided target pose (ZA), the
insertion fails as the center of the grasp is not aligned with the center of
the receptacle. ZAWF addresses this issue by using minimum pose refinement,
and can perform successful insertions. However, since the pose refinement is
specific to the human demonstration, we observe that the refinement is not
sufficiently granular, leading to failures when the new grasp configuration
has a large $y$-axis rotation. ZAWFG addresses this issue by performing pose
refinement for each of the grasps, resulting in consistent insertion
performance. The ZAWFG method requires a waiting time for the force
measurements to settle and thus takes longer to execute.

Setup | Trial 1 | Trial 2 | Trial 3 | Mean$\pm$Standard Error
---|---|---|---|---
ZA | 0/125 | 0/125 | 0/125 | 0.0$\pm$0.0$\%$
ZAWF | 125/125 | 125/125 | 34/125 | 75.7$\pm$19.8$\%$
ZAWFG | 125/125 | 125/125 | 125/125 | 100.0$\pm$0.0$\%$
TABLE III: Comparing data collection success rate. We measure the number of
successful insertions until failure for 125 different grasps configurations.
We compare Human Demonstration with axis alignment (ZA), Single Minimum Force-
Torque Refinement (ZAWF), and Minimum Force-Torque Refinement for all grasps
(ZAWFG). We report the mean success rate and the standard error for three
distinct human-provided target poses.

####  V-F2 Exploring Utility of Tactile and Vision Information

We perform another ablation study to highlight the relative benefits of using
tactile and vision for insertion tasks. In this case, we test 3 different
approaches: (1) A Tactile Only approach (2) A Vision Only approach trained
using a limited amount of the camera observation data and (3) a Combined
Approach. This ablation study differs from our earlier experiments by not
providing accurate target pose information to our policy. Instead, we inject a
uniformly sampled noise in the range $\pm 1$mm into the target pose’s $x,y$
translation to imitate imprecise knowledge of the target pose.

The Tactile Only approach attempts the entire insertion task in a single phase
using the tactile information to align the USB connector with the receptacle
and then move straight down to insert it. For the purpose of this study, we
modify the Vision Only model in the following manner. Recall that when
collecting camera observation data, for each waypoint we take three images:
one image with no additional gripper rotation and two with additional gripper
rotation. For training the new Vision Only model for this study, we only use a
third of the data set that does not have additional rotation. This mimics a
mono-view visuoservo model for receptacle localization and top-down insertion.
We modify the insertion motion from Sec. IV-B2 so the model only uses camera
observation and the $y$-axis rotation of the gripper, and it only outputs
translation in $x$ and $y$ based on the same regression objective as in Alg.
IV-B2. The combined approach sequences a Tactile Only approach in an Align
phase with the modified Vision Only approach in an Insert phase. We perform
experiments with the three different approaches with the same test set as in
Sec. V and report results in Table IV.

When the target pose is noisy, the Tactile Only model succeeds only 21/45
times. This is because the model only estimates the grasp pose but cannot
estimate the state of the target receptacle. The Vision Only model inserts the
part successfully in most cases where the grasps do not have any part rotation
(8/9 successes) but fails otherwise (for a total of 8/45 successes). This is
understandable since the policy is constrained to translation actions and is
trained on a limited set of data that does not include additional gripper
rotation. However, a separate Vision Only model trained with all the data is
similarly unsuccessful, with only 11/45 successes, showing that without the
ability to correct for grasp pose variation using tactile data, the system
will struggle to insert. The combination of the two models outperforms either
of the two models by leveraging an accurate rotation prediction of the part
(using tactile information) and implicitly estimating the state of the
environment (using visual information). This study suggests that with tactile
observation, we can reduce the complexity of an insertion problem by reducing
the uncertainty in the grasp pose of the part. It also highlights the benefits
of using visual information during the Insert phase to account for additional
uncertainty, e.g. a noisy target pose.

Algorithms |  Tactile Only |  Vision Only (No Rot) |  Tactile + Vision (No Rot)
---|---|---|---
Success/Total |  21/45 |  8/45 |  40/45
TABLE IV: Ablation study with noisy target poses comparing single-phase
Tactile Only, modified Vision Only, and a Combined two-phase approach
leveraging tactile and visual information.

##  VI Discussion

We present a safe self-supervised method to learn a visuo-tactile insertion
policy in real for industrial insertion with unknown grasp poses. We achieve
this by (1) using force-torque sensing to refine human demonstrated target
poses to discover ”reversible” insertion poses, enabling a safe self-
supervised method for data collection, and (2) constructing and learning a
two-phase approach to insertion by combining a policy for aligning the part
with the receptacle based on tactile information and for inserting the part
based on visual feedback. This separation of the insertion task into two
phases allows us to formulate the problem in a manner that is safe to learn in
comparison to a traditional reinforcement learning for insertion approach
where safe exploration is challenging. In future work, we aim to explore other
types of assembly tasks using this approach.

## References

  * [1] Lars Berscheid and Torsten Kröger  “Jerk-limited Real-time Trajectory Generation with Arbitrary Target States”  In _Robotics: Science and Systems XVII_ , 2021
  * [2] Arkadeep Narayan Chaudhury, Timothy Man, Wenzhen Yuan and Christopher G Atkeson  “Using Collocated Vision and Tactile Sensors for Visual Servoing and Localization”  In _IEEE Robotics and Automation Letters_ 7.2 IEEE, 2022, pp. 3427–3434
  * [3] LS Homem De Mello and Arthur C Sanderson  “A correct and complete algorithm for the generation of mechanical assembly sequences”  In _1989 IEEE International Conference on Robotics and Automation_ , 1989, pp. 56–57  IEEE Computer Society
  * [4] Yongxiang Fan, Jieliang Luo and Masayoshi Tomizuka  “A Learning Framework for High Precision Industrial Assembly”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 811–817
  * [5] Pete Florence et al.  “Implicit Behavioral Cloning”  In _Conf. on Robot Learning (CoRL)_ , 2021
  * [6] Scott Fujimoto, Herke Hoof and David Meger  “Addressing function approximation error in actor-critic methods”  In _International conference on machine learning_ , 2018, pp. 1587–1596  PMLR
  * [7] Kenneth Y Goldberg  “Orienting polygonal parts without sensors”  In _Algorithmica_ 10.2 Springer, 1993, pp. 201–225
  * [8] Johanna Hansen et al.  “Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 8298–8304  IEEE
  * [9] Hideyuki Ichiwara et al.  “Contact-rich manipulation of a flexible object based on deep predictive learning using vision and tactility”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 5375–5381  IEEE
  * [10] Tobias Johannink et al.  “Residual reinforcement learning for robot control”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 6023–6029  IEEE
  * [11] Tarik Kelestemur, Robert Platt and Taskin Padir  “Tactile Pose Estimation and Policy Learning for Unknown Object Manipulation”  In _Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS)_ , 2022
  * [12] K. Kimble et al.  “Benchmarking protocols for evaluating small parts robotic assembly systems”  In _IEEE Robotics and Automation Letters_ 5(2), 2020, pp. 883–889  IEEE
  * [13] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ , 2015
  * [14] Yotto Koga, Heather Kerrick and Sachin Chitta  “On CAD Informed Adaptive Robotic Assembly”  In _arXiv preprint arXiv:2208.01773_ , 2022
  * [15] Mike Lambeta et al.  “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation”  In _IEEE Robotics and Automation Letters_ , 2020
  * [16] Rui Li et al.  “Localization and manipulation of small parts using GelSight tactile sensing”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2014
  * [17] Wenzhao Lian et al.  “Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2021
  * [18] Tomas Lozano-Pérez  “Motion planning and the design of orienting devices for vibratory part feeders”  In _in IEEE Journal Of Robotics And Automation. MIT AI Laboratory_ , 1986
  * [19] Tomas Lozano-Perez, Matthew T Mason and Russell H Taylor  “Automatic synthesis of fine-motion strategies for robots”  In _The International Journal of Robotics Research_ 3.1 Sage Publications Sage CA: Thousand Oaks, CA, 1984, pp. 3–24
  * [20] Jianlan Luo et al.  “Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: A large-scale study”  In _Robotics: Science and Systems (RSS)_ , 2021
  * [21] KE McKee  “Automatic Assembly by G. Boothroyd, C. Poli and LE Murch, Marcel Dekker, New York, 378 pp., 1982”  In _Robotica_ 3.3 Cambridge University Press, 1985, pp. 195–196
  * [22] Philipp Moritz et al.  “Ray: A distributed framework for emerging AI applications”  In _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_ , 2018, pp. 561–577
  * [23] Yashraj Narang et al.  “Factory: Fast Contact for Robotic Assembly”  In _Robotics: Science and Systems (RSS)_ , 2022
  * [24] Balas K Natarajan  “Some paradigms for the automated design of parts feeders”  In _The International journal of robotics research_ 8.6 Sage Publications Sage CA: Thousand Oaks, CA, 1989, pp. 98–109
  * [25] Ryo Okumura, Nobuki Nishio and Tadahiro Taniguchi  “Tactile-Sensitive NewtonianVAE for High-Accuracy Industrial Connector-Socket Insertion”  In _arXiv preprint arXiv:2203.05955_ , 2022
  * [26] Hong Qiao, BS Dalay and RM Parkin  “Fine motion strategies for robotic peg-hole insertion”  In _Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science_ 209.6 SAGE Publications Sage UK: London, England, 1995, pp. 429–448
  * [27] Ilija Radosavovic et al.  “Designing network design spaces”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10428–10436
  * [28] Gerrit Schoettler et al.  “Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 5548–5555  IEEE
  * [29] Gerrit Schoettler et al.  “Meta-reinforcement learning for robotic industrial insertion tasks”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 9728–9735  IEEE
  * [30] Oren Spector and Dotan Di Castro  “InsertionNet - A Scalable Solution for Insertion”  In _IEEE Robotics and Automation Letters_ , 2021
  * [31] Oren Spector, Vladimir Tchuiev and Dotan Di Castro  “InsertionNet 2.0: Minimal Contact Multi-Step Insertion Using Multimodal Multiview Sensory Input”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2022
  * [32] Shaoxiong Wang, Mike Lambeta, Po-Wei Chou and Roberto Calandra  “TACTO: A Fast, Flexible, and Open-source Simulator for High-resolution Vision-based Tactile Sensors”  In _IEEE Robotics and Automation Letters (RA-L)_ 7.2, 2022, pp. 3930–3937  DOI: 10.1109/LRA.2022.3146945
  * [33] Yikai Wang et al.  “Elastic tactile simulation towards tactile-visual perception”  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 2690–2698
  * [34] Bowen Wen, Wenzhao Lian, Kostas E. Bekris and Stefan Schaal  “You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration”  In _Robotics: Science and Systems (RSS)_ , 2022
  * [35] Wenzhen Yuan, Siyuan Dong and Edward H Adelson  “Gelsight: Highresolution robot tactile sensors for estimating geometry and force”  In _Sensors 17_ , 2017
  * [36] Tony Z Zhao et al.  “Offline meta-reinforcement learning for industrial insertion”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 6386–6393  IEEE