Maria Bauza,
Massachusetts Institute of Technology, Cambridge, MA 02139, USA.

# Tac2Pose: Tactile Object Pose Estimation from the First Touch.

Maria Bauza  Antonia Bronars  and Alberto Rodriguez  Massachusetts Institute
of Technology, Cambridge, MA 02139, USA
* Authors with equal contribution<EMAIL_ADDRESS>
###### Abstract

In this paper, we present Tac2Pose, an object-specific approach to tactile
pose estimation from the first touch for known objects. Given the object
geometry, we learn a tailored perception model in simulation that estimates a
probability distribution over possible object poses given a tactile
observation. To do so, we simulate the contact shapes that a dense set of
object poses would produce on the sensor. Then, given a new contact shape
obtained from the sensor, we match it against the pre-computed set using an
object-specific embedding learned using contrastive learning. We obtain
contact shapes from the sensor with an object-agnostic calibration step that
maps RGB tactile observations to binary contact shapes. This mapping, which
can be reused across object and sensor instances, is the only step trained
with real sensor data. This results in a perception model that localizes
objects from the first real tactile observation. Importantly, it produces pose
distributions and can incorporate additional pose constraints coming from
other perception systems, multiple contacts, or priors.

We provide quantitative results for 20 objects. Tac2Pose provides high
accuracy pose estimations from distinctive tactile observations while
regressing meaningful pose distributions to account for those contact shapes
that could result from different object poses. We extend and test Tac2Pose in
multi-contact scenarios where two tactile sensors are simultaneously in
contact with the object, as during a grasp with a parallel jaw gripper. We
further show that when the output pose distribution is filtered with a prior
on the object pose, Tac2Pose is often able to improve significantly on the
prior. This suggests synergistic use of Tac2Pose with additional sensing
modalities (e.g. vision) even in cases where the tactile observation from a
grasp is not sufficiently discriminative. Given a coarse estimate of an
object’s pose, even ambiguous contacts can be used to determine an object’s
pose precisely.

We also test Tac2Pose on object models reconstructed from a 3D scanner, to
evaluate the robustness to uncertainty in the object model. We show that even
in the presence of model uncertainty, Tac2Pose is able to achieve fine
accuracy comparable to when the object model is the manufacturer’s CAD model.
Finally, we demonstrate the advantages of Tac2Pose compared with three
baseline methods for tactile pose estimation: directly regressing the object
pose with a neural network, matching an observed contact to a set of possible
contacts using a standard classification neural network, and direct pixel
comparison of an observed contact with a set of possible contacts.

Website: mcube.mit.edu/research/tac2pose.html

###### keywords:

Tactile Sensing, Object Pose Estimation, Manipulation, Perception, Machine
Learning, Grasping, Contrastive Learning

††This paper extends the paper that appeared in the proceedings of the 2020
Conference on Robot Learning Bauza et al. (2020).

##  1 Introduction

Robotics history sends a clear lesson: accurate and reliable perception is an
enabler of progress in robotics. From depth cameras to convolutional neural
networks, we have seen how advances in perception foster the development of
new techniques and applications. For instance, the invention of high-
resolution LIDAR fueled self-driving cars, and the generalization capacity of
deep neural networks has dominated progress in perception and grasp planning
in warehouse automation (Zeng et al., 2018; Milan et al., 2018; Schwarz et
al., 2018). The long term goal of our research is to understand the key role
that tactile sensing plays in that progress. In particular, we are interested
in robotic manipulation applications, where occlusions challenge accurate
object pose estimation, and where object dynamics are dominated by contact
interactions.

Figure 1: Tactile pose estimation with Tac2Pose. (Bottom row) In simulation,
we render geometric contact shapes of the object from a dense set of possible
contacts between object and tactile sensor. (Top row) The real sensor
generates a tactile image from which we estimate its geometric contact shape.
We then match it against the simulated set of contact shapes to find the
distribution of contact poses that are more likely to have generated it. For
efficiency and robustness, we do the contact shape matching in an embedding
learned for that particular object.

In this paper, we propose Tac2Pose, a framework to estimate the pose of a
touched object, as illustrated in Figure 1. Given a 3D model of the object,
Tac2Pose learns an object-specific perception model in simulation, tailored at
estimating the pose of the object from one–or possibly multiple–tactile
images. As a result, the approach localizes the object from the first touch,
i.e., without requiring any previous interaction. The perception model is
based on merging two key ideas:

  * •

Geometric contact rendering: we use the object model to render the contact
shapes that the tactile sensor would observe for a dense set of contact poses.

  * •

Contact shape matching: given the estimated contact shape from a tactile
observation, we match it against a precomputed dense set of simulated contact
shapes. The comparison happens in an object-specific embedding for contact
shapes learned in simulation using contrastive learning tools (He et al.,
2019). This provides robustness and speed compared to other methods based on
direct pixel comparisons.

Enabled by the discriminative power of tactile sensing, the proposed approach
is motivated by scenarios where the main requirement is estimation accuracy
and where object models will be available beforehand. Many industrial
scenarios fit this category. We also consider cases where the object model is
not known exactly, but reconstructed apriori with a 3D scanner. This broadens
the set of scenarios to which Tac2Pose can be applied. Many previous solutions
to tactile pose estimation require prior exploration of the object (Li et al.,
2014; Bauza et al., 2019). Acquiring this tactile experience can be expensive,
and in many cases, unrealistic. In this paper, instead, we learn the
perception model directly from the object mesh model. The results in Sec. 4.2
show that the model learned in simulation directly transfers to the real
world.

We attribute this to the object-specific nature of the learned model, the high
resolution nature of the tactile sensors, and the contrastive-based perception
framework. Tac2Pose uses an intermediate representation of the tactile image,
a binary contact mask, which we are able to generate with high fidelity both
in simulation and using the real sensor. Furthermore, because Tac2Pose matches
observed contact shapes to a discrete set of simulated contacts, the match
doesn’t need to be perfect - just better than the other possible options - for
the localization to be accurate.

Also key to the approach is that, by simulating a dense set of tactile
imprints, the algorithm can reason over pose distributions, not only the best
estimate. The learned embedding allows us to efficiently compute the
likelihood of each contact shape in the simulated dense set to match with the
predicted contact shape from the tactile sensor. Predicting distributions is
key given that tactile sensing provides local observations, which often do not
discriminate pose globally (see, for example, the first contact in Figure
7(a)).

Finally, by maintaining probability distributions in pose space, we can
incorporate extra constraints over the likelihood of each pose. We illustrate
it in the case of multi-contact, where information from multiple tactile
observations must be combined simultaneously, and in the case of filtering the
distribution with a prior estimate of the object pose. In practice, this prior
estimate could come from previous tactile observations, kinematics, or other
perception systems (e.g., vision).

We evaluate the performance of Tac2Pose on real datasets collected for 20
different real objects of varying size and complexity in Section 4.2. The
labelled datasets evaluated in this work are available on the project website.
For 5 of the 20 objects, we also reconstruct the object models using a 3D
scanner. We compare the localization accuracy on the same real datasets, when
using a reconstructed model versus the manufacturer’s CAD model, in Section
4.3.

Finally, we compare the performance of Tac2Pose with three baseline methods
for 5 objects in Section 4.4: direct regression of the object pose using a
neural network, matching an observed contact to a set of possible contacts
using a standard classification network, and matching an observed contact to a
set of possible contacts using direct pixel comparison.

In summary, the main contribution of this work is a framework for tactile pose
estimation for objects with known geometry, with the following primary
strengths:

  * 1.

Provides accurate pose estimation from the first touch, without requiring any
previous interactions with the object.

  * 2.

Reasons over pose distributions by efficiently computing probabilities between
a real contact shape and a dense set of simulated contact shapes. This allows
for integration of pose constraints, such as those arising from multi-contact
scenarios or prior estimates of the object pose.

We evaluate Tac2Pose’s accuracy with respect to object geometry, uncertainty
(from reconstructed object geometry and tactile sensor noise), and other
methods for tactile localization. In particular, we show:

  * 1.

Quantitative results for 20 objects on three ablations of Tac2Pose using real
data: single contact, parallel jaw grasping, and parallel jaw grasping with a
pose prior (Section 4.2). We achieve high localization accuracy when making
contact with distinct object features. Complete results for each object can be
seen in Table 1.

  * 2.

Comparisons for results on 5 objects using reconstructed object shapes instead
of manufacturer’s CAD models (Section 4.3). The localization error on
reconstructed models increases between $0.2$ to $1.5$mm compared to
manufacturer’s CAD models. Complete results can be seen in Table 2.

  * 3.

Better performance of Tac2Pose when compared with three baseline methods
(Section 4.4). Results on real contact shapes for 5 objects can be seen in
Table 4.

##  2 Related Work

Tactile sensing has been extensively explored in the robotics community.
Relevant to this paper, this has resulted in the development of high-
resolution tactile sensors and their use in a wide range of robotic
manipulation applications. In this section, we review works that study tactile
pose estimation and refer the reader to (Luo et al., 2017a) for a more in-
depth review of other applications of tactile sensing.

While we propose to use high-resolution tactile sensors that are
discriminative and rich in contact information, most initial works in tactile
localization were focused on low-resolution tactile sensors (Schaeffer and
Okamura, 2003; Corcoran, 2010; Petrovskaya and Khatib, 2011; Chalon et al.,
2013; Bimbo et al., 2016; Saund et al., 2017; Javdani et al., 2013; Chebotar
et al., 2014). Some works explore how to combine multiple tactile readings and
reason in the space of contact manifolds (Koval et al., 2017, 2015). However,
these are based on binary contact/no-contact signals, and require many tactile
readings to narrow pose estimates.

Given the challenges from the locality of tactile sensing, recent works have
gravitated towards two different approaches. Combining tactile and vision to
obtain better global estimates of the object pose or using higher-resolution
tactile sensors that can better discriminate different contacts. Among the
solutions that combine vision and tactile, most rely on tactile sensors as
binary contact detectors whose main purpose is to refine the predictions from
vision (Bimbo et al., 2015; Allen et al., 1999; Ilonen et al., 2014; Falco et
al., 2017; Yu and Rodriguez, 2018).

Other works, more in line with Tac2Pose, have focused on using high-resolution
tactile sensors as the main sensing source for object localization. Initial
works in this direction used image-based tactile sensors to recover the
contact shape of an object and then use it to filter the object pose (Platt et
al., 2011; Pezzementi et al., 2011; Luo et al., 2017b). However, these
approaches only provide results on planar objects and require previous tactile
exploration. There has also been some recent work on highly deformable tactile
sensors for object localization (Kuppuswamy et al., 2019). These sensors are
large enough to fully cover the touched objects, which eases localization.

In this work, we use the image-based tactile sensor GelSlim 3.0 (Taylor et
al., 2021). The sensing capabilities of high-resolution sensors of this kind
have already proven useful in multiple robotic applications, including
assessing grasp quality (Hogan et al., 2018), improving 3D shape perception
(Wang et al., 2018) or directly learning from tactile images how to do contour
following (Lepora et al., 2019) or tactile servoing (Tian et al., 2019).

For the task of tactile object localization, Li et al. (2014) proposed to
extract local contact shapes from objects to build a map of the object and
then use it to localize new contacts. The approach is meant to deal with small
parts with discriminative features. Later Izatt et al. (2017) proposed to
compute pointclouds from the sensor and use them to complement a vision-based
tracker. Their tracker is fused with vision to deal with the uncertainty that
arises from the locality of tactile sensing. In previous work (Bauza et al.,
2019), we proposed to extract local contact shapes from the sensors and match
them to the tactile map of the objects to do object pose estimation. This
approach requires the estimation of a tactile map for each object by
extensively exploring them with the sensor.

Sodhi et al. (2020) estimated object pose during planar pushing from a stream
of tactile imprints through a factor graph-based estimation framework. Their
tactile observation model is trained to predict the relative pose of the
object between a pair of non-sequential tactile images. This approach is
designed to track the drift of an object from an initial well-known pose.
Similarly, Sodhi et al. (2021) estimated 3D object pose over a contact
sequence through a factor graph-based estimation framework. The tactile
observation model maps from tactile images to surface normals using an image-
to-image translation network, and uses ICP to determine the relative pose
between 3D contact geometry in a contact sequence. This approach is designed
to track the drift of object from an initial well-known pose, where the object
itself is arbitrary and unknown. In comparison, Tac2Pose assumes known object
geometry and estimates the pose of the object from scratch, while also being
able to exploit prior information when available.

Tac2Pose moves all object-specific computations to simulation and only
requires an object-agnostic calibration step of the tactile sensor to predict
contact shapes from RBG images from the GelSlim sensor (Bauza et al., 2020).
As a result, we can render in simulation contact shapes and learn object-
specific models for pose estimation that translate well to the real world and
achieve good accuracy from the first contact.

Recent work on tactile sensing applied to object insertion (Kim and Rodriguez,
2021) also uses an intermediate representation of the tactile image as the
input to their insertion policy. Because this intermediate representation
(pose of the contact line between the object and the cavity during an
insertion attempt) can be simulated easily, the insertion policy can be
trained entirely in simulation. Both this work and ours demonstrate the value
of relying on geometric intermediate representations of contact shapes to
allow high fidelity transfer of models trained on simulated data to real
robotic systems.

Finally, Tac2Pose to tactile pose estimation is related to methods recently
explored in the computer vision community where they render realistic images
of objects and learn how to estimate the orientation of an object given a new
image of it (Sundermeyer et al., 2018; Jeon and Kim, 2020). Li et al. (2018)
designs a network to predict a relative pose between an observed image, and a
rendered image of the object in a known pose. The method is used to
iteratively improve on a coarse initial estimate of object pose. Labbé et al.
(2020) leverages a modified version of Li et al. (2018) to estimate the pose
of multiple known objects from a sequence of images from multiple, unknown,
viewpoints. The approach consists of predicting object poses relative to the
camera within single views, then robustly matching objects and poses between
views, before performing a scene-level refinement of both object and camera
pose. Although this method does not reason explicitly over pose distributions
for a single-view estimate of pose, it is able to handle occlusions and
inaccurate single estimates by combining information from multiple viewpoints.

Other methods address occlusion and inaccurate estimates by reasoning over
pose distributions directly. Deng et al. (2019) estimates the 6D pose (3D
translation, and a distribution over a discretization of 3D rotations) of an
object in a Rao-Blackwellized particle filtering framework. The distributions
generated using this method function as scores on a codebook of possible
poses, rather than as well-defined probability distributions. Tac2Pose, in
contrast, regresses probability distributions which can be combined with
distributions coming from additional contacts, sensing modalities, or priors.

##  3 Method

We present an approach to object pose estimation based on tactile sensing and
known object models; illustrated in Fig. 1. In an object-specific embedding,
we match a dense set of simulated contact shapes against the estimated contact
shape from a real tactile observation. A contact shape is a binary mask over
contact regions on the tactile sensor. This results in a probability
distribution over contact poses that can be later refined using other pose
constraints. For example, we can combine information from the two contact
shapes and the gripper opening obtained by grasping an object with a parallel
jaw gripper.

We predict real contact shapes directly from the raw RGB tactile images that
the sensor outputs (Section 3.1). As seen in the top left quadrant of Figure
1, we predict a binary mask over the region of contact (a.k.a. the contact
shape) from an RBG tactile image with high fidelity.

The next steps of Tac2Pose exploit the object model to estimate the contact
pose, and are learned in simulation without using any real tactile
observations. First, we render the contact shape for a given object pose,
using the object model (Section 3.2). Examples of the correspondence between
object pose and contact shape obtained through geometric contact rendering can
be seen in the bottom half of Figure 1. Next, we use geometric contact
rendering to generate a dense set of object poses and their respective contact
shapes, and use contrastive learning to learn an embedding to match contact
shapes depending on the closeness of their object poses (Section 3.3). As a
result, given the estimated contact shape from a real tactile observation, we
can match it against this pre-computed dense set to obtain a probability
distribution over contact poses.

To predict contact poses beyond the resolution of the pre-computed set, it is
possible to combine Tac2Pose with registration techniques on the contact
shapes. While we do not explore this refinement step in detail in this paper,
we showed preliminary results on the efficacy of using FilterReg (Gao and
Tedrake, 2019) to further increase the localization resolution in (Bauza et
al., 2020). Finally, because our perception model outputs distributions over
contact poses, is not restricted to single tactile observations. The
distribution can be filtered or combined with other information coming from
multi-contact scenarios, other sensing modalities, or additional pose
constraints (Section 3.4).

###  3.1 Contact shape prediction from tactile observations

Given a tactile observation, our goal is to extract the contact shape that
produces it. To that aim, we train an image translation network (pix2pix)
based on Isola et al. (2018) to map RGB tactile observations to contact
shapes. We train pix2pix using pairs of real RGB tactile images and their
corresponding contact shapes. The training data is collected autonomously in a
controlled 4-axis stage that generates planned touches on known 3D-printed
shapes, following the approach proposed in Bauza et al. (2019). The dataset we
use to train pix2pix contains 10,000 RGB tactile image/contact shape pairs
from 32 distinct contact geometries. The dataset does not contain examples of
contact geometries from any of the objects we use to evaluate tactile
localization in Section 4.1. Note that the map between RGB tactile images and
contact shapes is independent of the object, and therefore we only need to
gather labelled data once. Empirically, we found that a model trained on
images from a single GelSlim sensor generalizes well across multiple instances
of GelSlim sensors.

We provide further implementation details in Section A of the Appendix,
including visualizations of the 32 contact geometries used for training
(Figure 9).

###  3.2 Contact shape rendering in simulation

Given the geometric model of an object and its pose w.r.t. the sensor, we use
rendering techniques to simulate contact shapes from object poses (Figure 1,
bottom). We refer to this process as geometric contact rendering. Below, we
describe how we compute contact poses, i.e., object poses w.r.t. the sensor
that would result in contact without penetration; and their associated contact
shapes:

  1. 1.

We create a rendering environment using the open-source library Pyrender (pyr,
). In this environment, we place a virtual depth camera at the origin looking
in the positive z-axis. The sensor can be then imagined as a flat surface (a
rectangle in our case) orthogonal to the z-axis and at a positive distance $d$
from the camera.

  2. 2.

We place the object in any configuration (6D pose) such that all points in the
object are at least at a distance $d$ in the z-axis from the origin. We then
compute the smallest translation in the z-axis that would make the object
contact the surface that represents the sensor.

  3. 3.

Finally, we move the object accordingly and render a depth image. The smallest
pixel value of the depth image corresponds to a depth $d$, and we consider
that only pixels between distances $d$ and $d+\Delta d$ are in contact with
the sensor, where $\Delta d$ represents the maximum deformation of the sensor.
The rest are marked as non-contact. We convert the depth image to a binary
image, where pixels with contact have a value of 1 and pixels with no contact
have a value of 0. This binary image is the contact shape.

For our sensor, depth images have a width and height of 640x640 (later re-
scaled to 160x160 for faster to compute), the distance to the origin is
$d=10$mm, and the maximum sensor deformation is $\Delta d=1.3$mm.

###  3.3 Global tactile pose estimation

Once we know how to compute contact shapes both in simulation and from real
tactile imprints, we reduce the problem of object pose estimation to finding
what contact poses are more likely to produce a given contact shape. We solve
this problem by first discretizing the space of possible contact poses as a
parametrized grid, and then learning a similarity function that compares
contact shapes.

Object-dependent grids. Using the 3D model of an object, we discretize the
space of object poses in a multidimensional grid. Building a grid in the space
of poses is a well-studied problem (Yershova et al., 2010; Roşca et al., 2014)
that makes finding nearest neighbors trivial. It also allows each point on the
grid to be seen as the representative of a volumetric part of the space which
helps to reason over distributions. We prune the grid by only keeping object
poses that result in contact, and then pair each of them with their respective
contact shape. Since we only consider poses that result in contact, this
reduces the space of 6D poses to a 5D manifold. Using a grid, a discrete
structured set of poses, allows us to easily account for object symmetries
which can significantly reduce the grid size.

Our grids cover regions of the object which correspond to feasible grasp
locations from the set of stable resting poses of the object on a surface. In
particular, we include faces that correspond to feasible grasp approach
directions, where the grasp approach direction specifies the axis of the grasp
relative to the object. For some objects, we also include additional grasp
approach directions that have interesting tactile features. For each grasp
approach direction, We compute a dense set of contacts with 2.5mm
translational resolution, and 6 degrees of rotational resolution around the
axis of the grasp. As a result, the grid spans four coordinates (grasp
approach direction, x, y, $\theta$) and a contact pose will be no more than
1.25mm from an element of the grid, and often closer. A visualization of the
grid dimensions can be seen in Figure 2. The number of elements in each grid
varies between 3.8K and 181.3K, depending on the object size, shape, and
number of grasp approach directions.

Figure 2: Object grids. The four dimensions of the grid with respect to the
object are visualized (left): grasp approach direction, which is defined as
the direction of the axis of the grasp (blue arrow), x translation (red
arrow), y translation (green error), and angle in the plane of the grasp
(yellow arrow). Samples of grid elements on the object long grease are shown
as black dots, where each black dot represents the center location of the
gripper during a grasp. Tac2Pose assigns likelihoods to each gripper location
(right), given an observed contact (top). The most likely gripper locations
for an observed contact are colored green, while the least likely are colored
blue. The ground truth gripper location is shown as a black dot (right,
center).

Similarity metric for contact shapes. Given a new contact shape, we compare it
to all pre-computed contact shapes in the grid to find what poses are more
likely to produce it. To that aim, we modify Momentum Contrast (MoCo) (He et
al., 2019), a widely-used algorithm in contrastive learning, to encode contact
shapes into a low dimensional embedding based on the distance between contact
poses.

MoCo is able to learn unsupervised embeddings by building ”a dynamic
dictionary with a queue and a moving-averaged encoder”. Instead, Tac2Pose is
supervised and the elements in the queue are fixed and assigned to each of the
poses in the object’s grid. Given a new contact shape, our model learns to
predict the likelihood that each pose in the grid has produced the given
shape. This likelihood is computed by taking the softmax over the distances
between the embedding of the new contact shape and the embeddings saved in the
queue, which correspond to the embeddings of each contact shape in the grid.
Compared to the original MoCo, Tac2Pose is supervised because during training,
given a new contact shape, we can compute which element in the queue is
closest to it.

To implement the encoder in our model, we use a ResNet-50 (He et al., 2016)
cropped before the average-pooling layer to preserve spatial information,
making it a fully-convolutional architecture. The loss function is the
categorical cross-entropy loss which allows us to ensure that the output of
the softmax is a well-defined probability distribution. The training data
comes from selecting a random contact pose and finding its closest element in
the dense grid. Then, we use as desired probabilities a vector of all zeros
except for the closest element which gets assigned to probability one (see
Figure 3). Section C in the Appendix contains further details on the learning
method.

Once we have created a dense grid and trained a similarity encoder for an
object, given a new contact shape at test time, we can estimate which poses
from the grid are more likely to generate it. To run Tac2Pose in real-time, we
first encode the given contact shape and then compare it to all pre-computed
encodings from the grid, which requires a single matrix-vector multiplication.
Finally, we perform a softmax over the resulting vector of distances to obtain
a probability distribution over the contact poses in the grid. We take the
best match to be the contact pose with the highest probability, after (if
applicable) incorporating additional pose constraints.

Figure 3: Similarity function. We build a similarity function that learns to
encode contact shapes into a low dimensional space and predicts, given a new
contact shape, the likelihood of being the closest match of each contact shape
in the pre-computed set. By learning an encoder for the contact shapes, we can
compare them very efficiently.

###  3.4 Multi-contact pose estimation

In this section, we show how to extend Tac2Pose to multi-contact settings
where we simultaneously need to reason over several tactile observations. In
Section D of the Appendix we show that the likelihood of an object pose from
the grid, $x$, given the estimated contact shapes, $C_{1,...,N}$, from N
sensors is proportional to

$P(x|C_{1},...,C_{N})\propto P(x|C_{1})\cdot...\cdot P(x|C_{N})\cdot P(x)$ (1)

where $P(x)$ is a prior on the object pose, and the embedding network has been
trained using uniformly-sampled poses. $P(x|C_{i})$ is the likelihood that
pose $x$ produces the contact shape $C_{i}$ on sensor $i$. This allows
combining Tac2Pose with additional pose constraints such as the ones coming
from kinematics, previous tactile observations, or other perceptions systems.

The terms $P(x|C_{i})$ come directly from computing the similarity function
between $C_{i}$ and the contact shape from the grid of sensor $i$ that is
closest to the contact pose $x$. As a result, the computational cost of
considering multiple contacts scales linearly with the number of sensors. In
this paper, we consider the case of grasping with a parallel jaw gripper. We
include information from two tactile images and the opening of the gripper
during the grasp. We also consider the case of filtering the parallel jaw
grasp distribution with a prior on the object pose.

##  4 Results

###  4.1 Real data collection

While Tac2Pose is trained purely with simulated data, in this section we
describe how it can provide accurate pose estimation when evaluated on real
tactile data. To that aim, we design a system that collects tactile
observations on accurately-controlled poses. Below we describe the tactile
sensor, the robot platform, and the objects used to perform the experiments.

Tactile sensor. We consider the tactile sensor GelSlim 3.0 (Taylor et al.,
2021) which provides high-resolution tactile readings in the form of RGB
images. The sensor consists of a membrane that deforms when contacted and a
camera that records the deformation. The sensor publishes tactile observations
through ROS as 470x470 compressed images at a frequency of 90Hz (see top left
portion of Figure 1).

Robot platform. We collect labelled datasets of tactile observations of grasps
on 20 objects mounted in known positions and orientations in the world. Each
dataset contains pairs of RGB tactile images, and the corresponding ground-
truth object pose relative to the gripper. The robotic system we use to
collect the labelled datasets consists of a dual arm ABB Yumi with two WSG-32
grippers and GelSlim 3.0 tactile sensing fingers. We provide further details
on the collection of labelled real data in Section B of the Appendix.

We collect labelled observations of feasible grasps on each each object. We
compute the contacts in each dataset by considering the stable poses of each
object. Stable poses are the poses that an object is most likely to fall in
when dropped onto a table. We then determine a set of feasible grasp approach
directions for each stable pose. We define the grasp approach direction as the
axis of the grasp during a parallel jaw grasp (Figure 2). For some objects, we
also include additional grasp approach directions that have interesting
tactile features. For each grasp approach direction, we collect a set of
equispaced real observations, that correspond to a given grasp approach
direction at various x,y locations relative to the object. In total, these
observations correspond to the set of contacts that we are likely to encounter
when picking up each object from a table.

Figure 4: Normalized pose errors, i.e., pose errors w.r.t. to the average
random error, for the object grease. The closest distance in the grid for this
object is 0.03, similar to the first case. The median pose error when using
parallel jaw contact information (0.10) corresponds to an error like the
second case , and the average random error (1) would match with the last case.
Finally, the example with 0.84 normalized error depicts a non-unique contact
shape, i.e., the two object poses result in very similar contact shapes at the
center that are not possible to distinguish without additional information.

Objects. We test Tac2Pose on 20 objects, derived from CAD models of objects
available on McMaster. During the selection of objects, we aimed at covering
object features that differently impact tactile localization. The features we
considered include:

  1. 1.

Contact non-uniqueness: Contact non-uniqueness is the degree to which a
contact is ambiguous, i.e., the contact in one pose resembles the contacts
from other object poses (that are not equivalent under any symmetry). Non-
unique contacts are more difficult to localize. An example of a non-unique
contact on the object grease is shown in Figure 4 (second from right).

  2. 2.

Symmetry: Object symmetries are sets of transformations under which the object
pose is indistinguishable. Symmetry is a desirable property because it reduces
the size of the grid.

  3. 3.

Object size: Larger objects are generally more challenging to localize with
tactile sensing alone, since a single touch corresponds to a more local view
of the object.

  4. 4.

Contact size: Large regions of contact with the sensor are generally more
difficult to localize than smaller ones, because most tactile sensors are less
likely to produce crisp imprints for large contacts.

We share the CAD models of each of the objects we use on the project website.

###  4.2  Real pose estimation results

We evaluate the accuracy of Tac2Pose at estimating object poses from real
tactile images. We note that Tac2Pose , even for large grids (100K elements),
can easily run at 50Hz allowing real-time estimation of pose distributions.
For each object, we evaluate Tac2Pose on 90 to 400 (varies depending on object
size and number of grasp approach directions) pairs of real tactile images and
object poses per object using the approach described in 4.1.

Given the ground truth object pose and our best estimate (the object pose with
highest likelihood), we measure the resulting pose error (or distance between
the two poses) by sampling a pointcloud of 10K points from the object 3D model
and averaging the distance between these points when the object is at either
of the two poses. This distance is sometimes called ADD (average 3D distance)
but, for simplicity, we just refer to it as the pose error. To compare errors
across shapes and object sizes, we also compute the normalized pose error
which divides the original pose error by the average error obtained from
predicting a random contact pose. Figure 4 shows examples of different
normalized pose errors.

For each object, we evaluate the pose error for three ablations of Tac2Pose :

  1. 1.

Single Contact: Estimate object pose from a single tactile image.

  2. 2.

Parallel Jaw: Estimate object pose from a pair of tactile images, collected
during a parallel jaw grasp on the object. The estimate also factors in the
opening of the gripper during the grasp.

  3. 3.

Parallel Jaw + Prior: Filter the distribution from the parallel jaw estimate
to remove any poses that are more than a given pose distance from ground
truth. This approximates cases in which an object pose is known within a
margin of error, and is relevant when tactile localization is used to refine a
coarse estimate of object pose from another sensing modality, like vision. We
evaluate prior distances of 10mm and 5mm.

The median error for each ablation for each object is shown in Table 1.

Table 1: Median error of the most likely pose. We take the error corresponding to the most likely pose for each grasp on an object, and report the median. The blue line next to each object is 20mm long, to show relative object scale. We consider the error when using a single contact, parallel jaw contacts, and parallel jaw contacts + a pose prior (prior distances of 10mm and 5mm). We report the median error in mm, and as a normalized error (in parenthesis). To normalize the distributions filtered with a prior, we use the expected random error from the filtered distribution. Objects that can be localized accurately (more than twice as good as random) with a single contact are colored in green, objects that can be localized accurately after inclusion of parallel jaw information are colored in blue, and objects that can be localized accurately after inclusion of parallel jaw information and a 10mm prior on the object pose are colored in purple. Median values referenced in the text are bolded. | Tactile Only | Pose Prior
---|---|---
|  Single Contact mm (norm) |  Parallel Jaw mm (norm) |  10mm Prior mm (norm) |  5mm Prior mm (norm)
Snap Ring |  | 1.5 (0.10) | 1.4 (0.10) | 1.4 (0.17) | 1.4 (0.38)
Grease |  | 1.3 (0.12) | 1.2 (0.10) | 1.2 (0.15) | 0.9 (0.26)
Slotted Shim |  | 4.0 (0.15) | 3.0 (0.12) | 2.4 (0.30) | 2.3 (0.57)
Round Clip |  | 3.4 (0.17) | 11.5 (0.58) | 2.0 (0.24) | 1.9 (0.50)
Hanger |  | 6.6 (0.19) | 2.6 (0.07) | 2.4 (0.30) | 2.4 (0.57)
Pin |  | 6.5 (0.20) | 5.6 (0.17) | 4.7 (0.66) | 3.6 (1.02)
Big Head |  | 7.8 (0.20) | 6.1 (0.16) | 4.9 (0.72) | 3.9 (1.19)
Holder |  | 5.8 (0.26) | 2.2 (0.10) | 1.8 (0.23) | 1.8 (0.45)
Round Couple |  | 13.6 (0.53) | 10.9 (0.43) | 6.0 (0.75) | 3.5 (0.91)
Hydraulic |  | 14.0 (0.67) | 4.9 (0.23) | 2.5 (0.31) | 2.0 (0.49)
Long Grease |  | 26.6 (0.76) | 3.3 (0.09) | 2.3 (0.33) | 2.3 (0.61)
Stud |  | 33.7 (0.85) | 13.4 (0.34) | 4.8 (0.57) | 3.4 (0.82)
Cotter |  | 19.0 (0.49) | 19.6 (0.51) | 2.9 (0.38) | 2.7 (0.70)
Cable Clip |  | 10.2 (0.59) | 11.7 (0.67) | 2.5 (0.30) | 1.9 (0.45)
Hook |  | 24.5 (0.77) | 27.2 (0.85) | 3.0 (0.38) | 2.4 (0.63)
Couple |  | 20.7 (0.79) | 19.9 (0.76) | 3.5 (0.42) | 2.6 (0.66)
Hose |  | 39.0 (0.62) | 41.6 (0.66) | 7.8 (1.00) | 4.2 (1.03)
Pencil |  | 38.1 (0.69) | 41.6 (0.76) | 5.0 (0.62) | 3.9 (1.05)
Round Hose |  | 37.6 (0.70) | 37.3 (0.69) | 5.5 (0.71) | 4.2 (1.05)
Long Pencil |  | 77.5 (0.96) | 78.5 (0.97) | 6.9 (0.85) | 4.3 (1.01)
Figure 5: Error distributions for snap ring (top row), long grease (second
row), hydraulic (third row), and cotter (bottom row). Each violin plot
contains three distributions. At left, the single contact error distribution
is visualized in green. At right, the parallel jaw contact distribution is
visualized in blue, while the parallel jaw distribution filtered with a 10mm
prior is overlaid in transparent purple. The median closest grid error is
visualized as a red line. This corresponds to the best possible median
performance. The median random error corresponds to a normalized pose error of
1. To the left and right of each violin plot, we show the localization errors
for a sample grasp on each object. In the green boxes, the true object pose
(grey mesh) is visualized with the best match when using a single contact
(green pointcloud). The blue boxes and purple box show the same true object
pose (grey mesh) with the best match when using parallel jaw contacts (blue
pointcloud), and parallel jaw contacts + a 10mm pose prior (purple
pointcloud), respectively. A single contact is enough to, on average, localize
snap ring accurately using Tac2Pose (top row). Parallel jaw contacts are
enough to localize long grease and hydraulic accurately (second and third
rows), even though a single contact is not. Parallel jaw contacts and a coarse
(10mm) pose prior are enough to localize cotter accurately (bottom row), even
though single contact and parallel jaw contact information are not enough.

We also aggregate the pose errors corresponding to the best estimate for each
grasp, then visualize the distribution of errors as a violin plot in Figure 5,
for selected objects. The medians reported in Table 1 correspond to the
medians of the distributions visualized in the violin plots.

For each of the selected objects, we show three different error distributions,
corresponding to the best estimates when using: a single contact (green
distribution), parallel jaw contacts (blue distribution), and parallel jaw
contacts plus a 10mm pose prior (purple distribution). To facilitate
comparison between the objects, we plot normalized pose errors. Note that a
normalized pose error of 1 corresponds to the expected error from selecting a
pose at random from the grid. The red line measures the median pose error
between the ground truth pose and its closest pose in the object’s grid of
simulated contacts (a.k.a. the closest error). This sets a lower bound on the
median performance for any given method.

Single Contact. We first analyze the performance of Tac2Pose with a single
contact. Of the 20 objects we evaluate, 9 have median localization error with
a single contact that is at least twice as good as random. These 9 objects
have single contact normalized errors less than 0.5 in Table 1.

The objects that tend to perform best with a single contact are small objects
with unique tactile features. For these objects, a single tactile imprint is
often discriminative enough to fully determine the object pose. Consider the
error distribution for snap ring, visualized in top row of Figure 5 as an
example. Even with a single contact (green distribution), the error
distribution has a clear primary mode which is centered near the closest error
from the grid (red line). When incorporating additional constraints from a
parallel jaw grasp, the distribution (blue) tightens around the closest error
from the grid (red line), and secondary, higher error modes disappear. The
median normalized error is approximately 0.10 for both the single contact and
parallel jaw contact cases (Table 1). In other words, choosing the most likely
pose using Tac2Pose results in, on average, approximately ten times less error
than selecting a pose at random from the grid.

A sample contact that is localized within 1.3mm (0.09 normalized error) of the
true pose using a single contact is visualized at left of the violin plot in
Figure 5. This provides a sense of the scale of the errors.

Parallel Jaw. We next consider objects that perform significantly better with
parallel jaw contacts. Of the 20 objects we evaluate, 11 have a median
localization error with parallel jaw contacts that is more than twice as good
as random. Of these 11 objects, 5 perform much better with parallel jaw
contacts than with a single contact. The 5 objects satisfying both criteria
are those in Table 1 where the parallel jaw normalized error is less than 0.5
and the single contact normalized error is more than double the parallel jaw
contact normalized error. Two such objects are long grease and hydraulic.

The objects that perform significantly better with parallel jaw contacts
compared with a single contact tend to be larger. Contacts with the object may
appear as large, featureless patches from the local perspective of the tactile
sensor. In general, large, flat contacts are more challenging to reconstruct
from tactile images because they do not deform the gel membrane of the tactile
sensor as much. Including the constraints of a second contact and the gripper
opening provides higher robustness to noisy contact reconstructions.
Furthermore, objects that vary significantly in width depending on where they
are grasped benefit from parallel jaw contacts. In these cases, the additional
constraint of the gripper opening during grasp can disambiguate between
otherwise non-unique contacts. Recall that a contact is non-unique if its look
similar to contacts where the object is in a different pose. In these cases,
the contact is not a unique indicator of the object pose.

As an example, the error distribution for long grease is visualized as a
violin plot in the second row of Figure 5. Considering parallel jaw
information shifts the mode of the distribution much closer to the closest
grid match (red line), and significantly reduces the prevalence of higher
error modes. The medians of the error distributions for the single contact and
parallel cases are significantly different; with a single contact, the median
pose error is 26.6mm (0.76 normalized error) while with parallel jaw contacts,
the median pose error is 3.3mm (0.10 normalized error). The random error, in
comparison, is 35.2mm. With a single contact, Tac2Pose only performs about 1.3
times better than random. When including parallel jaw contacts, however,
Tac2Pose is more 10 times better than random. The second row of Figure 5 (left
of the violin plot) shows a sample contact on long grease, and the
corresponding best match when using only a single contact (green pointcloud).
The best match using a single contact is 39.3mm (1.19 normalized error) away
from the true pose. The same contact, and the corresponding best match when
using parallel jaw contacts (blue pointcloud) is visualized to the right of
the violin plot. The best match using parallel jaw contacts is only 1.2mm
(0.04 normalized error) away from the true pose. In this case, the additional
information from the second contact and gripper opening is enough to resolve
the ambiguity and substantially improve the localization.

Parallel jaw contacts can also be an important tool for disambiguating
ambiguity inherent to the object geometry. As an example, consider a grasp on
the object hydraulic, visualized at left of the violin plot in the third row
of Figure 5. The true object pose is visualized as a grey mesh, whereas the
best match using a single contact is visualized as a green pointcloud. The
localization error with a single contact, in this case, is 16.6mm (0.84
normalized error). When considering only contacts from one of the fingers, the
contact shapes corresponding to the true object pose and the best match are
indistinguishable. Considering parallel jaw contacts, on the other hand,
resolves the ambiguity.

The same contact, and the corresponding best match using parallel jaw
contacts, is visualized at the right of the violin plot in the third row of
the same figure. The localization (blue pointcloud) using parallel jaw
contacts is much better because the contact on the second finger provides
critical information about the object’s pose. The localization improves from
16.6mm (0.84 normalized) error with a single contact, to 1.1mm (0.06
normalized) error with parallel jaw contacts.

All grasps on hydraulic in contact with this non-unique feature are subject to
the same problem when considering information from only a single contact, and
therefore the non-uniqueness impacts the overall localization quality; the
median localization error for hydraulic with a single contact is 14mm (0.67
normalized error), while with parallel jaw contact is 4.9mm (0.23 normalized
error). The violin plot in the third row of Figure 5 shows that the
distribution of errors when using a single contact (green) is bimodal, with a
portion of the second, high error, mode corresponding to flipped
localizations, like the one shown at left of the violin plot. When considering
the distribution of best match errors using parallel jaw contacts (blue
distribution), we see that the density of the high error mode corresponding to
the flipped localization decreases, and more probability mass shifts into the
low error mode centered around the closest error.

In the case of hydraulic, it is informative to break out the results by grasp
approach direction. The primary reason that the localization error with
parallel jaw contacts is, on average, nearly 3 times better than with a single
contact is that including parallel jaw contacts resolves the aforementioned
non-uniqueness, which only impacts one of the two grasp approach directions.
For the grasp approach direction that contains the non-unique feature,
including parallel jaw contacts reduces the localization error by nearly 8
times. For the other grasp approach direction, including parallel jaw contacts
reduces the localization error by 1.6 times.

Parallel Jaw + Prior. Finally, we consider the performance of Tac2Pose with
parallel jaw contacts plus a prior on the object pose. Pose priors, in
practice, can be obtained using additional sensing modalities (e.g. vision),
kinematics, or previous estimates of the object’s pose. Because tactile
sensing is inherently local, an object can have non-unique contacts which
won’t be possible to fully disambiguate even with parallel jaw contacts. In
such cases, a prior on the object’s pose can help resolve these ambiguities.
To test the effect of a prior on Tac2Pose , we take the prediction
distribution over possible object poses, and filter any pose that is more than
a given distance from ground truth. Note that this implementation of a pose
prior requires knowledge of the ground truth object pose, but it is a useful
proxy for cases in which the object pose is known roughly, but not accurately.
We evaluate localization accuracy for prior distances of 10mm and 5mm. Results
for both prior distances for each of the 20 objects are listed in Table 1.

The objects that benefit most from incorporating a prior on the object pose
are those with discrete non-uniqueness. An object with discrete non-uniqueness
has features that are unique (i.e. create discriminate tactile imprints)
relative to most other possible contacts on the object, with a discrete number
of exceptions. This type of discrete non-uniqueness can be an inherent feature
of the object, or an artifact of noisy and incomplete contact shapes. Because
there is a discrete number of possible object poses that are likely to produce
such a contact shape, we expect the distribution over object poses to have a
discrete number of modes where the majority of probability mass is
concentrated. In these cases, incorporating a pose prior can truncate the
distribution in such a way that only one of the modes is left. In other words,
because there is a discrete number of possible poses that are likely to
produce a given contact, if we eliminate the higher error options using a pose
prior, we are likely to get a near perfect match. Therefore, for some objects,
even a coarse prior on the object pose results in precise localization when
combined with tactile information.

We evaluate which objects benefit most from incorporating a pose prior by
comparing the pose error of the best match after filtering the pose
distribution, with the expected error from selecting a pose at random from the
filtered distribution. The normalized error we report in Table 1 for the 10mm
and 5mm prior ablations uses the expected random error from the filtered
distribution. For the remainder of this section, we consider just the case of
a 10mm pose prior. For 12 of the 20 objects we evaluate, using Tac2Pose on top
of a 10mm pose prior results in performance more than twice as good as
selecting a pose at random from the filtered distribution. For 5 of the 12
objects, incorporating a pose prior plays an important role in driving down
the localization error. With just parallel jaw contact information, the median
localization error for these 5 objects is not significantly better than
selecting a pose at random from the grid (where here we take significantly
better to be more than twice as good as random). However, after filtering the
distribution with a 10mm prior on the object pose, Tac2Pose selects the best
pose with more than twice as much accuracy as selecting a pose at random from
the filtered distribution. The objects that benefit most from incorporation of
a pose prior are those in Table 1 with parallel jaw normalized error greater
than 0.5, but 10mm prior normalized errors less than 0.5. An example of an
object for which the incorporation of a 10mm prior leads to significantly
better localization is cotter.

By examining the multi-modal error distributions with a single contact, or
parallel jaw contact (violin plot in the bottom section of Figure 5), we can
notice that the object cotter is likely to benefit from using pose prior. Its
error distributions have a discrete number of modes (three) where the majority
of probability mass is concentrated. Note that the symmetry of cotter is such
that parallel jaw contacts do not provide much new information relative to a
single contact, so the distributions of error for a single contact and
parallel jaw contacts are very similar. The median error is 19.0mm (0.49
normalized error) when using a single contact, and 19.6mm (0.51 normalized
error) when using parallel jaw contacts. Incorporation of a 10mm prior on the
object pose eliminates the two higher error modes, and shifts the median of
the distribution toward the closest error. The median error becomes 2.9mm
(0.38 normalized error), which is nearly seven times lower than using parallel
jaw contact information alone. Recall that after incorporating a pose prior,
the median error is normalized by the expected error from selecting a pose at
random from the filtered distribution rather than from all contact poses.

We show an example of a grasp that benefits from a pose prior at left of the
violin plots in the bottom row of Figure 5. With single or parallel jaw
contacts, the localization error is 16.5mm (0.43 normalized). The contacts
when cotter is in the given pose (grey mesh) are informative, but not
completely unique relative to other possible contacts on the object.
Incorporation of a coarse 10mm prior on the object pose, though, resolves the
ambiguity, and the localization improves from 16.5mm (0.43 normalized) to
1.8mm (0.23 normalized) error for this grasp. The best match after filtering
the parallel jaw distribution is visualized as a purple pointcloud on the
right side of the violin plot in the bottom row of Figure 5.

Comparing Grasp Approach Directions. We evaluate multiple grasp approach
directions for 13 of the 20 objects. For these objects, we compare the median
localization error using parallel jaw contacts of each grasp approach
directions. In practice, knowing which sets of grasps lead to lower
localization errors could be leveraged in a grasp planning framework to select
more informative grasps. There are 6 objects for which one direction has half
as much error or less as the other(s): long grease, grease, hanger, hydraulic,
round clip and round couple. Full results are shown in Table 6 in Appendix
Section E. As example, we compare the two grasp approach directions for round
clip in Figure 6. The first grasp approach direction, visualized in the left
half of the figure, has non-unique contacts. As a result, the median
normalized error is 15.2mm (0.85 normalized) when using parallel jaw contacts.
The second grasp approach direction, visualized in the right half of the
figure, has much more unique contacts; the median normalized error is 2.2mm
(0.11 normalized), and the primary mode of the error distribution is
concentrated near the closest grid error. The second grasp approach direction
consists of grasps that, on average, lead to about eight times lower
localization errors.

Figure 6: Error distributions for the first (left) and second (right) grasp
approach directions on round clip using parallel jaw contacts. The red line
represents the median closest grid error. The first grasp approach direction
has more non-unique contacts (sample contacts visualized at left of the plot),
and therefore has higher median error. Contacts in the second grasp approach
direction, on the other hand, are much more unique (sample contacts visualized
at right of the plot).

Comparing Individual Grasps. Some objects are difficult to localize even in
the presence of a pose prior. For these objects, Tac2Pose does not reduce the
median error much beyond the prior on the object pose. These objects are
characterized by large, continuous, regions of non-unique contacts, and their
error distributions tend to be broader, rather than having a discrete number
of modes.

(a) Real RGB tactile images from the sensors with contact masks overlaid for
three grasps on long pencil (top). Distributions over possible contacts on the
object, using parallel jaw information from the real grasp (bottom).

(b) Simulated contact shapes for three grasps on long pencil (top).
Distributions over possible contacts on the object, using parallel jaw
information from the simulated grasp (bottom).

Figure 7: Output pose distributions for three grasps on long pencil, using
real (7(a)) and simulated (7(b)) parallel jaw contacts. Each dot represents a
possible grasp center location when the object is oriented parallel to the
table, and its color represents the likelihood that a given grasp generated
the tactile observation. The true grasp location is shown as a black dot. The
left grasp in both sub-figures is on a non-unique region of the object; many
grasps along the length of the object have high probability (green dots). On
the other hand, the grasps at center and right of both sub-figures are on
unique regions of the object. Only grasps immediately surrounding the true
grasp (black dot) have high probability (green dots). The distributions from
the real and simulated contact masks are qualitatively similar.

Long pencil (Figure 7(a)), for example, suffers from having large regions of
non-unique contacts. Many of the contact poses along the length of the object
result in the same information, and thus on average the match is chosen
essentially at random from a large set of possible contacts. Note that this is
not a limitation of Tac2Pose for tactile localization, but rather results from
the object geometry and the local nature of tactile feedback; most contacts on
long pencil are not sufficient to uniquely determine the object pose.

Even for objects with a large amount of non-uniqueness on average, there can
be regions of the object which create distinctive and unique tactile imprints.
Contacts near either end of long pencil, for example, are easier to localize
accurately. Grasping on these regions leads to better localization. Depending
on the downstream application, it might be beneficial to plan for grasps on an
object that are expected to produce more unique tactile imprints, and thus
lower localization errors. The existence of good grasps is something we can
detect and exploit to avoid large regions of non-uniqueness, and achieve low
localization errors. The distribution over contact poses for selected contacts
on unique and non-unique regions of long pencil is shown in Figure 7(a). The
top row of the figure shows three possible grasps on the object. The bottom
row shows the output of Tac2Pose - a distribution over possible contact poses.
Each dot overlaid on long pencil represents a possible grasp location, and the
color represents the likelihood that a given grasp generated the input tactile
observation. The green dots are grasps with highest likelihood, while the blue
dots are grasps with lowest likelihood. The black dot represents the true
grasp location. Note that for ease of visualization, we show only points that
represent center locations of the grasp when long pencil is oriented
horizontally. This is a small subset (935) of the total number of contacts
(¿65k) Tac2Pose reasons over. For contacts near the middle of the object (left
of Figure 7(a), for example), many grasps along the length of the object have
high probability (green dots). For instance, the left contact in Figure 7(a)
on its own is not enough to uniquely determine the object pose. In fact, the
pose with the highest likelihood results in 77.9mm (0.99 normalized) error for
this case. This aligns with our intuitive understanding that the best match is
chosen essentially at random from a large set of likely contacts (green dots)
along the middle of the object. Cases like this highlight the importance of
outputting meaningful distributions over possible contact poses; the true
contact pose has high likelihood, so Tac2Pose could be used in combination
with information from other sensing modalities, kinematics, or previous
tactile estimates to converge on a unique estimate of the true pose.

For contacts near the tips of the pencil (center and right of Figure 7(a), for
example), only grasps immediately surrounding the correct tip have high
probability. The cases illustrated result in 6mm (0.09 normalized) for the
center contact, and 7mm (0.09 normalized) for the right.

In Figure 7(b), we show distributions over contact poses for simulated
versions of the same three contacts on long pencil. The output distributions
when using simulated contacts are qualitatively similar as when using real
contacts. Simulated contact shapes can therefore be used to detect regions of
non-uniqueness before ever encountering the object. This feature could be used
in a grasp planning framework to avoid regions of non-unique contacts and
promote accurate, unique pose estimation from a single grasp.

###  4.3  Comparison with Scanned Object Models

Table 2: Pose error and normalized pose error (in parenthesis) for Tac2Pose on
manufacturer’s CAD models versus scanned object models. We compare results
with parallel jaw contacts + 10mm pose prior.

|  Scanned Model mm (norm) |  Manufacturer’s Model mm (norm)
---|---|---
Snap Ring |  | 1.6 (0.19) |  | 1.4 (0.17)
Hanger |  | 3.0 (0.39) |  | 2.4 (0.30)
Long Grease |  | 2.5 (0.33) |  | 2.3 (0.33)
Cotter |  | 4.0 (0.51) |  | 2.9 (0.38)
Big Head |  | 6.4 (0.88) |  | 4.9 (0.72)

Tac2Pose relies on apriori knowledge of the object geometry, in the form of a
3D CAD model. In Section 4.2, we evaluate Tac2Pose on manufacturer’s CAD
models. Here, we consider the case where the object model is reconstructed
using a 3D scanner. In particular, we use a SOL 3D Scanner by Scan Dimension,
to generate the scanned object models.

We simulate a grid of possible object contacts and contact poses using the
scanned object model. At test time, we compare contacts observed on the real
object (same real datasets used in Section 4.2) against this imperfect set of
simulated contacts.

We evaluate the accuracy of Tac2Pose when using scanned object models for 5 of
the 20 objects, long grease, snap ring, big head, cotter, and hanger. We
qualitatively compare the scanned and manufacturer’s CAD models for each of
the 5 objects in Table 2. The left column shows the result of the 3D scan, and
the right column shows the true object model.

When evaluating on the scanned object models, we relax the parameter that
controls our confidence in the measurement of the gripper opening to reflect
the fact that we expect the scanned object models to provide noisy information
about the object geometry. The intuition for this decision is the following:
when the object model is known, the gripper opening is a reliable signal of
object pose. Therefore, we only want to assign high likelihood to those
contact poses that match the observed gripper opening closely. When the object
model is reconstructed with a scanner, we cannot be as confident that the
gripper opening we measure corresponds closely to the opening of the true
nearest contact pose in the grid. This is because the gripper opening computed
using the scanned model will be noisy. Therefore, we may want to assign high
likelihood to a contact pose, even if the observed gripper opening does not
match the opening of the contact pose as closely.

Table 2 compares the localization accuracy on scanned models versus
manufacturer’s CAD models for the 5 objects, when using parallel jaw contact
information with a 10mm pose prior. Although globally the geometry of the
object looks similar between the scanned and manufacturer’s CAD models, there
are clear differences at the local level. This ablation of Tac2Pose allows us
to evaluate whether we are still able to achieve significant pose refinement,
even when the local geometries are noisy.

We find that the error is comparable to that when using a manufacturer’s CAD
model for all 5 objects. The normalized error is between 1.01 times (long
grease) and 1.33 times (cotter) higher when using a scanned model compared
with the manufacturer’s CAD model. Cotter is most impacted by using a scanned
object model because the contact non-uniqueness becomes less discrete. When
the object model is known exactly, many of the contacts look unique with a
discrete number of exceptions. Therefore, even a coarse prior on the object
pose leads to precise localization. When the object model is noisy, the
discrete nature of the non-uniqueness is impacted for a fraction of the
contacts, and the median localization error increases.

For snap ring, hanger, long grease, and cotter Tac2Pose is able to improve on
the 10mm prior by a significant amount. For the first three objects, including
parallel jaw information on top of the 10mm prior results in localization
errors twice as low as selecting a pose at random from the filtered
distribution (in Table 2, the normalized pose errors for the scanned objects
are less than 0.5). For cotter, the localization error is nearly twice as low
as selecting a pose at random from the filtered distirbution (in Table 2, the
normalized pose error is 0.51). For these four objects, Tac2Pose is able to
refine the object pose within a small amount of error, even when local
geometry is noisy. For big head, the amount of error is comparable (1.22 times
higher) to that when using a manufacturer’s CAD model. Because big head has a
symmetry-breaking feature around its principle axis, many contacts are
continuously non-unique, and therefore the localization error does not improve
much when incorporating a coarse prior, even when the object model is known.

###  4.4  Comparison with Baselines

Next, we compare Tac2Pose to pose estimation from tactile images with three
baseline methods:

  1. 1.

Pixel: We perform direct pixel comparison between the observed contact mask,
and each of the contact shapes in the grid.

    1. 1.1.

Single Contact: We select the grid shape that has the most pixels in common
with the observed contact mask as the best match. We take the object pose to
be that corresponding to the best match in the grid.

    2. 1.2.

Parallel Jaw: We take into account both tactile images and the gripper opening
during a parallel jaw grasp. To do so, we compare the pair of observed contact
masks, and the observed gripper opening, with triplets of contact masks and
gripper openings from the grid. We sum the pixel error from each of the
contacts, and the normalized error between the observed opening and the grid
opening, to score each of the parallel jaw triplets in the grid. We take the
object pose to be the contact pose corresponding to the triplet with the
lowest score.

  2. 2.

Classification: We formulate the task of pose prediction as a standard
classification problem between the elements of the grid. We train a
convolutional neural network (CNN) based on ResNet-50 to predict a
distribution over the grid elements from a given contact mask. Recall that in
Tac2Pose, the neural network is an encoder that maps contact shapes to vectors
and we obtain a distribution over object poses by comparing the encoding from
an observed contact shape to all the encodings from the contact shapes in the
grid. In contrast, the neural network in the classification baseline learns to
predict the distributions over object poses directly. We train this baseline
by using the same data generation as in Tac2Pose. The inputs are simulated
contacts that we obtained from slightly perturbing a pose from the grid and
rendering its corresponding contact. For each of these new contacts, we
compute its closest grid element and use as training label a vector with
length equal to the number of elements as the grid, where all its values are
zero except the entry corresponding to the index of the closest element, which
has value one. This vector represents a probability distribution that measures
the likelihood of each element of the grid to be the closest to the new
contact. The loss function is the cross-entropy loss between the predicted
likelihood and the classification label. At test time, from an observed
tactile image in the form of a contact mask we obtain a distribution over grid
elements.

  3. 3.

Pose: We train a CNN based on ResNet-50 to regress a nine-element
representation of the object pose based on an observed tactile image. The
training data consists of contact shapes as inputs, and the contact poses that
generated such contacts as labels. Each label is a pose represented by three
translational elements, and six rotational elements that can be mapped into a
rotation matrix. In comparison with quaternions, our 6D representation of
rotations is continuous (meaning that similar orientations are close together
in 6D representation), and therefore better suited to regression. We construct
the last column of the rotation matrix from the 6D representation by applying
Gram-Schmidt as a post-process on the first two columns, and taking the third
column as the cross product of the first two (Zhou et al., 2018). The loss
function is the mean squared error between the output and true nine-element
pose. This method for pose prediction does not rely on matching to elements on
the grid, but rather predicts the pose of the object directly using supervised
regression.

Both Tac2Pose and pixel are compatible with using parallel jaw information,
while classification and pose only work with single contacts.

We evaluate the performance of Tac2Pose compared to baselines for 5 of the 20
objects: long grease, snap ring, big head, cotter, and hanger. We select these
5 objects to cover a range of difficulty in terms of grid size and degrees of
symmetry and non-uniqueness.

Baseline comparison with simulated data.

Table 3: Pose error and normalized pose error (in parenthesis) for Tac2Pose versus baseline methods for a range of grid sizes, using 100 randomly simulated contacts per grid. Columns labelled SC are evaluated with a single contact, while columns labelled PJ use parallel jaw information. | Tac2Pose | Pixel | Classification | Pose
---|---|---|---|---
|  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  SC mm (norm)
Long Grease |  |
Mini One Face | 2.3 (0.09) | 2.3 (0.09) | 2.7 (0.11) | 2.4 (0.09) | 2.1 (0.08) | 16.1 (0.57)
Bigger Mini One Face | 18.1 (0.51) | 11.5 (0.31) | 27.1 (0.76) | 16.6 (0.45) | 34.8 (0.99) | 25.2 (0.71)
One Face | 1.5 (0.04) | 1.3 (0.04) | 12.0 (0.35) | 4.8 (0.14) | 32.7 (0.96) | 7.5 (0.22)
Full | 1.3 (0.04) | 1.3 (0.04) | 5.0 (0.14) | 2.2 (0.06) | 33.8 (0.92) | 21.6 (0.59)
Snap Ring |  |
Mini One Face | 2.0 (0.16) | 2.0 (0.16) | 2.1 (0.14) | 2.1 (0.15) | 1.2 (0.11) | 8.5 (0.76)
Bigger Mini One Face | 3.1 (0.20) | 3.1 (0.22) | 6.9 (0.44) | 6.9 (0.45) | 2.6 (0.16) | 10.1 (0.63)
Full | 1.0 (0.06) | 1.0 (0.07) | 1.2 (0.08) | 1.2 (0.08) | 7.2 (0.49) | 4.9 (0.33)
Big Head |  |
Mini One Face | 2.5 (0.09) | 2.3 (0.09) | 3.5 (0.13) | 3.2 (0.12) | 13.4 (0.50) | 21.5 (0.81)
Bigger Mini One Face | 37.5 (0.83) | 14.6 (0.34) | 37.9 (0.89) | 31.1 (0.77) | 43.4 (1.04) | 30.5 (0.73)
One Face | 1.7 (0.05) | 1.6 (0.05) | 4.8 (0.15) | 4.7 (0.14) | 31.4 (0.97) | 15.6 (0.48)
Full | 1.6 (0.04) | 1.6 (0.04) | 9.9 (0.23) | 9.9 (0.23) | 39.6 (0.97) | 13.8 (0.34)
Cotter |  |
Mini One Face | 2.5 (0.08) | 2.5 (0.10) | 15.5 (0.52) | 17.7 (0.60) | 2.1 (0.07) | 19.6 (0.71)
Bigger Mini One Face | 8.3 (0.21) | 8.8 (0.21) | 34.0 (0.87) | 34.0 (0.87) | 15.2 (0.39) | 22.2 (0.56)
One Face | 1.3 (0.03) | 1.4 (0.03) | 5.8 (0.15) | 6.2 (0.16) | 35.2 (0.90) | 8.5 (0.22)
Full | 13.8 (0.34) | 8.9 (0.22) | 17.1 (0.44) | 18.2 (0.42) | 34.9 (0.88) | 18.2 (0.46)
Hanger |  |
Mini One Face | 2.3 (0.08) | 2.2 (0.09) | 2.5 (0.09) | 2.6 (0.10) | 2.8 (0.12) | 9.8 (0.42)
Bigger Mini One Face | 9.1 (0.24) | 5.1 (0.14) | 25.2 (0.71) | 25.1 (0.71) | 21.1 (0.60) | 19.6 (0.56)
One Face | 1.5 (0.04) | 1.4 (0.04) | 4.0 (0.11) | 3.3 (0.09) | 30.9 (0.88) | 9.0 (0.26)
Full | 2.6 (0.08) | 1.9 (0.06) | 6.6 (0.19) | 4.2 (0.12) | 34.2 (0.95) | 6.4 (0.18)

(a) Results for 100 randomly sampled simulated contacts across four hanger
grid sizes. Mini One Face (MOF) is the smallest and simplest grid, while Full
is the largest and most complex.

(b) Results on the hanger dataset (Section 4.2) with over 100 contacts, using
the full-sized grid for all methods. We evaluate all methods on the same real
(left) and simulated (right) contact shapes.

Figure 8: Normalized pose error for the object hanger using tactile matching
(Tac2Pose) and three baseline methods. We compare the performance of each
method on simulated contacts from different grid sizes in (8(a)), and on real
vs. simulated versions of the contacts in the hanger dataset (Section 4.2) in
(8(b)).

We first evaluate each method on sets of 100 simulated contacts per object. We
sample 100 contact poses from the object’s grid, add noise to each pose, and
render the contacts corresponding to the new poses.

For each of the 5 baseline objects, we create three smaller, simpler, grids in
addition to the original. The reduced grids serve to test each method’s
ability to scale to different problem sizes and types of complexity.

Recall that the original grids are determined by 4 spatial coordinates (grasp
approach direction, x, y, $\theta$) as illustrated in Figure 2. They have
angular resolution of 6 degrees, and x, y translational resolution of 2.5mm.
The size of the original grids for the baseline objects ranges from 8.1k to
91.5k poses. The reduced grids are modified as follows:

  1. 1.

Mini One Face: We include only one grasp approach direction, and one angle. We
consider translations in x, y with 5mm resolution. This grid is much smaller
than the original (ranging from 29 to 124 poses, depending on the object), and
is effectively 2D (x, y).

  2. 2.

Bigger Mini One Face: We include only one grasp approach direction, and 10
angles (angular resolution of 36 degrees). We consider translations in x, y
with 5mm resolution. This grid is smaller than the original, and ranges from
331 to 1137 poses, depending on the object. It is effectively 3D (x, y,
$\theta$).

  3. 3.

One Face: We include only one grasp approach direction, with full angular (6
degrees) and translational (2.5mm) resolution. This grid ranges in size from
4.8k to 26.8k poses, depending on the object. It is effectively 3D (x, y,
$\theta$). For snap ring, which only has one grasp approach direction even in
the full sized grid, this grid is equivalent to the original and is therefore
omitted.

Comparing results between the reduced grids can provide insight into each
method’s sensitivity to problem size, resolution, and complexity. For example,
comparing mini one face and one face can indicate the sensitivity of each
method to grid size and resolution. Comparing one face and the original grid
(called full from here on out), which have the same resolution but cover
different faces, provides insight into how each method handles complexity in
the form of 3D rotations. Mini one face and bigger mini one face have the same
translational resolution, but bigger mini one face varies the angle in the
plane of the grasp. Comparing results on these two grids provides insight into
how incorporating rotations in the plane of the grasp impacts the accuracy of
each method.

We first conduct experiments using only simulated data. The full set of
results on the 5 baseline objects are listed in Table 3. We consider the
results for hanger, visualized in Figure 8(a), for simulated contacts from
mini one face, bigger mini one face, one face, and full. The trends we note
for this object are representative of the trends for the remaining objects.

The performance of classification (green bar) is inversely correlated with
grid size; its performance is comparable to Tac2Pose only for mini one face
(MOF in Figure 8(a)), which has 124 contacts. The normalized error is 0.12,
compared with 0.08 for the single contact case of Tac2Pose. This indicates
that although classification could be effective for very small grid sizes, the
grids quickly become too large. Even with bigger mini one face, which contains
794 transformations, classification has a normalized error of 0.6, making it
more than six times worse than Tac2Pose (single contact), which has 0.09
normalized error.

The classification baseline struggles with larger grid sizes because it needs
to provide a classification over all elements in the grid, which often
contains thousands of elements (the full grid for hanger, for example, has
more than 90k contact poses). In comparison with the classification baseline,
Tac2Pose scales better to large grid sizes because it learns to generate an
embedding space based on the distance between contact poses. The encoder in
Tac2Pose learns to push distant contacts away in embedding space, and
therefore when comparing a new encoding to all the encodings in the grid, this
direct comparison scales better than classification when the set of contacts
in the grid is large. Instead, in classification the learned NN needs to
encapsulate all the information relevant about all poses solely in the NN
weights. Another way to think about the difference is to consider Tac2Pose as
describing each contact in the grid with 1000 parameters (the encoder),
whereas with classification, each contact in the grid is described by a single
parameter (0 or 1).

The pose baseline (purple bar), on the other hand, is most comparable to
Tac2Pose for the larger grids with high resolution. For one face (OF in Figure
8(a)), which covers one object face with the same translational and rotational
resolution as the full grid, the normalized pose error using pose is 0.26,
compared with 0.04 for the single contact version of Tac2Pose. For the full
grid, the normalized error for pose and the single contact version of Tac2Pose
is 0.18 and 0.08, respectively. In the absence of contact shape noise
(simulated contacts), Tac2Pose is about 2.3 times better on the full grid.
Recall that the pose baseline does not match observed contacts to contact
poses on the grid, but instead regresses the corresponding contact pose
directly. In pose, the learned NN needs to encapsulate all information
relevant about all poses solely in the NN weights, as it doesn’t have access
to embeddings it can compare against.

The pixel baseline, for both single contact and parallel jaw contact cases,
performs most comparably to Tac2Pose for the mini one face grid. The mini one
face grid is small and simple enough (one angle, one face) that all methods
(excluding pose) perform comparably. For one face and full, Tac2Pose
outperforms pixel by about 2-3 times for both single contact and parallel jaw
cases, which is still relatively comparable. This implies that for pixel is
most successful for high resolution grids. The performance of the pixel
baseline method degrades most when the grid is low resolution, particularly
when the grid contains object rotations. Bigger mini one face (shown as BMOF
in the figure) has 0.71 median normalized error, compared with 0.24 for
Tac2Pose (single contact). With parallel jaw contacts, the pixel baseline has
0.71 median normalized error, compared with 0.14 forTac2Pose. Tac2Pose
outperforms pixel by nearly 3 times in the single contact case, and more than
5 times in the parallel jaw case. Furthermore, pixel (both single contact and
parallel jaw contact cases) is the least effective method when the grid is low
resolution but complex.

When the grid resolution is low, the nearest grid match will be farther (in
pixel distance) from an observed contact than when the grid resolution is
high. This is particularly harmful for pixel when rotations are introduced,
because a rotated version of a remote contact may, by chance, have more pixel
overlap with the observed contact than the true closest match. This is much
less likely to occur for high resolution grids, in which there exists a very
close match on the grid to any observed contact. When the grid is simple (does
not contain planar or 3D rotations) pixel can be successful even when the grid
resolution is low (as is the case for mini one face) because the best match on
the grid is still likely to have the smallest pixel distance. If the grid is
sufficiently complex, it must also be high resolution for pixel to be
relatively successful (as is the case for one face and full). It is worth
noting that a drawback of pixel, particularly with high resolution grids, is
the execution time. In order to choose the best match, an observed contact is
compared with contact shapes corresponding to every contact pose in the grid.
Execution time is doubled when considering parallel jaw information, because
two contacts are compared for each contact pose. Pixel is therefore the
slowest method we evaluated by a significant margin. In practice, matching a
contact to a high resolution grid in real-time using pixel is likely
infeasible. Instead, Tac2Pose only compares low-dimensional embeddings
resulting in at least an order of magnitude speed-up.

Finally, Tac2Pose performs consistently for both single and parallel jaw
simulated contacts for all grid sizes for hanger. Tac2Pose also outperforms
all baselines for all grid sizes. The parallel jaw case of Tac2Pose slightly
outperforms the single contact case for all hanger grid sizes, except mini one
face, in which the single contact case outperforms parallel jaw by 0.08 versus
0.09 normalized error. Tac2Pose (both single contact and parallel jaw) also
performs slightly better for high resolution grids (one face and full).

Baseline comparison with real data.

Table 4: Pose error and normalized pose error (in parenthesis) for Tac2Pose versus baseline methods, on real datasets. Columns labelled SC are evaluated with a single contact, while columns labelled PJ use parallel jaw information. | Tac2Pose | Pixel | Classification | Pose
---|---|---|---|---
|  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  SC mm (norm)
Long Grease |  | 26.6 (0.76) | 3.3 (0.09) | 32.8 (0.93) | 6.0 (0.17) | 33.3 (0.95) | 25.3 (0.72)
Snap Ring |  | 1.5 (0.10) | 1.4 (0.10) | 5.6 (0.39) | 2.2 (0.15) | 6.0 (0.42) | 5.9 (0.41)
Big Head |  | 7.8 (0.20) | 6.1 (0.16) | 27.6 (0.70) | 11.7 (0.30) | 35.0 (0.89) | 33.8 (0.86)
Cotter |  | 19.0 (0.49) | 19.6 (0.51) | 31.5 (0.81) | 36.7 (0.95) | 35.8 (0.93) | 38.1 (0.99)
Hanger |  | 6.6 (0.19) | 2.6 (0.07) | 31.3 (0.90) | 20.5 (0.59) | 34.2 (0.98) | 18.3 (0.53)
Table 5: Pose error and normalized pose error (in parenthesis) for Tac2Pose versus baseline methods, on simulated versions of the real contacts used in Table 4. Columns labelled SC are evaluated with a single contact, while columns labelled PJ use parallel jaw information. | Tac2Pose | Pixel | Classification | Pose
---|---|---|---|---
|  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  PJ mm (norm) |  SC mm (norm) |  SC mm (norm)
Long Grease |  | 1.1 (0.03) | 1.1 (0.03) | 1.8 (0.05) | 1.7 (0.05) | 33.0 (0.94) | 17.0 (0.48)
Snap Ring |  | 1.0 (0.07) | 1.0 (0.07) | 1.2 (0.08) | 1.2 (0.08) | 6.0 (0.41) | 4.5 (0.31)
Big Head |  | 3.9 (0.10) | 3.3 (0.08) | 8.7 (0.22) | 9.6 (0.24) | 34.0 (0.87) | 13.8 (0.35)
Cotter |  | 1.3 (0.03) | 1.3 (0.03) | 17.9 (0.46) | 17.7 (0.46) | 35.0 (0.91) | 13.2 (0.34)
Hanger |  | 1.2 (0.03) | 1.2 (0.03) | 5.1 (0.15) | 2.4 (0.07) | 33.6 (0.96) | 6.0 (0.17)

We next compare Tac2Pose against the three baselines, using the real datasets
evaluated in Section 4.2 and full sized grids. The full set of results are
listed in Table 4. We also evaluate simulated versions of the contacts in the
real datasets, to assess each method’s sensitivity to noise in the contact
shapes. The results on simulated versions of the contacts in the real datasets
are listed in Table 5. We discuss in detail the results for hanger, visualized
in Figure 8(b). The trends for this object are representative of the trends we
see overall.

For real data, and full-sized grids, the performance classification (green
bar), and pixel with a single contact (dark orange bar) are similar to
selecting a contact pose randomly from the grid. The median normalized error
is 0.98 for classification, and 0.90 for pixel with a single contact. Pixel
with parallel jaw contacts and pose perform better, and have errors almost two
times lower than selecting a pose at random from the grid; the median
normalized error is 0.59 for pixel with parallel jaw contacts, and 0.53 for
pose. Tac2Pose (dark blue bar for single contact, light blue bar for parallel
jaw contacts) significantly outperforms all baselines. The median normalized
error for the single contact case is 0.19, and for the parallel jaw contact
case is 0.07. Tac2Pose outperforms the next best baseline (pose) by nearly
three times for the single contact case, and nearly eight times for the
parallel jaw contact case.

On simulated versions of the same contacts, pose and pixel (single contact and
parallel jaw contact cases) both perform better than they do with real
contacts. The median normalized error for pose shrinks by over three times,
dropping to 0.17 with simulated data. Similarly, for pixel, the normalized
error in simulation shrinks six times for single contact, and over eight times
for parallel jaw contacts, dropping to 0.15 and 0.07, respectively. This
indicates the sensitivity of both pose and pixel to noise in the contact
shapes. Tac2Pose, in comparison, is more robust to contact shape noise,
particularly in the parallel jaw contact case. With simulated contacts, the
median normalized error for the parallel jaw case of Tac2Pose shrinks about
two times, dropping to 0.03. Furthermore, despite the better performance of
pose and pixel on simulated contacts compared with real contacts, Tac2Pose
yields the best performance on simulated contacts as well. With simulated
contacts, Tac2Pose is more than twice as good as the next best baseline (pixel
with parallel jaw contacts).

The primary reason for pixel’s sensitivity to noise in the contact shapes is
that it only evaluates the exact 2D location of the contact on the sensor.
Tac2Pose, in comparison, is trained to match contact shapes resulting from
slight perturbations of grid poses to the closest match on the grid. This
makesTac2Pose more robust to slight discrepancies between observed and grid
contact shapes.

The sensitivity of pose to contact shape noise has to do, instead, with the
method’s ability to generalize to out of distribution data in the form of real
contact shapes. Both pose and Tac2Pose are trained on simulated versions of
contact shapes, but Tac2Pose generalizes to real contact shapes much better.
Tac2Pose computes the likelihood of an observed contact matching to a discrete
set of simulated shapes. Therefore, the match does not need to be exact, just
better than other possible options, for the localization to be accurate. The
structure imposed by the grid therefore improves the ability of Tac2Pose to
generalize to real, noisy contact shapes. Because pose regresses object poses
directly, without the structure of the grid, its performance is more brittle
and degrades when using real contacts.

The normalized error of classification, even with simulated contacts, is 0.96,
remaining similar to selecting a pose randomly from the grid. The size of the
grid is too large to be handled with classification (as addressed in Section
4.4), so it is not possible to comment on the impact of noisy contacts.

For the 5 objects we evaluate (long grease, snap ring, big head, hanger, and
cotter), Tac2Pose performs significantly better than random (here, we define
significantly better as more than twice as good, which can be identified as a
median normalized error less than 0.5 in Table 4) for 4/5 objects in both the
single contact and parallel jaw contact cases. In the parallel jaw contact
case, the fifth object, cotter, has median normalized error of 0.51, which is
just below twice as a good as random. In comparison, pose, classification, and
pixel with a single contact are only significantly better than random for 1/5
objects. Pixel with parallel jaw contacts is significantly better than random
for 3/5 objects. Furthermore, with a single contact, Tac2Pose outperforms all
baselines for 4/5 objects. Tac2Pose is between 1.5 and 4 times better than the
next best baseline for all objects except for long grease, which performs
marginally worse than pose in the single contact case (0.72 median normalized
error for pose, compared with 0.76 for Tac2Pose). With parallel jaw contacts,
Tac2Pose outperforms pixel, which is the only other method compatible with
parallel jaw contacts, with by between 1.5 and 9 times, depending on the
object.

A final observation is that for some objects, parallel jaw contact versions of
pixel and Tac2Pose are more robust to contact shape noise than single contact
versions. We consider long grease (results listed in Table 4) as an example.
Tac2Pose with a single real contact has 0.76 median normalized error, compared
with 0.09 for the parallel jaw contact case. This means that the localization
performance improves around eight times when using parallel jaw contacts.
Similarly, pixel with a single contact has 0.93 median normalized error,
compared with 0.17 median normalized error with parallel jaw contacts; the
localization performance is about 5.5 times better in the parallel jaw contact
case. We also consider the performance of Tac2Pose, and pixel, on simulated
contacts to evaluate the impact of contact shape noise on the discrepancy
between single and parallel jaw contact cases. With simulated contacts, the
localization performance is the same for Tac2Pose for both a single contact
and parallel jaw contacts; the median normalized error is 0.03 in both cases.
The outcome is similar for pixel; the median normalized error is 0.05 in both
the single contact and parallel jaw contact cases. The consistent localization
performance between methods and contact configurations when using simulated
contacts indicates that contact shape noise is the differentiating factor
between single contact and parallel jaw contact cases when using real
contacts. This means that robustness to noise is a key advantage of using
parallel jaw contacts over a single contact, for both methods. Both Tac2Pose
and pixel have similar performance in the parallel jaw case with real contacts
as they do with idealized (simulated) contacts; it is 3 (Tac2Pose ) to 3.5
(pixel) times easier to localize simulated contacts than real contacts. The
discrepancy between simulated and real results is much more pronounced in the
single contact case of both methods. This implies that the sim-to-real gap can
be bridged, in part, by the inclusion of parallel jaw information.

We conclude this section with some remarks about the challenge of estimating
an object’s pose from a single noisy contact. Tac2Pose outperforms all
baselines on 4/5 objects with a single contact, and all objects with parallel
jaw contacts. Ultimately, though, even Tac2Pose struggles to perform
significantly better than selecting a pose at random from the grid with a
single contact for 1/5 objects (where significantly better than random is
defined as more than twice as good). Estimating the pose of an object from a
single, noisy, often non-unique contact is challenging. Therefore, perhaps the
most important feature of Tac2Pose is that it outputs meaningful pose
distributions over possible object poses (Figure 7(a)). This creates a natural
framework for incorporating constraints from additional contacts, measurements
of the robot state (such as the gripper opening), information from additional
sensing modalities, or even previous tactile estimates of the object pose.

##  5 Discussion

This paper presents an approach to tactile pose estimation for objects with
known geometry. Tac2Pose relies on learning an embedding completely in
simulation that facilitates comparing real and simulated contact shapes. We
can reconstruct contact shapes with high fidelity, using images from the real
sensor and in simulation. We compare the embeddings of an observed contact
shape with those of a precomputed dense set of simulated contact shapes to
obtain the distribution over a dense set of possible object poses. The
approach therefore allows to reason over pose distributions and to handle
additional pose constraints.

We evaluate Tac2Pose on real grasp datasets for 20 objects, and report the
accuracy for three ablations of Tac2Pose, corresponding to increasing amounts
of information from a given grasp. First, we evaluate the accuracy using a
single tactile image which corresponds to the case where only one contact is
available to the algorithm. Second, we consider the case where two tactile
images (corresponding to a parallel jaw grasp on the object) and the gripper
opening are available. Third, we evaluate the accuracy after filtering the
distribution obtained using parallel jaw information with a coarse prior on
the object pose, approximating the case where information from an additional
sensing modality (e.g. vision) is available.

We find that the amount of information needed to localize an arbitrary grasp
on an object accurately is highly dependent on the object geometry. Of the 20
objects we evaluate, 9 can be localized accurately with a single contact. For
these objects, more than half of arbitrary grasps on the object can be
localized accurately. The objects that can be localized accurately with a
single contact tend to be smaller, and have significant regions of unique
contacts. When including parallel jaw information, the number of objects that
can be localized accurately using the best match from Tac2Pose increases to
12. The objects that can be localized accurately after including parallel jaw
information tend to have pseudosymmetrical features which can be disambiguated
with a second contact, or vary significantly in width across the length of the
object, which makes the gripper opening a discriminative source of
information. After filtering the parallel jaw pose distribution with a coarse
10mm prior on the object pose, the number of objects that can be localized
accurately increases to 16. The objects that can be localized accurately after
including a coarse prior on the object pose tend to be larger, and have
discrete nonuninquess (features that are unique, with a discrete number of
exceptions on remote regions of the object).

There are, however, four objects which cannot be localized accurately on
average even after filtering the parallel jaw distribution with a 10mm prior
and selecting the resulting most likely object pose. These objects are large,
and have significant, continuous regions of non-unique contacts. The majority
of contacts on such objects do not provide enough information to uniquely
determine their pose.

Even objects which can be, on average, localized accurately from an arbitrary
grasp have regions of non-unique contacts that do not provide enough
information to uniquely determine the object pose. It is therefore important
to evaluate the localization accuracy of specific grasps on objects, as well
as arbitrary grasps. To this end, we compare the localization accuracy
separated out by grasp approach direction, and the localization accuracy of
individual grasps on selected objects. We find that many objects have one
direction which is significantly easier to localize (half as much error or
less) than others. By examining individual grasps on the object long pencil,
we find that even objects with large, continuous regions of non-unique
contacts have regions that are more unique and easier to localize.

Both of these breakdowns (sets broken out by grasp approach direction, or
individual grasps) could ultimately be leveraged in a grasp planning
framework, in which grasps that are easier to localize are specifically
targeted. In practice, grasps within a given grasp approach direction could be
targeted with a very coarse prior on the object pose, while a specific
individual grasp could be targeted with somewhat finer information about the
object pose. Because Tac2Pose is trained entirely in simulation, it is
possible to evaluate which grasps lead to lower localization error in advance
of encountering the object. For instance, the output pose distributions we
obtain using simulated and real contact shapes on long pencil are
qualitatively similar (Figure 7(a) and 7(b)). Simulated contacts can therefore
be used to identify unique and non-unique candidate grasps. This feature, too,
could be important to leverage in a manipulation planning framework.

Our approach also assumes we have access to accurate geometric models of
objects. We demonstrate that Tac2Pose can be effectively combined with object
models reconstructed from a 3D scanner. A key feature of Tac2Pose is the
ability to refine coarse estimates of object pose with high resolution tactile
information. We find that moderate shape noise does not significantly
compromise that ability for any of the 5 objects we evaluate.

Future work could learn embeddings that are even more robust to shape
uncertainty by corrupting the contact shapes with noise during training. This
would not inherently improve the localization accuracy when considering only
information from a single grasp, but could yield distributions that are more
representative of our true confidence in the estimate. These distributions
could be combined effectively with other information (e.g. pose priors,
additional tactile measurements, or dynamics) to converge on a unique estimate
of the object pose.

We demonstrate the advantages of Tac2Pose compared with baseline methods for
estimating object pose from tactile images. Compared with a standard
classification approach, direct pose regression, and direct pixel comparison,
Tac2Pose scales better to larger, and more complex problems (e.g. regressing
6D object pose rather than simply translation or rotation in plane) and is
more robust to contact shape noise.

In summary, we demonstrate the effectiveness of Tac2Pose at pose estimation
for 20 objects with information gathered from a single, real grasp on the
object. We consider known and reconstructed object shapes, and show that in
both cases Tac2Pose outputs meaningful distributions over object pose that can
be maximized directly, or combined with additional information to converge on
a unique estimate of object pose. While Tac2Pose is trained entirely in
simulation, it is robust to real and noisy contact shapes arising from both
sensor noise and object shape noise. This robustness stands in contrast
baseline methods, which are more sensitive to contact shape noise. This
suggests that matching estimated contact shapes to a dense precomputed set
opens the door to moving many computations into simulation, without loss of
robustness, and improving how robots learn to perceive and manipulate their
environment.

## Acknowledgments

We thank Ian Taylor and Siyuan Song for helping with the real setup. We also
thank Ferran Alet for providing detailed insights and reviewing the paper.

## References

  * [1] Pyrender.  https://github.com/mmatl/pyrender.
  * Allen et al. [1999] Peter K. Allen, Andrew T. Miller, Paul Y. Oh, and Brian S. Leibowitz.  Integration of vision, force and tactile sensing for grasping.  _Int. J. Intelligent Machines_ , 4:129–149, 1999.
  * Bauza et al. [2019] Maria Bauza, Oleguer Canal, and Alberto Rodriguez.  Tactile mapping and localization from high-resolution tactile imprints.  In _International Conference on Robotics and Automation (ICRA)_. IEEE, 2019.
  * Bauza et al. [2020] Maria Bauza, Eric Valls, Bryan Lim, Theo Sechopoulos, and Alberto Rodriguez.  Tactile object pose estimation from the first touch with geometric contact rendering.  _CoRR_ , abs/2012.05205, 2020.  URL https://arxiv.org/abs/2012.05205.
  * Bimbo et al. [2016] J. Bimbo, S. Luo, K. Althoefer, and H. Liu.  In-hand object pose estimation using covariance-based tactile to geometry matching.  _IEEE Robotics and Automation Letters_ , 2016.
  * Bimbo et al. [2015] Joao Bimbo, Petar Kormushev, Kaspar Althoefer, and Hongbin Liu.  Global estimation of an object’s pose using tactile sensing.  _Advanced Robotics_ , 29(5):363–374, 2015.
  * Chalon et al. [2013] Maxime Chalon, Jens Reinecke, and Martin Pfanne.  Online in-hand object localization.  In _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 2977–2984. IEEE, 2013.
  * Chebotar et al. [2014] Yevgen Chebotar, Oliver Kroemer, and Jan Peters.  Learning robot tactile sensing for object manipulation.  In _International Conference on Intelligent Robots and Systems_. IEEE, 2014.
  * Corcoran [2010] Craig Corcoran.  Tracking object pose and shape during robot manipulation based on tactile information.  In _International Conference on Robotics and Automation (ICRA)_ , 2010\.
  * Deng et al. [2019] Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox.  Poserbpf: A rao-blackwellized particle filter for 6d object pose tracking.  _CoRR_ , abs/1905.09304, 2019.  URL http://arxiv.org/abs/1905.09304.
  * Falco et al. [2017] Pietro Falco, Shuang Lu, Andrea Cirillo, Ciro Natale, Salvatore Pirozzi, and Dongheui Lee.  Cross-modal visuo-tactile object recognition using robotic active exploration.  In _Robotics and Automation (ICRA), 2017 IEEE International Conference on_ , pages 5273–5280. IEEE, 2017.
  * Gao and Tedrake [2019] Wei Gao and Russ Tedrake.  Filterreg: Robust and efficient probabilistic point-set registration using gaussian filter and twist parameterization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 11095–11104, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * He et al. [2019] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  _arXiv preprint arXiv:1911.05722_ , 2019.
  * Hogan et al. [2018] Francois Robert Hogan, Maria Bauzá, Oleguer Canal, Elliott Donlon, and Alberto Rodriguez.  Tactile regrasp: Grasp adjustments via simulated tactile transformations.  _CoRR_ , abs/1803.01940, 2018.
  * Ilonen et al. [2014] Jarmo Ilonen, Jeannette Bohg, and Ville Kyrki.  Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing.  _The International Journal of Robotics Research_ , 2014.
  * Isola et al. [2018] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks, 2018\.
  * Izatt et al. [2017] Gregory Izatt, Geronimo Mirano, Edward Adelson, and Russ Tedrake.  Tracking objects with point clouds from vision and touch.  In _International Conference on Robotics and Automation_ , 2017.
  * Javdani et al. [2013] Shervin Javdani, Matthew Klingensmith, J Andrew Bagnell, Nancy S Pollard, and Siddhartha S Srinivasa.  Efficient touch based localization through submodularity.  In _International Conference on Robotics and Automation (ICRA)_. IEEE, 2013.
  * Jeon and Kim [2020] MyungHwan Jeon and Ayoung Kim.  Prima6d: Rotational primitive reconstruction for enhanced and robust 6d pose estimation.  _IEEE Robotics and Automation Letters_ , PP:1–1, 06 2020\.
  * Kim and Rodriguez [2021] Sangwoon Kim and Alberto Rodriguez.  Active extrinsic contact sensing: Application to general peg-in-hole insertion, 2021.
  * Koval et al. [2017] M. C. Koval, M. Klingensmith, S. S. Srinivasa, N. S. Pollard, and M. Kaess.  The manifold particle filter for state estimation on high-dimensional implicit manifolds.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4673–4680, May 2017.
  * Koval et al. [2015] Michael C Koval, Nancy S Pollard, and Siddhartha S Srinivasa.  Pose estimation for planar contact manipulation with manifold particle filters.  _The International Journal of Robotics Research_ , 2015.
  * Kuppuswamy et al. [2019] Naveen Kuppuswamy, Alejandro Castro, Calder Phillips-Grafflin, Alex Alspach, and Russ Tedrake.  Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors.  _IEEE Robotics and Automation Letters_ , PP:1–1, 12 2019\.  10.1109/LRA.2019.2961050.
  * Labbé et al. [2020] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  _CoRR_ , abs/2008.08465, 2020.  URL https://arxiv.org/abs/2008.08465.
  * Lepora et al. [2019] Nathan F Lepora, Alex Church, Conrad De Kerckhove, Raia Hadsell, and John Lloyd.  From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor.  _IEEE Robotics and Automation Letters_ , 4(2):2101–2107, 2019.
  * Li et al. [2014] Rui Li, Robert Platt, Wenzhen Yuan, Andreas ten Pas, Nathan Roscup, Mandayam A. Srinivasan, and Edward Adelson.  Localization and manipulation of small parts using gelsight tactile sensing.  In _Intelligent Robots and Systems (IROS), 2014 IEEE/RSJ International Conference on._ , 2014.
  * Li et al. [2018] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox.  Deepim: Deep iterative matching for 6d pose estimation.  _CoRR_ , abs/1804.00175, 2018.  URL http://arxiv.org/abs/1804.00175.
  * Luo et al. [2017a] Shan Luo, Joao Bimbo, Ravinder Dahiya, and Hongbin Liu.  Robotic tactile perception of object properties: A review.  _Mechatronics_ , 48:54–67, 2017a.
  * Luo et al. [2017b] Shan Luo, Wenxuan Mou, Kaspar Althoefer, and Hongbin Liu.  Localizing the object contact through matching tactile features with visual map.  _CoRR_ , abs/1708.04441, 2017b.
  * Milan et al. [2018] Anton Milan, Trung Pham, K Vijay, Douglas Morrison, Adam W Tow, L Liu, J Erskine, R Grinover, A Gurman, T Hunn, N Kelly-Boxall, D Lee, M McTaggart, G Rallos, A Razjigaev, T Rowntree, T Shen, R Smith, S Wade-McCue, Z Zhuang, C Lehnert, G Lin, I Reid, P Corke, and J Leitner.  Semantic segmentation from limited training data.  In _International Conference on Robotics and Automation (ICRA)_. IEEE, 2018.
  * Petrovskaya and Khatib [2011] Anna Petrovskaya and Oussama Khatib.  Global localization of objects via touch.  _IEEE Transactions on Robotics_ , 27(3):569–585, 2011.
  * Pezzementi et al. [2011] Zachary Pezzementi, Caitlin Reyda, and Gregory D Hager.  Object mapping, recognition, and localization from tactile geometry.  In _International Conference on Robotics and Automation (ICRA)_ , 2011\.
  * Platt et al. [2011] Robert Platt, Frank Permenter, and Joseph Pfeiffer.  Using bayesian filtering to localize flexible materials during manipulation.  _IEEE Transactions on Robotics_ , 27(3):586–598, 2011.
  * Roşca et al. [2014] D Roşca, A Morawiec, and M De Graef.  A new method of constructing a grid in the space of 3d rotations and its applications to texture analysis.  _Modelling and Simulation in Materials Science and Engineering_ , 22(7):075013, 2014.
  * Saund et al. [2017] Brad Saund, Shiyuan Chen, and Reid Simmons.  Touch based localization of parts for high precision manufacturing.  In _International Conference on Robotics and Automation (ICRA)_. IEEE, 2017.
  * Schaeffer and Okamura [2003] Monika A Schaeffer and Allison M Okamura.  Methods for intelligent localization and mapping during haptic exploration.  In _International Conference on Systems, Man and Cybernetics._ IEEE, 2003.
  * Schwarz et al. [2018] Max Schwarz, Christian Lenz, Germán Martín García, Seongyong Koo, Arul Selvam Periyasamy, Michael Schreiber, and Sven Behnke.  Fast object learning and dual-arm coordination for cluttered stowing, picking, and packing.  In _International Conference on Robotics and Automation (ICRA)_. IEEE, 2018.
  * Sodhi et al. [2020] Paloma Sodhi, Michael Kaess, Mustafa Mukadam, and Stuart Anderson.  Learning tactile models for factor graph-based state estimation.  _CoRR_ , abs/2012.03768, 2020.  URL https://arxiv.org/abs/2012.03768.
  * Sodhi et al. [2021] Paloma Sodhi, Michael Kaess, Mustafa Mukadam, and Stuart Anderson.  Patchgraph: In-hand tactile tracking with learned surface normals.  _CoRR_ , abs/2111.07524, 2021.  URL https://arxiv.org/abs/2111.07524.
  * Sundermeyer et al. [2018] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel.  Implicit 3d orientation learning for 6d object detection from rgb images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 699–715, 2018.
  * Taylor et al. [2021] Ian Taylor, Siyuan Dong, and Alberto Rodriguez.  Gelslim3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger.  _CoRR_ , abs/2103.12269, 2021.  URL https://arxiv.org/abs/2103.12269.
  * Tian et al. [2019] Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, and Sergey Levine.  Manipulation by feel: Touch-based control with deep predictive models.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 818–824. IEEE, 2019.
  * Wang et al. [2018] S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson.  3D Shape Perception from Monocular Vision, Touch, and Shape Priors.  _ArXiv e-prints_ , August 2018.
  * Yershova et al. [2010] Anna Yershova, Swati Jain, Steven M Lavalle, and Julie C Mitchell.  Generating uniform incremental grids on so (3) using the hopf fibration.  _The International journal of robotics research_ , 2010.
  * Yu and Rodriguez [2018] Kuan-Ting Yu and Alberto Rodriguez.  Realtime state estimation with tactile and visual sensing. application to planar manipulation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 7778–7785. IEEE, 2018.
  * Zeng et al. [2018] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois Robert Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, Ian J. Taylor, Weber Liu, Thomas A. Funkhouser, and Alberto Rodriguez.  Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2018.
  * Zhou et al. [2018] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  _CoRR_ , abs/1812.07035, 2018.  URL http://arxiv.org/abs/1812.07035.

##  Appendix A Contact shape prediction from tactile observations

Given a tactile observation, our goal is to extract the contact shape that
produces it. To that aim, we train an image translation netwrok (pix2pix)
based on Isola et al. [2018] to map from RGB tactile observations to contact
shapes, described below for completeness.

As shown in Figure 9, the input to the NN is an RGB tactile image of size
160x120x3. The output corresponds to a binary contact mask of size 160x120x3,
which we flatten to 160x120 to represent the contact shape.

The training data is collected autonomously in a controlled 4-axis stage that
generates controlled touches on known 3D-printed shapes. In our case, we
collect the calibration data from two 3D-printed boards with simple geometric
shapes on them (see the top of Figure 9). From each controlled touch, we
obtain a tactile observation and the pose of the board w.r.t. to the sensor.
From this pose, we can later simulate the corresponding contact shape to the
tactile observation using geometric contact rendering and the 3D model of the
board.

Note that to do tactile localization on any object, we only need to gather
calibration data once because the map between tactile observations and contact
shapes is object-independent. Empirically, we find that the map is also
independent of sensor instance.

The training input is a 160x240x3 image, consisting of the raw tactile image
and the ground truth contact mask stitched together.

Both the generator and descriminator networks of the NN are in the form of
convolution-BatchNorm-ReLu modules, as in Isola et al. [2018]. The optimizer
is Adam with an initial learning rate of 0.0002, that decays linearly. The
momentum of Adam is 0.5. The GAN objective is the least squares loss function.

In practice, we collected 10,000 pairs of tactile observations in total (9,000
from the board at left, and 1,000 from the board at right), and trained the NN
for 86 epochs.

Figure 9: Training data set-up to estimate real contact shapes from real
tactile observations. We use the boards at the top to collect real tactile
observations from them as well as simulated contact shapes. By placing the
boards on a robotic platform, we can move them w.r.t. the sensor to perform
calibrated touches on them. From each touch, we recover both a tactile image
and a simulated contact shape that we compute using the 3D model of the boards
and its pose w.r.t the sensor when the touch happened.

##  Appendix B Collecting ground-truth data for real objects

We collect labelled datasets of tactile observations of grasps on 20 objects
mounted in known positions and orientations in the world. Each dataset
contains pairs of RGB tactile images, and the corresponding ground-truth
object pose relative to the gripper. This allows us to compute pose errors
between Tac2Pose predictions from tactile observations, and the true
calibrated object poses.

To get calibrated real pairs of contact poses and tactile observations, we
first obtain a coarse estimate of the object pose using an overhead camera. We
then refine that estimate using Tac2Pose for geometric contact rendering: we
move the robot to our coarse estimate of the object pose, grasp the object,
and compare the observed contact with the simulated contact shape in the
estimated pose. We adjust the estimate of the object pose until the observed
contact matches the simulated one.

The robotic system we use to collect the labelled datasets consists of a dual
arm ABB Yumi with two WSG-32 grippers and GelSlim 3.0 tactile sensing fingers.
Figure 10 shows a still of the data collection procedure for the object stud.

We collect labelled observations of feasible grasps on each each object. To
determine which contacts should appear in the datasets, we first find the
stable poses of each object. Stable poses are the poses that an object is most
likely to fall in when dropped onto a table. We then determine a set of
feasible grasp approach directions for each stable pose. We define the grasp
approach direction as the axis of the grasp during a parallel jaw grasp. We
break out the labelled datasets by grasp approach direction, to facilitate
insight about what categories of grasps are easier to localize for different
objects. For each grasp approach direction, we collect a set of equispaced
observations, that correspond to a given grasp approach direction at various
x,y locations relative to the object. In total, these observations correspond
to the set of contacts that we are likely to encounter when picking up each
object from a table.

Figure 10: Collecting ground-truth data on real objects. We 3D-print object
models mounted on posts and fix them in known location relative to the robot.
As a result, we can collect ground-truth data for test objects (the object
stud is shown).

##  Appendix C Similarity metric for contact shapes

Given a new contact shape, we want to compare it to all pre-computed contact
shapes in the grid to find what poses are more likely to produce it. To that
aim, we use MOCO [He et al., 2019], a widely used tool in contrastive
learning, to train a NN that encodes contact shapes into a low dimensional
embedding allowing to compare them using the distance between their encodings.

The NN encoder is based on a pre-trained Resnet-50 [He et al., 2016] cropped
before the average-pooling layer to preserve spatial information. It encodes
contact shapes into vectors of dimension 1000. To train, we use the same
optimizer as He et al. [2019]: stochastic gradient descent with a learning
rate that starts at 0.03 and decays over time, momentum of 0.999, and weight
decay of 0. Each training datapoint comes from selecting a random contact
pose, rendering its contact shape, and finding its closest pose in the pre-
computed grid. Then our input is the contact shape. The output or label is a
vector with all entries zeros except the one that corresponds to the closest
element that gets a one. This vector represents the probability of each poses
in the grid to be the closest to the target pose. The loss function is the
cross-entropy loss between the label and the softmax between the target
encoder and the ones from the grid (see Fig. 3). As the NN gets trained, we
recompute the encoding of the queue following [He et al., 2019]. We train the
models for 15 epochs, as the model tends to overfit to simulated data after
that.

To make the training data resemble more the real distribution of contact
shapes, we train using as target, contact shapes with different depth
thresholds, $\Delta d$, selected randomly between 1 and 2 mm. This accounts
for changes in the contact force applied to the sensor that can make contacts
more or less deep (see Fig. 3). Finally, before encoding the contact shapes,
we converted them into binary masks to facilitate the learning process and
prevent that numerical mismatches between real and simulated contact shapes
affect the results.

##  Appendix D Multi-contact for N sensors

To derive Equation 1 in Section 3.4, we will prove a more general equation for
the object pose $x$ when it is in contact with N sensors:

$P(x|C_{1},...,C_{N})=\\\ \frac{P(x|C_{1})\cdot...\cdot P(x|C_{N})\cdot
P_{ta}(x)/P_{tr}(x)^{N}}{\sum_{x^{\prime}}P(x^{\prime}|C_{1})\cdot...\cdot
P(x^{\prime}|C_{N})\cdot P_{ta}(x^{\prime})/P_{tr}(x^{\prime})^{N}}$ (2)

where $P(x|C_{i})$ is the likelihood that pose $x$ has produced a contact
shape $C_{i}$ on sensor $i$. $P_{ta}(x)$ is a prior over all possible contact
poses that would result in contact with the N sensors in the current task.
$P_{tr}(x)$ is the distribution over the contact poses used to train the
similarity function in Section 3.1.

Next, we prove Equation (2).

Assuming that we have access to the poses of the N sensors, given an object
pose $x$, the contact shapes $\\{C_{i}\\}$ become determined. This allows us
to write the joint distribution of pose and contact shapes as:

$P(x,C_{1},...,C_{N})=\\\ P_{ta}(x)\cdot P(C_{1}|x)\cdot...\cdot P(C_{N}|x)$
(3)

Now we can use Bayes theorem on the $P(C_{i}|x)$ terms:

$P(C_{i}|x)=P(x|C_{i})\cdot P(C_{i})/P_{tr}(x)$ (4)

and obtain:

$P(x,C_{1},...,C_{N})=P(x|C_{1})\cdot P(C_{1})\cdot...\\\ \cdot
P(x|C_{N})\cdot P(C_{N})\cdot P_{ta}(x)/P_{tr}(x)^{N}$ (5)

where we take into account that the $P(x|C_{i})$ have been trained using a
predefined distribution of poses $P_{tr}(x)$ that does not necessarily match
$P_{ta}(x)$. Now, we can compute $P(x|C_{1},...,C_{N})$ using again the Bayes
theorem:

$P(x|C_{1},...,C_{N})=\frac{P(x,C_{1},...,C_{N})}{P(C_{1},...,C_{N})}\\\
=\frac{P(x,C_{1},...,C_{N})}{\sum_{x^{\prime}}P(x^{\prime},C_{1},...,C_{N})}$
(6)

and thus

$P(x|C_{1},...,C_{N})=\\\ \frac{\left(\Pi_{i}^{N}P(x|C_{i})\cdot
P(C_{i})\right)\cdot
P_{ta}(x)/P_{tr}(x)^{N}}{\sum_{x^{\prime}}\left(\Pi_{i}^{N}P(x^{\prime}|C_{i})\cdot
P(C_{i})\right)\cdot P_{ta}(x^{\prime})/P_{tr}(x^{\prime})^{N}}$ (7)

We can now cancel the terms $P(C_{i})$ that appear both in the numerator and
denominator, getting:

$P(x|C_{1},...,C_{N})=\\\ \frac{\left(\Pi_{i}^{N}P(x|C_{i})\right)\cdot
P_{ta}(x)/P_{tr}(x)^{N}}{\sum_{x^{\prime}}\left(\Pi_{i}^{N}P(x^{\prime}|C_{i})\right)\cdot
P_{ta}(x^{\prime})/P_{tr}(x^{\prime})^{N}}$ (8)

This concludes the proof of Equation (2). In practice, we discretize the set
of contact poses $\\{x\\}$ that result in contact with the N sensors using the
grid over of sensor 1 (this is an arbitrary choice). Then, we use the
transformation between sensor 1 and sensor $i$ to compute each $P(x|C_{i})$
using the closest pose to $x$ in the grid of sensor $i$. Note that often a
pose that results in contact with sensor $1$ will not contact sensor $i$. In
that case, we do not consider that pose as we are only interested in poses
that contact the N sensors. Finally, because the grids are dense and
structured, finding the closest pose to $x$ in a grid has a minor effect on
performance and is fast to compute.

We can use Equation (2) to derive Equation (1) in Section 3.4 under two extra
assumptions. First, we observe that the denominator is constant because all
contact shapes $C_{i}$ are given. Next, if we have no prior over the contact
poses during training, then $P_{tr}(x)$ is constant, and that leads to
Equation 1 repeated here for completeness:

$P(x|C_{1},...,C_{N})\propto P(x|C_{1})\cdot...\cdot P(x|C_{N})\cdot
P_{ta}(x)$ (9)

##  Appendix E Results by grasp approach direction

Table 6: Results by grasp approach direction for Parallel Jaw ablation of
Tac2Pose

| Grasp Approach Direction mm (norm)
---|---
|  | 1 | 2 | 3 | 4
Long Grease |  | 1.2 (0.0) | 9.9 (0.3) |  |
Grease |  | 0.8 (0.1) | 2.0 (0.2) |  |
Stud |  | 13.6 (0.3) | 13.2 (0.3) |  |
Pin |  | 5.5 (0.2) | 5.7 (0.2) |  |
Hanger |  | 2.7 (0.1) | 1.1 (0.0) | 1.3 (0.0) | 6.2 (0.2)
Hydraulic |  | 9.1 (0.4) | 1.7 (0.1) |  |
Hose |  | 46.1 (0.7) | 37.2 (0.6) |  |
Round Hose |  | 35.2 (0.7) | 38.9 (0.7) |  |
Holder |  | 2.0 (0.1) | 2.9 (0.1) |  |
Round Cable |  | 15.2 (0.8) | 2.2 (0.1) |  |
Couple |  | 13.9 (0.6) | 19.6 (0.7) | 21.4 (0.8) |
Round Couple |  | 4.8 (0.2) | 14.8 (0.6) | 17.8 (0.7) |
Cable Clip |  | 10.6 (0.6) | 12.1 (0.7) |  |