# TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction
using Vision-Based Tactile Sensing

Mauro Comi1 Yijiong Lin1,2 Alex Church1,2 Alessio Tonioni3 Laurence Aitchison1
Nathan F. Lepora1,2

1University of Bristol 2Bristol Robotics Laboratory 3Google Zürich
<EMAIL_ADDRESS>
###### Abstract

Humans rely on their visual and tactile senses to develop a comprehensive 3D
understanding of their physical environment. Recently, there has been a
growing interest in exploring and manipulating objects using data-driven
approaches that utilise high-resolution vision-based tactile sensors. However,
3D shape reconstruction using tactile sensing has lagged behind visual shape
reconstruction because of limitations in existing techniques, including the
inability to generalise over unseen shapes, the absence of real-world testing,
and limited expressive capacity imposed by discrete representations. To
address these challenges, we propose TouchSDF, a Deep Learning approach for
tactile 3D shape reconstruction that leverages the rich information provided
by a vision-based tactile sensor and the expressivity of the implicit neural
representation DeepSDF. Our technique consists of two components: (1) a
Convolutional Neural Network that maps tactile images into local meshes
representing the surface at the touch location, and (2) an implicit neural
function that predicts a signed distance function to extract the desired 3D
shape. This combination allows TouchSDF to reconstruct smooth and continuous
3D shapes from tactile inputs in simulation and real-world settings, opening
up research avenues for robust 3D-aware representations and improved
multimodal perception in robotics. Code and supplementary material are
available at: https://touchsdf.github.io/

##  1 Introduction

The current state of 3D shape reconstruction research is primarily concerned
with the sense of vision [3, 37]. However, training Computer Vision algorithms
for object manipulation is challenging due to the high-dimensional observation
space, which suffers from occlusion and external lighting conditions.
Methodologies based on high-resolution tactile sensing have recently been
proposed as additional solutions for object exploration and manipulation [44,
17]. This type of perception presents several benefits over pure camera-based
sensing, including the ability to capture contact-rich information, as well as
being effective despite the presence of occlusions, homogeneous surfaces and
transparent or reflective object materials. Additionally, tactile observations
simplify the translation of simulated tasks to real-world scenarios as they
require a smaller observation space [4].

Recently, several data-driven methodologies that utilise vision-based tactile
sensors for 3D understanding have been proposed [11, 27, 17]. To the best of
our knowledge, Smith et al. [31] proposed the first and only approach for
vision-based tactile reconstruction, which utilised DIGIT vision-based tactile
sensors [16] to extract contact-rich information. Although that work serves as
a foundation for tactile-driven 3D reconstruction, a lack of testing in a
real-world scenario makes it challenging to assess its applicability to
robotic manipulation. Furthermore, the resulting reconstruction consists of
separate meshes instead of smooth surfaces, potentially introducing
discontinuities in the reconstructed shape. Notably, Smith et al. employ both
touch-only and visuo-tactile data into their approach. For the purposes of our
comparison, we focus on the touch-only model to investigate the possibility of
3D reconstruction using only tactile data, considering the advantages
previously discussed.

In this work, we propose TouchSDF, a novel approach for 3D shape
reconstruction that leverages an implicit neural function for vision-based
tactile sensing. Unlike existing methods that utilise discrete
representations, TouchSDF employs DeepSDF [24], an implicit neural
representation that encodes a smooth and continuous surface, enabling more
accurate and robust reconstructions. DeepSDF has demonstrated the ability to
reconstruct 3D geometry from partial point clouds, which is useful in our
context as tactile sensors provide partial observations of an object. A key
step for 3D reconstruction in the real world is to have effective real-to-sim
tactile image translation, where we extend previous work with a GAN-based
method [4] to handle complex surfaces and 6D poses. Through evaluation on both
simulated and real objects, we demonstrate the effectiveness of TouchSDF in
tactile-based 3D shape reconstruction and highlight its potential in object
exploration and manipulation tasks.

Contribution In summary, our work makes the following principal contributions:
1) We propose TouchSDF, an approach for 3D shape reconstruction using vision-
based tactile sensing. By leveraging the implicit neural representation
DeepSDF, our method encodes smooth and continuous surfaces that enable
accurate and robust reconstructions from partial observations.
2) We demonstrate the ability to generalise across unseen objects and poses,
both in simulation and in reality, by conditioning on latent variables, thus
encoding a wide range of geometrical variations.
3) We give the first evaluation of 3D shape reconstruction using purely
vision-based tactile sensing on some real-world 3D-printed and household
objects.

##  2 Related work

Object reconstruction using tactile sensing is a new, active area of research
that combines and adapts reconstruction methodologies from computer vision,
graphics and robotics to a distinct perceptual modality: high-resolution
touch.

The development of low-cost and open-source tactile sensors has opened new
avenues of study in 3D reconstruction over the past few years [20, 14, 43,
12]. To the best of our knowledge, [41] proposed the first work on touch
exploration for object reconstruction that employs vision-based tactile
sensing (using a GelSight sensor [44]). However, this technique yields a
voxel-based reconstruction that is not always suitable for retrieving fine
details. In contrast, Smith et al. [30, 31] proposed an approach that
decouples vision and tactile sensing (using a DIGIT [16]) by leveraging a
series of Graph Convolutional Networks to deform a default mesh into the
desired object [32]. However, the lack of testing in the real world makes it
challenging to assess the applicability of this reconstruction method in a
robotic setting. In contrast, our approach results in a smooth and continuous
surface reconstruction on both simulated and real-world shapes.

Figure 1: (Top) Data collection procedure for extracting the tactile image at
the touch location, the sensor pose as an end effector to the robot arm, and
local point cloud. (Bottom) Procedure to map a tactile image into a local
point cloud with a CNN conditioned on the previously extracted data.

Implicit neural representations in robotic manipulation. Implicit Neural
Representations (INRs), such as DeepSDF [24], Occupancy Networks [21] and NeRF
[22], have recently been proposed as state-of-the-art approaches for many
tasks in graphics, vision and robotics. In particular, numerous works have
explored the use of INRs in manipulation tasks [11, 5, 42, 29, 7, 15].
However, while NeRF-based approaches excel in view-synthesis tasks, they are
not suited for reconstruction tasks based purely on tactile sensing, as the
output produced by tactile sensors is a local depth map rather than a global
view of the scene. Consequently, our proposed approach relies on DeepSDF, an
efficient INR for encoding and reconstructing 3D surfaces using solely
geometric information.

Tactile sim-to-real. Physics engines approximate sensor deformation, resulting
in a disparity between real and simulated tactile images. Numerous approaches
have been suggested to bridge this simulation-to-reality gap, such as finite-
element methods [23, 28] and rigid-body physics approximation [4, 1, 40].
Recently, Higuera et al. [10] proposed a diffusion model-based technique [33]
to generate realistic tactile images from simulated contact depths. However,
diffusion models have high computational requirements which limit their
suitability in real-time and low resource environments. Hence, we follow the
approach proposed by [4] in utilising a fast rigid-body physics simulator to
simulate depth images and a one-step image translation model trained using a
Generative Adversarial Network (GAN) [9], which is more suitable for real-time
robotics applications.

##  3 Methodology

Our reconstruction procedure has three steps
(LABEL:fig:architecture_vertical). In the first step, we collect tactile
images by sampling the surface of the object we intend to reconstruct. We also
store the sensor (end-effector) pose at contact alongside the local point
cloud describing the contact geometry. Real-world tactile images are converted
into simulated tactile images using a real-to-sim GAN model [4]. In the second
step, we predict the contact geometry based on the simulated tactile image and
sensor pose. Finally, we employ a pre-trained DeepSDF model to reconstruct
objects using the predicted contact geometry (local point cloud). Both our
contact geometry prediction and DeepSDF models were trained separately on
objects sampled from the ShapeNet and ABC datasets. While ShapeNet consists of
3D models of various object categories, the ABC dataset lacks categorical
divisions and includes objects synthetically generated to encompass a wide
range of geometries.

###  3.1 Simulation

Collection of tactile images and local point cloud. Tactile images, local
point clouds and sensor poses were collected for training purposes using a
PyBullet-based tactile simulator (Tactile Gym [4] [18]). The data collection
procedure involved four steps. First, the objects to be reconstructed are
loaded into our simulator and the sampling domain is automatically restricted
to the shape boundaries of the object. Second, a virtual sphere is created
around the object (Sample surface in Fig. 1), from which a random location is
sampled. Third, the robot arm moves the end effector-mounted tactile sensor to
this location while keeping the sensor oriented towards the centre of the
object. Last, as the sensor touches the object, the surface contact area is
transduced into a 256$\times$256 pixel tactile image. This image is obtained
by rendering the contact depth at the touch location using the PyBullet’s
synthetic camera. To control the touch depth, the average image intensity
$\mathrm{D}_{t}\in[0,255]$ is checked every timestep $t$; then the sampling
procedure is stopped when $\mathrm{D}_{t}>1$. In addition to the tactile
image, the sensor’s pose and a local point cloud are extracted. This point
cloud serves as the ground truth to train a model that maps a tactile image
into the corresponding touched surface.

Tactile images to point cloud prediction. We adapted the approach of Smith et
al. [31] to learn a mapping between tactile images and local point clouds. Our
data collection involved a simulated 6-DoF robot arm equipped with a simulated
TacTip tactile sensor [4, 18] to sample the surfaces of the objects used for
training and validation. To train our local chart prediction model, we
gathered single-channel 2D tactile images in simulation, with each image
representing the depth map of the touched area on the object’s surface
$\mathcal{S}$. Additionally, we extracted a ground truth point cloud of the
surface at the touch location. Our objective is to employ a Convolutional
Neural Network (CNN) to predict the contact geometry by deforming a base mesh.
To achieve this, we followed the procedure outlined in [31] and shown in Fig.
1. Initially, we define a base triangular mesh with 25 fixed vertices and
faces, serving as the initial state for subsequent deformations. Then, a CNN
receives a tactile image as input and produces the relative displacement for
each vertex in the base mesh, which is deformed by displacing each vertex
according to its corresponding predicted displacement value. Next, we
uniformly sample a point cloud on the deformed mesh, generating a set of
points representing the shape at touch location. Additionally, we enhanced the
point cloud by estimating the normals of the surface and sampling $m$
additional points along them, both in the positive and negative directions.
Finally, the CNN is optimised to minimise the Chamfer Distance [34] between
the point cloud sampled on the predicted mesh and the ground truth point
cloud.

3D shape reconstruction. DeepSDF [24] is an implicit neural representation
that encodes 3D shapes as Signed Distance Functions (SDF), enabling smooth
shape reconstruction from partial observations. In this work, we utilised
DeepSDF to reconstruct 3D shapes, leveraging partial object surfaces estimated
from tactile sensor observations. Following the methodology outlined in [24]
on DeepSDFs, we constructed a dataset of SDF pairs
$\\{(\mathbf{x_{j}},s_{j})\\}_{j=1}^{N}$ per object, where
$\mathbf{x}\in\mathbb{R}^{3}$ are the coordinates sampled within a closed
volume around the objects and $s_{j}\in\mathbb{R}$ corresponds to the
associated signed distances. We adopted the architecture introduced by [24]
with added positional encoding $\gamma(\cdot)$ [22, 38], where
$\gamma:\mathbb{R}^{3}\rightarrow\mathbb{R}^{m}$ is a Fourier feature
representation encoding each spatial coordinate $\mathbf{x_{j}}$ into a set of
high-frequency sinusoidal functions
$\gamma(\mathbf{x_{j}})=[\gamma_{1}(\mathbf{x_{j}}),\gamma_{2}(\mathbf{x_{j}}),...,\gamma_{m}(\mathbf{x_{j}})]$.
Positional encoding, which we use in place of raw 3D coordinates, has proved
to be extremely efficient in encoding high-frequency functions [35], as
demonstrated in recent advancements in neural radiance fields [22]. This
aligns with recent findings [25] indicating that deep networks tend to bias
towards lower frequency functions. Positional encoding projects inputs into a
higher-dimensional space using high-frequency functions, thus improving the
model’s ability to capture high-frequency variations in the data. During
training, the DeepSDF model learns a continuous signed distance function
$f_{\theta}(\mathbf{x},\mathbf{z})$ conditioned on a shape embedding
$\mathbf{z}$. To ensure the encoding of shape information, an embedding
$\mathbf{z_{i}}$ is jointly optimized with the network parameters $\theta$ for
each shape $i$. As a result, similar shapes are encoded by similar latent
vectors, enabling interpolations in the latent space that facilitate the
reconstruction of unseen objects. The objective minimised during the training
process follows [24]:

$\mathcal{L}(\theta,\mathbf{z})=\underset{\theta,\mathbf{z_{i}}}{\text{arg
min}}\sum_{i}\sum_{j=1}^{N}||f_{\theta}(\gamma\mathbf{(x_{j})},\mathbf{z_{i}})-s_{j}||_{1}+\alpha\cdot||\mathbf{z_{i}}||_{2}^{2}$
(1)

Here the regularisation term $||\mathbf{z_{i}}||_{2}^{2}$ weighted by $\alpha$
is crucial to avoid exploding gradients when optimising latent vectors.
Additionally, this term serves as a prior on the shape embeddings, enhancing
their retrieval during inference.

Reconstruction. The CNN-predicted point cloud $\mathcal{O}$ provides a partial
observation of the geometry at the touch location, including samples along the
normals of the estimated contact surface. Points directly on the surface are
assigned a signed distance value of zero, while those along the normals
receive positive or negative distances. This approach enhances the robustness
of the DeepSDF model at inference time, as it provides not only surface points
but also samples inside and outside the shape. To reconstruct the complete
shape of the target object, a DeepSDF auto-decoder [24] is conditioned on
$\mathcal{O}$ to optimise a 128-dimensional latent code $\mathbf{z_{i}}$ by
solving $\text{arg
max}_{\mathbf{z_{i}}}\mathbb{P}(\mathbf{z_{i}}|\mathcal{O})$ via first-order
optimisation. This is achieved by freezing the model’s parameters $\theta$ and
solving $\text{arg min}_{\mathbf{z_{i}}}\mathcal{L}(\mathbf{z_{i}})$. The
inferred global latent vector represents a compact representation of the
entire shape that best describes the partial observation. Inspired by GAN
inversion [26], we additionally freeze the latent vector and fine-tune the
model’s parameters on the partial observation. The signed-distance function is
defined by conditioning a pre-trained DeepSDF model on coordinates
$\mathbf{x}$ and inferred latent vector $\mathbf{z_{i}}$. Finally, a surface
$\mathcal{S}$ is extracted by employing the deterministic Marching Cubes
algorithm [19], which extracts the zero-level set of the predicted signed-
distance function
$\mathcal{S}=\\{\mathbf{x}\in\mathbb{R}^{3}|f_{\theta}(\mathbf{x},\mathbf{z})=0\\}$.

###  3.2 Real world

Hardware. We conducted real-world experiments with a 6-DoF industrial robot
arm (ABB IRB 120). The robot was equipped with a high-resolution vision-based
tactile sensor, for which we used a TacTip 3D-printed soft biomimetic tactile
sensor [17]. For real-world evaluation, we 3D-printed four objects (two
different bottles, a camera, and a bowl) selected from ShapeNetCore.V2 [2] and
selected two household objects (a transparent jar and a mug). Note that all
these objects are unseen during training. The 3D-printed objects were designed
with mounting sockets bolted onto a table as a secure fixture during tactile
interactions and the everyday objects were fixed with a clamp.

Real-to-Sim Image Transfer. The point cloud prediction model, which predicts
the coordinates that DeepSDF uses for shape completion, is trained on
simulated tactile images. These exhibit significant deviations from real
tactile images (Fig. LABEL:fig:architecture_vertical, top-left box). When
reconstructing objects in the real world, the robot collects real tactile
images, which the point cloud prediction model is not trained to process.
Therefore, we map the real tactile images to simulated ones using the
translation approach proposed in [4], which utilizes a Generative Adversarial
Network (GAN) framework for image-to-image translation. The pix2pix GAN [13]
is trained with pairwise simulated (depth map) and real tactile images (marker
patterns) collected from the objects set introduced in [8] and not part of the
test data. We adopted the hyperparameters from [4], which gave good
performance in the translation process [4] [18].

Sim-to-Real Object Reconstruction. In this step, we combine the point cloud
prediction model, DeepSDF and real-to-sim image transfer to achieve sim-to-
real object reconstruction. We collect real tactile images by performing
random contacts with the robotic arm-mounted tactile sensor onto the real-
world objects, using the same contact criteria as the simulated data gathering
(Sec. 3.1). For each touch, the real image is translated into a simulated
tactile image and labeled with the corresponding end effector pose. The CNN is
used to predict a local point cloud describing the contact geometry. Our
method is evaluated by computing the mean absolute error between the predicted
point clouds and the object (Sec. 4.3), which is possible due to the
availability of CAD models for those 3D-printed objects. Finally, our pre-
trained DeepSDF model is conditioned on the predicted point clouds and
estimated signed distance to extract the shape of an object (see Sec. 3.1).

Figure 2: Comparison of TouchSDF and Smith et al. (grasp-only) performance
after 20 touches. TouchSDF achieves a better score on EMD, Surface error, and
visual quality, despite occasionally registering lower CD values.

##  4 Results

###  4.1 Metrics

Our experimental evaluation utilised three metrics: Chamfer Distance [34],
Earth Mover’s Distance (EMD) [6], and Surface error. While CD is a common
metric in 3D reconstruction tasks, it has documented limitations in accurately
measuring the visual quality of reconstructed meshes [39] [36]. This
limitation stems from CD’s focus on point-wise distance rather than on the
distribution of the predicted point cloud. This can be observed in Fig. 2,
where shapes with lower visual quality reconstructions achieve better (lower)
CD scores. To address this shortcoming, EMD optimises a mapping that minimises
the average distance between corresponding points in the predicted and ground
truth point sets. Because EMD focuses on the entire point cloud distribution,
it is more faithful to visual quality. However, these metrics do not provide a
measure of the average error across surface reconstruction, which is
informative in assessing the quality of a reconstructed mesh. Therefore, we
introduced the Surface Error, which quantifies the average error on the
surface reconstruction relative to each mesh. Specifically:

$\text{Surface error (\%)}=\frac{|S_{p}-S_{gt}|}{S_{gt}}\cdot 100$

where $S_{gt}$ and $S_{p}$ are the surface of the ground truth and predicted
mesh respectively.

###  4.2 Simulation

Figure 3: Reconstruction results of TouchSDF for 1, 10 and 20 touches. The
Chamfer Distance (CD) between the reconstructed and ground truth meshes is
reported. Predicted point clouds referring to contact geometries are shown in
purple (darker tones represent positive and lighter tones negative distances).

We trained and tested our models using objects from the ABC and the ShapeNet
[2] datasets. While for the ABC dataset we were able to benchmark our model’s
performance against the grasp-only model proposed by Smith et al., a direct
comparisons on the ShapeNet dataset was not performed as Smith et al.’s
evaluation and provided model were not tailored to this specific dataset.
However, our results could serve as a benchmark for future assessments.

ABC dataset. We sampled 3500 shapes for training, 350 for validation and 200
for testing. To ensure a fair comparison with Smith et al. [31], shapes were
randomly sampled from the training, validation, and testing sets used by those
authors. Table 1 shows the CD, EMD and Surface error obtained by TouchSDF and
Smith et al. after 20 touches. To evaluate their approach, we used the grasp-
only model provided by the authors. Both EMD and Chamfer Distance were
computed using 4096 points. TouchSDF achieves better EMD and surface
reconstruction error, but slightly lower CD despite better visual quality
(Fig. 2). It is important to highlight that, in contrast to our approach,
Smith et al.’s reconstruction model was trained to explicitly minimise the CD
between predicted and ground truth meshes.

|  CD ($\downarrow$) |  EMD ($\downarrow$) |  Surface error ($\downarrow$)
---|---|---|---
Smith et al. | 0.003 | 0.19 | 2785%
(grasp-only)
TouchSDF (ours) | 0.006 | 0.07 | 36%
Table 1: Comparison of TouchSDF and Smith et al. after 20 touches.  Figure 4:
Reconstructions of seen and unseen objects (ShapeNet). The consistent EMD
scores demonstrate TouchSDF’s ability to generalise across categories.

ShapeNet dataset. We collected a dataset consisting of 1,650 objects, which
were then divided into three subsets: 1,100 objects for training, 200 for
validation and 350 for testing. The chosen objects span six diverse categories
considered previously in related work: bowls, bottles, cameras, jars, guitars
and mugs. Fig. 3 and Tab. 2 show an improvement in reconstruction quality as
the number of touches increases. This results from a higher coverage of the
object surface. Fig. 3 displays a set of point clouds predicted from tactile
images. We present the object point clouds of the reference shape (in blue)
solely for visualisation purpose, as our model exclusively relies on the
predicted point clouds (in purple). By leveraging points with both positive
and negative distance, we increased the robustness of the latent code
inference procedure.

| Unseen objects and poses
---|---
# Touches | 1 | 10 | 20
Bottle |  0.113 $\pm$ 0.085 |  0.082 $\pm$ 0.058 | 0.047 $\pm$ 0.024
Mug |  0.091 $\pm$ 0.030 |  0.072 $\pm$ 0.024 | 0.066 $\pm$ 0.018
Bowl |  0.085 $\pm$ 0.034 |  0.073 $\pm$ 0.029 | 0.048 $\pm$ 0.017
Camera |  0.131 $\pm$ 0.035 |  0.101 $\pm$ 0.046 | 0.092 $\pm$ 0.043
Guitar |  0.195 $\pm$ 0.102 |  0.177 $\pm$ 0.104 | 0.155 $\pm$ 0.087
Jar |  0.164 $\pm$ 0.113 |  0.136 $\pm$ 0.107 | 0.071 $\pm$ 0.038
Average |  0.136 $\pm$ 0.092 |  0.112 $\pm$ 0.086 | 0.081 $\pm$ 0.062
Table 2: EMD for unseen objects and poses in simulation (lower is better).
Results are reported for 1, 10 and 20 touches. Reconstruction quality improves
with number of touches.

To assess the accuracy of the reconstructed shapes, we considered 300 unseen
objects and poses across the six distinct categories previously mentioned,
using 1, 10, and 20 touches per object. To measure the reconstruction quality,
we computed the EMD between predicted and ground truth objects and then
computed the average metrics for each category (Table 2). Our results indicate
that increasing the number of touches leads to much better reconstruction
quality across all categories. The Guitar category has a higher EMD than
average, likely due to the substantial geometric variations in objects
belonging to this category. As a result, it is challenging for the auto-
decoder framework to optimise robust global shape embeddings for these
objects.

Moreover, the relatively low difference in EMD scores for seen and unseen
objects obtained for 20 randomly-sampled touches (Fig. 4) shows that our model
is able to generalise effectively to unseen objects and poses within known
categories. The obtained reconstructions retain details that could be useful
for downstream tasks in robot learning, such as cup handles and concave
surfaces.

###  4.3 Real world

Figure 5: Examples of real-world touches, with the corresponding real and
simulated tactile images, and predicted point cloud superimposed on the
object, for the 3D-printed camera, bowl and bottle. Figure 6: Reconstruction
of four real 3D-printed objects and two everyday objects. The Earth Mover’s
Distance (EMD) calculation requires a CAD model, hence it is applicable to
3D-printed objects but not to everyday objects. The cube-shaped mounting
sockets are not part of the objects.

Real-to-Sim. To evaluate our sim-to-real procedure, we collected 160 real-
world tactile images from the four 3D-printed objects illustrated in Fig. 6.
We employed the Real-to-Sim Image Transfer approach described in Section 3.2
to convert these real tactile images into simulated tactile images. Next, our
local surface prediction CNN was utilised to predict local point clouds based
on the translated tactile images. The evaluation of our method involved
calculating the Mean Absolute Error (MAE) between the predicted local point
clouds and the corresponding contact geometries obtained from our tactile
simulator at the touch location. The results (Table 3) demonstrate that our
method achieved an average error from 2-4 mm over the considered objects when
predicting contact geometry. Figure 5 shows examples of collected tactile
images, their translations from real to simulated images, and the resulting
predicted local point clouds superimposed onto the original CAD model.

| Camera | Bowl | Bottle A | Bottle B
---|---|---|---|---
MAE (mm) | 4.0 | 3.1 | 2.5 | 2.3
Table 3: Mean average error between predicted local point clouds and contact
geometries.

Reconstruction. To reconstruct the 3D-printed objects, we collected real
tactile images from random locations on the object surfaces and translated
them into their corresponding simulated images. The reconstruction procedure
followed the same methodology employed in simulation. As the embeddings
optimised by DeepSDF are not $SE(3)$-invariant, it was necessary to ensure
that the pose of the object being reconstructed is consistent with the poses
observed during training. To achieve this, the world reference frame was set
manually to the centre of the objects. Our method successfully reconstructed
both real 3D-printed objects and additional everyday objects (a mug and a
transparent jar), achieving low EMD values that are comparable to those
obtained in simulation (Fig. 6). The EMD could not be computer for everyday
objects to the lack of CAD models.

##  5 Discussion and limitations

In this study, we introduced TouchSDF, a new approach for reconstructing 3D
objects using tactile images captured by a high-resolution vision-based
tactile sensor (the TacTip), both in simulation and in reality. To the best of
our knowledge, this is the first work capable of reconstructing smooth and
continuous 3D objects in the real world using solely tactile sensing. Our work
was enabled by advancements in real-to-sim translation techniques for vision-
based tactile images [4] coupled with progress in neural implicit
representations [24]. By leveraging TouchSDF, which was trained entirely in
simulation, we successfully reconstructed real-world objects via real-to-sim
translation. In combination with a CNN capable of mapping tactile images to
local point clouds, we obtained precise local surface reconstructions with an
average error of around 2-4 mm. Additionally, we demonstrated that the
reconstruction quality improves notably with increasing the number of touches.
Overall, we expect that our approach can be adapted to various types of
vision-based tactile sensors, as the real-to-sim technique we used here has
also been shown to work well with a GelSight-type sensor [18].

Limitations and future directions. Because DeepSDF is not inherently
$SE(3)$-invariant, the performance of our method relies on the object’s pose
and the ability to interpolate in the latent space to generalise over unseen
poses. Moreover, objects with unique shapes and features are harder to
reconstruct, as retrieving their corresponding latent vector at test-time
optimisation is challenging. Potential solutions involve augmenting the
dataset with objects in different poses and exploring $SE(3)$-invariant
methodologies for shape completion using DeepSDF. Another limitation lies in
the use of an auto-decoder [24] within the DeepSDF framework, which results in
slow latent code optimisation. The implementation of a standard encoder-
decoder architecture could potentially expedite the process, as it does not
rely on the optimisation of a latent code at test time. Furthermore, the
random selection of touch locations is motivated by the lack of initial
knowledge regarding the object to be reconstructed, and it would be
interesting to explore strategies for inferring the next optimal touch
location that maximises reconstruction quality as in [31]. Lastly, a potential
avenue for future research in 3D understanding lies in integrating vision with
tactile sensing. By leveraging the rich information captured by both
modalities, we would expect that the techniques could be extended to enable
sophisticated object exploration and manipulation tasks.

## References

  * [1] Arpit Agarwal, Timothy Man, and Wenzhen Yuan.  Simulation of vision-based tactile sensors using physics based rendering.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1–7. IEEE, 2021.
  * [2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [3] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628–644. Springer.
  * [4] Alex Church, John Lloyd, Nathan F Lepora, and others.  Tactile sim-to-real policy transfer via real-to-sim image translation.  In Conference on Robot Learning, pages 1645–1654. PMLR.
  * [5] Danny Driess, Jung-Su Ha, Marc Toussaint, and Russ Tedrake.  Learning models as functionals of signed-distance fields for manipulation planning.  In Conference on Robot Learning, pages 245–255. PMLR, 2022.
  * [6] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  * [7] Jiahui Fu, Yilun Du, Kurran Singh, Joshua B Tenenbaum, and John J Leonard.  Neuse: Neural se (3)-equivariant embedding for consistent spatial understanding with objects.  arXiv preprint arXiv:2303.07308, 2023.
  * [8] Daniel Fernandes Gomes, Paolo Paoletti, and Shan Luo.  Generation of gelsight tactile images for sim2real learning.  IEEE Robotics and Automation Letters, 6(2):4177–4184, 2021.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [10] Carolina Higuera, Byron Boots, and Mustafa Mukadam.  Learning to read braille: Bridging the tactile reality gap with diffusion models.  arXiv preprint arXiv:2304.01182, 2023.
  * [11] Carolina Higuera, Siyuan Dong, Byron Boots, and Mustafa Mukadam.  Neural contact fields: Tracking extrinsic contact with tactile sensing.  arXiv preprint arXiv:2210.09297, 2022.
  * [12] Jarmo Ilonen, Jeannette Bohg, and Ville Kyrki.  Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing.  The International Journal of Robotics Research, 33(2):321–341, 2014.
  * [13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [14] Yan-Bin Jia and Jiang Tian.  Surface patch reconstruction from “one-dimensional” tactile data.  IEEE Transactions on Automation Science and Engineering, 7(2):400–407, 2009.
  * [15] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu.  Synergies between affordance and geometry: 6-dof grasp detection via implicit representations.  arXiv preprint arXiv:2104.01542, 2021.
  * [16] Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al.  Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation.  IEEE Robotics and Automation Letters, 5(3):3838–3845, 2020.
  * [17] Nathan F Lepora.  Soft biomimetic optical tactile sensing with the tactip: A review.  IEEE Sensors Journal, 21(19):21131–21143, 2021.
  * [18] Yijiong Lin, John Lloyd, Alex Church, and Nathan F Lepora.  Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch.  IEEE Robotics and Automation Letters, 7(4):10754–10761, 2022.
  * [19] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM siggraph computer graphics, 21(4):163–169, 1987.
  * [20] Martin Meier, Matthias Schopfer, Robert Haschke, and Helge Ritter.  A probabilistic approach to tactile shape reconstruction.  IEEE Transactions on Robotics, 27(3):630–635, 2011.
  * [21] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019.
  * [22] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [23] Yashraj Narang, Balakumar Sundaralingam, Miles Macklin, Arsalan Mousavian, and Dieter Fox.  Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6444–6451. IEEE, 2021.
  * [24] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [25] Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron Courville.  On the spectral bias of deep neural networks.  arXiv preprint arXiv:1806.08734, 4, 2018.
  * [26] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM Transactions on graphics (TOG), 42(1):1–13, 2022.
  * [27] Lukas Rustler, Jens Lundell, Jan Kristof Behrens, Ville Kyrki, and Matej Hoffmann.  Active visuo-haptic object shape completion.  IEEE Robotics and Automation Letters, 7(2):5254–5261, 2022.
  * [28] Carmelo Sferrazza and Raffaello D’Andrea.  Sim-to-real for high-resolution optical tactile sensing: From images to 3d contact force distributions.  arXiv preprint arXiv:2012.11295, 2020.
  * [29] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.  Neural descriptor fields: Se (3)-equivariant object representations for manipulation.  In 2022 International Conference on Robotics and Automation (ICRA), pages 6394–6400. IEEE, 2022.
  * [30] Edward Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal.  3d shape reconstruction from vision and touch.  Advances in Neural Information Processing Systems, 33:14193–14206, 2020.
  * [31] Edward Smith, David Meger, Luis Pineda, Roberto Calandra, Jitendra Malik, Adriana Romero Soriano, and Michal Drozdzal.  Active 3d shape reconstruction from vision and touch.  Advances in Neural Information Processing Systems, 34:16064–16078, 2021.
  * [32] Edward J Smith, Scott Fujimoto, Adriana Romero, and David Meger.  Geometrics: Exploiting geometric structure for graph-encoded objects.  arXiv preprint arXiv:1901.11461, 2019.
  * [33] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [34] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2974–2983, 2018.
  * [35] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [36] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3405–3414, 2019.
  * [37] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2626–2634, 2017.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [39] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  * [40] Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra.  Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors.  IEEE Robotics and Automation Letters, 7(2):3930–3937, 2022.
  * [41] Shaoxiong Wang, Jiajun Wu, Xingyuan Sun, Wenzhen Yuan, William T. Freeman, Joshua B. Tenenbaum, and Edward H. Adelson.  3d shape perception from monocular vision, touch, and shape priors.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1606–1613. IEEE.
  * [42] Youngsun Wi, Pete Florence, Andy Zeng, and Nima Fazeli.  Virdo: Visio-tactile implicit representations of deformable objects.  In 2022 International Conference on Robotics and Automation (ICRA), pages 3583–3590. IEEE, 2022.
  * [43] Zhengkun Yi, Roberto Calandra, Filipe Veiga, Herke van Hoof, Tucker Hermans, Yilei Zhang, and Jan Peters.  Active tactile object exploration with gaussian processes.  In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4925–4930. IEEE, 2016.
  * [44] Wenzhen Yuan, Siyuan Dong, and Edward H Adelson.  Gelsight: High-resolution robot tactile sensors for estimating geometry and force.  Sensors, 17(12):2762, 2017.