# Overcoming Obstructions via Bandwidth-Limited
Multi-Agent Spatial Handshaking

Nathaniel Glaser, Yen-Cheng Liu, Junjiao Tian, and Zsolt Kira
Georgia Institute of Technology
<EMAIL_ADDRESS>
###### Abstract

In this paper, we address bandwidth-limited and obstruction-prone
collaborative perception, specifically in the context of multi-agent semantic
segmentation. This setting presents several key challenges, including
processing and exchanging unregistered robotic swarm imagery. To be
successful, solutions must effectively leverage multiple non-static and
intermittently-overlapping RGB perspectives, while heeding bandwidth
constraints and overcoming unwanted foreground obstructions. As such, we
propose an end-to-end learn-able Multi-Agent Spatial Handshaking network
(MASH) to process, compress, and propagate visual information across a robotic
swarm. Our distributed communication module operates directly (and
exclusively) on raw image data, without additional input requirements such as
pose, depth, or warping data. We demonstrate superior performance of our model
compared against several baselines in a photo-realistic multi-robot AirSim
environment, especially in the presence of image occlusions. Our method
achieves an absolute $11\%$ IoU improvement over strong baselines.

##  I INTRODUCTION

Deep learning has revolutionized robotic perception, especially in areas such
as object detection and semantic segmentation. As a result of these
improvements, we are increasingly seeing these algorithms applied at scale, as
with self-driving cars and autonomous delivery drones. However, many of these
core algorithms are developed around single agent perception. In a world where
physical hardware, compute power, and inter-connectivity are becoming
increasingly cheap, significant advantages exist in developing these
algorithms towards distributed multi-agent systems.

One such advantage is that of information redundancy–even if a single robot
faces a challenging input or failure mode (such as a local image occlusion),
it can leverage the information of its robot collaborators to overcome it.
However, several practical challenges remain: relevant information must be
identified, communication bandwidth constraints must be respected, and
computation must be effectively distributed across the networked robots.

In this paper, we address these challenges with a learned architecture that
identifies and exchanges information across several robots. Our problem
formulation and solution draw inspiration from the learning-based works of
several domains, including learned cost volume image correspondence [12, 7, 8,
21, 20], learned communication [11, 10], and learned multi-view fusion [16,
17, 9]. Based on these prior works, we develop a novel architecture that
effectively combines information from multiple robots in a distributed,
bandwidth-limited manner, without assuming known positions or transformations
between their sensors.

Unlike prior work, we introduce several novel features that allow for a tune-
able improvement in perception accuracy, as dictated by bandwidth constraints.
We extend the 1D handshaking mechanism of When2Com [10] to a 2D spatial
handshaking mechanism, allowing for patch-based identification, communication,
and exchange of visual information across agents. As with prior work, we match
image patches via cost volume correspondence; however, in contrast to previous
methods, we address multi-agent correspondence and introduce a novel
correspondence autoencoder, which infills the spurious matches and obstructed
patches that are common in an occlusion-prone, multi-agent setting. Finally,
we duplicate our cross-agent patch-matching pipeline across several agents and
perform multi-agent fusion on warped and exchanged segmentation data.

Figure 1: Collaborative Perception. Alone, an isolated perception system may
produce an undesirable output in the presence of an unexpected obstruction. On
the other hand, a collaborative perception system may leverage multiple views
to overcome potentially dangerous misclassifications.

To demonstrate the efficacy of our approach, we perform extensive
experimentation in a photo-realistic simulation environment (AirSim [15]),
specifically investigating how a group of mobile robots can communicate their
observations to overcome unexpected foreground obstructions, such as occluding
vegetation and wildlife. Our method increases performance on a multi-agent
semantic segmentation task by an absolute $11\%$ IoU over strong baselines,
and it approaches upper bounds that utilize ground truth transformations
across the agents’ sensors, while saving significant bandwidth.

We summarize the contributions of our paper as follows:

  * •

We introduce a variant on the Multi-Robot Collaborative Perception task [10,
11] in which a group of independent robotic agents captures synchronized 2D
RGB data as they travel through a shared 3D environment. Unlike prior work, we
do not provide any pose and depth information during inference.

  * •

To address this challenge, we propose an end-to-end learn-able Multi-Agent
Spatial Handshaking network, MASH. We contribute a set of novel techniques
that address challenges that the current state of art do not address,
including extending distributed communication mechanisms into 3D, introducing
a smoothing autoencoder to address potential inaccuracies in cost volume
estimation, and adding a tune-able compression mechanism for trading-off
bandwidth and perception accuracy.

  * •

We test our algorithm on the multi-robot collaborative perception task within
a photorealistic simulator, demonstrating superior performance. We
specifically highlight a real-world application of collaborative perception
with an obstruction-prone multi-drone dataset.

##  II RELATED WORK

Our work builds on several domains, including image correspondence, optical
flow, visual odometry, visual SLAM, communication, inpainting, and multi-view
fusion. Here, we highlight the learning-based approaches from each domain and
describe how each contributes to the design of our network architecture.

Image Correspondence methods explore how to correspond visual information
between pairs of images. To perform this correspondence, learning-based
approaches typically use learned feature extraction and feature comparison
modules. Prior work ranges from sparse matching via learned interest points
[6, 5] to dense matching with patches [14] and cost volumes [12]. These
techniques are particularly relevant to our work, which seeks to correspond
spatial image information between pairs of independent agents. Specifically,
we extend the cost volume decoder of DGC [12] into a cost volume autoencoder.
We train the autoencoder to use spatial context and matching uncertainty to
infill a warping grid and produce warping scores between pairs of agents.

Visual SLAM corresponds image features across observations but also creates a
spatial representation (i.e. a map) of these features. This spatial
representation allows for the features to be more reliably corresponded and
referenced for localization estimates. A notable learning-based example of
visual SLAM is EMPNet [1] which uses a dense depth map to reproject image
patches (and associated learned feature embeddings) into a 3D map. Subsequent
3D reprojections are then corresponded via weighted point cloud alignment to
build the map and to produce localization estimates. Though inspired by the
learned correspondence of visual SLAM, our work addresses a different problem
setting. Rather than building an accurate map from a continuous sequence of
observations collected over time, we instead focus on how to best leverage
currently available sensor information for an immediate prediction task,
especially from several independent and disparate viewpoints.

Learned Communication methods exchange information across multiple observers
via learned features. Who2Com [11] and When2Com [10] use a learned handshake
communication protocol to pass complementary information between degraded and
non-degraded agents. Specifically, each agent broadcasts a learned query
vector to the others; the other agents compare this query against an internal
key vector to produce a matching score; this matching score is used to forward
and fuse feature vectors from contributing agents for eventual decoding into
the final task output. Our work extends this one-dimensional handshaking
mechanism into a two-dimensional spatial handshaking mechanism. This extension
allows the network to efficiently flag patches of an image for bandwidth-
limited exchange across agents.

Inpainting methods attempt to replace missing or corrupted regions of an image
based on surrounding image context. The blind inpainting convolutional neural
network (BICNN) [3] overcomes image corruption by learning from a dataset that
contains corrupted images and their non-corrupted counterparts. The work of
Purkait et al [13] employs a similar method to infill the occluded semantic
masks of foreground and background objects. Similarly, our work addresses the
same task of overcoming image occlusions and degradations. However, unlike
inpainting works which “hallucinate” plausible replacements for occlusions,
our work leverages the perspectives of multiple agents to “see past” those
occlusions.

Multi-View Fusion combines multiple views into a shared representation that
can then be decoded for some task. For instance, the Multi-View Convolutional
Neural Network (MVCNN) [16] classifies 3D objects by combining the features of
multiple views into a single representation with a view-pooling operation.
Non-learned multi-view fusion, as with multi-view inpainting [17, 9], use 3D
reconstruction techniques or depth data to warp image data between viewpoints;
additionally, these works tackle the issue of missing correspondences via
patch-based or exemplar-based infilling. As with these prior works, our work
fuses information from multiple views. However, unlike MVCNN and other learned
multi-view methods, our approach explicitly learns how to align spatial images
features, rather than implicitly learning these transformations as part of the
network weights. This explicit alignment allows for better interpretability
and more flexibility with the features selected for cross-agent exchange and
fusion, as shown by our experimental results. Furthermore, unlike many of
these multi-view works, our approach does not require depth data, relative
pose data, static viewpoints, or expensive 3D reconstruction techniques.
Without these restrictions, our method is sufficiently lightweight for
distributed deployment across multiple mobile agents.

##  III METHOD

Figure 2: Multi-Agent Spatial Handshake Network: This figure depicts the
spatial handshake between a target agent (red block) and one of its supporting
agents (blue block). Without loss of generality, we describe the spatial
handshaking procedure for one pair of agents: (1ab) a target agent and a set
of supporting agents use a segmentation network (orange trapezoids) to compute
spatial feature maps and a semantic segmentation mask; (2a) the target agent
compresses its feature map with a query encoder and broadcasts it to all
supporting agents; (2b) each supporting agent receives this compressed map and
reshapes it with a query-key transform to handle any asymmetry between the
query and key; (2c) each then compares this reshaped map with its own feature
map similarly compressed by a key encoder; (3) the resulting similarity volume
is smoothed with an autoencoder smoothing network (purple trapezoids); (4)
channel-wise Max and ArgMax operators then generate a 2D matching score and
warping grid, respectively; (5) each supporting agent uses its warping grid to
sample its output distribution; and (6) the corresponding match scores are
used as the spatial attention weights during multi-agent fusion. The green
boxes highlight the ground truth information used to train the network.

###  III-A Multi-Agent Spatial Handshaking

The Multi-Agent Collaborative Perception task involves producing the most
accurate egocentric semantic segmentation prediction for one robot (or set of
robots), given bandwidth-limited access to the current observations of its
collaborators. To achieve this goal, each robot must identify complementary
visual information despite intermittently and partially overlapping fields of
view with the others. Additionally, these robots must be able to correspond
their observations in the presence of unknown obstructions (i.e. close-up
objects that are distinct from the known classes). Together, these components
probe one of the challenges that faces a networked autonomous vehicles:
namely, how can we efficiently leverage peers to improve perception
performance and resistance to visual occlusions and outliers?

The data associated with this task includes high-resolution inputs (RGB
images), ground truth outputs (semantic segmentation masks), and auxiliary
training information (pairwise pixel correspondences). Additionally,
communication bandwidth limitations prevent direct transmission of the raw
input data; therefore, information must be efficiently processed and
exchanged.

To this end, we propose an extensible neural pipeline that leverages several
components inspired by prior work. Our overall system uses segmentation
networks [2] to extract features, handshake communication [10] to compress
these features for transmission, cost volume correspondences [7, 8, 21, 4, 19]
to associate them, and attention mechanisms [18] to fuse the associated data.
In order to solve the challenging collaborative perception problem with
unknown correspondences, we introduce three significant novel aspects: (1) we
extend the 1D key-query-feature handshaking mechanism of When2Com [10] into a
2D spatial handshaking mechanism for data compression and transmission; (2) we
introduce a novel smoothing autoencoder that smooths spurious matches and
infills obstructed matching regions; and (3) we duplicate our dense
correspondence pipeline across several agents and perform multi-agent fusion
on the warped and scored segmentation results.

Architecture Overview The Multi-Agent Spatial Handshaking MASH network guides
the exchange of spatially-structured image data across several agents. The
general layout of the network consists of $N$ identical modules that are
deployed across $N$ separate agents. Each module consists of (1) a semantic
segmentation backbone, (2) a pair of query and key encoders, (3) a dense
correspondence calculation, (4) a context-based smoothing network, (5) an
image sampling operation, and (6) an output fusion operation. Each of these
$N$ modules is processed locally per robot, allowing for distributed
computation across the robot group.

Inputs and Outputs The input of MASH consists of $N$ RGB images ($H\times
W\times 3$) captured from the viewpoints of $N$ independent agents. For sake
of explanation, we denote one of these agents as the target agent and the
others as supporting agents. The final output of MASH is an obstruction-free
semantic segmentation for the target agent, where “obstruction” refers to an
unexpected (i.e. out-of-distribution) foreground object. Note that all agents
may simultaneously be a target and supporting agent, with shared computation
and communication for each role. Without loss of generality, we describe our
method for one target agent.

Semantic Segmentation Backbone We use SegNet [2] as the backbone of our
spatial handshaking network. This network architecture uses a convolutional
encoder-decoder to transform a RGB input ($H\times W\times 3$) into a full-
resolution semantic segmentation mask ($H\times W\times C$). For our purposes,
we do not alter the structure of the backbone network; rather, we extract and
forward two of its layers to the rest of MASH. Specifically, we use (A) an
intermediate spatial feature map and (B) the output semantic segmentation
distribution. We note that our SegNet backbone is interchangeable with most
semantic segmentation architectures of a similar structure.

Data Compression After extracting the intermediate feature map from the
segmentation backbone, we use a pair of encoders to further process and
compress it into a query and key feature map. These two encoders consist of a
sequence of learned convolutional kernels of spatial size $1\times 1$ which
transform the original spatial feature map $f$ ($H_{s}\times W_{s}\times F$)
into channel-reduced maps $q$ ($H_{s}\times W_{s}\times Q$) and $k$
($H_{s}\times W_{s}\times K$), respectively. Following this compression, the
target agent transmits $q$, and the supporting agent (or agents) receives $q$
and compares it against its own $k$.

This compression step reduces the bandwidth required for transmitting the
feature map $q$, and it reduces the complexity of computing dense
correspondences between the two feature maps, $q$ and $k$. Furthermore, by
having separate query and key encoders, the agents have a mechanism for
describing the input image in the form of a question (query) and an answer
(key), similar to the mechanism explored by When2Com [10]. Previously shown
for 1D feature vectors, asymmetric encoders allow the transmitted query to be
much smaller than the key, even for spatial data and dense correspondence
settings.

Data Correspondence The spatial feature maps ($q$ and $k$) from the preceding
step have asymmetrical channel dimensions ($Q$ and $K$). To compare these two
quantities, we first convolve $q$ with $K$ learned convolutional kernels of
size $1\times 1\times Q$ to produce a spatial feature map $p$. Next, using the
$p$ map from the target robot (subscript $T$) and the $k$ map from the
supporting robot (subscript $S$), we compute a dense pairwise distance volume:

$\mathcal{D}_{TS}[x,y,x^{\prime},y^{\prime}]=-d(p_{T}[x,y],k_{S}[x^{\prime},y^{\prime}]),$
(1)

where $d$ defines a distance metric (e.g. $L_{2}$) between two feature vectors
of dimension $K$. Next, we reshape this 4D similarity volume into a 3D tensor
of size $H_{T}\times W_{T}\times(H_{S}\times W_{S})$. This new representation
describes how each compressed query feature from the target robot $A$ matches
to all key features from the supporting robot $B$. Additionally, we compute a
“no-match” score:

$\mathcal{D}_{T\O}[x,y]=-d(p_{T}[x,y],\vec{0}).$ (2)

We append this “no-match” tensor ($H_{T}\times W_{T}$) to the 3D similarity
tensor, yielding a final volume of $H_{T}\times W_{T}\times(H_{S}\times
W_{S}+1)$. This added score gives the target robot a self-attention mechanism
for evaluating the descriptiveness of its own features, as used in prior work
[1, 10]. Finally, we convert these per-cell matching scores into a matching
distribution by applying the Softmax operation across the channel dimension.

To summarize the preceding three steps, first, the SegNet backbone encodes a
low-resolution, high-dimensional spatial representation of the original image.
Next, the data compression step reduces the channel dimension of this
representation for more efficient communication and computation. We then
exchange these compressed representations across agents. Finally, the data
association step compares pairs of spatial feature maps to produce a dense
correspondence volume.

Figure 3: Simplified cost volume pipeline. The target and supporting agents
encode their visual inputs into query and key spatial feature maps,
respectively. We compute a smoothed cost volume from these spatial maps. The
channel-wise argmax of this volume describes how each source pixel maps to the
target; the channel-wise max describes the strength of this mapping.

The following steps outline how we identify and exchange image patches for
multi-agent fusion:

Correspondence Smoothing and Infilling The correspondence volume produced by
the previous step highlights visually similar patches between pairs of agents.
To generate this volume, each patch from the target agent is matched
independently to all of the patches from the source agent; moreover, the
spatial location of the target patch and the matching distributions of its
neighboring patches do not factor into generating the correspondence volume.
In essence, the raw correspondence volume does not directly leverage strong
prior knowledge that relates to the spatial arrangement of image patches, such
as matching continuity and dataset priors. If a given target patch $X_{T}$
matches to a particular source patch $Y_{S}$, we would expect patches in the
neighborhood of $X_{T}$ to match to patches roughly in the neighborhood of
$Y_{S}$. Furthermore, for instance, if agents always capture gravity-oriented
images, that should serve as a strong prior for corresponding unseen image
matches.

To this end, we propose a convolutional autoencoder that smooths and infills
noisy correspondence volumes. This module takes a raw correspondence volume as
input ($H_{T}\times W_{T}\times(H_{S}\times W_{S}+1)$) and produces an
identically sized correspondence volume as output. Our convolutional
architecture exploits both the spatial arrangement of the cost volume
($H_{T}\times W_{T}$) and its corresponding matching distribution
($H_{S}\times W_{S}+1$). We train this encoder to match the ground truth
warping map using a cross entropy classification objective.

Figure 4: Warping pipeline. The patch match index is used to warp patches from
the supporting frame into the target frame. Prior to re-transmission back to
the target agent, the patch match confidence is used to mask (or weight) the
warped data. Here we visualize the pipeline using RGB images, but the same
technique applies to any image-structured data, such as segmentation masks.

Output Sampling and Fusion We use the smoothed correspondence volume
($H_{T}\times W_{T}\times(H_{S}\times W_{S}+1)$) to directly sample and weight
the output contributions of the supporting robot(s). Specifically, we compute
the channel-wise ArgMax and Max of the correspondence volume to produce a 2D
warping grid and 2D warping confidence map, as highlighted in Fig 3. The
warping grid describes how each spatial cell from a target agent $T$ could be
complemented by a corresponding cell in a source agent $S$. If the ArgMax
selects the last “no-match” channel, then the original spatial cell from $T$
is used as the complementary cell. The 2D warping confidence map describes the
confidence associated with that best match.

As shown in Fig 4, we use the 2D warping grid to align visual information from
the supporting agent(s) into the target frame. In our case, we use the warping
grid to align the pre-ArgMax semantic segmentation output distribution from
the supporting agent(s).

Next, we use the 2D match confidence map to determine how to best combine the
semantic segmentation distributions from each agent (including itself).
Namely, we want image patches that have a high confidence of being warped
properly (i.e. those patches with a large confidence map score) to more
prominently contribute to the final output. We consider several variants that
fuse the segmentation contributions from each agent:

HardSelection: For each spatial cell, hard selection uses the semantic
segmentation distribution from the agent with the strongest warping confidence
score.

SoftSelection: For each spatial cell, soft selection scales the distribution
of each agent based on its warping confidence score. The scaled semantic
segmentation distributions are then fused together via addition.

StackedFusion: Stacked fusion concatenates the warped segmentation masks from
each output. Then, it passes the resulting stack to a small convolutional
network that generates the final fused segmentation mask.

Finally, after the semantic segmentation output distributions are fused, we
generate a semantic mask prediction via channel-wise ArgMax.

In summary, we propose a neural architecture that learns both compression and
warping in an end-to-end manner, allowing us to directly train towards the
task of semantic segmentation. Moreover, our design decisions improve
resilience to obstructions (via the learned costmap autoencoder) and allow us
to efficiently deploy across any number of agents in a bandwidth-limited
collaborative swarm.

Training We require two sets of ground truth signals to train MASH: (1) per-
agent semantic segmentation masks and (2) patch correspondences between the
target and supporting agents. We use a cross entropy loss for both semantic
segmentation and patch correspondence. Additionally, though the network is
trained in a dense, centralized manner, it can be deployed in a distributed
manner. During inference, each agent receives the same trained network
weights; and unlike centralized models, full network connectivity is not a
hard requirement. We show results for optional top-1 communication (where we
communicate the feature with the best warping score, referred to as
HardSelection). We emphasize that, during inference, the only inputs to the
network are a single RGB image per robot.

Bandwidth and Computation We designed the MASH architecture to reduce
communication bandwidth while leveraging a swarm’s access to distributed
computing. Regarding communication bandwidth, we reduce the naive
communication of all raw images ($(N-1)\times H\times W\times 3$) to:

$\begin{split}\overbrace{\underbrace{(H_{s}\times W_{s}\times Q)}_{\text{Query
Map}}}^{\text{Target Agent
(Tx)}}+\underbrace{(N-1)}_{\text{Agents}}\times\overbrace{\underbrace{(H\times
W\times C)}_{\text{Segmentation Map}}\times\underbrace{r}_{\text{Selection
Factor}}}^{\text{Supporting Agent (Rx)}}\end{split}$ (3)

with $H$ and $W$ image dimensions, $C$ classes, $H_{s}$ and $W_{s}$
downsampled spatial dimensions, $Q$ compressed channel dimension ($Q\leq K$),
$N$ agents, and $r$ selection factor ($0\leq r\leq 1$).

Note, the selection factor $r$ denotes the empirical fraction of spatial
information transmitted by the supporting agent, and it reflects the degree of
overlap between pairs of agents. This factor ranges from $0$ (for pairs with
no overlap) to $1$ (for pairs with full overlap); it is generally quite small
($r<<1$), especially for our dataset that includes intermittently and
partially overlapping agents.

Regarding computation, rather than perform processing operations in a
centralized manner, we distribute and parallelize several operations across
the available agents, reducing both latency and computational burden. Please
refer to Fig 2 to see how operations are grouped and parallelized.

|  | Segmentation | MASH Parameters | Bandwidth | Compute Time
---|---|---|---|---|---
|  |  Full Image |  Obstructed Region | Query | Key | Per | Per | Per | Per
|  | Accuracy | IoU | Accuracy | IoU | Dim | Dim | Target | Supporting | Target | Supporting
Baselines | Inpainting [3] (Trained on Target View) | 50.29 | 40.70 | 27.54 | 18.66 | - | - | - | - | 1.00$t$ | -
| Inpainting [3] (Trained on All Views) | 55.24 | 44.77 | 22.85 | 15.01 | - | - | - | - | 1.00$t$ | -
| InputStack [4, 19] | 34.85 | 26.88 | 23.35 | 15.83 | - | - | - | 1.00$x$ | 1.00$t$ | -
| OutputStack | 51.35 | 40.94 | 27.65 | 17.21 | - | - | - | 1.00$y$ | 1.08$t$ | 1.00$t$
| WarpedOutputStack* [11] | 64.16 | 54.55 | 38.31 | 28.27 | - | - | - | - | - | -
MASH | vs Bandwidth Consumption | 57.95 | 43.29 | 26.16 | 16.63 | 16 | 64 | 0.08$x$ | 0.09$y$ | 1.09$t$ | 1.69$t$
|  | 57.34 | 45.68 | 25.70 | 16.86 | 32 | 64 | 0.17$x$ | 0.08$y$ | 1.09$t$ | 1.70$t$
|  | 65.15 | 55.70 | 33.59 | 23.73 | 64 | 64 | 0.33$x$ | 0.04$y$ | 1.09$t$ | 1.69$t$
|  | 64.52 | 53.76 | 31.04 | 21.63 | 64 | 128 | 0.33$x$ | 0.03$y$ | 1.09$t$ | 1.71$t$
|  | 65.60 | 56.30 | 32.45 | 22.00 | 64 | 256 | 0.33$x$ | 0.03$y$ | 1.09$t$ | 1.68$t$

TABLE I: Segmentation Results for Full Image and Obstructed Region. We compare
baselines and MASH variants using metrics such as bandwidth consumption,
compute time, mean accuracy, and mean IoU. Here, the bandwidth used by the
target agent is based on raw input size ($x=256\times 256\times 3$); the
bandwidth used by each supporting agent is based on output semantic mask size
($y=256\times 256$); and the compute time is based on SegNet’s compute time
($t\approx 0.2s$ on an Nvidia RTX 2080 GPU), averaged over 50 inference runs.
The transmitted bandwidth is determined by the channel dimension of the
compression encoders, and the received bandwidth is empirically calculated
based on the matched fraction of the image. *This baseline uses the auxiliary
ground truth warping grid (i.e. extra ground truth information) and is an
upper bound.

##  IV EXPERIMENTS

Dataset. Our dataset consists of synchronized RGB and semantic segmentation
images captured from 6 moving agents in a simulated photo-realistic urban
environment. The fields of view of these agents periodically and non-trivially
overlap, and their cameras capture both static and dynamic scene objects.
Additionally, to provide an intermediate training signal, our dataset also
includes ground truth dense pixel correspondences between pairs of images,
computed from the raw pose and depth provided by the simulator.

The task associated with this dataset is the following: using RGB data
captured from single snapshot in time (i.e. 6 synchronized images from
different parts of the swarm), the target agent must generate the most
accurate semantic segmentation mask. Additionally, we constrain this task by
restricting the direct exchange of raw RGB data; namely, the target agent may
access its own RGB data at no cost, but communication with other agents is
penalized based on bandwidth consumption. Furthermore, to better tailor our
dataset to investigating the task of perceptual resilience in a swarm, we
spawn random obstructions in the 3D scene (such as birds and trees). Each
target agent must then predict an unobstructed semantic segmentation mask,
despite the presence of a foreground obstruction.

To produce this dataset, we spawned a swarm of 6 drones in the photo-
realistic, multi-robot AirSim [15] simulator. We commanded the drones to move
roughly together throughout the environment, capturing synchronized images
along the way. To ensure dynamically overlapping views, we commanded different
drones to travel with different yaw rates. Additionally, each time the swarm
arrived at an intersection, we commanded each drone to randomly and
independently explore the intersection, further ensuring interesting frame
overlaps. We repeated this process until the swarm explored all of the road
intersections in the simulator.

Futhermore, we synthesize an obstruction-prone variant. We post-processed each
snapshot of the base AirSim dataset, randomly placing a 3D object into the
scene and rendering this object in the views of each observer. We used the
pose and depth information from each agent and the pose and depth information
of the 3D obstruction model to perform the insertion rendering.

In summary, our dataset consists of $18,588$ snapshots across $6$ non-static
robots. The robots have an average pose difference of $8.7m$ and $50^{\circ}$.
Sporadic motion and yaw of the robots creates intermittently and partially
overlapping views. A random pair of agents has an average pixel overlap of
$37\%$ with a standard deviation of $15\%$. For our experiments, we select one
of the $6$ agents as the target and allow it to communicate with any number of
the $5$ supporting agents.

Baselines. We compare against several baselines:

Inpainting uses the SegNet architecture to directly transform a (potentially
obstructed) target image into an unobstructed semantic segmentation mask, as
inspired by [3]. The model is trained to “hallucinate” a reasonable prediction
for obstructed pixels, rather than query the other frames that have a glimpse
at those occluded pixels.

InputStack concatenates the unwarped RGB images from all agents, and it
forwards this stack of images to SegNet for semantic segmentation. This
procedure resembles that of Deep Homography and Visual Odometry methods [4,
19].

OutputStack uses a shared SegNet model to process each unwarped RGB input into
a semantic segmentation output distribution. Next, it concatenates these
output distributions and then fuses them through a learned convolutional
kernel.

WarpedOutputStack is identical to OutputStack with the exception that the
output distributions are warped to the target frame using the ground truth
warping grid. This warping grid is normally provided during training only, so
this baseline approximates the upper-bound performance of a network which
learns that warping.

Results. As shown in Table I, MASH achieves superior performance on
segmentation metrics, while simultaneously improving bandwidth consumption and
computational parallelism. For ease of comparison, we show bandwidth and
compute time in terms of fixed quantities, such as input image size ($x$),
output image size ($y$), and SegNet compute time ($t$). We also show
segmentation accuracy and mean IoU evaluated both on the full image and within
its obstructed pixels. Please refer to Fig 1 for a reference on how the
obstructed region is scored–for this metric, the mean IoU score is only
computed on the pixels within the inserted obstruction (e.g. the unknown teal
eagle class in Fig 1).

Segmentation Performance. We observe a couple of interesting trends. Despite
not having access to unobstructed data, the Inpainting baselines produce
reasonable semantic masks, even in the obstructed region of the image. This
trend likely results from their ability to “hallucinate” plausible outputs
from the surrounding input context. The tested dataset also has a strong scene
prior for static classes, further enhancing the effectiveness of inpainting
methods. The InputStack and OutputStack baselines have the opposite problem to
Inpainting; even though they can access unobstructed data, their simple,
stacked architectures do not explicitly correspond complementary image
features, and therefore the extra information detracts from overall
performance. The importance of image correspondence is demonstrated by the
strong performance of the WarpedOutputStack upper bound, which uses the
otherwise inaccessible ground truth warping information to align the output
feature masks.

In the bottom of Table I, we present the results for our MASH network. Of our
three proposed multi-agent fusion mechanisms, we highlight the results for the
HardSelection mechanism, which yielded the best trade-off between segmentation
performance and bandwidth consumption. Each row shows the performance of this
variant under various bandwidth constraints. Our MASH models achieve a
$~{}11\%$ improvement in Full Image accuracy and mean IoU over several key
baselines. We similarly observe a $~{}6\%$ improvement in Obstructed Region
accuracy and mean IoU.

These significant performance gains support several of our design choices. We
emphasize that MASH does not fall prey to the shortcomings of the baselines:
our network is able to benefit from multiple, unaligned sources of image data
via a learned, data-efficient warping mechanism. Furthermore, our novel cost
volume autoencoder enables obstructed image regions to be properly
corresponded across multiple viewpoints, resulting in improved segmentation of
degraded regions.

Additionally, we demonstrate comparable performance of the strong
WarpedOutputStack upper bound. We attribute this narrow gap to the ability of
MASH to properly align contributing data as well as its ability to
intelligently select between these contributions. Both of these features are
present within the baseline, but the MASH network has a slightly deeper
network depth which yields slightly stronger inference capabilities.

Bandwidth. In the bottom of Table I, we show that we are able to tune
bandwidth consumption and segmentation performance by manipulating the channel
dimension of the transmitted query, as shown by the Bandwidth Per Target
column. As we relax our bandwidth constraint (by increasing the query
dimension), we observe an improvement in perception performance. This
improvement matches expectations, especially since each agent can then pack
more information into its query map.

By altering the query dimension, we can dictate how a target agent consumes
bandwidth. However, MASH itself moderates how each supporting agent consumes
bandwidth. Namely, the warping score produced by MASH guides how much output
information is selectively transmitted from the supporting agents to the
target agent. The Bandwidth Per Supporting column shows the amount of data (on
average across the dataset) that MASH passes from each supporting agent to the
target agent. Our dataset consists of largely non-overlapping data sources,
and MASH is able to selectively transmit data that improves its perception
performance, while avoiding a substantial fraction of distractor data.

Compute. We show that MASH yields improved performance (comparable to a ground
truth augmented variant, WarpedOutputStack) at a reasonable, distributed
computational overhead. The target agent only introduces an additional $9\%$
of compute time in addition to a backbone SegNet architecture. And each
supporting agent only incurs a $70\%$ increase in compute time. Note that
rather than burdening the target agent with each per-agent correspondence,
warping, and fusion module, these computations are effectively distributed
among the supporting agents.

##  V CONCLUSION AND FUTURE WORK

We propose a variant on the multi-agent collaborative perception task in which
robots must communicate visual information to improve their local perception
performance. We adapt this general task to target bandwidth-limited and
obstruction-prone semantic segmentation for networked vehicles. To identify
and exchange relevant information in this restricted multi-agent setting, we
introduce a neural network architecture that uses learned (1) spatial
handshake communication, (2) cost volume correspondence, (3) context-aware
cost volume smoothing, and (4) attention-based output fusion. Furthermore, we
show that our model is able match the performance of warping-based upper
bound, with reduced bandwidth and computational overhead. Future work in this
domain will explore sparse image correspondence methods to further reduce
transmission of unnecessary matching data between agents. Our work primarily
addresses the task of multi-agent perception without active control; however,
future extensions will involve active multi-agent perception and in-the-loop
control.

##  VI Acknowledgement

This work was supported by ONR grant N00014-18-1-2829.

## References

  * [1] Gil Avraham, Yan Zuo, Thanuja Dharmasiri and Tom Drummond  “EMPNet: Neural Localisation and Mapping Using Embedded Memory Points”, 2019, pp. 8119–8128  DOI: 10.1109/ICCV.2019.00821
  * [2] Vijay Badrinarayanan, Alex Kendall and Roberto Cipolla  “Segnet: A deep convolutional encoder-decoder architecture for image segmentation”  In _IEEE transactions on pattern analysis and machine intelligence_ 39.12 IEEE, 2017, pp. 2481–2495
  * [3] Nian Cai et al.  “Blind inpainting using the fully convolutional neural network”  In _The Visual Computer_ 33.2 Springer, 2017, pp. 249–261
  * [4] Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich  “Deep image homography estimation”  In _arXiv preprint arXiv:1606.03798_ , 2016
  * [5] Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich  “Superpoint: Self-supervised interest point detection and description”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 224–236
  * [6] Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich  “Toward geometric deep SLAM”  In _arXiv preprint arXiv:1707.07410_ , 2017
  * [7] Philipp Fischer et al.  “Flownet: Learning optical flow with convolutional networks”  In _arXiv preprint arXiv:1504.06852_ , 2015
  * [8] Eddy Ilg et al.  “Flownet 2.0: Evolution of optical flow estimation with deep networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2462–2470
  * [9] F. Li, G… Ricardez, J. Takamatsu and T. Ogasawara  “Multi-view Inpainting for RGB-D Sequence”  In _2018 International Conference on 3D Vision (3DV)_ , 2018, pp. 464–473  DOI: 10.1109/3DV.2018.00060
  * [10] Yen-Cheng Liu, Junjiao Tian, Nathan Glaser and Zsolt Kira  “When2com: Multi-agent perception via Communication Graph Grouping”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [11] Yen-Cheng Liu et al.  “Who2com: Collaborative Perception Via Learnable Handshake Communication”  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2020
  * [12] Iaroslav Melekhov et al.  “Dgc-net: Dense geometric correspondence network”  In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2019, pp. 1034–1042  IEEE
  * [13] Pulak Purkait, Christopher Zach and Ian D. Reid  “Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions”  In _CoRR_ abs/1906.02885, 2019  arXiv: http://arxiv.org/abs/1906.02885
  * [14] Ignacio Rocco, Relja Arandjelovic and Josef Sivic  “Convolutional neural network architecture for geometric matching”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6148–6157
  * [15] Shital Shah, Debadeepta Dey, Chris Lovett and Ashish Kapoor  “AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles”  In _Field and Service Robotics_ , 2017  URL: https://arxiv.org/abs/1705.05065
  * [16] Hang Su, Subhransu Maji, Evangelos Kalogerakis and Erik Learned-Miller  “Multi-view convolutional neural networks for 3d shape recognition”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 945–953
  * [17] Theo Thonat, Eli Shechtman, Sylvain Paris and George Drettakis  “Multi-View Inpainting for Image-Based Scene Editing and Rendering”, 2016, pp. 351–359  DOI: 10.1109/3DV.2016.44
  * [18] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ , 2017, pp. 5998–6008
  * [19] Sen Wang, Ronald Clark, Hongkai Wen and Niki Trigoni  “Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks”  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 2043–2050  IEEE
  * [20] Yang Wang et al.  “Occlusion aware unsupervised learning of optical flow”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4884–4893
  * [21] Jia Xu, René Ranftl and Vladlen Koltun  “Accurate optical flow via direct cost volume processing”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1289–1297


# Development and accuracy evaluation of Coded Phase-shift 3D scanner

Pranav Kant Gaur  Computer Division
Bhabha Atomic Research Centre
Mumbai,INDIA
Email<EMAIL_ADDRESS>D.M.Sarode  Computer Division
Bhabha Atomic Research Centre
Mumbai,INDIA
Email<EMAIL_ADDRESS>S.K.Bose  Computer Division
Bhabha Atomic Research Centre
Mumbai,INDIA
Email<EMAIL_ADDRESS>
###### Abstract

In this paper, we provide an overview of development of a structured light
3D-scanner based on combination of binary-coded patterns and sinusoidal phase-
shifted fringe patterns called Coded Phase-shift technique. Further, we
describe the experiments performed to evaluate measurement accuracy and
precision of the developed system. A study of this kind is expected to be
helpful in understanding the basic working of current structured-light 3D
scanners and the approaches followed for their performance assessment.

##  I Introduction

Phase-shift approach[1][2] encodes the projector pixel coordinates with the
phase of projected sinusoidal fringe patterns. When this pattern is projected
onto the scene of interest it codifies the scene as well using the phase of
the incident sinusoidal signal. Camera captures the scene with projected
patterns. Recovery of phase information at all captured points in the scene
provides information regarding source projector pixel(using phase value).
Hence stereo-correspondence between camera and projector pixels can be
established. But since the sinusoidal patterns are periodic in nature, the
phase value repeats after every $2\pi$(or a period) therefore binary coded
patterns[3] are used in addition to assist in recovering original phase at
each point in the captured scene by assigning unique period number to each
cycle of sinusoidal fringe[4]. This approach eliminates the ambiguity due to
periodicity of sinusoidal fringes.
This combined approach has the advantage of high accuracy of phase-shift
technique and high robustness against noise as of binary-coded pattern based
technique[3].
Development work to realize this approach was initiated because in our
knowledge there is no Open-source implementation of this technique which can
be used as a starting point for accurate 3D measurements. Authors in [5-6]
have developed such systems but in [5] there are no provisions for system
calibration and the work found in [6] do not have documentation and is
developed only for Windows platform.
Furthermore, the performance of a metrology equipment has been conventionally
measured in terms of its measurement accuracy and precision(or repeatability).
Recently, there have been many attempts to evaluate performance of another
popular 3D sensor Microsoft Kinect[7-10]. Same methods can be applied to
evaluate performance of our developed system. But these approaches assess
measurement accuracy either with respect to 3D data obtained from laser
scanner[7] which requires accurate calibration of laser scanner itself, or
confirm to VDI/VDE 2634 standard which requires accurate fabrication of sphere
balls and hexagonal structures which again requires sophisticated
fabrication[9]. To avoid such overheads, this paper reports a simple and
straightforward method for assessing measurement accuracy. Further, we report
the precision of the developed system.
In section II, process for estimating stereo- correspondence using Coded
phase-shift technique is described followed by description of process of
system calibration and finally, triangulation. In section III, approach used
for evaluation of measurement accuracy and precision has been described.
Section IV concludes the paper.

##  II Development

A structured-light 3D scanner based on optical triangulation(shown in Figure
1) needs to determine the correspondence between camera and projector pixels.
This goal is achieved by projection of a known pattern on object of interest.
This pattern(e.g. light stripes in Figure 1) assign a unique code to each
point on the surface of object. Camera captures a view of scene and recovers
this code at each pixel. This process indirectly relates a projector
column(strip number in Figure 1) or row or a unique combination of row and
column to a camera pixel.
Once the correspondence is known, equations for optical rays emanating from
any corresponding pair of camera and projector pixels needs to be known in
real world units, for which system calibration is performed.
Given stereo-correspondence and system calibration information, optical
triangulation can be performed to compute 3D coordinates for each real-world
point seen by both camera and projector.

Figure 1: Optical triangulation(Source:Wikipedia)

###  II-A System setup

In this work, we have used Logitech Quickcam Sphere AF webcam at 1600X1200
resolution, Sharp PG-F200X projector at 1024X768 resolution. Figure 2 shows
our system setup.

Figure 2: System setup

###  II-B System calibration

Camera and projector calibration is a process of estimating their
intrinsic(i.e. focal length:($f_{x},f_{y}$), principle-point:($c_{x},c_{y}$))
and extrinsic geometry(i.e. rotation:($r_{x},r_{y},r_{z}$),
translation:($t_{x},t_{y},t_{z}$))[11-13].
In this work we have used OpenCV library for camera calibration and pose
estimation algorithms for intrinsic and extrinsic calibration respectively.
For projector intrinsic calibration we have used VPCLib[14] since we have
experimentally observed unacceptably lower repeatability of OpenCV calibration
algorithm when used for projector.

####  II-B1 Camera calibration

A checkerboard of known dimension is shown at different distances and
orientations with respect to camera. Figure 10 shows the plot of the views
used for camera calibration. This process gives sample 2D image point and
corresponding 3D points. This information is used to estimate calibration
parameters using OpenCV method. Table I shows the estimated calibration
parameters for camera.

####  II-B2 Projector calibration

Projector can be modeled as inverse-camera[15]. Projector calibration is
performed with camera as a feedback device. Initially, camera-to-screen(or any
planerboard)homography(projective-mapping between 2 planes) is computed. Then
projector projects the checkerboard pattern. Camera captures the projected
checkerboard pattern and detects its inner-corners. Using camera-to-screen
homography world coordinates for detected checkerboard corners are computed.
This process gives set of 2D-3D correspondences required for projector
calibration. Thereafter, an identical procedure as of camera calibration can
be applied for estimating projector calibration parameters. Figure 11 shows
plot of the views used for projector calibration. Table I shows estimated
calibration parameters for projector.

####  II-B3 Extrinsic camera-projector calibration

To bring optical-ray from a camera-pixel and optical-ray emanating from
corresponding projector-pixel to a common coordinate system, relative rotation
and translation between camera and projector coordinate system needs to be
known. This is required for performing optical triangulation since computed
3D-coordinate is the intersection point of optical rays from camera and
projector, requiring both rays to be in a common coordinate system.
Since intrinsic parameters are already known for both projector and camera,
similar procedure as used for camera(and projector) calibration is applied but
for a single view of physical(and projected) checkerboard to get 2D-3D
mappings for camera and projector separately. These mapping are used to
estimate rotation and translation transformation for camera and projector
coordinate systems with respect to world coordinate system. These are then
combined to get projector-to-camera rotation and translation parameters.

TABLE I: Camera and Projector intrinsic calibration parameters Parameter | Camera | Projector
---|---|---
$f_{x}$ | 1362.2 | 2261.7
$f_{y}$ | 1372.2 | 2262.8
$c_{x}$ | 803.9 | 522.7
$c_{y}$ | 590.1 | 713.8
$k_{1}$ | 0.07 | 0.0
$k_{2}$ | -0.14 | 0.0

###  II-C Stereo correspondence

Once system intrinsic and extrinsic geometry is defined, stereo-correspondence
which pairs the points in camera and projector viewing a common 3D point is
estimated. Following subsections describe the working of modules for
estimating stereo-correspondence.

####  II-C1 Pattern generation module

This module generates phase-shifted sinusoidal fringes and binary- coded
patterns. Equation 1 shows the relation used for generating the phase-shifted
sinusoidal fringes.

$\displaystyle I_{1}=I_{dc}+I_{mod}*cos(\phi+\theta)$ (1) $\displaystyle
I_{2}=I_{dc}+I_{mod}*cos(\phi)$ $\displaystyle
I_{3}=I_{dc}+I_{mod}*cos(\phi-\theta)$

where, $I_{1}$,$I_{2}$,$I_{3}$ represent 3 sinusoidal signals at any point
represented by phase $\phi$ and successively shifted in phase by $\theta$.
$I_{dc}$ models a biasing factor, $I_{mod}$ models the modulation intensity of
the sinusoid. Further, binary-coded patterns are designed such that width of a
bit-plane equals the width of a fringe(or one sinusoidal cycle).

####  II-C2 Pattern projection and capture module

This module sequentially projects and captures the sinusoidal phase-shifted
and binary-coded patterns. Figure 3 shows some captured vertical and
horizontal fringe and binary coded pattern images.

|  |
---|---
|

(a)

(b)

(c)

(d)

Figure 3: Captured fringe and binary patterns

####  II-C3 Phase wrapping module

As already mentioned, periodic nature of sinusoidal leads to a recovered value
of phase which repeats itself after period of $2\pi$. This makes
correspondence ambiguous since multiple points with common phase value exists.
This situation is shown in (2) where computed phase wraps-up after every
$2\pi$ interval. Hence, the computed phase is called wrapped phase. Figure 4
shows wrapped phase across the scene of interest.

$\phi=tan^{-1}\bigg{[}\frac{\sqrt[2]{3}(I_{1}-I_{3})}{2I_{2}-I_{1}-I_{3}}\bigg{]}$
(2)

where, $-\pi\leq\phi\leq\pi$

|  |
---|---

(a) Vertical wrapped phase

(b) Horizontal wrapped phase

Figure 4: Computed wrapped phase

####  II-C4 Phase unwrapping module

Wrapped phase computed in (2) repeats its value after every 2$\pi$ interval
thereby making discrimination between pixels from different intervals non-
trivial. This problem can be resolved by assigning unique period number to
each sinusoidal cycle(or interval). As described earlier, binary coded
patterns with width of a bit-plane equal to the width of a fringe(or one
sinusoidal cycle) are used for this purpose. Therefore the true phase of
incident signal at any camera pixel can be represented as in equation 3. Since
wrapped phase is unrolled/unwrapped by this process, computed phase in this
process is called unwrapped phase. Figure 5 shows the corresponding vertical
and horizontal unwrapped phase.

$\delta(x,y)=\phi(x,y)+2\pi*C(x,y)$ (3)

where, $\delta(x,y)$ represents the unwrapped phase at pixel (x,y) and C(x,y)
represents the decoded binary code at pixel (x,y)

|  |
---|---

(a) Vertical unwrapped phase

(b) Horizontal unwrapped phase

Figure 5: Computed unwrapped phase

Typically, thresholding techniques are used to recover the codeword at each
pixel from the captured images. Major issue in accurate codeword extraction is
faced at edges of the strips in these patterns. Practically, this region has a
smooth gradient instead of a hard edge, leading to ambiguity in codeword
extraction in this region as mentioned in [16-17].

####  II-C5 Absolute phase computation

Vertical unwrapped phase $\delta_{v}(x_{c},y_{c})$ gives projector
X-coordinate($X_{p}$) whereas horizontal unwrapped phase
$\delta_{h}(x_{c},y_{c})$ gives projector Y-coordinate($Y_{p}$) corresponding
to camera-pixel $(x_{c},y_{c})$ for a fringe width $w_{fringe}$. Combining
this information gives projector pixel coordinates corresponding to camera
coordinates. Equation (4) explains this camera-to-projector coordinate
mapping. Figure 6 shows one example of estimated stereo-correspondence.

$\displaystyle X_{p}=\lfloor
w_{fringe}*\big{(}\frac{\delta_{v}(x_{c},y_{c})}{2\pi}\big{)}\rfloor$ (4)
$\displaystyle Y_{p}=\lfloor
w_{fringe}*\big{(}\frac{\delta_{h}(x_{c},y_{c})}{2\pi}\big{)}\rfloor$

|  |
---|---

(a) Camera image

(b) Projector image

Figure 6: Stereo correspondence between camera and projector: green spot in
(b) corresponds to selected point(cursor) in (a)

###  II-D Triangulation

System calibration parameters and camera-projector pixel-to-pixel
correspondence information can be used to compute the ray-ray intersections
[18]. Solution to these equations will give the 3D coordinates of real world
point with respect to world coordinate system. Figure 7 shows a example of 3D-
reconstructions obtained after solving these equation. However, non-linear
response of projector to input voltage was found to be adding waviness in the
3D reconstruction. This effect was also observed in [19].

|  |
---|---

(a) 2D face image

(b) 3D view

Figure 7: 3D scan of a human face

##  III Accuracy Evaluation

To evaluate measurement accuracy of system, 3D-distance between selected
feature-points in a planer checkerboard were measured and compared against
their true values. The distance between camera-projector base-line and
measurement object was decided based on common depth-of-field of camera-
projector system such that acceptably sharper details of projected patterns
can be acquired. In our case it was $\sim 2.2m$. Figure 8 shows the system
pose used for measurement experiment.

Figure 8: System pose for 3D measurement experiment

Four inner-corners A,B,C,D (shown in Figure 9) were used to define lengths
AB,BC,CD,DA,AC,BD. Measurement object was scanned 10 times to reduce the
effect of non-systematic errors in the measurements. Average percentage
absolute relative error defined in equation (5) was used as a measure of
accuracy of the system.

Figure 9: Measurement object with corners A,B,C,D marked in red

$Accuracy=\frac{\sum_{i=1}^{N}\Bigg{[}\bigg{(}\frac{|actual_{i}-measured_{i}|}{actual_{i}}\bigg{)}*100\Bigg{]}}{N}$
(5)

where, $|x|$ denotes the absolute value of $x$ , $actual_{i}$ is the true
value for $i^{th}$ length measurement, $measured_{i}$ is the corresponding
estimated value using 3D scanner data. $N$ denotes the total number of
measurements. To assess precision of the system, we determined average of %
deviation of those 10 samples(6 measurements per sample) with respect to their
mean values as defined in equation (6).

$Precision=\frac{\sum_{p=1}^{vp}\Bigg{[}\frac{\sum_{i=1}^{vs_{p}}\Bigg{[}\bigg{(}\frac{|mean_{p}-sample_{i}|}{mean_{p}}\bigg{)}*100\Bigg{]}}{vs_{p}}\Bigg{]}}{vp}$
(6)

where, $vp$ denotes total number of length measurements, $vs_{p}$ denotes
total number of samples for $p^{th}$ length measurement, $mean_{p}$ and
$sample_{i}$ denote the corresponding mean value and the $i^{th}$ sample
respectively. Table II summarizes the observed results for measurement
accuracy and precision for our system with measurement object at a distance of
$\sim 2.2m$ from camera-projector baseline.

TABLE II: Measurement accuracy and precision of developed coded phase-shift 3D-scanner Metric | Value(in %)
---|---
Measurement accuracy | 0.61
Precision | 0.29

##  IV Conclusion

We have described a system for 3D scene reconstruction based on coded phase
shift approach. Measurement accuracy and precision of system was evaluated and
found to be within 1% of true and mean measurements respectively. Developed
system is designed to be experimental in nature allowing modification in
various structured light and system calibration parameters. This will provide
us a platform to investigate the problems related with system calibration
specifically the effect of relative pose of camera and projector used for
calibration on its accuracy(technically, sensor-planning). In addition, it
will allow us to study the problem of projector-camera system non-linearity
which is resulting in waviness in the 3D scan results. Recently, [20-21] have
reported studies on effect of global illumination and projector defocus on
accuracy of binary coded and phase shifting algorithms for stereo
correspondence. These works have motivated our ongoing investigation of these
issues in order to establish objective criteria for selecting a particular
spatial frequency for binary coded and phase shifting patterns. This will
reduce requirement of manual tweaking of system parameters to get optimal 3D
measurement accuracy. Further, we have planned to extend the system to be able
to do accurate 360 degree scans of the object.

## Acknowledgment

Authors would like to thank Computer Division,Bhabha Atomic Research Centre
technical staff and administration for providing them support and facilities
to pursue this work.

## References

  * [1] S.Zhang and S. Yau, “High-resolution, real-time 3D absolute coordinate measurement based on a phase- shifting method”, Opt. Express 14, 2644-2649 (2006).
  * [2] S.Zhang. High-resolution, Real-time 3-D Shape Measurement, PhD thesis,Stony Brook University,(2005)
  * [3] Jason Geng, “Structured-light 3D surface imaging: a tutorial”, Adv. Opt. Photon. 3, 128-160 (2011)
  * [4] Sansoni,G., Carocci,M., and Rodella,R.(1999). “Three- Dimensional Vision Based on a Combination of Gray-Code and Phase-Shift Light Projection: Analysis and Compensation of the Systematic Errors,” Appl. Opt. 38, 6565-6573 .
  * [5] ‘Structured-light’-Google code. http://code.google.com/p/structured-light/. Accessed 5 December 2012
  * [6] ‘open-light’-Google code http://code.google.com/p/open-light/. Accessed 5 December 2012
  * [7] Khoshelham, K., Elberink, S.O.(2012). Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications. Sensors, 12, 1437-1454.
  * [8] Molnar,B., Toth,C.K., and Detrekoi,A.(2012). ACCURACY TEST OF MICROSOFT KINECT FOR HUMAN MORPHOLOGIC MEASUREMENTS, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XXXIX-B3, 543-547, doi:10.5194/isprsarchives-XXXIX- B3-543-2012
  * [9] Boehm,J.(2012).NATURAL USER INTERFACE SENSORS FOR HUMAN BODY MEASUREMENT, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XXXIX-B3, 531-536, doi:10.5194/isprsarchives-XXXIX- B3-531-2012
  * [10] Chow,J.C.K.,Ang,K.D.,Lichti,D.D.,and Teskey,W.F. (2012).PERFORMANCE ANALYSIS OF A LOW- COST TRIANGULATION-BASED 3D CAMERA: MICROSOFT KINECT SYSTEM. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B5 XXII ISPRS Congress
  * [11] Tsai,Roger Y. (1987) :A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4,August 1987, pp. 323-344.
  * [12] Zhang,Z.(2000).A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000
  * [13] Camera Calibration Toolbox For Matlab http://www.vision.caltech.edu/bouguetj/calib_doc/index.h tml. Accessed:3 December 2012
  * [14] J. Draréni, S. Roy, and P. Sturm, “Methods for Geometrical Video Projector Calibration”, Machine Vision and Applications, 2012.
  * [15] Song Zhang and Peisen S. Huang.“Novel method for structured light system calibration”, Opt. Eng. 45(8), 083601 (August 21, 2006). ; http://dx.doi.org/10.1117/1.2336196
  * [16] Trobina, Marjan. “Error model of a coded-light range sensor”. Technique Report, Communication Technology Laboratory (1995).
  * [17] Qican Zhang, Xianyu Su, Liqun Xiang, Xuezhen Sun, 3- D shape measurement based on complementary Gray- code light, Optics and Lasers in Engineering, Volume 50, Issue 4, April 2012, Pages 574-579, ISSN 0143-8166, 10.1016/j.optlaseng.2011.06.024.
  * [18] Brett R.,Jones(2010).AUGMENTING COMPLEX SURFACES WITH PROJECTOR-CAMERA SYSTEMS,M.S.Thesis,University of Illinois at Urbana- Champaign
  * [19] T. Hoang, B. Pan, D. Nguyen, and Z. Wang, “Generic gamma correction for accuracy enhancement in fringe- projection profilometry”, Opt. Lett. 35, 1992-1994 (2010).
  * [20] M. Gupta, S. K. Nayar, “Micro Phase Shifting”, cvpr, pp.813-820, 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012
  * [21] Gupta, Mohit and Agrawal, Amit and Veeraraghavan, Ashok and Narasimhan, SrinivasaG. “A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus”, International Journal of Computer Vision, Vol. 102,no. 1-3,pp. 33-55,2013

Figure 10: Views used for camera calibration Figure 11: Views used for
projector calibration