(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  National Taiwan University
11email<EMAIL_ADDRESS>22institutetext: NVIDIA
22email<EMAIL_ADDRESS>

# DOrA: 3D Visual Grounding
with Order-Aware Referring

Tung-Yu Wu 11 0000-1111-2222-3333 Sheng-Yu Huang 11 1111-2222-3333-4444 Yu-
Chiang Frank Wang 1122 2222–3333-4444-5555

###### Abstract

3D visual grounding aims to identify the target object within a 3D point cloud
scene referred to by a natural language description. While previous works
attempt to exploit the verbo-visual relation with proposed cross-modal
transformers, unstructured natural utterances and scattered objects might lead
to undesirable performances. In this paper, we introduce DOrA, a novel 3D
visual grounding framework with Order-Aware referring. DOrA is designed to
leverage Large Language Models (LLMs) to parse language description,
suggesting a referential order of anchor objects. Such ordered anchor objects
allow DOrA to update visual features and locate the target object during the
grounding process. Experimental results on the NR3D and ScanRefer datasets
demonstrate our superiority in both low-resource and full-data scenarios. In
particular, DOrA surpasses current state-of-the-art frameworks by 9.3% and
7.8% grounding accuracy under 1% data and 10% data settings, respectively.

###### Keywords:

3D Computer Vision 3D Visual Grounding

##  1 Introduction

Visual grounding is an emerging task that aims to ground a target object in a
given 2D/3D scene from a natural description, where the description contains
information to identify the target object (e.g., color, shape, or relations to
other anchor objects). This task is potentially related to industrial
applications to AR/VR and robotics [27, 25, 3]. Compared to object detection,
the main challenge of visual grounding lies in the requirement to find the
only one object described in the given natural description, while there might
be multiple objects with the same class of the target object appearing in the
scene. Therefore, the model is expected to identify the relations between all
objects in the scene to find the ideal target object according to the given
description. In recent years, significant progress has been made in image-
based 2D visual grounding [23, 51, 29, 38, 45, 54]. However, comparatively
fewer efforts are directed towards addressing the more intricate challenge of
3D visual grounding, raised from joint consideration of the unstructuredness
of natural language descriptions and scattered object arrangements in the 3D
scene. The complications of the two modalities make it challenging to directly
refer to the target object with plain cross-modal interaction between the
features of the scene point cloud and the referring description, showing the
need for additional research in the field of 3D visual grounding.

As pioneers of 3D visual grounding, Referit3D [2] and ScanRefer [8] are two
benchmark datasets that build upon the point cloud scene provided by the
ScanNet [12] dataset. The former presents a graph neural network (GNN)
[36]-based framework to explicitly learn the object spatial relations as the
baseline. The latter designs a verbo-visual cross-modal feature extraction and
fusion pipeline as the baseline. Several subsequent methods are presented [20,
46, 4, 31, 21, 15, 19, 11, 40], which either augment the visual features [20,
46, 4, 31], utilize specialized cross-modal feature alignment frameworks [21],
or exploit spatial relations with designed frameworks/modules [17, 15, 19, 11,
40] are proposed to solve the 3D visual grounding problem. However, these
approaches use a referring head to localize the target object directly.
Without explicitly considering any additional information about the anchor
objects mentioned in the description, models must implicitly discover the
relation between the anchor objects and the target object. They may be misled
by other similar objects presented, as pointed out in [5]. To overcome this
issue, some approaches [49, 1, 42] propose to incorporate anchor objects
during training by including their label annotations [18, 5]. Nonetheless,
human annotators are usually required to obtain this additional linguistic
information [49, 1], causing potential difficulty in scaling up to larger
datasets for real-world applications.

To unload the need for human annotators, recent approaches [18, 5] leverage
large language models (LLMs) for automatic dissection of the descriptions and
generation of prior linguistic knowledge. For example, NS3D [18] utilizes
Codex [10] to parse descriptions into nested expressions and designs a neuro-
symbolic framework to find the target object step-by-step. However, NS3D is
restricted to fixed-template relations between objects (e.g., below/above,
near/far, etc.) and cannot be easily extended to arbitrary natural
descriptions. Inspired by the mechanism of human perception system [32, 9],
CoT3DRef [5] generates the referential order of a description that points from
anchor objects to the final target object. For example, for a description
“Find the water bottle on the table nearest to the door.”, the referential
order is generated as { “door” (anchor), “table” (anchor), “water bottle”
(target) }. A rule-based algorithm is presented to establish matching between
each class name in the referential order and a corresponding anchor/target
object in the scene at once, with the matching results being used to guide a
transformer-based module to predict the final target object. However, if the
matching results calculated by the rule-based method are not accurate, the
performance might be degraded.

In this paper, we propose a 3D Visual Grounding Framework with Order-Aware
Referring (DOrA). By utilizing a Large Language Model (LLM) of GPT-3.5-Turbo
[48], DOrA exploits the awareness of anchor objects with the associated
referential order from the textual description, as depicted in
LABEL:fig:teaser. With this referential order of anchor objects as guidance, a
series of Object Referring blocks is deployed in DOrA to process the extracted
anchor/target objects, each performing feature enhancement to update the
visual features of corresponding objects. Since only the ground-truth identity
of the target object is available during training (no ground-truth referential
order observed), we additionally introduce a pre-training strategy to augment
accurate labels and referential orders for anchor/target objects. By
conducting extensive experiments on real-world benchmark datasets, we show
that our DOrA is comparable to current state-of-the-art 3D visual-grounding
methods on the standard setting and further outperforms them significantly
when training data is limited.

We now summarize our contribution as follows:

  * •

We present a 3D Visual Grounding Framework with Order-Aware Referring (DOrA)
for 3D visual grounding via natural descriptions.

  * •

With a two-stage in-context learning technique on LLM, DOrA infers a series of
anchor/target objects in a referential order to guide the grounding process.

  * •

DOrA deploys a series of Object-Referring blocks that follows the referential
order and progressively refines the features of anchor/target objects for
improved grounding performances.

  * •

A pre-training scheme is introduced for DOrA, which augments reliable labels
and accurate referential orders of anchor/target objects as additional
training examples, ensuring the awareness of the identities and referential
orders of anchor objects.

##  2 Related Work

###  2.1 2D Visual Grounding

2D visual grounding aims to locate the target object in an image referred to
by a natural language description, with various approaches being proposed in
recent years [23, 28, 51, 29, 47, 39, 38, 45]. Among them, verbo-visual
feature alignment frameworks [23, 28, 51, 29] have proven themselves to be an
effective way to equip models with abilities of tackling description contexts
and image semantics simultaneously. Particularly, as one of the pioneers,
MDETR [23] extends DETR [7], an end-to-end object detection framework, to
incorporate text modalities with the proposed text-image alignment contrastive
losses. GLIP [51] takes a step forward to improve the performance of visual
grounding by proposing unified multi-task learning that includes object
localization and scene understanding tasks, showing that these tasks could
gain mutual benefits from each other. Grounding DINO [29] further designs the
large-scale grounding pretraining for DINO [16], reaching the capability of
open-set grounding. Although great progress is achieved, extending these 2D
visual grounding methods to 3D scenarios is not easy due to the additional
depth information in 3D data that triggers more complicated object
arrangements and more complex descriptions to describe the relations between
objects, leaving 3D visual grounding as an unsolved research area.

###  2.2 3D Visual Grounding

In 3D visual grounding, models are designed to jointly handle complicated
natural language descriptions and scattered objects within a point cloud
scene. Previous approaches attempt to solve this task by either constructing
text-point-cloud feature alignment frameworks [21], designing pipelines to
better exploit the 3D spatial relations of objects [50, 2, 15, 19, 44, 11], or
bringing in auxiliary visual features [46, 4, 31, 20]. Specifically, BUTD-DETR
[21] extends MDETR [23] to 3D visual grounding by adapting a text-point-cloud
alignment loss to pull the features of point cloud and text together. To
exploit the 3D spatial relations, graph-based methods [50, 2, 15, 19] utilize
GNNs to model the 3D scene, with nodes and edges being the objects and object-
to-object relations, to learn their correlations explicitly. Also, some
studies craft specialized modules [17, 44, 11, 40], such as the spatial self-
attention presented in ViL3DRel [11] and relation matching network in
CORE-3DVG [44], to capture spatial relations among objects.

To better identify the target object of interest, some [46, 4, 31, 20] intend
to produce richer input semantic information for the model. For example, [46,
4, 31] introduce image features by acquiring 2D images of the scene to obtain
more color/shape information. On the other hand, MVT [20] projects the point
cloud into multiple views for more position information. Although these
approaches have achieved great progress in dealing with scattered object
arrangements, such methods typically extract a global, sentence-level feature
[13, 30] from the given natural description. As a result, detailed information
within the description (_e.g_., the target object, anchor objects, and their
relations) may not be preserved and leveraged properly, potentially reducing
training efficiency and prediction accuracy of the model as discussed in [5].

To address the above issue, some works have put their efforts into mining the
natural descriptions to acquire additional prior knowledge for more efficient
and effective learning [49, 1, 42, 18, 5]. Specifically, ScanEnts3d [1] and
3DPAG [49] recruit human annotators to establish one-to-one matching between
each anchor object mentioned in the description and the corresponding object
entity in the 3D scene. With this additional information, they design dense
word-object alignment losses to improve the training. However, annotating one-
to-one correlation requires considerable labor. For example, it takes 3664
hours of workforce commitment in 3DPAG to annotate anchor objects for 88k
descriptions. To eliminate the need for human annotators, NS3D [18] makes the
first attempt to use the LLM. It leverages Codex [10] to parse fixed-template
descriptions into nested logical expressions, followed by a neural-symbolic
framework to execute the logical expressions implemented as programmatic
functions.

Unfortunately, NS3D cannot handle arbitrary natural descriptions well, which
might generate complicated expressions and unforeseen functions, hindering the
framework from successful execution [18]. Recently, CoT3DRef [5] proposes to
acquire the referential order of the description, listing from anchor objects
to the final target object, using LLMs. Also, the authors design a rule-based
searching method using a traditional sentence parser [37] to construct the
one-to-one matching between class names in the order and potential
anchor/target objects in the scene at once. The matching information is
encoded by positional encoding as additional inputs for the proposed
transformer-based CoT module to learn the referring process implicitly and
predict the final target object. Nevertheless, since rule-based matching is
not guaranteed to be correct, such additional inputs might result in noisy
information and degrade the grounding process, impeding correct target
prediction and affecting training stability.

##  3 Methodology

###  3.1 Problem Formulation and Framework Overview

####  3.1.1 Problem formulation

We first define the setting and notations used in this paper. For each indoor
scene, we have a set of colored point cloud $C\in\mathbb{R}^{N\times 6}$,
where $N$ denotes the number of the points in the scene, with each point
represented in terms of its three-dimensional coordinate and RGB spaces. $C$
is processed to acquire $K$ object proposals
$P=\\{\boldsymbol{p}_{1},\ldots,\boldsymbol{p}_{K}\\}$ that represent possible
objects in the scene, with each proposal containing $I$ points (_i.e_.,
$\boldsymbol{p}_{n}\in\mathbb{R}^{I\times 6}$, $n\in[1,\cdots,M]$ ). $P$ is
obtained either by pre-trained object segmentation networks [22, 34, 43] or
directly from the dataset. Along with $P$, the class labels
$L=\\{l_{1},\cdots,l_{K}\\}$ for all proposals are additionally predicted by a
Pointnet++ [35] classifier. For 3D grounding, a text description $D$ is given,
illustrating the target object in $C$ by describing its color, shape, or
relations to other anchor objects. Given the above inputs, our goal is to
identify the exact target object that matches $D$ among all objects in the
scene by predicting a $K$-dimensional confidence score
$S=\\{s_{1},\cdots,s_{K}\\}$ for classifying the target object.

####  3.1.2 Framework overview

As shown in Fig. 1, DOrA is composed of $B$ consecutive Object-Referring
blocks $\\{R_{1},\cdots,R_{B}\\}$ to progressively locate the target object.
In particular, by taking both $P$ and $D$ as the inputs, DOrA utilizes the
Object-Referring blocks $R_{1:B}$ in Fig. 1 to sequentially produce anchor
objects to guide the grounding process. Each $R_{i}$ takes the object feature
$F_{i}\in\mathbb{R}^{K\times d_{i}}$ and the text feature
$T\in\mathbb{R}^{(|D|+1)\times 768}$ as the inputs. Note that $T$ contains a
${1\times 768}$-dimensional sentence-level feature and ${|D|\times
768}$-dimensional word-level feature, where $|D|$ denotes the length of $D$
after tokenization. By observing $F_{i}$ and $T$, the Object-Referring block
$R_{i}$ aims to produce the refined feature $F_{i+1}$ along with the updated
anchor/target objects and their relations for grounding purposes.

To enable our Object-Referring blocks to capture proper information about the
anchor/target objects, we apply a Large Language Model (LLM) to $D$ to
generate a Referential Order $O_{1:B}=\\{O_{1},\cdots,O_{B}\\}$ that mimics
human perception system of searching target object [32, 9] by extracting and
arranging the class names of the anchor and target objects. Specifically,
$\\{O_{1},\cdots,O_{B-1}\\}$ represent the class names of the anchor objects,
and $O_{B}$ is the class name of the target object. Note that $O_{i:B}$ is
observed by the $i$-th Object-Referring block $R_{i}$ as guidance, which
updates the features of anchor/target objects with the proposed Feature
Enhancing (FE) module. Since there is no ground truth referential order of the
produced anchor objects, we also introduce a pre-training strategy for
training DOrA. This is achieved by synthesizing accurate referential order and
anchor/target object labels. With the above design, DOrA can be applied for 3D
grounding tasks and achieve satisfactory performances. We now detail the
design of our DOrA in the following subsections.

Figure 1: Architecture of our 3D Visual Grounding Framework with Order-Aware
Referring (DOrA). By taking a point cloud scene $C$ and a natural description
$D$ as inputs, our DOrA produces a referential order of anchor objects
$O_{1:B}$ and conduct Object-Referring blocks $R_{1:B}$ to locate the target
object progressively.

###  3.2 Referential Ordering using LLM

To have Object-Referring blocks $\\{R_{1},\cdots,R_{B}\\}$ to locate the
target object properly, it is desirable to extract a proper referential order
$O_{1:B}$ from the input description $D$. With such a referring path
constructed, visual features of the associated objects can be updated for
grounding purposes. This is inspired by the idea of Chain-of-Thoughts [41, 26]
in LLM, as noted in CoT3DRef [5]. To achieve such a parsing task, we apply
GPT-3.5-Turbo [48] as the description parser using in-context learning (ICL)
[14], as depicted in Fig. 2(a).

However, it is not trivial to have LLM output a referential order from $D$ due
to lengthy and noisy descriptions. For example, for an input description “Look
at the king-size bed in the room next to a green chair, Find the pillow on the
bed. Not the pillow on the sofa which is also beside the chair.”, one would
expect first to find the green chair then the king-size bed next to the chair,
and finally, the pillow on the bed. Therefore, the ideal referential order is
{“chair”, “bed”, “pillow”}. In the above example, the sentence “Not the pillow
on the sofa which is also beside the chair.” is redundant since one can find
the correct target object without this information. If we apply the LLM
directly to the original description, the model may be misled by this
redundant information and generate a referential order containing the object
“sofa”.

To tackle the above problem, we conduct a two-stage in-context learning (ICL)
scheme to remove redundant information in $D$ before producing $O_{1:B}$, as
depicted in Fig. 2(a). With the given target object $O_{B}$, we predict a
summarized description $D^{{}^{\prime}}$ to remove redundant information in
$D$ in the first stage. Then, the entire referential order $O_{1:B}$ given
$D^{{}^{\prime}}$ and $O_{B}$ is produced in the second stage. For each stage
of our ICL, 10 examples are provided as demonstrations of the input prompts
for LLM to predict $O_{1:B}$. Due to page limitations, demonstration examples
and the complete prefix prompts are presented in the supplementary materials.
With $O_{1:B}$ obtained, the order-aware object referring process can be
processed accordingly, as we detail next.

(a) Referential order generation via LLM. A two-stage ICL is deployed to
remove redundant information from $D$, forming a descriptive order $O_{1:B}$
for locating the target object $O_{B}$.

(b) Feature Enhancement (FE). Taking $R_{1}$ as an example, FE processes the
objects described mentioned in $O_{1:B}$ and the relations between them by
attending the masked feature $F^{mask}_{1}$. Thus, only object features
related to $O_{1:B}$ would be refined as $F_{2}$.

Figure 2: Illustration of referential order generation and feature enhancement
in DOrA.

###  3.3 3D Visual Grounding with Order-Aware Object Referring

####  3.3.1 Object-referring blocks.

Given the object proposals $P$, the corresponding labels
$L=\\{l_{1},\cdots,l_{K}\\}$, the encoded text features $T$, and the derived
referential order $O_{1:B}$ the as inputs, our DOrA deploys a series of
Object-Referring blocks $\\{R_{1},\cdots,R_{B}\\}$ to perform the grounding
task. As depicted in Fig. 1, this referring process is conducted by leaving
out an anchor object and updating the visual features in each step, until only
the final step locates the target object of interest. Thus, the deployment of
Object-Referring blocks allows one to focus on the anchor/target objects, so
that their visual features and spatial relations between them can be exploited
with those of irrelevant objects disregarded.

Take the $i$-th referring block observing $O_{i:B}$ for example, a masked
feature $F^{mask}_{i}=F_{i}\odot M_{i}$ is derived by applying a Hadamard
product between $F_{i}$ and a $K$-dimensional binary mask $M_{i}$ to replace
features of object proposals in $F_{i}$ not belonging to any of the object
classes in $O_{i:B}$. Such a masking strategy ensures that $F^{mask}_{i}$
contains objects described in $O_{i:B}$ and hence explicitly suppresses
effects of irrelevant objects that are not in our interests. Thus, the $j$-th
entry of $M_{i}$ (denoted as $m_{ij}$) is defined as:

$m_{ij}=\begin{cases}1&\text{if class name of }l_{j}\text{ is in }O_{i:B},\\\
0&\text{otherwise}.\end{cases}$ (1)

$F_{i}$ and $F^{mask}_{i}$ are refined into $F_{i+1}$ for the next referring
block via the feature enhancement module (as discussed later). At the final
stage, the output $F_{B+1}$ of $R_{B}$ is utilized to predict the confidence
score $S$ that represents the identity of the target object, supervised by a
cross-entropy loss $\mathcal{L}_{ref}$. Additionally, to ensure the text
feature $T$ properly describing the anchor/target objects, we follow CoT3DRef
[5] and apply the language classification loss $\mathcal{L}_{text}$ to $T$ for
matching the associated class labels.

####  3.3.2 Object feature enhancement.

With the above masking process, each object-referring block is expected to
update the object features related to the anchor and target objects. This is
realized by our attention-based Feature Enhancement (FE) module. To be more
precise, in order to update the features associated with the anchor and target
objects in $R_{i}$ according to $F_{i}$, $F^{mask}_{i}$, $T$ and $O_{i:B}$,
our FE module aims to exploit their visual features and spatial relations
through attention mechanisms.

Take the FE module in $R_{1}$ as an example, as depicted in Fig. 2(b), we
start with the lower branch which locally emphasizes the features of the
potential anchor/target objects via a cross-attention layer by treating
$F^{mask}_{i}$ as value/key and encoded text feature of $O_{1:B}$ (denoted as
$T_{O_{1:B}}$) as query. On the other hand, the upper branch of Fig. 2(b)
explores the spatial relations between all objects by treating the self-
attended $F_{1}$ as key/value of another cross-attention, with the
concatenation of $T_{O_{1:B}}$ and $T$ being query. Finally, an additional
cross-attention layer is applied to the previous output features of both
branches to obtain the enhanced proposal feature $F_{2}$, which enriches not
only the information of anchor/target objects but also the relations between
them.

On the other hand, to prevent the information extracted from the anchor/target
objects from vanishing (i.e., $F_{2}$ becomes identical to $F_{1}$) during FE,
we introduce an additional masking loss $\mathcal{L}_{mask}$ by projecting
$F_{2}$ from $K\times d_{2}$-dimensional to $K\times 1$-dimensional digits
with MLPs to classify if each proposal in $F_{2}$ is previously masked in
$F^{mask}_{1}$. The masking loss $\mathcal{L}_{mask}$ is defined as:

$\mathcal{L}_{mask}=\mathcal{L}_{BCE}(MLP(F_{2}),M_{1}),$ (2)

where $M_{1}$ represents the $K$-dimensional binary mask as defined in Eq. 1.
It is worth noting that, $\mathcal{L}_{mask}$ is applied to output features of
each referring block with a similar formulation to ensure each output feature
contains the information of the current anchor/target objects correspondingly.

###  3.4 Order-Aware Pre-training with Synthetic Referential Order

Although DOrA is designed to produce a referential order of anchor objects for
localizing the target object, only the ground truth point cloud information of
the target object is given during training. Thus, the above framework is
viewed as a weakly-supervised learning scheme since there is no ground truth
referential order available during training. To provide better training
supervision, we pre-train DOrA with a simple yet proper synthetic 3D visual
grounding task, where the ground-truth labels of anchor/target objects and
descriptions with accurate referential orders can be obtained. This pre-
training strategy is presented below.

####  3.4.1 Augmenting plausible referential order and description

To provide better training supervision and to ensure the reliability of our
synthesized data, the constructed description and the corresponding
referential order need to be easily and uniquely determined based on
anchor/target objects. In our work, we choose to consider spatial relations
between objects that are independent of viewpoint (e.g., “nearest” or
“farthest”) as the constructing descriptions, suggesting the referential order
of anchor/target objects during this data augmentation stage. As highlighted
in Algorithm A1 of our supplementary material, given $P$, $L$, and $B$, we
construct an augmented referential order $O^{aug}_{1:B}$ by choosing $B$
different class labels $\\{l^{aug}_{1},\cdots,l^{aug}_{B}\\}$ from $L$ and
extracting their class names. The augmented description $D^{aug}$ is then
derived as:

“There is a $\\{O^{aug}_{1}\\}$ in the room, find the $\\{O^{aug}_{2}\\}$
farthest to it, and then find the $\\{O^{aug}_{3}\\}$ farthest to that
$\\{O^{aug}_{2}\\}$, {…}, finally you can see the $\\{O^{aug}_{B}\\}$ farthest
to that $\\{O^{aug}_{B-1}\\}$.”

Since $D^{aug}$ is constructed following the appearing sequence of object
names in $O^{aug}_{1:B}$, it is guaranteed that $O^{aug}_{1:B}$ is a correct
referential order w.r.t. $D^{aug}$ and thus can be served as ground truth
supervision for pre-training DOrA.

It is worth noting that, to have each object in $D^{aug}$ uniquely defined, we
only keep one proposal $p^{aug}_{1}$ with the class name of
$\\{O^{aug}_{1}\\}$ in $P$ and remove all the other proposals with that class
name. As a result, all the anchor and target objects in $P$ (denoted as
$p^{aug}_{1:B}=\\{p^{aug}_{1},\cdots,p^{aug}_{B}\\}$) according to $D^{aug}$
and their corresponding identities are uniquely determined (i.e.,
$p^{aug}_{2}$ is assigned by finding the farthest proposal against
$p^{aug}_{1}$ with label $l^{aug}_{2}$, and the rest of the anchor/target
objects are determined consecutively with the same strategy).

####  3.4.2 Pre-training objectives

To have DOrA follow $O^{aug}_{1:B}$ to refer $p^{aug}_{i}$ in the $i$-th
referring block $R_{i}$, we design a coordinate loss $\mathcal{L}_{crd}$ to
encourage the output feature $F_{i+1}$ of $R_{i}$ to identify the coordinate
of all proposals in $P$ w.r.t. $p^{aug}_{i}$. Thus, we calculate
$\mathcal{L}_{crd}$ as:

$\mathcal{L}_{crd}=\frac{1}{B}\sum_{i=1}^{B}\mathcal{L}_{MSE}(MLP(F_{i+1}),V-\mathbb{I}\cdot{{v}_{i}}),$
(3)

where $MLP(\cdot)$ represents MLP layers applied to $F_{i+1}$, $V$ is a
$K\times 3$ matrix representing center coordinates of calculated bounding-
boxes of all $K$ proposals in $P$, $\mathbb{I}$ stands for a $K\times
1$-dimensional identity vector, and ${v}_{i}$ is the $1\times 3$-dimensional
center coordinate of $p^{aug}_{i}$.

We note that, the referential loss $\mathcal{L}_{ref}$ mentioned in Sec. 3.3
is also extended to classify both the identity of anchor objects using
$F_{2:B}$ and the identity of the target object for $F_{B+1}$ during the pre-
training. To this end, we can define the objectives used during our pre-
training by summing up the referential loss $\mathcal{L}_{ref}$ (for both
anchor and target objects), the masking loss $\mathcal{L}_{mask}$, the
language classification loss $\mathcal{L}_{text}$ and the coordinate loss
$\mathcal{L}_{crd}$.

###  3.5 Overall Training Pipeline

We now summarize the training of DOrA. With the pre-training stage noted in
Sec. 3.4 to warm up the model, we take point cloud data with real-world
natural descriptions to continue the training process. Since the identity of
anchor objects is unknown, we only apply $\mathcal{L}_{ref}$ (for the target
object only), $\mathcal{L}_{mask}$, and $\mathcal{L}_{text}$ as supervision.
The overall training pipeline is summarized in Algorithm A2 of our
supplementary material.

##  4 Experiments

###  4.1 Datasets

####  4.1.1 NR3D

NR3D [2] dataset consists of 707 indoor scenes in ScanNet [12] with 28715/7485
description-target pairs in the training/testing set, where the descriptions
are collected from human annotators. There are 524 different object classes in
the scenes in total. NR3D assumes perfect class-agnostic object proposals,
where each point in the scene is properly assigned to its corresponding
proposal. As a result, models are only required to classify the target object
that uniquely matches the description among all proposals in the scene, with
classification accuracy (Acc in %) being the metric. Please refer to the
supplementary material for the details of the implementation.

####  4.1.2 ScanRefer

ScanRefer [8] contains 36665/9508 description-target pairs across a total of
800 indoor scenes in its training/validation set, where the descriptions are
also collected from human annotators. Also derived from ScanNet [12] but
different from NR3D, perfect object proposals are not available in ScanRefer,
and therefore additional object proposers are required for all methods.
Nevertheless, Acc in % under 0.25 and 0.5 intersection over union (IoU) are
used as the metrics for ScanRefer. Please refer to the supplementary material
for the details of the implementation.

###  4.2 Quantitative Results

Table 1: Grounding accuracy (%) on NR3D. Note that each column shows the
results trained with a specific amount of training data.

Method | Labeled Training Data
---|---
1% | 2.5% | 5% | 10% | 100%
Referit3D [2] | 4.4 | 13.6 | 20.3 | 23.3 | 35.6
TransRefer3D [17] | 11.0 | 16.1 | 21.9 | 25.7 | 42.1
SAT [46] | 11.6 | 16.0 | 21.4 | 25.0 | 49.2
BUTD-DETR [21] | 24.2 | 28.6 | 31.2 | 33.3 | 54.6
MVT [20] | 9.9 | 16.1 | 21.6 | 26.5 | 55.1
MVT [20] \+ CoT3DRef [5] | - | - | - | 38.2 | 60.4
DOrA (Ours) | 33.5 | 36.1 | 41.5 | 46.0 | 59.7

We present the quantitative results on NR3D and ScanRefer, with consideration
of different amounts of available training samples against several baselines
by reproducing from their official implementation. As shown in Tab. 1, DOrA
possesses a considerably superior performance when training with 1% (287),
2.5% (717), 5% (1435), and 10% (2871) NR3D training set samples. Specifically,
using only 10% data, DOrA achieves 46.0 overall Acc, which outperforms
Referit3D and TransRefer3D with 100% (28715) data. When training with 100%
data, DOrA achieves comparable results to MVT augmented with
CoT3DRef111Performance of MVT+CoT3DRef on 1%, 2.5%, and 5% training data are
not available since its referential order and 1-to-1 class-identity relation
are not publicly available. Only their results of 10% and 100% data are
presented., which requires additional prior linguistic knowledge about the
dataset for its rule-based algorithm to construct a one-to-one relation
between each anchor’s class name and corresponding object identity.

Tab. 2 further displays the detailed performance on different official subsets
of NR3D. Among the subsets, the Hard subset contains samples with more than 2
distractors, where a distractor is an object having the same class name as the
target object, and the Easy subset is the contrary. View-dependent samples
contain relations where rotating the point cloud scene will affect the
referred ground-truth target object (e.g., left and right), and View-
independent samples are contrary. DOrA accentuates itself with decent
capabilities on different subsets, consistently followed by BUTD-DETR [21]
that also features a self-designed DETR-based data augmentation for 3D visual
grounding.

Finally, Tab. 3 presents DOrA’s performance on ScanRefer, where ground-truth
object proposals are unavailable. We can see that our DOrA demonstrates
slightly preferable results with only 5% and 10% of data compared to other
approaches. It is comparable to SOTA methods when given 100% data. Note that
we only report CORE-3DVG’s [44] results on 100% (36665) ScanRefer training
samples since its implementation is currently unavailable.

Table 2: Grounding accuracy on the official NR3D subsets [2]. For
implementation and comparison purposes, only the setting of 10% of training
data is considered.

Method | Hard | Easy | View-Dep. | View-Indep. | Overall
---|---|---|---|---|---
Referit3D [2] | 19.5 | 27.3 | 21.2 | 24.2 | 23.3
TransRefer3D [17] | 21.6 | 29.9 | 22.9 | 27.0 | 25.7
SAT [46] | 22.5 | 27.6 | 21.7 | 26.6 | 25.0
BUTD-DETR [21] | 25.9 | 41.9 | 29.1 | 34.8 | 33.3
MVT [20] | 22.9 | 30.3 | 25.4 | 27.1 | 26.5
DOrA (Ours) | 39.1 | 53.3 | 45.3 | 46.4 | 46.0

Table 3: Grounding accuracy (%) on the ScanRefer validation set. Note that
different amounts of training data are considered.

Method | Labeled Training Data
---|---
5% | 10% | 100%
|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
ScanRefer [8] | 23.0 | 12.0 | 27.4 | 15.0 | 35.5 | 22.4
3DVG-Trans. [53] | 35.3 | 23.3 | 39.0 | 29.0 | 45.9 | 34.5
3D-SPS [31] | 28.4 | 16.9 | 32.9 | 22.9 | 48.8 | 37.0
BUTD-DETR [21] | 38.2 | 26.2 | 40.3 | 28.3 | 52.2 | 39.8
M3DRef-CLIP [52] | 37.2 | 29.9 | 40.0 | 32.4 | 52.5 | 44.7
CORE-3DVG [44] | - | - | - | - | 56.8 | 43.8
DOrA (Ours) | 39.8 | 31.5 | 43.6 | 34.9 | 52.8 | 44.8

###  4.3 Ablation Studies

####  4.3.1 Component analysis

We ablate different components of our DOrA in Tab. 4, with performances under
1% and 100% training samples available. Baseline model A extracts class names
of anchor/target objects from $D$ using a language parser [6] and constructs
the referential order according to the appearance of the names in $D$
directly. Also, we employ each of our Referring Blocks $R_{i}$ in model A
without the FE module, _i.e_., directly using $F_{i}$ to calculate attention
with text features of $O_{i:B}$ and $D$. Model B enhances the accuracy on 1%
training data by conducting our two-stage object ordering with LLM. When
further applying the order-aware pre-training in model C, significant
improvements in both settings, especially for 1% available data, are observed.
The scarcity of training data often impedes the ability of the model to learn
the concept of order and relations among objects implicitly. By conducting our
pre-training strategy, our DOrA is able to learn foundational concepts of
ordering and relations to locate the target objects progressively. Finally,
our full model in the last row, incorporating FE modules each $R_{i}$ to
enhance features of anchor/target objects with $F^{mask}_{i}$, achieves
optimal results on both settings. This verifies the success of our proposed
modules and pre-training strategy.

Table 4: Ablation studies of proposed components in DOrA. For methods without
LLM Object Ordering, we form $O_{1:B}$ according to the appearance of
anchor/target object names in $D$. Both cases of 1% and 100% training data are
considered.

|  FE Module |  Order-Aware Pre-training |  LLM Object Ordering | 1% | 100%
---|---|---|---|---|---
A |  |  |  | 8.9 | 53.9
B |  |  | ✓ | 10.3 | 53.8
C |  | ✓ | ✓ | 25.8 | 58.0
Ours | ✓ | ✓ | ✓ | 33.5 | 59.7

(a) Grounding accuracies with different maximum (padded) referential order
length $B$. Note that $B$ is equal to and denoted as the number of Object-
Referring blocks.

(b) Statistics of the referential order length in NR3D. A longer referential
order denotes a more complex referring process for grounding.

Figure 3: (a) Grounding accuracies with varying Object-Referring block numbers
$B$ and (b) statistics of referential order length of NR3D. It can be seen
that the model performance saturates at $B$=4, matching length statistics of
NR3D.

####  4.3.2 Length of referential order

We additionally analyze the effectiveness of padded order length $B$ of
$O_{1:B}$ (and also the number of Referring Blocks) in 3, which illustrates
the performances with different $B$ using 1% and 100% training data in Fig.
3(a). We also show the statistics of the original order length generated by
our two-stage referential ordering in Fig. 3(b). We observe that although
performances gradually increase with larger $B$ (_i.e_., considering more
potential anchor objects appeared in $D$ achieves better accuracy), the
performance gain saturates at $B=4$. This can be explained by looking at Fig.
3(b), which shows that only a very few amount of data exceeds an order length
of $4$ in both training and testing pairs. This suggests that our selection of
$B=4$ is reasonable, optimally balancing computational efficiency with
prediction accuracy.

###  4.4 Qualitative Results

Fig. 4 demonstrates the qualitative results of DOrA on NR3D, with MVT being
the baseline. We display four successful cases and two failed cases. It is
shown that DOrA can successfully identify the target object referred by one to
multiple anchor objects, even in lengthy descriptions. Failed cases include
those that refer by shapes or have a very small target object that is hard for
Pointnet++ to capture visual information.

Figure 4: 3D grounding examples of NR3D. Note that blue/green/red boxes denote
ground truth/correct/incorrect predictions. While both MVT and DOrA fail on
the last two cases, it is due to the fact that the size of the target object
is extremely small (e.g., cup) and the description does not describe any
anchor objects.

##  5 Conclusions and Limitations

####  5.0.1 Conclusions

We presented a 3D visual grounding framework with Order-Aware referring (DOrA)
in this paper. DOrA leverages LLM to identify anchor/target objects from the
input description as a referential order, guiding the updates of the
associated object features for grounding purposes. The above process is
realized by stacked Object-Referring blocks in DOrA, which progressively
refine the features for the objects of interest in the referential order. In
addition, we also designed a pre-training scheme which augments a pseudo yet
correct series of anchor/target objects for warming up DOrA in a supervised
fashion. Experiments on benchmark datasets demonstrate that our DOrA performed
favorably against SOTA 3D grounding works across different settings.

####  5.0.2 Limitations

Since DOrA utilizes LLMs to generate the referential order from the input
description, despite of our introduced pre-training strategy to warm the
training process, the correctness of the extracted referential order would
affect the training and the performance of DOrA.

## References

  * [1] Abdelreheem, A., Olszewski, K., Lee, H.Y., Wonka, P., Achlioptas, P.: Scanents3d: Exploiting phrase-to-3d-object correspondences for improved visio-linguistic models in 3d scenes. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 3524–3534 (2024)
  * [2] Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., Guibas, L.: Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 422–440. Springer (2020)
  * [3] Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 3674–3683 (2018)
  * [4] Bakr, E., Alsaedy, Y., Elhoseiny, M.: Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 35 (2022)
  * [5] Bakr, E.M., Ayman, M., Ahmed, M., Slim, H., Elhoseiny, M.: Cot3dref: Chain-of-thoughts data-efficient 3d visual grounding. arXiv preprint arXiv:2310.06214 (2023)
  * [6] Bird, S., Klein, E., Loper, E.: Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc." (2009)
  * [7] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Proceedings of the European conference on computer vision (ECCV). pp. 213–229. Springer (2020)
  * [8] Chen, D.Z., Chang, A.X., Nießner, M.: Scanrefer: 3d object localization in rgb-d scans using natural language. In: Proceedings of the European conference on computer vision (ECCV). pp. 202–221. Springer (2020)
  * [9] Chen, L., Lambon Ralph, M.A., Rogers, T.T.: A unified model of human semantic knowledge and its disorders. Nature human behaviour p. 0039 (2017)
  * [10] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)
  * [11] Chen, S., Guhur, P.L., Tapaswi, M., Schmid, C., Laptev, I.: Language conditioned spatial relation reasoning for 3d object grounding. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 35 (2022)
  * [12] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 5828–5839 (2017)
  * [13] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). pp. 4171–4186 (2019)
  * [14] Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., Sui, Z.: A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022)
  * [15] Feng, M., Li, Z., Li, Q., Zhang, L., Zhang, X., Zhu, G., Zhang, H., Wang, Y., Mian, A.: Free-form description guided 3d visual graph network for object grounding in point cloud. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3722–3731 (2021)
  * [16] Hao, Z., Feng, L., Shilong, L., Lei, Z., Hang, S., Jun, Z., Ni, L.M., Heung-Yeung, S.: DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In: Proceedings of the International Conference on Learning Representations (ICLR) (2023)
  * [17] He, D., Zhao, Y., Luo, J., Hui, T., Huang, S., Zhang, A., Liu, S.: Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding. In: Proceedings of the 29th ACM International Conference on Multimedia (MM). pp. 2344–2352 (2021)
  * [18] Hsu, J., Mao, J., Wu, J.: Ns3d: Neuro-symbolic grounding of 3d objects and relations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2614–2623 (2023)
  * [19] Huang, P.H., Lee, H.H., Chen, H.T., Liu, T.L.: Text-guided graph neural networks for referring 3d instance segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 1610–1618 (2021)
  * [20] Huang, S., Chen, Y., Jia, J., Wang, L.: Multi-view transformer for 3d visual grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15524–15533 (2022)
  * [21] Jain, A., Gkanatsios, N., Mediratta, I., Fragkiadaki, K.: Bottom up top down detection transformers for language grounding in images and point clouds. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 417–433. Springer (2022)
  * [22] Jiang, L., Zhao, H., Shi, S., Liu, S., Fu, C.W., Jia, J.: Pointgroup: Dual-set point grouping for 3d instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4867–4876 (2020)
  * [23] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detection for end-to-end multi-modal understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1780–1790 (2021)
  * [24] Kingma, D.P., Jimmy, B.: Adam: A method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (ICLR) (2015)
  * [25] Kochanski, R.B., Lombardi, J.M., Laratta, J.L., Lehman, R.A., O’Toole, J.E.: Image-guided navigation and robotics in spine surgery. Neurosurgery pp. 1179–1189 (2019)
  * [26] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 35 (2022)
  * [27] Lee, L.H., Braud, T., Zhou, P., Wang, L., Xu, D., Lin, Z., Kumar, A., Bermejo, C., Hui, P.: All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda. arXiv preprint arXiv:2110.05352 (2021)
  * [28] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10965–10975 (2022)
  * [29] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
  * [30] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [31] Luo, J., Fu, J., Kong, X., Gao, C., Ren, H., Shen, H., Xia, H., Liu, S.: 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16454–16463 (2022)
  * [32] McVay, J.C., Kane, M.J.: Conducting the train of thought: working memory capacity, goal neglect, and mind wandering in an executive-control task. Journal of Experimental Psychology: Learning, Memory, and Cognition p. 196 (2009)
  * [33] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 32 (2019)
  * [34] Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for 3d object detection in point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9277–9286 (2019)
  * [35] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Proceedings of Advances in neural information processing systems (NeurIPS). vol. 30 (2017)
  * [36] Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Transactions on Neural Networks pp. 61–80 (2008)
  * [37] Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., Manning, C.D.: Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In: Proceedings of the fourth workshop on vision and language. pp. 70–80 (2015)
  * [38] Shtedritski, A., Rupprecht, C., Vedaldi, A.: What does clip know about a red circle? visual prompt engineering for vlms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 11987–11997 (2023)
  * [39] Subramanian, S., Merrill, W., Darrell, T., Gardner, M., Singh, S., Rohrbach, A.: Reclip: A strong zero-shot baseline for referring expression comprehension. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (2022)
  * [40] Wang, Z., Huang, H., Zhao, Y., Li, L., Cheng, X., Zhu, Y., Yin, A., Zhao, Z.: 3DRP-net: 3D relative position-aware network for 3D visual grounding. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2023)
  * [41] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 35 (2022)
  * [42] Wu, Y., Cheng, X., Zhang, R., Cheng, Z., Zhang, J.: Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19231–19242 (2023)
  * [43] Wu, Y., Shi, M., Du, S., Lu, H., Cao, Z., Zhong, W.: 3d instances as 1d kernels. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 235–252. Springer (2022)
  * [44] Yang, L., Zhang, Z., Qi, Z., Xu, Y., Liu, W., Shan, Y., Li, B., Yang, W., Li, P., Wang, Y., et al.: Exploiting contextual objects and relations for 3d visual grounding. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 36 (2024)
  * [45] Yang, L., Wang, Y., Li, X., Wang, X., Yang, J.: Fine-grained visual prompting. In: Proceedings of Advances in Neural Information Processing Systems (2024)
  * [46] Yang, Z., Zhang, S., Wang, L., Luo, J.: Sat: 2d semantics assisted training for 3d visual grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1856–1866 (2021)
  * [47] Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T.S., Sun, M.: Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797 (2021)
  * [48] Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., Shen, Y., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420 (2023)
  * [49] Yuan, Z., Yan, X., Li, Z., Li, X., Guo, Y., Cui, S., Li, Z.: Toward explainable and fine-grained 3d grounding through referring textual phrases. arXiv preprint arXiv:2207.01821 (2022)
  * [50] Yuan, Z., Yan, X., Liao, Y., Zhang, R., Wang, S., Li, Z., Cui, S.: Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1791–1800 (2021)
  * [51] Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. In: Proceedings of Advances in Neural Information Processing Systems (NeurIPS). vol. 35 (2022)
  * [52] Zhang, Y., Gong, Z., Chang, A.X.: Multi3drefer: Grounding text description to multiple 3d objects. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 15225–15236 (2023)
  * [53] Zhao, L., Cai, D., Sheng, L., Xu, D.: 3dvg-transformer: Relation modeling for visual grounding on point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2928–2937 (2021)
  * [54] Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., Kang, B.: Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581 (2023)

##  Appendix 0.A More Details of DOrA

This section provides more details regarding model implementations and
experimental setups. In particular, we first demonstrate the pseudocode of
order-aware sample synthesis for pre-training in Sec. 3.4 and the complete
training pipeline in Sec. 3.5. Then, we elaborate on the hyperparameters of
DOrA and implementations of baselines for experiments in Sec. 4.

###  0.A.1 Pre-Training Sample Synthesis and Training Pipeline

This section provides the pseudocode of synthesizing order-aware samples for
DOrA’s pre-training in Sec. 3.4 and the complete training pipeline in Sec. 3.5
in our main paper, respectively. Specifically, the Algorithm A1 demonstrates
the pipeline to synthesize an order-aware pre-training sample given simply
object proposals $P$ and predicted object labels $L$. On the other hand,
Algorithm A2 illustrates the complete pipeline to train DOrA with synthesized
samples and natural-description samples, such as NR3D and ScanRefer.

Algorithm A1 Order-Aware Sample synthesis for DOrA Pre-training

Input: $P$ and $L$
Hyperparameters: $B$
Output: $D^{aug}$, $O^{aug}_{1:B}$, and $p^{aug}_{1:B}$

1:randomly sample and arrange $\\{l^{aug}_{1},\cdots,l^{aug}_{B}\\}$ from $L$.

2:extract class names of $\\{l^{aug}_{1},\cdots,l^{aug}_{B}\\}$ as
$O^{aug}_{1:B}=\\{O^{aug}_{1},\ldots,O^{aug}_{B}\\}$.

3:$D^{aug}=$ _“There is a $\\{O^{aug}_{1}\\}$ in the room, find the
$\\{O^{aug}_{2}\\}$ farthest to it, and then find the $\\{O^{aug}_{3}\\}$
farthest to that $\\{O^{aug}_{2}\\}$, {…}, finally you can see the
$\\{O^{aug}_{B}\\}$ farthest to that $\\{O^{aug}_{B-1}\\}$.”_

4:get $p^{aug}_{1}$ by randomly removing objects in $P$ with class of
$O^{aug}_{1}$ until only one of them is left.

5:initialize the anchor/target objects set as $\\{p^{aug}_{1}\\}$.

6:for $i=2,3,\ldots,B$ do

7: for all objects in $P$ with the class of $O^{aug}_{i}$, find the one
farthest from $p^{aug}_{i-1}$

8: to be $p^{aug}_{i}$.

9: append $p^{aug}_{i}$ to $\\{p^{aug}_{1},\ldots,p^{aug}_{i-1}\\}$.

10:end for

11:$p^{aug}_{1:B}\leftarrow\\{p^{aug}_{1},\cdots,p^{aug}_{B}\\}$

12:return $D^{aug}$, $O^{aug}_{1:B}$, and $p^{aug}_{1:B}$

###  0.A.2 Implementation and Details and Experimental Settings

For both datasets, we use PyTorch [33] library to implement DOrA. We train
DOrA using Adam [24] optimizer with a single NVIDIA Tesla V100 GPU. To conduct
batch-wise training with a fixed number of Object-Referring blocks, we set the
length of referential order $B$ to be 4, _i.e_., we trim the original
referential order from the front if its length exceeds 4 and pad it if its
length is lower than 4. We adopt BERT [13] as the text encoder to extract $T$
and Pointnet++ [35] as the visual encoder to acquire $F_{1}$. We sample
$I$=1024 points for each object proposal in the scene. Object proposal number
$K$ and token number $|D|$ are sample-dependent. Object feature dimension
$d_{i}$ for $i$-th Object-Referring block is set to 768, aligning with BERT’s
768 dimension, to conduct cross-attention.

Algorithm A2 DOrA Training Pipeline

Input: scene-description paired training samples
$\\{\\{C_{1},D_{1}\\},\\{C_{2},D_{2}\\},\ldots\\}$
Hyperparameters: pre-training step $S_{p}$, official training step $S_{o}$,
and $B$

1:initialize DOrA’s model weights $\phi$.

2:$L_{pre}\leftarrow\\{\mathcal{L}_{text},\mathcal{L}_{mask},\mathcal{L}_{ref},\mathcal{L}_{crd}\\}$

3:for $i=2,3,\ldots,S_{p}$ do

4: sample a scene point cloud $C$ from paired training samples.

5: acquire $P$ and $L$ of $C$.

6: use Algorithm A1 with $\\{P,L,B\\}$ as inputs to synthesize
$\\{D^{aug},O^{aug}_{1:B},p^{aug}_{1:B}\\}$.

7: use $\\{P,L,D^{aug},O^{aug}_{1:B},p^{aug}_{1:B}\\}$ to update $\phi$ with
$L_{pre}$.

8:end for

9:$L_{train}\leftarrow\\{\mathcal{L}_{text},\mathcal{L}_{mask},\mathcal{L}_{ref}\\}$

10:for $i=2,3,\ldots,S_{o}$ do

11: sample a data point $\\{C,D\\}$ in the training samples

12: acquire $P$ and $L$ of $C$.

13: apply LLM to acquire $O_{1:B}$.

14: use $\\{P,L,D,O_{1:B}\\}$ to update $\phi$ with $L_{train}$.

15:end for

####  0.A.2.1 NR3D

For NR3D, we use pre-trained Pointnet++ to classify all object proposals as
object labels $L$ following BUTD-DETR [21]. We pre-train DOrA for 15k steps on
ScanNet scene point cloud and our augmented samples in Sec. 3.4 and continue
on real-world pairs in NR3D for 120k steps using a batch size of 24.

####  0.A.2.2 ScanRefer

For ScanRefer, since ground-truth object proposals are unavailable, we adopt
the visual encoder of M3DRef-CLIP [52] that adopts PointGroup [22] to perform
object segmentation as proposals $P$ and object classification as labels $L$.
However, We do not perform object-aware pre-training for ScanRefer since
imperfect object proposals may lead to noisy synthetic samples and therefore
hinder the training stability of DOrA.

####  0.A.2.3 Baselines

For baselines [2, 17, 46, 21, 20, 5, 8, 53, 31, 52, 44] on NR3D and ScanRefer
in Table 1, 2, and 3, we utilize their official public implementations with
different amounts of available training samples and the full testing set to
evaluate their low-resource performance. For the full-data ($100$%) scenario,
we acquire their performance either on the official leaderboard of
NR3D/ScanRefer or their published papers. Especially, the implementation of
CORE-3DVG [44] is currently unavailable, so we only report its full-data
ScanRefer performance, which is presented in their paper. On the other hand,
CoT3DRef [5] does not release the parsed datasets required to train their
referential-order-based framework, so we only report its performance under
settings of full- and 10% training data, which are presented in their original
paper.

##  Appendix 0.B Prefix Prompt Examples of Two-Stage ICL for Deriving
Referential Order

This section lists the complete prompts of our two-stage ICL using
GPT-3.5-Turbo [48] for referential order generation in Sec. 3.2. Also, we
exhibit parsing results of 4 samples in the NR3D testing set and compare with
CoT3DRef [5] that also establishes the extraction and usage of referential
order.

###  0.B.1 Prefix Prompt of First-Stage ICL

The first-stage prompt in Sec. 3.2 is used to acquire the summarized
description $D^{{}^{\prime}}$ and target object $O_{B}$ of the original
description. Our first-stage prompt is as follows:

I have some descriptions, each describing a specific target object in a room.
However, they may have some redundant clauses or words. Your task is to
summarize them into a shorter description. Also tell me what is the target
object.
Below are 10 examples:
description 1: Assume you are facing the door in the room, find the larger
cabinet to its left.
summarized description 1: When facing the door, the cabinet on the right of
it.
target object 1: cabinet

description 2: The water bottle that is above the easy chair. NOT the smaller
water bottle that is above the orange table.
summarized description 2: The water bottle that is above the easy chair.
target object 2: water bottle

description 3: In the bedroom, you will see a sheer curtain, beside the
curtain is the steel window you need to find.
summarized description 3: The steel window beside a sheer curtain.
target object 3: window

description 4: Please find the towel hanging on the wall in the bathroom with
other three towels. You should find the one which is nearest to the door. Or
say it is on the door’s right side.
summarized description 4: The towel on the wall nearest to the door.
target object 4: towel

description 5: Between a pencil and a desk lamp on the desk is the backpack
you need to find.
summarized description 5: The backpack between a pencil and a desk lamp on the
desk.
target object 5: backpack

description 6: In the living room we have three bookshelves. Choose the
bookshelf to the right of the clock facing a cabinet.
summarized description 6: The bookshelf to the right of the clock facing a
cabinet.
target object 6: bookshelf

description 7: The person wearing a white T-shirt, not the man who is also
sitting on the bed but with a jacket.
summarized description 7: The person wearing a white T-shirt on a bed.
target object 7: person

description 8: The purple pillow on the right side of the bed on when facing
it. Not the one the left side and the one on the middle of the bed.
summarized description 8: The purple pillow on the right side of the bed when
facing it.
target object 8: pillow

description 9: The brown door at the end of the living room, next to the trash
cans which is full of garbage.
summarized description 9: The brown door next to the full trash can.
target object 9: door

description 10: The shoes that is placed in the middle of five shoes near the
door in the room.
summarized description 10: The middle shoes near the door.
target object 10: shoes
Now for the description [DESCRIPTION], give me the summarized description and
the target object. Your answer must be in the form "summarized description:
target object:"

###  0.B.2 Prefix Prompt of Second-Stage ICL

The second-stage prompt in Sec. 3.2 is used to acquire the referential order
$O_{1:B}$ based on the target object $O_{B}$ and the summarized description
$D^{\prime}$. The complete prompt is as follows:

I have some descriptions, each describing a specific target object with some
supporting anchor objects helping the localization. We can find the specific
target object by tracing the referential order of anchor objects step by step.
Your task is to provide a correct referential order. Also tell me what are the
mentioned anchor objects.
Below are 10 examples:
description 1: The water bottle that is above the easy chair.
target object 1: water bottle
anchor objects 1: easy chair
referential order 1: easy chair->water bottle

description 2: The steel window beside a sheer curtain.
target object 2: window
anchor objects 2: curtain
referential order 2: curtain->window

description 3: The trash can that is on the right of the king size bed.
target object 3: trash can
anchor objects 3: bed
referential order 3: bed->trash can

description 4: The backpack between a pencil and a desk lamp. They are all on
a wooden desk.
target object 4: backpack
anchor objects 4: pencil, desk lamp, desk
referential order 4: desk->pencil->desk lamp->backpack

description 5: The cabinet on the right of the door.
target object 5: cabinet
anchor objects 5: door
referential order 5: door->cabinet

description 6: The bookshelf to the right of the clock facing a cabinet.
target object 6: bookshelf
anchor objects 6: clock, cabinet
referential order 6: cabinet->clock->bookshelf

description 7: The person wearing a white T-shirt on a bed.
target object 7: person
anchor objects 7: bed
referential order 7: bed->person

description 8: The purple pillow on the right side of the bed when facing it.
target object 8: pillow
anchor objects 8: bed
referential order 8: bed->pillow

description 9: The brown door next to the full trash can.
target object 9: door
anchor objects 9: trash can
referential order 9: trash can->door

description 10: Please find the towel hanging on the wall in the bathroom with
other three towels. You should find the one nearest to the door. Or say it is
on the door’s right side.
target object 10: towel
anchor objects 10: wall, door
referential order 10: wall->door->towel

Now for the description: [DESCRIPTION], give me the anchor objects and the
referential order. Your answer must be in the form "referential order: ,
anchor objects: ".

###  0.B.3 Examples of Derived Referential Order

To show that our two-stage ICL produces reasonable referential orders, we
provide examples and comparisons between ours and CoT3DRef [5]’s parsing
results. Note that CoT3DRef currently does not release their complete parsing
results of NR3D, so we manually leverage its released prompt files in their
paper and official repository to query the GPT-3.5-Turbo. We display 4
examples in Tab. A5, where CoT3DRef misses an anchor object “bed” in the third
example and includes a redundant object “shelves” as an anchor object in the
fourth example, while our DOrA produces proper results in both cases. This
shows the effectiveness of two-stage ICL strategy.

Table A5: Examples of LLM-parsed referential order from CoT3DRef [5] and DOrA. Note that the blue text represent the ideal anchor/target objects and the red text represent the redundant object that should not appear in the referential order. We can see that MVT misses one anchor object in the third example and includes a redundant object in the fourth example, while DOrA predicts proper order for both cases. Description |  CoT3DRef [5] |  DOrA (Ours)
---|---|---
The pillow closest to the foot of the bed. |  bed -> pillow |  bed -> pillow
Facing the bed, it’s the large white pillow on the right. The second one from the headboard. |  bed -> headboard -> pillow |  bed -> headboard -> pillow
The front pillow on the bed with the laptop. |  bed -> pillow |  laptop -> bed -> pillow
The window near the table, not the one near the shelves. |  table -> shelves -> window |  table -> window

##  Appendix 0.C Ablation Studies on Training Objectives

This section performs the ablation study on several training objectives used
for DOrA mentioned in Sec. 3. In particular, we investigate the influence of
$\mathcal{L}_{mask}$, $\mathcal{L}_{crd}$, and $\mathcal{L}_{text}$ for low-
resource (1% training data) and full-resource (100% training data) scenarios,
as shown in Tab. A6. Note that we only conduct target object classification on
the text feature $T$ when deactivating $\mathcal{L}_{text}$, and apply
classification on both anchor and target objects when adopting
$\mathcal{L}_{text}$, as mentioned in Sec. 3.3. Also, for all experiments in
Tab. A6, the pre-training with synthetic data in Sec. 3.4 is applied for fair
comparison. In summary, $\mathcal{L}_{mask}$, $\mathcal{L}_{crd}$, and
$\mathcal{L}_{text}$ all impose positive effects on models’ performances under
the two settings, verifying DOrA’s design in Sec. 3.

Table A6: Ablation studies on training objectives. Note that for all ablated settings, the proposed order-aware pre-training in Sec. 3.4 is applied for fair comparison. $\mathcal{L}_{mask}$ | $\mathcal{L}_{crd}$ | $\mathcal{L}_{text}$ | $\mathcal{L}_{ref}$ | 1$\%$ | 100$\%$
---|---|---|---|---|---
|  |  | ✓ | 31.1 | 54.2
|  | ✓ | ✓ | 31.6 | 54.8
| ✓ | ✓ | ✓ | 32.4 | 56.0
✓ | ✓ | ✓ | ✓ | 33.5 | 59.7

##  Appendix 0.D Model Performance Under Different Order Length

This section explores DOrA’s performance gain for descriptions with different
order lengths compared with MVT [20] to show the effectiveness of our design
of progressive location to the target object. We split the original NR3D
testing set into subsets that possess order lengths of 1, 2&3, and 4&5. An
order length of 1 (1092 samples in the testing set) means that only the target
object is mentioned in the description. Samples with an order length of 2 or 3
(6184 samples in the

Table A7: Grounding accuracy on NR3D subsets regarding different parsed referential order lengths. Note that referential order length $=1$ means only the target object is mentioned in the description. The improvement of DOrA over MVT [20] grows as the parsed order length increases. Method | Order Length | overall
---|---|---
1 | 2&3 | 4&5
MVT [20] | 59.4 | 55.0 | 46.7 | 55.1
DOrA (Ours) |  61.6 (+2.2) |  59.6 (+4.6) |  52.0 (+5.3) |  59.7 (+4.6)

testing set) have 1 or 2 anchor objects mentioned other than the target
object. Similarly, samples with an order length of 4 or 5 (209 samples in the
testing set) have 3 or 4 anchor objects mentioned other than the target
object. A longer referential order may generally denote a longer and more
complicated description. As shown in Tab. A7, though only a 2.2% performance
gain is obtained for target-only samples, DOrA is more advantageous when
dealing with lengthy descriptions, with 5.3% performance gain achieved for
descriptions with order length of 4 or 5.

##  Appendix 0.E Visualization of Responses in Each Referring Block

Figure A5: Visualization of Responses in Each Referring Block. An example in
NR3D is shown in this figure. We color each object proposal according to their
feature response in $F_{1:(B+1)}$, where larger response with brighter color.
Note that the blue bounding box represents the ground truth target object. We
can see that our DOrA progressively locate the target object by considering
the referential order by first focus the wall then the screen and finally the
trash can.

To show that our DOrA indeed progressively locates the target object following
the derived referential order, we visualize the feature response of
$F_{1:(B+1)}$ ($B=4$) in Fig. A5. The blue bounding box indicates the ground-
truth target object, and we color the object proposal according to the
response of their corresponding features in $F_{1:(B+1)}$, where a brighter
color represents a higher response. We can see that the responses to object
proposals are originally cluttered. As our referential blocks are applied, the
response of anchor/target objects becomes larger and finally locates the ideal
target object in the last feature $F_{5}$.