# Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point
Clouds

Minghua Liu1*, Xuanlin Li1*, Zhan Ling1*, Yangyan Li2, Hao Su1
1UC San Diego 2Alibaba
https://colin97.github.io/FrameMining/

###### Abstract

We study how choices of input point cloud coordinate frames impact learning of
manipulation skills from 3D point clouds. There exist a variety of coordinate
frame choices to normalize captured robot-object-interaction point clouds. We
find that different frames have a profound effect on agent learning
performance, and the trend is similar across 3D backbone networks. In
particular, the end-effector frame and the target-part frame achieve higher
training efficiency than the commonly used world frame and robot-base frame in
many tasks, intuitively because they provide helpful alignments among point
clouds across time steps and thus can simplify visual module learning.
Moreover, the well-performing frames vary across tasks, and some tasks may
benefit from multiple frame candidates. We thus propose FrameMiners to
adaptively select candidate frames and fuse their merits in a task-agnostic
manner. Experimentally, FrameMiners achieves on-par or significantly higher
performance than the best single-frame version on five fully physical
manipulation tasks adapted from ManiSkill and OCRTOC. Without changing
existing camera placements or adding extra cameras, point cloud frame mining
can serve as a free lunch to improve 3D manipulation learning.

†† *equal contribution

> Keywords: point cloud, coordinate frame, robot manipulation, 3D, RL

##  1 Introduction

Figure 1: A 3D point cloud of a dual-arm robot pushing a chair, which can be
represented in various coordinate frames without changing camera placements or
requiring extra camera views. Our FrameMiner takes as input a point cloud
represented in multiple candidate frames and adaptively fuses their merits,
resulting in better performance.

With the rapid development and proliferation of low-cost 3D sensors, point
clouds have become more accessible and affordable in robotics tasks [1]. Also,
the tremendous progress in building neural networks with 3D point clouds [2,
3, 4, 5, 6, 7] has enabled powerful and flexible frameworks for 3D visual
understanding tasks such as 3D object detection [5, 8, 9], 6D pose estimation
[10, 11], and instance segmentation [12, 13]. Very recently, point cloud
started to be used as the input to deep reinforcement learning (RL) for object
manipulation [14, 15, 16], which aims at learning mappings directly from raw
3D sensor observations of unstructured environments to robot action commands.
These end-to-end learning methods avoid highly structured pipelines and
laborious human engineering required by conventional robot manipulation
systems.

When building an agent with point cloud input, existing works [14, 15, 16]
typically incorporate off-the-shelf point cloud backbone networks (e.g.,
PointNet [2]) into the pipeline as a feature extractor of the scene. However,
some facets in constructing point cloud representations have been overlooked.
For example, in the literature of 3D deep learning, the choice of coordinate
frame significantly affects task performance [2, 17, 18, 19, 20, 21]. On 3D
instance segmentation benchmarks for autonomous driving, previous work such as
[5] showed a pipeline to process input point clouds in the camera frame,
frustum frame, and object frame subsequently, leading to a large performance
boost in comparison to using the camera frame alone. For our goal of
manipulation skill learning, point clouds describe dynamic interactions
between robots and objects, including frequent contacts and occlusions. This
is a novel and more complex setting that differs from well-explored scenarios
in 3D supervised learning (e.g., single objects, outdoor scenes for autonomous
driving). Under this setting, choices of coordinate frames are more flexible
and diverse as multiple entities (e.g., robot and manipulated object) and
dynamic movements are involved.

In this work, we first examine whether and how different coordinate frames may
impact the performance and sample efficiency of point cloud-based RL for
object manipulation tasks. We study four candidate coordinate frames: world
frame, robot-base frame, end-effector frame, and target-part frame. These
frames differ in positions of origin and orientations of axes, and
canonicalize inputs in different manners (e.g., a fixed third-view, ego-
centric, hand-centric, object-centric). The comparison and analysis are
performed on five distinct physical manipulation tasks adapted from ManiSkill
[22] and OCRTOC [23], covering various numbers of arms, robot mobilities, and
camera settings. Results show that the choice of frames has profound effects.
In particular, the end-effector frame and the target-part frames, rarely
considered in previous works, lead to significantly better sample efficiency
and final convergence than the widely used world frame and robot-base frame on
many tasks. Visualization and analysis indicate that, by using different
coordinate frames to represent input point clouds, we are actually performing
various alignments of input scenes through $\mathbb{SE}(3)$ transformations,
which may simplify the learning of visual modules.

However, the well-performing single coordinate frame may vary from task to
task, and in many cases, we may need coordination between decisions made
according to multiple coordinate frames. For example, tasks equipped with
dual-arm robots may benefit from both left-hand and right-hand frames. For
mobile manipulation tasks involving both navigation and manipulation,
different frames could favor different skills (e.g., robot-base frame for
navigation skills, end-effector frame for manipulation skills). We thus
propose three task-agnostic strategies to adaptively select from multiple
candidate coordinate frames and fuse their merits, leading to more efficient
and effective object manipulation policy learning. Because we do not need to
capture additional camera views or rely on task-specific frame selections, our
frame mining strategies can be used as a free lunch to improve existing
methods on point cloud-based policy learning. We call these fusion approaches
as FrameMiners. Experimentally, we find that it matters to fuse information
from multiple frames, but the specific FrameMiner to choose does not create
much performance difference. In particular, we use one of the FrameMiners,
MixAction, to interpret the importance of different frames in the policy
execution process, and the interpretation agrees with our intuitions.

In summary, the main contributions of this work are as follows:

  * •

We find that the choice of coordinate frame has a profound impact on point
cloud-based object manipulation learning. In particular, the end-effector
frame and the target-part frame lead to much better sample efficiency than the
widely-used world frame and robot-base frame on many tasks;

  * •

We find that well-performing frames differ task by task, necessitating task-
agnostic ways to select and fuse frames. This observation is consistent across
3D backbone networks;

  * •

We propose FrameMiners, a collection of methods to fuse information from
multiple candiate frames. FrameMiners provide a free lunch to improve existing
point cloud-based manipulation learning methods without changing camera
placements or requiring additional camera views.

##  2 Related Work

##### Manipulation Learning with Point Clouds

Visual representation learning for object manipulation has been extensively
studied [24, 25, 26, 27, 28, 29, 30, 31]. With the flourishment of 3D deep
learning [2, 3, 4, 5, 6, 7], a major line of work learns representations from
3D point clouds for object manipulation [32, 33, 34, 35, 36, 37, 38].
Recently, people have also started to incorporate point clouds into deep
reinforcement learning (RL) pipelines for manipulation learning [14, 15, 16].
However, existing point cloud-based manipulation learning methods have not
paid enough attention to coordinate frame selections of input point clouds,
which is fundamental in 3D visual learning. Some very recent work [39, 40]
explored placement and selection of camera views and fusion of multi-view
images. We differ from them in that we focus on the preprocessing of captured
input point clouds without modifying existing camera configurations or adding
additional cameras.

##### Normalization and View Fusion in Point Cloud Learning

Normalizing input point clouds is a common practice in 3D deep learning
literature. For example, in single object analysis (e.g., classification and
part segmentation), people often normalize input point clouds into a
categorical canonical pose with unit scale [2, 3], simplifying network
training. Prior works find that existing point cloud networks [2, 3, 7, 41,
42] are very sensitive to input normalization [21, 43, 44], and many recent
attempts explore rotation invariant [45, 46, 47] and equivariant methods [21,
48, 49] for 3D deep learning. Compared to well-studied scenarios (e.g., single
object and autonomous driving), normalizations of point clouds under robot-
object interactions are under-explored.

In LiDAR point cloud learning for autonomous driving, many work focuses on the
fusion of multiple views [17, 18, 19, 20]. Unlike fusing multiple camera
scans, there is only one point cloud. They propose to process the point cloud
from different views (e.g., perspective view and birds-eye view) to combine
their merits, which has proven to be helpful. Our work shares a similar idea.
However, we focus on robotic object manipulation settings, and the choice of
coordinate systems is more diverse.

##  3 Point Cloud Coordinate Frame Selection Matters

###  3.1 Problem Setup

Figure 2: We study coordinate frame mining on manipulation tasks adapted from
OCRTOC [23] and ManiSkill [22] covering various setups (e.g., #arms, mobility,
camera). Simulation is fully physical. Figure 3: Architecture of a 3D point
cloud-based agent, which is optimized by actor-critic RL algorithms. We study
coordinate frame selection of input (fused) point cloud.

We aim to learn agents with point cloud input for object manipulation tasks
via Reinforcement / Imitation Learning (RL/IL). A task is formally defined as
a Partially-Observable Markov Decision Process (POMDP), which is represented
by a tuple $M=(S,A,\mu,T,R,\gamma,\Omega,O)$. Here $S$ and $A$ are the
environment state space and the action space. $\mu(s)$, $T(s^{\prime}|s,a)$,
$R(s,a)$, and $\gamma$ are the initial state distribution, state transition
probability, reward function, and discount factor, respectively.
$O(s):S\to\Omega$ is the observation function that maps environment states to
the observation space $\Omega$. Our agent is represented by a policy
$\pi:\Omega\to A$, which aims to maximize the expected accumulated return
given by $J(\pi\circ
O)=E_{\mu,T,\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})]$. Note that
$\pi$ does not have access to the environment state $s$ and only has access to
the observation $O(s)$. In this work, $O(s)$ consists of two parts: (1) a 3D
point cloud captured by depth cameras; (2) proprioceptive states for the
robot, such as joint positions and joint velocities. For the first part, if
there are multiple cameras, we fuse all point clouds into a single one by
transforming them into the same coordinate frame and concatenating the points
together.

Fig. 3 shows the architecture of a 3D point cloud-based agent, which we use to
discuss in this section. It first exploits a 3D backbone (e.g., PointNet [2])
to extract visual features from a 3D (fused) point cloud. The extracted
features are then concatenated with proprioceptive robot states and fed into
separate multi-layer perceptrons (MLP) for action and value prediction. The
input (fused) point cloud can be represented in different coordinate frames
before being fed into the 3D backbone network, and the choice of coordinate
frame is independent of camera views. For example, a point cloud captured by a
camera mounted on the robot’s head can be transformed into the end-effector
frame. In this work, we study how point cloud coordinate frames affect sample
efficiency and final convergence of object manipulation learning. Unlike prior
works [39, 40], we do not change robot camera configurations (e.g., camera
placement, inclusion of additional cameras).

As shown in Fig. 2, we exemplify the frame selection problem on five fully-
physical manipulation tasks, covering various numbers of robot arms,
mobilities, and camera settings. Among them, PickObject is adapted from OCRTOC
[23], and the other four tasks are adapted from ManiSkill [22]. On PickObject,
a fixed-base single-arm robot learns to physically grasp an object from the
table, lift it up to a target height, and keep it static for a while. Point
clouds are captured from a 3rd-view camera. On ManiSkill tasks, agents learn
generalizable physical manipulation skills (i.e., opening cabinet doors /
drawers, pushing chairs / moving buckets to target positions) across objects
with diverse topology, geometry, and appearance. We utilize mobile robots with
one or two arms. Point clouds come from a panoramic camera mounted on the
robot’s head. Action space includes joint velocities of the arm(s) and the
mobile robot base, along with joint positions of the gripper(s). More details
are presented in Appendix S.3.

###  3.2 Choices of Point Cloud Coordinate Frame

Figure 4: Illustration of four coordinate frames, which provide different
alignments across time steps. We visualize three point clouds (three time
steps) of an OpenCabinetDoor trajectory. Each row shows the same point cloud
represented in different coordinate frames. Please zoom in for details. Robot
arm, cabinet door handle, cabinet door, and cabinet body are colored in blue,
red, yellow, and brown, respectively. RGB arrows indicate the corresponding
origin and axes for each frame. Since the point clouds used for policy
learning can be rather sparse, we show dense point clouds here for better
visualization.

For 3D supervised learning tasks such as object classification and detection,
it’s a common practice to normalize input point clouds, and the choice of
coordinate frames significantly affects task performance [2, 21, 17, 18, 19,
20]. In point cloud-based manipulation learning, we are faced with an
underexplored, yet more challenging, setting. First, point clouds describe
more complex robot-object interactions, possibly including frequent contacts
and occlusions. Furthermore, compared to supervised learning, 3D visual
modules receive weaker supervision signals during RL training. Therefore, it
may become even more important to lessen the burden of visual module learning
by properly normalizing input point clouds. Unlike previous well-studied point
cloud learning scenarios (e.g., single-object point clouds, LiDAR point clouds
for autonomous driving), there exist more diverse choices of coordinate
frames. In this paper, we compare and analyze four candidates:

  * •

A world frame is attached to a fixed point in the world (e.g., the start point
of a trajectory).

  * •

A robot-base frame is attached to the robot base, offering an egocentric
perspective on a mobile robot. For a fixed-base robot, world frame and robot-
base frame could be equivalent.

  * •

In many object manipulation tasks, movements of robot end-effector(s) play
important roles, and we can attach an end-effector frame to each of them. Note
that for dual-arm robots, there are two end-effectors and thus two end-
effector frames.

  * •

A target-part frame is attached to the object part the robot intends to
interact with (e.g., target door handle for the OpenCabinetDoor task).

When we transform captured point clouds into the world frame, the robot-base
frame, and the end-effector frame, we may need proprioceptive robot states and
potential robot movement tracking, which is typically accessible in modern
robots. When we transform point clouds into the target-part frame, we may need
to leverage off-the-shelf 3D object detection and pose estimation techniques.
However, in this paper, we mainly focus on the choices of coordinate frames
themselves. In our simulated experiments, we use ground truth object poses for
the target-part frame.

In Fig. 4, we visualize an example trajectory under four coordinate frames. As
shown in the figure, different coordinate frames canonicalize inputs in
different manners (e.g., a fixed third-view, ego-centric, hand-centric, and
object-centric), which is essentially performing various alignments among
point clouds across multiple time steps. For example, in the end-effector
frame, the end-effector is always aligned at the origin throughout a
trajectory. Such alignments may simplify the learning of visual modules in
distinct ways. With the end-effector frame, the network does not need to
locate the end-effector in point clouds (always at the origin). Similarly,
with the target-part frame, it can be easier to determine the relative
position between the target part and the robot end-effector. The robot-base
frame naturally aligns its frame axes with the moving directions of the
robot’s base.

###  3.3 Single-Frame Comparison on Manipulation Tasks

Figure 5: Comparison of four coordinate frames on five fully-physical
manipulation tasks. The (fused) point cloud is transformed to a single
coordinate frame before being fed to the visual backbone network. For dual-arm
tasks (i.e., PushChair and MoveBucket), we use the right-hand frame as the
end-effector frame. For PickObject, which has a fixed base, the world frame is
the same as the robot-base frame. Mean and standard deviation over 5 seeds are
shown.

We compare the four coordinate frames on the five manipulation tasks by
training PPO [50] agents using PointNet [2] as the 3D visual backbone. In this
section, the (fused) point cloud is transformed into a single coordinate
frame. For PushChair and MoveBucket tasks that use a dual-arm robot, we use
the right hand frame as the end-effector frame (we observe almost identical
performance using the left hand frame). For the target-part frame, we choose
the handle frame for OpenCabinetDoor and OpenCabinetDrawer tasks, chair seat
frame for the PushChair task, bucket for the MoveBucket task, and the target
object for the PickObject task. Further details are presented in the
supplementary.

Fig. 5 shows the results. We observe that distinct coordinate frames lead to
very different agent training performances. Overall, the world frame is the
least effective, especially in PushChair and MoveBucket that involve more
pronounced movement of the robot base. This suggests that the alignment of a
static point in the world-frame is less helpful for the tasks. Compared to the
commonly-used world-frame and robot-base frame, the end-effector frame has
much higher sample efficiency on all single-arm tasks (i.e., OpenCabinetDoor,
OpenCabinetDrawer, and PickObject), demonstrating the benefits of end-effector
alignment. However, it shows similar or worse performance on PushChair and
MoveBucket, intuitively because these tasks rely on dual-arm coordination, but
our point cloud is normalized to a single end-effector frame (i.e., right hand
frame). In addition, the target-part frame achieves the best sample efficiency
on most tasks, suggesting that the target-part alignment across time could be
of great help for point cloud-based visual manipulation learning.

###  3.4 Further Analysis

In robot manipulation tasks, agents often need to infer binary relations
between subjects (e.g., relative pose between the end-effector and the cabinet
handle). By aligning point clouds under certain frames (e.g., end-effector
frame), these tasks may be reduced to single-subject location tasks (e.g.,
simply copying the handle pose), which become much easier to solve. To confirm
this hypothesis, we perform a diagnosis experiment on OpenCabinetDoor, where
we intentionally remove all robot points (i.e., blue points in Fig. 4) and see
its effect on different coordinate frames. As shown in Fig. 6, after the robot
points are removed, the end-effector frame performs the same, while the robot-
base frame performs worse (the task is still solvable since the end-effector
position is also provided in the proprioceptive robot state). This suggests
that the end-effector frame allows an agent to focus on the target object,
along with its interaction with the robot hand, which verifies our hypothesis.

Figure 6: Removing robot points does not harm the performance of the EE frame
on OpenCabinetDoor.

We utilized PointNet [2] as our 3D visual backbone for its fast speed and
general good performance. However, it’s unclear whether point cloud frame
selection is also crucial for other 3D backbones, especially those more
complex and powerful. Therefore, we conduct the same experiments as Sec 3.3
using SparseConvNet [51], a heavier 3D backbone network, on the four ManiSkill
tasks (further details in the supplementary). As shown in Fig. 7, we observe
similar relative performance between frames as before (e.g., the world frame
performs poorly; the end-effector frame outperforms the world frame and the
robot-base frame on OpenCabinetDoor and OpenCabinetDrawer). Interestingly,
using SparseConvNet doesn’t improve the overall performance over PointNet.

Figure 7: Using SparseConvNet [51] as the 3D visual backbone, we observe
similar trends as Fig 5. Mean and standard deviation over 5 seeds are shown.

##  4 Mining Multiple Coordinate Frames

Figure 8: The pipeline of FrameMiner-MixAction (FM-MA). Each frame outputs an
action proposal. Actions are then fused through input-dependent and joint-
specific weights.

We have shown that different point cloud coordinate frames lead to distinct
sample efficiencies and final performances in manipulation learning. However,
a single frame can perform well on some tasks but poorly on others, and we
wish to find good frames in a task-agnostic manner. Moreover, for complex
manipulation tasks, a single frame could be insufficient, and synergistic
coordination between multiple frames could provide unparalleled advantages.
For example, when robots are equipped with multiple arms, each arm may have
its preferred coordinate frame (e.g., left-hand frame and right-hand frame).
In addition, in tasks that involve simultaneous navigation and manipulation
(e.g., on PushChair and MoveBucket, an agent needs to move towards the target
while manipulating chairs or buckets), different frames could benefit
different skills (e.g., robot-base frame for navigation skills, and end-
effector frame for manipulation skills). Therefore, it is of great help to
propose a prior-agnostic method that can automatically select the best frame
from multiple candidates or combine the merits of them. Again, we are not
talking about fusing multiple camera views. Point clouds from multiple camera
views are first fused together into a single point cloud, before being
transformed to each coordinate frame.

In this section, we will present a collection of three strategies to
adaptively select and fuse multiple candidate coordinate frames, and we call
them FrameMiners. In particular, we will first introduce FrameMiner-MixAction
in Section 4.1 in detail to interpret the importance of different frames in
the policy execution process. We will then briefly introduce the other two
FrameMiners and compare different approaches with single-frame baselines.

###  4.1 FrameMiner-MixAction

Inspired by the idea of mixture of experts [52], we propose a general and
interpretable framework, FrameMiner-MixAction (FM-MA). As shown in Fig. 8, FM-
MA takes a (fused) point cloud and $n$ candidate coordinate frames as input,
and first transforms the point cloud into $n$ coordinate frames. For each
transformed point cloud, FM-MA employs an expert network, consisting of a 3D
visual backbone (e.g., PointNet [2]) and an MLP, to propose full robot actions
(e.g. target velocity of $m$ joints). Since different experts use different
coordinate frames, they are encouraged to specialize different skills and
controls of different joints. Finally, we combine actions from the $n$ experts
through input-dependent weights. Specifically, we concatenate extracted visual
features from all $n$ frames with the proprioceptive robot state, feed it into
an MLP, and predict a weight for each pair of expert and joint (there are
$n\times m$ weights in total). For each joint, we normalize the weights over
$n$ experts via softmax and fuse the actions through weighted linear
combination.

Figure 9: Left: learned weights of FrameMiner-MixAction over a MoveBucket
trajectory, where three coordinate frames are fused. We divide robot joints
into three groups and show the average weights of each group in each
coordinate frame. Right: three stages of the trajectory. (a) Approaching the
bucket. (b) Moving the bucket to the platform. (c) Placing the bucket on the
platform.

FM-MA fuses actions by predicting joint-specific weights, since we believe
that, for different joints, we need to extract information from different
coordinate frames. Furthermore, the weights are input-dependent, potentially
allowing the model to capture dynamic joint-frame relations at different
stages of a task. Fig. 9 confirms these hypotheses. On MoveBucket
trajectories, we observe distinct frame preferences between different robot
joints. The left and right-hand frames contribute significantly to their
respective joint actions. In addition, the weight distribution changes greatly
over different trajectory stages. Initially, when the robot is moving towards
the bucket but not interacting with it, the base frame contributes more.
However, when the hands start to manipulate the bucket, the hand frames’
weights increase. In particular, when the robot places the bucket onto the
platform, we need careful coordination among all joints, and thus similar
weights from each frame.

###  4.2 FrameMiners vs. Single Coordinate Frame

To study how network architectures influence coordinate frame fusion, we also
propose other two strategies: FrameMiner-FeatureConcat (FM-FC) and FrameMiner-
TransformerGroup (FM-TG). For each transformed point cloud, FM-FC uses an
individual PointNet to extract visual feature. All visual features are then
concatenated and fed into an MLP to predict robot action. FM-TG decomposes our
robot action into three groups: base joint actions, left-hand joint actions,
and right-hand joint actions (only two groups for single-arm tasks). After
visual features are extracted from PointNets, they are fused through a
Transformer [53] to produce a feature for each action group, which passes
through an MLP to predict its respective joint actions (see Appendix S.1 for
details).

| Base | EE | FM-FC | FM-MA | FM-TG
---|---|---|---|---|---
Door | 54$\pm$7 | 80$\pm$2 | 79$\pm$3 | 84$\pm$2 | 70$\pm$6
Drawer | 88$\pm$2 | 93$\pm$1 | 94$\pm$1 | 93$\pm$1 | 93$\pm$2
Chair | 7$\pm$3 | 2$\pm$1 | 32$\pm$4 | 36$\pm$4 | 34$\pm$6
Bucket | 23$\pm$6 | 19$\pm$4 | 77$\pm$5 | 81$\pm$3 | 90$\pm$2
Table 1: Success rates (%) on four ManiSkill tasks.

We compare our three FrameMiners with single-frame baselines. Specifically, in
this section, we focus on frame mining among the robot-base frame and end-
effector frame(s). For the dual-arm tasks (i.e., PushChair and MoveBucket),
the end-effector frames include both the left-hand frame and the right-hand
frame. We will discuss the inclusion of target-part frame in Section 4.3. Fig.
10 and Tab. 1 show the comparison results. On single-arm tasks (i.e.,
OpenCabinetDoor/Drawer), our FrameMiners perform on par with the end-effector
frame, which suggests that FrameMiners can automatically select the best
single frame. On dual-arm tasks, our FrameMiners significantly outperform
single-frame baselines, demonstrating the advantage of coordination between
multiple coordinate frames. We also demonstrate that our FrameMiners
outperform alternative designs and provide robust advantages (more details in
Appendix S.2.2, S.2.3, and S.2.4). While it matters to fuse information from
multiple frames, the specific FrameMiner to choose does not create much
performance difference. Empirically, we find FM-MA less sensitive to training
parameters, and FM-TG more computationally expensive.

Our previous experiments were conducted using online RL. To investigate
whether our previous findings can generalize to other algorithm domains, we
perform experiments on imitation learning, and more details are presented in
Appendix S.2.1. The results corroborate our previous findings, i.e., different
coordinate frames have a profound effect on point cloud-based manipulation
skill learning, and FrameMiners are capable of automatically selecting the
best coordinate frame or combining the merits from multiple frames and
outperforming single-frame results.

Figure 10: Comparison of different frame mining approaches on the four
ManiSkill tasks, where the robot-base frame and end-effector frame(s) are
fused. Black lines indicate single-frame baselines. Mean and standard
deviation over 5 seeds are shown.

###  4.3 Target-Part Frame

Figure 11: Fusion of target-part frame could further boost the performance.

In Section 4.2, we focus on the fusion of robot-base frame and end-effector
frames, since the target-part frame relies on pose estimation of target
objects, which requires extra efforts in real-world settings. As shown in Fig.
11, if object poses are estimated, we sometimes observe further performance
boost of FrameMiners from the target-part frame. For example, on PushChair, by
incorporating the target-part frame, the success rate of FM-MA increases from
36$\pm$4% to 53$\pm$3%. On PickObject, FM-MA already achieves good performance
without the target-part frame (94$\pm$2%); incorporating it slightly improves
the success rate to 97$\pm$1%.

##  5 Real World Experiments

Figure 12: Real robot setup.

To further verify that our learned policies can be deployed on real-world
robots without introducing extra domain gaps, we test on PickObject with a
Kinova Jaco2 Spherical 7-DoF robot, an Intel RealSense [54] D435 camera for
uncolored point cloud capture, and a Rubik’s cube from YCB objects [55, 56]
(see Fig. 12). We use a 3-DoF end-effector position controller and a 1-DoF
gripper position controller. We train the FM-MA policy (Section 4.1) by fusing
the robot-base frame and the end-effector frame. At test time, we build a
digital twin in the simulator over 25 sampled initializations of the real
environment with a vision-based pipeline like Jiang et al. [57]. By following
trajectories from policy rollout, we obtain a 84% success rate with FM-MA,
compared to an 80% success rate with the end-effector frame. The robot-base
frame is unable to achieve successful picks under our training budget. Note
that the performance differences in the real world are very similar to the
simulation environment, indicating that point cloud frame selection or mining
does not affect original domain gap. More details are presented in Appendix
S.5.

##  6 Conclusion and Limitations

We find that choices of point cloud coordinate frames have a profound impact
on learning manipulation skills. Our proposed FrameMiners can adaptively
select and fuse multiple candidate frames, serving as a free lunch for 3D
point cloud-based manipulation learning. Currently, our FrameMiners need to
process each frame separately, leading to more network computation. In the
future, we would like to explore more advanced fusion strategies to further
improve network efficiency as well as performance. In addition, the target-
part frame requires human judgment to determine the target part candidates and
6D pose estimation of the target parts, although we have shown our method can
also achieve great improvements without the target-part frame (Section 4.2).

#### Acknowledgments

This work is supported in part by gifts from Qualcomm. We would like to thank
Jiayuan Gu and Zhiao Huang for their helpful discussion and manuscript
proofreading.

## References

  * Kin [2022] Kinova.  https://www.kinovarobotics.com, 2022.  Accessed: 2022-06-14.
  * Qi et al. [2017a] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017a.
  * Qi et al. [2017b] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017b.
  * Chang et al. [2015] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Qi et al. [2018] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas.  Frustum pointnets for 3d object detection from rgb-d data.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 918–927, 2018.
  * Fan et al. [2017] H. Fan, H. Su, and L. J. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 605–613, 2017.
  * Wang et al. [2019] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon.  Dynamic graph cnn for learning on point clouds.  _Acm Transactions On Graphics (tog)_ , 38(5):1–12, 2019.
  * Lang et al. [2019] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12697–12705, 2019.
  * Qi et al. [2019] C. R. Qi, O. Litany, K. He, and L. J. Guibas.  Deep hough voting for 3d object detection in point clouds.  In _proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9277–9286, 2019.
  * He et al. [2020] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun.  Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11632–11641, 2020.
  * Wang et al. [2019] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese.  Densefusion: 6d object pose estimation by iterative dense fusion.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3343–3352, 2019.
  * Yang et al. [2019] B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni.  Learning object bounding boxes for 3d instance segmentation on point clouds.  _Advances in neural information processing systems_ , 32, 2019.
  * Yi et al. [2019] L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas.  Gspn: Generative shape proposal network for 3d instance segmentation in point cloud.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3947–3956, 2019.
  * Huang et al. [2021] W. Huang, I. Mordatch, P. Abbeel, and D. Pathak.  Generalization in dexterous manipulation via geometry-aware multi-task learning.  _arXiv preprint arXiv:2111.03062_ , 2021.
  * Chen et al. [2022] T. Chen, J. Xu, and P. Agrawal.  A system for general in-hand object re-orientation.  In _Conference on Robot Learning_ , pages 297–307. PMLR, 2022.
  * Wu et al. [2022] Y.-H. Wu, J. Wang, and X. Wang.  Learning generalizable dexterous manipulation from human grasp affordance.  _arXiv preprint arXiv:2204.02320_ , 2022.
  * Zhou et al. [2020] Y. Zhou, P. Sun, Y. Zhang, D. Anguelov, J. Gao, T. Ouyang, J. Guo, J. Ngiam, and V. Vasudevan.  End-to-end multi-view fusion for 3d object detection in lidar point clouds.  In _Conference on Robot Learning_ , pages 923–932. PMLR, 2020.
  * Alnaggar et al. [2021] Y. A. Alnaggar, M. Afifi, K. Amer, and M. ElHelw.  Multi projection fusion for real-time semantic segmentation of 3d lidar point clouds.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1800–1809, 2021.
  * Gerdzhev et al. [2021] M. Gerdzhev, R. Razani, E. Taghavi, and L. Bingbing.  Tornado-net: multiview total variation semantic segmentation with diamond inception module.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9543–9549. IEEE, 2021.
  * Liong et al. [2020] V. E. Liong, T. N. T. Nguyen, S. Widjaja, D. Sharma, and Z. J. Chong.  Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation.  _arXiv preprint arXiv:2012.04934_ , 2020.
  * Deng et al. [2021] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas.  Vector neurons: A general framework for so (3)-equivariant networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12200–12209, 2021.
  * Mu et al. [2021] T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su.  Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.  _arXiv preprint arXiv:2107.14483_ , 2021.
  * Liu et al. [2021] Z. Liu, W. Liu, Y. Qin, F. Xiang, M. Gou, S. Xin, M. A. Roa, B. Calli, H. Su, Y. Sun, et al.  Ocrtoc: A cloud-based competition and benchmark for robotic grasping and manipulation.  _IEEE Robotics and Automation Letters_ , 7(1):486–493, 2021.
  * Zhu et al. [2018] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, et al.  Reinforcement and imitation learning for diverse visuomotor skills.  _arXiv preprint arXiv:1802.09564_ , 2018.
  * Chen et al. [2021] B. Chen, P. Abbeel, and D. Pathak.  Unsupervised learning of visual 3d keypoints for control.  In _International Conference on Machine Learning_ , pages 1539–1549. PMLR, 2021.
  * Vecerik et al. [2020] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz.  S3k: Self-supervised semantic keypoints for robotic manipulation via multi-view consistency.  _arXiv preprint arXiv:2009.14711_ , 2020.
  * Cheng et al. [2018] R. Cheng, A. Agarwal, and K. Fragkiadaki.  Reinforcement learning of active vision for manipulating objects under occlusions.  In _Conference on Robot Learning_ , pages 422–431. PMLR, 2018.
  * Zaky et al. [2020] Y. Zaky, G. Paruthi, B. Tripp, and J. Bergstra.  Active perception and representation for robotic manipulation.  _arXiv preprint arXiv:2003.06734_ , 2020.
  * Shah and Kumar [2021] R. Shah and V. Kumar.  Rrl: Resnet as representation for reinforcement learning.  _arXiv preprint arXiv:2107.03380_ , 2021.
  * Pari et al. [2021] J. Pari, N. Muhammad, S. P. Arunachalam, L. Pinto, et al.  The surprising effectiveness of representation learning for visual imitation.  _arXiv preprint arXiv:2112.01511_ , 2021.
  * Nair et al. [2022] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Lin et al. [2021] X. Lin, Y. Wang, Z. Huang, and D. Held.  Learning visible connectivity dynamics for cloth smoothing.  In _Conference on Robot Learning_ , pages 256–266. PMLR, 2021.
  * Alliegro et al. [2022] A. Alliegro, M. Rudorfer, F. Frattin, A. Leonardis, and T. Tommasi.  End-to-end learning to grasp from object point clouds.  _arXiv preprint arXiv:2203.05585_ , 2022.
  * Yarats et al. [2020] D. Yarats, I. Kostrikov, and R. Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations_ , 2020.
  * Lv et al. [2022] J. Lv, Q. Yu, L. Shao, W. Liu, W. Xu, and C. Lu.  Sagci-system: Towards sample-efficient, generalizable, compositional, and incremental robot learning.  In _2022 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2022.
  * James and Davison [2022] S. James and A. J. Davison.  Q-attention: Enabling efficient learning for vision-based robotic manipulation.  _IEEE Robotics and Automation Letters_ , 2022.
  * Sundermeyer et al. [2021] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox.  Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 13438–13444. IEEE, 2021.
  * Eisner et al. [2022] B. Eisner, H. Zhang, and D. Held.  Flowbot3d: Learning 3d articulation flow to manipulate articulated objects.  _Robotics Science and Systems (RSS)_ , 2022.
  * Hsu et al. [2022] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn.  Vision-based manipulators need to also see from their hands.  _International Conference on Learning Representations_ , 2022.
  * Jangir et al. [2022] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang.  Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation.  _IEEE Robotics and Automation Letters_ , 2022.
  * Atzmon et al. [2018] M. Atzmon, H. Maron, and Y. Lipman.  Point convolutional neural networks by extension operators.  _arXiv preprint arXiv:1803.10091_ , 2018.
  * Li et al. [2018] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen.  Pointcnn: Convolution on x-transformed points.  _Advances in neural information processing systems_ , 31, 2018.
  * Sun et al. [2022] J. Sun, Q. Zhang, B. Kailkhura, Z. Yu, C. Xiao, and Z. M. Mao.  Benchmarking robustness of 3d point cloud recognition against common corruptions.  _arXiv preprint arXiv:2201.12296_ , 2022.
  * Lorenz et al. [2021] T. Lorenz, A. Ruoss, M. Balunović, G. Singh, and M. Vechev.  Robustness certification for point cloud models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7608–7618, 2021.
  * Chen et al. [2019] C. Chen, G. Li, R. Xu, T. Chen, M. Wang, and L. Lin.  Clusternet: Deep hierarchical cluster network with rigorously rotation-invariant representation for point cloud analysis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4994–5002, 2019.
  * Li et al. [2021] X. Li, R. Li, G. Chen, C.-W. Fu, D. Cohen-Or, and P.-A. Heng.  A rotation-invariant framework for deep point cloud analysis.  _IEEE Transactions on Visualization and Computer Graphics_ , 2021\.
  * Zhang et al. [2019] Z. Zhang, B.-S. Hua, D. W. Rosen, and S.-K. Yeung.  Rotation invariant convolutions for 3d point clouds deep learning.  In _2019 International Conference on 3D Vision (3DV)_ , pages 204–213. IEEE, 2019.
  * Fuchs et al. [2020] F. Fuchs, D. Worrall, V. Fischer, and M. Welling.  Se (3)-transformers: 3d roto-translation equivariant attention networks.  _Advances in Neural Information Processing Systems_ , 33:1970–1981, 2020.
  * Thomas et al. [2018] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  _arXiv preprint arXiv:1802.08219_ , 2018.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Graham and van der Maaten [2017] B. Graham and L. van der Maaten.  Submanifold sparse convolutional networks.  _arXiv preprint arXiv:1706.01307_ , 2017.
  * Masoudnia and Ebrahimpour [2014] S. Masoudnia and R. Ebrahimpour.  Mixture of experts: a literature survey.  _Artificial Intelligence Review_ , 42(2):275–293, 2014.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Keselman et al. [2017] L. Keselman, J. Iselin Woodfill, A. Grunnet-Jepsen, and A. Bhowmik.  Intel realsense stereoscopic depth cameras.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 1–10, 2017.
  * Calli et al. [2015a] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar.  Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols.  _arXiv preprint arXiv:1502.03143_ , 2015a.
  * Calli et al. [2015b] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar.  The ycb object and model set: Towards common benchmarks for manipulation research.  In _2015 international conference on advanced robotics (ICAR)_ , pages 510–517. IEEE, 2015b.
  * Jiang et al. [2022] Z. Jiang, C.-C. Hsu, and Y. Zhu.  Ditto: Building digital twins of articulated objects from interaction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5616–5626, 2022.
  * Wen et al. [2022] B. Wen, W. Lian, K. Bekris, and S. Schaal.  You only demonstrate once: Category-level manipulation from single visual demonstration.  In _Proceedings of Robotics: Science and Systems_ , 2022.
  * Simeonov et al. [2022] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann.  Neural descriptor fields: Se(3)-equivariant object representations for manipulation.  2022\.
  * Tang et al. [2020] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han.  Searching efficient 3d architectures with sparse point-voxel convolution.  In _European Conference on Computer Vision_ , 2020.

###  S.1 Architecture of the other two FrameMiners

Fig. 13 shows architectures of the other two FrameMiners, FrameMiner-
FeatureConcat (FM-FC) and FrameMiner-TransformerGroup (FM-TG).

(a) FrameMiner-FeatureConcat

(b) FrameMiner-TransformerGroup

Figure 13: Architectures of FrameMiner-FeatureConcat and FrameMiner-
TransformerGroup.

###  S.2 Additional Experiment Results and Discussions

####  S.2.1 Imitation Learning

In the main text, we analyzed the profound impact of coordinate frames on
point cloud-based object manipulation learning through online RL algorithms.
Apart from online RL, some previous work [58] have shown that dynamic
selection of coordinate frames could benefit demonstration-based manipulation
learning as well. In this section, we conduct experiments on imitation
learning and investigate whether our previous findings can generalize to other
algorithm domains.

For each task, we use an expert RL policy to generate 100 successful
demonstrations. We then perform Behavior Cloning (BC) by representing input
point clouds under different coordinate frames, along with using our proposed
FrameMiner-MixAction (FM-MA). We utilize the same network architectures as
online RL, and we use MSE loss for training. For FM-MA, the robot-base frame
and the end-effector frame(s) are fused. As shown in Table 2, we observe
similar findings to Section 3.3 and Section 4.2. Specifically, the end-
effector frame has much higher performance on single-arm tasks
(OpenCabinetDoor/Drawer), demonstrating the benefits of end-effector
alignment. Our proposed FrameMiner is capable of automatically selecting the
best single frame or combining the merits from multiple frames and
outperforming single-frame baselines.

| Robot-Base | End-Effector | FM-MA
---|---|---|---
OpenCabinetDoor | 50$\pm$3 | 85$\pm$3 | 83$\pm$4
OpenCabinetDrawer | 72$\pm$4 | 88$\pm$2 | 88$\pm$2
PushChair | 38$\pm$3 | 28$\pm$2 | 42$\pm$4
MoveBucket | 76$\pm$4 | 80$\pm$2 | 91$\pm$2
Table 2: Behavior Cloning (BC) success rates (%) on four ManiSkill tasks. Mean
and standard deviation over 5 seeds are shown.

####  S.2.2 Alternative Designs in FM-MA (Weighted Linear Combination vs.
Maximum Weight)

Figure 14: Comparison between FM-MA-WLC and FM-MA-MW on MoveBucket. Mean and standard deviation over 5 seeds are shown. FM-MA-WLC achieves 81$\pm$3% final success rate, while FM-MA-MW only has 9$\pm$2% final success rate.  | FM-MA (WLC eval) | FM-MA (MW eval)
---|---|---
OpenCabinetDoor | 84$\pm$2 | 45$\pm$5
OpenCabinetDrawer | 93$\pm$1 | 93$\pm$2
PushChair | 36$\pm$4 | 20$\pm$3
MoveBucket | 81$\pm$3 | 14$\pm$3
Table 3: Success rate (%) comparison between the same FM-MA checkpoint
evaluated using weighted linear combination of actions (WLC) and using
maximum-weighted action (MW) on four ManiSkill tasks. Mean and standard
deviation over 5 seeds are shown.

In the main paper, FrameMiner-MixAction (FM-MA) uses weighted linear
combination to fuse action proposals from each coordinate frame (see Figure
8). For simplicity, we name this variant FM-MA-WLC. An alternative design is
to choose the max-weighted action proposal for each joint (we name this
variant FM-MA-MW). Formally, let $A\in\mathbb{R}^{n\times m}$, where $A_{ij}$
denotes the action proposal for the $j$-th robot joint from the $i$-th
coordinate frame. Let $W\in\mathbb{R}^{n\times m}$ be the weight matrix
predicted by the network. In FM-MA-MW, the output action
$\mathbf{a}=(a_{1},a_{2},\dots,a_{m})$ satisfies $a_{j}$ = $A_{kj}$ where
$k=\textrm{argmax}_{k=1}^{n}W_{kj}$. Note that FM-MA-WLC uses SoftMax to
normalize the weights; thus FM-MA-WLC can be regarded as a “soft version” of
FM-MA-MW.

To compare the two designs, we conduct two experiments: (1) We train FM-MA-MW
from scratch. Results are shown in Fig. 14. (2) We resume from the final
checkpoint of the original FM-MA-WLC. During evaluation, we use the max-
weighted action proposal as the action output. Results are shown in Table 3.
We observe that for both experiments, using FM-MA-MW deteriorates performance.
We conjecture that FM-MA-WLC alleviates optimization difficulty, which likely
comes from the fact that it is a “soft version” of FM-MA-MW with well-behaving
gradients. On the other hand, since FM-MA-MW uses argmax operation over
columns of $W$, there is a lack of gradient for $W$ during training, which
leads to more difficult optimization.

####  S.2.3 Ablation Study on Camera Placements

As a recap, the five tasks analyzed in our main paper cover both static and
moving camera settings. The experiments in the main paper were conducted using
default camera placements shown in Fig. 2. For the four tasks with moving
cameras, a panoramic camera is mounted on the robot head.

While FrameMiners do not require changing existing camera placements, camera
placements could still matter, since different camera placements affect the
point clouds being captured (due to different occlusion and sparsity
patterns). Therefore, we perform an experiment where we move the panoramic
camera from the robot head to the robot base. As shown in Fig. 15, we observe
similar phenomena as in Fig. 10. Specifically, fusing multiple coordinate
frames with our FrameMiners still leads to better sample efficiency and final
performance, demonstrating that FrameMiners are robust under different camera
placements.

Figure 15: Results on MoveBucket with a panoramic camera mounted on the robot
base. The “Robot-Base Frame” and the “End-Effector Frame” indicate the
coordinate frames used to represent captured input point clouds. FM-MA fuses
the two end-effector frames (left and right arms) and the robot-base frame.
Mean and standard deviation over 5 seeds are shown.

####  S.2.4 Learning Adaptive Frame Transformations from Observations

In our paper, we use known transformations (e.g., end-effector pose in robot
state) to align input point clouds in different coordinate frames and propose
FrameMiners to fuse merits of multiple coordinate frames. A potential baseline
is to learn a transformation adaptively based on input point clouds. To
examine the effectiveness of this baseline, we add an additional network
before the PointNet backbone to learn an adaptive $\mathbb{SE}(3)$
transformation based on the input point cloud. This transformation is then
applied to the input point cloud before passing it through the PointNet
backbone (note that we remove spatial transformation layers from the original
PointNet in all of our experiments). However, as shown in Figure 16, adding
this $\mathbb{SE}(3)$ transformation layer barely improves performance.

We conjecture that it’s very difficult to predict a $\mathbb{SE}(3)$
transformation for aligning the input point cloud across time due to the large
search space (where most transformations are ineffective) and weak supervision
from RL training loss. Moreover, in many challenging tasks, we may need to
fuse information simultaneously from multiple coordinate frames (e.g., left-
hand and right-hand frames). This is not achievable through learning a single
transformation. In contrast, for FrameMiners, we take advantage of easily-
accessible frame information (e.g. end-effector poses) without relying on
transformation prediction. We then fuse the merits of multiple candidate
coordinate frames.

Figure 16: Ablation study on adding an adaptive $\mathbb{SE}(3)$
transformation prediction layer. When the input point cloud is represented in
the robot-base frame, adding such transformation layer barely improves
performance, while representing the point cloud in the end-effector frame
significantly improves performance.

####  S.2.5 $\mathbb{SO}(3)$ and $\mathbb{SE}(3)$ Equivariant Point Cloud
Backbones

Recently, there have been several works on designing $\mathbb{SO}(3)$ and
$\mathbb{SE}(3)$ equivariant/invariant backbone networks for point cloud
learning [21, 59]. While they are of great benefit for analysis within each
object (e.g., shape classification, part segmentation, and 6D pose
estimation), our robot-object interaction setting is a bit different.

In robot manipulation scenarios, a particular challenge comes from inferring
the relations between two object parts (e.g., relative pose between the end-
effector and the cabinet handle). This binary relation inference task is
challenging under the weak RL loss supervision, even using $\mathbb{SO}(3)$
and $\mathbb{SE}(3)$ equivariant/invariant backbones. FrameMiners explicitly
approach this challenge by aligning point clouds (across multiple time steps)
with the known transformation matrices (e.g., the end-effector pose). This
reduces many binary relation inference tasks to single-subject location tasks,
which has much lower difficulty. For example, when using the end-effector
frame in the OpenCabinet task, the network only needs to copy the handle pose
to infer the relative pose between the handle and the end-effector, as the
end-effector is always at the frame origin.

###  S.3 More Details of Manipulation Tasks

#### Task Descriptions:

  * •

In OpenCabinetDoor, a single-arm mobile agent needs to approach a cabinet, use
the handle to fully open the designated cabinet door, and then keep the door
static for a while.

  * •

In OpenCabinetDrawer, a single-arm mobile agent needs to approach a cabinet,
use the handle to fully open the designated cabinet drawer, and then keep the
drawer static for a while.

  * •

In PushChair, a dual-arm mobile agent needs to approach the chair, push the
chair to a target location, and then keep the chair static for a while.

  * •

In MoveBucket, a dual-arm mobile agent needs to approach the bucket, move the
bucket to a target platform, place the bucket onto the platform, and then keep
the bucket static for a while.

  * •

In PickObject, a single-arm fixed-base agent needs to grasp an object from the
table, lift it up to a certain target height, and keep it static for a while.

Simulations are fully physical. For OpenCabinetDoor, OpenCabinetDrawer,
PushChair, and MoveBucket, there are 66, 49, 26, and 29 different objects
(designated parts) during training, respectively.

#### Observations and Actions:

For all ManiSkill tasks, the proprioceptive robot state includes:

  * •

Positions of all (two if single-arm, four if dual-arm) fingers

  * •

Velocities of all (two or four) fingers

  * •

x, y position of the mobile robot base

  * •

Mobile robot base’s rotation around the z-axis

  * •

x, y velocity of the mobile robot base

  * •

Angular velocity of the mobile robot base around the z-axis

  * •

Joint angles of the robot, excluding the joints in the mobile base

  * •

Joint velocities of the robot, excluding the joints in the mobile base

  * •

Indicator of whether each joint receives an external torque

The action space includes:

  * •

x, y velocity of the mobile robot base

  * •

Angular velocity of the mobile robot base around the z-axis

  * •

Height of the robot body

  * •

Joint velocities of the robot, excluding joints of the mobile base and the
gripper fingers

  * •

Joint positions of the gripper fingers

Joint positions of the gripper fingers are controlled by position PID. All
other action components are controlled by velocity PID.

For the PickObject task, the proprioceptive robot state includes:

  * •

Joint angles of the robot,

  * •

Joint velocities of the robot,

  * •

1D gripper joint position,

  * •

Target xyz positions of object.

The action space includes 3 DoF end-effector position and 1 DoF gripper joint
position.

For all tasks, input point cloud features include xyz coordinates, RGB colors,
and one-hot segmentation masks for each part category.

### Motivations for Our Task Choice

We aim to cover a wide range of factors that may influence the selection of
point cloud coordinate frames. Specifically, the tasks are chosen to cover
various robot mobilities, numbers of robot arms, and camera settings, as
demonstrated in Figure 1.

Different robot mobility results in differences in world frame and robot base
frame. These two frames are aligned in static robots but not in mobile robots.
The robot’s mobility can also change the focus of tasks (e.g., navigation or
object interaction), which may place different requirements on the choice of
point cloud frame.

We cover both single-arm and dual-arm environments, as they pose different
requirements for point cloud frame selection. In single-arm environments,
using the only end-effector frame may already be able to achieve good
performance. However, in dual-arm environments, there are two end-effector
frames, and these tasks require precise coordination between the two robot
arms, which pose significant challenges for manipulation learning. As each
end-effector may have a preferred frame, the necessity of frame fusion becomes
more pronounced.

Last but not least, camera placements determine sources of point clouds, which
may potentially influence the selection of coordinate frames. In our
experiments, we cover both static camera settings and moving camera settings
(mounted on robots).

Figure 17: RGB images and 3D point clouds captured in both simulation and the
real world. Colored point clouds for better illustration.

###  S.4 Detailed Experimental Settings and Hyperparameters

For our visual backbones, our PointNets are implemented with a three-layer MLP
with dimensions $[64,128,300]$ followed by a max-pooling layer. We do not
apply any spatial transformation to the inputs. Our SparseConvNets are
implemented as a SparseResNet10 using TorchSparse [60]. SparseResNet10 has a
4-stage pipeline with kernel size 3 and hidden channels $[64,128,256,512]$
respectively. We use kernel size 3 and stride 2 for downsampling. Initial
voxel size is 0.05. Final features in the final-stage voxels are maxpooled as
output visual feature.

All of our agents are trained with PPO (hyperparameters in Tab. 4). Each
policy MLP that outputs actions has dimensions
$[192,128,\textrm{action\\_dim}]$. For FM-MA that uses input-dependent joint-
specific weights to fuse action proposals from different frames, the MLP has
dimension $[192,n\times m]$, where $n$ is the number of frames and $m$ is the
dimension of action space. For FM-TG that uses Transformer to fuse features
from different frames, the Transformer has 3 layers with hidden dimension 300
and feed-forward dimension 1024. For all network variants, the value head
takes the concatenation of all visual features from all frames as input and
passes through an MLP with dimensions $[192,128,1]$ to output value
prediction.

In addition, we found that zero-initializing the last layer of MLP before
action output along with the joint-specific weights in FM-MA to be very
helpful for stabilizing agent training.

For each task, we train an agent for a fixed number of environment steps.
Specifically, for OpenCabinetDoor, OpenCabinetDrawer, and MoveBucket, we train
for 15 million steps. For PushChair, we train for 20 million steps. For
PickObject, we train for 4 million steps. Success rates are calculated among
300 evaluation trajectories.

Hyperparameters | Value
---|---
Optimizer | Adam
Discount ($\gamma$) | 0.95
$\lambda$ in GAE | 0.95
PPO clip range | 0.2
Coefficient of the entropy loss term of PPO $c_{ent}$ | 0.0
Advantage normalization | True
Reward normalization | True
Number of threads for collecting samples | 5
Number of samples per PPO update | 40000
Number of epochs per PPO update | 2
Number of samples per minibatch | 330
Gradient norm clipping | 0.5
Max KL | 0.2
Policy learning rate | 3e-4 (non FM-TG); 1e-4 (FM-TG)
Value learning rate | 3e-4
Action MLP Last Layer Initialization | Zero-init
Table 4: Hyperparameters for PPO.

###  S.5 More Details of Real-World Experiments

Fig. 17 shows the captured RGB images and point clouds in both simulation and
the real world (by RealSense camera). For both simulation and the real-world
environment, the ground points are removed using z-coordinate threshold or
RANSAC, and the distant points are clipped. To reduce the sim-to-real gap, we
only use xyz coordinates as our input point cloud feature, and we discard RGB
colors.


# Decoy Effect in Search Interaction: A Pilot Study

Nuo Chen<EMAIL_ADDRESS>Waseda UniversityTokyoJapan ,  Jiqun Liu
<EMAIL_ADDRESS>The University of OklahomaNormanOK, USA ,  Tetsuya Sakai
<EMAIL_ADDRESS>Waseda UniversityTokyoJapan and  Xiao-Ming Wu  xiao-
<EMAIL_ADDRESS>The Hong Kong Polytechnic UniversityHong KongChina

(2018)

###### Abstract.

In recent years, the influence of cognitive effects and biases on users’
thinking, behaving, and decision-making has garnered increasing attention in
the field of interactive information retrieval. The decoy effect, one of the
main empirically confirmed cognitive biases, refers to the shift in preference
between two choices when a third option (the decoy) which is inferior to one
of the initial choices is introduced. However, it is not clear how the decoy
effect influences user interactions with and evaluations on Search Engine
Result Pages (SERPs). To bridge this gap, our study seeks to understand how
the decoy effect at the document level influences users’ interaction behaviors
on SERPs, such as clicks, dwell time, and usefulness perceptions. We conducted
experiments on two publicly available user behavior datasets and the findings
reveal that, compared to cases where no decoy is present, the probability of a
document being clicked could be improved and its usefulness score could be
higher, should there be a decoy associated with the document.

decoy effect, cognitive bias, interaction information retrieval

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: EVIA ’23: The 10th International Workshop
on Evaluating Information Access; December 12-15, 2023; Tokyo,
Japan††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Users and interactive retrieval

##  1\. Introduction

Understanding how users think, behave, and make decisions during interactions
with search systems represents a foundational research interest in the field
of interactive Information Retrieval (IR). A cognitive bias is a systematic
pattern of deviations in thinking which may lead to irrational judgements and
problematic decision-making (Tversky and Kahneman, 1974, 1992). In recent
years, the exploration of cognitive biases and their impact on the information
seeking and retrieval behaviors and outcomes has garnered increasing attention
(Liu, 2023a; Liu and Shah, 2019; Azzopardi, 2021).

The decoy effect, which is one kind of cognitive bias, describes a situation
in which individuals alter their preference between two initial choices when
introduced to a third (i.e. the decoy), which is asymmetrically inferior to
one of the initial choices (Huber et al., 1982). Figure 1 illustrates an
example of the decoy effect in shopping decision-making. In a shop, a customer
who wants to buy a drink might waver between a 500ml bottle of water (for
$\$1.19$) and a bottle of soda with a similar size (for $\$1.49$). The 500ml
water is cheap, but the soda tastes better, so it might be hard to make a
decision and which one the cosumer will choose might depend on the assessment
of their relative utility. Yet, with a 250ml bottle of water for $\$1.09$
presenting beside the 500ml water, the customer might lean towards the 500ml
water, as they perceive a substantial relative gain from the comparison of the
500ml water and the 250ml water: spending an additional $\$0.10$ to purchase a
500ml bottle of water, compared to the 250ml one, evidently presents a highly
economical deal. In the above example, the 250ml water serves as the decoy to
the target 500ml water.

Figure 1. An example of the decoy effect. The image is generated by the
authors using Midjourney.

In the field of information retrieval, Eickhoff (Eickhoff, 2018) examined the
impact of a decoy document on thresholds and strategies in crowdsourcing
relevance judgments, showing that assessors could increase the relevance
rating of target document when it is shown with the decoy document.
Nevertheless, Eickhoff (2018) only focuses on crowdsourcing assessors
operating within the annotation interface and few research currently addresses
how the decoy effect influences user interactions on Search Engine Result
Pages (SERPs).

To address this gap, our study seeks to understand how the decoy effect at the
document level influences users’ interaction behaviors on SERPs, such as
clicks, browsing dwell time, and usefulness perceptions. We conducted
experiments on two publicly available user behavior datasets and the
experimental results reveal that, compared to cases where no decoy is present,
the probability of a document being clicked could be improved and its
usefulness score could be higher, should there be a decoy associated with the
document.

This study, through the perspective of behavioral economics, delves deeper
into the behavioral patterns and decision-making processes of users
interacting with search engines. The findings stand to encourage IR
researchers to increase the explanatory power of formal search models from a
realistic behavioral and psychological foundation.

##  2\. Related Work

Insights from cognitive psychology and behavioral economics suggest that,
cognitive biases arise from one’s limited cognitive ability when there are no
enough resources to properly collect and process available information
(Kruglanski and Ajzen, 1983). Due to cognitive biases, one’s decisions under
uncertainty can systematically deviate from what is expected given rational
decision-making models (Tversky and Kahneman, 1974, 1991, 1992).

In search and recommendation contexts, interactions between individuals and
systems could lead to the incorporation of behavioral signals, influenced by
cognitive biases, into datasets used for training machine learning algorithms,
thereby potentially magnifying existing system biases (Azzopardi, 2021; Liu,
2023a). Cognitive biases might also result in significant deviations in users’
behaviors and judgements from optimal or desired outcomes. Consequently, this
could give rise to unfair decisions and outcomes between users who are more
susceptible to certain biases and contextual triggers and those who are not
(Liu, 2023b). Therefore, with an increasing number of individuals turning to
search and recommendation systems to access and utilize information for life
decisions, the influence of cognitive biases on the information retrieval
process is drawing heightened attention from IR researchers. Thus far, a lot
of studies have explored the influence of cognitive biases such as the
anchoring effect (Shokouhi et al., 2015), the priming effect (Scholer et al.,
2013; Chandar and Carterette, 2012), the ordering effect (Bansback et al.,
2014), and the reference dependence effect (Liu and Han, 2020; Wang and Liu,
2023) on document examination, relevance judgment, and evaluation of whole-
session search satisfaction. In recent years, with the growing knowledge about
users’ cognitive biases, some works began to introduce cognitive biases into
the construction and meta-evaluation of evaluation metrics (Zhang et al.,
2020; Chen et al., 2022, 2023; Liu and Han, 2022), but this is out of the
scope of this paper.

In this paper, we specifically shed light on one of the cognitive effects, the
decoy effect. The decoy effect, which is one kind of cognitive biases,
describes a situation in which individuals alter their preference between two
initial choices when introduced to a third (i.e. the decoy), which is
asymmetrically inferior to one of the initial choice (Huber et al., 1982). In
the fields of e-commerce and recommendation systems, there have been some
studies exploring the impact of the decoy effect (Teppan et al., 2010; Teppan
and Zanker, 2015; Mo et al., 2022). In the field of information retrieval, it
is not clear how the decoy effect influences user interactions with and
evaluations on Search Engine Result Pages. The work most closely related to
ours in theme is that of Eickhoff (Eickhoff, 2018), which shows that when a
relevant item is presented alongside two non-relevant items, with one non-
relevant item being distinctly inferior (i.e., the decoy), assessors tend to
rate the superior non-relevant document as more relevant. Nevertheless,
Eickhoff (2018) only focuses on crowdsourcing assessors operating within the
annotation interface and our study addresses how the decoy effect could
influence user interactions on Search Engine Result Pages (SERPs), which is a
broader scenario.

##  3\. Research Question

To address the research gap mentioned above, in this study, we seek to
understand how the decoy effect at the document level influences users’
interaction behaviors on SERPs, such as clicks, browsing dwell time, and
usefulness perceptions. Specifically, our work sought to answer following
research question (RQ):

How, and to what extent, the presence of a decoy influences the likelihood of
a document being clicked, the browsing duration on it, and its perceived
usefulness?

Figure 2.  Data processing flow in the first experiment.

Figure 3. The distribution of click probability, browsing time and usefulness
score on THUIR2016 dataset (top) and THU-KDD (bottom) dataset respectively.

Figure 4. The distribution of rank on THUIR2016 (left) dataset and THU-KDD
(right) dataset respectively.

##  4\. Experiment

In this section, we introduce the datasets employed and the methodology
adopted in the experiment, as well as the experimental results.

###  4.1. Datasets

The THUIR2016 dataset (Mao et al., 2016) is collected under controlled
laboratory setting, whereby participants were instructed to execute a given
intricate search tasks utilizing commercial search engines. This dataset
encompasses a total of $9$ topics, $225$ search sessions and $933$ queries,
along with the title and snippets on the SERP of each query. It also contains
$4$-level user self-rating usefulness scores for the items they clicked and
$5$-level graded relevance labels collected from external assessors.

The THU-KDD dataset (Liu et al., 2019) is also collected under controlled
laboratory setting similar to the THUIR2016 dataset. This dataset encompasses
a total of $9$ topics, $450$ search sessions and around $1100$ queries, along
with the title and snippets on the SERP of each query. It also contains
$4$-level user self-rating usefulness scores for the items they clicked and
$4$-level graded relevance labels collected from external assessors.

###  4.2. Data Processing Approach

To find out potential decoy instances from user logs, we first proffer a
definition of a decoy instance. A pair of documents, composed of the target
document and the decoy document $(t,d)$, constitutes a decoy instance if and
only if the following conditions are met: (1) $t$ and $d$ share certain degree
of similarity in content, albeit not identical, i.e.,
$S_{\min}\leq\text{similarity}(t,d)\leq S_{\max}$, where $S_{\min}$ and
$S_{\max}$ respectively represent the minimum and maximum similarity
thresholds; (2) $d$ is inferior in quality to $t$, i.e.,
$\text{quality}(t)>\text{quality}(d)$; (3) the position $t$ and $d$ within a
SERP is close enough, i. e., $|rank(t)-rank(d)|\leq\Delta_{\mathrm{rank}}$.

In our experiments, given that the documents are all in Chinese, we first
concatenate the title and snippet into a string. Subsequently, we employ a
tool named jieba111https://github.com/fxsjy/jieba to carry out word
segmentation, obtaining a token list. We then calculated the cosine similarity
with the vanilla definition (Han et al., 2012) between each pair of documents
under the same topic. We designated $S_{\min}$ as the 99th percentile of
document similarity in each dataset, setting $S_{\max}$ to $0.95$. In
THUIR2016 dataset, $S_{\min}$ stands at $0.626$ and in THU-KDD dataset,
$S_{\min}$ stands at $0.594$. For the second condition, we employ the
relevance scores given by external assessors as the measurement of document
quality, mandating that $\text{relevance}(t)-\text{relevance}(d)\geq 2$ to
ensure that the decoy is substantially inferior to the target. For the third
condition, we require the absolute value of the difference of the rank between
$t$ and $d$ is smaller than or equal to 5 ($\Delta_{\mathrm{rank}}=5$). We
processed the top 10 documents in each SERP on all datasets adhering to the
aforementioned three conditions, and we identified $982$ records of decoy
pairs involving $318$ distinct target documents in the THUIR2016 dataset; and
$922$ records of decoy pairs involving $376$ distinct target documents. in the
THU-KDD dataset. In the following discourse, we denote the set consisting of
all target documents in a corpus as $\mathcal{T}$.

To investigate whether user interactions with documents are disparate when no
decoy is present compared to situations with a decoy, we assign some documents
not in $\mathcal{T}$ to the control group (i.e., control documents), adhering
to the following condition: A document $c$ which is not in the set of target
documents (i.e., $c\notin\mathcal{T}$) is considered a control document if and
only if it matches a target document $t\in\mathcal{T}$ such that
$\text{similarity}(c,t)\geq S_{\mathbf{control}}$ and
$|\text{relevance}(c)-\text{relevance}(t)|<=2$. We denote the set of all such
$c$ as $\mathcal{C}$, and the set of all $t$ that can match with at least one
$c$ as $\mathcal{T}^{\prime}$, where $\mathcal{T}^{\prime}\subset\mathcal{T}$.
In our experiments, we set $S_{\mathbf{control}}$ to the $99.5$th percentile
of document similarity in each dataset. In THUIR2016 dataset,
$S_{\mathbf{control}}$ stands at $0.709$ and in THU-KDD dataset,
$S_{\mathbf{control}}$ stands at $0.676$. According to the aforementioned
condition, we have identified $741$ qualifying control documents in the
THUIR2016 dataset and $1790$ in the THU-KDD dataset.

We then extracted interaction records of all control documents in the
THUIR2016 and THU-KDD datasets, obtaining $1384$ and $2770$ records
respectively. Subsequently, from the records of decoy pairs in the THUIR2016
and THU-KDD datasets (with $982$ and $922$ records respectively), we filter
out all records where $t\in\mathcal{T}^{\prime}$, obtaining $739$ and $828$
records respectively. Note that, for decoy pairs from the same SERP
interaction $i$, there could be situations where the same target document
corresponds to multiple decoy documents. In our filtering process, we ensure
that for a given SERP interaction $i$ and a given target document $t$, only
one record is eventually extracted. We concatenate the interaction records of
target documents and control documents, ultimately obtaining document
interaction record lists of lengths $2123$ and $3598$ in the two respective
datasets. These two lists of interactions will be employed for the subsequent
data analysis. In the subsequent analysis, we process the interaction signals
as follows: for documents that have not been clicked, their usefulness score
is assigned a value of 0, and their browsing duration is also set to 0. Figure
2 provides a brief outline of our data processing workflow.

###  4.3. Data Analysis and Experimental Results

From Figure 3, one can see that in both datasets, there are more instances
receiving a score of $4$ (the maximum) in usefulness. On the THU-KDD dataset,
the target group exhibited fewer instances labeled as 0 and 1; the proportions
of instances labeled as 2 and 3 were approximately similar between the target
and control groups. On the THUIR2016 dataset, the target group exhibited fewer
instances labeled as 1 and 2. Additionally, the proportions of instances
labeled as 0 and 3 were approximately similar between the target and control
groups. For click probability, there is no difference in the THUIR2016
dataset, while in the THU-KDD dataset, the click probability of target group
is higher. Regarding browsing duration, on the THUIR2016 dataset, the target
group exhibited a higher proportion of instances with browsing times ranging
from 30 to 60 seconds. Apart from this observation, the distributions of
browsing durations for both the target and control groups were approximately
similar. On the THU-KDD dataset, the target group exhibited a higher
proportion of instances with browsing times ranging between 0 to 5 seconds and
30 to 60 seconds, while a smaller proportion of instances had browsing times
between 5 to 30 seconds.

From Figure 3, it is challenging to derive an intuitive conclusion, especially
regarding the relationship between the decoy effect and variables such as
click probability, browsing duration and usefulness. Nevertheless, Figure 4
shows that, across both datasets, the distribution of target documents and
control documents over ranks diverges. Hence, it is necessary to factor out
any latent effects stemming from position biases on our results in order to
draw a conclusion on the relationship between the presence of decoy and users’
behavior.

To control the impact caused by rank position, we employ regression analysis
to investigate the relationships between the presence of a decoy and the
probability of clicks, browsing duration, and usefulness scores. We
constructed three regression models, taking whether the document is clicked
(is_clicked), browsing duration (duration), and usefulness score (usefulness)
as dependent variables respectively, and the presence of a decoy (has_decoy),
the document’s rank (rank), task ID (task_id), and user ID (user_id) as
independent variables.

Note that contrary to computer science, in econometrics, regression models are
predominantly employed for interpretation rather than for prediction. In a
multiple regression model, each coefficient tells people the impact on the
dependent variable of a one-unit change in that independent variable, holding
all other independent variables constant (Smith, 2012). In this study, we
foucus on elucidating how, and to what extent, the presence of a decoy
influences the likelihood of a document being clicked, the browsing duration
on it, and its perceived usefulness. Hence, we do not partition the dataset
into training and test subsets; instead, we perform regression on the entirety
of the data. Including rank, task_id, and user_id as independent variables
serves to use them as control variables to mitigate the potential influences
from rank position, task type, and individual characteristics on the outcomes,
thus better elucidating how variations in has_decoy would affect the values of
is_clicked, duration, and usefulness. For is_clicked, we employ Logistic
regression, and for duration and usefulness, we resort to Ordinary Least
Squares (OLS) regression.

Table 1 shows the regression coefficient of the independent variable has_decoy
with the dependent variables is_clicked, duration and usefulness. As
previously stated, our focus in this research is to elucidate in what manner
and to what extent the presence of a decoy impacts whether a document is
clicked, the browsing duration, and the usefulness scores, and rank, task_id,
and user_id are included merely to control for the effects brought by rank
position, task, and individual characteristic respectively. Therefore, we opt
to omit the reporting of the constant as well as the regression coefficients
of rank, task_id, and user_id in the tables.

From Table 1, one can observe that: across the two datasets, the presence of a
decoy could exert a positive influence on the likelihood of being clicked
(coefficient = $0.363$ and $0.217$ respectively) and on the usefulness score
(coefficient = $0.136$ and $0.156$ respectively), all with a statistical
significance at the level of $p<0.05$. The existence of a decoy also seems to
render a positive impact on duration (coefficient = $1.916$ and $1.913$
respectively), however, the result on both datasets is not statistically
significant.

More precisely, the regression results can be interpreted as follows: Given
the document rank, type of task, and individual characteristics, when a decoy
is present, in comparison to when it is absent (1) the likelihood of being
clicked on the THUIR2016 and THU-KDD datasets would respectively increase by
$36.3\%$ ($p<0.05$) and $21.7\%$ ($p<0.05$); (2) browsing time duration would
rise by $1.92$s and $1.91$s respectively on the THUIR2016 and THU-KDD
datasets; (3) usefulness score would escalate by $0.136$ ($p<0.01$) and
$0.156$ ($p<0.01$) respectively on the THUIR2016 and THU-KDD datasets.

| THUIR2016 | THU-KDD
---|---|---
Dependent Variable | Coefficient | Coefficient
is_clicked | 0.363* | 0.217*
duration | 1.916 | 1.913
usefulness | 0.136 ** | 0.156 **
Table 1. The regression coefficient of the independent variable has_decoy with
the dependent variables is_clicked, duration, and usefulness. * and **
respectively indicate that the coefficients are significant at levels of
$p<0.05$, $p<0.01$.

##  5\. Conclusion

In this study, we seek to comprehend how the decoy effect at the document
level impacts users’ interaction behaviors on SERPs, such as clicks, dwell
time, and usefulness perceptions. We conducted descriptive data analysis and
regression analysis on two publicly available user behavior datasets. The
experimental results indicate that, the likehood of a document being clicked
could be improved and its usefulness score could be elevated, should there be
a decoy associated with the document. From the experimental results, we
observe that, given the document rank, type of task and individual
characteristics, when a decoy is present, in comparison to when it is absent,
there is a significant increment in the likehood of a document being clicked
and its perceived usefulness.

As far as we know, we are the first to addresses how the decoy effect
influences user interactions on Search Engine Result Pages. Our work extends
the endeavors of the IR community in exploring how cognitive biases impact
user behaviors in document examining and relevance judgment, providing
evidence from the perspective of the decoy effect.

However, our study is merely in a preliminary stage, with numerous aspects
awaiting further exploration. For instance: (1) Does the impact of the decoy
vary in size under different themes, topics, or search tasks? (2) Within the
same search session, does the impact of decoy results vary across different
cognitive states and information seeking intentions (e.g. exploring an
unfamiliar domain, seeking for a known item, evaluating retrieved information)
(Liu et al., 2020)? (2) The potential impact of the decoy effect on the
benefit of users and information providers, as well as possible social and
ethical issues it may bring about, such as fairness. Addressing these
questions will necessitate subsequent experiments to provide more evidence.

## Acknowledgement

We thank Prof. Kana Shimizu for providing valuable feedback for this study. We
also thank all reviewers of this paper for their valuable feedback and
constructive criticism. Jiqun Liu’s participation in this project is partially
supported by the Bridge Fuding Investment Program (BFIP) Award from the
University of Oklahoma Office of the Vice President for Research and
Partnerships.

## References

  * (1)
  * Azzopardi (2021) Leif Azzopardi. 2021.  Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval. In _Proceedings of the 2021 Conference on Human Information Interaction and Retrieval_ (Canberra ACT, Australia) _(CHIIR ’21)_. Association for Computing Machinery, New York, NY, USA, 27–37.  https://doi.org/10.1145/3406522.3446023
  * Bansback et al. (2014) Nick Bansback, Linda C. Li, Larry Lynd, and Stirling Bryan. 2014.  Exploiting order effects to improve the quality of decisions.  _Patient Education and Counseling_ 96, 2 (2014), 197–203.  https://doi.org/10.1016/j.pec.2014.05.021
  * Chandar and Carterette (2012) Praveen Chandar and Ben Carterette. 2012.  Using Preference Judgments for Novel Document Retrieval. In _Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Portland, Oregon, USA) _(SIGIR ’12)_. Association for Computing Machinery, New York, NY, USA, 861–870.  https://doi.org/10.1145/2348283.2348398
  * Chen et al. (2023) Nuo Chen, Jiqun Liu, and Tetsuya Sakai. 2023.  A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users. In _Proceedings of the ACM Web Conference 2023_ (Austin, TX, USA) _(WWW ’23)_. Association for Computing Machinery, New York, NY, USA, 3396–3405.  https://doi.org/10.1145/3543507.3583551
  * Chen et al. (2022) Nuo Chen, Fan Zhang, and Tetsuya Sakai. 2022.  Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Madrid, Spain) _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 2709–2714.  https://doi.org/10.1145/3477495.3531953
  * Eickhoff (2018) Carsten Eickhoff. 2018.  Cognitive Biases in Crowdsourcing. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ (Marina Del Rey, CA, USA) _(WSDM ’18)_. Association for Computing Machinery, New York, NY, USA, 162–170.  https://doi.org/10.1145/3159652.3159654
  * Han et al. (2012) Jiawei Han, Micheline Kamber, and Jian Pei. 2012.  2.4.7. Cosine Similarity.  In _Data Mining (Third Edition)_ (third edition ed.), Jiawei Han, Micheline Kamber, and Jian Pei (Eds.). Morgan Kaufmann, Boston, 77–78.  https://doi.org/10.1016/B978-0-12-381479-1.00002-2
  * Huber et al. (1982) Joel Huber, John W. Payne, and Christopher Puto. 1982.  Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis.  _Journal of Consumer Research_ 9, 1 (1982), 90–98.  http://www.jstor.org/stable/2488940
  * Kruglanski and Ajzen (1983) Arie W. Kruglanski and Icek Ajzen. 1983.  Bias and error in human judgment.  _European Journal of Social Psychology_ 13, 1 (1983), 1–44.  https://doi.org/10.1002/ejsp.2420130102 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/ejsp.2420130102
  * Liu (2023a) Jiqun Liu. 2023a.  _A Behavioral Economics Approach to Interactive Information Retrieval: Understanding and Supporting Boundedly Rational Users_. Vol. 48.  Springer Nature.
  * Liu (2023b) Jiqun Liu. 2023b.  Toward A Two-Sided Fairness Framework in Search and Recommendation. In _Proceedings of the 2023 Conference on Human Information Interaction and Retrieval_ (Austin, TX, USA) _(CHIIR ’23)_. Association for Computing Machinery, New York, NY, USA, 236–246.  https://doi.org/10.1145/3576840.3578332
  * Liu and Han (2020) Jiqun Liu and Fangyuan Han. 2020.  Investigating Reference Dependence Effects on User Search Interaction and Satisfaction: A Behavioral Economics Perspective. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 1141–1150.  https://doi.org/10.1145/3397271.3401085
  * Liu and Han (2022) Jiqun Liu and Fangyuan Han. 2022.  Matching search result diversity with user diversity acceptance in Web search sessions. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2473–2477.
  * Liu et al. (2020) Jiqun Liu, Shawon Sarkar, and Chirag Shah. 2020.  Identifying and predicting the states of complex search tasks. In _Proceedings of the 2020 conference on human information interaction and retrieval_. 193–202.
  * Liu and Shah (2019) Jiqun Liu and Chirag Shah. 2019.  Investigating the impacts of expectation disconfirmation on web search. In _Proceedings of the 2019 conference on human information interaction and retrieval_. 319–323.
  * Liu et al. (2019) Mengyang Liu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019.  Investigating Cognitive Effects in Session-Level Search User Satisfaction. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (Anchorage, AK, USA) _(KDD ’19)_. Association for Computing Machinery, New York, NY, USA, 923–931.  https://doi.org/10.1145/3292500.3330981
  * Mao et al. (2016) Jiaxin Mao, Yiqun Liu, Ke Zhou, Jian-Yun Nie, Jingtao Song, Min Zhang, Shaoping Ma, Jiashen Sun, and Hengliang Luo. 2016.  When Does Relevance Mean Usefulness and User Satisfaction in Web Search?. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Pisa, Italy) _(SIGIR ’16)_. Association for Computing Machinery, New York, NY, USA, 463–472.  https://doi.org/10.1145/2911451.2911507
  * Mo et al. (2022) Fan Mo, Tsuneo Matsumoto, Nao Fukushima, Fuyuko Kido, and Hayato Yamana. 2022.  Decoy Effect of Recommendation Systems on Real E-commerce Websites. In _CEUR Workshop Proceedings_ , Vol. 3222. CEUR-WS, 151–163.
  * Scholer et al. (2013) Falk Scholer, Diane Kelly, Wan-Ching Wu, Hanseul S. Lee, and William Webber. 2013.  The Effect of Threshold Priming and Need for Cognition on Relevance Calibration and Assessment. In _Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Dublin, Ireland) _(SIGIR ’13)_. Association for Computing Machinery, New York, NY, USA, 623–632.  https://doi.org/10.1145/2484028.2484090
  * Shokouhi et al. (2015) Milad Shokouhi, Ryen White, and Emine Yilmaz. 2015.  Anchoring and Adjustment in Relevance Estimation. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Santiago, Chile) _(SIGIR ’15)_. Association for Computing Machinery, New York, NY, USA, 963–966.  https://doi.org/10.1145/2766462.2767841
  * Smith (2012) Gary Smith. 2012.  Chapter 10 - Multiple Regression.  In _Essential Statistics, Regression, and Econometrics_ , Gary Smith (Ed.). Academic Press, Boston, 297–331.  https://doi.org/10.1016/B978-0-12-382221-5.00010-6
  * Teppan et al. (2010) Erich Christian Teppan, Gerhard Friedrich, and Alexander Felfernig. 2010.  Impacts of Decoy Effects on the Decision Making Ability. In _2010 IEEE 12th Conference on Commerce and Enterprise Computing_. 112–119.  https://doi.org/10.1109/CEC.2010.30
  * Teppan and Zanker (2015) Erich Christian Teppan and Markus Zanker. 2015.  Decision Biases in Recommender Systems.  _Journal of Internet Commerce_ 14, 2 (2015), 255–275.  https://doi.org/10.1080/15332861.2015.1018703 arXiv:https://doi.org/10.1080/15332861.2015.1018703
  * Tversky and Kahneman (1974) Amos Tversky and Daniel Kahneman. 1974.  Judgment under Uncertainty: Heuristics and Biases.  _Science_ 185, 4157 (1974), 1124–1131.  https://doi.org/10.1126/science.185.4157.1124 arXiv:https://www.science.org/doi/pdf/10.1126/science.185.4157.1124
  * Tversky and Kahneman (1991) Amos Tversky and Daniel Kahneman. 1991.  Loss Aversion in Riskless Choice: A Reference-Dependent Model.  _Quarterly Journal of Economics_ 106 (1991), 1039–1061.
  * Tversky and Kahneman (1992) Amos Tversky and Daniel Kahneman. 1992.  Advances in prospect theory: Cumulative representation of uncertainty.  _Journal of Risk and Uncertainty_ 5 (1992), 297–323.
  * Wang and Liu (2023) Ben Wang and Jiqun Liu. 2023.  Investigating the role of in-situ user expectations in Web search.  _Information Processing & Management_ 60, 3 (2023), 103300.
  * Zhang et al. (2020) Fan Zhang, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Min Zhang, and Shaoping Ma. 2020.  Cascade or Recency: Constructing Better Evaluation Metrics for Session Search. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 389–398.  https://doi.org/10.1145/3397271.3401163