# Inspecting Model Fairness in Ultrasound Segmentation Tasks

###### Abstract

With the rapid expansion of machine learning and deep learning (DL),
researchers are increasingly employing learning-based algorithms to alleviate
diagnostic challenges across diverse medical tasks and applications. While
advancements in diagnostic precision are notable, some researchers have
identified a concerning trend: their models exhibit biased performance across
subgroups characterized by different sensitive attributes. This bias not only
infringes upon the rights of patients but also has the potential to lead to
life-altering consequences. In this paper, we inspect a series of DL
segmentation models using two ultrasound datasets, aiming to assess the
presence of model unfairness in these specific tasks. Our findings reveal that
even state-of-the-art DL algorithms demonstrate unfair behavior in ultrasound
segmentation tasks. These results serve as a crucial warning, underscoring the
necessity for careful model evaluation before their deployment in real-world
scenarios. Such assessments are imperative to ensure ethical considerations
and mitigate the risk of adverse impacts on patient outcomes.

Index Terms—  Fairness, Segmentation, Ultrasound

##  1 Introduction

In recent years, the widespread adoption of machine learning and deep learning
(DL) has played a pivotal role in assisting medical professionals with a
spectrum of tasks related to the analysis of medical images. These tasks range
from fundamental processes such as reconstruction and denoising to more
intricate endeavors including classification, segmentation, and detection [1].

The integration of these advanced methodologies enables doctors to derive
disease predictions directly from raw medical images, leading to a reduction
in workload, improved work efficiency, and a significant enhancement in
diagnostic precision. The versatility of DL methods has been successfully
demonstrated across various imaging modalities, encompassing X-Ray [2],
Magnetic Resonance Imaging (MRI) [3], Computed Tomography (CT) [4], Ultrasound
(US) [5], and others. This broad applicability underscores the transformative
impact of DL in diverse medical imaging applications, promising continued
advancements in the realms of healthcare diagnostics and treatment planning.

However, there are also some unobtrusive shortcomings of DL models in medical
applications. One among them is model unfairness [6], which is defined as the
utility disparity of DL models when applied to sub-populations with different
sensitive attributes, such as Age, Sex, Race, and Marital. This phenomenon
occurs widely in medical classification tasks.

Nonetheless, there are inconspicuous limitations in the application of DL
models to medical scenarios. One notable concern is the issue of unfairness in
models, characterized by a discernible discrepancy in their utility when
employed across sub-populations distinguished by different sensitive
attributes, such as age, sex, race, or marital status. This phenomenon is
notably prevalent in medical classification tasks.

For instance, Larrazaba et al. [4] observed substantial performance variations
among three DL models when applied to distinct gender groups (male and female)
in a chest X-ray diagnosis task. Similarly, researchers have identified this
phenomenon in skin lesion detection tasks, revealing that state-of-the-art DL
models exhibit uneven performances between patients with dark and light skin
tones [7, 8].

In recent times, researchers have turned their attention to the critical task
of evaluating fairness in segmentation processes. Much of this research has
concentrated on brain MR images, uncovering instances of model unfairness,
particularly in brain tumor segmentation [9]. The underlying reason for this
phenomenon lies in the relative ease with which DL models can predict the sex
and age of subjects in MR images compared to directly predicting Alzheimer’s
state. Consequently, models may inadvertently exploit spurious correlations
between sex and the target label, resulting in observed unfairness [10].

Comparatively, obtaining sex and age information from US images is inherently
more challenging than from MR images. Furthermore, as of our current
knowledge, there is a lack of evidence confirming the absence of unfairness in
US segmentation tasks. Therefore, it is imperative to scrutinize the potential
existence of unfairness issues in US segmentation. This examination is crucial
not only for validating the origin of unfairness but also for alerting
patients to the necessity of careful consideration before embracing DL models
for US segmentation applications.

Our contributions to the field are outlined as follows:

(1) We conduct the first comprehensive assessment of model fairness in US
segmentation tasks. This evaluation encompasses seven state-of-the-art U-Net
based segmentation algorithms.

(2) Through rigorous statistical analysis, we identify varying degrees of
unfairness within the state-of-the-art segmentation methods. This underscores
the importance of acknowledging and addressing model unfairness as a critical
consideration before the deployment of these methods.

Dataset | TUSC | TUS
---|---|---
Age | Young | 479 (55.70%) | 884 (56.23%)
Old | 381 (44.30%) | 688 (43.77%)
Sex | Female | 722 (83.95%) | 1331 (84.67%)
Male | 138 (16.05%) | 241 (15.33%)
Table 1: Dataset distribution of TUSC and TUS

TUSC Dataset | TUS Dataset
---|---
Method | Dice $\uparrow$ | Sex (%) | Age (%) | Dice $\uparrow$ | Sex (%) | Age (%)
ESSP $\uparrow$ | DD $\downarrow$ | EOpp $\downarrow$ | ESSP $\uparrow$ | DD $\downarrow$ | EOpp $\downarrow$ | ESSP $\uparrow$ | DD $\downarrow$ | EOpp $\downarrow$ | ESSP $\uparrow$ | DD $\downarrow$ | EOpp $\downarrow$
CNN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |
U-Net [11] | 67.57 $\pm$ 0.71 | 67.47 | 0.30 | 4.06 | 66.20 | 4.15 | 3.32 | 82.06 $\pm$ 0.44 | 81.88 | 0.44 | 0.44 | 81.63 | 1.05 | 2.18
UNeXt [12] | 63.56 $\pm$ 1.31 | 63.51 | 0.17 | 0.83 | 62.56 | 3.19 | 1.93 | 79.14 $\pm$ 0.53 | 78.49 | 1.65 | 0.13 | 78.71 | 1.11 | 1.74
U-Net++ [13] | 71.78 $\pm$ 0.36 | 71.19 | 1.67 | 1.56 | 70.90 | 2.50 | 4.40 | 81.72 $\pm$ 0.68 | 81.68 | 0.10 | 1.25 | 81.45 | 0.67 | 1.83
Attention U-Net [14] | 68.22 $\pm$ 0.87 | 67.74 | 1.40 | 3.24 | 66.80 | 4.24 | 3.54 | 82.19 $\pm$ 0.42 | 82.09 | 0.25 | 0.98 | 81.75 | 1.07 | 1.27
CMU-Net [5] | 74.41 $\pm$ 1.26 | 74.09 | 0.84 | 1.04 | 73.57 | 2.26 | 3.82 | 82.54 $\pm$ 0.46 | 82.35 | 0.47 | 0.27 | 81.95 | 1.43 | 2.31
CMUNeXt [15] | 68.07 $\pm$ 0.74 | 67.37 | 2.07 | 1.28 | 66.65 | 4.26 | 4.12 | 81.94 $\pm$ 0.34 | 81.90 | 0.10 | 1.02 | 81.61 | 0.81 | 1.68
Transformer-based |  |  |  |  |  |  |  |  |  |  |  |  |  |
TransUnet [16] | 68.75 $\pm$ 1.29 | 68.38 | 1.07 | 1.16 | 67.13 | 4.80 | 5.87 | 81.66 $\pm$ 0.18 | 81.59 | 0.15 | -0.12 | 81.44 | 0.52 | 1.59

Table 2: Performance and fairness scores on Sex and Age attribute. The best
and second best are highligted.

##  2 Method

###  2.1 Problem Definition

Considering a medical segmentation task, where the input dataset $D$ consists
of $N$ samples, i.e. $D=\\{s_{1},s_{2},...,s_{N}\\}$. Each sample
$s_{i}=\\{X_{i},Y_{i},A_{i}\\}$ is a tuple of the input image $X_{i}\in
R^{C\times W\times H}$, the ground truth mask $Y_{i}\in R^{C\times W\times
H}$, and a sensitive attribute $A_{i}\in\\{0,1\\}$, which usually is a binary
variable that describes the metadata of the sample, such as Age, Sex, and
Race. In group fairness settings, the performance of the DL model is computed
per sample and averaged on the groups split by sensitive attributes. Ideal
group fairness requires that different subgroups to have the same performance,
which is hard to achieve. Thus, the performance disparity is computed across
subgroups to measure the unfairness.

###  2.2 Experiment Settings

In this paper, we conduct a group fairness evaluation on two US datasets: the
publicly available Thyroid UltraSound Cine-clip dataset [17] (TUSC) and a
private Thyroid Ultrasound dataset (TUS). Both datasets include sex and age
information for the patients, and their distributions are presented in the
table.

For the variable Age, treated as a continuous variable, we adopt a threshold
of 60 for the Age subgroup split in TUSC, while for TUS, we simplify the
threshold to 45 years old, following the processing outlined in [7]. The TUSC
dataset comprises ultrasound cine-clip images from 167 patients gathered at
Stanford University Medical Center. Due to slight differences in the images
extracted from the ultrasound video, we resample the dataset at a ratio of 5,
resulting in a dataset of 860 images. The TUS dataset is collected from the
Ultrasound Department of the Affiliated Hospital of Qingdao University,
consisting of 1,942 images from 192 cases, annotated by three experienced
radiologists. For both datasets, we randomly partition them into training and
testing sets with a ratio of 7:3.

Our evalutations are done with seven state-of-the-art image segmentation
methods, including CNN-baseds network such as: U-Net [11], U-Net++ [13], UNeXt
[12], Attention U-Net [14], CMU-Net [5], and CMUNeXt [15], and transformer-
based networks TransUnet [16].

For fair comparison, all of the networks are trained with the same loss
$\mathcal{L}$, consisting of binary cross-entropy loss and dice loss, given by
the following equation:

$\mathcal{L}=\mathcal{L}_{Dice}(y,\hat{y})+\alpha\mathcal{L}_{BCE}(y,\hat{y})$
(1)

where $y$ and $\hat{y}$ denote the ground truth mask and the model prediction,
respectively. $\alpha$ is the weight and is set as $\alpha=0.5$.

For utility evaluation, we employ the Dice metric directly. Regarding fairness
assessment, drawing inspiration from the concept introduced in [18], we
utilize the Equity-Scaled Segmentation Performance (ESSP) and Dice Disparity
(DD) to evaluate unfairness in US segmentation. The formulations for ESSP and
DD are provided by Equations 2 and 3.

$\text{ESSP}=\frac{\text{Dice}_{avg}}{1+\text{Dice}_{std}}$ (2)

$\text{DD}=\text{Dice}_{A=1}-\text{Dice}_{A=0}$ (3)

where $\text{Dice}_{A=0}$ denotes the average Dice of the unprivileged group
(lower performance) and $\text{Dice}_{A=1}$ denotes the average Dice of the
privileged group (higher performance).

We also use Equal Opportunity (EOpp) [19], which is a commonly used metric
designed for fairness evaluation on classification tasks, by regarding the
segmentation as pixel-level classification. The formula of EOpp is given by
Equ. 4.

$\text{EOpp}=\text{TPR}_{A=1}-\text{TPR}_{A=0}$ (4)

where $\text{EOpp}_{A=0}$ denotes the True Positive Rate (TPR) of the
unprivileged group (lower performance) and $\text{EOpp}_{A=1}$ denotes the TPR
of the privileged group (higher performance).

We resize all training cases of four datasets to 256$\times$256 and apply
random rotation and flip for simple image augmentations. In addition, we use
the Adam optimizer with a weight decay of 1e-4 and a momentum of 0.9 for
training. The initial learning rate is set to 0.01, and the poly strategy is
used to adjust the learning rate. The batch size is set to 8 and the training
epochs are 300. All the experiments are conducted using 8x NVIDIA GeForce
RTX3090Ti GPUs.

(a) Result on Sex on TUSC Dataset.

(b) Result on Age on TUSC Dataset.

(c) Result on Sex on TUS Dataset.

(d) Result on Age on TUS Dataset.

Fig. 1: Overall Dice for each of the seven models tested. Statistical
significance was found using a Mann-Whitney U test and is denoted by **** (
$p\leq$ 0.0001), *** (0.001 $<p\leq$ 0.0001), ** (0.01 $<p\leq$ 0.001), *
(0.01 $<p\leq$ 0.05), ns (0.05 $\leq p$ ).

##  3 Result

In our analysis of fairness regarding predictive performance and model
calibration, Table 2 presents various metrics illustrating the performance of
the DL models for US diagnosis. The metrics are averaged over the test set
across three runs, along with corresponding 95% confidence intervals.

Fig. 2: Male and Female images in an US dataset and a skin lesion dataset.
Compared to skin, US images are more difficult to distinguish the sex of the
image.

###  3.1 Quntitative Evaluation

TUSC Dataset: Among the seven methods evaluated on the TUSC, the average Dice
score is approximately 68%. CMU-Net achieves the highest performance with a
Dice score of 74.41%, while U-Net demonstrates the least favorable
performance. In terms of fairness criteria, CMU-Net exhibits the best ESSP for
both Sex and Age, which can be attributed to its high segmentation utility.
Conversely, UNeXt records the lowest DD and EOpp for Sex, and the lowest EOpp
for Age.

TUS Dataset: The trends are more distinct for the seven methods on the TUS.
CMU-Net and Attention U-Net demonstrate the best and second-best performance,
as well as fairness, across almost all metrics, followed by TransUnet and
U-Net++. Conversely, U-Net, UNeXt, and CMUNeXt exhibit higher levels of
unfairness on both Sex and Age.

###  3.2 Statistical Evaluation

From Fig. 1, it is evident that, across all seven algorithms evaluated on the
two datasets, the performance of the female group surpasses that of the male
group. Additionally, the performance of the young group is consistently higher
than that of the male group. The uniform performance disparities concerning
these two sensitive attributes underscore varying degrees of unfairness across
all seven algorithms.

To delve deeper, a Mann-Whitney U test is conducted on the Dice scores of the
seven algorithms for the two sensitive attributes on the test set. Notably,
U-Net, the foundational model for other sex-specific methods, exhibits
statistically significant performance disparities on Sex in the TUSC dataset
and on Age in both TUSC and TUS datasets. Furthermore, TransUnet (on TUSC),
Attention U-Net, and CMUNet (on TUS) show distinguishable performances on the
Female and Male groups ($0.01<p\leq 0.05$). UNeXt demonstrates significant
unfairness on Age (on TUSC). All algorithms, except for UNeXt and UNet++,
exhibit significant unfairness on the Age attribute in the TUS dataset.

In conclusion, the presence of model unfairness is evident across all seven
models, as observed through both quantitative evaluation and statistical
analysis. This sheds light on the importance of considering fairness alongside
the segmentation utility of deep learning models.

##  4 Discussion

To the best of our knowledge, this work represents the first comprehensive
analysis and comparison of fairness issues in US segmentation tasks. Through
the computation of three fairness metrics and rigorous statistical hypothesis
testing, we reveal that all seven state-of-the-art U-Net-based segmentation
models exhibit varying degrees of unfairness on the two US datasets.

This outcome is unexpected, considering that prior research on fairness
typically focused on tasks where sensitive attributes could be easily
distinguished. For instance, in skin lesion classification, differences in
skintone are readily apparent. Contrary to these tasks, US images present a
more challenging scenario for distinguishing between male and female, as
illustrated in Fig. 2.

Given that most fairness research in medical image analysis has been conducted
on classification tasks, where acknowledged fairness metrics exist, the
evaluation of unfairness in segmentation tasks, especially US segmentation,
requires careful consideration. While some studies adopt definitions from
classification tasks, applying metrics like the disparity of Dice or Hausdorff
Distance in segmentation, the unique challenges of evaluating unfairness in US
segmentation demand further exploration.

Our study has certain limitations. We independently consider Sex and Age,
without evaluating the intersection between these two variables, which may
introduce higher levels of unfairness due to the complex relationships among
segmentation targets, sex, and age. Additionally, assessing model unfairness
based on the actual age of patients, rather than a binary categorical label,
is an essential step in the analysis process.

In conclusion, the experiments in this paper uncover the presence of
unfairness in US segmentation tasks. Therefore, careful use of these deep
learning models is imperative to avoid producing unfair diagnoses. Future work
should focus on designing fairness mitigation algorithms tailored to address
this specific challenge in US segmentation.

##  5 Conclusion

In this paper, we present the inaugural in-depth evaluation of fairness issues
in US segmentation tasks, assessing seven state-of-the-art DL-based
segmentation models. The results reveal varying degrees of unfairness across
all these methods, underscoring the importance of considering model unfairness
alongside utility. This not only contributes to enhancing the reliability and
interpretability of deep learning-based models but also serves as a critical
step in preventing the unequal treatment of underprivileged patients.

## References

  * [1] S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L Prince, Daniel Rueckert, and Ronald M Summers,  “A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises,”  Proceedings of the IEEE, vol. 109, no. 5, pp. 820–838, 2021.
  * [2] Ivo M Baltruschat, Hannes Nickisch, Michael Grass, Tobias Knopp, and Axel Saalbach,  “Comparison of deep learning approaches for multi-label chest x-ray classification,”  Scientific reports, vol. 9, no. 1, pp. 6381, 2019.
  * [3] Zeynettin Akkus, Alfiia Galimzianova, Assaf Hoogi, Daniel L Rubin, and Bradley J Erickson,  “Deep learning for brain mri segmentation: state of the art and future directions,”  Journal of digital imaging, vol. 30, pp. 449–459, 2017.
  * [4] Agostina J Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone, and Enzo Ferrante,  “Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis,”  Proceedings of the National Academy of Sciences, vol. 117, no. 23, pp. 12592–12594, 2020.
  * [5] Fenghe Tang, Lingtao Wang, Chunping Ning, Min Xian, and Jianrui Ding,  “Cmu-net: A strong convmixer-based medical ultrasound image segmentation network,”  arXiv preprint arXiv:2210.13012, 2022.
  * [6] Ben Glocker, Charles Jones, Melanie Bernhardt, and Stefan Winzeck,  “Risk of bias in chest x-ray foundation models,”  arXiv preprint arXiv:2209.02965, 2022.
  * [7] Wenlong Deng, Yuan Zhong, Qi Dou, and Xiaoxiao Li,  “On fairness of medical image classification with multiple sensitive attributes via learning orthogonal representations,”  arXiv preprint arXiv:2301.01481, 2023.
  * [8] Zikang Xu, Shang Zhao, Quan Quan, Qingsong Yao, and S Kevin Zhou,  “Fairadabn: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classification,”  arXiv preprint arXiv:2303.08325, 2023.
  * [9] Tiarna Lee, Esther Puyol-Antón, Bram Ruijsink, Keana Aitcheson, Miaojing Shi, and Andrew P King,  “An investigation into the impact of deep learning model choice on sex and race bias in cardiac mr segmentation,”  in Workshop on Clinical Image-Based Procedures. Springer, 2023, pp. 215–224.
  * [10] Yongshuo Zong, Yongxin Yang, and Timothy Hospedales,  “Medfair: Benchmarking fairness for medical imaging,”  arXiv preprint arXiv:2210.01725, 2022.
  * [11] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234–241.
  * [12] Jeya Maria Jose Valanarasu and Vishal M Patel,  “Unext: Mlp-based rapid medical image segmentation network,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 23–33.
  * [13] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang,  “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,”  IEEE transactions on medical imaging, vol. 39, no. 6, pp. 1856–1867, 2019.
  * [14] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.,  “Attention u-net: Learning where to look for the pancreas,”  arXiv preprint arXiv:1804.03999, 2018.
  * [15] Fenghe Tang, Jianrui Ding, Lingtao Wang, Chunping Ning, and S Kevin Zhou,  “Cmunext: An efficient medical image segmentation network based on large kernel and skip fusion,”  arXiv preprint arXiv:2308.01239, 2023.
  * [16] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou,  “Transunet: Transformers make strong encoders for medical image segmentation,”  arXiv preprint arXiv:2102.04306, 2021.
  * [17] Stanford AIMI shared datasets,  “Thyroid ultrasound cine-clip,” https://stanfordaimi.azurewebsites.net/datasets/a72f2b02-7b53-4c5d-963c-d7253220bfd5.
  * [18] Yu Tian, Min Shi, Yan Luo, Ava Kouhana, Tobias Elze, and Mengyu Wang,  “Fairseg: A large-scale medical image segmentation dataset for fairness learning with fair error-bound scaling,”  arXiv preprint arXiv:2311.02189, 2023.
  * [19] Moritz Hardt, Eric Price, and Nati Srebro,  “Equality of opportunity in supervised learning,”  Advances in neural information processing systems, vol. 29, 2016.


# Digital Twin Tracking Dataset (DTTD): A New RGB+Depth 3D Dataset for Longer-
Range Object Tracking Applications

Weiyu Feng∗, Seth Z. Zhao∗, Chuanyu Pan∗, Adam Chang, Yichen Chen, Zekun Wang,
Allen Y. Yang
University of California, Berkeley
{weiyu_feng, sethzhao506, chuanyu_pan, adamchang2000, chenyc, zkw25, allenyang
<EMAIL_ADDRESS>
###### Abstract

Digital twin is a problem of augmenting real objects with their digital
counterparts. It can underpin a wide range of applications in augmented
reality (AR), autonomy, and UI/UX. A critical component in a good digital twin
system is real-time, accurate 3D object tracking. Most existing works solve 3D
object tracking through the lens of robotic grasping, employ older generations
of depth sensors, and measure performance metrics that may not apply to other
digital twin applications such as in AR. In this work, we create a novel RGB-D
dataset, called Digital-Twin Tracking Dataset (DTTD), to enable further
research of the problem and extend potential solutions towards longer ranges
and mm localization accuracy. To reduce point cloud noise from the input
source, we select the latest Microsoft Azure Kinect as the state-of-the-art
time-of-flight (ToF) camera. In total, 103 scenes of 10 common off-the-shelf
objects with rich textures are recorded, with each frame annotated with a per-
pixel semantic segmentation and ground-truth object poses provided by a
commercial motion capturing system. Through experiments, we demonstrate that
DTTD can help researchers develop future object tracking methods and analyze
new challenges. We provide the dataset, data generation, annotation, and model
evaluation pipeline as open source code at: https://github.com/augcog/DTTDv1.

**footnotetext: Equal Contribution

dataset | data type | ToF sensor | texture | depth | occlusion | average distance | light variation | # of frames | # of scenes | # of objects | # of annotations
---|---|---|---|---|---|---|---|---|---|---|---
FAT[25] | synthetic | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | $\checkmark$ | 60,000 | 3,075 | 21 | 205,359
ShapeNet6D[12] | synthetic | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | $\checkmark$ | 800,000 | - | 12,490 | -
StereoOBJ-1M[19] | real | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | - | $\checkmark$ | 393,612 | 182 | 18 | 1,508,327
TOD[20] | real | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | 0.65 | $\times$ | 64,000 | 10 | 20 | 64,000
LINEMOD[13] | real | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.88 | $\times$ | 18,000 | 15 | 15 | 15,784
YCB-Video[29] | real | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.85 | $\times$ | 133,936 | 92 | 21 | 613,917
LabelFusion[21] | real | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.99 | $\checkmark$ | 352,000 | 138 | 12 | 1,000,000
T-LESS[14] | real | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | 0.77 | $\times$ | 47,762 | - | 30 | 47,762
DTTD (Ours) | real | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 1.32 | $\checkmark$ | 55,691 | 103 | 10 | 136,226

Table 1: Comparison of DTTD dataset with other prior art. DTTD provides RGB-D
data using Microsoft Azure Kinect ToF sensor for longer-range object pose
estimation problems. DTTD also measures rich object texture in different
lighting conditions. To make a fair comparison, we only present the average
distance for real-world RGB-D datasets.

##  1 Introduction

Augmented reality (AR) is a growing area of research in both academia and
industry. Two main concerns in AR applications are localization and tracking.
While camera tracking in static environments is a relatively matured
technology as evidenced by the number of available SLAM solutions [6, 24],
tracking individual objects in unknown or potentially cluttered environments
is much more nascent. Potential solutions are also essential to underpin a
category of augmented reality applications called digital twin, for instance
replacing paper manuals for machine maintenance with virtual 3D digital
instructions and collaborative design of CAD models. 3D object tracking is an
essential part of any good digital twin solutions, aiming to estimate an
object’s rotation and translation relative to the camera with RGB or RGB-D
input and worn by the user.

Recent advances in precise 3D object tracking have been driven by deep neural
networks (DNN), which rely on high-quality datasets to be trained effectively.
Existing 3D object tracking datasets include synthetic datasets [12, 25] and
real-world datasets[13, 21, 29, 20, 14, 19], with many past demonstrations
focusing on robot grasping tasks. Recent DNN models are widely trained on
these datasets in an end-to-end manner to perform image semantic segmentation,
object classification, and object pose estimation tasks[27, 11, 10, 15, 22].
By employing fusion techniques on aligned RGB images and depth maps, these
approaches can be robust to varying lighting conditions and object occlusions.
Many recent works are capable of performing 3D object tracking with sufficient
precision in real-world applications [27], demonstrating up to centimeter
localization accuracy sufficient for the use cases of robot grasping.

However, these solutions when applied to AR applications would reveal a new
set of challenges. Real-world augmented reality applications usually require
accurate 3D object tracking at millimeter accuracy, up to one to two meters
distance, and under various lighting conditions. Unfortunately, most existing
real-world 3D object tracking datasets have been recorded with less than one
meter distance and under a fixed lighting condition (see Table 1).

Further compounding the above issues, these 3D object tracking datasets rely
on older generation depth sensors that are known to have high noise in depth
estimation beyond one meter, such as using low-power stereo cameras or
structured-light cameras. We believe these depth cameras have hindered
possible improvement of future algorithms to achieve accurate 3D pose
estimation, especially for digital twin AR applications. Finally, solving 3D
pose estimation for AR applications has a certain level of expectation of real
time algorithms. However, when depth data in existing datasets are sparse
given a 3D scene, then running 3D pose estimation could be time consuming [16,
11, 10].

In this paper, we introduce a new open-source dataset to address the above
gaps, called Digital Twin Tracking Dataset (DTTD). As summarized in Table 1,
DTTD is designed to be a six degree-of-freedom (6-DoF) pose estimation dataset
that aims to address the pose estimation problem for digital twin
applications. To this end, we select 10 common objects similar to [3, 4, 5],
purchasable online or at local grocery stores, placing them in an indoor room-
scale environment where users using AR could perform interaction tasks with
these objects from roughly 0.7 meter to 1.5 meters. Adopting Microsoft Azure
Kinect time-of-flight (ToF) sensor, we collect aligned RGB and depth frames
with the expectation of large field of view, high image resolution
($1280\times 720$) and increased depth accuracy[1, 17, 26]. The data is
collected under different lighting conditions with different levels of object
occlusions to simulate real-world digital-twin scenarios. We also create a
novel data annotation pipeline that makes use of an
OptiTrack111https://optitrack.com/ motion capture system to provide ground
truth object pose annotations as well as per-pixel semantic segmentation in
every frame.

Our main contributions are summarized as follows:

  1. 1.

DTTD is a real-world, high-quality 3D object tracking dataset with 10 objects
and 103 scenes designed for research in digital twin AR use cases. This
dataset provides image data with a state-of-the-art depth camera, providng
high resolution, large field of view, and accurate dense depth map. The data
are collected from a wide range of distances within 1.5 meters and with
various lighting conditions.

  2. 2.

We open source a pipeline for generating 3D object tracking measurements using
OptiTrack motion capture system and also provide manually annotated and
refined ground-truth scene labels.

  3. 3.

Finally, we evaluate selected state-of-the-art methods on DTTD to demonstrate
its utility and efficacy.

Figure 1: Illustration of data samples. DTTD dataset is collected under
various real-world occlusion and lighting conditions. The first row shows
scene images of four representative conditions (normal, occlusion, dim, dark).
The second and third rows show the corresponding semantic segmentation labels
and 3D bounding boxes, respectively. DTTD depth data have the same resolution
as the scene images.

##  2 Related Work

###  2.1 3D Object Tracking Datasets.

Existing object pose estimation algorithms are trained and evaluated primarily
on a handful synthetic and real-world 3D object tracking datasets[13, 21, 29,
25, 12, 20, 14, 19]. Synthetic datasets use computer graphics rendering
methods to generate photo-realistic images with various diverse shapes and
appearances. For example, FAT [25] renders 3D objects embedded in realistic
background images to generate measurement with pose annotation. Shapenet6D
[12] generates images with augmented object shapes, texture, and background.
Synthetic datasets typically require less human labor for annotation. However,
models trained on synthetic datasets may have difficulties dealing with the
domain-transfer problem when the systems are later applied in real world.

In comparison, real-world datasets collect frames from real scenes and then
employ manual annotation to label ground-truth 3D poses. Datasets such as YCB-
Video[29], LINEMOD[13], StereoOBJ-1M[19], and TOD[20] utilize depth-from-
stereo sensors to collect real 3D data. While previous generations of stereo
sensors primarily target use cases within one-arm distance (i.e., about 0.7
meter), the depth accuracy would degrade very rapidly as the range increases
[8]. Furthermore, stereo sensors also suffer from holes in the depth map when
stereo matching fails, requiring hole-filling algorithms[16] to preprocess the
data and hence sacrifice the solution speed.

TLess[14] is another option that collects data via time-of-flight (ToF)
sensors, which improves the quality of depth data in room scale. Our DTTD
dataset also uses state-of-the-art ToF camera, namely, Microsoft Azure Kinect,
to capture meter-scale RGB-D data, and uses OptiTrack system to collect the
ground-truth camera position.

Regarding the pose annotation pipeline, most existing RGB-D datasets manually
create their annotation by fitting 3D mesh models to 3D point cloud [13, 29,
14, 28]. In order to generate large quantities of labeled frames, annotation
errors up to a centimeter are tolerated. For example, LabelFusion[21] combines
human keypoint labeling with matching algorithms to generate annotation.
However, without per-frame manual refinement, centimeter-scale error may
frequently occur, causing insufficient accuracy for tracking tasks in AR and
digital-twin applications. DTTD relies on a combination of manual and
automated refinement on each frame to improve the annotation quality compared
to the existing works. More specifically, DTTD maintains accurate camera
positions using the Optitrack system to keep objects’ poses consistent among
frames, thus reducing the annotation workload.

###  2.2 6 DoF Object Pose Estimation.

Most data-driven methods for object pose estimation take RGB[18, 23, 29, 30]
or RGB-D images[10, 11, 27, 22, 15] as input. Xiang et al.[29] propose a
convolutional neural network for 3D pose estimation problem. To fuse different
data modalities more efficiently, Wang et al.[27] propose a network
architecture to extract and combine pixel-wise dense feature embedding for
both RGB and depth sources. Due to the architecture simplicity, this method
reaches high efficiency on predicting object poses. More recent works[11, 10,
12] improve the performance with more elaborate network architectures. For
example, He et al.[10] propose an improved bidirectional fusion network for
key-point matching, and achieve high accuracy on YCB-Video[29] and LINEMOD[13]
benchmarks. However, these methods are less efficient due to the complicated
hybrid network structure and processing stages. For symmetric objects, Mo et
al.[22] propose a symmetry-invariant pose distance metric to avoid local
minima issues. Jiang et al.[15] propose an L1-regularization loss, called abc
loss, which improves the pose estimation accuracy for non-symmetric objects.
Our work will focus on RGB-D data and evaluate some of the recent methods on
DTTD to create a baseline benchmark.

##  3 DTTD Dataset

DTTD collects RGB images and depth images from 10 rigid objects together with
their corresponding textured 3D models. In total, there are 103 scenes that
each includes one or more of such objects with different orientations. The
dataset provide ground-truth labels about 3D object poses and per-pixel
semantic segmentation. Camera specifications, pinhole camera projection
matrices, and distortion coefficients are also provided in the dataset. The
detailed dataset protocol is described below.

###  3.1 Scenes

A scene in DTTD consists of a static assortment of objects placed on a flat
surface captured by a single camera, whose movement is tracked by a motion
capture system. For each scene, we perform a half-circle revolution around the
object(s), exposing different faces of the object(s) and varying levels of
occlusion. Each scene contains up to five objects. As shown in Figure 1, we
purposefully collect scenes at multiple time of the day with different
lighting conditions. The metadata for each frame contains labeled ground-truth
object poses and camera intrinsic and distortion. Users of DTTD may reproject
the depth map into 3D point cloud using the camera parameters.

###  3.2 Camera

RGB frames and corresponding depth maps are collected from a Microsoft Azure
Kinect depth camera. During manual data collection, the camera is attached to
a cart and pushed smoothly around the scene, minimizing the shaking from
handheld operation. We then retrieve aligned RGB and depth frames from Azure
Kinect at a resolution of $1280\times 720$ at a framerate of around 30 frames
per second (fps), with some tolerance of latency. The frames are aligned and
synchronized and retrieved by us through the Azure Kinect SDK. The Azure
Kinect SDK also provides radial and tangential distortion coefficients that we
supply in addition to the frames.

Figure 2: Illustration of 3D CAD models of the 10 objects in DTTD. Notice that
{d,f} and {g,i} are two pairs of objects that are geometrically identical but
have different color texture.

###  3.3 Object Models

DTTD includes textured 3D model for the 10 DTTD objects. These models are
scanned using the Polycam222https://poly.cam/ app using an iPhone 12 Pro with
its camera and LIDAR sensors. For model refinement, we use
Blender333https://www.blender.org/ to fix surface holes and texture pixels
that are scanned incorrectly. To make the collected scenes compatible with
other existing datasets, part of the then DTTD objects overlap with the YCB-
Video dataset [29]. We use these 3D models to perform annotation of our data
as well as generate semantic segmentation. Figure 2 displays the collected
models. Figure 3 shows the distribution of the number of occurrence of the
objects appearing in all scenes. The distribution is nearly uniform.

Figure 3: Number of scenes that each object occurs in DTTD.

###  3.4 Pose Annotation

We have created a novel data annotation pipeline to annotate and refine
ground-truth poses for objects in the scene. The pipeline first reprojects
camera frames into corresponding 3D point cloud and aligns virtual models as
the initial ground-truth pose. Then a combination of _iterative closest
points_ (ICP)[2] point-cloud registration methods and other refinement
techniques are used to refine both the camera and object poses. Finally, a
global pose refinement step is performed manually. By reviewing the annotation
results of each frame and each object manually, investigators can ensure high
quality labels in the dataset. Annotating one scene can take as little as five
minutes, depending on the quality of the automatic refinement. In order to
generate the per-pixel semantic segmentation, the pipeline can further project
the virtual models onto the 2D camera plane using the camera intrinsic and
distortion parameters given the virtual models’ known poses.

###  3.5 Data Synthesizer and Scene Augmentation

The open-source DTTD code has the ability to further augment the dataset with
synthetic images for training sophisticated deep learning models. We build our
own data synthesizer using Open3D Visualization platform [31]. Users are able
to import custom CAD models, cameras, and backgrounds into the data
synthesizer. The virtual cameras in simulation captures five kinds of data,
including RGB images, depth images, semantic segmentation, object rotation and
translation. We also add Random Movement Component and Random Rotation
Component to objects to create diverse synthetic scenes. As part of our data
generation pipeline, we also provide the ability to render synthetic labeled
data selected from the original DTTD object models.

##  4 Data Generation Pipeline

###  4.1 Collection Setup

We collect the data in a single room outfitted with an OptiTrack motion
capture system with 10 Prime 17W cameras to cover the room space. Calibration
of the system using OptiTrack’s Motive software reports a mean 3D reprojection
error of 0.8mm. We calibrate the depth camera to the OptiTrack system using an
ARUCO marker in the center of the room. The camera is instrumented with six
OptiTrack markers to define a rigid body. Then the single rigid body can be
tracked by the OptiTrack software. The OptiTrack software returns a set of
translation and rotation parameters for each rigid body in each frame during
recording. Figure 4 illustrates the setup of a standard scene.

Figure 4: DTTD data collection. For setting up a scene in DTTD, the room is
outfitted with 10 Prime 17W OptiTrack cameras to provide the ground-truth
position and orientation in real time, with objects (such as cheez-it_box,
mustard, and mac_cheese) resting on a flat support platform. To track the
camera body, OptiTrack markers are glued onto the camera body. An ARUCO marker
is placed in the scene to calculate the offset between the camera coordinate
frame and the OptiTrack coordinate frame.

###  4.2 Sensor Timestamp Synchronization

The presence of multiple sensors during the data collection procedure requires
reasonable time synchronization, although the synchronization does not need to
be at high frame rates.

More specifically, the Azure Kinect camera captures frames at 30 fps, with
some error due to system latency. The OptiTrack system is set up to output
poses at 60 Hz. The two are then synchronized by matching computed poses from
camera frames computed using the ARUCO marker with OptiTrack poses. We
optimize the matching by minimizing the mean squared error. We use linear
interpolation to estimate sub-interval poses from the OptiTrack. We also
smooth the OptiTrack pose estimation jittering using a Kalman filter with a
constant velocity model. This method generally yields good data alignment.

###  4.3 Extrinsic Calibration

One important step of our data generation pipeline is to ensure the OptiTrack
is properly tracking the coordinate frame of the camera sensor itself, which
we denote as $g_{oc}$ (transform from camera sensor to OptiTrack). The
OptiTrack is directly tracking the coordinate transform of the rigid body
defined by its markers on the camera. This coordinate transform has a
translation and rotation frame with respect to the actual camera sensor. We
refer to this coordinate frame as $g_{ov}$ (transform from virtual camera to
OptiTrack), or the virtual camera pose.

We refer to the extrinsic calibration as calculating the transformation
between the OptiTrack’s virtual camera rigid body coordinate system and the
camera sensor’s own coordinate system, or $g_{cv}$ (transform from virtual
camera to camera sensor). In order to solve this extrinsic transformation, we
first measure the position and rotation of an ARUCO marker in the OptiTrack
world coordinate system, $g_{oa}$ (transform from ARUCO marker to OptiTrack).
We then capture a sequence of frames with the camera observing the ARUCO
marker and the OptiTrack observing the camera. Using the known pose of the
ARUCO marker in the camera sensor $g_{ca}$ (transform from camera sensor to
ARUCO marker), we can solve for the pose of the camera sensor in the OptiTrack
world coordinate system, $g_{oc}$ (transform from camera sensor to OptiTrack):

$g_{oc}=g_{oa}\cdot g_{ca}^{-1}$ (1)

Now that we have a virtual camera pose $g_{ov}$ and a camera sensor pose
$g_{oc}$, both in OptiTrack world coordinates, we can solve for an extrinsic
transform for each frame and save the average extrinsic:

$g_{cv}=g_{oc}^{-1}\cdot g_{ov}$ (2)

We perform averaging over the translation using the arithmetic mean and the
rotation in the quaternion space.

Once this extrinsic transformation has been computed, as long as the OptiTrack
system remains running, we can now directly track the pose of the camera
sensor itself through OptiTrack’s real-time tracking of the virtual camera
rigid body, which significantly expands our ability in DTTD to track long-
duration, large-transformation 3D motions while keeping precision of the pose
labeling automatically.

###  4.4 Annotation Pipeline

Initial Pose Annotation. Our initial pose annotation is performed in a 3D
environment using Open3D visualization framework [31]. We reproject RGB and
depth maps into point cloud using camera intrinsic and distortion
coefficients. We then place virtual models into the scene using ICP
initialized by a rough user approximation of the center of each object in the
scene. Annotators can then individually adjust position and rotation of
virtual models as well as perform incremental algorithmic alignment until
virtual models are fully aligned with the point cloud. Once the annotated
poses for the objects are satisfactory, they are saved and used to transform
the objects into every subsequent camera frame in the sequence.

Scene Pose Refinement. After transforming each virtual object into every
frame, we perform a frame-wide global pose refinement using ICP to refine
camera poses. Finally, we have another manual refinement phase, where
annotators can review each frame in 2D to ensure good alignment. The
environment allows the annotator to adjust the camera translation and rotation
in a very similar manner as in the object adjustments during the annotation
phase.

##  5 Experiments

###  5.1 Benchmark and Evaluation

Train/Test Split. DTTD provides a recommended train/test split as follows. The
training set contains 8726 keyframes extracted from 92 video sequences and the
testing set contains 1111 keyframes extracted from 11 video sequences. Scenes
with occluded objects and different lighting-conditions are split randomly
across the training/testing sets. In addition, 20000 synthetic images are
provided for training on our dataset by randomly placing objects in a scene as
described in section 3.5. Users of DTTD can opt to render additional synthetic
data using the data synthesizer provided by DTTD.

Evaluation Metrics. Following [29], we can evaluate baseline methods with the
average distance metrics ADD and ADD-S. Suppose $R$ and $T$ are ground truth
rotation and translation and $\tilde{R}$ and $\tilde{T}$ are predicted
rotation and translation. The ADD metric computes the mean of the pairwise
distances between the 3D model points using ground truth pose $(R,T)$ and
predicted pose $(\tilde{R},\tilde{T})$:

$\mathrm{ADD}=\frac{1}{m}\sum_{x\in M}\|(Rx+T)-(\tilde{R}x+\tilde{T})\|,$

where $M$ denotes the point set sampled from the object’s 3D model and $x$
denotes the point sampled from $M$.

The ADD-S metric is designed for symmetric objects when the matching between
points could be ambiguous. The formula is expressed through the closest point
distance between $x_{1},x_{2}\in M$:

$\mathrm{ADD{-}S}=\frac{1}{m}\sum_{x_{1}\in M}\min_{x_{2}\in
M}\|(Rx_{1}+T)-(\tilde{R}x_{2}+\tilde{T})\|$

Following [29, 27, 19], we consider a predicted 6D pose to be correct if the
average distance is smaller than a pre-defined threshold. However, simply
using a fixed threshold in computing ADD and ADD-S can’t reveal how well the
estimation method performs on poses that fall outside of the threshold. To
address this issue, we calculate AUC, the area under the accuracy-threshold
curve, by varying the distance threshold in evaluation. We calculate AUC of
ADD-S and ADD(S) as the performance metrics for comparisons of baseline
methods, where ADD(S) calculates the ADD distance for non-symmetric objects
and ADD-S distance for symmetric objects.

###  5.2 Baseline Methods

We evaluate DenseFusion [27] and DenseFusion with abc loss [15]. We also
provide evaluations on FFB6D [10] with and without hole filling algorithm [16]
applied. We train the above methods on the training sets with synthetic data
and report their performance on the test set.

DenseFusion[27] is a holistic method that directly regresses 3D pose of
objects in a given RGB-D image. The model extracts features from RGB images
and point clouds using separate CNN and point cloud encoders and then performs
per-pixel dense fusion with confidence score for pose estimation.

DenseFusion with abc loss[15]. Jiang et al. [15]’s work demonstrate the
benefits of abc loss on improving model’s performance on ADD(S) AUC metric.
The abc loss $\mathcal{L}_{\text{abc}}$ is defined as the L1 distance between
the points sampled on the objects model in the ground truth pose and
corresponding points on the same model transformed by the predicted pose:

$\mathcal{L}_{\text{abc}}=|a-a^{*}|+|b-b^{*}|+|c-c^{*}|$

where $(a,b,c)$ are coordinates of visible points after performing predicted
transformation, and $(a^{*},b^{*},c^{*})$ are coordinates of visible points
after performing ground-truth transformation. The overall learning objective
is then described as:

$\mathcal{L}=\mathcal{L}_{\text{rt}}+\alpha\mathcal{L}_{\text{abc}}$

where $\mathcal{L}_{\text{rt}}$ is the RT regression loss in [27] and $\alpha$
is the weight for abc loss.

FFB6D[10] is a keypoint-based method that recovers object pose parameters by
regressing on the most confident 3D keypoint features given an RGB-D image.
The model extracts the pointwise RGB-D features for 3D keypoints localization
of each object using a full flow bidirectional fusion network that facilitates
the learning of an object’s geometric and appearance representation. A Least-
Squares Fitting algorithm is applied on selected 3D keypoints in the object
coordinate system and camera coordinated system to obtain optimized pose
parameters. Note that the hole filling algorithm [16] becomes a design choice
when dealing with sparse depth data.

| DenseFusion[27] (per-pixel) | DenseFusion[27] (per-pixel) + abc loss[15] | FFB6D[10] | FFB6D[10] without hole filling[16]
---|---|---|---|---
Object | ADD-S AUC | ADD(S) AUC | ADD-S AUC | ADD(S) AUC | ADD-S AUC | ADD(S) AUC | ADD-S AUC | ADD(S) AUC
mac_cheese | 84.8607 | 59.4370 | 84.4954 | 62.3326 | 48.2619 | 18.9750 | 32.4646 | 14.2986
spam_can | 87.9855 | 38.8337 | 88.3336 | 51.7568 | 30.6463 | 8.7439 | 46.1623 | 15.3382
mustard | 84.3483 | 57.6207 | 84.4658 | 59.2276 | 46.6096 | 28.5681 | 24.4950 | 16.8288
pink_expo_marker | 89.9759 | 78.9417 | 92.3427 | 80.7950 | 47.1057 | 39.8565 | 35.8469 | 30.5378
cheerios_box | 71.3897 | 29.1047 | 69.1718 | 32.8976 | 35.4673 | 7.0891 | 41.6812 | 7.7160
black_expo_marker | 79.2332 | 62.8041 | 80.9540 | 62.6692 | 6.6660 | 5.6904 | 0.7048 | 0.6436
tomato_can | 84.9898 | 38.6220 | 87.1845 | 45.5703 | 24.3613 | 13.8302 | 7.7058 | 2.8534
cheez-it_box | 86.2357 | 60.0661 | 85.4747 | 49.1522 | 29.6672 | 7.9421 | 31.9884 | 13.9151
clam_can | 93.2859 | 71.6521 | 95.5851 | 76.8125 | 24.1573 | 9.6682 | 24.0257 | 12.8887
pop-tarts_box | 79.8134 | 17.4611 | 82.3599 | 24.1388 | 45.9755 | 2.9503 | 21.9485 | 0.5714
Average | 84.2118 | 51.4543 | 85.0368 | 54.5353 | 37.4527 | 14.3314 | 26.7023 | 11.5592

Table 2: Evaluation results of ADD-S AUC, ADD(S) AUC on the DTTD dataset.

###  5.3 Baseline Results

Implementation Details. We trained the DenseFusion and DenseFusion with abc
loss (we set $\alpha=1$ in overall learning objective described in section
5.2) with batch size of 1 for 64 epochs. Notice that we train and evaluate
both methods under given groundtruth segmentation of objects without post
refinement process. We also apply ImageNet [7] pretrained ResNet-18 [9] as the
CNN encoder for both methods. For FFB6D, we follow the proposed SIFT-FPS
algorithm[10] to obtain 8 target 3D keypoints for each object in DTTD. FFB6D
also applies a hole filling algorithm [16] as an expensive postprocessing step
on depth data. In our experiments, we train FFB6D with a batch size of 1 for
10 epochs for best checkpoint. We also choose to train FFB6D without applying
the hole filling algorithm with a batch size of 1 for 7 epochs for best
checkpoint. The performance of FFB6D with and without applying hole filling
algorithm is then compared. Since object segmentation is trained together with
FFB6D, we do not apply groundtruth mask like we did on training DenseFusion.
All methods are adjusted to input image resolution of $1280\times 720$. All
experiments are conducted with one GTX 1080-Ti GPU.

Evaluation on DTTD Dataset. Quantitative results on the DTTD dataset are shown
in Table 2. DenseFusion achieves average score of 84.21 ADD-S AUC and 51.45
ADD(S) AUC on all objects. DenseFusion with abc loss has better perforance
over DenseFusion on most of the objects over ADD(S) metric by an average of
3.1 ADD(S) score improvement. Some qualitative evaluation results are shown in
Figure 5. FFB6D achieves average score of 37.4527 ADD-S AUC and 14.3314 ADD(S)
AUC on all objects. FFB6D without hole filling has average scores of 26.7023
ADD-S AUC and 11.5592 ADD(S) AUC. Given that we do not apply groundtruth mask
to train both methods in FFB6D, we can observe some failure cases in FFB6D’s
segmentation network as shown in Figure 6. We suspect that FFB6D may have
performed incorrect key-point matching due to the complex background in
scenes. Also, we observe that dark lighting conditions can make the FFB6D
segmentation network harder to learn the detection of objects in a longer
range.

Figure 5: Qualitative evaluation result of DenseFusion. The left and right
columns show evaluation results of DenseFusion without and with abc loss,
respectively. Figure 6: Qualitative evaluation result of FFB6D. The left and
right columns show evaluation results of FFB6D with and without hole filling
algorithm applied, respectively. Notice that both methods fail the object
segmentation task of objects in dark lighting condition.

Effect of abc loss on DenseFusion. As shown in Table 2, DenseFusion with abc
loss trained on DTTD dataset outperforms DenseFusion without abc loss by 3.0%
of ADD(S) improvements. From an optimization standpoint, the difference in abc
loss and RT regression loss is the difference in L1-norm and L2-norm.
Numerically, L2-norm squares value of difference in distance, while L1-norm
only takes the absolute value of such. L2-norm is more outlier-prone since it
increases the cost of outliers exponentially whereas the L1 norm considers
them linearly. Thus, adding L1 regularization loss during training improves
model’s robustness towards outlier, which is especially important for the ADD
performance in 6-DoF pose estimation task on non-symmetric objects. Therefore,
adding an L1 regularization loss helps jump out of the local minima,
especially in non-symmetric objects cases.

Effect of hole filling algorithm on FFB6D. We notice that the performance of
FFB6D with or without hole filling is unstable in DTTD dataset. Since the
purpose of FFB6D employing depth hole-filling techniques is to deal with
stereo depth holes issues with some of data in YCB-Video, the hole filling
algorithm would enhances the point cloud representation on the data with
featureless areas and silhouettes of objects. However, the hole filling
algorithm itself would not affect the model’s performance if the depth data
does not contain above issues. Besides, at a longer distance, the hole filling
algorithm might induce a larger interpolation error in application. Thus we
observe a different performance in different objects and scenes.

Model Performance Difference with YCB-Video. With ground-truth segmentation
mask provided, the performance of DenseFusion in DTTD does not have a
comparable ADD-S AUC and ADD(S) AUC performance over the performance of
DenseFusion in YCB-Video, as shown in Table 3. We suppose that DTTD poses the
following challenges on 6-DoF pose estimation task 444Object correspondence
detail: cheez-it_box with 003_cracker_box, tomato_can with
005_tomato_soup_can, mustard with 006_mustard_bottle, spam_can with
010_potted_meat_can, black_expo_marker with 040_large_marker. Firstly,
compared to YCB-Video, objects are captured in a farther distance on average
with larger field of view. For RGB-D data input in DTTD, it is challenging for
the model to learn the object’s low-level features, such as geometrical and
appearance information. Secondly, DTTD provides more diverse light variations
than YCB-Video. The model also needs to bridge the gap of tracking 3D objects
under varying lighting conditions, e.g., estimate black_expo_marker’s pose
under dark lighting condition.

| DTTD | YCB-Video [29]
---|---|---
Object | ADD-S AUC | ADD(S) AUC | ADD-S AUC | ADD(S) AUC
spam_can | 87.9855 | 38.8337 | 90.3031 | 79.9354
mustard | 84.3483 | 57.6207 | 97.0297 | 94.9152
black_expo_marker | 79.2332 | 62.8041 | 97.2883 | 93.9526
tomato_can | 84.9898 | 38.6220 | 96.6725 | 87.7622
cheez-it_box | 86.2357 | 60.0661 | 93.1328 | 88.3662

Table 3: DenseFusion (per-pixel) performance on DTTD and YCB-Video dataset. We
select overlapping objects with similar appearances in both datasets.

##  6 Conclusions

In this paper, we introduce a 3D object tracking RGB-D dataset, called Digital
Twin Tracking Dataset (DTTD), to enable further research on extending
potential solutions to longer-range object tracking problems, which are common
in digital twin problem conditions. To reduce point cloud noise in longer-
range, we select Microsoft Azure Kinect, a state-of-the-art ToF camera. We
also provide a data collection pipeline that aims to minimize the ground-truth
labeling error by combining OptiTrack system, ARUCO marker, and manual
annotation together. In total, DTTD contains 103 scenes of 10 common off-the-
shelf textured objects, with each frame annotated with a per-pixel semantic
segmentation and ground-truth object poses. Our experiments have evaluated
different methods on DTTD, and demonstrated that DTTD can help evaluating
object tracking methods under common digital-twin conditions and analyzing new
challenges in longer-range applications. We provide our dataset, data
generation and annotation pipeline, and evaluation methods as open-source
code.

In the future, we plan to continue adding more objects, scenes, and possibly
adding newest RGB-D camera sensors in the DTTD dataset.

## Acknowledgement

We would like to thank Tianjian Xu, Han Cui, Yunhao Liu for assisting DTTD
data collection process.

## References

  * [1] Justin Amadeus Albert, Victor Owolabi, Arnd Gebel, Clemens Markus Brahms, Urs Granacher, and Bert Arnrich.  Evaluation of the Pose Tracking Performance of the Azure Kinect and Kinect v2 for Gait Analysis in Comparison with a Gold Standard: A Pilot Study.  Sensors, 20(18):5104, Sept. 2020.
  * [2] P.J. Besl and Neil D. McKay.  A method for registration of 3-d shapes.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239–256, 1992.
  * [3] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  Yale-cmu-berkeley dataset for robotic manipulation research.  The International Journal of Robotics Research, 36(3):261–268, 2017\.
  * [4] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar.  The ycb object and model set: Towards common benchmarks for manipulation research.  In 2015 International Conference on Advanced Robotics (ICAR), pages 510–517, 2015.
  * [5] Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar.  Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set.  IEEE Robotics & Automation Magazine, 22(3):36–52, 2015.
  * [6] Carlos Campos, Richard Elvira, Juan J. G’omez, Jos’e M. M. Montiel, and Juan D. Tard’os.  ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM.  IEEE Transactions on Robotics, 37(6):1874–1890, 2021.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [8] Hussein Haggag, Mohammed Hossny, D. Filippidis, Douglas C. Creighton, Saeid Nahavandi, and Vinod Puri.  Measuring depth accuracy in rgbd cameras.  2013, 7th International Conference on Signal Processing and Communication Systems (ICSPCS), pages 1–7, 2013.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [10] Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun.  Ffb6d: A full flow bidirectional fusion network for 6d pose estimation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [11] Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun.  Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [12] Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, and Qifeng Chen.  Fs6d: Few-shot 6d pose estimation of novel objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6814–6824, June 2022.
  * [13] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab.  Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes.  In Kyoung Mu Lee, Yasuyuki Matsushita, James M. Rehg, and Zhanyi Hu, editors, Computer Vision – ACCV 2012, pages 548–562, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
  * [14] Tomáš Hodaň, Pavel Haluza, Štěpán Obdržálek, Jiří Matas, Manolis Lourakis, and Xenophon Zabulis.  T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.  IEEE Winter Conference on Applications of Computer Vision (WACV), 2017.
  * [15] Xiaoke Jiang, Donghai Li, Hao Chen, Ye Zheng, Rui Zhao, and Liwei Wu.  Uni6d: A unified cnn framework without projection breakdown for 6d pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11174–11184, June 2022.
  * [16] Jason Ku, Ali Harakeh, and Steven L Waslander.  In defense of classical image processing: Fast depth completion on the cpu.  In 2018 15th Conference on Computer and Robot Vision (CRV), pages 16–22. IEEE, 2018.
  * [17] Gregorij Kurillo, Evan Hemingway, Mu-Lin Cheng, and Louis Cheng.  Evaluating the accuracy of the azure kinect and kinect v2.  Sensors, 22(7), 2022.
  * [18] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [19] Xingyu Liu, Shun Iwase, and Kris M. Kitani.  Stereobj-1m: Large-scale stereo image dataset for 6d object pose estimation.  In ICCV, 2021.
  * [20] Xingyu Liu, Rico Jonschkowski, Anelia Angelova, and Kurt Konolige.  Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020), 2020.
  * [21] Pat Marion, Peter R Florence, Lucas Manuelli, and Russ Tedrake.  Label fusion: A pipeline for generating ground truth labels for real rgbd data of cluttered scenes.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3325–3242. IEEE, 2018.
  * [22] Ningkai Mo, Wanshui Gan, Naoto Yokoya, and Shifeng Chen.  Es6d: A computation efficient and symmetry-aware 6d pose regression framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6718–6727, 2022.
  * [23] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In CVPR, 2019.
  * [24] Thomas Schöps, Torsten Sattler, and Marc Pollefeys.  Bad slam: Bundle adjusted direct rgb-d slam.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 134–144, 2019.
  * [25] Jonathan Tremblay, Thang To, and Stan Birchfield.  Falling things: A synthetic dataset for 3d object detection and pose estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [26] Michal Tölgyessy, Martin Dekan, Ľuboš Chovanec, and Peter Hubinský.  Evaluation of the azure kinect and its comparison to kinect v1 and kinect v2.  Sensors, 21(2), 2021.
  * [27] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese.  Densefusion: 6d object pose estimation by iterative dense fusion.  2019\.
  * [28] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642–2651, 2019.
  * [29] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  2018\.
  * [30] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  Dpod: 6d pose object detector and refiner.  pages 1941–1950, 10 2019.
  * [31] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3D: A modern library for 3D data processing.  arXiv:1801.09847, 2018.