# Cluster consistency: Simple yet effect robust learning algorithm on large-
scale photoplethysmography for atrial fibrillation detection in the presence
of real-world label noise

Cheng Ding Department of Biomedical Engineering, Georgia Institute of
Technology & Emory University, USA (chengding@gatech.edu). ZhiCheng Guo
Department of Electrical and Computer Engineering, Duke University, USA . Amit
Shah  Division of Cardiology, Department of Medicine, Emory University School
of Medicine, Atlanta, Georgia, USA. Gari Clifford  Department of Biomedical
Informatics, Emory University School of Medicine, USA. Cynthia Rudin
Department of Computer Science, Duke University, USA. Xiao Hu Professor and
Asa Griggs Candler Chair of Nursing Data Science, Associate Director — Center
for Data Science, Nell Hodgson Woodruff School of Nursing, Associated Faculty
of Biomedical Informatics, School of Medicine, Associated Faculty of Computer
Science, College of Arts and Sciences, Emory University, USA.

( )

###### Abstract

Obtaining large-scale well-annotated is always a daunting challenge,
especially in the medical research domain because of the shortage of domain
expert. Instead of human annotation, in this work, we use the alarm
information generated from bed-side monitor to get the pseudo label for the
co-current photoplethysmography (PPG) signal. Based on this strategy, we end
up with over 8 million 30-second PPG segment. To solve the label noise caused
by false alarms, we propose the cluster consistency, which use an unsupervised
auto-encoder (hence not subject to label noise) approach to cluster training
samples into a finite number of clusters. Then the learned cluster membership
is used in the subsequent supervised learning phase to force the distance in
the latent space of samples in the same cluster to be small while that of
samples in different clusters to be big. In the experiment, we compare with
the state-of-the-art algorithms and test on external datasets. The results
show the superiority of our method in both classification performance and
efficiency.

##  1 Introduction

Atrial fibrillation (AF) is one of the most common arrhythmias observed in
patients in the intensive care units (ICUs) of hospitals; between 4% to 30% of
ICU patients will exhibit AF [1]. AF can be diagnosed as the loss of the
P-wave and a change in the time between the RR intervals [2] in the
electrocardiography (ECG). In the ICU, many chronic diseases come from AF. It
has been linked to ischemic stroke, congestive heart failure, and other high-
risk cardiopulmonary conditions, as well as a higher risk of death [1].
Besides ECG, with the advancement of convenient and lower-cost,
photoplethysmography (PPG) has been ubiquitous in many wearable devices to
detect beat-to-beat blood volume changes and thus become an important
indicator for monitoring cardiovascular conditions, including AF. It has been
shown that deep learning (DL) algorithms achieve better results than
traditional machine learning algorithms [3, 4, 5, 6, 7] in AF detection with
PPG signal.

However, the difficulty of collecting an adequate amount of ground truth for
DL-based research spans a variety of medical domains & modalities. The fields
of computational genomics, digital pathology, & diagnostic imaging all
struggle with the same problems caused by a lack of labelled data. Even though
biomedical signal data is relatively standard compared to other sources, it is
not an exception. Datasets with clean, detailed, and adequately discrete
ground truth are challenging to come by. The most significant issue is that
there is a severe shortage of methods that are both cost-efficient and can
manually annotate the data. In non-medical machine learning domains, like
general computer vision, data annotation can be done by the crowd [8]. Even
with such effort, a considerable amount of label errors is still present in
the ImageNet [9]. Medical data often need to be interpreted by healthcare
professionals and healthcare staff, and it would have been too expensive to
hire these people for large-scale data annotation. The records of inpatients
in ICU are a rich and varied source of physiological signals for possible
analysis. These data consist of numerous vital signs, typically measured at
regular intervals, and continuous waveform signals gathered at a high
frequency. Some examples of these types of measurements include
electrocardiography (ECG) for heart rhythm analysis, and photoplethysmography
(PPG) from pulse oximetry. Tracking these waveforms in an intensive care unit
alone may produce gigabytes of data per day. Due to a large amount of data in
these streams, human inspection is usually limited, and nurses more rely on
the alarm information generated by the built-in ECG algorithms. If those
algorithms detect irregular pattern on the ECG, it will trigger an alarm.
These alarms therefore are valuable resources since they can used as the
labels for simultaneous PPG signal. With this mechanism, millions of signals
with related label can be generated. However, in significant numbers of cases,
these alarms are triggered due to the motion artifacts from patient movement.
The inaccurate alarm information would cause label errors when it is used to
auto-annotate the PPG signals. Although our previous work [10] concluded that
deep learning is more robust to label noise than conventional models, the need
for robust learning from noisy labels is still important.

Although the area of robust learning is getting more and more attention,
especially in the computer vision area [11], it remains three concerns to
directly adopt these state-of-the-art techniques to our situation. First, most
of the work are not dealing with real-world noise. Authors conducted the
experiments by generating synthetic wrong labels with either symmetric or
asymmetric flipping, which is not able fully represent the real-world
situations. Second, very few studies paid attention to the data quality in the
label noise study. We only able to find one work [12], in which authors tried
to estimate the label noise transition matrix for lower quality features based
on information theoretic approach. Datasets used in most of work are standard,
which are carefully preprocessed and noise-free. However, low-quality signals
are ubiquitous when collecting PPG due to patient movement or loose contact of
the device. Third, sophisticate algorithms may lead to higher performance but
hard to scale, such as INCV [13] and DivideMix [14], which requires warm-up
epochs. In our study, we have collected over 8 million 30-second PPG segments.
Efficiency of the robust learning algorithm is also top priority.

In this paper, we built by far the largest PPG datasets auto-labelled by
concurrent alarms from bedside monitor. To tackle the label noise caused by
false alarms, we propose a simple but effective method, cluster consistency
loss, to alleviate the impact of label errors. As inspired by [15], in which
they proposed the cluster-aligning loss and cluster-separating loss to solve
the domain shift problem, cluster consistency loss combines both unsupervised
cluster information and noisy supervised information. In cluster consistency
loss, as shown in Fig. 1, we first perform clustering on the latent space
trained on an auto-encoder to get the assigned cluster label. Second, we use
the cluster label to regulate the feature representation learned through the
convolutional layers. More specifically, the distance will be minimized
between feature representation within the same cluster, and distance of
features representations will be maximized for different clusters. We name
these two terms as intra-cluster distance and inter-cluster distance. Unlike
many studies which only perform their algorithms on public datasets with
artificial-created label noise, and some studies even need the level of noise
in order to perform, which may not fully reflect the real-world scenarios. In
this study, we train the Resnet-34 with the cluster consistency loss in
additional to the cross-entropy (CE) loss on the real-world PPG dataset with
unknown level of label noise. We also compare the performance of cluster
consistency loss with several the state-of-the-art methods, including co-
teaching [16], Early-Learning Regularization (ELR) [17], DivideMix [14] and
Sparse Over-Parameterization (SOP) [18], which are top runners in the 1st
Learning and Mining with Noisy Labels Challenge at
http://competition.noisylabels.com/. The results show that our simple method
can outperform the rest in most of the cases and on par with the DivideMix
(second in the competition) and SOP (first in the competition) in other cases.

The contribution of this study can be summarized:

  * •

To our knowledge, we are the first work conducting PPG-AF detection on the
data with over 8 million segments from over 3000 patients, which is the
largest scale study so far.

  * •

A framework to handle real-world data that has poor signal quality (i.e.,
motion artifacts) and poor label quality (i.e., many false alarms).

  * •

The framework discussed above includes a new ML algorithm for handling this
kind of data. Results show that the proposed cluster consistency loss is able
handle the label noise in such large-scale data in an accurate and efficient
way. We compare the performance with not just conventional CE but also top
runners from an open challenge. Data quality is also considered in this study,
which is missed by many robust learning algorithms. We show that the proposed
method still outperforms the baselines on lower quality signals.

  * •

insight into the domain

Figure 1: The overall flow of the cluster consistency loss. A pre-trained
cluster label is assigned to each sample. Based on the cluster label, intra-
cluster distance and inter-cluster distance of one-batch is calculated and
used as additional loss terms together with conventional cross-entropy.

##  2 Related work

Due to the huge amount of publications in the robust learning area, we briefly
provide reviews for a few selected studies which are highly related to our
work, including methods used as baselines.

##### Architecture design for robustness

Several studies have used dedicated model architecture design to achieve
robust learning. [19, 16, 14]. These methods usually use two branches of
networks. Probabilistic noise modeling [19] attempts to model the label
transition probability. Co-teaching [16] trains two branches of networks at
the same time. Each branch selects low-loss samples for the other branch for
further training. DivideMix [14] also leverages two branches of networks. Each
branch divides the data into clean and noisy portions for the other branch for
semi-supervised training. These methods tend to be computationally expensive.
We compare with DivideMix and co-teaching in our experiments.

##### Regularization for robustness

Regularization has been a crucial technique in combating noisy labels and
overfitting for deep learning models. Recent work Mixup [20] proposes using
linear-interpolated samples as additional training samples. Deep bilevel
learning [21] leverages a clean validation set, and set weights for each mini-
batch to optimize the clean validation set to reduce overfitting on the
training set. Both [17, 22] regularize the model against the memorization
effect. Robust early-learning [17] divides model parameters into critical and
non-critical ones and updates these two sets of parameters differently. ELR
[22] leverages the early learning effect to alleviate the effect of noisy
labels later in the training process. ODNL [23] artificially creates hard
samples at each epoch by assigning randomly selected labels to an open-set
auxiliary dataset, thus preventing the deep model from overfitting on noise
samples and achieving regularization. The regularization methods have the
potential to produce good robust learning results, but they do require careful
tuning and a balance between model complexity (depth) and regularization.

##### Sample selection for robustness

Sample selection is an intuitive way to solve the label noise problem. It aims
to distinguish between correctly labeled samples and incorrectly labeled
samples. Due to the memorization effect, before the model overfits, high-loss
samples are more likely to be noisy samples and low loss are likely to be
clean samples. INCV [13] from Chen et al. attempts to filter out clean samples
from the dataset-based loss values of samples. By iterative removal of high-
loss samples (noisy samples), it will produce a set of clean samples for
training. Several works [24, 25, 26, 16] also rely on loss to select clean
samples. Even though sample selection can achieve good results in ideal
situations, in practice mistakes in the selection process will accumulate and
enlarge the error in the end. These methods are often unable to distinguish
between hard samples and noisy samples; the fact that hard (bad quality)
samples are commonplace in physiological signals renders them unsuitable for
our task.

##### Loss functions for robustness

Popular loss functions such as cross-entropy loss perform well in noise-free
settings, however given noisy labels, these popular loss functions may not be
the optimal choice. Modifying or improving loss functions has shown to be
effective in combating noisy labels. GCE loss [23] combines the robust nature
of MAE and the advantage of traditional cross-entropy loss. Recently, SOP from
Liu et al. [18] modified the classic cross-entropy loss with an additional
sparse term to model label noise of each sample. Symmetric Cross Entropy (SCE)
[27] introduces a reverse CE term to the loss function. It improves learning
on hard samples and the robustness towards label noise. Curriculum loss (CL)
[28] provides a tight upper bound for robust 0-1 loss, as well as an extension
for multi-class label noises. Robust loss function designs are more adaptable
to existing network designs. They usually do not increase computational costs
and can achieve good results in the presence of label noises. Our work also
follows this path by modifying the existing CE loss to counteract overfitting
on noisy samples in training.

##### Robust learning on physiological signals

Besides the above-mentioned studies aimed at general data types or images,
several previous works also focused on the label noise problem in
physiological signals. Vázquez et al. [29] used self-learning for label
correction for the 12-lead ECG from PhysioNet challenge 2021 to alleviate the
label noise caused by inter-annotator variety [30] also considered cluster
information to handle label noise; the authors used K-nearest neighbor
iteratively to determine if one label is correct. [31] considers the ensemble
mechanism to combine the prediction from a set of under-trained models for
sleep apnea using PPG. They share the same assumption that the early learning
stage is less likely to be influenced by label noise. More recently, Negative-
ResNet [32] leverages a semi-supervised negative learning scheme on noisy-
labeled ECG signals.

###  2.1 Study data

####  2.1.1 Training data - auto-labelled from linked EHR and waveform dataset

The UCSF EHR datasets are collected from 28539 patients and contains 1,687,030
encounters from 2013-03 to 2018-12. The characteristics of the EHR database
are summarized in Table 1. From the table, we can observe that the prevalence
of men diagnosed with AF (60%) is higher than its prevalence in the general
admissions (53%), while it drops from 46% to 40% in the women group. This
phenomenon is in accordance with the conclusion from [33]. It is also reported
that the prevalence of AF from patients aged greater than 65 is higher than
younger group [34], we can see same situation in the UCSF HER that the
prevalence of AF increases greatly from 42% to 73% compared between general
and AF cohorts. We do not see much difference between two cohorts in any of
the race group, except the group ‘Other’ drops from 17% to 11% and the Native
Hawiian and pacific islander doubled from 1.45% to 3.1%. However, since the
shortage of further information of the ‘Other’ group, we are not able to
locate the exact race group causing this decreasing.

Table 1: Characteristics of Participants enrolled in UCSF Characteristics | Total Cohort (N = 28539) | AF group (N = 5779)
---|---|---
Sex |  |
Female | 13203 (46.2%) | 2304 (39.9%)
Male | 15330 (53.7%) | 3473 (60.1%)
Others | 6 (0%) | 2 (0%)
Age |  |
$\geq$65 yr | 12157 (42.6%) | 4251 (73.6%)
55-64 yr | 5370 (18.8%) | 928 (16.1%)
40-54 yr | 4372 (25.3%) | 419 (7.3%)
22-39 yr | 2715 (9.5%) | 147 (2.6%)
$>$22 yr | 3925 (13.8%) | 34 (0.6%)
Race or ethnic group |  |
White or Caucasian | 15890 (55.7%) | 3311 (56.5%)
Black or African American | 2159 (7.4%) | 435 (7.4%)
Asian | 4364 (15%) | 1097 (18.7%)
Unknown/Declined | 1149 (4%) | 191 (3.3%)
Others | 4913 (16.9%) | 671 (11.4%)
Native Hawaiian or Other Pacific Islander | 426 (1.46%) | 125 (3.1%)
American Indian or Alaska Native | 212 (0.7%) | 35 (0.6%)

####  2.1.2 Waveform datasets

Continuous waveform recordings are also collected from bedside monitors with
linked patient information with UCSF EHR. In this study, we focus on two types
of arrhythmia alarms: AF and PVCs, we also include the NSR into our dataset.
Detailed definition of each alarm is summarized in Table 2 and top 20 frequent
alarm types are reported in Figure 2.

Table 2: Schema for arrhythmia alarms. Alarms | Short label |  Definition
---|---|---
Atrial Fibrillation | AF |  Irregular timing of QRS complexes and absence of preceding P waves
Normal sinus rhythm | NSR |  Healthy and regular heartbeat
Premature Ventricular Contractions |  PVCs |  VT$>$2: 3-5 consecutive ventricular beats at rate$\geq$100
PVC: Isolated PVCs
R on T: PVC falls on the ST or T wave portion of previous beat
Couplet: Two consecutive PVCs with rate $\geq$ 100
Bigeminy: PVC alternates with a non-ventricular beat for $\geq$ 3 cycles
Trigeminy: PVC alternates with 2 non-ventricular beats for $\geq$ 3 cycles
PVC$\geq$X: PVC count is equal to or greater than user-defined limit
Figure 2: Statistics for top-20 most frequent alarm type in the UCSF waveform
database.

###  2.2 Labeling process

Figure 3: The illustration of algorithmic labeling process for three classes.
Each box indicates a specific alarm that was triggered over a specific period
of time for our example patient. Each time an alarm is triggered, we store 15
seconds of data both before and after the trigger. Note the presence of normal
PPG signals (labeled NSR) as well as the presence of many different alarms
(VFIB, VT, PVC), and motion artifacts (labeled “artifacts”). On the right, we
show that when an AFib alarm is triggered within 30 seconds of a PVC alarm, we
exclude the AFib alarm, because PVC and AFib are difficult to tell apart.

After we retrieve the alarm information, we use the alarm to auto-label the
concurrent PPG signals, detailed illustrated in Fig. 3. For each AF alarm, we
will take the alarm start time as the center of the signal and mark the 15
seconds before and 15 seconds after the center as the final 30-second signal.
To extract PVC signals, we use the same way as AF alarms. However, we exclude
PVC alarms if there are any AF alarms happen with 30 seconds. Since there is
no alarm on normal sinus rhythm, we consider one ‘NSR alarm’ exists if the
time gap between two alarms is longer than 30 seconds.

Figure 4: The statistics of training data

###  2.3 Testing data

The testing process is performed on three external datasets, as summarized in
Table 3. We also include the percentage of AF and Non-AF, signal type and the
setting of collection within each dataset. In UCLA and Simband, the percentage
of good quality is calculate based on labes predicted from our previous
developed binary PPG quality assessment model [35]. In Stanford dataset, the
percentage of good quality signals are calculated by the provided quality
label, which is also machine-generated.

UCLA dataset: The UCLA dataset was used as training data in [36]. It contains
continuous PPG signal collected from 126 patients from UCLA Medical center
using pulse oximeter. These patients were admitted between April 2010 to March
2013, with ages ranged from 18 to 95 years old. For each continuous recording,
the start and end time of each AF episode was annotated by a board certified
cardiac electrophysiologist.

Simband dataset: We also test our method on the dataset previously collected
from [7]. Totally number of 98 patients were recruited to the Emory University
Hospital (EUH), Emory University Hospital Midtown (EUHM), and Grady Memorial
Hospital (GMH). There patients were admitted between October 2015 to March
2016 aged from 18-89 years old. Approximately 5 minutes of PPG collected from
each patient using Samsung Simband in the ambulatory setting.

Stanford dataset: We also test our method on a publicly available dataset that
was collected in [37]. They recruited 107 patients diagnosed with AF (mean age
68), and 15 more prospective patients (mean age 67) with paroxysmal AF. An
additional 41 healthy participations (mean age 56) from exercise stress test
are also included.

Table 3: Characteristics of three external test set | UCLA medical center | Stanford dataset | Simband dataset
---|---|---|---
#Patients | 126 | 173 | 98
#PPG-AF | 36,855 (15.24%) | 4230 (24%) | 348 (40.7%)
#PPG-NonAF | 249,278 (84.75%) | 13387 (76%) | 506 (59.2%)
good quality(%) | 56.6% | 20.9% | 20.1%
Signal type | Fingertip | Wearable | Wearable
Setting | Hospital | Ambulatory | Ambulatory

##  3 Methods

###  3.1 Problem definition

We denote the training data as
$D=\left\\{\left(x_{i},y_{i}\right)\right\\}_{i=1}^{|D|}$, in which $x_{i}$
represents the $i^{th}$ signal and $y_{i}$ is the related label for the
$i^{th}$ signal from the concurrent alarm. Furthermore, $y_{i}\in\\{0,1\\}$ is
a categorical label, where 0 represents the signal does not contain atrial
fibrillation, and 1 when the signal contains atrial fibrillation. Given the
presence of label noise, $y_{i}$ can be not correct. Moreover, the
distribution of label errors is unknown.

###  3.2 Robust learning with Cluster consistency

####  3.2.1 Autoencoder

The Autoencoder (AE) is a neural network that attempts to recreate the input
as close to its original form as possible [38]. Encoder and decoder are the
two components that makeup AE. Figure 5 illustrates the fundamental concept
behind AE. The first step in using AE is encoding the input and transforming
it into latent space representation. Following this step, the decoder will
decode the representation. The training of the AE optimizing the parameters of
the encoder and the decoder such that the difference between output and the
actual input can be minimized. It can be expected that there is a particular
mapping between the input and the corresponding latent space vector if the
output is consistent with the original input. This would mean that the latent
space vector is an eigenvector of the input. In addition, the latent space
vector that the encoder code produces may be considered an efficient and novel
representation of the input in a much lower dimension. In this study, we use
AE as a pre-processing step for two reasons: feature extraction and dimension
reduction. We train the AE for the entire training data, then get the
representation from the latent space.

Figure 5: The architecture of Auto-encoder

The K-Means technique is a time-tested example of an unsupervised clustering
method [39]. K-Means will iteratively split the data points into K clusters as
per distance, group of data points denoted by
$D=\\{x_{1},x_{2},\cdots,x_{p}\\}$ and the number of clusters denoted by K.
K-Means requires both the number of clusters and the number of data points. We
can refer to the clusters using the notation $C=\\{C1,C2,CK\\}$. The purpose
of the method is to get the data in the same cluster as close together as
possible while bringing the data from other clusters as far apart as possible.
The distance in this case might be measured using the Euclidean distance,
which is calculated as follows:

$d_{ij}=\left\|\bm{H}\left(x_{i}\right)-\bm{H}\left(x_{j}\right)\right\|^{2}$
(1)

in which case $\|x\|^{2}$ refers to the 2-Norm of $x$, and $\bm{H}$ represents
the mapping function from AE.

####  3.2.2 Cluster consistency loss

Our proposed algorithm is predicated on the premise that the loss function to
train with noisy labels can be regularized by additional terms that capture
regularities intrinsic in the input feature manifold. In particular, we first
use an unsupervised learning (hence not subject to label noise) approach to
cluster training samples into a finite number of clusters. Then the learned
cluster membership is used in the subsequent supervised learning phase to
force the distance in the latent space of samples in the same cluster to be
small while that of samples in different clusters to be big. This approach can
be plugged into various neural network architectures by simply augmenting
cross-entropy loss with two additional terms:

$L_{\text{intra}}=\sum_{c=1}^{K}\left(\sum_{i=1}^{N_{c}}\sum_{j\neq
i}^{N_{c}}\left\|\bm{F}\left(x_{i}^{c};\theta\right)-\bm{F}\left(x_{j}^{c};\theta\right)\right\|\right)$
(2)
$L_{\text{inter}}=-\sum_{c_{1}=1}^{K}\sum_{c_{2\neq}c_{1}}^{K}\left(\sum_{i=1}^{N_{c_{1}}}\sum_{j=1}^{N_{c_{2}}}\left\|\bm{F}\left(x_{i}^{c_{1}};\theta\right)-\bm{F}\left(x_{j}^{c_{2}};\theta\right)\right\|\right)$
(3)

where the first term is the sum of distance among intra-cluster samples and
the second term is the sum of distance among inter-cluster samples, both in
latent space being learned with a function $\bm{F}$ parameterized by $\theta$,
which is learnt from the last convolution layer. This framework is flexible
and yet has a small extra computational cost for a mini-batch based learning.

The final loss is weighted combination with the cross-entropy loss and the
proposed cluster consistency loss,

$L=L_{CE}+\lambda_{1}\times L_{\text{intra}}+\lambda_{2}\times
L_{\text{inter}}$ (4)

$\lambda_{1}$ and $\lambda_{2}$ are the hyperparameters balancing the three
terms.

Algorithm 1 Training procedure with cluster consistency loss

0: Dataset, $\\{\bm{x}_{i},\bm{y}_{i}\\}$; $\bm{M}$, number of clusters;
$\bm{\varphi}$, model parameter set for Resnet; ${\lambda}_{1}$ and
${\lambda}_{2}$.

1: Generate embedding $\\{\bm{h}_{i}\\}$ for each $\bm{x}_{i}$ by an
autoencoder

2: Perform M-cluster K-means on $\\{\bm{h}_{i}\\}$, and then generate
$\left\\{\widehat{\bm{y}}_{i}\right\\}$ for each
$\bm{x}_{i}$.$\left\\{\widehat{\bm{y}}_{i}\right\\}\in\\{1,\cdots,M\\}$

3: while not convergent do

4: Calculate $\bm{L_{\rm{intra}}}$, $\bm{L_{\rm{interfor}}}$ each cluster in
$\\{1,\cdots,M\\}$

5: Loss $=L_{CE}+\lambda_{1}\times L_{\text{intra}}+\lambda_{2}\times
L_{\text{inter}}$

6: Update $\bm{\varphi}$ through backpropagation

7: end while

###  3.3 Data preprocessing and Training details

Continuous PPG recordings from UCLA and Simband are cut into 30-second
segments without overlap. Since signals from Stanford are in 25 second, we
augmented each signal by append the first 5-second to the end. Moreover, PPG
signals collected from UCSF and UCLA are both with 240hz, one 30-second
segment will result in 7200 data points. While the frequencies for signals in
Simband and Stanford are 128 and 25HZ, we upsampled signals from these two
test sets to 240hz. Minmax normalization then is applied to all the signals to
control the scale into [0,1].

We used Resnet-34 [40] as the base model for all the methods. For each
training process, we used total 50 epochs and 1000 as the batch size.
Validation set is generated by selecting 20 patients randomly from the
training data, which ensure no data leakage. The best epoch for testing is
selected based on the least loss from the validation set. Adam is used as the
optimizer with a learning rate at 0.00004. INCV, SCE, ELR and SOP were trained
on the GPU cluster from Emory Biomedical informatic department with vary
choices of GPU. INCV, since it is memory intensive, was trained on a large
memory machine with Nvidia Tesla P100, and SCE, ELR and SOP were trained with
Nvidia Tesla V100. The rest methods were trained on a standalone server with
one Intel Xeno 6212U (48 cores) and one Nvidia A2 Tensor Core GPU.

##  4 Results

###  4.1 Hyperparameter tuning

Table 4: The optimal hyperparameters selected for different cluster number #Cluster | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---
$\left\\{\lambda_{\text{intra}},\lambda_{\text{inter}}\right\\}$ | {1.8, 0.3} | {1.5, 0.4} | {1.3, 0.4} | {1.2, 0.6} | {1.8, 0.7}

Table 4 reports the values of optimal hyper-parameters selected for equation 4
by grid search. We set the range [0, 2] for $\lambda_{\text{intra}}$ and
$\lambda_{\text{inter}}$ with a step size of 0.1. Same model trained repeatly
with different combinations of
$\left\\{\lambda_{\text{intra}},\lambda_{\text{inter}}\right\\}$. From table
4, we can observe that $\lambda_{\text{intra}}$ plays more importance role
compared to $\lambda_{\text{inter}}$, with all $\lambda_{\text{intra}}$ are
greater than 1 and $\lambda_{\text{inter}}$ are less than 1. We can also see
that the importance of inter-cluster distance increases with the increased
cluster number, from 0.3 at cluster-2 to 0.7 when we use 6 clusters. The
performance reported in the rest of paper are all from the model trained with
selected optimal $\lambda_{\text{intra}}$ and $\lambda_{\text{inter}}$.

###  4.2 Performance improvement

Table 5: Performance comparison between our method with state-of-the-art algorithms | UCLA | Simband | Stanford
---|---|---|---
|  | whole
---
dataset
| bad
---
quality
| good
---
quality
| whole
---
dataset
| bad
---
quality
| good
---
quality
| whole
---
dataset
| bad
---
quality
| good
---
quality
CE | 0.9507 | 0.9282 | 0.9654 | 0.8216 | 0.7857 | 0.9595 | 0.5853 | 0.5277 | 0.9051
Symmetric CE | 0.955 | 0.9306 | 0.9621 | 0.8109 | 0.7793 | 0.9699 | 0.5859 | 0.5359 | 0.7429
Co-teaching | 0.9446 | 0.907 | 0.9655 | 0.7982 | 0.7556 | 0.9539 | 0.5448 | 0.4626 | 0.8217
INCV | 0.9543 | 0.9416 | 0.9646 | 0.844 | 0.8022 | 0.9697 | 0.6082 | 0.5472 | 0.9245
DivideMix | 0.9537 | 0.9459 | 0.9606 | 0.8977 | 0.8636 | 0.9987 | 0.7437 | 0.6983 | 0.9631
ELR | 0.9206 | 0.9167 | 0.9228 | 0.7959 | 0.7161 | 0.9256 | 0.503 | 0.4807 | 0.8266
SOP | 0.959 | 0.9387 | 0.9664 | 0.8672 | 0.8109 | 0.9644 | 0.6633 | 0.5752 | 0.9614
Cluster = 2 | 0.9545 | 0.947 | 0.9619 | 0.9073 | 0.8737 | 0.9965 | 0.7277 | 0.6779 | 0.9614
Cluster = 3 | 0.9522 | 0.9448 | 0.9554 | 0.8591 | 0.8225 | 0.9799 | 0.7054 | 0.6391 | 0.9503
Cluster = 4 | 0.9554 | 0.9493 | 0.9598 | 0.892 | 0.8629 | 0.9818 | 0.7327 | 0.6712 | 0.9554
Cluster = 5 | 0.9522 | 0.9449 | 0.9599 | 0.8825 | 0.8443 | 0.9833 | 0.7095 | 0.6759 | 0.9703
Cluster = 6 | 0.9563 | 0.9468 | 0.9589 | 0.8796 | 0.841 | 0.9793 | 0.7416 | 0.6847 | 0.9724

Table 5 summarizes the AF detection performance on the three external test
sets separately, including seven baseline methods and the proposed cluster
consistency with cluster number from 2 to 6. AUC is used as the metrics since
signal quality subgroup within each test sets are imbalanced.

In a general view, we can observe that AUCs on good quality signals are always
better than bad quality ones. We can also see a performance degradation from
Simband ($\sim$85%) and Stanford ($\sim$65%) compared to the AUCs from UCLA
($\sim$95%). Since signals from Simband and Stanford are all collected from
wrist in the ambulatory setting, while signals collected in UCLA and UCSF from
fingertip in the hospital setting, the data domain shift problem indeed causes
the performance degradation.

When tested on UCLA dataset, in which the percentage of good quality signals
is around 56%., both baseline methods and our method can achieve an AUC over
95%. We do see the cluster consistency loss shows about 0.5% AUC decreased
compared to baseline methods on all good quality signals, with 95% of AUC
while other baselines are all greater than 96% except ELR. However, co-
teaching degraded to 0.91% when tested on bad quality signals. At the same
time, the performance of CE and symmetric CE are also reduced to 92.5% while
cluster consistency loss still keeps an AUC over 94%.

When tested on Simband dataset, in which the percentage of good quality is
around 20%. Only DivideMix and our method can achieve $\sim$90% of AUC, which
are 10% higher than co-teaching and ELR. Our method and DivideMix also almost
achieve 100% of AUC when tested on good quality signals, this shows great
generalizability of the methods towards the data domain shift problem. ELR is
again the worst among all the methods, which is 16% less compared to our
methos when tested on poor quality signals.

When tested on Stanford dataset, in which only 20% of the signals are with
good quality. Different from Simband dataset which the number of AF and Non-AF
cases are balanced, there is around 1:3 ratio between AF and Non-AF in
Stanford dataset. As reported in Table 4, our method and DivideMix all have
AUC greater than 70% which show great resilience to the domain shift problem.
However, other methods are under 60% except SOP. ELR only has 50% of AUC which
is almost guess level when tested on the Stanford dataset.

###  4.3 Robustness to poor quality

Figure 6: The comparison of resilience to poor data quality between methods

We also want to investigate the influence of data quality. Fig. 6 here is
derived from Table 5 since we want to show the performance drop from good
quality to poor quality signals in a straightforward way. X-axis in each sub-
figure represents the percentage of reduced AUC when testing on poor quality
instead of good quality, Y-axis represents the AUC tested on the whole
datasets. An ideal method would have greater AUC while keeps lower degradation
respected to the bad data quality. As shown in Fig. 6, the points of cluster
consistency are all on the top-left corner in all three datasets. Along with
our method, points of DivideMix also locate on the top-left corner, even
better than us on the Stanford dataset. We can see that ELR has the worst and
unstable performance, which is on the bottom right in Simband and Stanford,
and bottom left on UCLA.

###  4.4 Statistical analysis

We used pair-wise T-test to test the difference between methods. For each
sampling process, one single signal was selected randomly for each patient,
which give us 126, 173, 98 samples each time for three test sets to get AUCs.
We then repeat this sampling process 100 time. 6 shows the results for three
test sets separately. Since we have repeated 100 times, the threshold of
p-value should be adjusted from 0.05 to 0.0005. From Table 6a for UCLA, we can
observe that, except ELR, the performance of most of the methods are not
significantly different from each other. Meanwhile, our method with different
cluster numbers also show consistency in AUC with all the p-values greater
than 0.0005. From Table 6b for Simband, we observe different phenomenon. Over
half of the pairs have significantly difference between each other, especially
for SCE, Co-teaching and ELR, which also have worse results from 5. Meanwhile
DivideMix and SOP has similar performance compared to our method. Also, there
is no significantly difference between our method with different cluster
number. From the Table 6c for Stanford, the difference is further expanded. We
can see both CE, SCE, Co-teaching and SOP are all different from the rest
methods. Even with DivideMix, it is different from our method with cluster
number of 3 and 6. Within our method, cluster number 6 is different from other
cluster numbers and only cluster 2 and 5 are similar with a p-value of 0.66.

Table 6: P-value of pair-wise test between different methods for all datasets

CE |
---|---
SCE | 0.56 |
Co-teaching |  $<$ 5e-04 | 0.001 |
Dividemix | 0.24 | 0.08 |  $<$ 5e-04 |
ELR |  $<$ 5e-04 |  $<$ 5e-04 |  $<$ 5e-04 |  $<$ 5e-04 |
SCP | 0.71 | 0.34 |  $<$ 5e-04 | 0.44 |  $<$ 5e-04 |
Closs2 | 0.005 | 0.02 | 0.62 |  $<$ 5e-04 |  $<$ 5e-04 | 0.002 |
Closs3 | 0.007 | 0.032 | 0.48 |  $<$ 5e-04 |  $<$ 5e-04 | 0.003 | 0.85 |
Closs4 | 0.28 | 0.61 | 0.01 | 0.03 |  $<$ 5e-04 | 0.16 | 0.07 | 0.1 |
Closs5 | 0.74 | 0.79 | 0.0005 | 0.13 |  $<$ 5e-04 | 0.49 | 0.01 | 0.03 | 0.44 |
Closs6 | 0.96 | 0.58 |  $<$ 5e-04 | 0.23 |  $<$ 5e-04 | 0.68 | 0.006 | 0.009 | 0.3 | 0.78 |
| CE | SCE | Co-teaching | Dividemix | ELR | SOP | Closs2 | Closs3 | Closs4 | Closs5 | Closs6

a. UCLA dataset
CE SCE 0.17 Co-teaching 0.06 0.41 Dividemix 0.002 $<$ 5e-04 $<$ 5e-04 ELR 0.04
0.25 0.72 $<$ 5e-04 SCP 0.16 0.008 0.004 0.05 0.002 Closs2 $<$ 5e-04 $<$ 5e-04
$<$ 5e-04 0.67 $<$ 5e-04 0.01 Closs3 0.16 0.003 0.002 0.015 0.001 0.83 0.0009
Closs4 0.003 $<$ 5e-04 $<$ 5e-04 0.66 $<$ 5e-04 0.1 0.36 0.03 Closs5 0.004 $<$
5e-04 $<$ 5e-04 0.699 $<$ 5e-04 0.10 0.38 0.04 0.97 Closs6 0.02 $<$ 5e-04 $<$
5e-04 0.46 $<$ 5e-04 0.11 0.13 0.03 0.8 0.78 CE SCE Co-teaching Dividemix ELR
SOP Closs2 Closs3 Closs4 Closs5 Closs6
b. Simband dataset
CE SCE $<$ 5e-04 Co-teaching $<$ 5e-04 $<$ 5e-04 Dividemix $<$ 5e-04 $<$ 5e-04
$<$ 5e-04 ELR $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 SCP $<$ 5e-04 $<$ 5e-04
$<$ 5e-04 $<$ 5e-04 $<$ 5e-04 Closs2 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 0.37 0.001
$<$ 5e-04 Closs3 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 0.65 $<$ 5e-04 $<$
5e-04 Closs4 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 0.001 0.41 $<$ 5e-04 0.016 0.2
Closs5 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 0.89 $<$ 5e-04 $<$ 5e-04 0.33 $<$ 5e-04
0.001 Closs6 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 $<$
5e-04 $<$ 5e-04 $<$ 5e-04 $<$ 5e-04 CE SCE Co-teaching Dividemix ELR SOP
Closs2 Closs3 Closs4 Closs5 Closs6
c. Stanford dataset

###  4.5 Training time

Figure 7: Training time of one epoch with same Resnet-34 model

In previous section, we show that we have better performance than all the
baseline models, and on par with DivideMix. We also recorded the training time
for one epoch to investigate the efficiency of each method. Methods run on the
BMI cluster were re-run for one epoch on the standalone server to ensure the
equality except INCV. Because INCV took more than three days to finish
running, the training time here is based on reasonable estimation. To make
fair comparison, we did not consider the time for pre-train steps, such as the
warm-up epochs for INCV and DivideMix, which cost much more extra time. The
training time for auto-encoder and clustering of our method is also not
considered, which is $\sim$20 minutes in practice. We can observe from Fig. 7
that, the additional calculation of inter-cluster and intra-cluster distance
only increase $\sim$2 minutes of training time. Since co-teaching and
DivideMix are training two models simultaneously, the training time is nearly
doubled compared to our method.

##  5 Visualization

Cluster-consistency + CE CE only

Figure 8: The visualization of top-10 neighbors from the learnt representation

We also randomly selected 4 samples, one sample each from each category - good
quality AF, good quality Non-AF, bad quality AF and bad quality Non-AF, and
generated their top-10 nearest neighbors from the training data based on the
representations learnt from last convolutional layer, as shown in 8. Then we
manually annotate the selected neighbors to see the true label rate. Neighbors
colored with green means the label of that signal from the training data is
true. We can observe that, compared to CE, the additional cluster-consistency
loss has more purity of neighborhood and hence helps aggregate signals with
same true labels.

##  6 Discussion

###  6.1 Promising to use alarm to auto-label the signal

To our knowledge, we are the first team trying to use EHR information to auto-
label the concurrent biomedical signal so human annotation effort can be
avoided. Although false alarms exist, we can still observe accurate
predictions on external test sets even with conventional CE loss. This
phenomenon shows deep learning is robust to label errors to certain level,
which is same as we concluded in the previous work [10]. The accurate
predictions do show great potential of using similar way to generate large-
scale dataset with no human expert involved. However, our strategy to auto-
label the signal is still trivial in this study since we only used alarm
information. More modalities from EHR, such as patient diagnosis, demographic
features and chart events, can be used to further filter false alarms and
hence improve the accuracy of the pseudo labels.

###  6.2 Consistency regulation can alleviate the influence of label error
from real-world noise.

The key finding of this study is cluster-consistency can alleviate the
influence of real-world label error. From table 5, first we can observe that
CE does not perform the worst compared to others, which indicates that deep
learning is robust to label noise. If the computation resource is limited, one
can still use CE and it will give comparable performance. Second, we can
conclude that both our method and DivideMix outperform the rest of methods.
The most plausible reason is we both include unsupervised information into
consideration. DivideMix treats the label noise issue as a semi-supervised
problem, which it divides the training data into clean and noisy subsets, then
re-label the noisy subsets by ‘co-guessing’ from two pre-trained network to
ensure the consistency regulation. Our method is similar but more intuitive
compared to DivideMix, we ensure the cluster consistency from pre-trained
clusters. The results indicate that adding extra regulation from unsupervised
information can magnificently reduce the influence caused by noisy labels.

###  6.3 Generalizability and Robustness to poor data quality.

When tested on Simband and Stanford, performance of all the methods decrease
significantly. It is highly because of different data collection environment
and study protocol. Since both Simband and Stanford data are collected from
wrist-worn devices from ambulatory setting while the training data from UCSF
are collected from fingertip in hospital setting, which patients are more
motionless. Also, the study protocol is different. Although Simband and
Stanford have similar collection environment, Stanford data also incorporated
41 patients from exercise stress test that includes more intensive movement.
According to table 3 and 5, although the percentage of poor quality signals
are all around 20%, the AUC is much higher on Simband than tested on Stanford
data. The performance of ELR and SOP, which are top-runner from the
competition, decrease significantly compared to our method and DivideMix.
Reasons can be summarized as follow: a) Real-world noise. Although studies
like ELR and SOP have tried symmetric and asymmetric label flipping to testify
their superiority, real-world noise cannot never been estimated. SOP was
developed based on the distribution of label noise is sparse. However, the
distribution of the label noise in our data is unknow, which could be non-
sparse. b) Poor data quality. Dataset used in label noise studies are well-
prepared image datasets in which the image quality is not considered. This is
the most plausible reason to the lower performance of these state-of-the-art
methods. The key assumption of ELR is early-stopping can prevent the model
from overfitting caused by label noise. However, the factor of data quality
has not been investigated, and our results show that early-stopping can be
trusted when poor data quality is involved. c) We use autoencoder as the
preliminary representation step for clustering. Auto-ender aims to lean an
informative compressed representation which can be recovered back to raw
input, which can be used for data denoising. In out study, the learnt
representation for each signal is robust to poor signal quality to certain
extent, which is a plausible reason for our good performance on poor quality
subsets.

###  6.4 Scalability and flexibility.

As shown in Fig. 8, the proposed cluster-consistency only cost extra 2 minutes
training time compared to conventional CE. However, to achieve the similar
performance, DivideMix has to train two models simultaneously with extra pre-
trained warm-up epochs. This scalability makes the cluster-consistency a
promising strategy to handle label noise issue for large-scale datasets.
Compared to co-teaching and DivideMix, which need extra step in order to adapt
to new task, our method is more flexible and plus-and-play for different model
architectures and tasks.

###  6.5 The choice of cluster number

We are aware there are techniques to find an optimal cluster number, such as
Silhouette analysis and Calinski-Harabasz Index. Due to the limited
computation resources, we did not investigate comprehensively on the choice of
cluster number. We chose 6 as the maximum cluster number because we used three
types of alarms, AF, PVC and NSR. Signals from each alarm type can have good
quality and poor quality, which gives us 6 categories totally. However, either
from performance from Table 5 or results of pair-wise T-test from Table. 6, we
did not observe significantly difference between different cluster numbers.
Also, the goal of this study is to propose a generic and applicable strategy
for label noise, the choice of cluster number is an hyperparameter and task
specific, which is beyond the goal of this study. However, a comprehensive
investigation of finding the optimal cluster number is also an importance
future work.

###  6.6 Limitations and future work

Our study also has limitations. First, the built-in algorithms from bedside
monitor are designed to take care all kinds of situations, such as PVC and VT,
and not for AF specifically, which leads to a lower true AF alarm rate. We
might need to adopt a more advanced AF-ECG algorithm to auto-label the
concurrent PPG in order to get purer dataset. Second, Resnet-34 is used as
base model in this study. Although it is beyond the scope of this study, a
deeper and more sophisticate architecture can be used to further improve the
classification accuracy. Third, we only focused on the AF and PVC alarm. Other
arrhythmias, such as VT and AF flutter, might be more difficult to distinguish
if we only reply on alarm information and with no human knowledge introduced.
Although our model is among the top in detecting AF in free living scenarios
using wearable ppg as shown in Simband and Stanford data, there is still room
for improvement in the future. First, we shall find a strategy to find the
optimal number of cluster. Second, we only used conventional K-means as the
clustering method, we will try different cluster techniques, such as spectral
clustering or other deep cluster methods. Third, Denoising auto-encoder or
variational auto-encoder could be more robust to the poor quality data and
domain shift, which are promising next steps for our future work. Lastly, we
used grid search to decide the weights for intra-cluster and inter-cluster, an
optimal approach is also needed to be investigated in the future.

##  7 Conclusion

In this study, we main achieve two goals. First, we manage to auto-annotate
over 8 million 30 second PPG segments using alarms from bed-side monitor,
which is by far the largest dataset to our knowledge. Second, with the
proposed cluster consistency loss, we are able to alleviate the performance
degradation caused by label errors. This whole pipeline is a promising
strategy for the research community when annotation effort is limited and
potential label error problem.

## References

  * [1] Nicholas A Bosch, Jonathan Cimini, and Allan J Walkey.  Atrial fibrillation in the icu.  Chest, 154(6):1424–1434, 2018.
  * [2] Gari D Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE Johnson, and Roger G Mark.  Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge 2017.  In 2017 Computing in Cardiology (CinC), pages 1–4. IEEE, 2017.
  * [3] Tania Pereira, Nate Tran, Kais Gadhoumi, Michele M Pelter, Duc H Do, Randall J Lee, Rene Colorado, Karl Meisel, and Xiao Hu.  Photoplethysmography based atrial fibrillation detection: a review.  NPJ digital medicine, 3(1):1–12, 2020.
  * [4] Yichen Shen, Maxime Voisin, Alireza Aliamiri, Anand Avati, Awni Hannun, and Andrew Ng.  Ambulatory atrial fibrillation monitoring using wearable photoplethysmography with deep learning.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1909–1916, 2019.
  * [5] Peng Cheng, Zhencheng Chen, Quanzhong Li, Qiong Gong, Jianming Zhu, and Yongbo Liang.  Atrial fibrillation identification with ppg signals using a combination of time-frequency analysis and deep learning.  IEEE Access, 8:172692–172706, 2020.
  * [6] Alireza Aliamiri and Yichen Shen.  Deep learning based atrial fibrillation detection using wearable photoplethysmography sensor.  In 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pages 442–445. IEEE, 2018.
  * [7] Supreeth Prajwal Shashikumar, Amit J Shah, Qiao Li, Gari D Clifford, and Shamim Nemati.  A deep learning approach to monitoring and detecting atrial fibrillation using wearable technology.  In 2017 IEEE EMBS international conference on biomedical & health informatics (BHI), pages 141–144. IEEE, 2017.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [9] Curtis G Northcutt, Anish Athalye, and Jonas Mueller.  Pervasive label errors in test sets destabilize machine learning benchmarks.  arXiv preprint arXiv:2103.14749, 2021.
  * [10] Cheng Ding, Tania Pereira, Ran Xiao, Randall J Lee, and Xiao Hu.  Impact of label noise on the learning based models for a binary classification of physiological signal.  Sensors, 22(19):7166, 2022.
  * [11] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee.  Learning from noisy labels with deep neural networks: A survey.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [12] Zhaowei Zhu, Jialu Wang, and Yang Liu.  Beyond images: Label noise transition matrix estimation for tasks with lower-quality features.  arXiv preprint arXiv:2202.01273, 2022.
  * [13] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang.  Understanding and utilizing deep neural networks trained with noisy labels.  In International Conference on Machine Learning, pages 1062–1070. PMLR, 2019.
  * [14] Junnan Li, Richard Socher, and Steven CH Hoi.  Dividemix: Learning with noisy labels as semi-supervised learning.  arXiv preprint arXiv:2002.07394, 2020.
  * [15] Ming Chen, Guijin Wang, Zijian Ding, Jiawei Li, and Huazhong Yang.  Unsupervised domain adaptation for ecg arrhythmia classification.  In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 304–307. IEEE, 2020\.
  * [16] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  Advances in neural information processing systems, 31, 2018.
  * [17] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda.  Early-learning regularization prevents memorization of noisy labels.  Advances in neural information processing systems, 33:20331–20342, 2020.
  * [18] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You.  Robust training under label noise by over-parameterization.  arXiv preprint arXiv:2202.14026, 2022.
  * [19] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang.  Learning from massive noisy labeled data for image classification.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2691–2699, 2015.
  * [20] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  CoRR, abs/1710.09412, 2017.
  * [21] Simon Jenni and Paolo Favaro.  Deep bilevel learning.  In Proceedings of the European conference on computer vision (ECCV), pages 618–633, 2018.
  * [22] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang.  Robust early-learning: Hindering the memorization of noisy labels.  In International conference on learning representations, 2020.
  * [23] Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An.  Open-set label noise can improve robustness against inherent label noise.  CoRR, abs/2106.10891, 2021.
  * [24] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Regularizing very deep neural networks on corrupted labels.  CoRR, abs/1712.05055, 2017.
  * [25] Yanyao Shen and Sujay Sanghavi.  Iteratively learning from the best.  CoRR, abs/1810.11874, 2018.
  * [26] Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang.  Understanding and utilizing deep neural networks trained with noisy labels.  CoRR, abs/1905.05040, 2019.
  * [27] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322–330, 2019.
  * [28] Yueming Lyu and Ivor W. Tsang.  Curriculum loss: Robust learning and generalization against label corruption.  CoRR, abs/1905.10045, 2019.
  * [29] Cristina Gallego Vázquez, Alexander Breuss, Oriella Gnarra, Julian Portmann, Antonio Madaffari, and Giulia Da Poian.  Label noise and self-learning label correction in cardiac abnormalities classification.  Physiological measurement, 43(9):094001, 2022.
  * [30] Pengfei Wu and Senping Tian.  Using semi-supervised cluster method to correct the mislabeled training samples of ecg signals.  In 2020 IEEE 9th Data Driven Control and Learning Systems Conference (DDCLS), pages 260–265. IEEE, 2020.
  * [31] Konstantinos Nikolaidis, Thomas Plagemann, Stein Kristiansen, Vera Goebel, and Mohan Kankanhalli.  Using under-trained deep ensembles to learn under extreme label noise: A case study for sleep apnea detection.  IEEE Access, 9:45919–45934, 2021.
  * [32] Zijiao Chen, Zihuai Lin, Peng Wang, and Ming Ding.  Negative-resnet: noisy ambulatory electrocardiogram signal classification scheme.  Neural Computing and Applications, 33(14):8857–8869, Jul 2021.
  * [33] Darae Ko, Faisal Rahman, Renate B Schnabel, Xiaoyan Yin, Emelia J Benjamin, and Ingrid E Christophersen.  Atrial fibrillation in women: epidemiology, pathophysiology, presentation, and prognosis.  Nature Reviews Cardiology, 13(6):321–332, 2016.
  * [34] Susan Colilla, Ann Crow, William Petkun, Daniel E Singer, Teresa Simon, and Xianchen Liu.  Estimates of current and future incidence and prevalence of atrial fibrillation in the us adult population.  The American journal of cardiology, 112(8):1142–1147, 2013.
  * [35] Tania Pereira, Cheng Ding, Kais Gadhoumi, Nate Tran, Rene A Colorado, Karl Meisel, and Xiao Hu.  Deep learning approaches for plethysmography signal quality assessment in the presence of atrial fibrillation.  Physiological measurement, 40(12):125002, 2019.
  * [36] Cheng Ding, Ran Xiao, Karl Meisel, Randall J Lee, and Xiao Hu.  Welch-gan: Generating realistic photoplethysmography signal from frequency-domain for atrial fibrillation detection.  arXiv preprint arXiv:2108.05272, 2021.
  * [37] Jessica Torres-Soto and Euan A Ashley.  Multi-task deep learning for cardiac rhythm detection in wearable devices.  NPJ digital medicine, 3(1):1–8, 2020.
  * [38] Umberto Michelucci.  An introduction to autoencoders.  arXiv preprint arXiv:2201.03898, 2022.
  * [39] J MacQueen.  Classification and analysis of multivariate observations.  In 5th Berkeley Symp. Math. Statist. Probability, pages 281–297, 1967.
  * [40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.


# An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator

Ravi Tejwani1, Chengyuan Ma1, Paolo Bonato2 and H. Harry Asada3 1Ravi Tejwani
and Chengyuan Ma are with Department of Electrical Engineering and Computer
Science (EECS), Massachusetts Institute of Technology, Cambridge, MA 02142 USA
{tejwanir<EMAIL_ADDRESS>Bonato is with Harvard Medical School
Department of Physical Medicine and Rehabilitation at Spaulding Rehabilitation
Hospital, Charlestown, MA 02129 USA<EMAIL_ADDRESS>Harry Asada is
with Department of Mechanical Engineering, Massachusetts Institute of
Technology, Cambridge, MA 02142 USA<EMAIL_ADDRESS>
###### Abstract

Although telepresence assistive robots have made significant progress, they
still lack the sense of realism and physical presence of the remote operator.
This results in a lack of trust and adoption of such robots. In this paper, we
introduce an Avatar Robot System which is a mixed real/virtual robotic system
that physically interacts with a person in proximity of the robot. The robot
structure is overlaid with the 3D model of the remote caregiver and visualized
through Augmented Reality (AR). In this way, the person receives haptic
feedback as the robot touches him/her. We further present an Optimal Non-
Iterative Alignment solver that solves for the optimally aligned pose of 3D
Human model to the robot (shoulder to the wrist non-iteratively). The proposed
alignment solver is stateless, achieves optimal alignment and faster than the
baseline solvers (demonstrated in our evaluations). We also propose an
evaluation framework that quantifies the alignment quality of the solvers
through multifaceted metrics. We show that our solver can consistently produce
poses with similar or superior alignments as IK-based baselines without their
potential drawbacks.

##  I INTRODUCTION

In a situation when a person (such as an older adult) needs physical
assistance, a care giver would be required to provide that support. In remote
interactions, physical assistance should be delivered without degrading the
quality of service. However, providing quality remote service is more
challenging than in-person care. This could be realized by using a physical
agent, i.e. a robot, being operated from a remote site. However, it might be
uncomfortable or even terrifying to have such robot touch a care recipient’s
body. Rather, it is desirable that a remote caregiver appears to be in contact
with the recipient while the robot is providing the physical assistance.

Advancements in telepresence assistive robots have enabled care givers to
remotely monitor and assist patients during the rehabilitation process by
observing and communicating with patients in real-time, even if they are not
physically present in the same location. These robots have been used in a
variety of rehabilitation services, including exercises, stretches, and
massages [1, 2, 3]. They have also helped patients to improve their mobility
and balance by providing feedback on their movements and posture [4, 5, 6].

The research on visual representation of the remote therapist on to the robot
has been limited in their ability to display the remote therapist in the form
of 2D screens [1, 2, 7] or using virtual and augmented reality to display the
remote person [8, 9, 10]. Even the state of the art telepresence avatar robots
[11, 12, 13] simply project the avatar (3D human model) of the remote
therapist on to the screens. Hence, there is a fundamental disconnect in the
research literature between the visual display of the remote care giver on the
tele-operated robot and the actual remote physical therapy performed by the
robot.

In this paper, we aim to bridge this gap by conceptualizing the development of
an Avatar Robot as a mixed real/virtual assistive tele-operated robot – a
physical robot that is overlaid with the 3D Human model of a remote care
giver. Therefore, the care receiver might feel as though he is in contact with
the remote operator.

Our contributions include:

  1. 1.

Avatar Robot System: To the best of our knowledge, we propose the first
general purpose end-to-end robotic system that provides the integration of the
trajectory control of the robot, cross-device pose synchronization, robot
tracking, alignment solver(s), and online 3D projection overlay using an
Augmented Reality (AR) device; it achieves efficient and reactive real-time
pose synchronization, alignment computation, and overlay projection.

  2. 2.

Alignment solver: An optimal non-iterative alignment solver that solves for
the optimally aligned pose of 3D Human model to the robot (shoulder to the
wrist). We show how existing iterative IK solvers such as Jacobian DLS [14]
and FABRIK [15] can be adapted for this purpose. Compared to these baselines,
our solver is intuitive, concise, deterministic, singularity-free, and
superior in alignment metrics. We also provide theoretical guarantees of the
solver;

  3. 3.

Empirical metrics and evaluation: Set of evaluation metrics that qualitatively
measures the quality of human-robot alignment from multiple perspectives
including visual overlap, structural deviation, and model distortion. Our
evaluation metrics could serve as a benchmark for evaluating future
telepresence robots and their performance.

##  II AVATAR ROBOT SYSTEM

An Avatar Robot is a mixed real/virtual robotic system that physically
interacts with a person in proximity of the robot. As shown in Fig. 1, a
remotely operated robot and the 3D model of a remote operator are incorporated
into our system. The robot structure is overlaid with the 3D model of the
remote caregiver using our solvers and visualized through Augmented Reality
(AR). In this way, the person receives haptic feedback as the robot is in
contact with him/her. Simultaneously, the person would visually observe the
caregiver’s presence through 3D rendering. Thus, care receivers will have a
highly immersive experience when haptic stimuli and visual observations are
synchronized.

Problem Given a 3D model of a human, obtained from the 3D scan of the physical
therapist (Fig. 1(a)), and a dual arm collaborative YuMi robot (Fig. 1(b))
which can provide physical assistance to the person. Our objective is to
optimally align and overlay them across their joints, axes and rotations as
the Avatar Robot interacts with the person.(Fig. 1(c)).

Human Model We prepare the 3D human model from a photogrammetry scan of a
Physical Therapist from Spaulding Rehabilitation Center, Charlestown, MA. The
therapist was in T-pose at the time of scan, and we manually annotated the
joints of the model so the model can be posed dynamically. In this research,
we are interested in dynamically computing the pose of the model’s arms to
create optimal alignment with the robot arms, so joints of particular concern
are the shoulder, elbow, and wrist joints on both sides.

Structural Correspondence between the Human Model and the Robot Since our
robot has anthropomorphic arms, with clearly identifiable “upper arm” and
“forearm” segments, our task is to align the human model’s shoulder, elbow,
and wrist joints to the corresponding _reference positions_ on the robot,
which we choose to be the center of axis 2, 3, 6 of the robot respectively. A
side-to-side comparison between the arms of our human model and the robot as
well as the reference positions that we chose is shown in Fig. 2.

As a preliminary step of alignment, we also compute the position and rotation
of the entire human model such that the segment connecting its shoulder joints
overlaps and share the same midpoint with the segment connecting the shoulder
reference positions on the robot. This step ensures the body /shoulder of the
human model is aligned with the body of the robot at large and allows us to
compute the best-aligning pose of two arms separately.

(a) 3D Scan of Physical Therapist

(c) Aligned Human model over robot

(b) Collaborative robot (YuMi)

Figure 1: Aligning Human model with Collaborative robot Figure 2: Side-by-side
comparison of our robot arm and the arm of our human model. We set the center
of axis 2, 3, and 6 of our robot as the shoulder, elbow, and wrist reference
points for the human model. The difference in proportions of arm segments
between the human model and the robot makes scaling necessary in order to
achieve good alignment. Figure 3: The pipeline of our Avatar Robot System. A
motion server and a pose server run in parallel on both arms of the robot. The
motion server executes commands from scripts on the control station. The
control station also runs a relay that aggregates joint angles from both arms
and stream dual-arm joint poses to the AR device over WLAN. The AR device
converts local-frame joint positions to world-frame by them combining with the
pose estimate of the robot from its cameras. An alignment solver then solves
for the human model pose; A gesture mapper maps gripper positions to hand
gestures of the human model. With the arm poses and gesture, the renderer
renders the human model accurately over the robot. Joint poses are
synchronized between robot, AR device and the control station at 100Hz, while
the rendering is at 60Hz.

Defining Input and Output for the Solvers The input to an alignment solver
consists of an joint positions of the human model in its default (unaligned)
configuration and the reference joint positions of the robot. The output will
be an _arm pose_ for each side, which include the rotations of joints as well
as stretch factor of forearm and upper arm of the human model. Allowing
stretching parts of the human model risks making the output pose un-
anthropormorphic, but it is necessary for better alignment as the proportions
of robot and human arms are inevitably different (see Fig. 2). We assume that
some extent of scaling is acceptable. We will revisit incorporating scaling
and other constraints in the description of the solvers.

Pipeline The Avatar Robot System integrates the alignment solvers into an end-
to-end real-time workflow that comprises trajectory control of the robot,
cross-device pose synchronization, robot tracking, alignment solving, and
overlay projection. The pipeline of robot System is presented in Fig. 3. It
consists of three components: the robot, the control station, and the
augmented reality (AR) device.

The robot, a dual-arm collaborative robot [16], runs a pose server and a
generic motion server in parallel on each arm, and communicates with the
control station through Ethernet. The control station stores control scripts
that issues specific trajectory commands to the robot to perform various
tasks, e.g., emulating a rehabilitation therapy. It also collects joint angles
and gripper orientations from both arms at real-time and hosts a relay server
to streatm this information through the local wireless network. We use
HoloLens 2 [17] as the AR device. The pose of the robot base is tracked using
the camera mounted on the AR device with a computer vision model. The raw pose
estimation is further passed to a low-pass filter to reduce noises and
stuttering. The filtered pose is combined with the robot-frame joint
coordinates retrieved from the control station real-time to compute the world-
frame joint positions. Our system then computes the shoulder, elbow, and wrist
reference positions for the human model and aligns its body with the robot.
The reference positions are then passed to a alignment solver, whose pose
output is then fed to the rendering pipeline which renders the human model and
projects it over the robot in front of the patient.

For completeness, we also designed a gripper mapper module that maps the state
of robot gripper into the gesture of the model. Our gripper is based on a 1DoF
linear gripper with fingers customized to resemble thumb and fingers, so we
map the position of the gripper linearly to the angle between the thumb and
index finger of the model. In practice, this system can align a human model
pose with a robot in real-time with minimal latency and high accuracy.

##  III SOLVERS

In this section, we first explore how existing iterative Inverse Kinematics
(IK) solvers can be adapted to perform the aforementioned alignment task.
Specifically, we show adaptation of two IK solvers: Jacobian Damped Least
Square [14], commonly used in robotics, and FABRIK [15], commonly used in
humanoid animation. These solvers serve as our baselines. We then introduce
our optimal non-iterative alignment (ONIA) solver.

###  III-A Inverse Kinematic Solvers

####  III-A1 Jacobian (adapted)

We first adapted the Jacobian method with damped least square (DLS) for the
alignment task. Given a parameterization $\bm{\theta}$ and the current end
effector position $\bm{p}$, the effect of perturbing $\bm{\theta}$ on $\bm{p}$
can be linearly approximated by $\bm{\Delta p}=J\bm{\Delta\theta}$ in
$\bm{p}$’s neighborhood, where $J$ is the Jacobian. Provided with a desired
$\bm{\Delta p}$, Jacobian with DLS computes $\bm{\Delta\theta}$ as follows,

$\bm{\Delta\theta}=J^{\mathsf{T}}\left(JJ^{\mathsf{T}}+\lambda^{2}I\right)^{-1}\bm{\Delta
p}.$ (1)

where the hyper-parameter $\lambda$ is the damping constant that must be
chosen with care to keep the solver both responsive and stable.

We model the human arm as the shown in Fig. 4. In particular, we parameterize
the swing of the upper arm around the shoulder as an exponential map [18, 19].
To incorporate the reference position of the elbow in Jacobian-based
algorithms, we start by noting that the Jacobian method is equivalent to
gradient descent to the objective

$\frac{1}{2}{\left\|\bm{p}-\bm{x}\right\|^{2}}$ (2)

where $\bm{x}$ is the target position. For a joint, $\bm{p}_{\textit{e}}$, we
want to be close to its reference, $\bm{x}_{\textit{e}}$, we add the term
$\frac{1}{2}w_{\textit{e}}^{2}\left\|\bm{p}_{\textit{e}}-\bm{x}_{\textit{e}}\right\|^{2}$
to the objective,

$\displaystyle\frac{1}{2}\left\|\bm{p}-\bm{x}\right\|^{2}+\frac{1}{2}w_{\textit{e}}^{2}\left\|\bm{p}_{\textit{e}}-\bm{x}_{\textit{e}}\right\|^{2}=\frac{1}{2}\left\|\begin{bmatrix}\bm{p}\\\
w_{\textit{e}}\bm{p}_{\textit{e}}\end{bmatrix}-\begin{bmatrix}\bm{x}\\\
w_{\textit{e}}\bm{x}_{\textit{e}}\end{bmatrix}\right\|^{2}$ (3)

where $w_{\textit{e}}$ is a small constant so
$\frac{1}{2}w_{\textit{e}}^{2}\left\|\bm{p}_{\textit{e}}-\bm{x}_{\textit{e}}\right\|^{2}$
becomes a secondary objective. We find $w_{\textit{e}}=0.1,\lambda=0.2$ to
work best.

####  III-A2 FABRIK (adapted)

Forward And Backward Reaching Inverse Kinematics (FABRIK) is a heuristic-based
IK solver that received wide use in humanoid animation due to its simplicity
and its natural poses [15]. Every iteration of FABRIK consists of a forward
and backward reach that traverse the kinematic chain in different directions.
It repositions a joint on the line connecting its current position and the
preceding joint’s updated position.

Our adaptation presented here enhances plain FABRIK by allowing every joints
to have a reference positions. It produces a pose that minimizes the maximum
deviation of any joint to its reference111We make the remark that a joint with
a reference is different from an end effector with a target, because an end
effector is assumed to be always able to reach its target in FABRIK but it may
very well be infeasible for every joint to be at its reference position, and
FABRIK is unstable when confronted with infeasbility. Hence, plain FABRIK’s
ability to handle multiple end effector does not help in our scenario.. Under
our adaptation, a joint with a reference position is given a deviation
tolerance $\epsilon$. In every iteration, the joint is re-positioned not on
the segment connecting its child and itself, but on the segment connecting the
child and the point closest to the joint _within a radius- $\epsilon$ sphere
around the reference position_. This heuristic is intuitive, local, minimal,
easy to implement, and works well empirically. The choice of $\epsilon$ is
critical: if too large, the constraint is too loose to be useful; if too
small, the solver may be unstable due to infeasibility. We determine the
optimal $\epsilon$ by binary search.

Both Jacobian and FABRIK allow incorporating joint rotational constraints into
the solution. Hence, we explored adding basic human joint constraints into
both baselines: We allow the upper arm to swing for at most $85^{\circ}$ in
all directions and twist for $\pm 75^{\circ}$ relative to the T-pose. The
hinge joint permits rotation from 0 to $150^{\circ}$ inward, with the T-pose
as the zero reference. These constraints are more permissive than anatomical
ground truth [18] so as not to over-constrain our model, as alignment is also
a tight constraint on its own. We allow a stretch factor from 0.8 to 1.3 for
both upper and forearm for reasons discussed in section II.

###  III-B Optimal Non-Iterative Alignment Solver

It is helpful when the input robot arm pose is not anthropomorphic to use IK
solvers that are generic and consider joint constraints. However, when the
robot arm poses are anthropomorphic, the benefits of these baseline IK solvers
fails to outweigh their limitations: they are prone to singularities, and
require careful tuning of its hyper parameters to be stable. We introduce an
Optimal, Non-Iterative, Alignment solver (ONIA) that solves for the optimally
aligned pose from shoulder to the wrist non-iteratively. ONIA is stateless,
achieves optimal alignment and faster than the baseline solvers (demonstrated
in the evaluation section). We describe ONIA’s design below.

##### Nomenclature

We begin by defining our notations. Let ${\mathcal{R}}_{\bm{x}}^{\bm{y}}$
denote the rotation that rotates $\bm{x}$ to $\bm{y}$ around
$\bm{x}\times\bm{y}$. In other words, ${\mathcal{R}}_{\bm{x}}^{\bm{y}}$ is the
rotation such that

${\mathcal{R}}_{\bm{x}}^{\bm{y}}\bm{x}=\bm{y},\quad{\mathcal{R}}_{\bm{x}}^{\bm{y}}(\bm{x}\times\bm{y})=\bm{x}\times\bm{y}.$
(4)

In addition, let

$\left(\bm{x}\right)_{\perp\,\bm{y}}=\bm{x}-\frac{\bm{x}\cdot\bm{y}}{\|\bm{y}\|^{2}}\bm{y}$
(5)

denote the projection of $\bm{x}$ onto the plane normal to $\bm{y}$.

As outlined in section II. The input to ONIA has two parts: the joint
positions of an arm of the human model (in its default unaligned pose), and
the corresponding reference positions of the robot. For clarity, through out
this section, we’ll use the superscripts $h$ to denote joints from the human
model and $r$ to denote references positions of the robot. We will also use
subscripts s, e, w, u, f for houlder, elbow, wrist, upper arm, and forearm.
All joints and rotations below are of the human model unless explicitly
specified otherwise. Fig. 4 visualizes the symbols we used below and presents
the kinematic chain of the human model which ONIA works on.

ONIA aligns the given arm of the human model to an arm of the robot from
shoulder to wrist through the following steps:

$\bm{x}_{s}^{h}$$\bm{x}_{e}^{h}$$\bm{x}_{w}^{h}$$R_{s}^{\mathrm{swing}}$$R_{s}^{\mathrm{twist}}$$\bm{a}^{h}$$R_{e}^{\mathrm{swing}}$$R_{e}^{\mathrm{twist}}$$\bm{d}_{u}^{h}$$\bm{d}_{f}^{h}$$S_{u}$$S_{f}$${\color[rgb]{0.25,0.46,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.46,0.02}\bm{x}_{e}^{r}}$${\color[rgb]{0.25,0.46,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.46,0.02}\bm{x}_{w}^{r}}$$\bm{d}_{u}^{r}$$\bm{d}_{f}^{r}$$\bm{{\color[rgb]{0.25,0.46,0.02}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.46,0.02}x}}{\color[rgb]{0.25,0.46,0.02}{}_{s}^{r}}$human
model armreference pose of the robotshoulderwristelbowleft sideright side
Figure 4: The kinematics model. We model the shoulder joint of a human model
as a ball-and-socket joint which can be decomposed into a swing and a
twist[18]. The elbow joint is composed of a hinge followed by a small of twist
around the forearm. The shoulder of the human model and the robot were assumed
to be aligned initially (see section II). The notations are used by the
description of our non-iterative solver below.

##### Computing arm axes

ONIA begins by computing the _arm axes_ of both the human model and the robot,
which are vector representations of the upper and forearm segments. Let
$\bm{x}_{\textit{s}}^{h},\bm{x}_{\textit{e}}^{h}$ and
$\bm{x}_{\textit{w}}^{h}$ be the positions of the human model’s shoulder
joint, and let $\bm{x}_{\textit{e}}^{r}$ and $\bm{x}_{\textit{w}}^{r}$ be the
reference elbow and wrist positions of the robot. The elbow axes are computed
as

$\displaystyle\bm{d}_{\textit{u}}^{h}$
$\displaystyle=\bm{x}_{\textit{e}}^{h}-\bm{x}_{\textit{s}}^{h},\qquad\bm{d}_{\textit{f}}^{h}=\bm{x}_{\textit{w}}^{h}-\bm{x}_{\textit{e}}^{h},$
(6) $\displaystyle\bm{d}_{\textit{u}}^{r}$
$\displaystyle=\bm{x}_{\textit{e}}^{r}-\bm{x}_{\textit{s}}^{h},\qquad\bm{d}_{\textit{f}}^{r}=\bm{x}_{\textit{w}}^{r}-\bm{x}_{\textit{e}}^{r}.$
(7)

Note $\bm{d}_{\textit{u}}^{r}$ starts at $\bm{x}_{\textit{s}}^{h}$ instead of
$\bm{x}_{\textit{s}}^{r}$ (see Fig. 4). Because the human model’s shoulder
width differs from the robot’s shoulder width. The human model’s and robot’s
shoulder joints cannot be perfectly aligned at the same time. Hence, we use
$\bm{x}_{\textit{s}}^{h}$ as the basis for guiding rotation of the upper arm.

##### Aligning direction of the upper arm

The direction of the upper arm is aligned with the reference axis
$\bm{d}_{\textit{u}}^{r}$ by swinging it around the shoulder,

$\displaystyle
R_{\textit{s}}^{\text{swing}}={\mathcal{R}}_{\bm{d}_{\textit{u}}^{r}}^{\bm{d}_{\textit{u}}^{h}}.$
(8)

##### Aligning scale of the upper arm

We further stretch(shrink) the upper arm segment so that its elbow joint
reaches the reference position $\bm{x}_{\textit{e}}^{r}$. This is done by
computing the scaling factor $S_{\textit{u}}$,

$S_{\textit{u}}=\frac{\|\bm{d}_{\textit{u}}^{r}\|}{\left\|\bm{d}_{\textit{u}}^{h}\right\|},$
(9)

such that

$\bm{d}_{\textit{u}}^{r}=R_{\textit{s}}^{\text{swing}}(S_{\textit{u}}\bm{d}_{\textit{u}}^{h}).$
(10)

##### Aligning the elbow axis

Further we twist the the upper arm so that its _elbow axis_ aligns with the
_reference elbow axis_ of the robot. The elbow axis of a human model’s arm is
the axis around which the elbow rotates, if treated as a hinge joint. We
annotate the elbow axis when prepossessing the model and denote it as
$\bm{a}^{h}$. The reference elbow axis of the robot is the cross product of
its reference arm axes,

$\displaystyle\bm{a}^{r}=\bm{d}_{\textit{u}}^{r}\times\bm{d}_{\textit{f}}^{r}.$
(11)

Consequently, the twist component of the human arm’s shoulder joint is

$\displaystyle R_{\textit{s}}^{\text{twist}}$
$\displaystyle={\mathcal{R}}_{\bm{a}_{\perp}^{h}}^{\bm{a}^{r}},$ (12)
$\displaystyle\text{where}\ \bm{a}_{\perp}^{h}$
$\displaystyle=\left(R_{\textit{s}}^{\text{swing}}\bm{a}^{h}\right)_{\perp\,\bm{d}_{\textit{u}}^{r}}.$
(13)

Essentially, we rotate the swung upper arm (and hence its elbow axis) around
itself so the projection of the elbow axis onto the plane normal to the axis
of rotation, $\bm{d}_{\textit{u}}^{r}$, coincides with the reference elbow
axis $\bm{a}^{r}$. This minimizes the angle between the human model’s elbow
axis and the reference elbow axis: in general, given vectors
$\bm{u},\bm{v},\bm{k}\in\mathbb{R}^{3}$, where $\bm{v}$ can rotate around
$\bm{k}$. The angle between $\bm{v}$ and $\bm{u}$ is minimized when
$\left(\bm{v}\right)_{\perp\,\bm{k}}$ and
$\left(\bm{u}\right)_{\perp\,\bm{k}}$ meet.

The overall rotation of the shoulder joint is the composition of the swing
followed by the twist,

$R_{\textit{s}}=R_{\textit{s}}^{\text{twist}}R_{\textit{s}}^{\text{swing}}.$
(14)

We are now done with the shoulder joint and the upper arm and now move on to
the elbow joint and the forearm.

##### Aligning the direction of the forearm

Similar to (8), we align the forearm — now
$R_{\textit{s}}\bm{d}_{\textit{f}}^{h}$ after rotating the shoulder — by
swinging it around the elbow so that it lines up with the reference forearm
axis,

$R_{\textit{e}}^{\text{swing}}={\mathcal{R}}_{R_{\textit{s}}\bm{d}_{\textit{f}}^{h}}^{\bm{d}_{\textit{f}}^{r}}R_{\textit{s}}.$
(15)

##### Aligning the scale of the forearm

We further stretch (shrink) the forearm, similar to (9), so that wrist joint
of the human model reaches the reference wrist position. The scaling factor of
the human model’s forearm is computed as

$S_{\textit{f}}=\frac{\|\bm{d}_{\textit{f}}^{r}\|}{\|\bm{d}_{\textit{f}}^{h}\|}.$
(16)

##### Aligning the wrist

We set the rotation of the wrist $R_{\textit{w}}$ to be the same as the end
effector rotation of the robot.

##### Reacting to wrist twist

Finally, we twist the forearm slightly around itself proportional to the twist
of the wrist joint. This leads to more realistic rendering by enabling the
forearm skin near the elbow to move in response to the wrist twist. We develop
the notion of _wrist twist_ through the following insight: Suppose in the
default configuration of the human model, there is some “wrist vector”
$\bm{w}$ that is roughly perpendicular to the palm. Then, after the wrist
joint rotates by $R_{\textit{w}}$, the twist component of $R_{\textit{w}}$ is
the rotation around the forearm axis that minimizes the angle between $\bm{w}$
and $R_{\textit{w}}\bm{w}$. Similar to (12), this is the rotation that brings
the projection of the two vectors together. With this insight, we formulate
the proportional twist of the elbow joint as follows,

$\displaystyle R_{\textit{e}}^{\text{twist}}$
$\displaystyle=\left({\mathcal{R}}_{\bm{w}_{\perp}^{h}}^{\bm{w}_{\perp}^{r}}\right)^{\alpha_{\textit{e}}}$
(17) $\displaystyle\text{where}\ \bm{w}_{\perp}^{h}$
$\displaystyle=\left(R_{\textit{e}}^{\text{swing}}\bm{w}\right)_{\perp\,\bm{d}_{\textit{f}}^{r}},\bm{w}_{\perp}^{r}=\left(R_{\textit{w}}\bm{w}\right)_{\perp\,\bm{d}_{\textit{f}}^{r}}.$
(18)

We set $\alpha_{e}=0.4$ empirically. Together, the overall rotation of the
elbow joint is

$R_{\textit{e}}=R_{\textit{e}}^{\text{twist}}R_{\textit{e}}^{\text{swing}},$
(19)

This concludes ONIA. Note that in the above procedure, ONIA starts with the
default configuration of the human arm and is not dependent on the current
poses. Being non-iterative makes the algorithm easy to analyze and ensures a
one-to-one, deterministic mapping from the robot poses to the human model
poses.

ONIA produces optimal alignment in two regards. First, the swinging ((8) and
(15)) and scaling ((9) and (16)) ensures that the elbow and wrist joints of
the human model reaches their reference positions on the robot _exactly_. In
addition, the twist in (12) _minimizes_ the angle between the human model’s
elbow axis and the robot’s elbow axis.

Figure 5: We evaluated the performance of the solvers on a set of 12 poses. We
chose these poses because they are representative of the space of possible arm
poses. Pose 1 to 3 all cover “hands down” but differ in forearm orientations.
Pose 4 to 6 likewise covers “hands up”. Pose 8 to 11 covers “arms in front,”
and pose 7 and 12 are T and A-poses. Note that for the unnatural “hands down”
poses, the Jacobian and FABRIK choose natural-looking approximations while
ONIA achieves best alignment. FABRIK was trapped in a singularity for pose 3.
For other poses, all solvers achieve similar alignment, with ONIA being the
fastest. See Fig. 6 for the metrics evaluated at each pose for every solver.
Detailed descriptions and results for each pose and each solver are available
on our online appendix: https://avatar-robots.github.io/poses

##  IV EVALUATION

###  IV-A Metrics

We define the following metrics in order to evaluate the performance of the
solvers in our Avatar Robot System:

  1. 1.

Overlay ratio of a solver at a certain frame is defined as

$O\coloneq\frac{\text{area of robot arms overlaid by human model}}{\text{total
area of robot arms}},$ (20)

where “area” is computed from the patient’s perspective, i.e. the area of
projection into patient’s eye view. This is an intuitive metric of visual
alignment: a value of 1 would place all robot arms behind the human model and
only the human model would be visible. We compute the overlay ratio by
replaying sessions in simulations with a special shader (see Fig. 7).

  2. 2.

Elbow (wrist) deviation: The elbow deviation $\Delta x_{\textit{e}}$ and the
wrist deviation $\Delta x_{\textit{w}}$ are defined as the distances of the
human model’s elbow and wrist joints from their corresponding reference
positions of the robot arm,

$\displaystyle\Delta x_{\textit{e}}$
$\displaystyle\coloneq\|\bm{x}_{\textit{s}}^{h}+R_{\textit{s}}(S_{\textit{u}}\bm{d}_{\textit{u}}^{h})-\bm{x}_{\textit{e}}^{r}\|,$
(21) $\displaystyle\Delta x_{\textit{w}}$
$\displaystyle\coloneq\|\bm{x}_{\textit{s}}^{h}+R_{\textit{s}}(S_{\textit{u}}\bm{d}_{\textit{u}}^{h})+R_{\textit{e}}(S_{\textit{f}}\bm{d}_{\textit{f}}^{h})-\bm{x}_{\textit{w}}^{r}\|.$
(22)

Smaller deviation should imply better alignment. ONIA is optimal in a sense by
ensuring $\Delta x_{\textit{e}}=\Delta x_{\textit{w}}=0$, but these metrics
still help compare other solvers.

  3. 3.

Upper & forearm shrink/stretch: The scaling factors of the human model’s arm
segments are $S_{\textit{u}}$ (9) and $S_{\textit{f}}$ (16). Scaling less
aggressively keeps the human model more anthropomorphic. In our plots we
present them by their absolute differences to 1 (i.e. $|S_{\textit{u}}-1|$ and
$|S_{\textit{f}}-1|$), the smaller the difference, the better.

  4. 4.

Computation time per frame: The average time a solver spent on computing
alignment pose per frame. Admittedly, the solvers are unlikely to be the
performance bottleneck of the entire system since rendering tend to be more
resource-intensive. Still, this metric reflects the complexity of different
solvers and a faster, simpler one should be preferred if multiple solvers
align the human model equally well.

We compute these metrics by logging the robot’s position, arm poses, and the
patient’s perspective (camera transform) of a session and replaying them later
in simulation. This replay-based approach allows us to compare multiple
solvers controlling for the robot trajectory and patient movement. We evaluate
our solvers with these metrics in two ways.

###  IV-B Evaluating solvers on a set of static poses

Figure 6: Violin plots of performance metrics of solvers evaluated from 12
poses. The poses are presented in Fig. 5 and marker colors here matches those
in Fig. 5. The minimums, maximums and medians are marked with black bars. We
see that ONIA is the fastest of all, achieves the best overlay ratio, and
scales the upper arm most conservatively and most consistently. Figure 7: The
key frames of the emulated therapy session. The first row shows the session as
seen from an external camera; The second row shows the recording of the AR
device the user is wearing, which, due to limitation of the recording
software, only captures a small central portion of actual user view; The third
row shows the replay scene of the session with ONIA, rendered with a special
shader, where green represents parts of the robot arm covered by the human
model, red represents uncovered area of the robot arm, and blue represents the
rest of the human model. The overlay ratio is then the green area over the red
and green area combined. See Fig. 8 for the quantitative analysis. The full
video of the session with detailed results can be found in our online
appendix: https://avatar-robots.github.io/sessions/emulated-therapy-session
Figure 8: Metrics of all solvers evaluated throughout the therapy session.
ONIA has the highest overlay ratio in average and runs the fastest. It scales
the upper arm similar to FABRIK but scales the forearm more aggressively,
consistent with section IV-B. FABRIK exhibits poor alignment metrics for
initially, and we find Jacobian to be lagging behind robot movements even with
the smallest damping necessary to keep it stable.

We evaluated the performance of the three solvers through a set of 12 static
poses, as shown in Fig. 5. Poses are chosen in group, with each group
representative of one partition of the space of all possible robot arm poses.
Poses 1 to 3 covers the general case of “hands down”, and vary in whether the
hand points forward, upright, or backward; Likewise, poses 4 to 6 covers the
case of “hands up.”; Poses 8 to 11 covers different cases where the arms are
in front of the body; Poses 7 and 12 are the standard T-pose and A-pose.

We compute metrics for every pose and each solver. The overlay ratio was
evaluated from looking at the robot and human model from the front (as shown
in Fig. 5). We present the results in Fig. 6.

We noticed that even though computational complexity varies greatly, all three
solvers achieve similar alignment. Jacobian and FABRIK has similar elbow and
wrist deviations. Alignment was poor for both baselines on poses 1 to 3 (the
“hands down”) poses, because these poses are unnatural for human and lie on
the boundary of human anatomical constraints. Incorporating these constraints,
both baseline solvers produces only natural-looking approximation to the given
robot arm pose. Whether a user prefers visually aligned or natural-looking
pose is the subject of future user studies. Still, even when _excluding the 3
outliers_ , ONIA achieves the highest median overlay ratio (hence best visual
alignment) 222Median overlay ratios for 3 solvers excluding poses 1-3 are not
shown in Fig. 6. They are .798, .773, and .783 for ONIA, Jacobian and FABRIK
respectively.. Even without explicit scaling constraints, ONIA solver scales
the upper arm less aggressively and more consistently (low variance) than both
baselines. As of forearm, ONIA tend to scale more aggressively while Jacobian
has the most consistent scaling. FABRIK was trapped at a singularity for pose
12, as shown in Fig. 5, with the arm straight and the back of the elbow facing
upwards. In general, we find that FABRIK is more likely to suffer from such
singularities than Jacobian. Being non-iterative, ONIA runs the fastest,
followed by Jacobian and FABRIK which are each an order of magnitude more
computationally expensive then one another.

###  IV-C Evaluating the Avatar Robot System sthrough an emulated physical
therapy session

We further evaluated our solvers through an emulated therapy session:
‘shoulder flexion with elbow extension therapy’ 333The full video of the
emulated therapy session can be found in our online appendix: https://avatar-
robots.github.io/sessions/emulated-therapy-session/ In the emulated therapy,
we program the robot to follow a trajectory that guides the user’s forearm to
rotate around his elbow joint. The key frames of the session are shown in Fig.
7, and the metrics evaluated at every frame for every solver are presented in
Fig. 8.

We noticed that ONIA has the best average overlay ratio throughout the session
and runs the fastest. It stretches the upper arm of the model similar to
FABRIK but stretches the forearm more aggressively. These are all consistent
with our previous findings from in section IV-B. Jacobian lagged behind
changes in robot pose, as shown in its spikes of elbow/wrist deviation when
the robot is moving the patient’s arm. This is caused by the damping factor of
DLS. Reducing the damping further than the current value may improve the
responsiveness of Jacobian but causes instability. The damping also causes
Jacobian to scale the arm segments differently than the other two solvers did.
The left-arm FABRIK solver was caught in a singularity similar to the one it
encountered in the static pose evaluation. This caused very poor alignment of
the left arm which was reflected in abnormal overlay ratio, elbow/wrist
deviations, and upper arm stretching. FABRIK recovered from this singularity
on its own in the middle of the session and achieved decent alignment for the
second half.

##  V LIMITATIONS

We list below the limitations of our approach , which can provide insights to
future investigations:

  1. 1.

The solvers consider the alignment of each arm separately. Future alignment
solvers could include dual arm poses with the torso to improve the alignment.

  2. 2.

ONIA solver does not limit the scaling of arm segments. While evaluations show
that it produces scaling comparable to the baselines in most cases, excessive
scaling as high as 1.5 is still observed in pose 8 of Fig. 5. It would be
ideal if a scaling limit could be incorporated into future ONIA versions
without significantly compromising its theoretical guarantees.

  3. 3.

For this research, we only considered the alignment of arms between the human
model and robot (shoulder, elbow, and wrist joints). The alignment of the hand
gesture (including the fingers) to the robot’s end effector needs to be
explored in the future;

  4. 4.

Our evaluation contains objective metrics only. While this allows precise
quantitative analysis of solvers, we expect to benefit from follow-up user
studies that can provide new insights about patient perceptions. Subjective
evaluations will complement our existing metrics and provide a more
comprehensive evaluation.

##  VI CONCLUSION

We presented an Avatar Robot System as a mixed real/virtual robotic system
that aligns the 3D human model (remote operator) to the robot structure.
Therefore, the care receiver might feel as though they are in close contact
with the remote operator. We proposed a non-iterative alignment solver that
optimally aligns arms poses of the human model with robot joints. We also
proposed a set of evaluation metrics that quantitatively measures alignment
quality. Our evaluation shows that our non-iterative solver consistently
achieves alignment quality comparable to the baselines, meanwhile avoiding
their limitations such as singularities, instability, and hyper-parameter
sensitivity.

## References

  * [1] A. Helal and B. Abdulrazak, “Tecarob: Tele-care using telepresence and robotic technology for assisting people with special needs,” _International Journal of ARM_ , vol. 7, no. 3, pp. 46–53, 2006.
  * [2] S. Koceski and N. Koceska, “Evaluation of an assistive telepresence robot for elderly healthcare,” _Journal of medical systems_ , 2016.
  * [3] P. D. Thacker, “Physician-robot makes the rounds,” _JAMA_ , vol. 293, no. 2, pp. 150–150, 2005.
  * [4] F. Bernardoni, Ö. Özen, K. Buetler, and L. Marchal-Crespo, “Virtual reality environments and haptic strategies to enhance implicit learning and motivation in robot-assisted training,” in _2019 IEEE 16th international conference on rehabilitation robotics (ICORR)_. IEEE, 2019.
  * [5] B. R. Ballester, L. S. Oliva, A. Duff, and P. Verschure, “Accelerating motor adaptation by virtual reality based modulation of error memories,” in _2015 IEEE International Conference on Rehabilitation Robotics (ICORR)_. IEEE, 2015, pp. 623–629.
  * [6] K. Brütsch, A. Koenig, L. Zimmerli, S. Mérillat-Koeneke, R. Riener, L. Jäncke, H. J. van Hedel, and A. Meyer-Heim, “Virtual reality for enhancement of robot-assisted gait training in children with central gait disorders,” _Journal of rehabilitation medicine_ , 2011.
  * [7] R. Agarwal, A. W. Levinson, M. Allaf, D. V. Makarov, A. Nason, and L.-M. Su, “The roboconsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface,” _Urology_ , 2007.
  * [8] B. Jones, Y. Zhang, P. N. Wong, and S. Rintel, “Vroom: virtual robot overlay for online meetings,” in _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–10.
  * [9] C. Neustaedter, G. Venolia, J. Procyk, and D. Hawkins, “To beam or not to beam: A study of remote telepresence attendance at an academic conference,” in _Proceedings of the 19th acm conference on computer-supported cooperative work & social computing_, 2016, pp. 418–431.
  * [10] Z. Erickson, Y. Gu, and C. C. Kemp, “Assistive vr gym: Interactions with real people to improve virtual assistive robots,” in _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2020, pp. 299–306.
  * [11] C. Lenz and S. Behnke, “Bimanual telemanipulation with force and haptic feedback through an anthropomorphic avatar system,” _Robotics and Autonomous Systems_ , 2023.
  * [12] R. Luo and e. a. Wang, “Towards robot avatars: Systems and methods for teleinteraction at avatar xprize semi-finals,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_.
  * [13] “Avatar xprize.” [Online]. Available: https://www.xprize.org/prizes/avatar/articles/avatar-technology-this-is-just-the-beginning
  * [14] S. R. Buss, “Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods,” _IEEE Journal of Robotics and Automation_ , vol. 17, no. 1-19, p. 16, 2004.
  * [15] A. Aristidou and J. Lasenby, “FABRIK: A fast, iterative solver for the inverse kinematics problem,” _Graphical Models_ , 2011.
  * [16] “ABB’s collaborative robot — YuMi.” [Online]. Available: https://new.abb.com/products/robotics/robots/collaborative-robots/yumi/irb-14000-yumi
  * [17] “HoloLens 2 — Overview, Features, and Specs - Microsoft.” [Online]. Available: https://www.microsoft.com/en-us/hololens/hardware
  * [18] P. Baerlocher and R. Boulic, “Parametrization and range of motion of the ball-and-socket joint,” in _Deformable Avatars: IFIP TC5/WG5. 10 DEFORM’2000 Workshop Geneva, Switzerland_. Springer, 2001.
  * [19] F. S. Grassia, “Practical parameterization of rotations using the exponential map,” _Journal of graphics tools_ , vol. 3, no. 3, pp. 29–48, 1998.