11institutetext: School of Computer Science and Engineering, Sun Yat-sen
University, China 22institutetext: Key Laboratory of Machine Intelligence and
Advanced Computing, MOE, China 22email<EMAIL_ADDRESS>33institutetext: The First Affiliated Hospital, Sun Yat-sen University,
Guangzhou, China

# Task-oriented Self-supervised Learning for Anomaly Detection in
Electroencephalography

Yaojia Zheng 1122 Zhouwu Liu 1122 Rong Mo 33 Ziyi Chen 33 Wei-shi Zheng 1122
Ruixuan Wang(✉) 1122

###### Abstract

Accurate automated analysis of electroencephalography (EEG) would largely help
clinicians effectively monitor and diagnose patients with various brain
diseases. Compared to supervised learning with labelled disease EEG data which
can train a model to analyze specific diseases but would fail to monitor
previously unseen statuses, anomaly detection based on only normal EEGs can
detect any potential anomaly in new EEGs. Different from existing anomaly
detection strategies which do not consider any property of unavailable
abnormal data during model development, a task-oriented self-supervised
learning approach is proposed here which makes use of available normal EEGs
and expert knowledge about abnormal EEGs to train a more effective feature
extractor for the subsequent development of anomaly detector. In addition, a
specific two-branch convolutional neural network with larger kernels is
designed as the feature extractor such that it can more easily extract both
larger-scale and small-scale features which often appear in unavailable
abnormal EEGs. The effectively designed and trained feature extractor has
shown to be able to extract better feature representations from EEGs for
development of anomaly detector based on normal data and future anomaly
detection for new EEGs, as demonstrated on three EEG datasets. The code is
available at https://github.com/ironing/EEG-AD.

###### Keywords:

Anomaly detection Self-supervised learning EEG.

##  1 Introduction

Electroencephalography (EEG) is one type of brain imaging technique and has
been widely used to monitor and diagnose brain status of patients with various
brain diseases (e.g., epilepsy) [12, 11, 16, 10, 2]. EEG data typically
consists of multiple sequences (or channels) of waveform signals, with each
sequence obtained by densely sampling electrical signals of brain activities
from a unique electrode attached to a specific position on patient’s head
surface. While brain activities can be recorded by EEG equipment over hours or
even days, clinicians often analyze EEG data at the level of seconds,
considering that cycles of most brain activities varies between 0.5Hz and
30Hz. Therefore, it is very tedious to manually analyze long-term EEG data and
automated analysis of EEG would largely alleviate clinician efforts in timely
monitoring patient statuses.

Currently, most automated analyses of EEGs focus on specific diseases [17, 8,
1], where labelled EEGs at the onset of disease and normal (healthy) status
are collected to train a classifier for prediction of patient status at the
level of seconds. However, such automated systems can only help analyze
specific diseases and would fail to recognize novel unhealthy statuses which
do not appear during classifier training. In contrast, developing an anomaly
detector based on only normal EEGs has the potential to detect any possible
unhealthy status (i.e., anomaly) in new EEG data. While multiple anomaly
detection strategies have been developed for both natural and medical image
analyses [22, 24], including statistical approaches [13, 18], discriminative
approaches [4, 25] reconstruction approaches [27, 19] and self-supervised
learning approaches [5, 15], very limited studies have been investigated on
anomaly detection based on normal EEGs only [26]. Furthermore, all these
strategies build anomaly detectors without considering any property of anomaly
due to absence of abnormal data during model development. One exception is the
recently proposed CutPaste method for anomaly detection in natural images
[14], where simulated abnormal images were generated by cutting and pasting
small image patches in normal images and then used to help train a more
effective feature extractor and anomaly detector.

Inspired by the CutPaste method, we propose a novel task-oriented self-
supervised learning approach to train an effective feature extractor based on
normal EEG data and expert knowledge (key properties including increased
amplitude and unusual frequencies) about unavailable abnormal EEGs. In
addition, a specific two-branch convolutional neural network (CNN) with larger
kernels is designed for effective extraction of both small-scale and large-
scale features, such that the CNN feature extractor can be trained to extract
features of both normal and abnormal EEGs. The feature extractor with more
powerful representation ability can help establish a better anomaly detector.
State-of-the-art anomaly detection performance was obtained on one internal
and two public EEG datasets, confirming the effectiveness of the proposed
approach.

##  2 Methodology

In this study, we try to solve the problem of anomaly detection in EEGs when
only normal EEG data is available for training. A two-stage framework is
proposed here, where the first stage aims to a train a feature extractor using
a novel self-supervised learning method, and the second stage can adopt any
existing generative or discriminative method to establish an anomaly detector
based on the feature representations from the well-trained feature extractor.

###  2.1 Task-oriented self-supervised learning

Various self-supervised learning (SSL) strategies have been proposed to train
feature extractors for downstream tasks in both natural and medical image
analysis. To make a feature extractor more generalizable for downstream tasks,
existing SSL strategies are often designed purposely without regard to any
specific downstream task. That means, SSL strategies often do not consider
characteristics in specific downstream tasks. However, when applying any such
SSL technique to an anomaly detection task, the feature extractor based on
only normal data would less likely learn to extract features of abnormal data,
and therefore may negatively affect model performance in the subsequent
anomaly detection task.

Different from most SSL strategies, a novel SSL strategy is proposed here to
train a feature extractor which can extract features of both normal and
abnormal EEG data. Specifically, considering that abnormal EEGs are
characterized by increased wave amplitude or temporally slowed or abrupt wave
signals, two special transformations are designed to generate simulated
abnormal EEG data. One transformation is to temporally locally increase
amplitude of normal EEG signals (Figure 1, Middle), and the other
transformation is to temporally increase or decrease the frequency of normal
EEG signals (Figure 1, Right). These amplitude-abnormal and frequency-abnormal
data, together with original normal EEGs, form a 3-class dataset for the
training of a 3-class CNN classifier. The feature extractor part of the well-
trained classifier would be expected to learn to extract features of both
normal and (simulated) abnormal EEGs. While the simulated abnormal data are
different from real abnormal EEGs, empirical evaluations show the feature
extractor trained with simulated abnormal EEGs helps improve the performance
of anomaly detection significantly.

Figure 1:  Transformation of normal EEG (Left) to generate simulated
amplitude-abnormal (Middle) and frequency-abnormal (Right) EEGs. One EEG data
consists of multiple sequences (channels, y-axis) of wave signals. x-axis:
time (seconds).

####  2.1.1 Generation of self-labeled abnormal EEG data:

Denote a normal EEG data by a matrix $\mathbf{X}\in\mathbb{R}^{K\times L}$,
where $K$ represents the number of channels (sequences) and $L$ represents the
length of each sequence. To generate an amplitude-abnormal EEG data based on
$\mathbf{X}$, a scalar amplitude factor $\alpha$ is firstly sampled from a
predefined range $[\alpha_{l},\alpha_{h}]$ where $1<\alpha_{l}<\alpha_{h}$,
followed by sampling a segment length $w$ from a predefined sequence segment
range $[w_{l},w_{h}]$ where $1<w_{l}<w_{h}<L$. Then $w$ consecutive columns in
$\mathbf{X}$ were randomly chosen, with each element in these columns
multiplied by the amplitude factor $\alpha$. Such transformed EEG data with
modified $w$ consecutive columns can be used as a simulated amplitude-abnormal
EEG data (Figure 1, Middle). On the other hand, to generate a lower-frequency
abnormal EEG data based on $\mathbf{X}$, a frequency scalar factor $\omega$ is
firstly randomly sampled from a predefined range $[\omega_{l},\omega_{h}]$
where $1<\omega_{l}<\omega_{h}$, and then each sequence (row) of signals in
$\mathbf{X}$ is linearly interpolated (i.e., upsampled) by the factor $\omega$
to generate an elongated EEG data $\mathbf{X}^{\prime}\in\mathbb{R}^{K\times
L^{\prime}}$, where $L^{\prime}=\lfloor\omega L\rfloor$ is the largest number
which is equal to or smaller than $\omega L$. One frequency-abnormal EEG data
can be generated by randomly choosing $L$ consecutive columns from
$\mathbf{X}^{\prime}$. Similarly, to alternatively generate a higher-frequency
abnormal EEG data from $\mathbf{X}$, the frequency scalar factor
$\omega^{\prime}$ is firstly randomly sampled from another predefined range
$[\omega^{\prime}_{l},\omega^{\prime}_{h}]$ where
$0<\omega^{\prime}_{l}<\omega^{\prime}_{h}<1$, and then each row of signals in
$\mathbf{X}$ is down-sampled by the factor $\omega^{\prime}$ to generate a
shortened EEG data. The shortened EEG data is then concatenated by itself
multiple times along the temporal (i.e., row) direction to obtain a temporary
EEG data $\mathbf{X}^{\prime\prime}\in\mathbb{R}^{K\times L^{\prime\prime}}$
where
$L^{\prime\prime}=\lfloor\omega^{\prime}L\rfloor\cdot\lceil\frac{1}{\omega^{\prime}}\rceil\geq
L$. One higher-frequency abnormal EEG data can be obtained by randomly
choosing $L$ consecutive columns from $\mathbf{X}^{\prime\prime}$ (when
$L^{\prime\prime}>L$) or just be $\mathbf{X}^{\prime\prime}$ (when
$L^{\prime\prime}=L$).

Using these simple transformations and based on multiple normal EEG data, two
classes of self-labeled abnormal data will be generated, with one class
representing anomaly in amplitude, and the other representing anomaly in
frequency. Although more complex transformations can be designed to generate
more realistic amplitude- and frequency-abnormal EEGs, empirical evaluations
show the simple transformations are sufficient to help train an anomaly
detector for EEGs.

Figure 2: The two-branch architecture for feature extractor training. The
second branch (lower part) consists of only one convolutional layer for
extraction of small-scale features. The part in the blue dotted box is the
feature extractor.

####  2.1.2 Architecture design for feature extractor:

To train a feature extractor with more powerful representation ability, we
designed a specific CNN classifier based on the ResNet34 backbone (Figure 2).
Considering that two adjacent channels (rows) in the EEG data do not indicate
spatial proximity between two brain regions, one-dimensional (1D)
convolutional kernels are adopted to learn to extract features from each
channel over convolutional layers as in previous studies [9]. However,
different from the previously proposed 1D kernels of size $1\times 3$, kernels
of longer size in time (i.e., $1\times 7$ here) is adopted in this study. Such
longer kernels are used considering that lower-frequency features may last for
a a longer period (i.e., longer sequence segment) in EEGs and therefore would
not be well captured by shorter kernels even over multiple convolutional
layers. On the other hand, considering that some other anomalies in EEGs may
last for very short time and therefore such abnormal features may be omitted
after multiple times of pooling or down-sampling over layers, we propose
adding a shortcut branch from the output of the first convolutional layer to
the penultimate layer. Specifically, the shortcut branch consists of only one
convolutional layer in which each kernel is two-dimensional (i.e., number of
EEG channels $\times$ 7) in order to capture potential correlation across all
the channels in short time interval. By combining outputs from the two
branches, both small-scale and large-scale features in time would be captured.
The concatenated features are fed to the last fully connected layer for
prediction of EEG category. The output of the classifier consists of three
values, representing the prediction probability of ‘normal EEG’, ‘amplitude-
abnormal EEG’, and ‘frequency-abnormal EEG’ class, respectively.

The feature extractor plus the 3-class classifier head can be trained by
minimizing the cross-entropy loss on the 3-class training set. After training,
the classifier head is removed and the feature extractor is used to extract
features from normal EEGs for the development of anomaly detector. Since the
feature extractor is trained without using real abnormal EEGs and the
simulated abnormal EEGs are transformed from normal EEGs and self-labelled,
the training is an SSL process. The SSL is task-oriented because it considers
the characteristics (i.e., crucial anomaly properties in both amplitude and
frequency of abnormal EEGs) in the subsequent specific anomaly detection task.
The feature extractor based on such task-oriented SSL is expected to be able
to extract features of both normal and abnormal EEGs for more accurate anomaly
detection.

###  2.2 Anomaly detection

While in principle any existing anomaly detection strategy can be applied
based on feature representations of normal EEGs from the well-trained feature
extractor, here the generative approach is adopted to demonstrate the
effectiveness of the proposed two-stage framework considering that a large
amount of normal EEG data are available to estimate the distribution of normal
EEGs in the feature space. Note discriminative approaches like one-class SVM
may be a better choice if normal data is limited. As one simple generative
approach, multivariate Gaussian distribution
$\mathcal{G}(\mbox{\boldmath$\mu$},\mathbf{\Sigma})$ is used here to represent
the distribution of normal EEGs, where the mean $\mu$ and the covariance
matrix $\mathbf{\Sigma}$ are directly estimated from the feature vectors of
all normal EEGs in the training set, with each vector being the output of the
feature extractor given a normal EEG input.

With the Gaussian model $\mathcal{G}(\mbox{\boldmath$\mu$},\mathbf{\Sigma})$,
the degree of abnormality for any new EEG data $\mathbf{z}$ can be estimated
based on the Mahalanobis distance between the mean $\mu$ and the feature
representation $f(\mathbf{z})$ of the new data $\mathbf{z}$, i.e.,

$A(\mathbf{z})=\sqrt{(f(\mathbf{z})-\mbox{\boldmath$\mu$})^{\top}\Sigma^{-1}(f(\mathbf{z})-\mbox{\boldmath$\mu$})}\,.$
(1)

Larger $A(\mathbf{z})$ score indicates that $\mathbf{z}$ is more likely
abnormal, and vice versa.

##  3 Experiments and results

Table 1: The statistics of three EEG datasets.

Dataset |  Sampling rate |  Channels |  Patients |  |  Normal EEGs
---
(seconds)
|  Abnormal EEGs
---
(seconds)
Internal | 1024 | 19 | 50 | 30008 | 14402
CHB-MIT | 256 | 18 | 21 | 70307 | 11506
UPMC | 400 | 16 | 4 | 9116 | 1087

Experimental settings: Three EEG datasets were used to evaluate the proposed
approach, including the public Children’s Hospital Boston–Massachusetts
Institute of Technology Database (‘CHB–MIT’) [21] and the UPenn and Mayo
Clinic’s Seizure Detection Challenge dataset (‘UPMC’) [23], and an internal
dataset from a national hospital (‘Internal’). See Table 1 for details. While
only the ictal stage of seizure is included as anomaly in UPMC, both ictal and
various interictal epileptiform discharges (IEDs) were included in Internal
and CHB-MIT. Particularly, the abnormal waveforms in Internal include
triphasic waves, spike-and-slow-wave complexes, sharp-and-slow-wave complex,
multiple spike-and-slow-wave complexes, multiple sharp-and-slow-wave complex
and ictal discharges. 2 out of 23 patients were removed from CHB–MIT due to
irregular channel naming and electrode positioning. For normal EEG recordings
lasting for more than one hour in CHB-MIT, only the first hour of normal EEG
recordings were included to partly balance data size across patients. In UPMC,
only dog data were used because of inconsistent recording locations across
human patients. For CHB–MIT and Internal, each original EEG recording was cut
into short segments of fixed length (3 seconds in experiments). For UPMC, one-
second short segments have been provided by the organizer. Each short segment
was considered as one EEG data during model development and evaluation.
Therefore the size of each EEG data is [number of channels, sampling rate
$\times$ segment duration]. For each dataset, signal amplitude in EEG data was
normalized to the range $[0,1]$ based on the minimum and maximum signal values
in the dataset.

On each dataset, while all abnormal EEG data were used for testing, normal EEG
data were split into training and test parts in two ways. One way (Setting I,
default choice) is to randomly choose the same number of normal EEGs as that
of abnormal EEGs for testing and the remaining is for training, without
considering patient identification of EEGs. The other (Setting II, subject
level) is to split patients with the cross-validation strategy, such that all
normal EEGs of one patient were used either for training or test at each round
of cross validation.

In training the 3-class classifier, for each batch of 64 normal EEGs,
correspondingly 64 amplitude-abnormal EEGs and 64 frequency-abnormal (32
higher-frequency and 32 lower-frequency) EEGs were generated (see Section
2.1.1). Adam optimizer with learning rate 0.0001 and weight decay coefficient
0.00003 were adopted, and training was consistently observed convergent within
300 epochs. $[\alpha_{l},\alpha_{h}]=[2.0,4.0]$, $[w_{l},w_{h}]=[4,L]$,
$[\omega_{l},\omega_{h}]=[2,4]$, and
$[\omega^{\prime}_{l},\omega^{\prime}_{h}]=[0.1,0.5]$. The ranges of amplitude
and frequency scalar factors were determined based on expert knowledge about
potential changes of abnormal brain wave compared to normal signals. The area
under ROC curve (AUC) and its average and standard deviation over five runs
(Setting I) or multiple rounds of validation (Setting II), the equal error
rate (EER), and F1-score (at EER) were reported.

Effectiveness evaluation: Our method was compared with well-known anomaly
detection methods including the one-class SVM (OC-SVM) [20], the statistical
kernel density estimation (KDE), and the autoencoder (AE) [7], the recently
proposed methods Multi-Scale Convolutional Recurrent Encoder-Decoder (MSCRED)
[28] and Unsupervised Anomaly Detection (USAD) [3] for multivariate time
series, and the recently proposed SSL methods for anomaly detection, including
ScaleNet [26] and CutPaste [14]. Note that ScaleNet [26] uses frequencies of
normal EEGs at multiple scales to help detect abnormal EEGs, without
considering any characteristics in abnormal EEGs. Similar efforts were taken
to tune relevant hyper-parameters for each method. In particular, to obtain
feature vector input for OC-SVM and KDE, every row in each EEG data was
reduced to a 64-dimensional vector by principal component analysis (PCA) based
on all the row vectors of all normal EEGs in each dataset, and then all the
dimension-reduced rows were concatenated as the feature representation of the
EEG data. For AE, the encoder consists of three convolutional layers and one
fully connected (FC) layer, and symmetrically the decoder consists of one FC
and three deconvolutional layers. For CutPaste, each normal EEG in each
training set is considered as a gray image of size $K\times L$ pixels, and the
suggested hyper-parameters from the original study [14] were adopted for model
training. For ScaleNet, the method was re-implemented with suggested hyper-
parameters [26]. As Table 2 shows, on all three datasets, our method (last
row) outperforms all the baselines by a large margin. Consistently, as Figure
3 (Left) demonstrates, our method performed best as well at the subject level
(i.e., Setting II), although the performance decreases a bit due to the more
challenging setting. All these results clearly confirm the effectiveness of
our method for anomaly detection in EEGs.

Table 2: Performance comparison on three datasets with Setting I. Bold face
indicates the best, and italic face for the second best. Standard deviations
are in brackets.

Method |  | Internal |  |  |  | CHB-MIT |  |  | UPMC
---|---|---|---|---|---|---|---|---|---
EER$\downarrow$ | F1$\uparrow$ | AUC$\uparrow$ |  | EER$\downarrow$ | F1$\uparrow$ | AUC$\uparrow$ |  | EER$\downarrow$ | F1$\uparrow$ | AUC$\uparrow$
OC-SVM | 0.30(0.008) | 0.71(0.008) | 0.75(0.02) |  | 0.33(0.001) | 0.69(0.001) | 0.73(0.003) |  | 0.33(0.01) | 0.51(0.02) | 0.74(0.01)
KDE | 0.24(0.001) | 0.76(0.001) | 0.87(0.001) |  | 0.32(0.002) | 0.69(0.002) | 0.75(0.003) |  | 0.24(0.003) | 0.70(0.004) | 0.83(0.003)
AE | 0.35(0.01) | 0.65(0.01) | 0.69(0.02) |  | 0.46(0.007) | 0.54(0.007) | 0.56(0.01) |  | 0.32(0.02) | 0.61(0.02) | 0.75(0.02)
MSCRED | 0.37(0.02) | 0.64(0.03) | 0.67(0.03) |  | 0.34(0.03) | 0.67(0.03) | 0.72(0.03) |  | 0.28(0.02) | 0.70(0.02) | 0.76(0.02)
USAD | 0.25(0.02) | 0.76(0.02) | 0.83(0.02) |  | 0.30(0.02) | 0.69(0.03) | 0.79(0.02) |  | 0.24(0.02) | 0.75(0.02) | 0.82(0.02)
ScaleNet | 0.18(0.01) | 0.82(0.01) | 0.90(0.01) |  | 0.26(0.03) | 0.73(0.03) | 0.81(0.03) |  | 0.20(0.03) | 0.75(0.03) | 0.89(0.03)
CutPaste | 0.26(0.02) | 0.74(0.02) | 0.83(0.03) |  | 0.26(0.01) | 0.74(0.01) | 0.83(0.01) |  | 0.21(0.01) | 0.74(0.01) | 0.87(0.006)
Ours | 0.11(0.01) | 0.89(0.01) | 0.95(0.004) |  | 0.16(0.01) | 0.84(0.02) | 0.92(0.02) |  | 0.13(0.007) | 0.83(0.01) | 0.95(0.006)

Figure 3: Performance comparison at the subject level (i.e., Setting II) on
each dataset (Left), and the ablation study of the feature extractor training
(Right). AUC values were included in figure for each method. Vertical line:
standard deviation.

Ablation studies: To specifically confirm the effectiveness of the proposed
task-oriented SSL strategy in training a feature extractor for EEG anomaly
detection, an ablation study is performed by replacing this SSL strategy with
several other SSL strategies, including (1) training an autoencoder and then
keeping the encoder part as the feature extractor, (2) training a ScaleNet and
then keeping its feature extractor part whose structure is the same as the
proposed one, and (3) contrastive learning of the proposed two-branch feature
extractor using the well-known SimCLR method [6]. As Figure 3 (Right) shows,
compared to these SSL strategies which do not consider any property of
anomalies in EEGs, our SSL strategy performs clearly better on all three
datasets.

Another ablation study was performed to specifically confirm the role of the
proposed two-branch backbone and self-labeled abnormal EEGs during feature
extractor training. From Table 3, it is clear that the inclusion of the two
types of simulated anomalies boosted the performance from AUC=0.71 to 0.93,
and additional inclusion of the second branch (‘Two-branch’) and change of
kernel size from $1\times 3$ to $1\times 7$ (‘Larger kernel’) further improved
the performance.

In addition, one more evaluation showed the proposed simple transformations
(based on simulated anomalies in frequency and amplitude individually)
performed better than the more complex transformations (based on combinations
of simulated abnormal frequency and amplitude), with AUC 0.954 vs. 0.901 on
dataset Internal, 0.924 vs. 0.792 on CHB-MIT, and 0.952 vs. 0.918 on UPMC.
Although simulated combinations of anomalies could be more realistic, they may
not cover all possible anomalies in real EEGs and so the feature extractor
could be trained to extract features of only the limited simulated complex
anomalies. In contrast, with simple transformations, the feature extractor is
trained to extract features which are discriminative enough between normal
EEGs and simulated abnormal EEGs based on only abnormal frequency or amplitude
features, therefore more powerful and effective in extracting discriminative
features.

Table 3: The effect of simulated anomaly classes and the two-branch
architecture on anomaly detection with ‘Internal’ dataset.

Amplitude-abnormal |  | ✓ |  | ✓ | ✓ | ✓ | ✓
---|---|---|---|---|---|---|---
Frequency-abnormal |  |  | ✓ | ✓ | ✓ | ✓ | ✓
Two-branch |  |  |  |  | ✓ |  | ✓
Larger kernel |  |  |  |  |  | ✓ | ✓
AUC | 0.71(0.04) | 0.91(0.001) | 0.89(0.01) | 0.93(0.01) | 0.94(0.006) | 0.94(0.005) | 0.95(0.004)

Sensitivity studies: The proposed task-oriented SSL strategy is largely
insensitive to the hyper-parameters for generating simulated abnormal EEGs.
For example, as shown in Figure 4, when respectively varying the range
$[\alpha_{l},\alpha_{h}]$ from $[2.0,3.0]$ to $[2.0,7.0]$, the range
$[w_{l},w_{h}]$ from $[4,L/5]$ to $[4,L]$, the range $[\omega_{l},\omega_{h}]$
from $[2,3]$ to $[2,7]$, and the range
$[\omega^{\prime}_{l},\omega^{\prime}_{h}]$ from $[0.1,0.4]$ to $[0.1,0.8]$,
the final anomaly performance changes in a relatively small range and all of
them are clearly better than the baselilne methods. These results support that
the proposed task-oriented SSL strategy is quite stable in improving anomaly
detection.

In addition, it is expected the proposed SSL strategy works stably even when
injecting incompatible transformations. For example, during training the
feature extractor, when a proportion ($5\%$, $10\%$, $15\%$, $20\%$) of
simulated abnormal EEGs were replaced by fake abnormal EEGs (each fake EEG
randomly selected from real normal EEGs but used as abnormal), the AUC is
respectively 0.927, 0.916, 0.898, and 0.883 on Internal, lower than original
0.954 but still kept at high level.

Figure 4: Sensitivity study of hyper-parameters.

##  4 Conclusion

In this study, we propose a two-stage framework for anomaly detection in EEGs
based on normal EEGs only. The proposed task-oriented self-supervised learning
together with the two-branch feature extractor from the first stage was shown
to be effective in helping improve the performance of the anomaly detector
learned at the second stage. This suggests that although only normal data are
available for anomaly detection in some scenarios, transformation of normal
data with embedded key properties of anomalies may generate simulated abnormal
data which can be used to greatly help develop a more effective anomaly
detector.

Acknowledgments. This work is supported by NSFCs (No. 62071502, U1811461), the
Guangdong Key Research and Development Program (No. 2020B1111190001), and the
Meizhou Science and Technology Program (No. 2019A0102005).

## References

  * [1] Achilles, F., Tombari, F., Belagiannis, V., Loesch, A.M., Noachtar, S., Navab, N.: Convolutional neural networks for real-time epileptic seizure detection. Computer methods in biomechanics and biomedical engineering. Imaging & visualization 6, 264–269 (2018)
  * [2] Alturki, F.A., AlSharabi, K., Abdurraqeeb, A.M., Aljalal, M.: EEG signal analysis for diagnosing neurological disorders using discrete wavelet transform and intelligent techniques. Sensors 20(9), 2505 (2020)
  * [3] Audibert, J., Michiardi, P., Guyard, F., Marti, S., Zuluaga, M.A.: Usad: Unsupervised anomaly detection on multivariate time series. In: KDD. pp. 3395–3404 (2020)
  * [4] Chalapathy, R., Menon, A.K., Chawla, S.: Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360 (2018)
  * [5] Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., Rueckert, D.: Self-supervised learning for medical image analysis using image context restoration. Medical Image Analysis 58, 101539–101539 (2019)
  * [6] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML. pp. 1597–1607 (2020)
  * [7] Chen, Z., Yeo, C.K., Lee, B.S., Lau, C.T.: Autoencoder-based network anomaly detection. In: WTS (2018)
  * [8] Craley, J., Johnson, E., Venkataraman, A.: A novel method for epileptic seizure detection using coupled hidden markov models. In: MICCAI (2018)
  * [9] Dai, G., Zhou, J., Huang, J., Wang, N.: Hs-cnn: a cnn with hybrid convolution scale for motor imagery classification. Journal of neural engineering 17, 016025 (2020)
  * [10] Dhar, P., Garg, V.K.: Brain-related diseases and role of electroencephalography (EEG) in diagnosing brain disorders. In: ICT Analysis and Applications, pp. 317–326 (2021)
  * [11] Fiest, K.M., Sauro, K.M., Wiebe, S., Patten, S.B., Kwon, C.S., Dykeman, J., Pringsheim, T., Lorenzetti, D.L., Jetté, N.: Prevalence and incidence of epilepsy: A systematic review and meta-analysis of international studies. Neurology 88(3), 296–303 (2017)
  * [12] Gemein, L.A., Schirrmeister, R.T., Chrabąszcz, P., Wilson, D., Boedecker, J., Schulze-Bonhage, A., Hutter, F., Ball, T.: Machine-learning-based diagnostics of EEG pathology. NeuroImage 220, 117021 (2020)
  * [13] Jia, W., Shukla, R.M., Sengupta, S.: Anomaly detection using supervised learning and multiple statistical methods. In: ICML (2019)
  * [14] Li, C.L., Sohn, K., Yoon, J., Pfister, T.: Cutpaste: Self-supervised learning for anomaly detection and localization. In: CVPR. pp. 9664–9674 (2021)
  * [15] Li, Z., Li, N., Jiang, K., Ma, Z., Wei, X., Hong, X., Gong, Y.: Superpixel masking and inpainting for self-supervised anomaly detection. In: BMVC (2020)
  * [16] Megiddo, I., Colson, A., Chisholm, D., Dua, T., Nandi, A., Laxminarayan, R.: Health and economic benefits of public financing of epilepsy treatment in india: An agent-based simulation model. Epilepsia 57, 464–474 (2016)
  * [17] Pérez-García, F., Scott, C., Sparks, R., Diehl, B., Ourselin, S.: Transfer learning of deep spatiotemporal networks to model arbitrarily long videos of seizures. In: MICCAI. pp. 334–344 (2021)
  * [18] Rippel, O., Mertens, P., Merhof, D.: Modeling the distribution of normal data in pre-trained deep features for anomaly detection. In: ICPR (2021)
  * [19] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis 54, 30–44 (2019)
  * [20] Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.: Support vector method for novelty detection. In: NeurIPS (1999)
  * [21] Shoeb, A.H.: Application of machine learning to epileptic seizure onset detection and treatment. Ph.D. thesis, Massachusetts Institute of Technology (2009)
  * [22] Shvetsova, N., Bakker, B., Fedulova, I., Schulz, H., Dylov, D.V.: Anomaly detection in medical imaging with deep perceptual autoencoders. IEEE Access 9, 118571–118583 (2021)
  * [23] Temko, A., Sarkar, A., Lightbody, G.: Detection of seizures in intracranial EEG: Upenn and mayo clinic’s seizure detection challenge. In: EMBC. pp. 6582–6585 (2015)
  * [24] Tian, Y., Pang, G., Liu, F., Chen, Y., Shin, S.H., Verjans, J.W., Singh, R., Carneiro, G.: Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images. In: MICCAI. pp. 128–140 (2021)
  * [25] Wang, J., Cherian, A.: Gods: Generalized one-class discriminative subspaces for anomaly detection. In: CVPR. pp. 8201–8211 (2019)
  * [26] Xu, J., Zheng, Y., Mao, Y., Wang, R., Zheng, W.S.: Anomaly detection on electroencephalography with self-supervised learning. In: BIBM (2020)
  * [27] Zavrtanik, V., Kristan, M., Skočaj, D.: Reconstruction by inpainting for visual anomaly detection. Pattern Recognition 112, 107706 (2021)
  * [28] Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., Chawla, N.V.: A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In: AAAI. pp. 1409–1416 (2019)


# Learning to Grasp Clothing Structural Regions
for Garment Manipulation Tasks

Wei Chen, Dongmyoung Lee, Digby Chappell, and Nicolas Rojas Wei Chen,
Dongmyoung Lee, Digby Chappell, and Nicolas Rojas are with the REDS Lab, Dyson
School of Design Engineering, Imperial College London, 25 Exhibition Road,
London, SW7 2DB, UK (w.chen21, d.lee20, d.chappell19,
<EMAIL_ADDRESS>work was supported in part by the China
Scholarship Council, and in part by the Dyson School of Design Engineering,
Imperial College London. Digby Chappell was supported by the UKRI CDT in AI
for Healthcare http://ai4health.io (Grant No. EP/S023283/1).

###### Abstract

When performing cloth-related tasks, such as garment hanging, it is often
important to identify and grasp certain structural regions—a shirt’s collar as
opposed to its sleeve, for instance. However, due to cloth deformability,
these manipulation activities, which are essential in domestic, health care,
and industrial contexts, remain challenging for robots. In this paper, we
focus on how to segment and grasp structural regions of clothes to enable
manipulation tasks, using hanging tasks as case study. To this end, a neural
network-based perception system is proposed to segment a shirt’s collar from
areas that represent the rest of the scene in a depth image. With a 10-minute
video of a human manipulating shirts to train it, our perception system is
capable of generalizing to other shirts regardless of texture as well as to
other types of collared garments. A novel grasping strategy is then proposed
based on the segmentation to determine grasping pose. Experiments demonstrate
that our proposed grasping strategy achieves 92%, 80%, and 50% grasping
success rates with one folded garment, one crumpled garment and three crumpled
garments, respectively. Our grasping strategy performs considerably better
than tested baselines that do not take into account the structural nature of
the garments. With the proposed region segmentation and grasping strategy,
challenging garment hanging tasks are successfully implemented using an open-
loop control policy. Supplementary material is available at
https://sites.google.com/view/garment-hanging

##  I Introduction

Manipulating items of clothing is a task that is critical to a broad spectrum
of tasks, ranging from domestic chores, assistive dressing, and industrial
automation. Humans are extremely gifted at tasks such as these, due to the
complex perception and sensorimotor systems present in our bodies enabling
precise, robust, and delicate manipulation of many objects [1]. Although
significant progress in the robotic manipulation of rigid body objects has
been realized, due to the deformable nature of many garments, robotic
manipulation of clothing still remains an open challenge. Robotic manipulators
lag far behind the human hand in terms of controllable degrees of freedom and
sensory capabilities, so an question is raised as to what level of clothing
manipulation is possible with existing technology, and how a priori knowledge
of the structure of garments can be utilized to supplement these missing
capabilities.

Clothing design elements are critical structures that affect the general
configuration of clothing (e.g. collar of a shirt, waistband of pants) [2].
Perception of such clothing structural regions is of particular importance for
robot manipulation tasks, none more so than when attempting to manipulate
deformable and continuum objects [3]. Without the ability to detect clothing
structural regions, downstream manipulation becomes inherently limited. For
example, folding or hanging clothes becomes increasingly difficult without
knowledge of suitable grasping locations on the garment. Many prior works on
the perception of generic cloth for manipulation have primarily focused on
detecting edges, wrinkles, or corners of the material in order to achieve a
successful grasp [4, 5, 6]. Both classical computer vision (CV)-based and
learning-based methods were applied in finding these edged or wrinkled
features. While grasp success is increased via these methods, such works are
only able to utilize the grasps in the form of relatively simple manipulation
tasks such as cloth folding and unfolding. For more complex tasks such as
garment-hanging, it would be challenging to apply these methods successfully
without further knowledge of the garment.

Figure 1: In the proposed method, a robot learns to automatically detect a
clothing structural region (collar in our case study) in severely crumpled
garments and execute its grasping. This is then used for cloth manipulation
(hanging the garment on a cloth tree in our case study).

This work presents a novel system for the robust robotic manipulation of
garments by leveraging structural elements of these garments during perception
and grasping as shown in Fig. 1. This system is tested extensively on the
manipulation of collared-garments such as shirts and jackets for a novel
garment-hanging task.

In order to successfully grasp structural areas of the garments a data
acquisition technique and deep learning-based segmentation model are required,
in which a 10-minute video of unstructured human manipulation of multiple
color-tagged garments is used to quickly extract ground truth segmentation
masks for training. A novel grasping strategy is then proposed that outputs a
suitable grasp pose based on the extracted skeleton and local geometric
structure of the segmented region of clothing. Finally, the suitability of the
grasping strategy for downstream manipulation tasks is tested on a garment-
hanging task. Our findings indicate that, by considering only the structural
regions of the garment that are vital for task success, robust clothing
manipulation can be achieved with standard perception and robotic hardware.

##  II Related Work

###  II-A Cloth Perception

In order for cloth manipulation to be realized, perception of key elements of
the cloth is vital. Several approaches have contributed to the area of cloth
perception. Due to the deformable characteristics of cloth, wrinkles [7, 8],
folds [9], and corners [10, 11, 12, 13] are generally considered to be
attractive grasping targets for manipulation tasks such as cloth folding or
flattening. Accordingly, early works of these approaches based on traditional
CV algorithms aimed to detect these cloth features. Typically, after some
image processing, corner and edge detection methods have been used to classify
cloths [14] or locate interaction points for cloth flattening tasks [6, 15].
In addition, [5] used shape index to analyse the topology of the garment
surface to detect wrinkles. More recent progress in cloth perception has
focused on learning-based methods to achieve the same goal. Notable works
include [10], which applied YOLO with pre-trained weights to identify corner
points for robotic bed making, and [12], which combined optical flow with a
neural network to learn grasping points for cloth folding.

Although detecting the corners and edges of a cloth is useful for basic tasks
such as flattening, the lack of semantic meaning makes it difficult to be
applied to tasks where structural elements of the cloth are important, as is
the case in many tasks involving items of clothing. This increases the
complexity of the problem twofold, as the perception of a structured garment
is more difficult, and also because the task and context in which the garment
must be perceived is likely to become more complicated as well. Some works
have focused on understanding the structure of the clothes in order to
facilitate a better grasp, attempting to reconstruct the garment in 3D
canonical space [16]. However, in the training of 3D reconstruction methods,
challenges arise regarding computational complexity and data acquisition,
since the training of such deep learning models requires high computational
power and high-quality manually labelled point cloud datasets.

###  II-B Cloth Grasping for Manipulation

During cloth grasping, the manipulation task to be performed provides context
that can either simplify or add complexity to the grasping problem [3]. For
example, a robot aiming to load laundry into a washing machine need not
perceive structural features of the clothes it is manipulating [17], whereas a
robot that is assisting dressing must consistently and accurately grasp
specific points on the item of clothing that is to be worn [18, 19]. For tasks
that do not consider the overall structure of the target clothes, some generic
features are preferred to be selected as the grasping point. For instance,
many simple cloth folding and unfolding setups have perception systems that
are devoted to locating previously mentioned generic features, such as edge or
corner points [6, 15, 12].

One of the largest challenges when grasping more complex items of clothing is
to estimate the 6D pose that the gripper must reach in order to achieve a
successful grasp. This is a complex perception task, requiring detailed
knowledge of the structure of the clothing, the current state of the clothing,
and the manipulation task to be performed. Rather than simplifying the garment
or the manipulation task itself, some works have conveniently simplified the
initial state of the clothing in order to facilitate a simpler grasping
strategy. This is typically achieved by semi-constraining the grasping pose to
an approximately planar setup [20, 10, 13], or by having the clothes pre-hung
on a peg to allow gravity to naturally restrict the orientation of possible
grasping points [18, 19, 21]. Furthermore, the vast majority of works that
focus on complex manipulation tasks only focus on a single garment at a time.

There is a notable lack of research investigating clothing manipulation from
more complex initial setups. In this work, we investigate the problem of
hanging shirts on a peg-based hanger from an unrestricted or crumpled initial
state. This is a relatively precise manipulation task requiring knowledge of
the structure of the garment. Furthermore, we explore the effect of the
initial setup on the ability of our proposed system to perform the task by
testing initial configurations with multiple folded and crumpled shirts.

Figure 2: Pipeline of Our Method: We collect depth image and its corresponding
RGB image of the diverse configuration of garments. The collected depth images
are labelled by extracting the blue pixels from RGB images and used as the
training set. During the running time, We use the depth image as input to
predict the collar by the trained segmentation neural network; The optimal
grasping point is obtained by fitting a skeleton to the predicted collar and
calculating the surface variation. A local PCA is finally conducted near this
point to find the grasping orientation. A set of action primitives, including
grasping and hanging, are designed for performing real-world robot execution.

##  III Methods

Successful manipulation always requires the garment to be grasped in a
specific region. In this paper, we propose a novel pipeline that detects the
collar region of crumpled garments to perform garment hanging. To this end, a
deep neural network is applied for the perception system. One of the most
significant challenges of training deep neural networks is data acquisition.
We propose a data collection process which uses video clips that record human
manipulating cloth for large dataset generation. This dataset can be directly
fed into the neural network for training with appropriate pre-processing
techniques. With the trained neural network, the collar region of garments can
be then segmented. Within the collar region, we estimate the most graspable
point to provide robust robotic cloth grasping. Specifically, we extract the
collar region’s center using a skeleton-based method. Then, the point with the
highest surface variation within the center region is finally chosen as the
grasping point, as this represents the sharp fold of the collar. Principal
component analysis (PCA) is applied for the pose estimation of the grasping
point. The insertion grasping policy and hanging execution are finally
conducted for the robotic cloth grasping and hanging. The overall system is
shown in Fig. 2.

###  III-A Cloth Region Segmentation

This section presents a deep neural network-based perception system that
detects the collar part of severely crumpled garments placed on a tabletop.
The neural network is trained on video clips recorded from a human freely
manipulating garments. The whole perception pipeline is illustrated in Fig. 2,
including data acquisition and model training.

####  III-A1 Data Acquisition

While training a deep neural network requires a huge number of datasets, such
datasets are not readily available. The traditional data acquisition method
requires much human effort for manual annotation [22]. While using a physics
engine to generate synthetic pictures to enrich the hybrid datasets will also
cause the sim-to-real gap [19]. We instead apply a mechanism similar to [4] to
obtain data directly from the real world.

Figure 3: Left: template shirts used for perception system training in Section
III-A. Right: Various garments were used in the testing section of our study.
The collar is pained with blue for groundtruth extraction.

As shown in Fig. 3, our data acquisition process uses white template shirts
(TSs) with long sleeves. We paint the collar of TSs to be blue. A human
operator randomly manipulates these TSs with an RGB-D camera fixed on the top,
simultaneously capturing RGB and depth images. The camera frame rate is set to
15Hz to ensure the TSs among each frame have appropriate deformation. We also
include multiple pieces (maximum three) of shirts to increase the clutter
level rather, than just one shirt during the data collection procedure. This
dataset frame order is then shuffled before being fed to the neural network to
break the temporal correspondence and make the dataset uniformly distributed.
After the shuffling, the ground truth can be obtained by extracting the blue
colour from the RGB images in HSV space. These RGB images are only used for
labelling and are not in the training procedure or for further segmentation.

####  III-A2 Semantic Segmentation with Neural Networks

We apply a deep neural network based on U-net [23]. Resnet-50 block [24] with
pre-trained weights on imagenet[25] is applied as the encoder to extract image
features. Due to the fact that the resnet-50 block uses weights pre-trained on
RGB pictures, we stack depth images three times to match the three channels of
RGB. We formulate the semantic segmentation as a pixel-level binary
classification task that only predicts the presence of collar pixels to
exclude other features in the collected depth image (e.g. tabletop, human
hand). Dice loss is applied here to reduce the effect of imbalanced class
distribution. Data augmentation procedures such as flipping, rotation and grid
distortion are also implemented to avoid model over-fitting.

###  III-B Grasping Pose Estimation

Choosing a centralized area with a wrinkled surface will give a higher
probability of successful grasp [3]. Rather than calculating graspable region
using a complex hierarchical vision architecture, such as one defined from
shape index in [5], here, an effective point cloud-based grasping pose
estimation algorithm is proposed. This proposed grasping strategy involves
three stages: center region extraction, grasping point selection, and
orientation estimation.

####  III-B1 Center Region Extraction

Grasping at the extreme edges of the collar may cause empty grasps or slippage
due to a smaller contact area between the gripper and the collar. To avoid
this, the center region of cloth is targeted, ensuring a stable grasping
action due to the larger contact area. However, the centroid or the center of
the convex hull may not provide a good candidate for grasping concave shape
garments, especially for the collar of garments which are typically concave.
Inspired by [3], we adopt a skeleton-based method to estimate the middle point
of the skeleton, therefore extracting the center region of the collar. Three
steps are involved in our center region extraction algorithm, shown in Fig. 4:

  * •

Clustering: After obtaining the prediction result, we first apply
agglomerative clustering to the prediction result based on the pixel locations
to identify individual collars from a scene potentially containing multiple
garments. The largest cluster is chosen for the following grasping pipeline.
After the largest cluster is obtained, an image dilation is applied to remove
noise from the clustered prediction result.

  * •

Skeletonize: The skeletonize algorithm is then applied to this largest
cluster. This skeletonize algorithm is originally from [26], and applied to
the prediction result to obtain a boundary edge map which is a single pixel
wide, in other words, a skeleton.

  * •

Center extraction: Closeness centrality is defined by calculating the inverse
of the total sum of shortest distances from a node to every other node [27].
We find the maximum closeness centrality here to represent the midpoint of the
whole skeleton. This pixel is selected as the center point of the collar.

Figure 4: Center Region Extraction: (a) Input depth image; (b) Prediction
result; (c) Clustering; (d) Skeletonize and calculate the center of skeleton.
We compare the centriod (blue dot) and skeleton center (red dot).

####  III-B2 Grasping Position Selection

Using the extracted collar pixels, we de-project these collar pixels from the
depth image to obtain the collar point cloud according to the known camera
intrinsic parameters. Pre-processing procedures such as voxel down-sampling
and radius outlier removal are implemented here to reduce computational
complexity[28]. Since cloth is highly deformable, a random point selection
near the skeleton center is insufficient to achieve a robust clothing grasp.
It has been shown that wrinkled points are good candidates for the gripping
action; successful grasping can be guaranteed by positioning the grippers at
either side of a crease [3]. To ensure a successful and stable grasp, wrinkles
with more significant curvature changes are favoured. We use surface variation
to approximate these curvature changes. Having the predicted collar point
cloud and estimated skeleton center, surface variation estimation near the
skeleton center of the local point cloud is applied [29].

For each point in the region of $N$ nearest points (here chosen to be $N=50$)
to the skeleton center $P$, we compute local surface variation. Surface
variation for point $\bm{p}^{(i)}$ is computed by selecting $n$ neighbouring
points, $[\bm{p}^{(i)}_{1},\ldots,\bm{p}^{(i)}_{n}]^{T}$ and first calculating
the centroid position $\bar{\bm{\mu}}^{(i)}$:

$\bar{\bm{\mu}}^{(i)}=\frac{1}{n}\sum_{j=1}^{n}\bm{p}^{(i)}_{j},$ (1)

where $n=50$ as in [29]. Then, the covariance matrix $\bm{C}^{(i)}$ is
obtained:

$\bm{C}^{(i)}=\frac{1}{n}\sum_{j=1}^{n}(\bm{p}^{(i)}_{j}-\bar{\bm{\mu}}^{(i)})(\bm{p}^{(i)}_{j}-\bar{\bm{\mu}}^{(i)})^{T}.$
(2)

The eigenvectors and eigenvalues of $\bm{C}^{(i)}$ are computed:

$\bm{C}^{(i)}\overrightarrow{\bm{v}}^{(i)}_{k}=\lambda_{k}\overrightarrow{\bm{v}}^{(i)}_{k},\quad
k\in\\{0,1,2\\},$ (3)

where the eigenvalue $\lambda_{k}$ quantifies the variance in the direction of
eigenvector $\overrightarrow{\bm{v}}^{(i)}_{k}$ at point $\bm{p}^{(i)}$.
Assuming the three eigenvalues are ordered,
$\lambda_{0}<\lambda_{1}<\lambda_{2}$, the eigenvector
$\overrightarrow{\bm{v}}^{(i)}_{0}$ then represents the axis normal to the
direction of maximum surface variation, and
$\overrightarrow{\bm{v}}^{(i)}_{2}$ to the axis of maximum surface variation
[30]. The three eigenvalues of the covariance matrix $\bm{C}$ are used to
approximate the local surface variation around point $\bm{p}^{(i)}$. The
surface variation at point $\bm{p}^{(i)}$ can be obtained, as in [29], by
computing:

$\sigma_{\bm{p}^{(i)}}=\frac{\lambda_{0}}{\lambda_{0}+\lambda_{1}+\lambda_{2}}.$
(4)

We compute the surface variation at each of the $N$ neighboring points around
skeleton center $P$, and select the point of largest surface variation,
$\bm{p}$, within the neighborhood as the grasping point, and its own $n$
nearest neighbors as the grasping region.

Figure 5: Grasping Pose Estimation: garment RGB image and extracted collar
region pointcloud. The point (red point) with highest surface variation in the
center region (dashed circle) will be selected as the grasping point. Three
eigenvectors ($\overrightarrow{\bm{v}}_{0}$, $\overrightarrow{\bm{v}}_{1}$,
$\overrightarrow{\bm{v}}_{2}$) from this point and its nearest neighbour are
used for the pose estimation. Table I: Perception Evaluation Results of Seen
and Unseen Garments

| TS (seen) | S1 (unseen) | S2 (unseen) | L1 (unseen) | L2 (unseen) | P1(unseen) | P2(unseen) | C1(unseen) | C2(unseen) | D1 (unseen) | D2 (unseen)
---|---|---|---|---|---|---|---|---|---|---|---
Recall | 0.873 | 0.799 | 0.665 | 0.778 | 0.751 | 0.304 | 0.603 | 0.623 | 0.585 | 0.681 | 0.772
Precision | 0.853 | 0.825 | 0.863 | 0.792 | 0.791 | 0.636 | 0.808 | 0.925 | 0.838 | 0.891 | 0.865
IoU | 0.760 | 0.684 | 0.602 | 0.645 | 0.626 | 0.255 | 0.527 | 0.593 | 0.525 | 0.628 | 0.689

Evaluation results of our perception system. Our perception system is trained
by dataset collected from template shirts (TS) and evaluated by testing
dataset from unseen garments: short sleeve shirts (S1, S2), long sleeve shirts
(L1, L2), polo shirts (P1, P2), coats (C1, C2) and denim jackets (D1, D2).

####  III-B3 Grasping Orientation Estimation

The grasping pose is defined by aligning the gripper pose to the grasping
point for the actual robotic cloth grasping instead of estimating the whole
pose of the cloth [19]. We use the eigenvector $\overrightarrow{\bm{v}}_{0}$
(correlates with the smallest eigenvalue), which is estimated from the mean
covariance matrix as the surface normal vector $\bm{n}$ of the selected
grasping region. This surface normal vector $\bm{n}$ is then defined as the
$Z$-axis of the orientation. As shown in illustration in Fig. 5, the major
component of eigenvector $\overrightarrow{\bm{v}}_{2}$ is aligned with the
average longitudinal direction of the selected grasping region in which we
define the $Y$-axis here. Since these three eigenvectors
$\overrightarrow{\bm{v}}_{0}$, $\overrightarrow{\bm{v}}_{1}$,
$\overrightarrow{\bm{v}}_{2}$ are orthogonal to each other, the $X$-axis will
be aligned with eigenvector $\overrightarrow{\bm{v}}_{1}$ at the lateral
direction.

###  III-C Insertion Grasping Execution

As shown in Fig. 6, this study uses $(\bm{c},\bm{o})$ to specify the goal
coordinate and orientation of the 6d grasping pose. We perform the calibration
according to the known camera extrinsic parameters to transfer the pose of
grasping region $(c_{goal},o_{goal})$ to the world frame. Based on the
selection of the goal grasping pose, we define a pre-grasp pose
$(c_{pre},o_{goal})$, which is 50 mm away from the target position along the
robot approaching direction ($Z$-axis). The robot arm will first reach the
pre-grasping pose, then perform the grasp insertion, in order to make sure the
collar of the garment is inserted between each gripper finger. Upon reaching
the grasping pose, the robot performs the grasp. The motion planning of the
robot is achieved by Moveit! [31].

Figure 6: Insertion Trajectory: (a) The robot will approach to
$(c_{pre},o_{goal})$; (b) Insertion will execute robot to
$(c_{goal},o_{goal})$; (c) Grasping.

###  III-D Garment-hanging

Garment-hanging is challenging due to the significant variation in appearance
and topology of garments. To approach this problem, we treat the grasping
point as the keypoint that represents the whole configuration of cloth, in a
similar method to [32]. By doing so, we assume that if the collar is grasped
successfully, then the garment will be in a suitable state to able to be hung
on a peg without the need for further regrasping. As shown in Robot Execution
of Fig. 2, after the robot grasps the cloth, the robot will execute a pre-
defined hanging trajectory. After reaching the intermediate position (blue
dot) and the center of the hook (green dot), the robot releases the garment
and finishes the hanging task.

Table II: Grasping Success Rate with Robot and A Parallel Gripper.

| Initial Configuration
---|---
Method | [fd1] | [cr1] | [cr2] | [cr3] | [cr4]
*B1: Predicted-Point | 48% | 28% | 22% | 20% | 16%
*B2: Predicted-Point-o | 80% | 44% | 34% | 26% | 20%
B3: Only-Centering | 58% | 46% | 34% | 30% | 24%
B4: Only-Surface-Variation | 80% | 66% | 56% | 38% | 28%
Our Method | 92% | 80% | 62% | 50% | 36%

[fd1], [cr1]-[cr4] correspond to experiment scenarios where folded garment (1
piece), crumpled garments (1-4 pieces) are randomly placed on the tabletop,
respectively. TS and S1 are used for grasping evaluation. The letter B
indicates a baseline method. *Baseline 1 and *Baseline 2 are based on[22, 19].

##  IV Experiment

Several experiments are conducted here to test our proposed cloth manipulation
system. We evaluate the proposed cloth manipulation system in three aspects:
1) the accuracy and robustness of our proposed learned perception model in
finding the collar of garments; 2) the performance of our proposed grasping
strategy; 3) the overall evaluation of the robotic hanging system.

####  IV-1 Perception Evaluation

In the experiment to evaluate the perception system, we use the method
described in Section III-A to prepare the training set, validation set and
testing set. The training set is comprised of 8212 depth images, the
validation set of 2023 depth images with the same garment, and the testing set
contains 5328 depth images from unseen garments. Our experiment uses template
shirts (TS) with two long sleeves to collect training and validation sets.
Short sleeve shirts (S1, S2), polo shirts (P1, P2), long sleeved shirts (L1,
L2), coats (C1, C2) and denim jackets (D1, D2) are used here for the testing
set. We use Intersection over Union (IoU), Recall and Precision to measure the
performance of the system. From Table I, it can be observed that our proposed
perception system can generalize to other collard garments. Specifically, our
proposed system has a high accuracy in seen garments and most unseen garments.
Although some accuracy gap between seen and unseen objects exists, the overall
performance of unseen garments is still satisfactory for downstream
manipulation tasks. More positive and negative examples of garment
segmentation are presented in the multimedia materials supplied with this
work.

Table III: Performance of Robotic Hanging

Initial Config. | TS | S1 | S2 | L1 | L2 | P1 | P2 | C1 | C2 | D1 | D2 | Overall
---|---|---|---|---|---|---|---|---|---|---|---|---
[fd] | 9/10 | 8/10 | 8/10 | 7/10 | 6/10 | 2/10 | 6/10 | 8/10 | 7/10 | 6/10 | 6/10 | 73/110
[cr] | 6/10 | 7/10 | 6/10 | 6/10 | 6/10 | 2/10 | 5/10 | 7/10 | 5/10 | 6/10 | 5/10 | 61/110

To demonstrate the robustness of our proposed system in real-world scenarios,
11 different garments used in perception experiments are applied for this
task. [fd] and [cr] represent folded and crumpled configuration of garments
respectively.

####  IV-2 Grasping Strategy Evaluation

The real-world garment grasping is implemented on a UR5 robot with a
customized parallel gripper. All gripper parts are 3d-printed, and the gripper
is driven by a NEMA-17 stepper motor. A soft silicone pad is attached to each
finger to ensure high friction during grasping. An Intel RealSense D435i RGB-D
camera is fixed above the scene during both data collection and experiment.
All training and inference is performed on a machine running Ubuntu 18.04 with
an NVIDIA GTX 3070 GPU.

A range of initial configurations of garments is essential to test the
robustness of the grasping strategy. In grasping evaluation experiments, we
follow the benchmark proposed by [33]. From this benchmark, two initial
configurations: folded [fd] and crumpled [cr] are adopted. For the [fd]
configuration, the garment is initialized with a relatively structured
configuration. The target collar region of the garment is completely visible
with minimal self-occlusion and deformation. For [cr], the garment is dropped
from height of 0.4 meters above the tabletop at the start of each trial,
resulting in a crumpled garment configuration. Generally, the target collar
region of the garment is partially visible with self-occlusion and
deformation. With more garments added for the experiment ([cr1] to [cr4]), the
degree of self-occlusion and deformation increases. For each scenario, 50
grasping trials are performed to obtain the grasping success rate. Success
cases are only counted when the collar regions are grasped and lifted 0.4
meters above the tabletop. In summary, our method uses a neural network to
segment the collar region of garments, determines the center of the segmented
collar pixels with a skeleton-based method, selects the pose aligning with the
maximum surface variation to grasp. Several baselines are also implemented to
evaluate the effectiveness of our proposed method and test the contribution of
each aspect of the method:

  * •

Predicted-Point uses TSs with a tagged collar center for data acquisition to
train a model that directly predicts the grasping point, similar to [22, 19].
We use the same architecture described in Section III-A for the model
training. A fixed top-down grasping orientation is executed for the grasping.

  * •

Predicted-Point-o uses the same point prediction model, but the grasping
orientation estimation is determined by the method described in Section
III-B-(3).

  * •

Only-Centering uses the segmented collar of the shirt. The grasping point is
randomly chosen within the center region. The grasping orientation is
determined by the method described in Section III-B-(3).

  * •

Only-Surface-Variation uses the segmented collar of the shirt. The point with
the maximum surface variation of the whole segmentation output is selected as
the grasping point. The grasping orientation is determined by using the method
described in Section III-B-(3).

Figure 7: Segmentation and predicted point for collar grasping methods. The
result of segmentation (a) and grasping estimation (b) of our method. Point
detection (c) and grasping estimation (d) of Predicted-Point-o.

.

As shown in Table II, segmentation-based methods outperform methods that rely
on image-to-grasping point prediction in unstructured ([cr1] \- [cr4])
scenarios. This is largely because the trained target point in point
prediction methods may not always be visible. This is illustrated further in
Fig. 7, which shows a failure case of point methods, where the target point is
occluded. Full grasping pose estimation (Predicted-Point vs Predicted-Point-o)
is shown to be critical for improved grasping performance, and is necessary
when the initial configuration is not restricted to a single orientation.
Identifying the collar fold is also important for successful grasping, with
methods that identify maximum surface variation (the proposed method and the
Only-Surface-Variation baseline) outperforming those that do not in all cases.
Finally, combining centering and surface variation allows our method to ensure
a good contact between gripper and garment, avoiding the extreme edges of the
collar which lead to failure in the Only-Surface-Variation baseline. The
success rate of these methods decreases as more garments are added in the
grasping scene. This strongly reduces segmentation performance because of more
occlusion and an increased garment configuration complexity. However, even in
the most complex scenario ([cr4]), the proposed method still outperforms other
baselines.

####  IV-3 Manipulation System Evaluation

We evaluate our proposed system based on a garment-hanging task. As shown in
Table III, with our proposed vision-based perception and grasping algorithm,
our system has 73 successful cases out of 110 trials for the [fd]
configuration, while the [cr] realizes 61 cases - a slight decrease.
Furthermore, we observed that failures of garment-hanging are mainly caused
by: (1) failed initial grasping; (2) garment dropping because of heavy weight;
(3) garment entangled during the hanging. These are shown in the multimedia
material. Of these, (1) represents a failure case regarding the proposed
algorithm, while (2) can be addressed with a stronger gripper, and (3) may
require a more sophisticated hanging policy as opposed to the offline policy
used here.

##  V Conclusions and Future Work

In this paper, we propose a novel cloth manipulation system that utilises the
structural elements of garments for both perception and grasping. The training
of this perception system is based on data acquisition of a 10-minute video of
human manipulation with automatic annotation. Experimental results indicate
the proposed perception system can generalize to other types of similarly
structured clothes. With the proposed pose estimation strategy, our system can
achieve 92% and 80% grasping success rates for a single piece of folded and
crumpled garment, respectively, which are significantly higher than other
baselines. We use a garment-hanging task as a case study to evaluate the
performance of our overall system. Results indicate that the challenging
garment-hanging task can be implemented with an offline hanging policy by only
considering the structural regions of garments used in grasping. Future work
will investigate how to explore and search scene in situations where garment
collar is completely non-visible. More work will be done by using our proposed
system to perceive and grasp other vital regions (e.g. hem or sleeves) of
garments for implementing different garment manipulation tasks.

## References

  * [1] A. R. Sobinov and S. J. Bensmaia, “The neural mechanisms of manual dexterity,” _Nature Reviews Neuroscience_ , vol. 22, no. 12, pp. 741–757, 2021.
  * [2] L. Hong, C. Yan, and D. Hongqin, “Eliciting and representing the knowledge of clothing design,” in _presented at Textile Bioengineering and Informatics Symposium Proceedings_ , 2008.
  * [3] P. Jiménez and C. Torras, “Perception of cloth in assistive robotic manipulation tasks,” _Natural Computing_ , vol. 19, no. 2, pp. 409–431, 2020\.
  * [4] J. Qian, T. Weng, L. Zhang, B. Okorn, and D. Held, “Cloth region segmentation for robust grasp selection,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9553–9560.
  * [5] L. Sun, G. Aragon-Camarasa, S. Rogers, and J. P. Siebert, “Accurate garment surface analysis using an active stereo robot head with application to dual-arm flattening,” in _2015 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2015, pp. 185–192.
  * [6] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 2308–2315.
  * [7] L. Sun, G. Aragon-Camarasa, S. Rogers, R. Stolkin, and J. P. Siebert, “Single-shot clothing category recognition in free-configurations with application to autonomous clothes sorting,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 6699–6706.
  * [8] W. Yuan, Y. Mo, S. Wang, and E. H. Adelson, “Active clothing material perception using tactile sensing and deep learning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 4842–4849.
  * [9] Y. Moriya, D. Tanaka, K. Yamazaki, and K. Takeshita, “A method of picking up a folded fabric product by a single-armed robot,” _ROBOMECH Journal_ , vol. 5, no. 1, pp. 1–12, 2018.
  * [10] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani, R. Berenstein, P. Baskaran, S. Iba, J. Canny, and K. Goldberg, “Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making,” in _International Symposium on Robotics Research (ISRR)_ , 2019.
  * [11] R. Jangir, G. Alenya, and C. Torras, “Dynamic cloth manipulation with deep reinforcement learning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4630–4636.
  * [12] T. Weng, S. M. Bajracharya, Y. Wang, K. Agrawal, and D. Held, “Fabricflownet: Bimanual cloth manipulation with a flow-based policy,” in _Conference on Robot Learning_. PMLR, 2022, pp. 192–202.
  * [13] J. Matas, S. James, and A. J. Davison, “Sim-to-real reinforcement learning for deformable object manipulation,” in _Conference on Robot Learning_. PMLR, 2018, pp. 734–743.
  * [14] B. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, 2011, pp. 1862–1868.
  * [15] D. Triantafyllou and N. A. Aspragathos, “A vision system for the unfolding of highly non-rigid objects on a table by one manipulator,” in _International Conference on Intelligent Robotics and Applications_. Springer, 2011, pp. 509–519.
  * [16] C. Chi and S. Song, “Garmentnets: Category-level pose estimation for garments via canonical space shape completion,” in _The IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [17] H. Shehawy, P. Rocco, and A. M. Zanchettin, “Estimating a garment grasping point for robot,” in _2021 20th International Conference on Advanced Robotics (ICAR)_ , 2021, pp. 707–714.
  * [18] K. Saxena and T. Shibata, “Garment recognition and grasping point detection for clothing assistance task using deep learning,” in _2019 IEEE/SICE International Symposium on System Integration (SII)_. IEEE, 2019, pp. 632–637.
  * [19] F. Zhang and Y. Demiris, “Learning grasping points for garment manipulation in robot-assisted dressing,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 9114–9120.
  * [20] H. Ha and S. Song, “Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding,” in _Conference on Robot Learning_. PMLR, 2022, pp. 24–33.
  * [21] F. Zhang and Y. Demiris, “Learning garment manipulation policies toward robot-assisted dressing,” _Science robotics_ , vol. 7, no. 65, p. eabm6010, 2022.
  * [22] A. Ramisa, G. Alenya, F. Moreno-Noguer, and C. Torras, “Using depth and appearance features for informed robot grasping of highly wrinkled clothes,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 1703–1708.
  * [23] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [25] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [26] T. Y. Zhang and C. Y. Suen, “A fast parallel algorithm for thinning digital patterns,” _Communications of the ACM_ , vol. 27, no. 3, pp. 236–239, 1984\.
  * [27] L. C. Freeman, “Centrality in social networks conceptual clarification,” _Social networks_ , vol. 1, no. 3, pp. 215–239, 1978.
  * [28] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3D: A modern library for 3D data processing,” _arXiv:1801.09847_ , 2018.
  * [29] M. Pauly, M. Gross, and L. P. Kobbelt, “Efficient simplification of point-sampled surfaces,” in _IEEE Visualization, 2002. VIS 2002._ IEEE, 2002, pp. 163–170.
  * [30] R. B. Rusu, Z. C. Marton, N. Blodow, M. Dolha, and M. Beetz, “Towards 3d point cloud based object maps for household environments,” _Robotics and Autonomous Systems_ , vol. 56, no. 11, pp. 927–941, 2008.
  * [31] S. Chitta, “Moveit!: an introduction,” in _Robot Operating System (ROS)_. Springer, 2016, pp. 3–27.
  * [32] L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “kpam: Keypoint affordances for category-level robotic manipulation,” in _The International Symposium of Robotics Research_. Springer, 2019, pp. 132–157.
  * [33] I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Alenya, _et al._ , “Benchmarking bimanual cloth manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1111–1118, 2020.