# Self-supervised Regularization for Text Classification

Meng Zhou∗
Shanghai Jiao Tong University
<EMAIL_ADDRESS>Zechen Li
Northeastern University
<EMAIL_ADDRESS>Pengtao Xie
UC San Diego
<EMAIL_ADDRESS>Equal Contribution Corresponding Author

###### Abstract

Text classification is a widely studied problem and has broad applications. In
many real-world problems, the number of texts for training classification
models is limited, which renders these models prone to overfitting. To address
this problem, we propose SSL-Reg, a data-dependent regularization approach
based on self-supervised learning (SSL). SSL Devlin et al. (2019a) is an
unsupervised learning approach which defines auxiliary tasks on input data
without using any human-provided labels and learns data representations by
solving these auxiliary tasks. In SSL-Reg, a supervised classification task
and an unsupervised SSL task are performed simultaneously. The SSL task is
unsupervised, which is defined purely on input texts without using any human-
provided labels. Training a model using an SSL task can prevent the model from
being overfitted to a limited number of class labels in the classification
task. Experiments on 17 text classification datasets demonstrate the
effectiveness of our proposed method. Code is available at
https://github.com/UCSD-AI4H/SSReg

##  1 Introduction

Text classification Korde and Mahender (2012); Lai et al. (2015); Wang et al.
(2017); Howard and Ruder (2018) is a widely studied problem in natural
language processing and finds broad applications. For example, give clinical
notes of a patient, judge whether this patient has heart diseases. Given a
scientific paper, judge whether it is about NLP. In many real-world text
classification problems, texts available for training are oftentimes limited.
For instance, it is difficult to obtain a lot of clinical notes from hospitals
due to concern of patient privacy. It is well known that when training data is
limited, models tend to overfit to training data and perform less well on test
data.

To address overfitting problems in text classification, we propose a data-
dependent regularizer called SSL-Reg based on self-supervised learning (SSL)
Devlin et al. (2019a); He et al. (2019); Chen et al. (2020) and use it to
regularize the training of text classification models, where a supervised
classification task and an unsupervised SSL task are performed simultaneously.
Self-supervised learning (SSL) Devlin et al. (2019a); He et al. (2019); Chen
et al. (2020) is an unsupervised learning approach which defines auxiliary
tasks on input data without using any human-provided labels and learns data
representations by solving these auxiliary tasks. For example, BERT Devlin et
al. (2019a) is a typical SSL approach where an auxiliary task is defined to
predict masked tokens and a text encoder is learned by solving this task. In
existing SSL approaches for NLP, an SSL task and a target task are performed
sequentially. A text encoder is first trained by solving an SSL task defined
on a large collection of unlabeled texts. Then this encoder is used to
initialize an encoder in a target task. The encoder is finetuned by solving
the target task. A potential drawback of performing SSL task and target task
sequentially is that text encoder learned in SSL task may be overridden after
being finetuned in target task. If training data in the target task is small,
the finetuned encoder has a high risk of being overfitted to training data.

To address this problem, in SSL-Reg we perform SSL task and target task (which
is classification) simultaneously. In SSL-Reg, an SSL loss serves as a
regularization term and is optimized jointly with a classification loss. SSL-
Reg enforces a text encoder to jointly solve two tasks: an unsupervised SSL
task and a supervised text classification task. Due to the presence of the SSL
task, models are less likely to be biased to the classification task defined
on small-sized training data. We perform experiments on 17 datasets, where
experimental results demonstrate the effectiveness of SSL-Reg in alleviating
overfitting and improving generalization performance.

The major contributions of this paper are:

  * •

We propose SSL-Reg, which is a data-dependent regularizer based on SSL, to
reduce the risk that a text encoder is biased to a data-deficient
classification task on small-sized training data.

  * •

Experiments on 17 datasets demonstrate the effectiveness of our approaches.

The rest of this paper is organized as follows. Section 2 reviews related
works. Section 3 and 4 present methods and experiments. Section 5 concludes
the paper and discusses future work.

##  2 Related Works

###  2.1 Self-supervised Learning for NLP

Self-supervised learning (SSL) aims to learn meaningful representations of
input data without using human annotations. It creates auxiliary tasks solely
using input data and forces deep networks to learn highly-effective latent
features by solving these auxiliary tasks. In NLP, various auxiliary tasks
have been proposed for SSL, such as next token prediction in GPT Radford et
al. , masked token prediction in BERT Devlin et al. (2019a), text denoising in
BART Lewis et al. (2019), contrastive learning Fang et al. (2020), and so on.
These models have achieved substantial success in learning language
representations. The GPT model Radford et al.  is a language model (LM) based
on Transformer Vaswani et al. (2017). Different from Transformer which defines
a conditional probability on an output sequence given an input sequence, GPT
defines a marginal probability on a single sequence. In GPT, conditional
probability of the next token given a historical sequence is defined using a
Transformer decoder. Weight parameters are learned by maximizing likelihood on
token sequences. BERT Devlin et al. (2019a) aims to learn a Transformer
encoder for representing texts. BERT’s model architecture is a multi-layer
bidirectional Transformer encoder. In BERT, Transformer uses bidirectional
self-attention. To train the encoder, BERT masks some percentage of input
tokens at random, and then predicts those masked tokens by feeding hidden
vectors (produced by the encoder) corresponding to masked tokens into an
output softmax over word vocabulary. BERT-GPT Wu et al. (2019) is a model used
for sequence-to-sequence modeling where a pretrained BERT is used to encode
input text and GPT is used to generate output texts. In BERT-GPT, pretraining
of BERT encoder and GPT decoder is conducted separately, which may lead to
inferior performance. Auto-Regressive Transformers (BART) Lewis et al. (2019)
has a similar architecture as BERT-GPT, but trains BERT encoder and GPT
decoder jointly. To pretrain BART weights, input texts are corrupted randomly,
such as token masking, token deletion, text infilling, etc., then a network is
learned to reconstruct original texts. ALBERT Lan et al. (2019) uses
parameter-reduction methods to reduce memory consumption and increase training
speed of BERT. It also introduces a self-supervised loss which models inter-
sentence coherence. RoBERTa Liu et al. (2019a) is a replication study of BERT
pretraining. It shows that BERT’s performance can be greatly improved by
carefully tuning training processes, such as (1) training models longer, with
larger batches, over more data; (2) removing the next sentence prediction
objective; (3) training on longer sequences, etc. XLNet Yang et al. (2019)
learns bidirectional contexts by maximizing expected likelihood over all
permutations of factorization order and uses a generalized autoregressive
pretraining mechanism to overcome the pretrain-finetune discrepancy of BERT.
T5 Raffel et al. (2019) compared pretraining objectives, architectures,
unlabeled datasets, transfer approaches on a wide range of language
understanding tasks and proposed a unified framework that casts these tasks as
a text-to-text task. ERNIE 2.0 Sun et al. (2019) proposed a continual
pretraining framework which builds and learns incrementally pretraining tasks
through constant multi-task learning, to capture lexical, syntactic and
semantic information from training corpora. Gururangan et al. (2020) proposed
task adaptive pretraining (TAPT) and domain adaptive pretraining (DAPT). Given
a RoBERTa model pretrained on large-scale corpora, TAPT continues to pretrain
RoBERTa on training dataset of target task. DAPT continues to pretrain RoBERTa
on datasets that have small domain differences with data in target tasks. The
difference between our proposed SSL-Reg method with TAPT and DAPT is that SSL-
Reg uses a self-supervised task (e.g., mask token prediction) to regularize
the finetuning of RoBERTa where text classification task and self-supervised
task are performed jointly. In contrast, TAPT and DAPT use self-supervised
task for pretraining, where text classification task and self-supervised task
are performed sequentially. The connection between our method and TAPT is that
they both leverage texts in target tasks to perform self-supervised learning,
in addition to SSL on large-scale external corpora. Different from SSL-Reg and
TAPT, DAPT uses domain-similar texts rather than target texts for additional
SSL.

###  2.2 Self-supervised Learning in General

Self-supervised learning has been widely applied to other application domains,
such as image classification He et al. (2019); Chen et al. (2020), graph
classification Zeng and Xie (2021), visual question answering He et al.
(2020a), etc, where various strategies have been proposed to construct
auxiliary tasks, based on temporal correspondence Li et al. (2019); Wang et
al. (2019a), cross-modal consistency Wang et al. (2019b), rotation prediction
Gidaris et al. (2018); Sun et al. (2020), image inpainting Pathak et al.
(2016), automatic colorization Zhang et al. (2016), context prediction Nathan
Mundhenk et al. (2018), etc. Some recent works studied self-supervised
representation learning based on instance discrimination Wu et al. (2018) with
contrastive learning. Oord et al. (2018) proposed contrastive predictive
coding (CPC), which predicts the future in latent space by using powerful
autoregressive models, to extract useful representations from high-dimensional
data. Bachman et al. (2019) proposed a self-supervised representation learning
approach based on maximizing mutual information between features extracted
from multiple views of a shared context. MoCo He et al. (2019) and SimCLR Chen
et al. (2020) learned image encoders by predicting whether two augmented
images were created from the same original image. Srinivas et al. (2020)
proposed to learn contrastive unsupervised representations for reinforcement
learning. Khosla et al. (2020) investigated supervised contrastive learning,
where clusters of points belonging to the same class were pulled together in
embedding space, while clusters of samples from different classes were pushed
apart. Klein and Nabi (2020) proposed a contrastive self-supervised learning
approach for commonsense reasoning. He et al. (2020b); Yang et al. (2020)
proposed an Self-Trans approach which applied contrastive self-supervised
learning on top of networks pretrained by transfer learning.

Compared with supervised learning which requires each data example to be
labeled by humans or semi-supervised learning which requires part of data
examples to be labeled, self-supervised learning is similar to unsupervised
learning because it does not need human-provided labels. The key difference
between self-supervised learning (SSL) and unsupervised learning is that SSL
focuses on learning data representations by solving auxiliary tasks defined on
un-labeled data while unsupervised learning is more general and aims to
discover latent structures from data, such as clustering, dimension reduction,
manifold embedding Roweis and Saul (2000), etc.

###  2.3 Text Classification

Text classification Minaee et al. (2020) is one of the key tasks in natural
language processing and has a wide range of applications, such as sentiment
analysis, spam detection, tag suggestion, etc. A number of approaches have
been proposed for text classification. Many of them are based on RNNs. Liu et
al. (2016) use multi-task learning to train RNNs, utilizing the correlation
between tasks to improve text classification performance. Tai et al. (2015)
generalize sequential LSTM to tree-structured LSTM to capture the syntax of
sentences for achieving better classification performance. Compared with RNN-
based models, CNN-based models are good at capturing local and position-
invariant patterns. Kalchbrenner et al. (2014) proposed dynamic CNN (DCNN),
which uses dynamic k-max-pooling to explicitly capture short-term and long-
range relations of words and phrases. Zhang et al. (2015) proposed a
character-level CNN model for text classification, which can deal with out-of-
vocabulary words. Hybrid methods combine RNN and CNN to explore the advantages
of both. Zhou et al. (2015) proposed a convolutional LSTM network, which uses
a CNN to extract phrase-level representations, then feeds them to an LSTM
network to represent the whole sentence.

##  3 Methods

To alleviate overfitting in text classification, we propose SSL-Reg, which is
a regularization approach based on self-supervised learning (SSL), where an
unsupervised SSL task and a supervised text classification task are performed
jointly.

Figure 1: Illustration of SSL-Reg. Input texts are fed into a text encoder.
Encodings of these texts and their corresponding labels are fed into the head
of a target task (e.g., classification) which yields a classification loss. In
a self-supervised task, inputs are encodings of texts and outputs are
constructed on original texts (e.g., masked tokens). The classification task
and SSL task share the same text encoder and losses of these two tasks are
optimized jointly to learn the text encoder.

###  3.1 SSL-based Regularization

SSL-Reg uses a self-supervised learning task to regularize a text
classification model. Figure 1 presents an illustration of SSL-Reg. Given
training texts, we encode them using a text encoder. Then on top of text
encodings, two tasks are defined. One is a classification task, which takes
the encoding of a text as input and predicts the class label of this text.
Prediction is conducted using a classification head. The other task is SSL.
The loss of the SSL task serves as a data-dependent regularizer to alleviate
overfitting. The SSL task has a predictive head. These two tasks share the
same text encoder. Formally, SSL-Reg solves the following optimization
problem:

$\begin{array}[]{l}\mathcal{L}^{(c)}(D,L;\mathbf{W}^{(e)},\mathbf{W}^{(c)})+\lambda\mathcal{L}^{(p)}(D,\mathbf{W}^{(e)},\mathbf{W}^{(p)})\end{array}$
(1)

where $D$ represents training texts and $L$ represents their labels.
$\mathbf{W}^{(e)}$, $\mathbf{W}^{(c)}$, and $\mathbf{W}^{(p)}$ denote text
encoder, classification head in classification task, and prediction head in
SSL task respectively. $\mathcal{L}^{(c)}$ denotes classification loss and
$\mathcal{L}^{(p)}$ denotes SSL loss. $\lambda$ is a tradeoff parameter.

At the core of SSL-Reg is using SSL to learn a text encoder that is robust to
overfitting. Our methods can be used to learn any text encoder. In this work,
we perform the study using a Transformer encoder, while noting that other text
encoders are also applicable.

Domain | Dataset | Label Type | Train | Dev | Test | Classes
---|---|---|---|---|---|---
BioMed | ChemProt | relation classification | 4169 | 2427 | 3469 | 13
RCT | abstract sent. roles | 180040 | 30212 | 30135 | 5
CS | ACL-ARC | citation intent | 1688 | 114 | 139 | 6
SciERC | relation classification | 3219 | 455 | 974 | 7
News | HyperPartisan | partisanship | 515 | 65 | 65 | 2
AGNews | topic | 115000 | 5000 | 7600 | 4
Reviews | Helpfulness | review helpfulness | 115251 | 5000 | 25000 | 2
IMDB | review sentiment | 20000 | 5000 | 25000 | 2
Table 1:  Statistics of datasets used in (Gururangan et al., 2020). Sources: ChemProt (Kringelum et al., 2016), RCT (Dernoncourt and Lee, 2017), ACL-ARC Jurgens et al. (2018), SciERC (Luan et al., 2018), HyperPartisan (Kiesel et al., 2019), AGNews (Zhang et al., 2015), Helpfulness (McAuley et al., 2015), IMDB (Maas et al., 2011). This table is taken from (Gururangan et al., 2020).  | CoLA | RTE | QNLI | STS-B | MRPC | WNLI | SST-2 |  MNLI (m/mm) | QQP | AX
---|---|---|---|---|---|---|---|---|---|---
Train | 8551 | 2490 | 104743 | 5749 | 3668 | 635 | 67349 | 392702 | 363871 | -
Dev | 1043 | 277 | 5463 | 1500 | 408 | 71 | 872 | 9815/9832 | 40432 | -
Test | 1063 | 3000 | 5463 | 1379 | 1725 | 146 | 1821 | 9796/9847 | 390965 | 1104
Table 2: GLUE dataset statistics.

###  3.2 Self-supervised Learning Tasks

In this work, we use two self-supervised learning tasks – masked token
prediction (MTP) and sentence augmentation type prediction (SATP) – to perform
our studies while noting that other SSL tasks are also applicable.

  * •

Masked Token Prediction (MTP) This task is used in BERT. Some percentage of
input tokens are masked at random. Texts with masked tokens are fed into a
text encoder which learns a latent representation for each token including the
masked ones. The task is to predict these masked tokens by feeding hidden
vectors (produced by the encoder) corresponding to masked tokens into an
output softmax over word vocabulary.

  * •

Sentence Augmentation Type Prediction (SATP) Given an original text $o$, we
apply different types of augmentation methods to create augmented texts from
$o$. We train a model to predict which type of augmentation was applied to an
augmented text. We consider four types of augmentation operations used in (Wei
and Zou, 2019), including synonym replacement, random insertion, random swap,
and random deletion. Synonym replacement randomly chooses 10% of non-stop
tokens from original texts and replaces each of them with a randomly selected
synonym. In random insertion, for a randomly chosen non-stop token in a text,
among the synonyms of this token, one randomly selected synonym is inserted
into a random position in the text. This operation is performed for 10% of
tokens. Synonyms for synonym replacement and random insertion are obtained
from Synsets in NLTK (Bird and Loper, 2004) which are constructed based on
WordNet (Miller, 1995). Synsets serve as a synonym dictionary containing
groupings of synonymous words. Some words have only one Synset and some have
several. In synonym replacement, if a selected word in a sentence has multiple
synonyms, we randomly choose one of them, and replace all occurrences of this
word in the sentence with this synonym. Random swap randomly chooses two
tokens in a text and swaps their positions. This operation is performed for
10% of token pairs. Random deletion randomly removes a token with a
probability of 0.1. In this SSL task, an augmented sentence is fed into a text
encoder and the encoding is fed into a 4-way classification head to predict
which operation was applied to generate this augmented sentence.

###  3.3 Text Encoder

We use a Transformer encoder to perform the study while noting that other text
encoders are also applicable. Different from sequence-to-sequence models
Sutskever et al. (2014) that are based on recurrent neural networks (e.g.,
LSTM Hochreiter and Schmidhuber (1997), GRU Chung et al. (2014)) which model a
sequence of tokens via a recurrent manner and hence is computationally
inefficient, Transformer eschews recurrent computation and instead uses self-
attention which not only can capture dependency between tokens but also is
amenable for parallel computation with high efficiency. Self-attention
calculates the correlation among every pair of tokens and uses these
correlation scores to create “attentive" representations by taking weighted
summation of tokens’ embeddings. Transformer is composed of building blocks,
each consisting of a self-attention layer and a position-wise feed-forward
layer. Residual connection He et al. (2016) is applied around each of these
two sub-layers, followed by layer normalization Ba et al. (2016). Given an
input sequence, an encoder – which is a stack of such building blocks – is
applied to obtain a representation for each token.

##  4 Experiments

###  4.1 Datasets

We evaluated our method on the datasets used in (Gururangan et al., 2020),
which are from various domains. For each dataset, we follow the
train/development/test split specified in (Gururangan et al., 2020). Dataset
statistics are summarized in Table 1.

In addition, we performed experiments on the datasets in the GLUE benchmark
Wang et al. (2018). The General Language Understanding Evaluation (GLUE)
benchmark has 10 tasks, including 2 single-sentence tasks, 3 similarity and
paraphrase tasks, and 5 inference tasks. For each GLUE task, labels in
development sets are publicly available while those in test sets are not
released. We obtain performance on test sets by submitting inference results
to GLUE evaluation server111https://gluebenchmark.com/leaderboard. Table 2
shows the statistics of data split in each task.

###  4.2 Experimental Setup

####  4.2.1 Baselines

For experiments on datasets used in Gururangan et al. (2020), text encoders in
all methods are initialized using pretrained RoBERTa Liu et al. (2019a). For
experiments on GLUE datasets, text encoders are initialized using pretrained
BERT Liu et al. (2019a) or pretrained RoBERTa. We compare our proposed SSL-Reg
with the following baselines.

  * •

Unregularized RoBERTa Liu et al. (2019b). In this approach, the Transformer
encoder is initialized with pretrained RoBERTa. Then the pretrained encoder
and a classification head form a text classification model, which is then
finetuned on a target classification task. Architecture of the classification
model is the same as that in Liu et al. (2019b). Specifically, representation
of the [CLS] special token is passed to a feedforward layer for class
prediction. Nonlinear activation function in the feedforward layer is tanh.
During finetuning, no SSL-based regularization is used. This approach is
evaluated on all datasets used in Gururangan et al. (2020) and all datasets in
GLUE.

  * •

Unregularized BERT. This approach is the same as unregularized RoBERTa, except
that the Transformer encoder is initialized by pretrained BERT Devlin et al.
(2019a) instead of RoBERTa. This approach is evaluated on all GLUE datasets.

  * •

Task adaptive pretraining (TAPT) Gururangan et al. (2020). In this approach,
given the Transformer encoder pretrained using RoBERTa or BERT on large-scale
external corpora, it is further pretrained by RoBERTa or BERT on input texts
in a target classification dataset (without using class labels). Then this
further pretrained encoder is used to initialize the encoder in the text
classification model and is finetuned to perform classification tasks which
use both input texts and their class labels. Similar to SSL-Reg, TAPT also
performs SSL on texts in target classification dataset. The difference is:
TAPT performs SSL task and classification task sequentially while SSL-Reg
performs these two tasks jointly. TAPT is studied for all datasets in this
paper.

  * •

Domain adaptive pretraining (DAPT) Gururangan et al. (2020). In this approach,
given a pretrained encoder on large-scale external corpora, the encoder is
further pretrained on a small-scale corpora whose domain is similar to that of
texts in a target classification dataset. Then this further pretrained encoder
is finetuned in a classification task. DAPT is similar to TAPT, except that
TAPT performs the second stage pretraining on texts $T$ in the classification
dataset while DAPT performs the second stage pretraining on external texts
whose domain is similar to that of $T$ rather than directly on $T$. The
external dataset is usually much larger than $T$.

  * •

TAPT+SSL-Reg. When finetuning the classification model, SSL-Reg is applied.
The rest is the same as TAPT.

  * •

DAPT+SSL-Reg. When finetuning the classification model, SSL-Reg is applied.
The rest is the same as DAPT.

Task | Epoch |  Learning Rate |  Regularization Parameter
---|---|---|---
CoLA | 10 | 3e-5 | 0.2
SST-2 | 3 | 3e-5 | 0.05
MRPC | 5 | 4e-5 | 0.05
STS-B | 10 | 4e-5 | 0.1
QQP | 5 | 3e-5 | 0.2
MNLI | 3 | 3e-5 | 0.1
QNLI | 4 | 4e-5 | 0.5
RTE | 10 | 3e-5 | 0.1
WNLI | 5 | 5e-5 | 2

Table 3: Hyperparameter settings for BERT on GLUE datasets, where the SSL task
is MTP.

Task | Epoch |  Learning Rate |  Regularization Parameter
---|---|---|---
CoLA | 6 | 3e-5 | 0.4
SST-2 | 3 | 3e-5 | 0.8
MRPC | 5 | 4e-5 | 0.05
STS-B | 10 | 4e-5 | 0.05
QQP | 5 | 3e-5 | 0.4
MNLI | 4 | 3e-5 | 0.5
QNLI | 4 | 4e-5 | 0.05
RTE | 8 | 3e-5 | 0.6
WNLI | 5 | 5e-5 | 0.1

Table 4: Hyperparameter settings for BERT on GLUE datasets, where the SSL task
is SATP.

Task | Epoch |  Learning Rate |  Regularization Parameter
---|---|---|---
CoLA | 10 | 1e-5 | 0.8
SST-2 | 3 | 1e-5 | 1.0
MRPC | 10 | 1e-5 | 0.01
STS-B | 10 | 2e-5 | 0.01
QQP | 10 | 1e-5 | 0.1
MNLI | 3 | 1e-5 | 0.1
QNLI | 3 | 1e-5 | 0.1
RTE | 10 | 2e-5 | 0.1
WNLI | 10 | 2e-5 | 0.02

Table 5: Hyperparameter settings for RoBERTa on GLUE datasets, where the SSL task is MTP. Dataset | RoBERTa | DAPT | TAPT | SSL-Reg | TAPT+SSL-Reg | DAPT+SSL-Reg |
---|---|---|---|---|---|---|---
ChemProt |  81.91.0 |  84.20.2 |  82.60.4 |  83.10.5 |  83.50.1 | 84.40.3 |
RCT |  87.20.1 |  87.60.1 | 87.70.1 |  87.40.1 | 87.70.1 | 87.70.1 |
ACL-ARC |  63.05.8 |  75.42.5 |  67.41.8 |  69.34.9 |  68.12.0 | 75.71.4 |
SciERC |  77.31.9 |  80.81.5 |  79.31.5 |  81.40.8 |  80.40.6 | 82.30.8 |
HyperPartisan |  86.60.9 |  88.25.9 |  90.45.2 |  92.31.4 | 93.21.8 |  90.73.2 |
AGNews |  93.90.2 |  93.90.2 | 94.50.1 |  94.20.1 |  94.40.1 |  94.00.1 |
Helpfulness |  65.13.4 |  66.51.4 |  68.51.9 |  69.40.2 | 71.01.0 |  68.31.4 |
IMDB |  95.00.2 |  95.40.1 |  95.50.1 |  95.70.1 | 96.10.1 |  95.40.1 |
Table 6: Results on datasets used in (Gururangan et al., 2020). For vanilla (unregularized) RoBERTa, DAPT, and TAPT, results are taken from (Gururangan et al., 2020). For each method on each dataset, we run it for four times with different random seeds. Results are in $m_{s}$ format, where $m$ denotes mean and $s$ denotes standard derivation. Following (Gururangan et al., 2020), for ChemProt and RCT, we report micro-F1; for other datasets, we report macro-F1.  Dataset | RoBERTa | SSL-Reg
---|---|---
ChemProt | 13.05 | 13.57
ACL-ARC | 28.67 | 25.24
SciERC | 19.51 | 18.23
HyperPartisan | 7.44 | 5.64
Table 7: Difference between F1 score on training set and F1 score on test set
with or without SSL-Reg (MTP). Bold denotes a smaller difference, which means
overfitting is less severe.

####  4.2.2 Hyperparameter Settings

Hyperparameters were tuned on development datasets.

##### Hyperparameter settings for RoBERTa on datasets used in

(Gururangan et al., 2020). For a fair comparison, most of our hyperparameters
are the same as those in (Gururangan et al., 2020). The maximum text length
was set to 512. Text encoders in all methods are initialized using pretrained
RoBERTa Liu et al. (2019a) on a large-scale external dataset. For TAPT, DAPT,
TAPT+SSL-Reg, and DAPT+SSL-Reg, the second-stage pretraining on texts $T$ in a
target classification dataset or on external texts whose domain is similar to
that of $T$ is based on the pretraining approach in RoBERTa. In SSL-Reg, the
SSL task is masked token prediction. SSL loss function only considers the
prediction of masked tokens and ignores the prediction of non-masked tokens.
Probability for masking tokens is 0.15. If a token $t$ is chosen to be masked,
80% of the time, we replace $t$ with a special token [MASK]; 10% of the time,
we replace $t$ with a random word; and for the rest 10% of the time, we keep
$t$ unchanged. For the regularization parameter in SSL-Reg, we set it to 0.2
for ACL-ARC, 0.1 for SciERC, ChemProt, AGNews, RCT, Helpfulness, IMDB, and
0.01 for HyperPartisan. For ACL-ARC, ChemProt, RCT, SciERC and HyperPartisan,
we trained SSL-Reg for 10 epochs; for Helpfulness, 5 epochs; for AGNews, RCT
and IMDB, 3 epochs. For all datasets, we used a batch size of 16 with gradient
accumulation. We used the AdamW optimizer (Loshchilov and Hutter, 2017) with a
warm-up proportion of 0.06, a weight decay of 0.1, and an epsilon of 1e-6. In
AdamW, $\beta_{1}$ and $\beta_{2}$ are set to 0.9 and 0.98, respectively. The
maximum learning rate was 2e-5.

##### Hyperparameter settings for BERT on GLUE datasets.

The maximum text length was set to 128. Since external texts whose domains are
similar to those of the GLUE texts are not available, we did not compare with
DAPT and DAPT+SSL-Reg. For each method applied, text encoder is initialized
using pretrained BERT Devlin et al. (2019a) (with 24 layers) on a large-scale
external dataset. In TAPT, the second-stage pretraining is performed using
BERT. As we will show later on, TAPT does not perform well on GLUE datasets;
therefore, we did not apply TAPT+SSL-Reg on these datasets further. In SSL-
Reg, we studied two SSL tasks: masked token prediction (MTP) and sentence
augmentation type prediction (SATP). In MTP, we randomly mask 15% of tokens in
each text. Batch size was set to 32 with gradient accumulation. We use the
AdamW optimizer (Loshchilov and Hutter, 2017) with a warm-up proportion of
0.1, a weight decay of 0.01, and an epsilon of 1e-8. In AdamW, $\beta_{1}$ and
$\beta_{2}$ are set to 0.9 and 0.999, respectively. Other hyperparameter
settings are presented in Table 3 and Table 4.

##### Hyperparameter settings for RoBERTa on GLUE datasets.

Most hyperparameter settings follow those in RoBERTa experiments performed on
datasets used in (Gururangan et al., 2020).

We set different learning rates and different epoch numbers for different
datasets as guided by Liu et al. (2019b). In addition, we set different
regularization parameters for different datasets. These hyperparameters are
listed in Table 5.

Figure 2: Training dynamics of unregularized RoBERTa and SSL-Reg (denoted by
“Regularized”) on HyperPartisan and ACL-ARC. In SSL-Reg, we experimented with
two values of the regularization parameter $\lambda$: 0.1 and 1.

###  4.3 Results

####  4.3.1 Results on the datasets used in Gururangan et al. (2020).

Performance of text classification on datasets used in Gururangan et al.
(2020) is reported in Table 6. Following (Gururangan et al., 2020), for
ChemProt and RCT, we report micro-F1; for other datasets, we report macro-F1.
From this table, we make the following observations. First, SSL-Reg
outperforms unregularized RoBERTa significantly on all datasets. We used a
double-sided t-test to perform significance tests. The p-values are less than
0.01, which indicate strong statistical significance. This demonstrates the
effectiveness of our proposed SSL-Reg approach in alleviating overfitting and
improving generalization performance. To further confirm this, we measure the
difference between F1 scores on the training set and test set in Table 7. A
larger difference implies more overfitting: performing well on the training
set and less well on the test set. As can be seen, the train-test difference
under SSL-Reg is smaller than that under RoBERTa. SSL-Reg encourages text
encoders to solve an additional task based on SSL, which reduces the risk of
overfitting to the data-deficient classification task on small-sized training
data. In Figure 2, we compare the training dynamics of unregularized RoBERTa
and SSL-Reg (denoted by “Regularized"). As can be seen, under a large
regularization parameter $\lambda=1$, our method achieves smaller differences
between training accuracy and validation accuracy than unregularized RoBERTa;
our method also achieves smaller differences between training accuracy and
test accuracy than unregularized RoBERTa. These results show that our proposed
SSL-Reg indeed acts as a regularizer which reduces the gap between
performances on training set and validation/test set. Besides, when increasing
$\lambda$ from 0.1 to 1, the training accuracy of SSL-Reg decreases
considerably. This also indicates that SSL-Reg acts as a regularizer which
penalizes training performance. Second, on 6 out of the 8 datasets, SSL-Reg
performs better than TAPT. On the other two datasets, SSL-Reg is on par with
TAPT. This shows that SSL-Reg is more effective than TAPT. SSL-Reg and TAPT
both leverage input texts in classification datasets for self-supervised
learning. The difference is: TAPT uses these texts to pretrain the encoder
while SSL-Reg uses these texts to regularize the encoder during finetuning. In
SSL-Reg, the encoder is learned to perform classification tasks and SSL tasks
simultaneously. Thus the encoder is not completely biased to classification
tasks. In TAPT, the encoder is first learned by performing SSL tasks, then
finetuned by performing classification tasks. There is a risk that after
finetuning, the encoder is largely biased to classification tasks on small-
sized training data, which leads to overfitting. Third, on 5 out of the 8
datasets, SSL-Reg performs better than DAPT, although DAPT leverages
additional external data. The reasons are two-fold: 1) similar to TAPT, DAPT
performs SSL task first and then classification task separately; as a result,
the encoder may be eventually biased to classification task on small-sized
training data; 2) external data used in DAPT still has a domain shift with
target dataset; this domain shift may render the text encoder pretrained on
external data not suitable for target task. To verify this, we measure the
domain similarity between external texts and target texts by calculating
cosine similarity between the BERT embeddings of these texts. The similarity
score is 0.14. As a reference, the similarity score between texts in the
target dataset is 0.27. This shows that there is indeed a domain difference
between external texts and target texts. Fourth, on 6 out of 8 datasets,
TAPT+SSL-Reg performs better than TAPT. On the other two datasets, TAPT+SSL-
Reg is on par with TAPT. This further demonstrates the effectiveness of SSL-
Reg. Fifth, on all eight datasets, DAPT+SSL-Reg performs better than DAPT.
This again shows that SSL-Reg is effective. Sixth, on 6 out of 8 datasets,
TAPT+SSL-Reg performs better than SSL-Reg, indicating that it is beneficial to
use both TAPT and SSL-Reg: first use the target texts to pretrain the encoder
based on SSL, then apply SSL-based regularizer on these target texts during
finetuning. Seventh, DAPT+SSL-Reg performs better than SSL-Reg on 4 datasets,
but worse on the other 4 datasets, indicating that with SSL-Reg used, DAPT is
not necessarily useful. Eighth, on smaller datasets, improvement achieved by
SSL-Reg over baselines is larger. For example, on HyperPartisan which has only
about 500 training examples, improvement of SSL-Reg over RoBERTa is 5.7%
(absolute percentage). Relative improvement is 6.6%. As another example, on
ACL-ARC which has about 1700 training examples, improvement of SSL-Reg over
RoBERTa is 6.3% (absolute percentage). Relative improvement is 10%. In
contrast, on large datasets such as RCT which contains about 180000 training
examples, improvement of SSL-Reg over RoBERTa is 0.2% (absolute percentage).
Relative improvement is 0.2%. On another large dataset AGNews which contains
115000 training examples, improvement of SSL-Reg over RoBERTa is 0.3%
(absolute percentage). Relative improvement is 0.3%. The reason that SSL-Reg
achieves better improvement on smaller datasets is that smaller datasets are
more likely to lead to overfitting and SSL-Reg is more needed to alleviate
this overfitting.

Figure 3: How regularization parameter in SSL-Reg affects text classification F1 score. |  CoLA (Matthew Corr.) |  SST-2 (Accuracy) |  RTE (Accuracy) |  QNLI (Accuracy) |  MRPC (Accuracy/F1)
---|---|---|---|---|---
_The median result_ |  |  |  |  |
BERT, Lan et al. 2019 | 60.6 | 93.2 | 70.4 | 92.3 | 88.0/-
BERT, our run | 62.1 | 93.1 | 74.0 | 92.1 | 86.8/90.8
TAPT | 61.2 | 93.1 | 74.0 | 92.0 | 85.3/89.8
SSL-Reg (SATP) | 63.7 | 93.9 | 74.7 | 92.3 | 86.5/90.3
SSL-Reg (MTP) | 63.8 | 93.8 | 74.7 | 92.6 | 87.3/90.9
_The best result_ |  |  |  |  |
BERT, our run | 63.9 | 93.3 | 75.8 | 92.5 | 89.5/92.6
TAPT | 62.0 | 93.9 | 76.2 | 92.4 | 86.5/90.7
SSL-Reg (SATP) | 65.3 | 94.6 | 78.0 | 92.8 | 88.5/91.9
SSL-Reg (MTP) | 66.3 | 94.7 | 78.0 | 93.1 |  89.5/92.4
Table 8: Results of BERT-based experiments on GLUE development sets, where results on MNLI and QQP are the median of five runs and results on other datasets are the median of nine runs. The size of MNLI and QQP is very large, taking a long time to train on. Therefore, we reduced the number of runs. Because we used a different optimization method to re-implement BERT, our median performance is not the same as that reported in Lan et al. (2019).  |  MNLI-m/mm (Accuracy) |  QQP (Accuracy/F1) |  STS-B (Pearson Corr./Spearman Corr.) |  WNLI (Accuracy)
---|---|---|---|---
_The median result_ |  |  |  |
BERT, Lan et al. 2019 | 86.6/- | 91.3/- | 90.0/- | -
BERT, our run | 86.2/86.0 | 91.3/88.3 | 90.4/90.0 | 56.3
TAPT | 85.6/85.5 | 91.5/88.7 | 90.6/90.2 | 53.5
SSL-Reg (SATP) | 86.2/86.2 | 91.6/88.8 | 90.7/90.4 | 56.3
SSL-Reg (MTP) | 86.6/86.6 | 91.8/89.0 | 90.7/90.3 | 56.3
_The best result_ |  |  |  |
BERT, our run | 86.4/86.3 | 91.4/88.4 | 90.9/90.5 | 56.3
TAPT | 85.7/85.7 | 91.7/89.0 | 90.8/90.4 | 56.3
SSL-Reg (SATP) | 86.4/86.5 | 91.8/88.9 | 91.1/90.8 | 59.2
SSL-Reg (MTP) | 86.9/86.9 | 91.9/89.1 | 91.1/90.8 | 57.7
Table 9:  Continuation of Table 8.  | BERT | TAPT | SSL-Reg (SATP) | SSL-Reg (MTP)
---|---|---|---|---
CoLA (Matthew Corr.) | 60.5 | 61.3 | 63.0 | 61.2
SST-2 (Accuracy) | 94.9 | 94.4 | 95.1 | 95.2
RTE (Accuracy) | 70.1 | 70.3 | 71.2 | 72.7
QNLI (Accuracy) | 92.7 | 92.4 | 92.5 | 93.2
MRPC (Accuracy/F1) | 85.4/89.3 | 85.9/89.5 | 85.3/89.3 | 86.1/89.8
MNLI-m/mm (Accuracy) |  86.7/85.9 | 85.7/84.4 | 86.2/85.4 | 86.6/86.1
QQP (Accuracy/F1) | 89.3/72.1 | 89.3/71.6 | 89.6/72.2 | 89.7/72.5
STS-B (Pearson Corr./Spearman Corr.) | 87.6/86.5 |  88.4/87.3 | 88.3/87.5 | 88.1/87.2
WNLI (Accuracy) | 65.1 | 65.8 | 65.8 | 66.4
AX(Matthew Corr.) | 39.6 | 39.3 | 40.2 | 40.3
Average | 80.5 | 80.6 | 81.0 | 81.3
Table 10: Results of BERT-based experiments on GLUE test sets, which are scored by the GLUE evaluation server (https://gluebenchmark.com/leaderboard). Models evaluated on AX are trained on the training dataset of MNLI.  |  CoLA (Matthew Corr.) |  SST-2 (Accuracy) |  RTE (Accuracy) |  QNLI (Accuracy) |  MRPC (Accuracy/F1)
---|---|---|---|---|---
_The median result_ |  |  |  |  |
RoBERTa, Liu et al. 2019 | 68.0 | 96.4 | 86.6 | 94.7 | 90.9/-
RoBERTa, our run | 68.7 | 96.1 | 84.8 | 94.6 | 89.5/92.3
SSL-Reg (MTP) | 69.2 | 96.3 | 85.2 | 94.9 |  90.0/92.7
_The best result_ |  |  |  |  |
RoBERTa, our run | 69.2 | 96.7 | 86.6 | 94.7 | 90.4/93.1
SSL-Reg (MTP) | 70.2 | 96.7 | 86.6 | 95.2 |  91.4/93.8
Table 11: Results of RoBERTa-based experiments on GLUE development sets, where the median results are the median of five runs. Because we used a different optimization method to re-implement RoBERTa, our median performance is not the same as that reported in Liu et al. (2019b).  |  MNLI-m/mm (Accuracy) |  QQP (Accuracy) |  STS-B (Pearson Corr./Spearman Corr.) |  WNLI (Accuracy)
---|---|---|---|---
_The median result_ |  |  |  |
RoBERTa, Liu et al. 2019 | 90.2/90.2 | 92.2 | 92.4/- | -
RoBERTa, our run | 90.5/90.5 | 91.6 | 92.0/92.0 | 56.3
SSL-Reg (MTP) | 90.7/90.7 | 91.6 | 92.0/92.0 | 62.0
_The best result_ |  |  |  |
RoBERTa, our run | 90.7/90.5 | 91.7 | 92.3/92.2 | 60.6
SSL-Reg (MTP) | 90.7/90.5 | 91.8 | 92.3/92.2 | 66.2
Table 12:  Continuation of Table 11.  |  CoLA |  SST-2 |  RTE |  QNLI |  MRPC |  STS-B
---|---|---|---|---|---|---
SR+RD+RI+RS | 63.6 | 94.0 | 74.8 | 92.2 | 86.8/90.6 | 90.6/90.3
SR+RD+RI | 63.4 | 93.8 | 72.8 | 92.1 | 86.9/90.8 | 90.6/90.2
SR+RD | 61.6 | 93.6 | 72.5 | 92.2 |  87.2/91.0 |  90.6/90.3
Table 13: Ablation study on sentence augmentation types in SSL-Reg (SATP),
where SR, RD, RI and RS denotes synonym replacement, random deletion, random
insertion, and random swap respectively. Results are averaged over 5 runs with
different random initialization.

Figure 3 shows how classification F1 score varies as we increase
regularization parameter $\lambda$ from 0.01 to 1.0 in SSL-Reg. As can be
seen, starting from 0.01, when the regularizer parameter is increasing, F1
score increases. This is because a larger $\lambda$ imposes a stronger
regularization effect, which helps to reduce overfitting. However, if
$\lambda$ becomes too large, F1 score drops. This is because the
regularization effect is too strong, which dominates classification loss.
Among these 4 datasets, F1 score drops dramatically on HyperPartisan as
$\lambda$ increases. This is probably because this dataset contains very long
sequences. This makes MTP on this dataset more difficult and therefore yields
an excessively strong regularization outcome that hurts classification
performance. Compared with HyperPartisan, F1 score is less sensitive on other
datasets because their sequence lengths are relatively smaller.

####  4.3.2 Results on the GLUE benchmark

Table 8 and Table 9 show results of BERT-based experiments on development sets
of GLUE. As mentioned in (Devlin et al., 2019b), for the 24-layer version of
BERT, finetuning is sometimes unstable on small datasets, so we run each
method several times and report the median and best performance. Table 10
shows the best performance on test sets. Following Wang et al. (2018), we
report Matthew correlation on CoLA, Pearson correlation and Spearman
correlation on STS-B, accuracy and F1 on MRPC and QQP. For the rest datasets,
we report accuracy. From these tables, we make the following observations.
First, SSL-Reg methods including SSL-Reg-SATP and SSL-Reg-MTP outperform
unregularized BERT (our run) on most datasets: 1) on test sets, SSL-Reg-SATP
performs better than BERT on 7 out of 10 datasets and SSL-Reg-MTP performs
better than BERT on 9 out of 10 datasets; 2) in terms of median results on
development sets, SSL-Reg-SATP performs better than BERT (our run) on 7 out of
9 datasets and SSL-Reg-MTP performs better than BERT (our run) on 8 out of 9
datasets; 3) in terms of best results on development sets, SSL-Reg-SATP
performs better than BERT (our run) on 8 out of 9 datasets and SSL-Reg-MTP
performs better than BERT (our run) on 8 out of 9 datasets. This further
demonstrates the effectiveness of SSL-Reg in improving generalization
performance. Second, on 7 out of 10 test sets, SSL-Reg-SATP outperforms TAPT;
on 8 out of 10 test sets, SSL-Reg-MTP outperforms TAPT. On most development
datasets, SSL-Reg-SATP and SSL-Reg-MTP outperform TAPT. The only exception is:
on QQP development set, the best F1 of TAPT is slightly better than that of
SSL-Reg-SATP. This further demonstrates that performing SSL-based
regularization on target texts is more effective than using them for
pretraining. Third, overall, SSL-Reg-MTP performs better than SSL-Reg-SATP.
For example, on 8 out of 10 test datasets, SSL-Reg-MTP performs better than
SSL-Reg-SATP. MTP works better than SATP probably because it is a more
challenging self-supervised learning task that encourages encoders to learn
more powerful representations. Fourth, improvement of SSL-Reg methods over
BERT is more prominent on smaller training datasets, such as CoLA and RTE.
This may be because smaller training datasets are more likely to lead to
overfitting where the advantage of SSL-Reg in alleviating overfitting can be
better played.

Table 11 and 12 show results of RoBERTa-based experiments on development sets
of GLUE. From these two tables, we make observations that are similar to those
in Table 8 and Table 9. In terms of median results, SSL-Reg (MTP) performs
better than unregularized RoBERTa (our run) on 7 out of 9 datasets and
achieves the same performance as RoBERTa (our run) on the rest 2 datasets. In
terms of best results, SSL-Reg (MTP) performs better than RoBERTa (our run) on
5 out of 9 datasets and achieves the same performance as RoBERTa (our run) on
the rest 4 datasets. This further demonstrates the effectiveness of our
proposed SSL-Reg approach which uses an MTP-based self-supervised task to
regularize the finetuning of RoBERTa.

In SSL-Reg (SATP), we perform an ablation study on different types of sentence
augmentation. Results are shown in Table 13, where SR, RD, RI and RS denote
synonym replacement, random deletion, random insertion, and random swap,
respectively. SR+RD+RI+RS means that we apply these four types of operations
to augment sentences; given an augmented sentence $a$, we predict which of the
four types of operations was applied to an original sentence to create $a$.
SR+RD+RI+RS and SR+RD hold similar meanings. From this table, we make the
following observations. First, as the number of augmentation types increases
from 2 (SR+RD) to 3 (SR+RD+RI) then to 4 (SR+RD+RI+RS), the performance
increases in general. This shows that it is beneficial to have more
augmentation types in SATP. The reason is that more types make the SATP task
more challenging and solving a more challenging self-supervised learning task
can enforce sentence encoders to learn more powerful representations. Second,
SR+RD+RI+RS outperforms SR+RD+RI on 5 out of 6 datasets. This demonstrates
that leveraging random swap (RS) for SATP can learn more effective
representations of sentences. The reason is: SR, RD, and RI change the
collection of tokens in a sentence via synonym replacement, random deletion,
and random insertion; RS does not change the collection of tokens, but changes
the order of tokens; therefore, RS is complementary to the other three
operations; adding RS can bring in additional benefits that are complementary
to those of SR, RD, and RI. Third, SR+RD+RI performs much better than SR+RD on
CoLA and is on par with SR+RD on the rest five datasets. This shows that
adding RI to SR+RD is beneficial. Unlike synonym replacement (SR) and random
deletion (RD) which do not increase the number of tokens in a sentence, RI
increases token number. Therefore, RI is complementary to SR and RD and can
bring in additional benefits.

##  5 Conclusions and Future Work

In this paper, we propose to use self-supervised learning to alleviate
overfitting in text classification problems. We propose SSL-Reg, which is a
regularizer based on SSL and a text encoder is trained to simultaneously
minimize classification loss and regularization loss. We demonstrate the
effectiveness of our methods on 17 text classification datasets.

For future works, we will use other self-supervised learning tasks to perform
regularization, such as contrastive learning, which predicts whether two
augmented sentences stem from the same original sentence.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems_ , pages 15509–15519.
  * Bird and Loper (2004) Steven Bird and Edward Loper. 2004.  NLTK: The natural language toolkit.  In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_ , pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_.
  * Dernoncourt and Lee (2017) Franck Dernoncourt and Ji Young Lee. 2017.  Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts.  In _IJCNLP_.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _NAACL-HLT_.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020.  Cert: Contrastive self-supervised learning for language understanding.  _arXiv e-prints_ , pages arXiv–2005.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of ACL_.
  * He et al. (2019) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019.  Momentum contrast for unsupervised visual representation learning.  _arXiv preprint arXiv:1911.05722_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * He et al. (2020a) Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020a.  Pathological visual question answering.  _arXiv preprint arXiv:2010.12435_.
  * He et al. (2020b) Xuehai He, Xingyi Yang, Shanghang Zhang, Jinyu Zhao, Yichen Zhang, Eric Xing, and Pengtao Xie. 2020b.  Sample-efficient deep learning for covid-19 diagnosis based on ct scans.  _medRxiv_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Howard and Ruder (2018) J. Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _ACL_.
  * Jurgens et al. (2018) David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018.  Measuring the evolution of a scientific field through citation frames.  _TACL_.
  * Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014.  A convolutional neural network for modelling sentences.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 655–665, Baltimore, Maryland. Association for Computational Linguistics.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019.  SemEval-2019 Task 4: Hyperpartisan news detection.  In _SemEval_.
  * Klein and Nabi (2020) Tassilo Klein and Moin Nabi. 2020.  Contrastive self-supervised learning for commonsense reasoning.  _arXiv preprint arXiv:2005.00669_.
  * Korde and Mahender (2012) V. Korde and C. Mahender. 2012.  Text classification and classifiers: A survey.  _International Journal of Artificial Intelligence & Applications_, 3:85–99.
  * Kringelum et al. (2016) Jens Kringelum, Sonny Kim Kjærulff, Søren Brunak, Ole Lund, Tudor I. Oprea, and Olivier Taboureau. 2016.  ChemProt-3.0: a global chemical biology diseases mapping.  In _Database_.
  * Lai et al. (2015) Siwei Lai, L. Xu, Kang Liu, and Jun Zhao. 2015.  Recurrent convolutional neural networks for text classification.  In _AAAI_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2019) Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. 2019.  Joint-task self-supervised learning for temporal correspondence.  In _Advances in Neural Information Processing Systems_ , pages 317–327.
  * Liu et al. (2016) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016.  Recurrent neural network for text classification with multi-task learning.  _arXiv preprint arXiv:1605.05101_.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A robustly optimized BERT pretraining approach.  arXiv:1907.11692.
  * Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter. 2017.  Fixing weight decay regularization in adam.  _ArXiv_ , abs/1711.05101.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _EMNLP_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _ACL_.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015\.  Image-based recommendations on styles and substitutes.  In _ACM SIGIR_.
  * Miller (1995) G. Miller. 1995.  Wordnet: a lexical database for english.  _Commun. ACM_ , 38:39–41.
  * Minaee et al. (2020) Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2020.  Deep learning based text classification: A comprehensive review.  _arXiv preprint arXiv:2004.03705_.
  * Nathan Mundhenk et al. (2018) T Nathan Mundhenk, Daniel Ho, and Barry Y Chen. 2018.  Improvements to context based self-supervised learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9339–9348.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pathak et al. (2016) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2536–2544.
  * (41) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Roweis and Saul (2000) Sam T Roweis and Lawrence K Saul. 2000.  Nonlinear dimensionality reduction by locally linear embedding.  _science_ , 290(5500):2323–2326.
  * Srinivas et al. (2020) Aravind Srinivas, Michael Laskin, and Pieter Abbeel. 2020.  Curl: Contrastive unsupervised representations for reinforcement learning.  _arXiv preprint arXiv:2004.04136_.
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019.  Ernie 2.0: A continual pre-training framework for language understanding.  _arXiv preprint arXiv:1907.12412_.
  * Sun et al. (2020) Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. 2020.  Test-time training with self-supervision for generalization under distribution shifts.  In _ICML_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , pages 3104–3112.
  * Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015.  Improved semantic representations from tree-structured long short-term memory networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1556–1566, Beijing, China. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wang et al. (2017) Jin Wang, Zhongyuan Wang, D. Zhang, and Jun Yan. 2017.  Combining knowledge with deep convolutional neural networks for short text classification.  In _IJCAI_.
  * Wang et al. (2019a) Xiaolong Wang, Allan Jabri, and Alexei A Efros. 2019a.  Learning correspondence from the cycle-consistency of time.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2566–2576.
  * Wang et al. (2019b) Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019b.  Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6629–6638.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6383–6389, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2019) Qingyang Wu, Lei Li, Hao Zhou, Ying Zeng, and Zhou Yu. 2019.  Importance-aware learning for neural headline editing.  _arXiv preprint arXiv:1912.01114_.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3733–3742.
  * Yang et al. (2020) Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang, Shanghang Zhang, and Pengtao Xie. 2020.  Transfer learning or self-supervised learning? a tale of two pretraining paradigms.  _arXiv preprint arXiv:2007.04234_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5754–5764.
  * Zeng and Xie (2021) Jiaqi Zeng and Pengtao Xie. 2021.  Contrastive self-supervised learning for graph classification.  _AAAI_.
  * Zhang et al. (2016) Richard Zhang, Phillip Isola, and Alexei A Efros. 2016.  Colorful image colorization.  In _European conference on computer vision_ , pages 649–666. Springer.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _NeurIPS_.
  * Zhou et al. (2015) Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and F. C. M. Lau. 2015.  A c-lstm neural network for text classification.  _ArXiv_ , abs/1511.08630.


# Evidence of Dark Contents in the Center of NGC 6517

Yi Xie 1 Dejiang Yin2 Lichun Wang1 Yujie Lian3,4 Liyun Zhang2 Zhichen Pan5
1School of Science, Jimei University, Xiamen 361021, Fujian Province, China
2College of Physics, Guizhou University, Guiyang 550025, China
3Institute for Frontiers in Astronomy and Astrophysics, Beijing Normal
University, Beijing 102206, China
4Department of Astronomy, Beijing Normal University, Beijing 100875, China
5National Astronomical Observatories, Chinese Academy of Sciences, Beijing
100101, China
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Millisecond pulsars can serve as effective probes to investigate the presence
of Intermediate-mass Black Holes (IMBHs) within Galactic globular clusters
(GCs). Based on the standard structure models for GCs, we conduct simulations
to analyze the distributions of pulsar accelerations within the central region
of NGC 6517. By comparing the measured accelerations of pulsars obtained from
their period derivatives $\dot{P}$ to the simulated distribution profiles, we
demonstrate that a central excess of dark mass is required to account for the
measured accelerations. Our analysis, which relies on existing pulsar timing
observations, is currently unable to differentiate between two possible
scenarios: an IMBH precisely situated at the core of the cluster with mass
$\gtrsim 9000^{+4000}_{-3000}~{}M_{\odot}$, or a central concentration of
stellar mass dark remnants with a comparable total mass. However, with
additional acceleration measurements from a few more pulsars in the cluster,
it will be possible to differentiate the source of the nonluminous matter.

###### keywords:

(stars:) pulsars: general – (Galaxy:) globular clusters: general – stars:
black holes

††pubyear: 2022††pagerange: Evidence of Dark Contents in the Center of NGC
6517–Evidence of Dark Contents in the Center of NGC 6517

##  1 Introduction

Intermediate-mass black holes (IMBHs), possessing masses within the range of
$10^{2}$ and $10^{5}M_{\odot}$, are the critical bridge to connect stellar-
mass black holes (BHs) and supermassive BHs. Although conclusive evidence for
the existence of IMBHs is still lacking, they are proposed to be situated
within dwarf galaxies (Greene & Ho, 2004; Baldassare et al., 2015), nearby
globular clusters (GCs) (Portegies Zwart et al., 2004; Kızıltan et al., 2017),
or appearing as ultraluminous X-ray sources (Madau & Rees, 2001; Farrell et
al., 2009). The existence of IMBHs within GCs has been extensively
investigated for a long time (Silk & Arons, 1975; Bahcall & Ostriker, 1975).
Theoretical models suggested that IMBHs could be formed as a result of runaway
core collapse and subsequent merger of stars within highly dense star clusters
(Portegies Zwart & McMillan, 2002). Therefore, the central regions of GCs
present an ideal environment for the formation of IMBHs, given the high
frequency of stellar collisions and mergers (Miller & Hamilton, 2002; Gürkan
et al., 2004).

Three types of evidence have been widely invoked to demonstrate the existence
of IMBHs in GCs. Firstly, surface photometry for the Galactic GCs discovered
that about one fifth of the known GCs display indications of a collapsed core
(Djorgovski & King, 1986). In other word, their surface brightness profiles
exhibit a central power-law cusp. A potential interpretation could be that the
IMBHs might reside at the core of cusp clusters (Newell et al., 1976; Noyola
et al., 2008). Through optical observations, one can also measure various
aspects of projected density and kinematical data, including projected
position, line-of-sight velocity and proper motion of the stars. A
comprehensive dynamical analysis of the data has inferred upper limits on the
mass of IMBHs and led to several tentative detections (Gebhardt et al., 2000;
van der Marel et al., 2002; McLaughlin et al., 2006; Lanzoni et al., 2007; van
der Marel & Anderson, 2010).

The second line of evidence relies on observations in the radio and X-ray
region. Numerous investigations focused on identifying IMBHs within GCs by
analyzing their distinctive radiation signatures during the accretion phase
(Gebhardt et al., 2002; Maccarone, 2004; Ulvestad et al., 2007; Bash et al.,
2008). However, the gas and dust within GCs are extremely sparse (Spergel,
1991; Freire et al., 2001), which may cause a very faint accretion signature.
Consequently, only a few debatable findings have been acquired (Ulvestad et
al., 2007; Maccarone & Servillat, 2008; Lu & Kong, 2011; Tremou et al., 2018;
Häberle et al., 2021). A convincing evidence came from a luminous X-ray
outburst in a massive star cluster, resulting from a tidal disruption event
and subsequent accretion. The analysis of the spectra and light curve from
this event demonstrates that the extragalactic cluster hosts an IMBH with a
mass of $\sim 10^{4}~{}M_{\odot}$ (Lin et al., 2018).

The third piece of evidence comes from pulsar timing observations. To date,
305 pulsars have been discovered in 40 Galactic GCs111https://www3.mpifr-
bonn.mpg.de/staff/pfreire/GCpsr.html for an up-to-date list, with the
overwhelming majority being millisecond pulsars (MSPs). This number continues
to increase steadily. Pulsars are powerful tools to probe the GC dynamics
(Blandford et al., 1987; Phinney, 1992). Owing to their remarkably stable spin
periods, measurements of pulsar acceleration and the local GC potential are
feasible through the Doppler effect (Phinney, 1993; Perera et al., 2017;
Kızıltan et al., 2017; Freire et al., 2017; Prager et al., 2017; Abbate et
al., 2019). The high-eccentricity orbit of PSR B1820-30A indicates that an
IMBH with a mass $>7500~{}M_{\odot}$ could potentially be located at the
center of NGC 6624 (Perera et al., 2017). The measurement of the gravitational
potential in the GC 47 Tucanae, using pulsars as indicators, offers evidence
for a central IMBH with an estimated mass of $2300^{+1500}_{-850}~{}M_{\odot}$
(Kızıltan et al., 2017). By analyzing the measured accelerations of the
pulsars in NGC 6266, Abbate et al. (2019) found that the center of the cluster
displays an excess of dark mass ranging from $1200$ to $6000~{}M_{\odot}$.
However, from a modeling perspective, it is worth noting that the central
concentration of the stellar mass dark remnants may also account for all of
the IMBH signatures identified in these investigations.

In this study, we concentrate on the GC NGC 6517, which is a metal-poor bulge
cluster situated approximately $4.2~{}{\rm kpc}$ from the Galactic Center and
around $10.6~{}{\rm kpc}$ from the Sun (Harris, 1996). So far, there is no
positive report on the presence of an IMBH in this cluster. This includes all
available data from optical, X-ray, and radio observations, as well as the
pulsar timing method. NGC 6517 contains 15 confirmed radio pulsars, 9 of which
have both period and period time derivative measurements (Lynch et al., 2011;
Pan et al., 2021a, b). Five pulsars are located very close to the center of
the GC ($\lesssim 0.2~{}{\rm pc}$), which might be considerably influenced by
the possible IMBH in the center. We perform simulations to analyze the
acceleration distributions of pulsars within the cluster, by utilizing optical
observation data for the core radius and central density of the cluster
(Harris, 1996; Baumgardt & Hilker, 2018). We compare the simulated results
with the measured accelerations of the nine pulsars, which can offer a lower
limit on the mass for the IMBH or central dark remnants. In Section 2, we
outline the simulation methods, followed by the presentation of fitting
results in Section 3. Finally, a summary of our findings can be found in
Section 4.

##  2 Methods

In this study, a coordinate system is proposed to determine the positions of
pulsars in the GC NGC 6517, as shown in Figure 1. The plane denoted as $O$ is
defined as the one perpendicular to the line-of-sight and passing through the
center of gravity (COG) of the cluster. The separation along the line-of-sight
direction between the pulsar and plane $O$ is represented by the variable $l$.
The projected distance on the plane of the sky from the pulsar to the COG is
indicated as $R_{\perp}^{\prime}$. Finally, the radial distance of the pulsar
from the COG is referred to as $r^{\prime}$.

Figure 1: Schematic illustration indicating the placement of a pulsar in
relation to the central region of the cluster NGC 6517. The optical image of
the cluster comes from the _Hubble Space Telescope_ (https://hla.stsci.edu/).
The plane intersecting the cluster’s COG and standing perpendicular to our
line-of-sight is defined as plane $O$. The pulsar’s line-of-sight distance
from this plane is indicated as $l$. The symbol $R_{\perp}^{\prime}$
represents the measurement of the pulsar’s projected distance from the
cluster’s gravitational center, while $r^{\prime}$ signifies the radial
distance between them. Core radius is denoted as $r_{c}$. Accordingly, when
the pulsar settles into the foremost half of the cluster, the GC potential
induces a positive fluctuation in the period’s time derivative. In contrast,
if it positions on the rear half, it will contribute to a negative value.

The gravitational acceleration experienced by a test star, specifically a
pulsar, within GCs stems from the combined influence (mean field) of all other
stars contained in the cluster. The density distribution of these stars,
extending from the center to a few core radii of the GC, can be formulated as
King density profile (King, 1962):

$\rho(r^{\prime})=\rho_{\rm c}[1+(r^{\prime}/r_{\rm c})^{2}]^{-\frac{3}{2}},$
(1)

where $\rho_{\rm c}$ is the core density of the cluster, its core radius is
$r_{\rm c}$.

When an IMBH is located at the center of a cluster, it significantly alters
the spatial distribution of stars surrounding it. The IMBH with a specified
mass $M_{\rm BH}$, has an associated influence radius (Baumgardt et al.,
2004a)

$r_{i}=\frac{3M_{\rm BH}}{8\pi\rho_{c}r_{c}^{2}}.$ (2)

The standard King model governs the star distribution at and beyond $r_{i}$
(up to a few core radii). However, within $r_{i}$, it is the density profile
that complies with the formula (Baumgardt et al., 2004b),

$\rho_{\rm BH}\propto r^{-1.55}.$ (3)

This power law distribution is deemed most suitable when modelling cusp
clusters (Djorgovski & King, 1986).

The gravitational attraction felt by a pulsar at radius $r^{\prime}_{\ast}$ is
generated by the stars enclosed within the spherical volume of radius
$r^{\prime}_{\ast}$. By integrating Equation 1 and 3 in radial direction, we
can determine the total interior mass inside the sphere of radius
$r^{\prime}_{\ast}$. Multiplying $G/r^{\prime 2}_{\ast}$ by the total interior
mass yields the accerelation (Xie & Wang, 2020; Wang & Xie, 2021, 2022)

$a(r^{\prime}_{\ast})=\frac{4\pi G}{r^{\prime 2}_{\ast}}[M_{\rm
BH}/4\pi+\int_{0}^{r_{i}}r^{\prime 2}\rho_{\rm
BH}dr^{\prime}+\int_{r_{i}}^{r^{\prime}_{\ast}}r^{\prime
2}\rho(r^{\prime})dr^{\prime}],$ (4)

in which $G$ is the gravitational constant. The mean field line-of-sight
acceleration $a_{l}$, can be calculated by multiplying $a(r^{\prime}_{\ast})$
with a factor of $l/r^{\prime}_{\ast}$.

To investigate the distributions of $a_{l}$ with relation to
$R_{\perp}^{\prime}$ for pulsars within the central region of the GC, we
accomplish a thorough distribution profile through Monte Carlo simulations.
Pulsars, being among the heavier stellar populations, are predominantly
located in the innermost sector of the cluster due to mass segregation effects
(Spitzer, 1987). This occurrence is well endorsed by notable observations made
within the GCs 47 Tucanae and Terzan 5 (Freire et al., 2017; Prager et al.,
2017). In the observed scenario, the column number density profile associated
with these deposited pulsars fits the description provided by (Lugger et al.,
1995)

$n(x_{\perp})=n_{0}(1+x_{\perp}^{2})^{\beta/2},$ (5)

where the central number density is denoted by $n_{0}$, the projected distance
between the pulsar and the COG is given by $x_{\perp}\equiv
R_{\perp}^{\prime}/r_{c}$. The mass segregation is described by the parameter
$\beta$, which follows a Gaussian distribution centered at $-3$ with a
dispersion of $0.5$ (Abbate et al., 2019; Wang & Xie, 2021).

The time derivative of the pulsar spin period can be greatly influenced by the
line-of-sight accelerations as a result of the Doppler effect. The first
derivative of period $\dot{P}$ relates with the various accelerations in the
following way (Phinney, 1993; Prager et al., 2017; Wang & Xie, 2021, 2022):

$\frac{\dot{P}}{P}=\frac{\dot{P}_{0}}{P_{0}}+\frac{a_{l}}{c}+\frac{a_{\rm
g}}{c}+\frac{a_{\rm s}}{c}+\frac{a_{\rm DM}}{c}$ (6)

in which $\dot{P}_{0}/P_{0}$ is the intrinsic spin-down caused by the magnetic
braking of a pulsar, $a_{l}$ is the line-of-sight acceleration due to GC
potential, $a_{\rm g}$ is the acceleration from the Galactic potential,
$a_{\rm s}$ accounts the centrifugal acceleration due to the Shklovskii
effect, $a_{\rm DM}$ is the apparent acceleration due to the measuring error
from the changes of dispersion measure (DM), and $c$ is the speed of light.

For a standard dipole radiation model, the intrinsic period derivative for a
typical MSP can be estimated as

$\frac{\dot{P}_{0}}{P_{0}}=3.1\times 10^{-10}~{}{\rm yr}^{-1}(\frac{B}{2\times
10^{8}~{}{\rm G}})^{2}(\frac{2~{}{\rm ms}}{P})^{2}$ (7)

where $B$ is the magnetic field strength at the surface of the pulsar.

By substituting typical values for GCs such as NGC 6517, and considering the
case without an IMBH at the core, we can calculate the acceleration $a_{l}$
using Equation 4, which is expressed as follows (Freire et al., 2005; Prager
et al., 2017):

$\begin{split}\frac{a_{l}}{c}=-3.7\times 10^{-9}~{}{\rm
yr}^{-1}(\frac{\rho_{c}}{10^{5}~{}M_{\odot}~{}{\rm
pc}^{-3}})(\frac{l}{0.2~{}{\rm pc}})(\frac{r}{r_{c}})^{-3}\\\
\times[\sinh^{-1}(\frac{r}{r_{c}})-\frac{r}{r_{c}\sqrt{1+(r/r_{c})^{2}}}].\end{split}$
(8)

The acceleration due to the Galactic potential can be derived from the pattern
of Galactic rotation (Nice & Taylor, 1995). We assume a flat rotation curve
and use a distance for the Sun from the Galactic center $R_{0}=8.34\pm
0.16~{}{\rm kpc}$ and rotation velocity $\Theta_{0}=240\pm 8~{}{\rm
km~{}s^{-1}}$ (Sharma et al., 2014). The line-of-sight acceleration between
the Sun and each pulsar is given by (Nice & Taylor, 1995)

$\boldsymbol{a_{\rm
g}}\cdot\boldsymbol{n}=-\cos(b)(\frac{\Theta_{0}^{2}}{R_{0}})[\cos(l)+\frac{\beta}{\sin^{2}(l)+\beta^{2}}],$
(9)

where $\beta=(d/R_{0})\cos(b)-\cos(l)$, and the distance of NGC 6517 from the
Sun is $d=10.6~{}{\rm kpc}$. The Galactic coordinates at COG of the GC are
$l=19^{\circ}.23$ and $b=6^{\circ}.76$. Then we find the acceleration is
$a_{\rm g}/c=-5.75\times 10^{-11}~{}{\rm yr}^{-1}$.

The Shklovskii effect is a centrifugal acceleration due to proper motion of
the pulsar, which reads (Shklovskii, 1970)

$\frac{a_{\rm s}}{c}=8.1\times 10^{-13}~{}{\rm yr}^{-1}(\frac{d}{10.6~{}{\rm
kpc}})(\frac{\mu_{T}}{{\rm mas~{}yr^{-1}}})^{2}.$ (10)

As NGC 6517 is situated close to the Galactic center, the proper motion
$\mu_{T}$ may be estimated as (Prager et al., 2017)

$\mu_{T}=4.7~{}{\rm mas~{}yr^{-1}}(\frac{\Theta_{0}}{240~{}{\rm
km~{}s^{-1}}})(\frac{10.6~{}{\rm kpc}}{d}).$ (11)

The value is in good agreement with the precise proper motion of the cluster
as measured in Vasiliev & Baumgardt (2021), which gives $\mu_{T}\simeq 4.73\pm
0.03~{}{\rm mas~{}yr^{-1}}$. Using Equation 10, we find the value $a_{\rm
s}/c\sim 1.8\times 10^{-11}~{}{\rm yr}^{-1}$.

Table 1: List of Pulsars with Reported Period and its Time Derivative in NGC
6517

Pulsar | $P$ | $\dot{P}$ | $\dot{P}/P$ | $\theta_{\perp}$ | $R_{\perp}^{\prime}(1)$ | $R_{\perp}^{\prime}(2)$ | $R_{\perp}^{\prime}(3)$ | References
---|---|---|---|---|---|---|---|---
| (ms) | (10${}^{-19})$ | ($10^{-9}$ yr-1) | (arcmin) | (pc) | (pc) | (pc) |
A | 7.17 | -5.13 | -2.250 | 0.022 | 0.068 | $0.059\pm 0.004$ | $0.065\pm 0.011$ | 1
B | 28.96 | 21.91 | 2.368 | 0.022 | 0.068 | $0.059\pm 0.004$ | $0.065\pm 0.011$ | 1
C | 3.74 | -0.65 | -0.548 | 0.057 | 0.176 | $0.153\pm 0.009$ | $0.169\pm 0.028$ | 1
D | 4.23 | 0.07 | 0.051 | 1.202 | 3.706 | $3.227\pm 0.196$ | $3.566\pm 0.595$ | 1
E | 7.60 | -10.37 | -4.297 | 0.025 | 0.078 | $0.068\pm 0.004$ | $0.075\pm 0.012$ | 2
F | 24.89 | -26.72 | -3.385 | 0.055 | 0.168 | $0.147\pm 0.008$ | $0.162\pm 0.027$ | 2
G | 51.59 | 0.80 | 0.049 | 0.121 | 0.373 | $0.325\pm 0.020$ | $0.359\pm 0.060$ | 2
H | 5.64 | 0.34 | 0.192 | 0.562 | 1.734 | $1.509\pm 0.091$ | $1.667\pm 0.277$ | 3
I | 3.25 | -0.14 | -0.139 | 0.757 | 2.335 | $2.032\pm 0.124$ | $2.246\pm 0.374$ | 4

Notes. $\theta_{\perp}$ is the angular offset from the centre of the cluster
for the pulsars in NGC 6517. The projected distances, denoted as
$R_{\perp}^{\prime}(1)$, $R_{\perp}^{\prime}(2)$, and $R_{\perp}^{\prime}(3)$,
are computed using $\theta_{\perp}$ and the distinct measurements of the
distance of the cluster from the Sun. Each symbol correlates with specific
distances: $d=10.6~{}{\rm kpc}$, $9.23\pm 0.56~{}{\rm kpc}$, and $10.2\pm
1.7~{}{\rm kpc}$, respectively.

_References:_ (1) Lynch et al. (2011); (2) Pan et al. (2021a); (3) Pan et al.
(2021b); (4) Yin et al. 2023 (in preparation).

The apparent acceleration caused by the stochastic DM error can be expressed
as (Prager et al., 2017)

$\frac{a_{\rm DM}}{c}=-6.4\times 10^{-15}~{}{\rm yr}^{-1}(\frac{\Delta t_{\rm
DM}}{1~{}\rm{\mu s}})(\frac{10~{}\rm yr}{T})^{2}$ (12)

where $\Delta t_{\rm DM}$ is the delay time of the dispersion, $T$ is the
timescale for the DM error measurement. For typical values of $\Delta t_{\rm
DM}=100~{}\rm{\mu s}$ and $T=10~{}\rm yr$, one can find the apparent
acceleration is $a_{\rm DM}/c=-6.4\times 10^{-13}~{}\rm{yr}^{-1}$.

Figure 2: The comparison of the measured $a_{l}$ with the simulated values,
for pulsars in the innermost region of NGC 6517. Upper panel: $d=10.6~{}{\rm
kpc}$, $r_{c}=0.185~{}{\rm pc}$, $\rho_{c}=1.95\times 10^{5}~{}M_{\odot}{\rm
pc}^{-3}$ are taken from optical observation of Harris (1996). Middle panel:
$d=9.23\pm 0.56~{}{\rm kpc}$, $r_{c}=0.14~{}{\rm pc}$, $\rho_{c}=4.79\times
10^{5}~{}M_{\odot}{\rm pc}^{-3}$ are taken from Baumgardt’s catalog. Bottom
panel: the distance from Gaia data, $d=10.2\pm 1.7~{}{\rm kpc}$, and the core
radius $r_{c}=0.178~{}{\rm pc}$, and the central density derived from central
velocity dispersion of the cluster $\rho_{c}=4.14\times 10^{5}~{}M_{\odot}{\rm
pc}^{-3}$ are assumed. The central IMBH is not included in the simulations.

It is evident that the terms of $a_{\rm g}$, $a_{\rm s}$ and $a_{\rm DM}$
contribute minimally to the measured acceleration of each pulsar.
Consequently, Equation 6 can be simplified to (Wang & Xie, 2022)

$\frac{\dot{P}}{P}=\frac{\dot{P}_{0}}{P_{0}}+\frac{a_{l}}{c}.$ (13)

In the Australian Telescope National Facility (ATNF) catalog
222https://www.atnf.csiro.au/research/pulsar/psrcat/ (Manchester et al.,
2005), when we examine the MSPs, we find that $\dot{P}_{0}$ varies between
$10^{-21}$ and $10^{-19}$ for $P_{0}<10~{}{\rm ms}$, resulting in a maximum
value of $\dot{P}0/P0\sim 3\times 10^{-9}~{}{\rm yr^{-1}}$. This value is of
the same order as the $a_{l}$ term. Due to the difficulty in distinguishing
the contribution of the intrinsic spin-down term from the measured
accelerations, this issue could introduce significant uncertainties in any
research focused on measuring the GC potential.

Figure 3: The line-of-sight acceleration $a_{l}$ with respect to the projected
distance $R_{\perp}^{\prime}$. The measured $a_{l}$ for pulsars from
J1801-0857A to I are represented by red points and marked appropriately. All
the pulsars with positive $\dot{P}$ are indicated with red arrows. The
simulated results are caused by the gravitational potential of NGC 6517. The
distance and core radius in the simulations are maintained at constant values
of $d=10.2\pm 1.7~{}{\rm kpc}$ and $r_{c}=0.178~{}{\rm pc}$, respectively.
Upper panel: the core density $\rho_{c}=8.2\times 10^{5}~{}M_{\odot}{\rm
pc}^{-3}$ is adopted. The central IMBH is not included in the simulation.
Bottom panel: the core density derived from the central velocity dispersion
$\rho_{c}=4.14\times 10^{5}~{}M_{\odot}{\rm pc}^{-3}$ is taken, and the
central IMBH has a mass of $M_{\rm BH}=9000~{}M_{\odot}$ is assumed.

It is important to note that $\dot{P}_{0}$ is always positive. In cases where
a pulsar is situated in the rear half of the cluster, the gravitational
potential of the GC negatively impacts the time derivative of the period
(i.e., $\delta\dot{P}<0$). This is illustrated in Figure 1. Consequently,
pulsars with a negative $\dot{P}$ are likely influenced by the GC potential as
the dominant term, represented by $\frac{\dot{P}}{P}\approx\frac{a_{l}}{c}$.
This relationship can be utilized to provide a minimum estimate for the GC
potential and the mass of a possible IMBH.

##  3 Results

In NGC 6517, nine pulsars with measured $P$ and $\dot{P}$ have been observed,
as shown in Table 1. Among them, five pulsars, J1801-0857A, C, E, F and I,
have negative $\dot{P}$, which makes the cluster a very good candidate for
gravitational field estimation.

In Table 1, we display the angular offsets $\theta_{\perp}$ from the centre of
the cluster for the nine pulsars. The projected distances for pulsars, can be
determined by using the offsets $\theta_{\perp}$ and the distance $d$ of the
cluster from the Sun. We utilize three distinct measurements of the distance
$d$ in determining the projected distances: the distance taken from the
optical observation of Harris (1996) is $10.6~{}{\rm kpc}$ (without an error
bar), the mean distance given by Baumgardt’s list of fundamental globular
cluster parameters (Baumgardt’s catalog)
333https://people.smp.uq.edu.au/HolgerBaumgardt/globular/parameter.html is
$9.23\pm 0.56~{}{\rm kpc}$, and the recent value derived from Gaia data is
$10.2\pm 1.7~{}{\rm kpc}$ (Baumgardt & Vasiliev, 2021). Employing these
specified measurements, we derived the respective projected distances,
represented as $R_{\perp}^{\prime}(1)$, $R_{\perp}^{\prime}(2)$, and
$R_{\perp}^{\prime}(3)$, which have been displayed in the same table.

We calculate the line-of-sight acceleration $a_{l}$ for pulsars located at
various radial distances in NGC 6517 with Equations 1-5. We perform Monte
Carlo simulations on the acceleration $a_{l}$ with respect to the projected
distance, for pulsars in the innermost region of the cluster. The simulated
$a_{l}$ distributions are shown in Figure 2. At first, we adopt the cluster
distance from the Sun $d=10.6~{}{\rm kpc}$, in association with the core
radius $r_{c}=0.185~{}{\rm pc}$, which is derived from the angular core radius
$\theta_{\rm c}=0.06~{}{\rm arcmin}$, and core density $\rho_{c}=1.95\times
10^{5}~{}M_{\odot}{\rm pc}^{-3}$, as determined from the optical observations
for NGC 6517 (Harris, 1996), and the simulated results are shown in the upper
panel of Figure 2. When compared to the measured accelerations, the simulated
results are noticeably smaller. This indicates that the central density
$\rho_{c}$ from the optical observation is significantly lower than the mass
density generating the GC potential.

We then take the following parameters from Baumgardt’s catalog: the distance
of $d=9.23\pm 0.56~{}{\rm kpc}$, the core radius of $r_{c}=0.14~{}{\rm pc}$,
and the core density of $\rho_{c}=4.79\times 10^{5}~{}M_{\odot}{\rm pc}^{-3}$.
The last parameter was estimated using N-body simulations as detailed in the
same catalog. Applying these values, we proceed with the simulation, and show
the simulated results in the middle panel of Figure 2. The error bars of the
measured values for the position of pulsars are derived from the distance $d$.
One can see that the simulated distribution remains below the measured $a_{l}$
values for certain pulsars.

We utilize the distance from Gaia data, $d=10.2\pm 1.7~{}{\rm kpc}$, and the
core radius $r_{c}=0.178~{}{\rm pc}$. The central density can be estimated by
using the central velocity dispersion of the cluster, which is defined as
(King, 1966; McLaughlin & van der Marel, 2005; Freire et al., 2017)

$\rho_{c}=\frac{9\sigma_{0}^{2}}{4\pi Gr_{c}^{2}},$ (14)

where $\sigma_{0}$ is the central velocity dispersion, and for NGC 6517,
$\sigma_{0}=9.2~{}{\rm km/s}$ is taken from Baumgardt’s catalog. Then we
obtain $\rho_{c}=4.14\times 10^{5}~{}M_{\odot}{\rm pc}^{-3}$. The bottom panel
of Figure 2 shows the simulated results. It is evident that, for specific
pulsars, the simulated distribution persistently falls short of the measured
$a_{l}$ values.

By retaining $d=10.2\pm 1.7~{}{\rm kpc}$ and $r_{c}=0.178~{}{\rm pc}$, and
adjusting the mass density to $\rho_{c}=8.2^{+0.2}_{-0.2}\times
10^{5}~{}M_{\odot}{\rm pc}^{-3}$, we find that all the measured values are
encompassed by the simulated outcomes, as shown in upper panel of Figure 3.
The uncertainty arises due to the adjustment process with respect to the error
bar of $R_{\perp}^{\prime}(3)$ for Pulsar J1801-0857E. This strongly implies
that some optically nonluminous matters are necessary for explaining the
measured accelerations of the pulsars. Potential candidates include massive
white dwarfs, neutrons stars or stellar mass BHs that formed in the early
phase of the GC. In addition to a concentration of dark remnants in the
central region, it is also plausible that an IMBH exists at the core of the
GC.

For the models, we consider the presence of an IMBH at the center of the GC.
We perform the simulations using various masses of the IMBH, while maintaining
$d=10.2\pm 1.7~{}{\rm kpc}$, $r_{c}=0.178~{}{\rm pc}$. We adopt the core
density derived from the central velocity dispersion, $\rho_{c}=4.14\times
10^{5}~{}M_{\odot}{\rm pc}^{-3}$. This value is in approximate consistency
with results obtained from N-body simulations of Baumgardt’s catalog. The
results of the simulations can be found in the bottom panel of Figure 3. The
pulsars J1801-0857E and F, exhibiting the highest measured accelerations, can
be well covered with the simulated data for the mass of the IMBH $M_{\rm
BH}=9000^{+4000}_{-3000}~{}M_{\odot}$. The uncertainty emerges as a
consequence of the adjustment process with respect to the error bar of Pulsar
J1801-0857F. The analysis also reveals that the simulated data cannot account
for the measured acceleration of pulsar I. However, density model used for the
entire cluster (Equation 1) describes the density profile only within a few
core radii of a GC. It is found from Equations (14) and (27) reported in King
(1962) setting the tidal radius to infinity. It is not a good approximation of
the actual density far from the centre of the GC. For this reason, the fitting
should not be applied to pulsars H, I and D without substantial
justifications.

It should be pointed out that an IMBH of the mass $M_{\rm BH}\sim
9000~{}M_{\odot}$ would have an influence radius of $0.08~{}{\rm pc}$ ($\sim
1.7$ arcseconds), about $46\%$ of the core radius of the cluster. This would
have strong effects on central stars of the GC and the radial profile would
show a clear cusp in the central region. However, such a cusp is not seen in
the surface density profiles of the cluster (see for example Trager et al.
(1995)). This absence suggests that the dark remnants model could be a more
plausible interpretation. However, currently determining this is challenging.
This is due to the fact that the segregation of stellar mass BHs, neutron
stars and massive white dwarfs induces a steep increase in dark mass,
subsequently causing a softening effect on the luminosity cusp. Furthermore,
the irregular movement of an IMBH, when positioned relative to the cluster
centre, may cause additional distortion, thereby further lessening the ascent
of a centrally rising cusp (Kızıltan et al., 2017).

We would like to emphasize that the simulated $a_{l}$ distribution profiles
for the dark remnants model (upper panel of Figure 3) and the IMBH model
(bottom panel of Figure 3) exhibit considerable differences. These distinct
distribution profiles could potentially be employed to differentiate between
the two scenarios, once a sufficient number of pulsars have been discovered
within the cluster.

##  4 Summary

In NGC 6517, there are five pulsars exhibiting negative $\dot{P}$, rendering
the cluster highly suitable for gravitational potential estimation. Utilizing
the measured accelerations of these pulsars, we have predicted the
gravitational potential and mass density of the cluster. Moreover, we
conducted a Monte Carlo simulation on the acceleration distributions for
pulsars, and compared them to the measured values. Our findings indicate a
concentration of non-luminous mass in the central region of the cluster, and
the source of this dark mass excess could be either a system of stellar dark
remnants or a single IMBH with a mass of approximately $\gtrsim 9\times
10^{3}~{}M_{\odot}$. Although our analysis cannot currently distinguish
between these two scenarios, the simulations predict distinct acceleration
distribution profiles for each. As a result, it may be possible to
differentiate between the two once enough pulsars with negative $\dot{P}$ are
discovered within the cluster.

## Acknowledgements

We appreciate valuable discussions with C.Cheng and J.Y.Liao. We would like to
thank the referee for comments that led to a significant improvement of this
paper. ZP is supported by National Key R&D Program of China, No.
2022YFC2205202, and the CAS “Light of West China” Program. This work is
supported by National Natural Science Foundation of China under grant Nos.
11803009, 11603009, 11703047, 11773041, U2031119, and 12173052, and by the
Natural Science Foundation of Fujian Province under grant Nos. 2023J01806,
2018J05006, and 2018J01416.

## Data Availability

Data generated in this research will be shared on reasonable request to the
corresponding author.

## References

  * Abbate et al. (2019) Abbate F., Possenti A., Colpi M., Spera M., 2019, ApJ, 884, L9
  * Bahcall & Ostriker (1975) Bahcall J. N., Ostriker J. P., 1975, Nature, 256, 23
  * Baldassare et al. (2015) Baldassare V. F., Reines A. E., Gallo E., Greene J. E., 2015, ApJ, 809, L14
  * Bash et al. (2008) Bash F. N., Gebhardt K., Goss W. M., Vanden Bout P. A., 2008, AJ, 135, 182
  * Baumgardt & Hilker (2018) Baumgardt H., Hilker M., 2018, MNRAS, 478, 1520
  * Baumgardt & Vasiliev (2021) Baumgardt H., Vasiliev E., 2021, MNRAS, 505, 5957
  * Baumgardt et al. (2004a) Baumgardt H., Makino J., Ebisuzaki T., 2004a, ApJ, 613, 1133
  * Baumgardt et al. (2004b) Baumgardt H., Makino J., Ebisuzaki T., 2004b, ApJ, 613, 1143
  * Blandford et al. (1987) Blandford R. D., Romani R. W., Applegate J. H., 1987, MNRAS, 225, 51P
  * Djorgovski & King (1986) Djorgovski S., King I. R., 1986, ApJ, 305, L61
  * Farrell et al. (2009) Farrell S. A., Webb N. A., Barret D., Godet O., Rodrigues J. M., 2009, Nature, 460, 73
  * Freire et al. (2001) Freire P. C., Kramer M., Lyne A. G., Camilo F., Manchester R. N., D’Amico N., 2001, ApJ, 557, L105
  * Freire et al. (2005) Freire P. C. C., Hessels J. W. T., Nice D. J., Ransom S. M., Lorimer D. R., Stairs I. H., 2005, ApJ, 621, 959
  * Freire et al. (2017) Freire P. C. C., et al., 2017, MNRAS, 471, 857
  * Gebhardt et al. (2000) Gebhardt K., Pryor C., O’Connell R. D., Williams T. B., Hesser J. E., 2000, AJ, 119, 1268
  * Gebhardt et al. (2002) Gebhardt K., Rich R. M., Ho L. C., 2002, ApJ, 578, L41
  * Greene & Ho (2004) Greene J. E., Ho L. C., 2004, ApJ, 610, 722
  * Gürkan et al. (2004) Gürkan M. A., Freitag M., Rasio F. A., 2004, ApJ, 604, 632
  * Häberle et al. (2021) Häberle M., et al., 2021, MNRAS, 503, 1490
  * Harris (1996) Harris W. E., 1996, AJ, 112, 1487
  * King (1962) King I., 1962, AJ, 67, 471
  * King (1966) King I. R., 1966, AJ, 71, 64
  * Kızıltan et al. (2017) Kızıltan B., Baumgardt H., Loeb A., 2017, Nature, 542, 203
  * Lanzoni et al. (2007) Lanzoni B., Dalessandro E., Ferraro F. R., Miocchi P., Valenti E., Rood R. T., 2007, ApJ, 668, L139
  * Lin et al. (2018) Lin D., et al., 2018, Nature Astronomy, 2, 656
  * Lu & Kong (2011) Lu T.-N., Kong A. K. H., 2011, ApJ, 729, L25
  * Lugger et al. (1995) Lugger P. M., Cohn H. N., Grindlay J. E., 1995, ApJ, 439, 191
  * Lynch et al. (2011) Lynch R. S., Ransom S. M., Freire P. C. C., Stairs I. H., 2011, ApJ, 734, 89
  * Maccarone (2004) Maccarone T. J., 2004, MNRAS, 351, 1049
  * Maccarone & Servillat (2008) Maccarone T. J., Servillat M., 2008, MNRAS, 389, 379
  * Madau & Rees (2001) Madau P., Rees M. J., 2001, ApJ, 551, L27
  * Manchester et al. (2005) Manchester R. N., Hobbs G. B., Teoh A., Hobbs M., 2005, AJ, 129, 1993
  * McLaughlin & van der Marel (2005) McLaughlin D. E., van der Marel R. P., 2005, ApJS, 161, 304
  * McLaughlin et al. (2006) McLaughlin D. E., Anderson J., Meylan G., Gebhardt K., Pryor C., Minniti D., Phinney S., 2006, ApJS, 166, 249
  * Miller & Hamilton (2002) Miller M. C., Hamilton D. P., 2002, MNRAS, 330, 232
  * Newell et al. (1976) Newell B., Da Costa G. S., Norris J., 1976, ApJ, 208, L55
  * Nice & Taylor (1995) Nice D. J., Taylor J. H., 1995, ApJ, 441, 429
  * Noyola et al. (2008) Noyola E., Gebhardt K., Bergmann M., 2008, ApJ, 676, 1008
  * Pan et al. (2021a) Pan Z., et al., 2021a, Research in Astronomy and Astrophysics, 21, 143
  * Pan et al. (2021b) Pan Z., et al., 2021b, ApJ, 915, L28
  * Perera et al. (2017) Perera B. B. P., et al., 2017, MNRAS, 468, 2114
  * Phinney (1992) Phinney E. S., 1992, Philosophical Transactions of the Royal Society of London Series A, 341, 39
  * Phinney (1993) Phinney E. S., 1993, in Djorgovski S. G., Meylan G., eds, Astronomical Society of the Pacific Conference Series Vol. 50, Structure and Dynamics of Globular Clusters. p. 141
  * Portegies Zwart & McMillan (2002) Portegies Zwart S. F., McMillan S. L. W., 2002, ApJ, 576, 899
  * Portegies Zwart et al. (2004) Portegies Zwart S. F., Baumgardt H., Hut P., Makino J., McMillan S. L. W., 2004, Nature, 428, 724
  * Prager et al. (2017) Prager B. J., Ransom S. M., Freire P. C. C., Hessels J. W. T., Stairs I. H., Arras P., Cadelano M., 2017, ApJ, 845, 148
  * Sharma et al. (2014) Sharma S., et al., 2014, ApJ, 793, 51
  * Shklovskii (1970) Shklovskii I. S., 1970, Soviet Ast., 13, 562
  * Silk & Arons (1975) Silk J., Arons J., 1975, ApJ, 200, L131
  * Spergel (1991) Spergel D. N., 1991, Nature, 352, 221
  * Spitzer (1987) Spitzer L., 1987, Dynamical evolution of globular clusters
  * Trager et al. (1995) Trager S. C., King I. R., Djorgovski S., 1995, AJ, 109, 218
  * Tremou et al. (2018) Tremou E., et al., 2018, ApJ, 862, 16
  * Ulvestad et al. (2007) Ulvestad J. S., Greene J. E., Ho L. C., 2007, ApJ, 661, L151
  * Vasiliev & Baumgardt (2021) Vasiliev E., Baumgardt H., 2021, MNRAS, 505, 5978
  * Wang & Xie (2021) Wang L.-C., Xie Y., 2021, Research in Astronomy and Astrophysics, 21, 270
  * Wang & Xie (2022) Wang L.-C., Xie Y., 2022, Research in Astronomy and Astrophysics, 22, 115007
  * Xie & Wang (2020) Xie Y., Wang L.-C., 2020, Research in Astronomy and Astrophysics, 20, 191
  * van der Marel & Anderson (2010) van der Marel R. P., Anderson J., 2010, ApJ, 710, 1063
  * van der Marel et al. (2002) van der Marel R. P., Gerssen J., Guhathakurta P., Peterson R. C., Gebhardt K., 2002, AJ, 124, 3255