# CC2Vec: Combining Typed Tokens with Contrastive Learning for Effective Code
Clone Detection

Shihan Dou  0009-0002-6013-3035 Fudan UniversityShanghaiChina
<EMAIL_ADDRESS>,  Yueming Wu  0000-0002-1515-3558 Nanyang
Technological UniversitySingaporeSingapore<EMAIL_ADDRESS>,  Haoxiang
Jia  0009-0005-6027-231X Huazhong University of Science and
TechnologyWuhanChina<EMAIL_ADDRESS>,  Yuhao Zhou
0009-0008-8665-3999 Fudan UniversityShanghaiChina<EMAIL_ADDRESS>,
Yan Liu  0009-0002-5741-336X Fudan UniversityShanghaiChina
<EMAIL_ADDRESS>and  Yang Liu  0000-0001-7300-9215 Nanyang Technological
UniversitySingaporeSingapore<EMAIL_ADDRESS>
(2024; 2023-09-28; 2024-04-16)

###### Abstract.

With the development of the open source community, the code is often copied,
spread, and evolved in multiple software systems, which brings uncertainty and
risk to the software system (e.g., bug propagation and copyright
infringement). Therefore, it is important to conduct code clone detection to
discover similar code pairs. Many approaches have been proposed to detect code
clones where token-based tools can scale to big code. However, due to the lack
of program details, they cannot handle more complicated code clones, i.e.,
semantic code clones. In this paper, we introduce _CC2Vec_ , a novel code
encoding method designed to swiftly identify simple code clones while also
enhancing the capability for semantic code clone detection. To retain the
program details between tokens, _CC2Vec_ divides them into different
categories (i.e., typed tokens) according to the syntactic types and then
applies two self-attention mechanism layers to encode them. To resist changes
in the code structure of semantic code clones, _CC2Vec_ performs contrastive
learning to reduce the differences introduced by different code
implementations. We evaluate _CC2Vec_ on two widely used datasets (i.e.,
BigCloneBench and Google Code Jam) and the results report that our method can
effectively detect simple code clones. In addition, _CC2Vec_ not only attains
comparable performance to widely used semantic code clone detection systems
such as _ASTNN_ , _SCDetector_ , and _FCCA_ by simply fine-tuning, but also
significantly surpasses these methods in both detection efficiency.

Code clone detection, contrastive learning, self-attention mechanism

††copyright: rightsretained††doi: 10.1145/3660777††journalyear:
2024††submissionid: fse24main-p109-p††journal: PACMSE††journalvolume:
1††journalnumber: FSE††article: 70††publicationmonth: 7††ccs: Software and its
engineering Software maintenance tools

##  1\. Introduction

Code clone, also known as duplicate code or similar code, refers to two or
more identical or similar source code fragments that exist in the code base.
According to the syntactic or semantic level differences, code clones can be
classified into four types (Roy and Cordy, 2007; Bellon et al., 2007). Type-1
to Type-3 code clones belong to syntactic code clones, while Type-4 code
clones are semantic code clones. Most of the existing code clone detection
methods aim to detect syntactic code clones because these clones are easier to
identify compared to semantic code clones.  For example, to detect the first
two types of code clones, _CCFinder_ (Kamiya et al., 2002) first performs
lexical analysis to extract the token sequence of a code snippet. After
obtaining all tokens, it conducts several code transformations to convert the
tokens into corresponding forms and then uses the transformed tokens to finish
code clone detection. Since _CCFinder_ (Kamiya et al., 2002) only applies
certain simple code transformations, it cannot detect Type-3 code clones. To
address the issue, other token-based methods are designed by analyzing the
token features between the two codes within a pair from different
perspectives. For example, _SourcererCC_ (Sajnani et al., 2016) computes the
overlap ratio of tokens to detect near-miss Type-3 code clones. However, due
to the lack of preservation of program semantics, these token-based methods do
badly in detecting semantic code clones since the code structure may change a
lot after using another way of implementation. For example, when implementing
the same loop-related functionality, some developers may use _for_ statements
while others may prefer to use _while_ statements.

To tackle the issue, researchers perform complex program analysis to transform
the code fragments into different intermediate representations such as
_abstract syntax tree_ (AST) (Zhang et al., 2019) and _control flow graph_
(CFG) (Wu et al., 2020b). These intermediate representations can maintain the
program semantics from different perspectives. Empirical experiments have
demonstrated the capability of tree-based and graph-based methods (Krinke,
2001; Komondoor and Horwitz, 2001; Wang et al., 2017) on detecting semantic
code clones. However, both tree processing and graph analysis are time-
consuming, making it difficult for them to find semantic code clones
efficiently. A recent report (ope, 2021a) has shown that open source has
become an unstoppable trend, leading to an increasing scale of code reuse.
These cloned codes can bring uncertainty and risks to the software system,
such as bug propagation, and copyright infringement (ope, 2021b). Therefore,
there is an urgent need to develop a tool that can be used for large-scale
semantic clone detection.

In this paper, we propose _CC2Vec_ , a novel code encoding approach to
efficiently detect syntactic code clones, while enhancing the ability of
semantic code clone detection by further combining with a few neural networks.
Specifically, we mainly address two challenges.

  * •

_Challenge 1: How to retain the program semantics by purely analyzing the
source code tokens?_

  * •

_Challenge 2: How to differentiate codes based on their functionalities
instead of their structures?_

To address the first challenge, we parse the source code into corresponding
tokens and divide them into 15 categories based on syntactic types by lexical
analysis. We call tokens in different categories as _typed tokens_ and observe
that these typed tokens have different weights in detecting code clones
(Details are in Section 2). Based on the observation, we first apply one self-
attention mechanism layer to encode the tokens in the same category into a
vector representation. By this, the source code is converted into 15 vectors,
each corresponding to a token category. After obtaining 15 vectors, we then
apply another self-attention mechanism layer to encode them into one vector,
which is the output of this phase. Using self-attention mechanism layers can
not only interpret the detection results but also preserve the potential
relationships between tokens. These relationships may represent some program
details of a method.

To solve the second challenge, we conduct contrastive learning to train the
program encoder.  The use of contrastive learning is to maximize the
similarity between representations of a code snippet and its clone codes
(i.e., positive samples), while minimizing the representations’ similarity
between this code snippet and its non-clone codes (i.e., negative samples).
As for semantic code clones, although their code structures may change a lot,
the implemented functionality remains unchanged. In other words, they can be
treated as clone samples of each other. Therefore, we can use contrastive
learning to decrease the distance between semantic code pairs while enlarging
the difference between dissimilar code pairs. After contrastive learning, the
learned encoder _CC2Vec_ is robust to code changes introduced by semantic code
cloning, making it possible to distinguish semantic code clones although they
are syntactically different.

We design a token-based code clone detector by simply calculating the cosine
similarity of the two codes’ vector encoded by _CC2Vec_ , to detect syntactic
code clones efficiently. In addition, we can effectively detect more
complicated code clones (i.e., semantic code clones) by combining _CC2Vec_
with a few neural networks.  To examine the ability of _CC2Vec_ , we conduct
evaluations on two widely used datasets, namely BigCloneBench (big, 2020;
Svajlenko et al., 2014) and Google Code Jam (gcj, 2017).  Compared to five
pretrain-based methods, _CC2Vec_ can effectively capture the tokens
relationship information by pre-training from the training code corpus to
achieve an improvement in detecting code clones.  Compare to nine traditional
code clone detectors, the recall of _CC2Vec_ is 64% while the comparative
tools can only maintain very low recall, respectively. Compared to six deep-
learning-based code clone detectors, _CC2Vec_ can achieve the best F1 score
when using only a simple three-layer neural network as the classifier. As for
scalability, _CC2Vec_ outperforms most pretrain-based and deep learning-based
methods in terms of runtime efficiency. _CC2Vec_ is about 100 times faster
than _ASTNN_ in predicting code clone pairs 111 Our tool is available on our
website: _https://github.com/CC2Vector/CC2Vec_..

In summary, this paper makes the following contributions:

  * •

We introduce a novel encoding approach _CC2Vec_ to enhance the ability on code
clone detection by using contrastive learning. _CC2Vec_ applies two self-
attention mechanisms layers to encode source code tokens and contrastive
learning to learn a robust encoder.

  * •

We improve the detection accuracy of syntactic code clones by simply computing
the cosine similarity of two codes’ vectors encoding by _CC2Vec_.
Additionally, we can also effectively detect the semantic code clones by
combining _CC2Vec_ with a few neural networks.

  * •

We check the ability of _CC2Vec_ by conducting experiments on two widely used
datasets (i.e., BigCloneBench and Google Code Jam). Experimental results show
that our proposed detectors based on _CC2Vec_ can achieve comparable
performance to widely used code clone detectors with efficient training and
detecting phase.

Paper organization. The remainder of the paper is organized as follows.
Section 2 presents our background. Section 3 introduces our system. Section 4
reports the experimental results. Section 5 shows our discussions. Section 6
describes the related work. Section 7 concludes the present paper.

##  2\. Background

In this section, we first discuss four types of code clones and then clarify
the key insight of _CC2Vec_.

###  2.1. Clone Types

As aforementioned, code clones can be divided into four types (Roy and Cordy,
2007; Bellon et al., 2007). The first three types belong to syntactic code
clone while the last type is semantic code clone.

  * •

Type-1: The same code snippets, except for differences in white space and
comments.

  * •

Type-2: The same code snippets, except for differences in identifier names,
literal values, white space, and comments.

  * •

Type-3: Syntactically similar code snippets that differ at the statement
level. The snippets have statements added, modified and/or removed.

  * •

Type-4: Syntactically dissimilar code snippets that implement the same
functionality.

Figure 1. Examples of four code clone types

To illustrate the differences between four types of code clones, we present
one method and its four types of code clones in Figure 1. They implement the
same functionality which is to calculate the greatest common divisor of two
numbers. Method 1 is the Type-1 code clone of the original method, with no
changes between their source code. Method 2 is the Type-2 code clone of the
original method, and the difference between them is only in the identifiers
name (i.e., $m$ and $n$ instead of $a$ and $b$). The few code changes make
them easy to detect. Method 3 is the Type-3 code clone of the original method,
it differs at the statement level. Their method names and parameter types are
different and the order of the statements also changes a little. This type of
code clone is more difficult to detect than the previous two types. Method 4
is the Type-4 code clone of the original method, it implements the same
functionality in a different way. This type of clone is also called semantic
clone and is the most difficult to discover since the code structure may
change a lot.

###  2.2. Motivation

We propose an example to illustrate the key insight of our method. This
example is a clone pair in BigCloneBench (big, 2020), it consists of two
methods and their functionalities are both to compute the greatest common
divisor of two integers. As shown in Figure 2, although they are syntactically
dissimilar, the functionalities are the same. Therefore, they can be treated
as a semantic clone pair.

Figure 2.  Source code and corresponding tokens of _gcd 1.java_ and _gcd
2.java_ after lexical analysis

As aforementioned, _SourcererCC_ (Sajnani et al., 2016) is one of the most
scalable token-based code clone detector, which can scale to analyze more than
428 million files on Github (Lopes et al., 2017). It uses a simple overlap
similarity calculation to measure the distance between two methods. For
example, given two methods _M 1_ and _M 2_, the overlapping similarity
$S(M_{1},M_{2})$ is computed as the quotient of the number of same tokens
shared by _M 1_ and _M 2_ and the maximum number of tokens in _M 1_ and _M 2_,
which can be written as $S(M_{1},M_{2})=\frac{|M_{1}\bigcap
M_{2}|}{max(|M_{1}|,|M_{2}|)}$.

To compute the overlapping similarity of methods in List 1 and List 2, we
first perform lexical analysis to collect the corresponding source code
tokens. Then the number of the same tokens shared by these two methods is
obtained by statistical analysis. After completing the analysis, we find that
there are 19 tokens shared by _gcd 1.java_ and _gcd 2.java_. Therefore, the
similarity computed by _SourcererCC_ is 0.5 (i.e., 19/38=0.5) since the
maximum number of tokens of _gcd 1.java_ and _gcd 2.java_ is 38. Because 0.5
is smaller than 0.7 which is the default similarity threshold in _SourcererCC_
, _gcd 1.java_ and _gcd 2.java_ will not be reported as a clone pair. In other
words, it will be a false negative when we use _SourcererCC_ to detect _gcd
1.java_ and _gcd 2.java_.

To detect the semantic code pair, we conduct further analysis on their source
code tokens.  We observe that these tokens can be divided into different
categories. Figure 2 presents the tokens in different categories of _gcd
1.java_ and _gcd 2.java_ after applying lexical analysis. Through the results,
we can see that the tokens in some categories are similar, while the tokens in
certain categories are different.  For example, only four “ _long_ ” tokens
belong to _BasicType_ category in _gcd 1.java_, while the tokens belonging to
_BasicType_ category in _gcd 2.java_ are three “ _int_ ” tokens. As for tokens
in _DecimalInterger_ category, _gcd 1.java_ and _gcd 2.java_ are the same,
that is, only one “ _0_ ” token.  To measure the similarity of tokens in
different categories, we apply Eq. LABEL:equ:sou to the tokens within the
single category to compute the category-level overlapping similarity of these
two methods. To measure the similarity of tokens in different categories, we
compute the above category-level overlapping similarity of these two methods
for each token category. For instance, the overlapping similarity for the
_Separator_ category can be calculated as 0.69 (i.e.,
(2+2+1+2+1+1)/max((2+2+1+4+2+2), (3+3+2+2+1+1)) = 9/13 = 0.69).  After
computing all similarities, we find that they are different in different
categories. Some categories have high similarities, while others have low
similarities. In other words, if we assign a higher weight to a category with
high similarity and a smaller weight to a category with low similarity, then
_gcd 1.java_ and _gcd 2.java_ are likely to be detected as a clone pair. For
example, if the weights of seven categories in Figure 2 are 0, 0.6, 0.15,
0.05, 0, 0.05, and 0.15, respectively. The similarity will be calculated as
0.82 (i.e., 0$*$0 + 1$*$0.6 + 0.58$*$0.15 + 0.33$*$0.05 + 0$*$0 + 0.20$*$0.05
+ 0.69$*$0.15 = 0.82. In each multiplication, the first value represents the
category overlap similarity, and the second value denotes its assigned weight)
which is higher than the default similarity threshold (i.e., 0.7) in
_SourcererCC_.

Therefore, inspired by the observation, we introduce the attention mechanism
to design a novel technique to train an encoder that can assign suitable
weights for different categories, to detect all kinds of code clones. The
attention mechanism can automatically focus on important tokens, which has the
potential to filter the useless tokens and categories in detecting code
clones, while concentrating the attention on these useful tokens and
categories.

Figure 3. System of _CC2Vec_

##  3\. System

In this section, we introduce our code encoding approach, namely _CC2Vec_ ,
and how to utilize it to detect syntactic and semantic code clones.

###  3.1. _CC2Vec_ Overview

As shown in Figure 3, _CC2Vec_ is made up of three main components: _Token
Categorization_ , _Program Encoding_ , and _Optimization Process_.

  * •

_Token Categorization_ : Given the source code of a method, we first apply
lexical analysis to parse them into tokens with corresponding categories,
namely _typed tokens_. The input of this component is a method, while the
output is certain typed tokens.

  * •

_Program Encoding_ : This component consists of two attention mechanism
layers. The first layer encodes tokens in each category into a vector, and the
second layer encodes all categories into the final vector. The input of this
component is certain typed tokens, while the output is a vector
representation.

  * •

_Optimization Process_ : In this phase, we employ contrastive learning as the
optimization approach to train the program encoder which is robust to code
changes introduced by different implementations.

Table 1. Selected 15 token types “ _Annotation_ ”, “ _BasicType_ ”, “
_BinaryInteger_ ”, “ _Boolean_ ”, “ _DecimalFloatingPoint_ ”, “ _Modifier_ ”
---
, “ _Operator_ ”, “ _DecimalInteger_ ”, “ _HexFloatingPoint_ ”, “ _HexInteger_
”, “ _Identifier_ ”,
“ _Keyword_ ”, “ _OctalInteger_ ”, “ _Separator_ ”, “ _Null_ ”

###  3.2. Token Categorization

This component aims to parse the source code of a method into tokens with
corresponding categories (i.e., _typed tokens_). To complete the purpose, we
conduct a lexical analysis to analyze the source code to divide it into
different categories based on syntactical types. Since the experimental
dataset is BigCloneBench (big, 2020) and the programming language is _Java_ ,
we leverage a python library _javalang_ (jav, 2022) to accomplish the lexical
analysis of _CC2Vec_. In practice, token categorization aims to divide a
method into _typed tokens_ , which can be achieved by lexical analysis.
Therefore, _CC2Vec_ is not limited to programming languages (e.g., _C_ and
_Java_) because different languages have related lexical analysis tools (e.g.,
_pycparser_ (pyc, 2022) for _C_ language and _javalang_ (jav, 2022) for _Java_
language).

In practice, listing all token types parsed by _javalang_ can be challenging
due to the potential existence of some types that are used infrequently. To
identify the most relevant token types, we conducted a lexical analysis on the
top 10,000 Java projects on GitHub, ranked by their criticality score. This
score can be used to describe the impact and importance of an open source
project (cri, 2023). After our statistical analysis, we found that 14 types
collectively constitute over 99.5% of all tokens. Therefore, we choose these
14 types as the final token types and add a _Null_ type to represent other
types as shown in Table 1.

Figure 4. Token categorization of _CC2Vec_

To better describe the different steps of our system, we present one simple
example in Figure 4. It shows that tokens of the method in List 2 can be
divided into seven categories after applying lexical analysis and each
category contains the corresponding tokens and the number of tokens. For
example, in “ _BasicType_ ” category, it only contains one token “ _int_ ”,
and the number of tokens “ _int_ ” is three. The output of this component is
tokens with corresponding categories, namely _typed tokens_.  Since different
tokens may determine whether two pieces of code are clones, as discussed in
Section 2, if we can identify the optimal tokens and increase their weights
automatically, we will be able to detect code clones more effectively.

Figure 5. Program encoding of _CC2Vec_

Figure 6. Contrastive learning of _CC2Vec_

###  3.3. Program Encoding

This component aims to encode _typed tokens_ into a vector representation. As
shown in Figure 5, it consists of two steps: _Token Encoding_ and _Type
Encoding_.

####  3.3.1. Token Encoding.

This step aims to encode tokens within a single category into vector
representations. To achieve this, we initially employ a word embedding
technique to map tokens to their corresponding vector representations.
Subsequently, these vectors are input into an attention mechanism layer for
analysis. Specifically, to encode tokens with similar meanings into high-
dimensional feature spaces with comparable distances, we utilize the source
code from BigCloneBench (big, 2020) as the training set to train a _Word2Vec_
model (Mikolov et al., 2013). _Word2Vec_ stands as one of the most widely used
techniques for learning word embeddings through shallow neural networks. After
obtaining corresponding vector representations of all tokens in one category,
we multiply them by the number of corresponding tokens one by one. These
multiplied vectors will be fed into one attention mechanism layer to be
encoded. Note that each category is assigned an attention mechanism layer, and
the layers are independent of each other. In other words, we design 15
independent attention mechanism layers to encode tokens in 15 categories,
respectively. Some semantic relationships exist between statements in a method
(e.g., , data flow), resulting in potential relationships between tokens. To
consider these potential relationships while program encoding, we use the
self-attention mechanism as our encoding layer since it can not only describe
the importance or weights but also consider the relationships between tokens.
Our self-attention mechanism layer consists of one multi-head attention layer
and one feed-forward layer. More details of them are shown in _Transformer_
(Vaswani et al., 2017). After token encoding, each category can output a
vector.

####  3.3.2. Type Encoding.

This step aims to encode vectors obtained from _Token Encoding_ step into a
vector representation. Due to some potential relationships between tokens,
there are some relationships between categories. Therefore, we also apply one
self-attention mechanism layer to finish the final encoding. It is worth
noting that some methods might not encompass all 15 categories. For example,
after lexical analysis, the method in Figure 4 can only be divided into seven
categories. To ensure appropriate weight allocation across all categories, we
assign a value of 0 to vectors corresponding to absent categories.  Setting
each position of absent types of token to zero, instead of category 0 is
intended to ensure that the results of subsequent matrix operations in
attention are 0, preventing this token from interfering with others.  By this,
the inputs of _Type Encoding_ step are 15 vectors corresponding to 15
categories. The self-attention mechanism layer used in this step is the same
as that used in _Token Encoding_ step. After _Type Encoding_ , we can obtain
the final vector representation of a given method.

###  3.4. Optimization Process

Given a dollar bill, it is difficult for us to draw it exactly the same,
although we have seen it many times. However, we can easily use our drawing to
convey the essence of the dollar bill to others (Anand, 2020). Drawing
inspiration from such commonplace observations, researchers have designed a
learning algorithm. Instead of focusing on every sample detail, it extracts
enough abstract features to distinguish it from other samples. This learning
algorithm is also known as contrastive learning, its goal is to make the
distance between different types of inputs get larger and larger, and the same
types of inputs get closer and closer. Since contrastive learning only needs
to learn to distinguish the sample in the feature space of the abstract
semantic level, the model and its optimization become simpler and the
generalization ability is stronger. Formally, contrastive learning aims to
allow the encoder to generate more unique data features without losing its
essence. For any data sample $x$ and its feature $f(x)$, it intends to
complete a task as equation (1).

(1) $\small\text{SIM}(f(x),f(x^{+}))>>\text{SIM}(f(x),f(x^{-}))$

Here $x^{+}$ is a data sample similar to $x$ (referred to as the positive
sample) and $x^{-}$ is a data sample dissimilar to $x$ (referred to as the
negative sample). $f(x^{+})$, $f(x^{-})$ is the feature of $x^{+}$, $x^{-}$
separately, and $\text{SIM}(\cdot)$ is a metric to measure the similarity
between two features.

We utilize contrastive learning as the optimization method to train the
program encoder. Specifically, the model training and optimization process is
shown in Figure 6, where code method $A_{1}$ and code method $A_{2}$ are a
clone pair, and code method $B_{1}$ is dissimilar to code methods $A_{1}$ and
$A_{2}$. Therefore, the optimization goal of the model is to reduce the
distance between the encoded vectors of the code method and its similar pairs
(i.e., positive samples), such as $A_{1}$ and $A_{2}$, and increase the
distance between the encoded vectors of the code method and its dissimilar
pairs (i.e., negative samples), such as $A_{1}$ and $B_{1}$. As for semantic
code clones, although their code structures may change a lot, the implemented
functionality remains unchanged. Therefore, they can be treated as positive
samples of each other.

###  3.5. Code Clone Detection

After completing the training phase by using contrastive learning, the learned
encoder can be robust to code changes introduced by different implementations,
making it possible to identify the subtle difference between the two code
snippets. The code encoder can be directly used for discovering the simple
syntactic code clones, while can detect semantic code clones by continuing
fine-tuning.

Firstly, _CC2Vec_ introduces two self-attention mechanisms layers to capture
potential relationships between tokens, which means that it can encode more
important information. Therefore, we can directly design a simple code clone
detector to identify code clones efficiently, and easily to scale to big code.
Specifically, for the Type-1 to Type-3 code clones (i.e., syntactic code
clones), we obtain the two code representations by using _CC2Vec_ and
calculate the cosine similarity between these two vectors. If their cosine
similarity is higher than 70% (Sajnani et al., 2016), they will be reported as
a clone pair. Otherwise, they will be treated as a non-clone pair.

Secondly, for the Type-4 code clones (i.e., semantic code clones), we combine
_CC2Vec_ with a few neural networks (i.e., fully connected layer) to enhance
the ability to detect more complicated samples. As _CC2Vec_ has learned
potential information from code data, fine-tuning it with one to three fully
connected layers can significantly improve the detection ability on semantic
code clones. Specifically, we obtain the vectors corresponding to each of the
two code snippets within a pair and then concatenate them to form a single
vector representation. This vector is then input to multiple fully connected
layers to detect whether the code pair is a clone pair or not. For instance,
_CC2Vec-3L_ denotes that we connect our pre-trained encoder _CC2Vec_ with
three fully connected layers. Notably, we insert a ReLu activation layer
between each fully connected layer to ensure the non-linear capability of the
neural networks.

##  4\. Experiments

In this section, we undertake comprehensive experiments to assess _CC2Vec_ ’s
ability to detect code clones from three key perspectives: effectiveness,
interpretability, and scalability. For effectiveness, we not only evaluate
_CC2Vec_ ’s performance in detecting both syntactic and semantic code clones
but also demonstrate the contributions of its different components (i.e.,
self-attention and contrastive learning) to code clone detection. For
interpretability, we provide examples to illustrate how _CC2Vec_ interprets
the detection results, shedding light on its decision-making process. For
scalability, we present a detailed experiment to show the runtime overhead of
_CC2Vec_ , providing insights into its computational efficiency. Specifically,
we aim to answer the following research questions:

  * •

_RQ1: Can CC2Vec detect code clones effectively?_

  * •

_RQ2: How do self-attention mechanism layers and contrastive learning
contribute to CC2Vec?_

  * •

_RQ3: Can CC2Vec interpret the detection results?_

  * •

_RQ4: Can CC2Vec scale to big code?_

###  4.1. Experimental Settings

####  4.1.1. Dataset.

To check the ability of _CC2Vec_ to detect code clones, we choose
BigCloneBench (big, 2020) as our first dataset since it is the largest dataset
and widely used by many researchers (Sajnani et al., 2016; Zhang et al., 2019;
Wang et al., 2018). Experts assign all pairs in BigCloneBench a clone type
according to a similarity score calculated by line-level and token-level after
code normalization. As for Type-3 and Type-4 code clones, they are divided
into four parts due to the ambiguous boundary between them, that is, 1) _Very
Strongly Type-3_ (VST3) with a similarity between [0.9, 1.0), 2) _Strongly
Type-3_ (ST3) with a similarity between [0.7, 0.9), 3) _Moderately Type-3_
(MT3) with a similarity between [0.5, 0.7), and 4) _Weakly Type-3/Type-4_
(WT3/T4) with a similarity between [0.0, 0.5).  BigCloneBench contains about
270,000 non-clone pairs and over eight million tagged clone pairs. Finally, we
obtain 48,116 T1 clones, 4,234 T2 clones, 4,577 VST3 clones, 16,818 ST3
clones, 86,341 MT3 clones and 7,943,729 WT3/T4 clones. To balance our dataset,
we select almost 270, 000 clone pairs from them. Specifically, we select all
of the syntactic pairs to balance the number of each code clone pair type, and
randomly select 109,914 clone pairs from a total of about eight million
semantic clones. As a result, there are 48,116 T1 clones, 4,234 T2 clones,
4,577 VST3 clones, 16,818 ST3 clones, 86,341 MT3 clones, and 109,914 WT3/T4
clones in our dataset.

Similar to previous work (Wu et al., 2020b), for the more challenging semantic
clones, we also evaluate our proposed method on another widely used dataset
namely Google Code Jam (Zhao and Huang, 2018), which is primarily composed of
semantic clones, to better demonstrate the effectiveness of our method.
Google Code Jam is derived from an online programming competition held by
Google and contains 1,669 projects from 12 different competition problems
which are written by different programmers. So projects of the same competing
problem are almost syntactically different but semantically similar, and we
treat them as clone pairs. Projects that solve different problems are not
similar, and we regard them as non-clone pairs.  Finally, we obtain 275,570
semantic clone pairs and 1,116,376 non-clone pairs. We randomly select 270,000
pairs from all non-clone pairs to balance our dataset.

####  4.1.2. Implementations.

For detecting the syntactic code clones, we only need to train a code encoder
_CC2Vec_.  We use _Word2Vec_ (Mikolov et al., 2013) to embed the token to the
vector. The size of the embedding layer and the hidden size of the two
attention layers are 100. We average the hidden cell of the attention layer to
obtain the output vector, so the vector size is also 100. During the pre-
trained phase, we freeze the embedding layer and only the attention layer is
trainable.  For contrastive learning, we apply the SupCon loss function
(Khosla et al., 2020) as the training objective. The temperature in this loss
function is set to 0.07. During the training phase, we use the RMSProp
optimization algorithm (RMS, [n. d.]), with a momentum of 0.9 and a weight
decay of 0.0001, to train the encoder. The learning rate and the training
epochs are set to 0.0001 and 10, respectively. The training batch size is 64.
After training, we can obtain a program encoder to extract high-level
potential features from code snippets. The threshold is set to 0.7, based on
the previous research (Sajnani et al., 2016), indicating that if the cosine
similarity between the two code snippets within a pair exceeds 70%, they are
classified as a clone pair. Otherwise, they are considered a non-clone pair.
For fully connected layers used to enhance the ability to detect semantic code
clones, we need to fine-tune the code encoder _CC2Vec_ and the newly
introduced neural networks. The learning rate and optimizer are the same as
above. Notably, the training dataset utilized is the same dataset used in
training the encoder _CC2Vec_ , to avoid any leakage of validation and test
data into the training process.

####  4.1.3. Comparative Tools.

We also compare _CC2Vec_ with some state-of-the-art code clone detection
systems. The choice of our baseline follows these principles: The method is
popular and extensively used, with a high citation in the code clone detection
and widely compared in other works.  Specifically, we choose five pretrain-
based methods (i.e., _Word2Vec_ (Mikolov et al., 2013), _Doc2Vec_ (Le and
Mikolov, 2014), _Code2Vec_ (Alon et al., 2019), _CodeBERT_ (Feng et al.,
2020), and _CodeT5_ (Wang et al., 2021)), nine traditional code clone
detectors (i.e., _SourcererCC_ (Sajnani et al., 2016), _CCFinder_ (Kamiya et
al., 2002), _Nicad_ (Roy and Cordy, 2008), _Decakrd_ (Jiang et al., 2007),
_CCAligner_ (Wang et al., 2018), _Oreo_ (Saini et al., 2018), _LVMapper_ (Wu
et al., 2020a), _NIL_ (Nakagawa et al., 2021), and _Tamer_ (Hu et al., 2023)),
and six deep-learning-based code clone detectors (i.e., _RtvNN_ (White et al.,
2016), _CDLH_ (Wei and Li, 2017), _TBCNN_ (Mou et al., 2016), _ASTNN_ (Zhang
et al., 2019), _SCDetector_ (Wu et al., 2020b), and _FCCA_ (Hua et al.,
2020)).  Note that we use the parameters recommended in each paper for these
works, and the experimental results are also close to the best result reported
in their paper to ensure credibility.

####  4.1.4. Experimental Environment and Metrics.

For token categorization, we make use of a Python library (i.e., _javalang_
(jav, 2022)) to complete our lexical analysis. After parsing the source code
into typed tokens, we leverage _Pytorch_ to implement the self-attention
mechanism layers and contrastive learning to train the encoder. Given two
vectors of two methods, we use another Python library (i.e., _Sklearn_ (skl,
2022)) to compute the cosine similarity. To measure the effectiveness of
_CC2Vec_ , we adopt the following widely used metrics. Precision is defined as
$P=TP/(TP+FP)$. Recall is defined as $R=TP/(TP+FN)$. F1 is defined as
$F1=2*P*R/(P+R)$. Among them, _true positive_ (TP) represents the number of
samples correctly classified as clone pairs, _false positive_ (FP) represents
the number of samples incorrectly classified as clone pairs, and _false
negative_ (FN) represents the number of samples incorrectly classified as non-
clone pairs. We employ ten-fold cross-validation to record the F1 score,
precision, and recall for each validation run on all experiments. We calculate
the average of these metrics across the ten validations and consider this
average as the final performance.

###  4.2. RQ1: Detection Effectiveness

In this subsection, we evaluate our proposed system by comparing it to three
types of code clone detectors, i.e., pretrain-based methods, traditional
scalable methods, and deep learning-based methods. Firstly, we assess _CC2Vec_
’s ability to detect syntactic code clones compared to other popular pretrain-
based methods and traditional scalable tools. Secondly, we fine-tune _CC2Vec_
with neural networks and evaluate its semantic code clones detection
effectiveness compared to widely used deep learning-based detectors.

####  4.2.1. Performance of CC2Vec Compared to Pretrain-based Methods.

We first analyze our method’s effectiveness in detecting syntactic code clones
compared to pretrain-based methods.  Similar to most of the code clone
detectors (Sajnani et al., 2016), when we set the detection threshold of
_CC2Vec_ to 0.7, the precision under this threshold is 98%. At the same time,
we find that the detection thresholds of different pretrain-based methods are
not the same, and to ensure a reasonable comparison, we adapt the thresholds
of the other detection methods to make all of them have an accuracy rate of
98%. The threshold of _Word2Vec_ , _Doc2Vec_ , _Code2Vec_ , _CodeBERT_ , and
_CodeT5_ is 0.85, 0.75, 0.91, 0.95 and 0.87, respectively. At the same time,
we analyze the recall on cloned code pairs to evaluate the ability of these
pretrain-based methods.

Table 2. Results of pretrain-based methods and our method _CC2Vec_ on BigCloneBench dataset Methods | Recall | Precision
---|---|---
T1 | T2 | VST3 | ST3 | MT3 | T4
Word2Vec | 1 | 1 | 0.99 | 0.85 | 0.53 | 0.1 | 0.98
Doc2Vec | 1 | 0.95 | 0.86 | 0.57 | 0.21 | 0.02 | 0.98
Code2Vec | 1 | 1 | 0.95 | 0.82 | 0.54 | 0.17 | 0.98
CodeBERT | 1 | 1 | 0.95 | 0.89 | 0.71 | 0.23 | 0.98
CodeT5 | 1 | 1 | 0.98 | 0.91 | 0.77 | 0.49 | 0.98
CC2Vec | 1 | 1 | 0.97 | 0.93 | 0.81 | 0.64 | 0.98

As shown in Table 2, all methods achieved good performance on easier-to-find
clones like Type-1, Type-2, and _Very Strongly Type-3_ (VST3), being able to
detect and recall the vast majority of them. However, we also see that the
recall scores of other pretrain-based methods on _Moderately Type-3_ (MT3) and
Type-4 perform poorly and are far lower than their precision scores. Although
these code clone detectors model the code language through code corpus, they
do not capture the semantic information in the pre-training corpus, but only
learn the surface token-based information of the code and obtain an efficient
method representation. The recall scores of _CodeBERT_ and _CodeT5_ on Type-4
clones are 23% and 49%, respectively, which indicates that they are capable of
detecting Type-4 code clones. This is reasonable since _CodeBERT_ and _CodeT5_
are pre-trained on a massive code corpus (Feng et al., 2020; Wang et al.,
2021; Dou et al., 2024) and have a much higher number of parameters than the
other four pretraining-based methods.  On the other hand, despite _CodeBERT_
and _CodeT5_ having a large number of parameters and being pre-trained on
extensive code corpora, they do not specifically incorporate the code clone
detection task into pre-training tasks, nor do they design optimization goals
for code clone detection. As a result, they face difficulty in accurately
detecting Type-4 code clones.

For _CC2Vec_ , the experimental results show that the recall score on Type-4
clones is 64% with 98% precision, achieving the best performance in pretrain-
based methods on BigCloneBench datasets.  This is reasonable that _CC2Vec_
considers the potential relationships between tokens, thereby extracting more
information than other vanilla pre-trained models during the training phase.
Meanwhile, contrastive learning provides a better way to learn the information
between the code snippets and its clone code snippets (i.e., positive samples)
and other non-clone segments (i.e., negative samples) within a batch.
Moreover, in contrast to _CodeBERT_ , _CC2Vec_ do not require a massive pre-
training code corpus and a large number of parameters.  _CC2Vec_ is only pre-
trained on the training dataset, such as _Word2Vec_ and _Doc2Vec_.

In summary, the results indicate that _CC2Vec_ outperforms other pretrained-
based methods in detecting syntactic code clones, while providing the
capability to detect semantic code clones. However, the ability to detect
semantic code clones by only using CC2Vec remains limited.

####  4.2.2. Performance of CC2Vec Compared to Traditional Tools.

To further examine the effectiveness of _CC2Vec_ in detecting syntactic code
clones, we also evaluate _CC2Vec_ compared to nine traditional scalable
detection tools (i.e., _SourcererCC_ (Sajnani et al., 2016), _CCFinder_
(Kamiya et al., 2002), _NiCad_ (Roy and Cordy, 2008), _Deckard_ (Jiang et al.,
2007), _CCAligner_ (Wang et al., 2018), _Oreo_ (Saini et al., 2018),
_LVMapper_ (Wu et al., 2020a), _NIL_ (Nakagawa et al., 2021), and _Tamer_ (Hu
et al., 2023)).

The experimental results are described in Table 3. Many detectors achieve
perfect or near-perfect recall for T1 and T2 scenarios, while challenges
remain for more semantic clone types like Type-3 and Type-4. The results show
that _SourcererCC_ (Sajnani et al., 2016) achieves very low recall scores on
MT3 and Type-4 clones. It is reasonable that _SourcererCC_ only considers the
overlap similarity of tokens between two methods. Therefore, it can not handle
semantic clones because it does not consider program semantics. Similar
performance effects are found in all other traditional code clone detectors
(i.e., the recall of them on Type-4 clones is approximately zero) that are
poor at detecting clone pairs at the semantic level (i.e., Type-4 clone
pairs). On the other hand, _CC2Vec_ can outperform all other traditional
methods on MT-3 and Type-4 clones (i.e., the recall is 81% and 64%,
respectively) and achieve a high precision (i.e., the precision is 98%).

In summary, these results further indicate that _CC2Vc_ can learn more
potential relationship information between tokens to detect syntactic code
clones, while enhancing a certain ability to detect semantic code clones.

Table 3. Results of the other traditional methods and _CC2Vec_ on BigCloneBench dataset Methods | Recall | Precision
---|---|---
T1 | T2 | VST3 | ST3 | MT3 | T4
SourcererCC | 1 | 0.97 | 0.93 | 0.6 | 0.05 | 0 | 0.98
CCFinder | 1 | 0.93 | 0.62 | 0.15 | 0.01 | 0.0 | 0.72
NiCad | 1 | 0.99 | 0.98 | 0.52 | 0.02 | 0 | 0.99
Deckard | 0.6 | 0.52 | 0.62 | 0.31 | 0.12 | 0.01 | 0.35
CCAligner | 1 | 1 | 0.99 | 0.65 | 0.14 | 0 | 0.61
Oreo | 1 | 1 | 1 | 0.89 | 0.3 | 0.01 | 0.89
LVMapper | 1 | 1 | 0.98 | 0.81 | 0.19 | 0 | 0.59
NIL | 1 | 0.97 | 0.88 | 0.66 | 0.19 | 0 | 0.94
Tamer | 1 | 1 | 1 | 0.99 | 0.53 | 0.03 | 0.96
CC2Vec | 1 | 1 | 0.97 | 0.93 | 0.81 | 0.64 | 0.98

####  4.2.3. Performance of CC2Vec Compared to Deep Learning-based Code Clone
Detectors.

To further evaluate our proposed method’s efficacy in identifying semantic
code clones, we carry out comparative experiments on two extensively used
datasets (i.e., BigCloneBench and GoogleCodeJam), against six popular deep
learning-based code clone detection techniques (i.e., _RtvNN_ (White et al.,
2016), _CDLH_ (Wei and Li, 2017), _TBCNN_ (Mou et al., 2016), _ASTNN_ (Zhang
et al., 2019), _SCDetector_ (Wu et al., 2020b) and _FCCA_ (Hua et al., 2020)).
Notably, we enhance our method’s ability to detect semantic clone pairs by
integrating the pre-trained encoder _CC2Vec_ with several fully connected
layers, as described in Section 3.4. The integrated model is subsequently
fine-tuned utilizing the identical training dataset employed during the pre-
training phase of _CC2Vec_. This approach mitigates any potential leakage of
validation and test data into the training process, thereby safeguarding the
integrity and reliability of the evaluation.

Table 4. Results of _CC2Vec_ and comparative systems for each clone type on BigCloneBench dataset and GoogleCodeJam dataset Methods | BigCloneBench | GoogleCodeJam
---|---|---
Recall | Precision | F1 | Recall | Precision | F1
RtvNN | 0.01 | 0.95 | 0.01 | 0.90 | 0.20 | 0.33
CDLH | 0.74 | 0.92 | 0.82 | 0.70 | 0.46 | 0.55
TBCNN | 0.81 | 0.90 | 0.85 | 0.89 | 0.91 | 0.90
ASTNN | 0.94 | 0.92 | 0.93 | 0.87 | 0.95 | 0.91
SCDetector | 0.92 | 0.97 | 0.94 | 0.87 | 0.81 | 0.82
FCCA | 0.92 | 0.98 | 0.95 | 0.90 | 0.95 | 0.92
CC2Vec-1L | 0.96 | 0.92 | 0.94 | 0.94 | 0.91 | 0.93
CC2Vec-3L | 0.97 | 0.94 | 0.96 | 0.95 | 0.93 | 0.94
CC2Vec-5L | 0.97 | 0.94 | 0.96 | 0.95 | 0.93 | 0.94

Table 4 describes the Recall, Precision, and F1 scores of our proposed method
and six comparative deep learning-based code clone detectors on two datasets.
As in the above experiments, we also use ten-fold cross-validation to record
these metrics and report the average scores.  _CC2Vec-1L_ , _CC2Vec-3L_ and
_CC2Vec-5L_ represent the encoder _CC2Vec_ followed by one, three and five
fully connected layers, respectively. A ReLU activation layer is inserted in
the middle of every two fully connected layers, while for _CC2Vec-1L_ , there
is no activation layer included.  Among the semantic code clone detectors
evaluated,  _RtvNN_ , _CDLH_ , _TBCNN_ and _ASTNN_ are all AST-based clone
detection methods. As for _RtvNN_ , it applies the RNN model to encode both
source code tokens and AST to detect code clones. However, the tree structure
may change significantly even when the code changes slightly. The F1 score of
_RtvNN_ is 1% and 33% for BigCloneBench dataset and GoogleCodeJam dataset,
respectively. The other three AST-based methods performed well on both
datasets. For example, the F1 of _ASTNN_ on the BigCloneBench dataset and
GoogleCodeJam dataset achieves 93% and 91%, respectively.  Another type of
semantic code clone detection tools are graph-based, such as _SCDetector_ and
_FCCA_.  These two graph-based methods also have good detection performance.
For the BigCloneBench dataset, the recall of _SCDetector_ and _FCCA_ is 94%
and 82%, respectively. As for the GoogleCodeJam dataset, their recall score is
95% and 92%, respectively.

For _CC2Vec_ , when the fully connected layer number is three (i.e.,
_CC2Vec-3L_), the F1 score is 96% and 94% for BigCloneBench dataset and
GoogleCodeJam dataset, respectively, achieving comparable performance to other
popular deep learning-based methods. At the same time, we found that
increasing the number of fully connected layers to five (i.e., _CC2Vec-5L_)
does not result in a significant performance improvement.

In summary, the experimental results demonstrate that through fine-tuning the
integrated model, our method can effectively detect all types of code clones
including syntactic and semantic code clones. By using token categorization
and program encoding, the trainable encoder _CC2Vec_ can capture token-level
potential relationship information in code segments from code corpus.
Meanwhile, contrastive learning enables the encoder can not only learn
information from clone pairs, but also learn from non-clone pairs dynamically
constructed during the training phase. This learning approach enhances the
encoder’s ability to identify subtle semantic differences between code
snippets, to further achieve an improvement in detecting code clones.

###  4.3. RQ2: Self-Attention and Contrastive Learning

In this subsection, we conduct three single-factor experiments to validate the
use of self-attention mechanism layers and contrastive learning can contribute
to the effectiveness of _CC2Vec_ on clone detection. Our first experiment
adopts two GRU layers to encode the program source code. After training the
encoder by using an encoder-decoder model, we can use it to embed other code
methods. The output of the learned encoder is the corresponding vector of a
method. Given two methods, their cosine similarity will be computed as the
standard to detect code clones. If it is higher than 70%, they will be
reported as a clone pair. Our second experiment replaces the two GRU layers
with two self-attention mechanism layers as the program encoder. However, we
do not perform contrastive learning to train the encoder but use an encoder-
decoder model to train it. Similar to the first experiment, we also compute
the cosine similarity of the two methods’ vectors to check whether they are a
clone pair or not. Our third experiment is to implement _CC2Vec_. In other
words, we use two self-attention mechanism layers as the program encoder and
leverage contrastive learning to train it. The clone detection phase is the
same as in the former two experiments. In summary, the difference between the
first and second experiments is only in the program encoder, and the
difference between the second and third experiments is only in the training
phase.

Table 5. Results on BigCloneBench dataset | Recall | Precision | F1
---|---|---|---
Gated Recurrent Unit (GRU) | 0.31 | 0.24 | 0.27
Self-Attention | 0.44 | 0.75 | 0.55
Self-Attention + Contrastive Learning (_CC2Vec_) | 0.74 | 0.98 | 0.84

We describe the results of these three comparative experiments in Table 5. It
shows that the use of self-attention mechanism layers can increase the
detection accuracy of _CC2Vec_. For example, the F1 score of the first
experiment is only 27%, while it increases to 55% when we adopt self-attention
mechanism layers to encode source code. It is reasonable because the adoption
of self-attention mechanism layers can retain some potential relationships
between tokens, and these relationships may contain the program details,
resulting in better detection effectiveness in detecting code clones.
Moreover, when detecting similar code fragments, self-attention mechanism
layers can assign larger weights to similar categories while reducing
dissimilar categories’ weights. These weights can boost the ability of code
clone detection.

Meanwhile, we also find that integrating contrastive learning enhances the
robustness of _CC2Vec_ to code changes of different implementations.
Specifically, when we employ contrastive learning in training the program
encoder, the F1 score sees a significant jump to 84%, compared to just 55%
without its use. The distance between such semantically similar code pairs is
minimized, making it robust to the code changes introduced by different
implementations.

In summary, through the results in Table 5, we see that both self-attention
mechanism layers and contrastive learning can boost the ability of _CC2Vec_ on
code clone detection.

###  4.4. RQ3: Interpretability

Since we leverage self-attention mechanism layers as our program encoder, the
assigned weights of different categories can interpret the detection results.
As a matter of fact, the dimension of the attention weight matrix is 15×15,
corresponding to 15 categories of tokens, and the value of each position ($i$,
$j$) in the matrix represents the correlation between category $i$ and
category $j$. If the matrix column items are summed, and input into a
_Softmax()_ layer, the importance (i.e., weight) of each category can be
obtained. To validate the interpretability of _CC2Vec_ , we use methods in
List 1 and List 2 as our examples. Table 6 shows the weights of different
token categories in List 1 calculated from the second self-attention layer.
Since the tokens of the method in List 1 can only be divided into seven
categories, the weights of the other eight categories are all 0.

Through the results in Table 6, we observe that the weight of _BasicType_
token category is the largest, which means that tokens in _BasicType_ category
have the greatest effect in determining whether List 1 and List 2 are code
clones or not. To further check whether the weights assigned by _CC2Vec_ are
suitable or not, we choose tokens in _BasicType_ category and _Keyword_
category as our objects because the weights of these two categories are the
largest and smallest in Table 6. More specifically, we first obtain the
corresponding vectors of all tokens in _BasicType_ category and _Keyword_
category by word2vec (Mikolov et al., 2013). To show the vectors more
intuitively, we perform a visualization technique (i.e., TSNE (tsn, 2022)) to
visualize them in Figure 7. The figure shows that the vector representations
of the token “int” and the token “long” are similar. To obtain more
determinate results, we calculate the cosine distance between them. After
calculation, we find that their similarity is 87%. For List 1 and List 2, the
tokens in _BasicType_ category only have token “int” and token “long”,
respectively. Therefore, the similarity between the two categories is the
similarity between token “int” and token “long”, which is 87%. Such a high
similarity makes _CC2Vec_ assign a higher weight. Moreover, we also compute
the cosine similarity between the tokens “while” and “if”, finding a mere 12%
similarity, indicating their dissimilarity. Meanwhile, results in Figure 7
also indicate that the similarity between token “return”, token “while”, and
token “if” are low, resulting in the similarity of _Keyword_ category between
List 1 and List 2 is low. It is the reason why _CC2Vec_ assigns a smaller
weight to _Keyword_ category than _BasicType_ category.

In summary, incorporating self-attention mechanism layers in CC2Vec provides
interpretability for code clone detection outcomes.

Table 6. Weights of different categories

Token Category | Attention Weight
---|---
BasicType | 0.1922
Operator | 0.1739
Separator | 0.1573
Identifier | 0.1424
Modifier | 0.1424
DecimalInterger | 0.1055
Keyword | 0.0864

Figure 7. The visualization of token vectors

###  4.5. RQ4: Scalability

Our final experiment aims to examine the ability of _CC2Vec_ on large-scale
code analysis. Specifically, we randomly choose one million code pairs from
BigCloneBench and run _CC2Vec_ and our comparative systems on them to collect
the runtime overheads. We run these tools three times and report the average
as their final runtime. For pretrain-based methods (i.e., _Word2Vec_ ,
_Doc2Vec_ , _Code2Vec_ , _CodeBERT_ , _CodeT5_ , and _CC2Vec_), the training
time indicates the time for pre-trained with the code corpus and the
prediction time is the sum of the time to obtain vectors and the time spent in
computing the cosine similarity. For example, we should perform contrastive
learning to train a self-attention-based program encoder first. Otherwise, we
cannot detect code clones by using _CC2Vec_. For deep learning-based methods
(i.e., _RtvNN_ , _CDLH_ , _TBCNN_ , _ASTNN_ , _SCDetector_ , _FCCA_ and
_CC2Vec-3L_), they need to train a model first, and then the model can be used
to detect code clones. In other words, their runtime overheads are divided
into two parts, corresponding to training time and prediction time.

Table 7. Time performance of code clone detectors on analyzing one million code pairs Tools | Prediction Time | Training Time
---|---|---
Pretrain-based methods | Word2Vec | 25s | 7,523s
Doc2Vec | 28s | 9,126s
Code2Vec | 30s | 17,345s
CodeBERT | 274s | -
| CodeT5 | 873s | -
Traditional methods | SourcererCC | 16s | 0s
CCFinder | 23s | 0s
NiCad | 14s | 0s
Deckard | 72s | 0s
CCAligner | 10s | 0s
Oreo | 15s | 0s
LVMapper | 11s | 0s
NIL | 9s | 0s
Tamer | 10s | 0s
Deep learning-based methods | RtvNN | 35s | 5,206s
CDLH | 90s | 45,317s
TBCNN | 86s | 41,168s
ASTNN | 2,894s | 16,096s
SCDetector | 139s | 2,937s
FCCA | 91s | 56,769s
Ours | CC2Vec | 28s | 3,649s
CC2Vec-3L | 83s | 4,871s

The runtime overheads of our proposed methods (i.e., _CC2Vec_ and _CC2Vec-3L_)
and other detection systems are presented in Table 7. _CC2Vec_ is trained
faster than other pretrain-based methods. For example, the training time of
_Code2Vec_ is about 17,345 seconds, and _CC2Vec_ takes about 3,649 seconds.
The training process for _CodeBERT_ is notably resource-intensive and time-
consuming (Feng et al., 2020; Wang et al., 2021), hence its training duration
is not measured.  The prediction time of _CodeT5-base_ exceeds that of
_CodeBERT_. It is reasonable that _CodeT5_ has a greater number of parameters
compared to _CodeBERT_ and it is also an encoder-decoder model (Wang et al.,
2021).  Since traditional methods do not need to train, their training time is
zero. As for prediction time, the token-based method _NIL_ consumes the least
runtime because it only uses an N-gram representation of token sequences for
clone detection. Compared to _NIL_ , _CC2Vec_ and _CC2Vec-3L_ take more
runtime to predict the same code pairs. It is reasonable because _CC2Vec_
conducts lexical analysis to divide the tokens into different categories,
which consumes some time to complete the step. And _CC2Vec-3L_ fine-tuned
based on _CC2Vec_ combined with neural networks. Compared to deep learning-
based methods, _CC2Vec-3L_ are more scalable. It takes about 4,871 seconds to
finish the training procedure and 83 seconds to detect one million code pairs.
The training phase consumes more time than _SCDetector_ (i.e., 2,937 seconds)
and more prediction time than _RtvNN_ (i.e., 35 seconds), but is faster than
other deep learning-based cloned code detection tools. Compared to _RtvNN_ ,
whether from the training or prediction phase, the runtime overhead of
_CC2Vec_ is lower. For _ASTNN_ , it requires 2,894 seconds to predict the one
million code pairs, which is about 34 times slower than _CC2Vec-3L_.  As for
_FCCA_ , our method has a significant advantage in detection efficiency during
the training phase. In addition, the detecting time of _CC2Vec-3L_ is also
less than _FCCA_ (i.e., the detecting time of _CC2Vec-3L_ and _FCCA_ are 83s
and 91s, respectively).

In summary, due to self-attention mechanism layers and contrastive learning,
_CC2Vec_ and _CC2Vec-3L_ are not as fast as traditional methods.  However, it
consumes less time than most other pretrain-based methods and deep learning-
based methods and also can achieve comparable performance to these deep
learning-based methods. Efficient detectors can reduce time and resource
consumption and have the potential to detect large-scale code clones.

##  5\. Discussions

###  5.1. Threats to Validity

The first threat comes from the BigCloneBench dataset, a recent study (Krinke
and Ragkhitwetsagul, 2022) has shown that the dataset has some quality issues
which may cause negative effects on _CC2Vec_. To mitigate the threat and make
our experimental results more trustworthy, we also validate _CC2Vec_ on
another widely used code clone benchmark (i.e., the Google Code Jam dataset).
The experimental results still show that _CC2Vec_ can perform well on
detecting code clones. The second threat comes from the Google Code Jam
dataset, it may contain several inaccuracies since some participants may have
misunderstood the task and implemented something semantically different. To
mitigate the threat, we randomly choose 10% samples from each competition
problem and manually verify the functionality. The results show that these
samples all correctly address the problem. The third threat comes from the
token types, in the token categorization phase, we select a total of 15
categories to commence our experiments. The selection of these categories may
cause some inaccuracies since the total number of token categories parsed by
_javalang_ is not clear. To mitigate the situation, we perform a statistical
analysis to select the categories with a high number of occurrences and add a
_Null_ category to represent the remaining categories. The fourth threat comes
from the runtime overhead. The calculation of runtime overheads of _CC2Vec_
and its comparative tools may also cause some inaccuracies due to the
different machine states such as CPU usage. We mitigate the threat by
conducting evaluations three times and reporting the average runtime overhead
in our paper.

###  5.2. Differences From C4

The most similar work to _CC2Vec_ is _C4_ (Tao et al., 2022) which uses
contrastive learning to detect cross-language code clones. However, our method
greatly differs from _C4_. Firstly, _CC2Vec_ processes code snippets through
syntactic analysis, tokenization, and categorization. Secondly, it utilizes a
two-layer attention mechanism to capture the relationships between tokens of
the same category and different categories within the language. Finally, we
use contrastive learning to derive the code vector. Our primary contribution
is how we capture the relationships among tokens in the code, while the
subsequent contrastive learning aims to obtain a better code vector. In
contrast, _C4_ directly tokenizes code snippets and uses CodeBERT for
representation, with training via contrastive learning. Contrastive learning
is a general method, but our contribution is to capture the relationships
between tokens, which is not explored in _C4_. Additionally, our research
field differs from that of _C4_. _C4_ investigates cross-language clone
detection, whereas _CC2Vec_ focuses on obtaining better code vectors for more
rapid clone detection. We do not use CodeBERT, a large model, but instead
train with a lightweight network having only two layers of attention, making
our detection more efficient.

###  5.3. Why does CC2Vec Perform Better?

The reasons why _CC2Vec_ is superior to other token-based systems are mainly
two-fold. First, _CC2Vec_ uses self-attention mechanism layers as the program
encoder which can not only compute the corresponding weights to interpret the
detection results but also retain some potential relationships between tokens.
It is just like a data flow, when some tokens are defined or used earlier, the
attention mechanism can establish connections between the definition and the
reference locations. These connections can better aid in the detection of
clones. In addition, the attention mechanism is a double-direction neural
network, which means that it can capture the relationships between tokens
better. This method is similar to language in natural language processing,
where attention can establish connections between two tokens that are far
apart. These potential relationships between tokens can retain some program
details between source code tokens.  Second, the use of contrastive learning
in _CC2Vec_ can resist the changes of code structures introduced by different
implementations, making it possible to detect some semantic code clones.

##  6\. Related Work

According to the extraction of different representations, existing code clone
detection methods can be divided into five main categories: text-based, token-
based, tree-based, graph-based, and metrics-based approaches.

Text-based methods (Dou et al., 2023; Ducasse et al., 1999; Roy and Cordy,
2008; Wettel and Marinescu, 2005; Lee and Jeong, 2005; Barbour et al., 2010;
Marcus and Maletic, 2001) compute the similarity between two code fragments in
the form of text. Wettel et al. (Wettel and Marinescu, 2005) propose a method
for clone detection based on code line comparison. Lee et al. (Lee and Jeong,
2005) design _SDD_ to detect large-scale code clones. Token-based methods
(Kamiya et al., 2002; Göde and Koschke, 2009; Sajnani et al., 2016; Li et al.,
2017; Wang et al., 2018; Murakami et al., 2013) need to transform the source
code into tokens by lexical analysis, then the similarity can be computed by
analyzing tokens. Murakami et al. (Murakami et al., 2013) use a hash algorithm
to convert the code fragment into a token sequence and introduce a classic and
highly efficient Smith-Waterman algorithm to detect code clones. Sainj et al.
(Sajnani et al., 2016) implement _SourcererCC_ to capture the tokens’ overlap
similarity between different methods to detect near-miss Type-3 clones.
_SourcererCC_ is the most scalable code clone detector which can scale to scan
more than 428 million files on Github (Lopes et al., 2017). Due to the lack of
program details, these text-based and token-based methods are difficult to
handle semantic clones.

Tree-based methods (Jiang et al., 2007; Wei and Li, 2017; Zhang et al., 2019;
Baxter et al., 1998; Yang, 1991; Wahler et al., 2004; Koschke et al., 2006; Wu
et al., 2022; Hu et al., 2022) need to extract the _abstract syntax tree_
(AST) of a code method and then use it to detect code clones. Baxter et al.
(Baxter et al., 1998) design a tree-based code clone detector _CloneDR_. It
first uses a compiler tool to generate the AST of the code, then calculates
the hash code of each subtree, and finally performs similarity matching based
on the obtained hash code. Wei et al. (Wei and Li, 2017) introduce _CDLH_
which leverages Tree-LSTM (Tai et al., 2015) on generated binary trees to
encode them. Although tree-based methods can detect semantic code clones,
however, they suffer from low scalability because of the large execution times
(Zhang et al., 2019). Graph-based methods (Krinke, 2001; Komondoor and
Horwitz, 2001; Wang et al., 2017; Zhao and Huang, 2018; Chen et al., 2014; Wu
et al., 2020b; Shan et al., 2023) distill the program semantics into different
graph representations, such as _control flow graph_ (CFG) and _program
dependency graph_ (PDG). Komondoor et al. (Komondoor and Horwitz, 2001) and
Krinke et al. (Krinke, 2001) both use PDG as the standard to detect code
clones. They apply graph analysis to discover isomorphic subgraphs and use
these subgraphs to detect semantic code clones. Wu et al. (Wu et al., 2020b)
apply centrality analysis to transform the CFG of a method into certain
semantic tokens and detect code clones by analyzing these semantic tokens.
Graph-based methods can also detect semantic code clones, but due to the
complexity of graph isomorphism and the heavy-weight time consumption of graph
matching, they can not scan codes on a large scale.

##  7\. Conclusion

In this paper, we propose to boost the capability of code clone systems to
efficiently detect syntactic and semantic code clones. To complete the
purpose, we introduce _CC2Vec_ , a code encoding method for code clone
detection. _CC2Vec_ uses two self-attention mechanism layers as the program
encoder and applies contrastive learning to train the encoder. We can detect
simple code clones by directly utilizing _CC2Vec_ , while identifying more
complicated code clones by combining it with a few neural networks.  Through
the experimental results, we find that _CC2Vec_ can detect more code clones
than five pretrain-based methods and 15 state-of-the-art code clone detectors.
As for scalability, although _CC2Vec_ is not as fast as traditional methods,
it surpasses the majority of pretrain-based and deep learning-based methods.

## References

  * (1)
  * RMS ([n. d.]) [n. d.].  RMSProp Optimizer.  https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
  * gcj (2017) 2017\.  Google Code Jam.  https://code.google.com/codejam/past-contests.
  * big (2020) 2020\.  BigCloneBench.  https://github.com/clonebench/BigCloneBench.
  * ope (2021a) 2021a.  Open Source Technology Trends 2021 – What’s there for you?  https://www.hiddenbrains.com/blog/open-source-technology-trends.html.
  * ope (2021b) 2021b.  What the 2021 OSSRA report tells us about the state of open source in commercial software.  https://www.synopsys.com/blogs/software-security/open-source-trends-ossra-report/.
  * jav (2022) 2022\.  javalang is a pure Python library for working with Java source code.  https://github.com/c2nes/javalang/.
  * skl (2022) 2022\.  Machine Learning in Python.  https://scikit-learn.org/stable/.
  * pyc (2022) 2022\.  pycparser is a complete parser of the C language.  https://pypi.python.org/pypi/pycparser/.
  * tsn (2022) 2022\.  t-distributed Stochastic Neighbor Embedding.  https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html.
  * cri (2023) 2023\.  Criticality score.  https://github.com/ossf/criticality_score.
  * Alon et al. (2019) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.  code2vec: Learning distributed representations of code.  _Proceedings of the ACM on Programming Languages_ 3, POPL (2019), 1–29.
  * Anand (2020) Ankesh Anand. 2020.  Contrastive Self-Supervised Learning.  https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html.
  * Barbour et al. (2010) Liliane Barbour, Hao Yuan, and Ying Zou. 2010.  A technique for just-in-time clone detection in large scale systems. In _2010 IEEE 18th International Conference on Program Comprehension_. IEEE, 76–79.
  * Baxter et al. (1998) Ira D Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant’Anna, and Lorraine Bier. 1998.  Clone detection using abstract syntax trees. In _Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272)_. IEEE, 368–377.
  * Bellon et al. (2007) Stefan Bellon, Rainer Koschke, Giulio Antoniol, Jens Krinke, and Ettore Merlo. 2007.  Comparison and evaluation of clone detection tools.  _IEEE Transactions on Software Engineering_ 33, 9 (2007), 577–591.
  * Chen et al. (2014) Kai Chen, Peng Liu, and Yingjun Zhang. 2014.  Achieving accuracy and scalability simultaneously in detecting application clones on android markets. In _Proceedings of the 36th International Conference on Software Engineering (ICSE’14)_. 175–186.
  * Dou et al. (2024) Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Junjie Shan, Caishuang Huang, Wei Shen, Xiaoran Fan, Zhiheng Xi, et al. 2024\.  StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback.  _arXiv preprint arXiv:2402.01391_ (2024).
  * Dou et al. (2023) Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang. 2023.  Towards understanding the capability of large language models on code clone detection: a survey.  _arXiv preprint arXiv:2308.01191_ (2023).
  * Ducasse et al. (1999) Stéphane Ducasse, Matthias Rieger, and Serge Demeyer. 1999.  A language independent approach for detecting duplicated code. In _Proceedings of the 1999 International Conference on Software Maintenance (ICSM’99)_. 109–118.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  Codebert: A pre-trained model for programming and natural languages.  _arXiv preprint arXiv:2002.08155_ (2020).
  * Göde and Koschke (2009) Nils Göde and Rainer Koschke. 2009.  Incremental clone detection. In _Proceedings of the 2009 European Conference on Software Maintenance and Reengineering (ECSMR’09)_. 219–228.
  * Hu et al. (2023) Tiancheng Hu, Zijing Xu, Yilin Fang, Yueming Wu, Bin Yuan, Deqing Zou, and Hai Jin. 2023.  Fine-Grained Code Clone Detection with Block-Based Splitting of Abstract Syntax Tree. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_. 89–100.
  * Hu et al. (2022) Yutao Hu, Deqing Zou, Junru Peng, Yueming Wu, Junjie Shan, and Hai Jin. 2022.  TreeCen: Building tree graph for scalable semantic code clone detection. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Hua et al. (2020) Wei Hua, Yulei Sui, Yao Wan, Guangzhong Liu, and Guandong Xu. 2020.  Fcca: Hybrid code representation for functional clone detection using attention networks.  _IEEE Transactions on Reliability_ 70, 1 (2020), 304–318.
  * Jiang et al. (2007) Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007.  Deckard: Scalable and accurate tree-based detection of code clones. In _Proceedings of the 29th International Conference on Software Engineering (ICSE’07)_. 96–105.
  * Kamiya et al. (2002) Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002.  CCFinder: A multilinguistic token-based code clone detection system for large scale source code.  _IEEE Transactions on Software Engineering_ 28, 7 (2002), 654–670.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_ (2020).
  * Komondoor and Horwitz (2001) Raghavan Komondoor and Susan Horwitz. 2001.  Using slicing to identify duplication in source code. In _Proceedings of the 2001 International Static Analysis Symposium (ISAS’01)_. 40–56.
  * Koschke et al. (2006) Rainer Koschke, Raimar Falke, and Pierre Frenzel. 2006.  Clone detection using abstract syntax suffix trees. In _2006 13th Working Conference on Reverse Engineering_. IEEE, 253–262.
  * Krinke (2001) Jens Krinke. 2001.  Identifying similar code with program dependence graphs. In _Proceedings of the 8th Working Conference on Reverse Engineering (WCRE’01)_. 301–309.
  * Krinke and Ragkhitwetsagul (2022) Jens Krinke and Chaiyong Ragkhitwetsagul. 2022.  BigCloneBench Considered Harmful for Machine Learning. In _2022 IEEE 16th International Workshop on Software Clones (IWSC)_. IEEE, 1–7.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. PMLR, 1188–1196.
  * Lee and Jeong (2005) Seunghak Lee and Iryoung Jeong. 2005.  SDD: high performance code clone detection system for large scale source code. In _Companion to the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications_. 140–141.
  * Li et al. (2017) Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara Ryder. 2017.  CClearner: A deep learning-based clone detection approach. In _Proceedings of the 2017 International Conference on Software Maintenance and Evolution (ICSME’17)_. 249–260.
  * Lopes et al. (2017) Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017.  DéjàVu: a map of code duplicates on GitHub.  _Proceedings of the ACM on Programming Languages (OOPSLA’17)_ (2017), 1–28.
  * Marcus and Maletic (2001) Andrian Marcus and Jonathan I Maletic. 2001.  Identification of high-level concept clones in source code. In _Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001)_. IEEE, 107–114.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _Computer Science_ (2013).
  * Mou et al. (2016) Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016.  Convolutional neural networks over tree structures for programming language processing. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * Murakami et al. (2013) Hiroaki Murakami, Keisuke Hotta, Yoshiki Higo, Hiroshi Igaki, and Shinji Kusumoto. 2013.  Gapped code clone detection with lightweight source code analysis. In _2013 21st International Conference on Program Comprehension (ICPC)_. IEEE, 93–102.
  * Nakagawa et al. (2021) Tasuku Nakagawa, Yoshiki Higo, and Shinji Kusumoto. 2021.  Nil: large-scale detection of large-variance clones. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 830–841.
  * Roy and Cordy (2008) CK. Roy and JR. Cordy. 2008.  NICAD: Accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization. In _Proceedings of the 2008 International Conference on Program Comprehension (ICPC’08)_. 172–181.
  * Roy and Cordy (2007) Chanchal Kumar Roy and JR. Cordy. 2007.  A survey on software clone detection research.  _Queen’s School of Computing TR_ 541, 115 (2007), 64–68.
  * Saini et al. (2018) Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Pierre Baldi, and Cristina V Lopes. 2018.  Oreo: Detection of clones in the twilight zone. In _Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE’18)_. 354–365.
  * Sajnani et al. (2016) Hitesh Sajnani, Vaibhav Saini, Jeffrey Svajlenko, CK. Roy, and CV. Lopes. 2016.  SourcererCC: Scaling code clone detection to big code. In _Proceedings of the 38th International Conference on Software Engineering (ICSE’16)_. 1157–1168.
  * Shan et al. (2023) Junjie Shan, Shihan Dou, Yueming Wu, Hairu Wu, and Yang Liu. 2023.  Gitor: Scalable Code Clone Detection by Building Global Sample Graph. In _Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 784–795.
  * Svajlenko et al. (2014) Jeffrey Svajlenko, JF. Islam, Iman Keivanloo, CK. Roy, and Mohammad Mamun Mia. 2014.  Towards a big data curated benchmark of inter-project code clones. In _Proceedings of the 2014 International Conference on Software Maintenance and Evolution (ICSME’14)_. 476–480.
  * Tai et al. (2015) Kai Sheng Tai, Richard Socher, and CD. Manning. 2015.  Improved semantic representations from tree-structured long short-term memory networks.  _arXiv preprint arXiv:1503.00075_ (2015).
  * Tao et al. (2022) Chenning Tao, Qi Zhan, Xing Hu, and Xin Xia. 2022.  C4: Contrastive cross-language code clone detection. In _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension_. 413–424.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wahler et al. (2004) Vera Wahler, Dietmar Seipel, J Wolff, and Gregor Fischer. 2004.  Clone detection in source code by frequent itemset techniques. In _Source Code Analysis and Manipulation, Fourth IEEE International Workshop on_. IEEE, 128–135.
  * Wang et al. (2017) Min Wang, Pengcheng Wang, and Yun Xu. 2017.  CCSharp: An efficient three-phase code clone detector using modified pdgs. In _Proceedings of the 24th Asia-Pacific Software Engineering Conference (APSEC’17)_. 100–109.
  * Wang et al. (2018) Pengcheng Wang, Jeffrey Svajlenko, Yanzhao Wu, Yun Xu, and CK. Roy. 2018.  CCAligner: A token based large-gap clone detector. In _Proceedings of the 40th International Conference on Software Engineering (ICSE’18)_. 1066–1077.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  _arXiv preprint arXiv:2109.00859_ (2021).
  * Wei and Li (2017) Huihui Wei and Ming Li. 2017.  Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In _Proceedings of the 2017 International Joint Conferences on Artificial Intelligence (IJCAI’17)_. 3034–3040.
  * Wettel and Marinescu (2005) Richard Wettel and Radu Marinescu. 2005.  Archeology of code duplication: Recovering duplication chains from small duplication fragments. In _Seventh International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC’05)_. IEEE, 8–10.
  * White et al. (2016) Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016.  Deep learning code fragments for code clone detection. In _Proceedings of the 31st International Conference on Automated Software Engineering (ASE’16)_. 87–98.
  * Wu et al. (2020a) Ming Wu, Pengcheng Wang, Kangqi Yin, Haoyu Cheng, Yun Xu, and Chanchal K Roy. 2020a.  Lvmapper: A large-variance clone detector using sequencing alignment approach.  _IEEE access_ 8 (2020), 27986–27997.
  * Wu et al. (2022) Yueming Wu, Siyue Feng, Deqing Zou, and Hai Jin. 2022.  Detecting semantic code clones by building AST-based Markov chains model. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–13.
  * Wu et al. (2020b) Yueming Wu, Deqing Zou, Shihan Dou, Siru Yang, Wei Yang, Feng Cheng, Hong Liang, and Hai Jin. 2020b.  SCDetector: software functional clone detection based on semantic tokens analysis. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20)_. 821–833.
  * Yang (1991) Wuu Yang. 1991.  Identifying syntactic differences between two programs.  _Software: Practice and Experience_ 21, 7 (1991), 739–755.
  * Zhang et al. (2019) Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019.  A novel neural source code representation based on abstract syntax tree. In _Proceedings of the 41st International Conference on Software Engineering (ICSE’19)_. 783–794.
  * Zhao and Huang (2018) Gang Zhao and Jeff Huang. 2018.  Deepsim: Deep learning code functional similarity. In _Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE’18)_. 141–151.


* McBride et al. (2007) McBride, V. A., Wilms, J., Kreykenbohm, I., et al. 2007, A&A, 470, 1065, doi: 10.1051/0004-6361:20077238
  * McCollum & Laine (2019) McCollum, B., & Laine, S. 2019, The Astronomer’s Telegram, 13211, 1
  * McKinney (2010) McKinney. 2010, in Proceedings of the 9th Python in Science Conference, ed. Stéfan van der Walt & Jarrod Millman, 56 – 61, doi: 10.25080/Majora-92bf1922-00a
  * Mereghetti & La Palombara (2009) Mereghetti, S., & La Palombara, N. 2009, A&A, 504, 181, doi: 10.1051/0004-6361/200911944
  * Molkov et al. (2019) Molkov, S., Lutovinov, A., Tsygankov, S., Mereminskiy, I., & Mushtukov, A. 2019, ApJ, 883, L11, doi: 10.3847/2041-8213/ab3e4d
  * Morel & Grosdidier (2005) Morel, T., & Grosdidier, Y. 2005, MNRAS, 356, 665, doi: 10.1111/j.1365-2966.2004.08488.x
  * Motch et al. (1997) Motch, C., Haberl, F., Dennerl, K., Pakull, M., & Janot-Pacheco, E. 1997, A&A, 323, 853
  * Mukerjee & Antia (2021) Mukerjee, K., & Antia, H. M. 2021, ApJ, 920, 139, doi: 10.3847/1538-4357/ac11f1
  * Mukerjee et al. (2020) Mukerjee, K., Antia, H. M., & Katoch, T. 2020, ApJ, 897, 73, doi: 10.3847/1538-4357/ab97b6
  * Müller et al. (2012) Müller, S., Kühnel, M., Caballero, I., et al. 2012, A&A, 546, A125, doi: 10.1051/0004-6361/201219580
  * Mushtukov et al. (2015) Mushtukov, A. A., Suleimanov, V. F., Tsygankov, S. S., & Poutanen, J. 2015, MNRAS, 454, 2539, doi: 10.1093/mnras/stv2087
  * Nabizadeh et al. (2019a) Nabizadeh, A., Mönkkönen, J., Tsygankov, S. S., et al. 2019a, A&A, 629, A101, doi: 10.1051/0004-6361/201936045
  * Nabizadeh et al. (2019b) Nabizadeh, A., Tsygankov, S. S., Karasev, D. I., et al. 2019b, A&A, 622, A198, doi: 10.1051/0004-6361/201834635
  * Nabizadeh et al. (2022) Nabizadeh, A., Tsygankov, S. S., Molkov, S. V., et al. 2022, A&A, 657, A58, doi: 10.1051/0004-6361/202141608
  * Nagase (1989) Nagase, F. 1989, PASJ, 41, 1, doi: 1989PASJ…41….1N
  * Nagase et al. (1991) Nagase, F., Dotani, T., Tanaka, Y., et al. 1991, ApJ, 375, L49, doi: 10.1086/186085
  * Naik et al. (2013) Naik, S., Maitra, C., Jaisawal, G. K., & Paul, B. 2013, ApJ, 764, 158, doi: 10.1088/0004-637X/764/2/158
  * Negueruela (1998) Negueruela, I. 1998, A&A, 338, 505
  * Negueruela et al. (2008) Negueruela, I., Casares, J., Verrecchia, F., et al. 2008, The Astronomer’s Telegram, 1876, 1
  * Negueruela et al. (2003) Negueruela, I., Israel, G. L., Marco, A., Norton, A. J., & Speziali, R. 2003, A&A, 397, 739, doi: 10.1051/0004-6361:20021529
  * Negueruela & Okazaki (2001) Negueruela, I., & Okazaki, A. T. 2001, A&A, 369, 108, doi: 10.1051/0004-6361:20010146
  * Negueruela et al. (1999) Negueruela, I., Roche, P., Fabregat, J., & Coe, M. J. 1999, MNRAS, 307, 695, doi: 10.1046/j.1365-8711.1999.02682.x
  * Negueruela & Schurch (2007) Negueruela, I., & Schurch, M. P. E. 2007, A&A, 461, 631, doi: 10.1051/0004-6361:20066054
  * Nespoli et al. (2008) Nespoli, E., Fabregat, J., & Mennickent, R. E. 2008, A&A, 486, 911, doi: 10.1051/0004-6361:200809645
  * Nespoli et al. (2010) —. 2010, A&A, 516, A94, doi: 10.1051/0004-6361/200913410
  * Nespoli & Reig (2011) Nespoli, E., & Reig, P. 2011, A&A, 526, A7, doi: 10.1051/0004-6361/201015303
  * Nespoli et al. (2012) Nespoli, E., Reig, P., & Zezas, A. 2012, A&A, 547, A103, doi: 10.1051/0004-6361/201219586
  * Neumann et al. (2023) Neumann, M., Avakyan, A., Doroshenko, V., & Santangelo, A. 2023, arXiv e-prints, arXiv:2303.16137, doi: 10.48550/arXiv.2303.16137
  * Nikolov et al. (2017) Nikolov, Y. M., Zamanov, R. K., Stoyanov, K. A., & Martí, J. 2017, Bulgarian Astronomical Journal, 27, 10
  * Nowak et al. (2012) Nowak, M. A., Paizis, A., Rodriguez, J., et al. 2012, ApJ, 757, 143, doi: 10.1088/0004-637X/757/2/143
  * O’Connor et al. (2022) O’Connor, B., Göğüş, E., Huppenkothen, D., et al. 2022, ApJ, 927, 139, doi: 10.3847/1538-4357/ac5032
  * Oh & Kroupa (2016) Oh, S., & Kroupa, P. 2016, A&A, 590, A107, doi: 10.1051/0004-6361/201628233
  * Oja (1991) Oja, T. 1991, A&AS, 89, 415
  * Orlandini et al. (2012) Orlandini, M., Frontera, F., Masetti, N., Sguera, V., & Sidoli, L. 2012, ApJ, 748, 86, doi: 10.1088/0004-637X/748/2/86
  * Pacini (1970) Pacini, F. 1970, Nature, 226, 622, doi: 10.1038/226622a0
  * Pahari & Pal (2012) Pahari, M., & Pal, S. 2012, MNRAS, 423, 3352, doi: 10.1111/j.1365-2966.2012.21128.x
  * Parkes et al. (1978) Parkes, G. E., Murdin, P. G., & Mason, K. O. 1978, MNRAS, 184, 73P, doi: 10.1093/mnras/184.1.73P
  * Pearlman et al. (2019) Pearlman, A. B., Coley, J. B., Corbet, R. H. D., & Pottschmidt, K. 2019, ApJ, 873, 86, doi: 10.3847/1538-4357/aaf001
  * Piraino et al. (2000) Piraino, S., Santangelo, A., Segreto, A., et al. 2000, A&A, 357, 501
  * Postnov et al. (2015) Postnov, K. A., Mironov, A. I., Lutovinov, A. A., et al. 2015, MNRAS, 446, 1013, doi: 10.1093/mnras/stu2155
  * Pradhan et al. (2013) Pradhan, P., Maitra, C., Paul, B., & Paul, B. C. 2013, MNRAS, 436, 945, doi: 10.1093/mnras/stt1504
  * Pringle & Rees (1972) Pringle, J. E., & Rees, M. J. 1972, A&A, 21, 1
  * Puls et al. (2008) Puls, J., Vink, J. S., & Najarro, F. 2008, A&A Rev., 16, 209, doi: 10.1007/s00159-008-0015-8
  * Quaintrell et al. (2003) Quaintrell, H., Norton, A. J., Ash, T. D. C., et al. 2003, A&A, 401, 313, doi: 10.1051/0004-6361:20030120
  * Raddi et al. (2021) Raddi, R., Irrgang, A., Heber, U., Schneider, D., & Kreuzer, S. 2021, A&A, 645, A108, doi: 10.1051/0004-6361/202037872
  * Rahoui & Chaty (2008) Rahoui, F., & Chaty, S. 2008, A&A, 492, 163, doi: 10.1051/0004-6361:200810695
  * Rahoui et al. (2008) Rahoui, F., Chaty, S., Lagage, P. O., & Pantin, E. 2008, A&A, 484, 801, doi: 10.1051/0004-6361:20078774
  * Raichur & Paul (2010) Raichur, H., & Paul, B. 2010, MNRAS, 406, 2663, doi: 10.1111/j.1365-2966.2010.16862.x
  * Raman et al. (2021) Raman, G., Varun, Paul, B., & Bhattacharya, D. 2021, MNRAS, 508, 5578, doi: 10.1093/mnras/stab2835
  * Ray & Chakrabarty (2002) Ray, P. S., & Chakrabarty, D. 2002, ApJ, 581, 1293, doi: 10.1086/344300
  * Reed (2003) Reed, B. C. 2003, AJ, 125, 2531, doi: 10.1086/374771
  * Reig et al. (2008) Reig, P., Belloni, T., Israel, G. L., et al. 2008, A&A, 485, 797, doi: 10.1051/0004-6361:200809457
  * Reig et al. (2014a) Reig, P., Blinov, D., Papadakis, I., Kylafis, N., & Tassis, K. 2014a, MNRAS, 445, 4235, doi: 10.1093/mnras/stu2322
  * Reig & Coe (1999) Reig, P., & Coe, M. J. 1999, MNRAS, 302, 700, doi: 10.1046/j.1365-8711.1999.02179.x
  * Reig et al. (2014b) Reig, P., Doroshenko, V., & Zezas, A. 2014b, MNRAS, 445, 1314, doi: 10.1093/mnras/stu1840
  * Reig & Fabregat (2015) Reig, P., & Fabregat, J. 2015, A&A, 574, A33, doi: 10.1051/0004-6361/201425008
  * Reig & Fabregat (2022) —. 2022, A&A, 667, A18, doi: 10.1051/0004-6361/202243664
  * Reig et al. (2020) Reig, P., Fabregat, J., & Alfonso-Garzón, J. 2020, A&A, 640, A35, doi: 10.1051/0004-6361/202038333
  * Reig et al. (1997) Reig, P., Fabregat, J., Coe, M. J., et al. 1997, A&A, 322, 183
  * Reig et al. (2004) Reig, P., Negueruela, I., Fabregat, J., et al. 2004, A&A, 421, 673, doi: 10.1051/0004-6361:20035786
  * Reig et al. (2005a) Reig, P., Negueruela, I., Fabregat, J., Chato, R., & Coe, M. J. 2005a, A&A, 440, 1079, doi: 10.1051/0004-6361:20053124
  * Reig et al. (2005b) Reig, P., Negueruela, I., Papamastorakis, G., Manousakis, A., & Kougentakis, T. 2005b, A&A, 440, 637, doi: 10.1051/0004-6361:20052684
  * Reig et al. (2011) Reig, P., Nespoli, E., Fabregat, J., & Mennickent, R. E. 2011, A&A, 533, A23, doi: 10.1051/0004-6361/201117301
  * Reig & Roche (1999) Reig, P., & Roche, P. 1999, MNRAS, 306, 100, doi: 10.1046/j.1365-8711.1999.02473.x
  * Reig et al. (1996) Reig, P., Roche, P., Fabregat, J., & Coe, M. J. 1996, in Roentgenstrahlung from the Universe, ed. H. U. Zimmermann, J. Trümper, & H. Yorke, 181–182
  * Reig & Zezas (2014) Reig, P., & Zezas, A. 2014, MNRAS, 442, 472, doi: 10.1093/mnras/stu898
  * Reig & Zezas (2018) —. 2018, A&A, 613, A52, doi: 10.1051/0004-6361/201732533
  * Riquelme et al. (2012) Riquelme, M. S., Torrejón, J. M., & Negueruela, I. 2012, A&A, 539, A114, doi: 10.1051/0004-6361/201117738
  * Rodes-Roca et al. (2018) Rodes-Roca, J. J., Bernabeu, G., Magazzù, A., Torrejón, J. M., & Solano, E. 2018, MNRAS, 476, 2110, doi: 10.1093/mnras/sty333
  * Rodes-Roca et al. (2013) Rodes-Roca, J. J., Torrejón, J. M., Martínez-Núñez, S., Bernabéu, G., & Magazzú, A. 2013, A&A, 555, A115, doi: 10.1051/0004-6361/201321923
  * Romano et al. (2009) Romano, P., Sidoli, L., Cusumano, G., et al. 2009, ApJ, 696, 2068, doi: 10.1088/0004-637X/696/2/2068
  * Romano et al. (2015) Romano, P., Bozzo, E., Mangano, V., et al. 2015, A&A, 576, L4, doi: 10.1051/0004-6361/201525749
  * Rouco Escorial et al. (2018) Rouco Escorial, A., van den Eijnden, J., & Wijnands, R. 2018, A&A, 620, L13, doi: 10.1051/0004-6361/201834572
  * Roy et al. (2020) Roy, J., Agrawal, P. C., Singari, B., & Misra, R. 2020, Research in Astronomy and Astrophysics, 20, 155, doi: 10.1088/1674-4527/20/9/155
  * Sadakane et al. (1985) Sadakane, K., Hirata, R., Jugaku, J., et al. 1985, ApJ, 288, 284, doi: 10.1086/162791
  * Salganik et al. (2022) Salganik, A., Tsygankov, S. S., Djupvik, A. A., et al. 2022, MNRAS, 509, 5955, doi: 10.1093/mnras/stab3362
  * Santangelo et al. (1998) Santangelo, A., del Sordo, S., Segreto, A., et al. 1998, A&A, 340, L55–L59, doi: 10.1051/aas:1996164
  * Sarty et al. (2009) Sarty, G. E., Kiss, L. L., Huziak, R., et al. 2009, MNRAS, 392, 1242, doi: 10.1111/j.1365-2966.2008.14138.x
  * Sarty et al. (2011) Sarty, G. E., Pilecki, B., Reichart, D. E., et al. 2011, Research in Astronomy and Astrophysics, 11, 947, doi: 10.1088/1674-4527/11/8/007
  * Sato et al. (1986) Sato, N., Nagase, F., Kawai, N., et al. 1986, ApJ, 304, 241, doi: 10.1086/164157
  * Scott et al. (1997) Scott, D. M., Finger, M. H., Wilson, R. B., et al. 1997, ApJ, 488, 831, doi: 10.1086/304740
  * Segreto et al. (2013) Segreto, A., La Parola, V., Cusumano, G., et al. 2013, A&A, 558, A99, doi: 10.1051/0004-6361/201321892
  * Sguera et al. (2013) Sguera, V., Drave, S. P., Sidoli, L., et al. 2013, A&A, 556, A27, doi: 10.1051/0004-6361/201220785
  * Sguera et al. (2007) Sguera, V., Hill, A. B., Bird, A. J., et al. 2007, A&A, 467, 249, doi: 10.1051/0004-6361:20066762
  * Shakura (1973) Shakura, N. I. 1973, Soviet Ast., 16, 756
  * Shakura (1975) —. 1975, Soviet Astronomy Letters, 1, 223
  * Shaw et al. (2009) Shaw, S. E., Hill, A. B., Kuulkers, E., et al. 2009, MNRAS, 393, 419, doi: 10.1111/j.1365-2966.2008.14212.x
  * Shirke et al. (2021) Shirke, P., Bala, S., Roy, J., & Bhattacharya, D. 2021, Journal of Astrophysics and Astronomy, 42, 58, doi: 10.1007/s12036-021-09710-w
  * Shtykovsky et al. (2019) Shtykovsky, A. E., Lutovinov, A. A., Tsygankov, S. S., & Molkov, S. V. 2019, MNRAS, 482, L14, doi: 10.1093/mnrasl/sly182
  * Sidoli et al. (2016) Sidoli, L., Esposito, P., Motta, S. E., Israel, G. L., & Rodríguez Castillo, G. A. 2016, MNRAS, 460, 3637, doi: 10.1093/mnras/stw1246
  * Sidoli et al. (2017) Sidoli, L., Israel, G. L., Esposito, P., Rodríguez Castillo, G. A., & Postnov, K. 2017, MNRAS, 469, 3056, doi: 10.1093/mnras/stx1105
  * Sidoli et al. (2012) Sidoli, L., Mereghetti, S., Sguera, V., & Pizzolato, F. 2012, MNRAS, 420, 554, doi: 10.1111/j.1365-2966.2011.20063.x
  * Sidoli & Paizis (2018) Sidoli, L., & Paizis, A. 2018, MNRAS, 481, 2779, doi: 10.1093/mnras/sty2428
  * Sidoli et al. (2020) Sidoli, L., Postnov, K., Tiengo, A., et al. 2020, A&A, 638, A71, doi: 10.1051/0004-6361/202038078
  * Siuniaev & Shakura (1977) Siuniaev, R. A., & Shakura, N. I. 1977, Soviet Astronomy Letters, 3, 114
  * Staubert et al. (2011) Staubert, R., Pottschmidt, K., Doroshenko, V., et al. 2011, A&A, 527, A7, doi: 10.1051/0004-6361/201015737
  * Staubert et al. (2019) Staubert, R., Trümper, J., Kendziorra, E., et al. 2019, A&A, 622, A61, doi: 10.1051/0004-6361/201834479
  * Stevens et al. (1997) Stevens, J. B., Reig, P., Coe, M. J., et al. 1997, MNRAS, 288, 988, doi: 10.1093/mnras/288.4.988
  * Strader et al. (2019) Strader, J., Chomiuk, L., Swihart, S., & Aydi, E. 2019, The Astronomer’s Telegram, 12554, 1
  * Sugizaki et al. (2001) Sugizaki, M., Mitsuda, K., Kaneda, H., et al. 2001, ApJS, 134, 77, doi: 10.1086/320358
  * Sugizaki et al. (2015) Sugizaki, M., Yamamoto, T., Mihara, T., Nakajima, M., & Makishima, K. 2015, PASJ, 67, 73, doi: 10.1093/pasj/psv039
  * Swank & Markwardt (2003) Swank, J. H., & Markwardt, C. B. 2003, The Astronomer’s Telegram, 128, 1
  * Taylor (2005) Taylor, M. B. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 29
  * Thompson & Rothschild (2008) Thompson, T., & Rothschild, R. 2008, ApJ, 691, 1744–1753, doi: 10.1088/0004-637X/691/2/1744
  * Thompson et al. (2007) Thompson, T. W. J., Tomsick, J. A., in ’t Zand, J. J. M., Rothschild, R. E., & Walter, R. 2007, ApJ, 661, 447, doi: 10.1086/513458
  * Thompson et al. (2006) Thompson, T. W. J., Tomsick, J. A., Rothschild, R. E., in’t Zand, J. J. M., & Walter, R. 2006, ApJ, 649, 373, doi: 10.1086/506251
  * Tobrej et al. (2023) Tobrej, M., Rai, B., Ghising, M., Tamang, R., & Paul, B. C. 2023, MNRAS, 518, 4861, doi: 10.1093/mnras/stac3203
  * Torii et al. (1998) Torii, K., Kinugasa, K., Katayama, K., et al. 1998, ApJ, 508, 854, doi: 10.1086/306455
  * Torrejon et al. (2010) Torrejon, J., Negueruela, I., Smith, D., & Harrison, T. 2010, A&A, 510, 1, doi: 10.1051/0004-6361/200912619
  * Torrejón et al. (2004) Torrejón, J. M., Kreykenbohm, I., Orr, A., Titarchuk, L., & Negueruela, I. 2004, A&A, 423, 301, doi: 10.1051/0004-6361:20035743
  * Torrejón & Negueruela (2007) Torrejón, J. M., & Negueruela, I. 2007, in ESA Special Publication, Vol. 622, The Obscured Universe. Proceedings of the VI INTEGRAL Workshop, 503
  * Torrejón et al. (2010) Torrejón, J. M., Negueruela, I., Smith, D. M., & Harrison, T. E. 2010, A&A, 510, A61, doi: 10.1051/0004-6361/200912619
  * Tsygankov et al. (2019a) Tsygankov, S. S., Doroshenko, V., Mushtukov, A. A., Lutovinov, A. A., & Poutanen, J. 2019a, A&A, 621, A134, doi: 10.1051/0004-6361/201833786
  * Tsygankov et al. (2012) Tsygankov, S. S., Krivonos, R. A., & Lutovinov, A. A. 2012, MNRAS, 421, 2407, doi: 10.1111/j.1365-2966.2012.20475.x
  * Tsygankov et al. (2016) Tsygankov, S. S., Lutovinov, A. A., Krivonos, R. A., et al. 2016, MNRAS, 457, 258, doi: 10.1093/mnras/stv2849
  * Tsygankov et al. (2019b) Tsygankov, S. S., Rouco Escorial, A., Suleimanov, V. F., et al. 2019b, MNRAS, 483, L144, doi: 10.1093/mnrasl/sly236
  * Tsygankov et al. (2021) Tsygankov, S. S., Lutovinov, A. A., Molkov, S. V., et al. 2021, ApJ, 909, 154, doi: 10.3847/1538-4357/abddbd
  * Tutukov & Cherepashchuk (2020) Tutukov, A. V., & Cherepashchuk, A. M. 2020, Physics Uspekhi, 63, 209, doi: 10.3367/UFNe.2019.03.038547
  * Uchida et al. (2021) Uchida, N., Takahashi, H., Fukazawa, Y., & Makishima, K. 2021, PASJ, 73, 1389, doi: 10.1093/pasj/psab083
  * UKIDSS Consortium (2012) UKIDSS Consortium. 2012, VizieR Online Data Catalog, II/316
  * van den Heuvel (2017) van den Heuvel, E. P. J. 2017, in Handbook of Supernovae, ed. A. W. Alsabti & P. Murdin (Springer, Cham), 1527, doi: 10.1007/978-3-319-21846-5_75
  * van Paradijs (1995) van Paradijs, J. 1995, in X-ray Binaries, 536–577
  * Wang (2010) Wang, W. 2010, A&A, 516, A15, doi: 10.1051/0004-6361/200913196
  * Wang (2011) —. 2011, MNRAS, 413, 1083, doi: 10.1111/j.1365-2966.2010.18192.x
  * Wang (2014) —. 2014, Research in Astronomy and Astrophysics, 14, 565, doi: 10.1088/1674-4527/14/5/006
  * Wen et al. (2000) Wen, L., Remillard, R. A., & Bradt, H. V. 2000, ApJ, 532, 1119, doi: 10.1086/308604
  * Wenger et al. (2000) Wenger, M., Ochsenbein, F., Egret, D., et al. 2000, A&AS, 143, 9, doi: 10.1051/aas:2000332
  * Wilson et al. (2008) Wilson, C. A., Finger, M. H., & Camero-Arranz, A. 2008, ApJ, 678, 1263, doi: 10.1086/587134
  * Wilson et al. (2003) Wilson, C. A., Finger, M. H., Coe, M. J., & Negueruela, I. 2003, ApJ, 584, 996, doi: 10.1086/345791
  * Wilson et al. (2002) Wilson, C. A., Finger, M. H., Göǧüş, E., Woods, P. M., & Kouveliotou, C. 2002, ApJ, 565, 1150, doi: 10.1086/324707
  * Wilson et al. (1998) Wilson, C. A., Finger, M. H., Harmon, B. A., Chakrabarty, D., & Strohmayer, T. 1998, ApJ, 499, 820, doi: 10.1086/305677
  * Wilson et al. (1997) Wilson, C. A., Finger, M. H., Harmon, B. A., et al. 1997, ApJ, 479, 388, doi: 10.1086/303841
  * Wilson et al. (1999) Wilson, C. A., Finger, M. H., & Scott, D. M. 1999, ApJ, 511, 367, doi: 10.1086/306660
  * Wilson-Hodge (1999) Wilson-Hodge, C. A. 1999, PhD thesis, University of Alabama, Huntsville
  * Wilson-Hodge et al. (2018) Wilson-Hodge, C. A., Malacaria, C., Jenke, P. A., et al. 2018, ApJ, 863, 9, doi: 10.3847/1538-4357/aace60
  * Yamamoto et al. (2011) Yamamoto, T., Sugizaki, M., Mihara, T., et al. 2011, PASJ, 63, S751, doi: 10.1093/pasj/63.sp3.S751
  * Yan et al. (2012) Yan, J., Zurita Heras, J. A., Chaty, S., Li, H., & Liu, Q. 2012, ApJ, 753, 73, doi: 10.1088/0004-637X/753/1/73
  * Yungelson et al. (2019) Yungelson, L. R., Kuranov, A. G., & Postnov, K. A. 2019, MNRAS, 485, 851, doi: 10.1093/mnras/stz467
  * Zacharias et al. (2012) Zacharias, N., Finch, C. T., Girard, T. M., et al. 2012, VizieR Online Data Catalog, I/322A
  * Zhang et al. (2005) Zhang, S., Qu, J.-L., Song, L.-M., & Torres, D. F. 2005, ApJ, 630, L65, doi: 10.1086/462415
  * Zhao et al. (2019) Zhao, Y., Heinke, C. O., Tsygankov, S. S., et al. 2019, MNRAS, 488, 4427, doi: 10.1093/mnras/stz1946
  * Zurita Heras & Chaty (2008) Zurita Heras, J. A., & Chaty, S. 2008, A&A, 489, 657, doi: 10.1051/0004-6361:20079097
  * Zurita Heras et al. (2006) Zurita Heras, J. A., De Cesare, G., Walter, R., et al. 2006, A&A, 448, 261, doi: 10.1051/0004-6361:20053876