identify web-service groups for refactoring using K-means, cobweb, and
expectation maximization. Kosker et al. (2009) trained a Naive Bayes-based
classifier to identify classes that need refactoring. Kumar and Sureka (2017)
used Least Square-Support Vector Machine (ls-svm) along with smote as
classifier. They found that ls-svm with Radial Basis Function (rbf) kernel
gives the best results. Nyamawe et al. (2019) recommended refactorings based
on the history of requested features and applied refactorings. Their approach
involves two classification tasks; first, a binary classification that
suggests whether refactoring is needed or not and second, a multi-label
classification that suggests the type of refactoring. The authors used Linear
Regression, Multinomial Naive Bayes (mnb), Support Vector Machine, and Random
Forest classifiers. Yue et al. (2018) presented crec—a learning-based approach
that automatically extracts refactored and non-refactored clones groups from
software repositories, and trains an AdaBoost model to recommend clones for
refactoring. Kumar et al. (2019) employed a set of ml models such as Linear
Regression, Naive Bayes, Bayes Network, Random Forest, AdaBoost, and Logit
Boost to develop a recommendation system to suggest the need of refactoring
for a method. Amal et al. (2014) proposed the use of ann to generate a
sequence of refactoring. Aribandi et al. (2019) predicted the classes that are
likely to be refactored in the future iterations. To achieve their aim, the
authors used various variants of ann, Support Vector Machine, as well as Best-
in-training based Ensemble (bte) and Majority Voting Ensemble (mve) as
ensemble techniques. Kurbatova et al. (2020) proposed an approach to recommend
move method refactoring based on a path-based presentation of code using
Support Vector Machine. Similarly, Aniche et al. (2020) used Linear
Regression, Naive Bayes, Support Vector Machine, Decision Tree, Random Forest,
and Neural Network to predict applicable refactoring operations. Sidhu et al.
(2022); Xu et al. (2017); Wang and Godfrey (2014) used dnn, gradient boosting,
and Decision Tree respectively to identify refactoring candidate. Sagar et al.
(2021); Nyamawe et al. (2020) employed various classifiers such as Support
Vector Machine, Linear Regression, and Random Forest to predict commits with
refactoring operations.

###  4.9. Vulnerability analysis

The studies in this domain analyze source code to identify potential security
vulnerabilities. In this section, we point out the state-of-the-art in
software vulnerability detection using ml techniques. Figure 12 presents an
overview of a typical process to detect vulnerabilities with the help of ml
techniques. First, the studies prepare a dataset or identify an existing
dataset for ml training. Next, the studies extract relevant features from the
identified subject systems. Then, the features are fed into a ml model for
training. The trained model is then used to predict vulnerabilities in the
source code.

Figure 12. Overview of vulnerability detection process pipeline

Dataset preparation: Authors used existing labeled datasets as well as created
their own datasets to train ml models. Specifically, a set of studies (Pereira
et al., 2019; Milosevic et al., 2017; Rahman et al., 2017; Saccente et al.,
2019; Kim et al., 2019; Bilgin et al., 2020; Spreitzenbarth et al., 2014; Lin
et al., 2020; Yosifova et al., 2021; Sultana et al., 2021; Law and Grépin,
2010; Pang et al., 2017; Abunadi and Alenezi, 2015; Younis and Malaiya, 2014;
Vishnu and Jevitha, 2014; Khalid et al., 2019; Kim et al., 2018; Zhang and Li,
2020; Mateless et al., 2020; Du et al., 2019; Shiqi et al., 2018; Du et al.,
2015; Yang et al., 2019b; Batur Şahin and Abualigah, 2021; Narayanan et al.,
2018; Wang et al., 2020d; Chen et al., 2020b; Li et al., 2015; Ren et al.,
2019; Ban et al., 2019) used available labeled datasets for php, Java, C, C++,
and Android applications to train vulnerability detection models. In other
cases, Russell et al. (2018) extended an existing dataset with millions of C
and C++ functions and then labeled it based on the output of three static
analyzers (i.e., Clang, CppCheck, and Flawfinder).

Many studies (Ma et al., 2019; Ali Alatwi et al., 2016; Cui et al., 2020;
Ndichu et al., 2019; Elovici et al., 2007; Medeiros et al., 2014; Ferenc et
al., 2019; Piskachev et al., 2019; Kronjee et al., 2018; Pang et al., 2016;
Alves et al., 2016; Gupta et al., 2021d; Clemente et al., 2018; Chernis and
Verma, 2018; Moskovitch et al., 2009; Hou et al., 2010; Santos et al., 2013;
Yang et al., 2018; Zheng et al., 2020; Perl et al., 2015; Shar et al., 2015;
Jimenez et al., 2019; Lin et al., 2020; Lin et al., 2018a) created their own
datasets. Ma et al. (2019); Ali Alatwi et al. (2016); Cui et al. (2020), and
Gupta et al. (2021d) created datasets to train vulnerability detectors for
Android applications. In particular, Ma et al. (2019) decompiled and generated
cfgs of approximately 10 thousand, both benign and vulnerable, Android
applications from AndroZoo and Android Malware datasets; Ali Alatwi et al.
(2016) collected 5,063 Android applications where 1,000 of them were marked as
benign and the remaining as malware; Cui et al. (2020) selected an open-source
dataset comprised of 1,179 Android applications that have 4,416 different
version (of the 1,179 applications) and labeled the selected dataset by using
the Androrisk tool; and Gupta et al. (2021d) used two Android applications
(Android-universal-image-loader and JHotDraw) which they have manually labeled
based on the projects pmd reports (true if a vulnerability was reported in a
pmd file and false otherwise). To create datasets of php projects, Medeiros et
al. (2014) collected 35 open-source php projects and intentionally injected 76
vulnerabilities in their dataset. Shar et al. (2015) used phpminer to extract
15 datasets that include sql injections, cross-site scripting, remote code
execution, and file inclusion vulnerabilities, and labeled only 20% of their
dataset to point out the precision of their approach. Ndichu et al. (2019)
collected 5,024 JavaScript code snippets from d3m, jsunpack, and 100 top
websites where the half of the code snippets were benign and the other half
malicious. In other cases, authors (Yang et al., 2018; Rahman et al., 2017;
Perl et al., 2015) collected large number of commit messages and mapped them
to known vulnerabilities by using Google’s Play Store, National Vulnerability
Database (nvd), Synx, Node Security Project, and so on, while in limited cases
authors (Piskachev et al., 2019) manually label their dataset. Hou et al.
(2010); Moskovitch et al. (2009) and Santos et al. (2013) created their
datasets by collecting web-page samples from StopBadWare and VxHeavens. Lin et
al. (2020) constructed a dataset and manually labeled 1,471 vulnerable
functions and 1,320 vulnerable files from nine open-source applications, named
Asterisk, FFmpag, httpd, LibPNG, LibTIFF, OpenSSL, Pidgin, vlc Player, and
Xen. Lin et al. (2018a) have used more then 30,000 non-vulnerable functions
and manually labeled 475 vulnerable functions for their experiments.

Feature extraction: Authors used static source code metrics, cfgs, asts,
source code tokens, and word embeddings as features.

Source code metrics: A set of studies (Medeiros et al., 2014; Ferenc et al.,
2019; Alves et al., 2016; Gupta et al., 2021d; Clemente et al., 2018; Rahman
et al., 2017; Cui et al., 2020; Piskachev et al., 2019; Ren et al., 2019; Du
et al., 2019; Kim et al., 2018; Medeiros et al., 2016; Abunadi and Alenezi,
2015; Law and Grépin, 2010; Sultana et al., 2021) used more than 20 static
source code metrics (such as cyclomatic complexity, maximum depth of class in
inheritance tree, number of statements, and number of blank lines).

Data/control flow and ast: Ma et al. (2012); Kim et al. (2019); Bilgin et al.
(2020); Kronjee et al. (2018); Wang et al. (2020d); Du et al. (2015); Medeiros
et al. (2016) used cfgs, asts, or data flow analysis as features. More
specifically, Ma et al. (2019) extracted the api calls from the cfgs of their
dataset and collected information such as the usage of apis (which apis the
application uses), the api frequencies (how many times the application uses
apis) and api sequence (the order the application uses apis). Kim et al.
(2019) extracted asts and gfcs which they tokenized and fed into ml models,
while Bilgin et al. (2020) extracted asts and translated their representation
of source code into a one-dimensional numerical array to fed them to a model.
Kronjee et al. (2018) used data-flow analysis to extract features, while
Spreitzenbarth et al. (2014) used static, dynamic analysis, and information
collected from ltrace to collect features and train a linear vulnerability
detection model. Lin et al. (2018a) created asts and from there they extracted
code semantics as features.

Repository and file metrics: Perl et al. (2015) collected GitHub repository
meta-data (i.e., programming language, star count, fork count, and number of
commits) in addition to source code metrics. Other authors (Pereira et al.,
2019; Elovici et al., 2007) used file meta-data such as files’ creation and
modification time, machine type, file size, and linker version.

Code and Text tokens: Chernis and Verma (2018) used simple token features
(character count, character diversity, entropy, maximum nesting depth, arrow
count, “if” count, “if” complexity, “while” count, and “for” count) and
complex features (character n-grams, word n-grams, and suffix trees). Hou et
al. (2010) collected 10 features such as length of the document, average
length of word, word count, word count in a line, and  number of NULL
characters. The remaining studies (Russell et al., 2018; Pang et al., 2016;
Moskovitch et al., 2009; Santos et al., 2013; Yang et al., 2018; Saccente et
al., 2019; Zheng et al., 2020; Shar et al., 2015; Chen et al., 2020b;
Narayanan et al., 2018; Russell et al., 2018; Mateless et al., 2020; Fang et
al., 2020a; Zhang and Li, 2020; Pang et al., 2017; Ban et al., 2019; Yosifova
et al., 2021; Lin et al., 2020) tokenized parts of the source code or text-
based information with various techniques such as the most frequent
occurrences of operational codes, capture the meaning of critical tokens, or
applied techniques to reduce the vocabulary size in order to retrieve the most
important tokens. In some other cases, authors (Li et al., 2015) used
statistical techniques to reduce the feature space to reduce the number of
code tokens.

Other features: Ali Alatwi et al. (2016); Ndichu et al. (2019) and Milosevic
et al. (2017) extracted permission-related features. In other cases, authors
(Yang et al., 2019b) combined software metrics and N-grams as features to
train models and others (Shiqi et al., 2018) created text-based images to
extract features. Likewise, Sultana (2017) extracted traceable patterns such
as CompoundBox, Immutable, Implementor, Overrider, Sink, Stateless,
FunctionObject, and LimitSel and used Understand tool to extract various
software metrics. Wei et al. (2017) extracted system calls and function call-
related information to use as features, while Vishnu and Jevitha (2014)
extracted url-based features like number of chars, duplicated characters,
special characters, script tags, cookies, and re-directions. Padmanabhuni and
Tan (2015) extracted buffer usage patterns and defensive mechanisms statements
constructs by analyzing files.

Model training: To train models, the selected studies used a variety of
traditional ml and dl algorithms.

Traditional ML techniques: One set of studies (Ali Alatwi et al., 2016; Ndichu
et al., 2019; Pereira et al., 2019; Russell et al., 2018; Pang et al., 2016;
Moskovitch et al., 2009; Perl et al., 2015; Shar et al., 2015; Yosifova et
al., 2021; Sultana et al., 2021; Padmanabhuni and Tan, 2015; Law and Grépin,
2010; Abunadi and Alenezi, 2015; Younis and Malaiya, 2014; Sultana, 2017;
Vishnu and Jevitha, 2014; Wei et al., 2017; Du et al., 2019; Fang et al.,
2020a; Medeiros et al., 2016; Du et al., 2015; Narayanan et al., 2018; Wang et
al., 2020d; Chen et al., 2020b; Ren et al., 2019) used traditional ml
algorithms such as Naive Bayes, Decision Tree, Support Vector Machine, Linear
Regression, Decision Tree, and Random Forest to train their models.
Specifically, Ali Alatwi et al. (2016); Russell et al. (2018); Perl et al.
(2015) selected Support Vector Machine because it is not affected by over-
fitting when having very high dimensional variable spaces. Along the similar
lines, Ndichu et al. (2019) used Support Vector Machine to train their model
with linear kernel. Pereira et al. (2019) used Decision Tree, Linear
Regression, and Lasso to train their models, while (Abunadi and Alenezi, 2015)
found that Random Forest is the best model for predicting cross-project
vulnerabilities. Compared to the above studies, Shar et al. (2015) used both
supervised (i.e., Linear Regression and Random Forest) and semi-supervised
(i.e., Co-trained Random Forest) algorithms to train their models since most
of that datasets were not labeled. Yosifova et al. (2021) used text-based
features to train Naive Bayes, Support Vector Machine, and Random Forest
models. Du et al. (2019) created the leopard framework that does not require
prior knowledge about known vulnerabilities and used Random Forest, Naive
Bayes, Support Vector Machine, and Decision Tree to point them out.

Other studies (Medeiros et al., 2014; Ferenc et al., 2019; Piskachev et al.,
2019; Kronjee et al., 2018; Alves et al., 2016; Gupta et al., 2021d; Clemente
et al., 2018; Milosevic et al., 2017; Chernis and Verma, 2018; Hou et al.,
2010; Santos et al., 2013; Rahman et al., 2017; Cui et al., 2020) used up to
32 different ml algorithms to train models and compared their performance.
Specifically, Medeiros et al. (2014) experimented with multiple variants of
Decision Tree, Random Forest, Naive Bayes, K Nearest Neighbors, Linear
Regression, Multilayer Perceptron, and Support Vector Machine models and
identified Support Vector Machine as the best performing classifier for their
experiment. Likewise, Milosevic et al. (2017) and Rahman et al. (2017)
employed multiple ml algorithms, respectively, and found that Support Vector
Machine offers the highest accuracy rate for training vulnerability detectors.
In contrast to the above studies, Ferenc et al. (2019) showed that K Nearest
Neighbors offers the best performance for their dataset after experimenting
with dnn, K Nearest Neighbors, Support Vector Machine, Linear Regression,
Decision Tree, Random Forest, and Naive Bayes. In order to find out which is
the best model for the swan tool, Piskachev et al. (2019) evaluated the
Support Vector Machine, Naive Bayes, Bayes Network, Decision Tree, Stump, and
Ripper. Their results pointed out the Support Vector Machine as the best
performing model to detect vulnerabilities. Similarly, Kronjee et al. (2018),
Cui et al. (2020), and Gupta et al. (2021d) compared different ml algorithms
and found Decision Tree and Random Forest as the best performing algorithms.

DL techniques: A large number of studies (Yang et al., 2018; Saccente et al.,
2019; Kim et al., 2019; Lin et al., 2020; Ban et al., 2019; Kim et al., 2018;
Mateless et al., 2020; Lin et al., 2018a; Shiqi et al., 2018; Batur Şahin and
Abualigah, 2021) used dl methods such as cnn, rnn, and ann to train models. In
more details, Yang et al. (2018) utilized the bp-ann algorithm to train
vulnerability detectors. For the project Achilles, Saccente et al. (2019) used
an array of lstm models to train on data containing Java code snippets for a
specific set of vulnerability types. In another study, Kim et al. (2019)
suggested a dl framework that makes use of rnn models to train vulnerability
detectors. Specifically, the authors framework first feeds the code embeddings
into a bi-lstm model to capture the feature semantics, then an attention layer
is used to get the vector weights, and, finally, passed into a dense layer to
output if a code is safe or vulnerable. Compared to the studies that examined
traditional ml or dl algorithms, Zheng et al. (2020) examined both of them.
They used Random Forest, K Nearest Neighbors, Support Vector Machine, Linear
Regression among the traditional ml algorithms along with bi-lstm, gru, and
cnn. There results indicate bi-lstm as the best performing model. (Lin et al.,
2020) developed a benchmarking framework that can use bi-lstm, lstm, bi-gru,
gru, dnn and Text-cnn, but can be extended to use more deep learning models.
Kim et al. (2018) generating graphical semantics that reflect on code semantic
features and use them for Graph Convulutional Network to automatically
identify and learn semantic and extract features for vulnerability detection,
while Shiqi et al. (2018) created textual images and fed them to Deep Belief
Networks to classify malware.

##  5\. Datasets and tools

This section provides a consolidated summary of available datasets and tools
that are used by the studies considered in the survey. We carefully examined
each primary study and noted the used resources (i.e., datasets and tools). We
define the following criteria to include a resource in our catalog.

  * •

The referenced resource must have been used by at least one primary study.

  * •

The referenced resource must be publicly available at the time of writing this
paper (Jul 2022).

  * •

The resource provides bare-minimum usage instructions to build and execute
(wherever applicable) and to use the artifact.

  * •

The resource is useful either by providing an implementation of a ml
technique, helping the user to generate information/data which is further used
by a ml technique, or by providing a processed dataset that can be directly
employed in a ml study.

Table 5 lists all the tools that we found in this exploration. Each resource
is listed with it’s category, name and link to access the resource, number of
citations (as of Jul 2022), and the time when it was first introduced along
with the time when the resource was last updated. We collected the metadata
about the resources manually by searching the digital libraries, repositories,
and authors’ websites. The cases where we could not find the required
information, we mark the entry with ”–”. We also provide a short description
of the resource.

Table 2. A list of tools useful for analyzing source code and applying machine learning techniques Category |  Name |  #Citation |  Introd. |  Updated |  Description
---|---|---|---|---|---
Code Representation |  ncc (Ben-Nun et al., 2018) |  3 |  Dec 2018 |  Aug 2021 |  Learns representations of code semantics
Code2vec (Alon et al., 2019b) |  271 |  Jan 2019 |  Feb 2022 |  Generates distributed representation of code
Code2seq (Alon et al., 2019a) |  418 |  May 2019 |  Jul 2022 |  Generates sequences from structured representation of code
Vector representation for coding style (Kovalenko et al., 2020) |  1 |  Sep 2020 |  Jul 2022 |  Implements vector representation of individual coding style
CC2Vec (Hoang et al., 2020) |  23 |  Oct 2020 |  – |  Implements distributed representation of code changes
AutoenCODE (Tufano et al., 2018) |  1 |  – |  – |  Encodes source code fragments into vector representations
Graph-based code modeling (Allamanis et al., 2018b) |  544 |  May 2018 |  May 2021 |  Generates code modeling with graphs
Vocabulary learning on code (Cvitkovic et al., 2019) |  34 |  Jan 2019 |  – |  Generates an augmented ast from Java source code
User2code2vec (Azcona et al., 2019) |  14 |  Mar 2019 |  May 2019 |  Generates embeddings for developers based on distributed representation of code
Code Search |  Deep Code Search (Gu et al., 2018) |  160 |  May 2018 |  May 2022 |  Searches code by using code embeddings
Program Comprehension |  Obfuscated-code2vec (Compton et al., 2020) |  14 |  Oct 2022 |  – |  Embeds Java Classes with Code2vec
DeepTyper (Hellendoorn et al., 2018) |  54 |  Oct 2018 |  Feb 2020 |  Annotates types for JavaScript and TypeScript
CallNN (Liu et al., 2019c) |  6 |  Oct 2019 |  – |  Implements a code summarization approach by using call dependencies
NeuralCodeSum (Ahmad et al., 2020) |  147 |  May 2020 |  Oct 2021 |  Implements a code summarization method by using transformers
Summarization_tf (Shido et al., 2019) |  9 |  Jul 2019 |  – |  Summarizes code with Extended Tree-lstm
CoaCor (Yao et al., 2019) |  16 |  Jul 2019 |  May 2020 |  Explores the role of rich annotation for code retrieval
DeepCom (Li et al., 2020b) |  12 |  Nov 2020 |  May 2021 |  Generates code comments
Rencos (Zhang et al., 2020a) |  79 |  Oct 2020 |  – |  Generates code summary by using both neural and retrieval-based techniques
codes (Panichella et al., 2012) |  107 |  Jul 2012 |  Jul 2016 |  Extracts method description from StackOverflow discussions
cfs |  – |  – |  – |  Summarizes code fragments using svm and nb
tassal |  – |  – |  – |  Summarizes code using autofolding
ChangeScribe (Cortes-Coy et al., 2014) |  158 |  Dec 2014 |  Dec 2015 |  Generates commit messages
CodeInsight (Rahman et al., 2015) |  59 |  Nov 2015 |  May 2019 |  Recommends insightful comments for source code
CodeNN (Iyer et al., 2016) |  479 |  Aug 2016 |  May 2017 |  Summarizes code using neural attention model
Code2Que (Gao et al., 2020) |  3 |  Jul 2020 |  Aug 2021 |  Suggests improvements in question titles from mined code in StackOverflow
bi-tbcnn (Bui et al., 2018) |  30 |  Mar 2019 |  May 2019 |  Implements a bi-tbcnn model to classify algorithms
DeepSim (Zhao and Huang, 2018) |  150 |  Oct 2018 |  – |  Implements a dl approach to measure code functional similarity
FCDetector (Fang et al., 2020b) |  22 |  Jul 2020 |  – |  Proposes a fine-grained granularity of source code for functionality identification
Quality Assessment |  SonarQube |  – |  – |  – |  Analyzes code quality
svf (Sui and Xue, 2016) |  216 |  Mar 2016 |  Jul 2022 |  Enables inter-procedural dependency analysis for llvm-based languages
Designite (Sharma et al., 2016) |  18 |  Mar 2016 |  Jul 2022 |  Detects code smells and computes quality metrics in Java and C# code
CloneCognition (Mostaeen et al., 2018) |  4 |  Nov 2018 |  May 2019 |  Proposes a ml framework to validate code clones
smad (Barbez et al., 2020) |  25 |  Mar 2020 |  Feb 2021 |  Implements smell detection (God class and Feature envy) using ml
Checkstyle |  – |  – |  – |  Checks for coding convention in Java code
FindBugs |  – |  – |  – |  Implements a static analysis tool for Java
pmd |  – |  – |  – |  Finds common programming flaws in Java and six other languages
ML Clone Validation Framework (Mostaeen et al., 2018) |  11 |  Aug 2019 |  Aug 2019 |  Implements a ml framework for automatic code clone validation
py-ccflex (Ochodek et al., 2019) |  6 |  Mar 2017 |  Oct 2020 |  Mimics code metrics by using ml
Deep learning smells (Sharma et al., 2021) |  18 |  Jul 2021 |  Nov 2020 |  Implements dl (cnn, rnn, and Autoencoder-based models) to identify four smells
crec (Yue et al., 2018) |  26 |  Nov 2018 |  – |  Recommends clones for refactoring
ml for software refactoring (Aniche et al., 2020) |  31 |  Sep 2020 |  – |  Recommends refactoring by using ml
Program Synthesis |  CoCoNuT (Lutellier et al., 2020) |  86 |  Jul 2020 |  Sep 2021 |  Repairs Java programs
DeepFix (Gupta et al., 2017) |  359 |  Feb 2017 |  Dec 2017 |  Fixes common C errors
Testing |  AppFlow (Hu et al., 2018b) |  47 |  Oct 2018 |  – |  Automates ui tests generation
DeepFuzz (Liu et al., 2019b) |  42 |  Jul 2019 |  Mar 2020 |  Grammar fuzzer that generates C programs
Agilika (Utting et al., 2020) |  7 |  Aug 2020 |  Mar 2022 |  Generates tests from execution traces
BugDetection (Li et al., 2019b) |  66 |  Oct 2019 |  May 2021 |  Trains models for defect prediction
dtldp (Chen et al., 2020a) |  21 |  Aug 2019 |  – |  Implements a deep transfer learning framework
DeepBugs (Pradel and Sen, 2018) |  210 |  Nov 2018 |  May 2021 |  Implements a framework for learning name-based bug detectors
Randoop |  – |  – |  Jul 2022 |  Generates tests automatic for Java code
TestDescriber |  – |  – |  – |  Implements test case summary generator and evaluator
Vulnerability Analysis |  wap (Medeiros et al., 2013) |  3 |  Oct 2013 |  Nov 2015 |  Detects and corrects input validation vulnerabilities
swan(Piskachev et al., 2019) |  6 |  Oct 2019 |  May 2022 |  Identifies vulnerabilities
vccfinder (Perl et al., 2015) |  174 |  Oct 2015 |  May 2017 |  Finds potentially dangerous code in repositories
General |  bert |  43462 |  Oct 2018 |  Mar 2020 |  nlp pre-trained models
bc3 Annotation Framework |  – |  – |  – |  Annotates emails/conversations easily
JGibLDA |  – |  – |  – |  Implements Latent Dirichlet Allocation
Stanford NLP Parser |  – |  – |  – |  A statistical NLP parser
srcML |  – |  – |  May 2022 |  Generates xml representation of sourcecode
CallGraph |  8 |  Oct 2017 |  Oct 2018 |  Generates static and dynamic call graphs for Java code
ML for programming |  – |  – |  – |  Offers various tools such as JSNice, Nice2Predict, and debin

The list of datasets found in this exploration are presented in Table 5.
Similar to Tools’ table, Table 5 lists each resource with it’s category, name
and link to access the resource, number of citations (as of Jul 2022), the
time when it was first introduced along with the time when the resource was
last updated, and a short description of the resource.

Table 3. A list of datasets useful for analyzing source code and applying machine learning techniques Category |  Name |  #Citation |  Introd. |  Updated |  Description
---|---|---|---|---|---
Code Representation |  Code2seq (Alon et al., 2019b) |  271 |  Jan 2019 |  Feb 2022 |  Sequences generated from structured representation of code
GHTorrent (Gousios, 2013) |  645 |  Oct 2013 |  Sep 2020 |  Meta-data from GitHub repositories
Code Completion |  Neural Code Completion |  148 |  Nov 2017 |  Sep 2019 |  Dataset and code for code completion with neural attention and pointer networks
TL-CodeSum (Hu et al., 2018a) |  150 |  Feb 2019 |  Sep 2020 |  Dataset for code summarization
Program Synthesis |  CoNaLa corpus (Yin et al., 2018) |  130 |  Dec 2018 |  Oct 2021 |  Python snippets and corresponding natural language description
IntroClass (Le Goues et al., 2015) |  144 |  Jul 2015 |  Feb 2016 |  Program repair dataset of C programs
Program Comprehension |  Program comprehension dataset (Stapleton et al., 2020) |  61 |  May 2018 |  Aug 2021 |  Contains code for a program comprehension user survey
CommitGen (Jiang et al., 2017) |  – |  – |  – |  Commit messages and the diffs from 1,006 Java projects
StaQC (Yao et al., 2018) |  – |  – |  Nov 2019 |  148K Python and 120K sql question-code pairs from StackOverflow
Quality Assessment |  src-d datasets |  – |  – |  – |  Various labeled datasets (commit messages, duplicates, DockerHub, and Nuget)
BigCloneBench (Svajlenko et al., 2014) |  187 |  Dec 2014 |  Mar 2021 |  Known clones in the IJaDataset source repository
Multi-label smells (Guggulothu and Moiz, 2020) |  28 |  May 2020 |  – |  A dataset of 445 instances of two code smells and 82 metrics
Deep learning smells (Sharma et al., 2021) |  15 |  Jul 2021 |  Nov 2020 |  A dataset of four smells in tokenized form from 1,072 C# and 100 Java repositories
ml for software refactoring (Aniche et al., 2020) |  6 |  Nov 2019 |  – |  Dataset for applying ml to recommend refactoring
QScored (Sharma and Kessentini, 2021) |  0 |  Aug 2021 |  – |  Code smell and metrics dataset for more than 86 thousand open-source repositories
Testing |  Defects4J (Just et al., 2014) |  858 |  Jul 2014 |  Jul 2022 |  Java reproducible bugs
promise (Sayyad Shirabad and Menzies, 2005) |  41 |  – |  Jan 2021 |  Various datasets including defect prediction and cost estimation
BugDetection (Li et al., 2019b) |  66 |  Oct 2019 |  May 2021 |  A bug prediction dataset containing 4.973M methods belonging to 92 different Java project versions
damt (Nair et al., 2019) |  15 |  Aug 2019 |  Dec 2019 |  Metamorphic testing dataset
dtldp (Chen et al., 2020a) |  21 |  Oct 2020 |  – |  Dataset for deep transfer learning for defect prediction
DeepBugs (Pradel and Sen, 2018) |  210 |  Oct 2018 |  Apr 2021 |  A JavaScript code corpus with 150K code snippets
Vulnerability Analysis |  wpscan |  – |  – |  – |  a php dataset for WordPress plugin vulnerabilities
Genome (Zhou and Jiang, 2012) |  1139 |  Jul 2012 |  Dec 2015 |  1,200 malware samples covering the majority of existing malware families
Juliet (Boland and Black, 2012) |  – |  – |  – |  81K synthetic C/C++ and Java programs with known flaws
AndroZoo (Allix et al., 2016) |  – |  – |  – |  15.7M apks from Google’s Play Store
trl (Lin et al., 2018b) |  108 |  Apr 2018 |  Jan 2019 |  Vulnerabilities in six C programs
Draper vdisc (Russell et al., 2018) |  247 |  Jul 2018 |  Nov 2018 |  1.27 million functions mined from c and c++ applications
samate (Black, 2007) |  – |  – |  – |  A set of known security flaws from nist for c, c++, and Java programs
jsVulner (Ferenc et al., 2019) |  – |  – |  – |  JavaScript Vulnerability Analysis dataset
swan (Piskachev et al., 2019) |  6 |  Jul 2019 |  Jul 2022 |  A Vulnerability Analysis collection of 12 Java applications
Project-KB (Ponta et al., 2019) |  49 |  Aug 2019 |  – |  A Manually-Curated dataset of fixes to vulnerabilities of open-source software
General |  GitHub Java Corpus (Allamanis and Sutton, 2013) |  333 |  – |  – |  A large collection of Java repositories
150k Python dataset (Raychev et al., 2016) |  – |  – |  – |  Contains parsed ast for 150K Python files
uci source code dataset (Lopes et al., 2010) |  38 |  Apr 2010 |  Nov 2013 |  Various large scale source code analysis datasets

##  6\. Challenges and perceived deficiencies

The aim of this section is to focus on the perceived deficiencies, challenges,
and opportunities in applying ml techniques in the context of source code
analysis observed from the primary studies. We document challenges or
deficiencies mentioned in the considered primary studies while studying and
summarizing them. After the summarization phase was over, we consolidated all
the documented notes and synthesized a summary that we present below.

  * •

Standard datasets: ml is by nature data hungry; specifically, supervised
learning methods need a considerably large, cleaned, and annotated dataset.
Though the size of available open software engineering artifacts is increasing
day by day, lack of high-quality datasets (i.e., clean and reliably annotated)
are one of the biggest challenges in the domain (Ghaffarian and Shahriari,
2017; Ucci et al., 2019; Gondra, 2008; Kumar and Singh, 2012; Durelli et al.,
2019; Chen et al., 2020a; Barbez et al., 2020; Alsolai and Roper, 2020;
Tsintzira et al., 2020; Soto and Le Goues, 2018; Tian et al., 2020;
Svyatkovskiy et al., 2020; Gopinath et al., 2016; Sakkas et al., 2020; Liu et
al., 2019a; Wan et al., 2019; Shen and Chen, 2020; Jimenez et al., 2019).
Therefore, there is a need for defining standardized datasets. Authors have
cited low performance, poor generalizability, and over-fitting due to poor
dataset quality as the results of the lack of standard validated high-quality
datasets.

  * •

Reproducibility and replicability: Reproducibility and replicability of any ml
implementation can be compromised by factors discussed below.

    * –

Insufficient information: Aspects such as ml model, their hyper-parameters,
data size and ratio (of benign and faulty samples, for instance) are needed to
understand and replicate the study. During our exploration, we found numerous
studies that do not present even the bare-minimum pieces of information to
replicate and reproduce their results. Likewise, Di Nucci et al. (2018)
carried out a detailed replication study and reported that the replicated
results were lower by up to 90% compared to what was reported in the original
study.

    * –

Handling of data imbalance: It is very common to have imbalanced datasets in
software engineering applications. Authors use techniques such as under-
sampling and over-sampling to overcome the challenge for training. However,
test datasets must retain the original sample ratio as found in the real world
(Di Nucci et al., 2018); carrying out a performance evaluation based on a
balanced dataset is flawed. Obviously, the model will perform significantly
inferior when it is put at work in a real-world context. We noted many studies
(Agnihotri and Chug, 2020; Oliveira et al., 2020; Guggulothu and Moiz, 2020;
Fontana et al., 2013, 2015; Thongkum and Mekruksavanich, 2020; Cunha et al.,
2020) that used balanced samples and often did not provide the size and ratio
of the training and testing dataset. Such improper handling of data imbalance
contributes to poor reproducibility.

  * •

Maturity in ml development: Development of ml systems are inherently different
from traditional software development (Wan et al., 2019). Phases of ml
development are very exploratory in nature and highly domain and problem
dependent (Wan et al., 2019). Identifying the most appropriate ml model, their
appropriate parameters, and configuration is largely driven by trial and error
manner (Wan et al., 2019; Azeem et al., 2019; Shen and Chen, 2020). Such an ad
hoc and immature software development environment poses a huge challenge to
the community.

A related challenge is lack of tools and techniques for ml software
development. It includes effective tools for testing ml programs, ensuring
that the dataset are pre-processed adequately, debugging, and effective data
management (Wan et al., 2019; Patel et al., 2008; Giray, 2021). In addition,
quality aspects such as explainability and trust-worthiness are new desired
quality aspects especially applicable for ml code where current practices and
knowledge is inadequate (Giray, 2021).

  * •

Data privacy and bias: Data hungry ml models are considered as good as the
data they are consuming. Data collection and preparation without data
diversity leads to bias and unfairness. Although we are witnessing more
efforts to understand these sensitive aspects (Zhang and Harman, 2021; Brun
and Meliou, 2018), the present set of methods and practices lack the support
to deal with data privacy issues at large as well as data diversity and
fairness (Brun and Meliou, 2018; Giray, 2021).

  * •

Effective feature engineering: Features represent the problem-specific
knowledge in pieces extracted from the data; the effectiveness of any ml model
depends on the features fed into it. Many studies identified the importance of
effective feature engineering and the challenges in gathering the same
(Tsintzira et al., 2020; Shen and Chen, 2020; Patel et al., 2008; Wan et al.,
2019; Ivers et al., 2019). Specifically, software engineering researchers have
notified that identifying and extracting relevant features beyond code quality
metrics is non-trivial. For example, Ivers et al. (2019) discusses that
identifying features that establishes a relationship among different code
elements is a significant challenge for ml implementations applied on source
code analysis. Sharma et al. (2021) have shown in their study that smell
detection using ml techniques perform poorly especially for design smells
where multiple code elements and their properties has to be observed.

  * •

Skill gap: Wan et al. (2019) identified that ml software development requires
an extended set of skills beyond software development including ml techniques,
statistics, and mathematics apart from the application domain. Similarly, Hall
and Bowes (2012) also reports a serious lack of ml expertise in academic
software engineering efforts. Other authors (Patel et al., 2008) have
emphasized the importance of domain knowledge to design effective ml models.

  * •

Hardware resources: Given the need of large training dataset and many hidden
layers, often ml training requires high-end processing units (such as gpus and
memory) (Wan et al., 2019; Giray, 2021). A user-survey study (Wan et al.,
2019) highlights the need to special hardware for ml training. Such
requirements poses a challenge to researchers constrained with limited
hardware resources.

##  7\. Discussion

This section provides a discussion on the top venues for articles belonging to
each selected category in our scope as well as on potential mitigations for
the challenges we identified in the previous section.

###  7.1. Venue and article categories

The goal of the exploration is to understand the top venues for each
considered category. We identified and manually curated the software
engineering venue for each primary study discussed in our literature review.
Figure 13 shows the venues for the considered categories. We show the most
prominent venues per category. Each label includes a number indicating the
number of articles published at the same venue in that category.

Figure 13. Top venues for each considered category

We observe that icse and ase are among the top venues, appearing in four and
three categories respectively. tse, jss, and ieee Access are the top journals
for the considered categories. Machine learning conferences such as iclr also
appear as the one of top venues for program synthesis category).

The categories of program comprehension and program synthesis exhibit the
highest concentration of articles to a relatively small list of top venues
where 45% and 35% of articles, respectively, come from the top venues. On the
other hand, researchers publish articles related to testing and vulnerability
in a rather large number of venues.

###  7.2. Model selection

Selecting a ml model for a given task depends on many factors such as nature
of the problem, properties of training and input samples, and expected output.
Below, we provide an analysis of employed ml models based on these factors.

  * •

One of the factors that influence the choice of ml models is the chosen
features and their properties. Studies in the quality assessment category
majorly relied on token-based features and code quality metrics. Such features
allowed studies in this categories to use traditional ml models. Some authors
applied dl models such as dnn when higher-granularity constructs such as cfg
and dfg are used as features.

  * •

Similarly, the majority of the studies in testing category relied on code
quality metrics. Therefore, they have fixed size, fixed meaning (for each
column) vectors to feed to a ml model. With such inputs, traditional ml
approaches, such as Random Forest and Support Vector Machine, work well. Other
studies used a variation of ast or ast of the changes to generate the
embeddings. dl models including dnn and rnn-based models are used to first
train a model for embeddings. A typical ml classifier use the embeddings to
classify samples in buggy or benign.

  * •

Typical output of a code representation study is embedding representing code
in the vector form. The semantics of the produced embeddings depends
significantly on the selected features. Studies in this domain identify this
aspect and hence swiftly focused to extract features that capture the relevant
semantics; for example, path-based features encode the order among the tokens.
The chosen ml model plays another important role to generate effective
embeddings. Given the success of rnn with text processing tasks, due to its
capability to identify sequence and pattern, rnn-based models dominate this
category.

  * •

Program repair is typically a sequence to sequence transformation i.e., a
sequence of buggy code is the input and a sequence of fixed code is the
output. Given the nature of the problem, it is not surprising to observe that
the majority of the studies in this category used Encoder-Decoder-based
models. rnn are considered a popular choice to realize Encoder-Decoder models
due to its capability to remember long sequences.

###  7.3. Mitigating the challenges

####  7.3.1. Availability of standard datasets:

Although available datasets have increased, given a wide number of software
engineering tasks and variations in these tasks as well as the need of
application-specific datasets, the community still looks for application-
specific, large, and high-quality datasets. To mitigate the issue, the
community has focused on developing new datasets and making them publicly
available by organizing a dedicated track, for example, the msr data showcase
track. Dataset search engines such as Google dataset
search333https://datasetsearch.research.google.com/ could be used to search
available datasets. Researchers may also propose generic datasets that can
serve multiple application domains or at least different variations of a
software engineering task. In addition, recent advancements in ml techniques
such as active learning (Prince, 2004; Settles, 2009; Ren et al., 2020) may
reduce the need of large datasets. Besides, the way the data is used for model
validation must be improved. For example, Jimenez et al. (Jimenez et al.,
2019) showed that previous studies on vulnerability prediction trained
predictive models by using perfect labelling information (i.e., including
future labels, as yet undiscovered vulnerabilities) and showed that such an
unrealistic labelling assumption can profoundly affect the scientific
conclusions of a study as the prediction performance worsen dramatically when
one fully accounts for realistically available labelling.

####  7.3.2. Reproducibility and replicability:

The importance of reproducibility and replicability has been emphasized and
understood by the software engineering community (Liu et al., 2020b). It has
lead to a concrete artifact evaluation mechanism adopted by leading software
engineering conferences. For example, fse artifact evaluation divides
artifacts into five categories—functional, reusable, available, results
reproduced, and results replicated.444https://2021.esec-
fse.org/track/fse-2021-artifacts Such thorough evaluation encouraging software
engineering authors to produce high-quality documentation along with easily
replicate experiment results using their developed artifacts. In addition,
efforts (such as model engineering process (Banna et al., 2021)) are being
made to support ml research reproducible and replicable.

####  7.3.3. Maturity in ML development:

The ad-hoc trial and error ml development can be addressed by improved tools
and techniques. Even though the variety of ml development environments
including managed services such as aws Sagemaker and Google Notebooks attempt
to make ml development easier, they essentially do not offer much help in
reducing the ad-hoc nature of the development. A significant research push
from the community would make ml development relatively systematic and
organized.

Recent advancements in the form of available tools not only help a developer
to comprehend the process but also let them effectively manage code, data, and
experimental results. Examples of such tools and methods include darviz
(Sankaran et al., 2017) for dl model visualization,
mlflow555https://mlflow.org/ for managing the ml lifecycle, and DeepFault
(Eniser et al., 2019) for identifying faults in dl programs. Such efforts are
expected to address the challenge.

Software Engineering for Machine Learning (se4ml) brings another perspective
to this issue by bringing best practices from software engineering to ml
development. Efforts in this direction not only can make ml specific code
maintainable and reliable but also can contribute back to reproducibility and
replicability.

####  7.3.4. Hardware resources:

ml development is resource hungry. Certain dl models (such as models based on
rnn) consume excessive hardware resources. The need for a large-scale hardware
infrastructure is increasing with the increase in size of the captured
features and the training samples. To address the challenge, infrastructure at
institution and country level are maintained in some countries; however, a
generic and widely-applicable solution is needed for more globally-inclusive
research.

The first internal threats to validity relates to the concern of covering all
the relevant articles in the selected domain. To mitigate the concern, we
defined our scope i.e., studies that use ml techniques to solve a software
engineering problem by analyzing source code. We also carefully defined
inclusion and exclusion criteria for selecting relevant studies. We carry out
extensive manual search process on commonly used digital libraries with the
help of a comprehensive set of search terms; we augment the search terms that
are used in related articles to maximize the chances of identifying the
relevant articles.

Another threat to validity is the validity of data extraction and their
interpretation applicable to the generated summary and metadata for each
primary study. We mitigated this threat by dividing the task of summarization
to all the authors and cross verifying the generated information. During the
manual summarization phase, metadata of each paper was reviewed by, at least,
two authors.

External validity concerns the generalizability and reproducibility of the
produced results and observations. We provide a spreadsheet (Sharma et al.,
2022) containing all the metadata for all the articles selected in each of the
phases of article selection. In addition, inspired by previous surveys
(Allamanis et al., 2018a; Hort et al., 2021), we have developed a
website666http://www.tusharma.in/ML4SCA as a living documentation and
literature survey to facilitate easy navigation, exploration, and extension.
The website can be easily extended as the new studies emerge in the domain; we
have made the repository777https://github.com/tushartushar/ML4SCA open-source
to allow the community to extend the living literature survey.

##  8\. Conclusions

With the increasing presence of ml techniques in software engineering
research, it has become challenging to have a comprehensive overview of its
advancements. This survey aims to provide a detailed overview of the studies
at the intersection of source code analysis and ml. We have selected 479
primary studies spanning from 2011 to 2021 (and to some extent 2022) covering
12 software engineering categories. We present a synthesized summary of the
selected studies arranged in categories, subcategories, and their
corresponding involved steps. Also, the survey consolidates useful resources
(datasets and tools) that could ease the task for future studies. Finally, we
present perceived challenges and opportunities in the field. The presented
opportunities invite practitioners as well as researchers to propose new
methods, tools, and techniques to make the integration of ml techniques for
software engineering applications easy, flexible, and maintainable.

###### Acknowledgements.

This work is supported by the erc Advanced fellowship grant no. 741278 (epic).

## References

  * (1)
  * mis (2020) 2020\.  GitHub archive.  https://www.gharchive.org/
  * Abbas et al. (2020) Raja Abbas, Fawzi Abdulaziz Albalooshi, and Mustafa Hammad. 2020.  Software change proneness prediction using machine learning. In _2020 International Conference on Innovation and Intelligence for Informatics, Computing and Technologies (3ICT)_. IEEE, 1–7.
  * Abdalkareem et al. (2020) Rabe Abdalkareem, Suhaib Mujahid, and Emad Shihab. 2020\.  A machine learning approach to improve the detection of ci skip commits.  _IEEE Transactions on Software Engineering_ (2020).
  * Abdeljaber et al. (2017) Osama Abdeljaber, Onur Avci, Serkan Kiranyaz, Moncef Gabbouj, and Daniel J Inman. 2017.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks.  _Journal of Sound and Vibration_ 388 (2017), 154–170.
  * Abuhamad et al. (2018) Mohammed Abuhamad, Tamer AbuHmed, Aziz Mohaisen, and DaeHun Nyang. 2018.  Large-Scale and Language-Oblivious Code Authorship Identification. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ (Toronto, Canada) _(CCS ’18)_. 101–114.  https://doi.org/10.1145/3243734.3243738
  * Abunadi and Alenezi (2015) Ibrahim Abunadi and Mamdouh Alenezi. 2015.  Towards Cross Project Vulnerability Prediction in Open Source Web Applications. In _Proceedings of the The International Conference on Engineering & MIS 2015_ (Istanbul, Turkey) _(ICEMIS ’15)_. Association for Computing Machinery, New York, NY, USA, Article 42, 5 pages.  https://doi.org/10.1145/2832987.2833051
  * Aggarwal (2019) Simran Aggarwal. 2019\.  Software Code Analysis Using Ensemble Learning Techniques. In _Proceedings of the International Conference on Advanced Information Science and System_ (Singapore, Singapore) _(AISS ’19)_. Article 9, 7 pages.  https://doi.org/10.1145/3373477.3373486
  * Agnihotri and Chug (2020) Mansi Agnihotri and Anuradha Chug. 2020.  Application of machine learning algorithms for code smell prediction using object-oriented software metrics.  _Journal of Statistics and Management Systems_ 23, 7 (2020), 1159–1171.  https://doi.org/10.1080/09720510.2020.1799576 arXiv:https://doi.org/10.1080/09720510.2020.1799576
  * Ahmad et al. (2020) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020.  A Transformer-based Approach for Source Code Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 4998–5007.  https://doi.org/10.18653/v1/2020.acl-main.449
  * Ahmed et al. (2018) Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. 2018.  Compilation Error Repair: For the Student Programs, from the Student Programs. In _Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training_ (Gothenburg, Sweden) _(ICSE-SEET ’18)_. 78–87.  https://doi.org/10.1145/3183377.3183383
  * Al-Jamimi and Ahmed (2013) H. A. Al-Jamimi and M. Ahmed. 2013.  Machine Learning-Based Software Quality Prediction Models: State of the Art. In _2013 International Conference on Information Science and Applications (ICISA)_. 1–4.  https://doi.org/10.1109/ICISA.2013.6579473
  * Al Qasem et al. (2020) Osama Al Qasem, Mohammed Akour, and Mamdouh Alenezi. 2020\.  The influence of deep learning algorithms factors in software fault prediction.  _IEEE Access_ 8 (2020), 63945–63960.
  * AL-Shaaby et al. (2020) A. AL-Shaaby, Hamoud I. Aljamaan, and M. Alshayeb. 2020\.  Bad Smell Detection Using Machine Learning Techniques: A Systematic Literature Review.  _Arabian Journal for Science and Engineering_ 45 (2020), 2341–2369.
  * Alazba and Aljamaan (2021) Amal Alazba and Hamoud Aljamaan. 2021.  Code smell detection using feature selection and stacking ensemble: An empirical investigation.  _Information and Software Technology_ 138 (2021), 106648.
  * Aleem et al. (2015) Saiqa Aleem, Luiz Fernando Capretz, Faheem Ahmed, et al. 2015\.  Comparative performance analysis of machine learning techniques for software bug detection. In _Proceedings of the 4th International Conference on Software Engineering and Applications_. AIRCC Press Chennai, Tamil Nadu, India, 71–79.
  * Aleti and Martinez (2021) Aldeida Aleti and Matias Martinez. 2021.  E-APR: mapping the effectiveness of automated program repair techniques.  _Empirical Software Engineering_ 26, 5 (2021), 1–30.
  * Alhusain et al. (2013) Sultan Alhusain, Simon Coupland, Robert John, and Maria Kavanagh. 2013.  Towards machine learning based design pattern recognition. In _2013 13th UK Workshop on Computational Intelligence (UKCI)_. IEEE, 244–251.
  * Ali Alatwi et al. (2016) Huda Ali Alatwi, Tae Oh, Ernest Fokoue, and Bill Stackpole. 2016\.  Android Malware Detection Using Category-Based Machine Learning Classifiers. In _Proceedings of the 17th Annual Conference on Information Technology Education_ (Boston, Massachusetts, USA) _(SIGITE ’16)_. 54–59.  https://doi.org/10.1145/2978192.2978218
  * Alikhashashneh et al. (2018) E. A. Alikhashashneh, R. R. Raje, and J. H. Hill. 2018\.  Using Machine Learning Techniques to Classify and Predict Static Code Analysis Tool Warnings. In _2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA)_. 1–8.  https://doi.org/10.1109/AICCSA.2018.8612819
  * Aljamaan and Alazba (2020) Hamoud Aljamaan and Amal Alazba. 2020.  Software defect prediction using tree-based ensembles. In _Proceedings of the 16th ACM international conference on predictive models and data analytics in software engineering_. 1–10.
  * Allamanis et al. (2015a) Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. 2015a.  Suggesting Accurate Method and Class Names. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. 38–49.  https://doi.org/10.1145/2786805.2786849
  * Allamanis et al. (2018a) Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018a.  A Survey of Machine Learning for Big Code and Naturalness.  _ACM Comput. Surv._ 51, 4, Article 81 (July 2018), 37 pages.  https://doi.org/10.1145/3212695
  * Allamanis et al. (2018b) Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018b.  Learning to Represent Programs with Graphs. In _International Conference on Learning Representations_.
  * Allamanis et al. (2016) Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016\.  A Convolutional Attention Network for Extreme Summarization of Source Code.  arXiv:1602.03001 [cs.LG]
  * Allamanis and Sutton (2013) M. Allamanis and C. Sutton. 2013.  Mining source code repositories at massive scale using language modeling. In _2013 10th Working Conference on Mining Software Repositories (MSR)_. 207–216.  https://doi.org/10.1109/MSR.2013.6624029
  * Allamanis and Sutton (2013) Miltiadis Allamanis and Charles Sutton. 2013.  Mining source code repositories at massive scale using language modeling. In _10th Working Conference on Mining Software Repositories (MSR)_. 207–216.  https://doi.org/10.1109/MSR.2013.6624029
  * Allamanis et al. (2015b) Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015b.  Bimodal Modelling of Source Code and Natural Language. In _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ (Lille, France) _(ICML’15)_. 2123–2132.
  * Allix et al. (2016) Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016\.  AndroZoo: Collecting Millions of Android Apps for the Research Community. In _Proceedings of the 13th International Conference on Mining Software Repositories_ (Austin, Texas) _(MSR ’16)_. 468–471.  https://doi.org/10.1145/2901739.2903508
  * Alon et al. (2019a) Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019a.  code2seq: Generating Sequences from Structured Representations of Code.  arXiv:1808.01400 [cs.LG]
  * Alon et al. (2018) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018.  A General Path-Based Representation for Predicting Program Properties.  _SIGPLAN Not._ 53, 4 (June 2018), 404–419.  https://doi.org/10.1145/3296979.3192412
  * Alon et al. (2019b) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019b.  Code2vec: Learning Distributed Representations of Code.  _Proc. ACM Program. Lang._ 3, POPL, Article 40 (January 2019), 29 pages.  https://doi.org/10.1145/3290353
  * Alrajeh et al. (2015) Dalal Alrajeh, Jeff Kramer, Alessandra Russo, and Sebastian Uchitel. 2015.  Automated Support for Diagnosis and Repair.  _Commun. ACM_ 58, 2 (January 2015), 65–72.  https://doi.org/10.1145/2658986
  * Alsolai and Roper (2020) Hadeel Alsolai and Marc Roper. 2020.  A systematic literature review of machine learning techniques for software maintainability prediction.  _Information and Software Technology_ 119 (2020), 106214.  https://doi.org/10.1016/j.infsof.2019.106214
  * Alves et al. (2016) H. Alves, B. Fonseca, and N. Antunes. 2016.  Experimenting Machine Learning Techniques to Predict Vulnerabilities. In _2016 Seventh Latin-American Symposium on Dependable Computing (LADC)_. 151–156.  https://doi.org/10.1109/LADC.2016.32
  * Amal et al. (2014) Boukhdhir Amal, Marouane Kessentini, Slim Bechikh, Josselin Dea, and Lamjed Ben Said. 2014.  On the Use of Machine Learning and Search-Based Software Engineering for Ill-Defined Fitness Function: A Case Study on Software Refactoring. In _Search-Based Software Engineering_ , Claire Le Goues and Shin Yoo (Eds.). 31–45.
  * Amorim et al. (2015) L. Amorim, E. Costa, N. Antunes, B. Fonseca, and M. Ribeiro. 2015.  Experience report: Evaluating the effectiveness of decision trees for detecting code smells. In _2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)_. 261–269.  https://doi.org/10.1109/ISSRE.2015.7381819
  * Amorim et al. (2018) L. A. Amorim, M. F. Freitas, A. Dantas, E. F. de Souza, C. G. Camilo-Junior, and W. S. Martins. 2018.  A New Word Embedding Approach to Evaluate Potential Fixes for Automated Program Repair. In _2018 International Joint Conference on Neural Networks (IJCNN)_. 1–8.  https://doi.org/10.1109/IJCNN.2018.8489079
  * Aniche et al. (2020) M. Aniche, E. Maziero, R. Durelli, and V. Durelli. 2020.  The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring.  _IEEE Transactions on Software Engineering_ (2020), 1–1.  https://doi.org/10.1109/TSE.2020.3021736
  * Arar and Ayan (2015) ”̈Omer Faruk Arar and K”̈urşat Ayan. 2015.  Software defect prediction using cost-sensitive neural network.  _Applied Soft Computing_ 33 (2015), 263–277.
  * Arcelli Fontana and Zanoni (2017) Francesca Arcelli Fontana and Marco Zanoni. 2017.  Code smell severity classification using machine learning techniques.  _Knowledge-Based Systems_ 128 (2017), 43 – 58.  https://doi.org/10.1016/j.knosys.2017.04.014
  * Aribandi et al. (2019) Vamsi Krishna Aribandi, Lov Kumar, Lalita Bhanu Murthy Neti, and Aneesh Krishna. 2019.  Prediction of Refactoring-Prone Classes Using Ensemble Learning. In _Neural Information Processing_ , Tom Gedeon, Kok Wai Wong, and Minho Lee (Eds.). 242–250.
  * Axelsson et al. (2009) S. Axelsson, D. Baca, Robert Feldt, Darius Sidlauskas, and Denis Kacan. 2009.  Detecting Defects with an Interactive Code Review Tool Based on Visualisation and Machine Learning. In _SEKE_.
  * Azcona et al. (2019) David Azcona, Piyush Arora, I-Han Hsiao, and Alan Smeaton. 2019\.  User2code2vec: Embeddings for Profiling Students Based on Distributional Representations of Source Code. In _Proceedings of the 9th International Conference on Learning Analytics & Knowledge_ (Tempe, AZ, USA) _(LAK19)_. 86–95.  https://doi.org/10.1145/3303772.3303813
  * Azeem et al. (2019) Muhammad Ilyas Azeem, Fabio Palomba, Lin Shi, and Qing Wang. 2019.  Machine learning techniques for code smell detection: A systematic literature review and meta-analysis.  _Information and Software Technology_ 108 (2019), 115 – 138.  https://doi.org/10.1016/j.infsof.2018.12.009
  * Bader et al. (2019) Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. 2019.  Getafix: Learning to Fix Bugs Automatically.  _Proc. ACM Program. Lang._ 3, OOPSLA, Article 159 (October 2019), 27 pages.  https://doi.org/10.1145/3360585
  * Ban et al. (2019) Xinbo Ban, Shigang Liu, Chao Chen, and Caslon Chua. 2019\.  A performance evaluation of deep-learnt features for software vulnerability detection.  _Concurrency and Computation: Practice and Experience_ 31, 19 (2019), e5103.  https://doi.org/10.1002/cpe.5103 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpe.5103.
  * Bandara and Wijayarathna (2011) U. Bandara and G. Wijayarathna. 2011.  A Machine Learning Based Tool for Source Code Plagiarism Detection.  _International Journal of Machine Learning and Computing_ (2011), 337–343.
  * Banna et al. (2021) Vishnu Banna, Akhil Chinnakotla, Zhengxin Yan, Anirudh Vegesana, Naveen Vivek, Kruthi Krishnappa, Wenxin Jiang, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis. 2021.  An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors.  _CoRR_ abs/2107.00821 (2021).  arXiv:2107.00821 https://arxiv.org/abs/2107.00821
  * Bansal et al. (2021) A. Bansal, S. Haque, and C. McMillan. 2021.  Project-Level Encoding for Neural Source Code Summarization of Subroutines. In _2021 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) (ICPC)_. IEEE Computer Society, 253–264.  https://doi.org/10.1109/ICPC52881.2021.00032
  * Barbez et al. (2020) Antoine Barbez, Foutse Khomh, and Yann-Gaël Guéhéneuc. 2020\.  A machine-learning based ensemble method for anti-patterns detection.  _Journal of Systems and Software_ 161 (2020), 110486.  https://doi.org/10.1016/j.jss.2019.110486
  * Barone and Sennrich (2017) Antonio Valerio Miceli Barone and Rico Sennrich. 2017.  A parallel corpus of Python functions and documentation strings for automated code documentation and code generation.
  * Batur Şahin and Abualigah (2021) Canan Batur Şahin and Laith Abualigah. 2021.  A Novel Deep Learning-Based Feature Selection Model for Improving the Static Analysis of Vulnerability Detection.  _Neural Comput. Appl._ 33, 20 (oct 2021), 14049–14067.  https://doi.org/10.1007/s00521-021-06047-x
  * Ben-Nun et al. (2018) Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018.  Neural Code Comprehension: A Learnable Representation of Code Semantics. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ (Montréal, Canada) _(NIPS’18)_. 3589–3601.
  * Bhandari and Gupta (2018) G. P. Bhandari and R. Gupta. 2018.  Machine learning based software fault prediction utilizing source code metrics. In _2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS)_. 40–45.  https://doi.org/10.1109/CCCS.2018.8586805
  * Bhatia et al. (2018) Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. 2018\.  Neuro-Symbolic Program Corrector for Introductory Programming Assignments. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. 60–70.  https://doi.org/10.1145/3180155.3180219
  * Bilgin et al. (2020) Z. Bilgin, M. A. Ersoy, E. U. Soykan, E. Tomur, P. Çomak, and L. Karaçay. 2020\.  Vulnerability Prediction From Source Code Using Machine Learning.  _IEEE Access_ 8 (2020), 150672–150684.  https://doi.org/10.1109/ACCESS.2020.3016774
  * Black (2007) Paul E. Black. 2007\.  Software Assurance with SAMATE Reference Dataset, Tool Standards, and Studies.  (Oct. 2007).
  * Boland and Black (2012) Frederick Boland and Paul Black. 2012.  The Juliet 1.1 C/C++ and Java Test Suite.  45 (2012-10-01 2012).  https://doi.org/10.1109/MC.2012.345
  * Bowes et al. (2016) David Bowes, Tracy Hall, Mark Harman, Yue Jia, Federica Sarro, and Fan Wu. 2016\.  Mutation-Aware Fault Prediction. In _Proceedings of the 25th International Symposium on Software Testing and Analysis_ (Saarbrücken, Germany) _(ISSTA 2016)_. Association for Computing Machinery, New York, NY, USA, 330–341.  https://doi.org/10.1145/2931037.2931039
  * Braga et al. (2018) Ronyérison Braga, Pedro Santos Neto, Ricardo Rabêlo, José Santiago, and Matheus Souza. 2018\.  A Machine Learning Approach to Generate Test Oracles. In _Proceedings of the XXXII Brazilian Symposium on Software Engineering_ (Sao Carlos, Brazil) _(SBES ’18)_. 142–151.  https://doi.org/10.1145/3266237.3266273
  * Brauckmann et al. (2020) Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jeronimo Castrillon. 2020.  Compiler-Based Graph Representations for Deep Learning Models of Code. In _Proceedings of the 29th International Conference on Compiler Construction_ (San Diego, CA, USA) _(CC 2020)_. 201–211.
  * Brockschmidt et al. (2019) Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. 2019.  Generative Code Modeling with Graphs. In _International Conference on Learning Representations_.
  * Bruch et al. (2009) Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009\.  Learning from Examples to Improve Code Completion Systems. In _Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering_ (Amsterdam, The Netherlands) _(ESEC/FSE ’09)_. 213–222.  https://doi.org/10.1145/1595696.1595728
  * Brun and Meliou (2018) Yuriy Brun and Alexandra Meliou. 2018.  Software Fairness. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Lake Buena Vista, FL, USA) _(ESEC/FSE 2018)_. Association for Computing Machinery, New York, NY, USA, 754–759.  https://doi.org/10.1145/3236024.3264838
  * Bui et al. (2018) Nghi D. Q. Bui, Lingixao Jiang, and Y. Yu. 2018.  Cross-Language Learning for Program Classification using Bilateral Tree-Based Convolutional Neural Networks. In _AAAI Workshops_.
  * Bui et al. (2019) N. D. Q. Bui, Y. Yu, and L. Jiang. 2019.  Bilateral Dependency Neural Networks for Cross-Language Algorithm Classification. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 422–433.  https://doi.org/10.1109/SANER.2019.8667995
  * Butgereit (2019) L. Butgereit. 2019\.  Using Machine Learning to Prioritize Automated Testing in an Agile Environment. In _2019 Conference on Information Communications Technology and Society (ICTAS)_. 1–6.  https://doi.org/10.1109/ICTAS.2019.8703639
  * Cai et al. (2019) Cheng-Hao Cai, Jing Sun, and Gillian Dobbie. 2019.  Automatic B-model repair using model checking and machine learning.  _Automated Software Engineering_ 26, 3 (Jan. 2019).  https://doi.org/10.1007/s10515-019-00264-4
  * Cambronero and Rinard (2019) José P Cambronero and Martin C Rinard. 2019.  AL: autogenerating supervised learning programs.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 1–28.
  * Caram et al. (2019a) Frederico Luiz Caram, Bruno Rafael De Oliveira Rodrigues, Amadeu Silveira Campanelli, and Fernando Silva Parreiras. 2019a.  Machine learning techniques for code smells detection: a systematic mapping study.  _International Journal of Software Engineering and Knowledge Engineering_ 29, 02 (2019), 285–316.
  * Caram et al. (2019b) Frederico Luiz Caram, Bruno Rafael De Oliveira Rodrigues, Amadeu Silveira Campanelli, and Fernando Silva Parreiras. 2019b.  Machine Learning Techniques for Code Smells Detection: A Systematic Mapping Study.  _International Journal of Software Engineering and Knowledge Engineering_ 29, 02 (2019), 285–316.  https://doi.org/10.1142/S021819401950013X arXiv:https://doi.org/10.1142/S021819401950013X
  * Cesare et al. (2013) Silvio Cesare, Yang Xiang, and Jun Zhang. 2013.  Clonewise – Detecting Package-Level Clones Using Machine Learning. In _Security and Privacy in Communication Networks_ , Tanveer Zia, Albert Zomaya, Vijay Varadharajan, and Morley Mao (Eds.). 197–215.
  * Cetiner and Sahingoz (2020) M. Cetiner and O. K. Sahingoz. 2020.  A Comparative Analysis for Machine Learning based Software Defect Prediction Systems. In _2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT)_. 1–7.  https://doi.org/10.1109/ICCCNT49239.2020.9225352
  * Ceylan et al. (2006) E. Ceylan, F. O. Kutlubay, and A. B. Bener. 2006\.  Software Defect Identification Using Machine Learning Techniques. In _32nd EUROMICRO Conference on Software Engineering and Advanced Applications (EUROMICRO’06)_. 240–247.  https://doi.org/10.1109/EUROMICRO.2006.56
  * Chakraborty et al. (2020) S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray. 2020\.  CODIT: Code Editing with Tree-Based Neural Models.  _IEEE Transactions on Software Engineering_ (2020), 1–1.  https://doi.org/10.1109/TSE.2020.3020502
  * Chakraborty et al. (2022) Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. 2022.  CODIT: Code Editing With Tree-Based Neural Models.  _IEEE Transactions on Software Engineering_ 48, 4 (2022), 1385–1399.  https://doi.org/10.1109/TSE.2020.3020502
  * Chakraborty and Ray (2021) Saikat Chakraborty and Baishakhi Ray. 2021.  On Multi-Modal Learning of Editing Source Code. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 443–455.  https://doi.org/10.1109/ASE51524.2021.1003_Chakraborty2021
  * CHALLAGULLA et al. (2008) VENKATA UDAYA B. CHALLAGULLA, FAROKH B. BASTANI, I-LING YEN, and RAYMOND A. PAUL. 2008.  EMPIRICAL ASSESSMENT OF MACHINE LEARNING BASED SOFTWARE DEFECT PREDICTION TECHNIQUES.  _International Journal on Artificial Intelligence Tools_ 17, 02 (2008), 389–400.  https://doi.org/10.1142/S0218213008003947 arXiv:https://doi.org/10.1142/S0218213008003947
  * Chappelly et al. (2017) T. Chappelly, C. Cifuentes, P. Krishnan, and S. Gevay. 2017.  Machine learning for finding bugs: An initial report. In _2017 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE)_. 21–26.  https://doi.org/10.1109/MALTESQUE.2017.7882012
  * Chaturvedi et al. (2018) Shivam Chaturvedi, Amrita Chaturvedi, Anurag Tiwari, and Shalini Agarwal. 2018.  Design pattern detection using machine learning techniques. In _2018 7th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO)_. IEEE, 1–6.
  * Chen et al. (2019a) Deyu Chen, Xiang Chen, Hao Li, Junfeng Xie, and Yanzhou Mu. 2019a.  Deepcpdp: Deep learning based cross-project defect prediction.  _IEEE Access_ 7 (2019), 184832–184848.
  * Chen et al. (2020a) Jinyin Chen, Keke Hu, Yue Yu, Zhuangzhi Chen, Qi Xuan, Yi Liu, and Vladimir Filkov. 2020a.  Software Visualization and Deep Transfer Learning for Effective Software Defect Prediction. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. 578–589.  https://doi.org/10.1145/3377811.3380389
  * Chen et al. (2019) Long Chen, Wei Ye, and Shikun Zhang. 2019.  Capturing Source Code Semantics via Tree-Based Convolution over API-Enhanced AST. In _Proceedings of the 16th ACM International Conference on Computing Frontiers_ (Alghero, Italy) _(CF ’19)_. 174–182.  https://doi.org/10.1145/3310273.3321560
  * Chen and Wan (2019) M. Chen and X. Wan. 2019.  Neural Comment Generation for Source Code with Auxiliary Code Classification Task. In _2019 26th Asia-Pacific Software Engineering Conference (APSEC)_. 522–529.  https://doi.org/10.1109/APSEC48747.2019.00076
  * Chen et al. (2019b) Qiuyuan Chen, Han Hu, and Zhaoyi Liu. 2019b.  Code Summarization with Abstract Syntax Tree. In _Neural Information Processing_ , Tom Gedeon, Kok Wai Wong, and Minho Lee (Eds.). 652–660.
  * Chen et al. (2021) Qiuyuan Chen, Xin Xia, Han Hu, David Lo, and Shanping Li. 2021.  Why my code summarization model does not work: Code comment improvement with category prediction.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 30, 2 (2021), 1–29.
  * Chen and Zhou (2018) Q. Chen and M. Zhou. 2018.  A Neural Framework for Retrieval and Summarization of Source Code. In _2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 826–831.  https://doi.org/10.1145/3238147.3240471
  * Chen et al. (2016) Xinyun Chen, Chang Liu, Richard Shin, Dawn Song, and Mingcheng Chen. 2016.  Latent Attention for If-Then Program Synthesis. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ (Barcelona, Spain) _(NIPS’16)_. 4581–4589.
  * Chen et al. (2020b) Yang Chen, Andrew E. Santosa, Ang Ming Yi, Abhishek Sharma, Asankhaya Sharma, and David Lo. 2020b.  _A Machine Learning Approach for Vulnerability Curation_.  Association for Computing Machinery, New York, NY, USA, 32–42.  https://doi.org/10.1145/3379597.3387461
  * Chen et al. (2019) Z. Chen, S. J. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus. 2019\.  SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair.  _IEEE Transactions on Software Engineering_ (2019), 1–1.  https://doi.org/10.1109/TSE.2019.2940179
  * Chernis and Verma (2018) Boris Chernis and Rakesh Verma. 2018.  Machine Learning Methods for Software Vulnerability Detection. In _Proceedings of the Fourth ACM International Workshop on Security and Privacy Analytics_ (Tempe, AZ, USA) _(IWSPA ’18)_. 31–39.  https://doi.org/10.1145/3180445.3180453
  * Chidamber and Kemerer (1994) S. R. Chidamber and C. F. Kemerer. 1994.  A Metrics Suite for Object Oriented Design.  _IEEE Transaction of Software Engineering_ 20, 6 (June 1994), 476–493.  https://doi.org/10.1109/32.295895
  * Choi et al. (2020) Y. Choi, S. Kim, and J. Lee. 2020.  Source Code Summarization Using Attention-Based Keyword Memory Networks. In _2020 IEEE International Conference on Big Data and Smart Computing (BigComp)_. 564–570.  https://doi.org/10.1109/BigComp48618.2020.00011
  * Choudhary et al. (2018) Garvit Rajesh Choudhary, Sandeep Kumar, Kuldeep Kumar, Alok Mishra, and Cagatay Catal. 2018\.  Empirical analysis of change metrics for software fault prediction.  _Computers & Electrical Engineering_ 67 (2018), 15–24.
  * Chug and Dhall (2013) A. Chug and S. Dhall. 2013.  Software defect prediction using supervised learning algorithm and unsupervised learning algorithm. In _Confluence 2013: The Next Generation Information Technology Summit (4th International Conference)_. 173–179.  https://doi.org/10.1049/cp.2013.2313
  * Clemente et al. (2018) C. J. Clemente, F. Jaafar, and Y. Malik. 2018\.  Is Predicting Software Security Bugs Using Deep Learning Better Than the Traditional Machine Learning Algorithms?. In _2018 IEEE International Conference on Software Quality, Reliability and Security (QRS)_. 95–102.  https://doi.org/10.1109/QRS.2018.00023
  * Compton et al. (2020) Rhys Compton, Eibe Frank, Panos Patros, and Abigail Koay. 2020\.  Embedding Java Classes with Code2vec: Improvements from Variable Obfuscation. In _Proceedings of the 17th International Conference on Mining Software Repositories_ (Seoul, Republic of Korea) _(MSR ’20)_. 243–253.  https://doi.org/10.1145/3379597.3387445
  * Cortes-Coy et al. (2014) Luis Fernando Cortes-Coy, M. Vásquez, Jairo Aponte, and D. Poshyvanyk. 2014.  On Automatically Generating Commit Messages via Summarization of Source Code Changes.  _2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation_ (2014), 275–284.
  * Cruz et al. (2020a) Daniel Cruz, Amanda Santana, and Eduardo Figueiredo. 2020a.  Detecting bad smells with machine learning algorithms: an empirical study. In _Proceedings of the 3rd International Conference on Technical Debt_. 31–40.
  * Cruz et al. (2020b) Daniel Cruz, Amanda Santana, and Eduardo Figueiredo. 2020b.  Detecting Bad Smells with Machine Learning Algorithms: An Empirical Study. In _Proceedings of the 3rd International Conference on Technical Debt_ (Seoul, Republic of Korea) _(TechDebt ’20)_. 31–40.  https://doi.org/10.1145/3387906.3388618
  * Cui et al. (2020) Jianfeng Cui, Lixin Wang, Xin Zhao, and Hongyi Zhang. 2020\.  Towards predictive analysis of android vulnerability using statistical codes and machine learning for IoT applications.  _Computer Communications_ 155 (2020), 125 – 131.  https://doi.org/10.1016/j.comcom.2020.02.078
  * Cunha et al. (2020) Warteruzannan Soyer Cunha, Guisella Angulo Armijo, and Valter Vieira de Camargo. 2020.  _Investigating Non-Usually Employed Features in the Identification of Architectural Smells: A Machine Learning-Based Approach_.  21–30.
  * Cvitkovic et al. (2019) Milan Cvitkovic, Badal Singh, and Animashree Anandkumar. 2019\.  Open Vocabulary Learning on Source Code with a Graph-Structured Cache _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). 1475–1485.
  * Dam et al. (2019) Hoa Khanh Dam, Trang Pham, Shien Wee Ng, Truyen Tran, John Grundy, Aditya Ghose, Taeksu Kim, and Chul-Joo Kim. 2019\.  Lessons Learned from Using a Deep Tree-Based Model for Software Defect Prediction in Practice. In _Proceedings of the 16th International Conference on Mining Software Repositories_ (Montreal, Quebec, Canada) _(MSR ’19)_. 46–57.  https://doi.org/10.1109/MSR.2019.00017
  * D’Ambros et al. (2012) Marco D’Ambros, Michele Lanza, and Romain Robbes. 2012\.  Evaluating Defect Prediction Approaches: A Benchmark and an Extensive Comparison.  _Empirical Softw. Engg._ 17, 4–5 (Aug. 2012), 531–577.  https://doi.org/10.1007/s10664-011-9173-9
  * Dantas et al. (2019) Altino Dantas, Eduardo F. de Souza, Jerffeson Souza, and Celso G. Camilo-Junior. 2019.  Code Naturalness to Assist Search Space Exploration in Search-Based Program Repair Methods. In _Search-Based Software Engineering_ , Shiva Nejati and Gregory Gay (Eds.). 164–170.
  * Dejaeger et al. (2012) Karel Dejaeger, Thomas Verbraken, and Bart Baesens. 2012\.  Toward comprehensible software fault prediction models using bayesian network classifiers.  _IEEE Transactions on Software Engineering_ 39, 2 (2012), 237–257.
  * Dewangan et al. (2021) Seema Dewangan, Rajwant Singh Rao, Alok Mishra, and Manjari Gupta. 2021.  A Novel Approach for Code Smell Detection: An Empirical Study.  _IEEE Access_ 9 (2021), 162869–162883.
  * Dhamayanthi and Lavanya (2019) N. Dhamayanthi and B. Lavanya. 2019.  Improvement in Software Defect Prediction Outcome Using Principal Component Analysis and Ensemble Machine Learning Algorithms. In _International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI) 2018_ , Jude Hemanth, Xavier Fernando, Pavel Lafata, and Zubair Baig (Eds.). 397–406.
  * Di Martino et al. (2011) Sergio Di Martino, Filomena Ferrucci, Carmine Gravino, and Federica Sarro. 2011.  A Genetic Algorithm to Configure Support Vector Machines for Predicting Fault-Prone Components. In _Product-Focused Software Process Improvement_ , Danilo Caivano, Markku Oivo, Maria Teresa Baldassarre, and Giuseppe Visaggio (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 247–261.
  * Di Nucci et al. (2018) D. Di Nucci, F. Palomba, D. A. Tamburri, A. Serebrenik, and A. De Lucia. 2018\.  Detecting code smells using machine learning techniques: Are we there yet?. In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 612–621.  https://doi.org/10.1109/SANER.2018.8330266
  * Dos Santos et al. (2020) Geanderson Esteves Dos Santos, E. Figueiredo, Adriano Veloso, Markos Viggiato, and N. Ziviani. 2020\.  Understanding machine learning software defect predictions.  _Autom. Softw. Eng._ 27 (2020), 369–392.
  * Du et al. (2019) Xiaoning Du, Bihuan Chen, Yuekang Li, Jianmin Guo, Yaqin Zhou, Yang Liu, and Yu Jiang. 2019.  LEOPARD: Identifying Vulnerable Code for Vulnerability Assessment Through Program Metrics. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. 60–71.  https://doi.org/10.1109/ICSE.2019.00024
  * Du et al. (2015) Yao Du, Xiaoqing Wang, and Junfeng Wang. 2015.  A Static Android Malicious Code Detection Method Based on Multi-Source Fusion.  _Sec. and Commun. Netw._ 8, 17 (nov 2015), 3238–3246.  https://doi.org/10.1002/sec.1248
  * Durelli et al. (2019) V. H. S. Durelli, R. S. Durelli, S. S. Borges, A. T. Endo, M. M. Eler, D. R. C. Dias, and M. P. Guimarães. 2019.  Machine Learning Applied to Software Testing: A Systematic Mapping Study.  _IEEE Transactions on Reliability_ 68, 3 (2019), 1189–1212.  https://doi.org/10.1109/TR.2019.2892517
  * Dwivedi et al. (2016) Ashish Kumar Dwivedi, Anand Tirkey, Ransingh Biswajit Ray, and Santanu Kumar Rath. 2016.  Software design pattern recognition using machine learning techniques. In _2016 ieee region 10 conference (tencon)_. IEEE, 222–227.
  * Efstathiou and Spinellis (2019) Vasiliki Efstathiou and Diomidis Spinellis. 2019.  Semantic Source Code Models Using Identifier Embeddings. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. 29–33.  https://doi.org/10.1109/MSR.2019.00015
  * Elovici et al. (2007) Yuval Elovici, Asaf Shabtai, Robert Moskovitch, Gil Tahan, and Chanan Glezer. 2007.  Applying Machine Learning Techniques for Detection of Malicious Code in Network Traffic. In _KI 2007: Advances in Artificial Intelligence_ , Joachim Hertzberg, Michael Beetz, and Roman Englert (Eds.). 44–50.
  * Eniser et al. (2019) Hasan Ferit Eniser, Simos Gerasimou, and Alper Sen. 2019\.  DeepFault: Fault Localization for Deep Neural Networks. In _Fundamental Approaches to Software Engineering_ , Reiner Hähnle and Wil van der Aalst (Eds.). Springer International Publishing, Cham, 171–191.
  * Erturk and Sezer (2015) Ezgi Erturk and Ebru Akcapinar Sezer. 2015.  A comparison of some soft computing methods for software fault prediction.  _Expert systems with applications_ 42, 4 (2015), 1872–1879.
  * Etemadi and Monperrus (2020) Khashayar Etemadi and Martin Monperrus. 2020.  On the Relevance of Cross-project Learning with Nearest Neighbours for Commit Message Generation. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_. 470–475.
  * Fakhoury et al. (2018) S. Fakhoury, V. Arnaoudova, C. Noiseux, F. Khomh, and G. Antoniol. 2018.  Keep it simple: Is deep learning good for linguistic smell detection?. In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 602–611.  https://doi.org/10.1109/SANER.2018.8330265
  * Falleri et al. (2014) Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014\.  Fine-Grained and Accurate Source Code Differencing. In _Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering_ (Vasteras, Sweden) _(ASE ’14)_. 313–324.  https://doi.org/10.1145/2642937.2642982
  * Fan et al. (2019) Guisheng Fan, Xuyang Diao, Huiqun Yu, Kang Yang, and Liqiong Chen. 2019.  Deep semantic feature learning with embedded static metrics for software defect prediction. In _2019 26th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 244–251.
  * Fang et al. (2020b) Chunrong Fang, Zixi Liu, Yangyang Shi, Jeff Huang, and Qingkai Shi. 2020b.  Functional Code Clone Detection with Syntax and Semantics Fusion Learning. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual Event, USA) _(ISSTA 2020)_. 516–527.  https://doi.org/10.1145/3395363.3397362
  * Fang et al. (2020a) Yong Fang, Yongcheng Liu, Cheng Huang, and Liang Liu. 2020a.  FastEmbed: Predicting vulnerability exploitation possibility based on ensemble machine learning algorithm.  _PLoS ONE_ 15 (Feb. 2020), e0228439.  https://doi.org/10.1371/journal.pone.0228439 ADS Bibcode: 2020PLoSO..1528439F.
  * Felix and Lee (2017) Ebubeogu Amarachukwu Felix and Sai Peck Lee. 2017.  Integrated approach to software defect prediction.  _IEEE Access_ 5 (2017), 21524–21547.
  * Ferenc et al. (2019) Rudolf Ferenc, Péter Hegedundefineds, Péter Gyimesi, Gábor Antal, Dénes Bán, and Tibor Gyimóthy. 2019.  Challenging Machine Learning Algorithms in Predicting Vulnerable JavaScript Functions. In _Proceedings of the 7th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering_ (Montreal, Quebec, Canada) _(RAISE ’19)_. 8–14.  https://doi.org/10.1109/RAISE.2019.00010
  * Ferreira et al. (2021) Fabio Ferreira, Luciana Lourdes Silva, and Marco Tulio Valente. 2021.  Software Engineering Meets Deep Learning: A Mapping Study. In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_ (Virtual Event, Republic of Korea) _(SAC ’21)_. Association for Computing Machinery, New York, NY, USA, 1542–1549.  https://doi.org/10.1145/3412841.3442029
  * Fontana et al. (2015) F. Fontana, M. Mäntylä, Marco Zanoni, and Alessandro Marino. 2015.  Comparing and experimenting machine learning techniques for code smell detection.  _Empirical Software Engineering_ 21 (2015), 1143–1191.
  * Fontana et al. (2013) F. A. Fontana, M. Zanoni, A. Marino, and M. V. Mäntylä. 2013.  Code Smell Detection: Towards a Machine Learning-Based Approach. In _2013 IEEE International Conference on Software Maintenance_. 396–399.
  * Gamma et al. (1994) Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994.  _Design Patterns: Elements of Reusable Object-Oriented Software_ (1st ed.).  Addison-Wesley Professional. Part of the Addison-Wesley Professional Computing Series series.  https://www.informit.com/store/design-patterns-elements-of-reusable-object-oriented-9780201633610?w_ptgrevartcl=Grady+Booch+on+Design+Patterns%2c+OOP%2c+and+Coffee_1405569
  * Gao et al. (2020) Zhipeng Gao, Xin Xia, John Grundy, David Lo, and Yuan-Fang Li. 2020.  Generating Question Titles for Stack Overflow from Mined Code Snippets.  _ACM Trans. Softw. Eng. Methodol._ 29, 4, Article 26 (September 2020), 37 pages.  https://doi.org/10.1145/3401026
  * Ghadhab et al. (2021) Lobna Ghadhab, Ilyes Jenhani, Mohamed Wiem Mkaouer, and Montassar Ben Messaoud. 2021.  Augmenting commit classification by using fine-grained source code changes and a pre-trained deep neural language model.  _Information and Software Technology_ 135 (2021), 106566.
  * Ghaffarian and Shahriari (2017) Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017.  Software Vulnerability Analysis and Discovery Using Machine-Learning and Data-Mining Techniques: A Survey.  _ACM Comput. Surv._ 50, 4, Article 56 (August 2017), 36 pages.  https://doi.org/10.1145/3092566
  * Gharbi et al. (2019) Sirine Gharbi, Mohamed Wiem Mkaouer, Ilyes Jenhani, and Montassar Ben Messaoud. 2019\.  On the classification of software change messages using multi-label active learning. In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_. 1760–1767.
  * Giray (2021) Görkem Giray. 2021\.  A software engineering perspective on engineering machine learning systems: State of the art and challenges.  _Journal of Systems and Software_ 180 (2021), 111031.  https://doi.org/10.1016/j.jss.2021.111031
  * Godefroid et al. (2017) P. Godefroid, H. Peleg, and R. Singh. 2017\.  Learn Fuzz: Machine learning for input fuzzing. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 50–59.  https://doi.org/10.1109/ASE.2017.8115618
  * Gondra (2008) Iker Gondra. 2008\.  Applying machine learning to software fault-proneness prediction.  _Journal of Systems and Software_ 81, 2 (2008), 186 – 195.  https://doi.org/10.1016/j.jss.2007.05.035 Model-Based Software Testing.
  * Gopalakrishnan et al. (2017) R. Gopalakrishnan, P. Sharma, M. Mirakhorli, and M. Galster. 2017.  Can Latent Topics in Source Code Predict Missing Architectural Tactics?. In _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_. 15–26.  https://doi.org/10.1109/ICSE.2017.10
  * Gopinath et al. (2014) Divya Gopinath, Sarfraz Khurshid, Diptikalyan Saha, and Satish Chandra. 2014.  Data-Guided Repair of Selection Statements. In _Proceedings of the 36th International Conference on Software Engineering_ (Hyderabad, India) _(ICSE 2014)_. 243–253.  https://doi.org/10.1145/2568225.2568303
  * Gopinath et al. (2016) D. Gopinath, K. Wang, J. Hua, and S. Khurshid. 2016\.  Repairing Intricate Faults in Code Using Machine Learning and Path Exploration. In _2016 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 453–457.  https://doi.org/10.1109/ICSME.2016.75
  * Goues et al. (2019) Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019\.  Automated Program Repair.  _Commun. ACM_ 62, 12 (November 2019), 56–65.  https://doi.org/10.1145/3318162
  * Gousios (2013) Georgios Gousios. 2013\.  The GHTorrent dataset and tool suite. In _Proceedings of the 10th Working Conference on Mining Software Repositories_ _(MSR ’13)_. IEEE Press, Piscataway, NJ, USA, 233–236.  http://dl.acm.org/citation.cfm?id=2487085.2487132
  * Grano et al. (2018) G. Grano, T. V. Titov, S. Panichella, and H. C. Gall. 2018.  How high will it be? Using machine learning models to predict branch coverage in automated testing. In _2018 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE)_. 19–24.  https://doi.org/10.1109/MALTESQUE.2018.8368454
  * Graves et al. (2013) Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013\.  Hybrid speech recognition with deep bidirectional LSTM. In _Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on_. IEEE, 273–278.
  * Greff et al. (2017) Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. 2017\.  LSTM: A search space odyssey.  _IEEE transactions on neural networks and learning systems_ 28, 10 (2017), 2222–2232.
  * Grodzicka et al. (2020) Hanna Grodzicka, Arkadiusz Ziobrowski, Zofia Łakomiak, Michał Kawa, and Lech Madeyski. 2020\.  _Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs_.  137–167.  https://doi.org/10.1007/978-3-030-34706-2_8
  * Gu et al. (2018) Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018\.  Deep Code Search. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. 933–944.  https://doi.org/10.1145/3180155.3180167
  * Guggulothu and Moiz (2020) Thirupathi Guggulothu and S. A. Moiz. 2020.  Code smell detection using multi-label classification approach.  _Software Quality Journal_ (2020), 1–24.
  * Gupta et al. (2021d) Aakanshi Gupta, Bharti Suri, Vijay Kumar, and Pragyashree Jain. 2021d.  Extracting rules for vulnerabilities detection with static metrics using machine learning.  _International Journal of System Assurance Engineering and Management_ 12 (2021), 65–76.
  * Gupta et al. (2021c) Aakanshi Gupta, Bharti Suri, and Lakshay Lamba. 2021c.  Tracing Bad Code Smells Behavior Using Machine Learning with Software Metrics.  _Smart and Sustainable Intelligent Systems_ (2021), 245–257.
  * Gupta et al. (2021a) Himanshu Gupta, Abhiram Anand Gulanikar, Lov Kumar, and Lalita Bhanu Murthy Neti. 2021a.  Empirical Analysis on Effectiveness of NLP Methods for Predicting Code Smell. In _International Conference on Computational Science and Its Applications_. Springer, 43–53.
  * Gupta et al. (2021b) Himanshu Gupta, Tanmay Girish Kulkarni, Lov Kumar, Lalita Bhanu Murthy Neti, and Aneesh Krishna. 2021b.  An empirical study on predictability of software code smell using deep learning models. In _International Conference on Advanced Information Networking and Applications_. Springer, 120–132.
  * Gupta et al. (2019) H. Gupta, L. Kumar, and L. B. M. Neti. 2019.  An Empirical Framework for Code Smell Prediction using Extreme Learning Machine*. In _2019 9th Annual Information Technology, Electromechanical Engineering and Microelectronics Conference (IEMECON)_. 189–195.  https://doi.org/10.1109/IEMECONX.2019.8877082
  * Gupta et al. (2019) Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019\.  Deep Reinforcement Learning for Syntactic Error Repair in Student Programs.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33 (07 2019), 930–937.  https://doi.org/10.1609/aaai.v33i01.3301930
  * Gupta et al. (2017) Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017\.  DeepFix: Fixing Common C Language Errors by Deep Learning.. In _AAAI_. 1345–1351.
  * Hadj-Kacem and Bouassida (2018) Mouna Hadj-Kacem and Nadia Bouassida. 2018.  A Hybrid Approach To Detect Code Smells using Deep Learning.. In _ENASE_. 137–146.
  * Hadj-Kacem and Bouassida (2019) Mouna Hadj-Kacem and Nadia Bouassida. 2019.  Deep representation learning for code smells detection using variational auto-encoder. In _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Hall and Bowes (2012) T. Hall and D. Bowes. 2012.  The State of Machine Learning Methodology in Software Fault Prediction. In _2012 11th International Conference on Machine Learning and Applications_ , Vol. 2. 308–313.  https://doi.org/10.1109/ICMLA.2012.226
  * Halstead (1977) Maurice H. Halstead. 1977\.  _Elements of Software Science (Operating and Programming Systems Series)_.  USA.
  * Hammad et al. (2021) Muhammad Hammad, ”̈Onder Babur, Hamid Abdul Basit, and Mark van den Brand. 2021.  Clone-advisor: recommending code tokens and clone methods with deep learning and information retrieval.  _PeerJ Computer Science_ 7 (2021), e737.
  * Hammouri et al. (2018) Awni Hammouri, Mustafa Hammad, Mohammad Alnabhan, and Fatima Alsarayrah. 2018.  Software Bug Prediction using Machine Learning Approach.  _International Journal of Advanced Computer Science and Applications_ 9 (01 2018).  https://doi.org/10.14569/IJACSA.2018.090212
  * Han et al. (2009) S. Han, D. R. Wallace, and R. C. Miller. 2009\.  Code Completion from Abbreviated Input. In _2009 IEEE/ACM International Conference on Automated Software Engineering_. 332–343.  https://doi.org/10.1109/ASE.2009.64
  * Han et al. (2011) Sangmok Han, David R. Wallace, and Robert C. Miller. 2011\.  Code Completion of Multiple Keywords from Abbreviated Input.  _Automated Software Engg._ 18, 3–4 (December 2011), 363–398.  https://doi.org/10.1007/s10515-011-0083-2
  * Hanif et al. (2021) Hazim Hanif, Mohd Hairul Nizam Md Nasir, Mohd Faizal Ab Razak, Ahmad Firdaus, and Nor Badrul Anuar. 2021\.  The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches.  _Journal of Network and Computer Applications_ 179 (April 2021), 103009.  https://doi.org/10.1016/j.jnca.2021.103009
  * Haque et al. (2021) Sakib Haque, Aakash Bansal, Lingfei Wu, and Collin McMillan. 2021.  Action word prediction for neural source code summarization. In _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 330–341.
  * Haque et al. (2020) Sakib Haque, Alexander LeClair, Lingfei Wu, and Collin McMillan. 2020.  Improved automatic summarization of subroutines via attention to file context. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 300–310.
  * Harman et al. (2014) Mark Harman, Syed Islam, Yue Jia, Leandro L. Minku, Federica Sarro, and Komsan Srivisut. 2014\.  Less is More: Temporal Fault Predictive Performance over Multiple Hadoop Releases. In _Search-Based Software Engineering_ , Claire Le Goues and Shin Yoo (Eds.). Springer International Publishing, Cham, 240–246.
  * Hellendoorn et al. (2018) Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. 2018.  Deep Learning Type Inference _(ESEC/FSE 2018)_. 152–162.  https://doi.org/10.1145/3236024.3236051
  * Hellendoorn and Devanbu (2017) Vincent J. Hellendoorn and Premkumar Devanbu. 2017.  Are Deep Neural Networks the Best Choice for Modeling Source Code?. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ (Paderborn, Germany) _(ESEC/FSE 2017)_. 763–773.  https://doi.org/10.1145/3106237.3106290
  * Heo et al. (2017) Kihong Heo, Hakjoo Oh, and Kwangkeun Yi. 2017.  Machine-Learning-Guided Selectively Unsound Static Analysis. In _Proceedings of the 39th International Conference on Software Engineering_ (Buenos Aires, Argentina) _(ICSE ’17)_. 519–529.  https://doi.org/10.1109/ICSE.2017.54
  * Hoang et al. (2020) Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall. 2020\.  CC2Vec: Distributed Representations of Code Changes. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. 518–529.  https://doi.org/10.1145/3377811.3380361
  * Hort et al. (2021) Max Hort, Maria Kechagia, Federica Sarro, and Mark Harman. 2021\.  A Survey of Performance Optimization for Mobile Applications.  _IEEE Transactions on Software Engineering (TSE)_ (2021).
  * Hou et al. (2010) Yung-Tsung Hou, Yimeng Chang, Tsuhan Chen, Chi-Sung Laih, and Chia-Mei Chen. 2010.  Malicious web content detection by machine learning.  _Expert Systems with Applications_ 37, 1 (2010), 55 – 60.  https://doi.org/10.1016/j.eswa.2009.05.023
  * Hu et al. (2018b) Gang Hu, Linjie Zhu, and Junfeng Yang. 2018b.  AppFlow: Using Machine Learning to Synthesize Robust, Reusable UI Tests. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Lake Buena Vista, FL, USA) _(ESEC/FSE 2018)_. 269–282.  https://doi.org/10.1145/3236024.3236055
  * Hu et al. (2018) X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin. 2018.  Deep Code Comment Generation. In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. 200–20010.
  * Hu et al. (2018a) Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018a.  Summarizing Source Code with Transferred API Knowledge. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_. International Joint Conferences on Artificial Intelligence Organization, 2269–2275.  https://doi.org/10.24963/ijcai.2018/314
  * Huang et al. (2020a) Yuan Huang, Xinyu Hu, Nan Jia, Xiangping Chen, Zibin Zheng, and Xiapu Luo. 2020a.  CommtPst: Deep learning source code for commenting positions prediction.  _Journal of Systems and Software_ 170 (2020), 110754.  https://doi.org/10.1016/j.jss.2020.110754
  * Huang et al. (2020b) Yuan Huang, Shaohao Huang, Huanchao Chen, Xiangping Chen, Zibin Zheng, Xiapu Luo, Nan Jia, Xinyu Hu, and Xiaocong Zhou. 2020b.  Towards automatically generating block comments for code snippets.  _Information and Software Technology_ 127 (2020), 106373.
  * Hussain et al. (2020) Yasir Hussain, Zhiqiu Huang, Yu Zhou, and Senzhang Wang. 2020\.  CodeGRU: Context-aware deep learning with gated recurrent unit for source code modeling.  _Information and Software Technology_ 125 (2020), 106309.  https://doi.org/10.1016/j.infsof.2020.106309
  * Ivers et al. (2019) J. Ivers, I. Ozkaya, and R. L. Nord. 2019.  Can AI Close the Design-Code Abstraction Gap?. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW)_. 122–125.  https://doi.org/10.1109/ASEW.2019.00041
  * Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016.  Summarizing Source Code using a Neural Attention Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2073–2083.  https://doi.org/10.18653/v1/P16-1195
  * Jain and Saha (2021) Shivani Jain and Anju Saha. 2021.  Improving performance with hybrid feature selection and ensemble machine learning techniques for code smell detection.  _Science of Computer Programming_ 212 (2021), 102713.
  * Ji et al. (2018) T. Ji, J. Pan, L. Chen, and X. Mao. 2018\.  Identifying Supplementary Bug-fix Commits. In _2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)_ , Vol. 01. 184–193.  https://doi.org/10.1109/COMPSAC.2018.00031
  * Jiang et al. (2018) Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018.  Shaping Program Repair Space with Existing Patches and Similar Code. In _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Amsterdam, Netherlands) _(ISSTA 2018)_. 298–309.  https://doi.org/10.1145/3213846.3213871
  * Jiang et al. (2019) Lin Jiang, Hui Liu, and He Jiang. 2019.  Machine Learning Based Recommendation of Method Names: How Far Are We. In _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering_ (San Diego, California) _(ASE ’19)_. 602–614.  https://doi.org/10.1109/ASE.2019.00062
  * Jiang et al. (2021) Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021\.  CURE: Code-aware neural machine translation for automatic program repair. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 1161–1173.
  * Jiang (2019) Shuyao Jiang. 2019\.  Boosting neural commit message generation with code semantic analysis. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 1280–1282.
  * Jiang et al. (2017) S. Jiang, A. Armaly, and C. McMillan. 2017.  Automatically generating commit messages from diffs using neural machine translation. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 135–146.  https://doi.org/10.1109/ASE.2017.8115626
  * Jiang and McMillan (2017) Siyuan Jiang and Collin McMillan. 2017.  Towards automatic generation of short summaries of commits. In _2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC)_. IEEE, 320–323.
  * Jie et al. (2016) Gong Jie, Kuang Xiao-Hui, and Liu Qiang. 2016.  Survey on Software Vulnerability Analysis Method Based on Machine Learning. In _2016 IEEE First International Conference on Data Science in Cyberspace (DSC)_. 642–647.  https://doi.org/10.1109/DSC.2016.33
  * Jimenez et al. (2019) Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman. 2019.  The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities. In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Tallinn, Estonia) _(ESEC/FSE 2019)_. Association for Computing Machinery, New York, NY, USA, 695–705.  https://doi.org/10.1145/3338906.3338941
  * Jing et al. (2014) Xiao-Yuan Jing, Shi Ying, Zhi-Wu Zhang, Shan-Shan Wu, and Jin Liu. 2014.  Dictionary learning based software defect prediction. In _Proceedings of the 36th international conference on software engineering_. 414–423.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst. 2014\.  Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ (San Jose, CA, USA) _(ISSTA 2014)_. Association for Computing Machinery, New York, NY, USA, 437–440.  https://doi.org/10.1145/2610384.2628055
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Learning and Evaluating Contextual Embedding of Source Code. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, 5110–5121.  https://proceedings.mlr.press/v119/kanade20a.html
  * Kang et al. (2019) Hong Jin Kang, Tegawendé F. Bissyandé, and David Lo. 2019.  Assessing the Generalizability of Code2vec Token Embeddings. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 1–12.  https://doi.org/10.1109/ASE.2019.00011
  * Karampatsis et al. (2020) Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. 2020\.  Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. 1073–1085.  https://doi.org/10.1145/3377811.3380342
  * Kaur et al. (2017) A. Kaur, S. Jain, and S. Goel. 2017.  A Support Vector Machine Based Approach for Code Smell Detection. In _2017 International Conference on Machine Learning and Data Science (MLDS)_. 9–14.  https://doi.org/10.1109/MLDS.2017.8
  * Kaur and Kaur (2015) Arvinder Kaur and Kamaldeep Kaur. 2015.  An empirical study of robustness and stability of machine learning classifiers in software defect prediction.  In _Advances in intelligent informatics_. Springer, 383–397.
  * Kaur et al. (2017) Arvinder Kaur, Kamaldeep Kaur, and Deepti Chopra. 2017\.  An empirical study of software entropy based bug prediction using machine learning.  _International Journal of System Assurance Engineering and Management_ 8, 2 (November 2017), 599–616.  https://doi.org/10.1007/s13198-016-0479-2
  * Kaur and Kaur (2021) Inderpreet Kaur and Arvinder Kaur. 2021.  A novel four-way approach designed with ensemble feature selection for code smell detection.  _IEEE Access_ 9 (2021), 8695–8707.
  * Khalid et al. (2019) Muhammad Noman Khalid, Humera Farooq, Muhammad Iqbal, Muhammad Talha Alam, and Kamran Rasheed. 2019.  Predicting Web Vulnerabilities in Web Applications Based on Machine Learning. In _Intelligent Technologies and Applications_ _(Communications in Computer and Information Science)_ , Imran Sarwar Bajwa, Fairouz Kamareddine, and Anna Costa (Eds.). Springer, Singapore, 473–484.  https://doi.org/10.1007/978-981-13-6052-7_41
  * Khan et al. (2020) Bilal Khan, Danish Iqbal, and Sher Badshah. 2020.  Cross-Project Software Fault Prediction Using Data Leveraging Technique to Improve Software Quality. In _Proceedings of the Evaluation and Assessment in Software Engineering_ (Trondheim, Norway) _(EASE ’20)_. 434–438.  https://doi.org/10.1145/3383219.3383281
  * Kim et al. (2019) Junae Kim, David Hubczenko, and Paul Montague. 2019\.  Towards Attention Based Vulnerability Discovery Using Source Code Representation. In _Artificial Neural Networks and Machine Learning – ICANN 2019: Text and Time Series_ , Igor V. Tetko, Věra Kůrková, Pavel Karpov, and Fabian Theis (Eds.). 731–746.
  * Kim et al. (2018) J. Kim, M. Kwon, and S. Yoo. 2018.  Generating Test Input with Deep Reinforcement Learning. In _2018 IEEE/ACM 11th International Workshop on Search-Based Software Testing (SBST)_. 51–58.
  * Kim et al. (2018) Sangwoo Kim, Seokmyung Hong, Jaesang Oh, and Heejo Lee. 2018\.  Obfuscated VBA Macro Detection Using Machine Learning. In _2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)_. 490–501.  https://doi.org/10.1109/DSN.2018.00057
  * Kim et al. (2009) Y. Kim, C. Jeong, A. Jeong, and H. S. Kim. 2009\.  Risky Module Estimation in Safety-Critical Software. In _2009 Eighth IEEE/ACIS International Conference on Computer and Information Science_. 967–970.  https://doi.org/10.1109/ICIS.2009.83
  * Knab et al. (2006) Patrick Knab, Martin Pinzger, and Abraham Bernstein. 2006\.  Predicting Defect Densities in Source Code Files with Decision Tree Learners. In _Proceedings of the 2006 International Workshop on Mining Software Repositories_ (Shanghai, China) _(MSR ’06)_. 119–125.  https://doi.org/10.1145/1137983.1138012
  * Kosker et al. (2009) Yasemin Kosker, Burak Turhan, and Ayse Bener. 2009\.  An expert system for determining candidate software classes for refactoring.  _Expert Systems with Applications_ 36, 6 (2009), 10000 – 10003.  https://doi.org/10.1016/j.eswa.2008.12.066
  * Kovalenko et al. (2020) Vladimir Kovalenko, Egor Bogomolov, Timofey Bryksin, and Alberto Bacchelli. 2020.  Building Implicit Vector Representations of Individual Coding Style. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ (Seoul, Republic of Korea) _(ICSEW’20)_. 117–124.  https://doi.org/10.1145/3387940.3391494
  * Krasniqi and Cleland-Huang (2020) Rrezarta Krasniqi and Jane Cleland-Huang. 2020.  Enhancing Source Code Refactoring Detection with Explanations from Commit Messages. In _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 512–516.  https://doi.org/10.1109/SANER48275.2020.770_Krasniqi2020
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  ImageNet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * Kronjee et al. (2018) Jorrit Kronjee, Arjen Hommersom, and Harald Vranken. 2018\.  Discovering Software Vulnerabilities Using Data-Flow Analysis and Machine Learning. In _Proceedings of the 13th International Conference on Availability, Reliability and Security_ (Hamburg, Germany) _(ARES 2018)_. Article 6, 10 pages.  https://doi.org/10.1145/3230833.3230856
  * Kumar et al. (2017) Lov Kumar, Santanu Kumar Rath, and Ashish Sureka. 2017\.  Using source code metrics to predict change-prone web services: A case-study on ebay services. In _2017 IEEE workshop on machine learning techniques for software quality evaluation (MaLTeSQuE)_. IEEE, 1–7.
  * Kumar et al. (2019) Lov Kumar, Shashank Mouli Satapathy, and Lalita Bhanu Murthy. 2019\.  Method Level Refactoring Prediction on Five Open Source Java Projects Using Machine Learning Techniques. In _Proceedings of the 12th Innovations on Software Engineering Conference (Formerly Known as India Software Engineering Conference)_ (Pune, India) _(ISEC’19)_. Article 7, 10 pages.  https://doi.org/10.1145/3299771.3299777
  * Kumar and Sureka (2017) L. Kumar and A. Sureka. 2017.  Application of LSSVM and SMOTE on Seven Open Source Projects for Predicting Refactoring at Class Level. In _2017 24th Asia-Pacific Software Engineering Conference (APSEC)_. 90–99.  https://doi.org/10.1109/APSEC.2017.15
  * Kumar and Sureka (2018) L. Kumar and A. Sureka. 2018.  An Empirical Analysis on Web Service Anti-pattern Detection Using a Machine Learning Framework. In _2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)_ , Vol. 01. 2–11.  https://doi.org/10.1109/COMPSAC.2018.00010
  * Kumar and Singh (2012) Pradeep Kumar and Yogesh Singh. 2012.  Assessment of Software Testing Time Using Soft Computing Techniques.  _SIGSOFT Softw. Eng. Notes_ 37, 1 (January 2012), 1–6.  https://doi.org/10.1145/2088883.2088895
  * Kurbatova et al. (2020) Zarina Kurbatova, Ivan Veselov, Yaroslav Golubev, and Timofey Bryksin. 2020.  Recommendation of Move Method Refactoring Using Path-Based Representation of Code. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ (Seoul, Republic of Korea) _(ICSEW’20)_. 315–322.  https://doi.org/10.1145/3387940.3392191
  * Lal and Pahwa (2017) H. Lal and G. Pahwa. 2017.  Code review analysis of software system using machine learning techniques. In _2017 11th International Conference on Intelligent Systems and Control (ISCO)_. 8–13.  https://doi.org/10.1109/ISCO.2017.7855962
  * Lal and Pahwa (2017) Harsh Lal and Gaurav Pahwa. 2017.  Root cause analysis of software bugs using machine learning techniques. In _2017 7th International Conference on Cloud Computing, Data Science & Engineering-Confluence_. IEEE, 105–111.
  * Laradji et al. (2015) Issam H Laradji, Mohammad Alshayeb, and Lahouari Ghouti. 2015\.  Software defect prediction using ensemble learning on selected features.  _Information and Software Technology_ 58 (2015), 388–402.
  * Law and Grépin (2010) Michael R. Law and Karen A. Grépin. 2010.  Is newer always better? Re-evaluating the benefits of newer pharmaceuticals.  _Journal of Health Economics_ 29, 5 (Sept. 2010), 743–750.  https://doi.org/10.1016/j.jhealeco.2010.06.007
  * Le et al. (2020) Triet H. M. Le, Hao Chen, and Muhammad Ali Babar. 2020.  Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges.  _ACM Comput. Surv._ 53, 3, Article 62 (June 2020), 38 pages.  https://doi.org/10.1145/3383458
  * Le et al. (2015) X. D. Le, T. B. Le, and D. Lo. 2015.  Should fixing these failures be delegated to automated program repair?. In _2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)_. 427–437.  https://doi.org/10.1109/ISSRE.2015.7381836
  * Le Goues et al. (2015) Claire Le Goues, Neal Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. 2015\.  The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs.  _IEEE Transactions on Software Engineering_ 41, 12 (2015), 1236–1256.  https://doi.org/10.1109/TSE.2015.2454513
  * LeClair et al. (2021) Alexander LeClair, Aakash Bansal, and Collin McMillan. 2021\.  Ensemble Models for Neural Source Code Summarization of Subroutines. In _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 286–297.
  * LeClair et al. (2020) Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020.  Improved Code Summarization via a Graph Neural Network. In _Proceedings of the 28th International Conference on Program Comprehension_ (Seoul, Republic of Korea) _(ICPC ’20)_. 184–195.  https://doi.org/10.1145/3387904.3389268
  * LeClair et al. (2019) Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019\.  A Neural Model for Generating Natural Language Summaries of Program Subroutines. In _Proceedings of the 41st International Conference on Software Engineering_ (Montreal, Quebec, Canada) _(ICSE ’19)_. 795–806.  https://doi.org/10.1109/ICSE.2019.00087
  * LeClair and McMillan (2019) Alexander LeClair and Collin McMillan. 2019.  Recommendations for datasets for source code summarization.
  * Lee et al. (2021) Suin Lee, Youngseok Lee, Chan-Gun Lee, and Honguk Woo. 2021\.  Deep Learning-Based Logging Recommendation Using Merged Code Representation. In _IT Convergence and Security_ , Hyuncheol Kim and Kuinam J. Kim (Eds.). 49–53.
  * Lee et al. (2017) Song-Mi Lee, Sang Min Yoon, and Heeryon Cho. 2017.  Human activity recognition from accelerometer data using Convolutional Neural Network. In _Big Data and Smart Computing (BigComp), 2017 IEEE International Conference on_. IEEE, 131–134.
  * Levin and Yehudai (2017) Stanislav Levin and Amiram Yehudai. 2017.  Boosting automatic commit classification into maintenance activities by utilizing source code changes. In _Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering_. 97–106.
  * Lewowski and Madeyski (2022) Tomasz Lewowski and Lech Madeyski. 2022.  Code smells detection using artificial intelligence techniques: A business-driven systematic review.  _Developments in Information I & Knowledge Management for Business Applications_ (2022), 285–319.
  * Li et al. (2020b) Boao Li, Meng Yan, Xin Xia, Xing Hu, Ge Li, and David Lo. 2020b.  DeepCommenter: A Deep Code Comment Generation Tool with Hybrid Lexical and Syntactical Information. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. 1571–1575.  https://doi.org/10.1145/3368089.3417926
  * Li et al. (2019a) Daoyuan Li, Li Li, Dongsun Kim, Tegawendé F Bissyandé, David Lo, and Yves Le Traon. 2019a.  Watch out for this commit! a study of influential software changes.  _Journal of Software: Evolution and Process_ 31, 12 (2019), e2181.
  * Li et al. (2017) Jian Li, Pinjia He, Jieming Zhu, and Michael R Lyu. 2017\.  Software defect prediction via convolutional neural network. In _2017 IEEE International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 318–328.
  * Li et al. (2021a) Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021a.  EditSum: A Retrieve-and-Edit Framework for Source Code Summarization. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 155–166.
  * Li et al. (2018) Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018\.  Code Completion with Neural Attention and Pointer Networks. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ (Stockholm, Sweden) _(IJCAI’18)_. 4159–25.
  * Li et al. (2011) M. Li, H. Zhang, Rongxin Wu, and Z. Zhou. 2011\.  Sample-based software defect prediction with active and semi-supervised learning.  _Automated Software Engineering_ 19 (2011), 201–230.
  * Li et al. (2015) Yuancheng Li, Rong Ma, and Runhai Jiao. 2015.  A Hybrid Malicious Code Detection Method based on Deep Learning.  _International journal of security and its applications_ 9 (2015), 205–216.
  * Li et al. (2020a) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020a.  DLFix: Context-Based Code Transformation Learning for Automated Program Repair. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. 602–614.  https://doi.org/10.1145/3377811.3380345
  * Li et al. (2021b) Yi Li, Shaohua Wang, and Tien N Nguyen. 2021b.  A Context-based Automated Approach for Method Name Consistency Checking and Suggestion. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 574–586.
  * Li et al. (2019b) Yi Li, Shaohua Wang, Tien N. Nguyen, and Son Van Nguyen. 2019b.  Improving Bug Detection via Context-Based Code Representation Learning and Attention-Based Neural Networks.  _Proc. ACM Program. Lang._ 3, OOPSLA, Article 162 (October 2019), 30 pages.  https://doi.org/10.1145/3360588
  * Li et al. (2019) Z. Li, D. Zou, J. Tang, Z. Zhang, M. Sun, and H. Jin. 2019\.  A Comparative Study of Deep Learning-Based Vulnerability Detection System.  _IEEE Access_ 7 (2019), 103184–103197.  https://doi.org/10.1109/ACCESS.2019.2930578
  * Liang et al. (2019) Hongliang Liang, Yue Yu, Lin Jiang, and Zhuosi Xie. 2019\.  Seml: A semantic LSTM model for software defect prediction.  _IEEE Access_ 7 (2019), 83812–83824.
  * Lim (2018) H. Lim. 2018.  Applying Code Vectors for Presenting Software Features in Machine Learning. In _2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)_ , Vol. 01. 803–804.  https://doi.org/10.1109/COMPSAC.2018.00128
  * Lima et al. (2020) R. Lima, A. M. R. da Cruz, and J. Ribeiro. 2020\.  Artificial Intelligence Applied to Software Testing: A Literature Review. In _2020 15th Iberian Conference on Information Systems and Technologies (CISTI)_. 1–6.  https://doi.org/10.23919/CISTI49556.2020.9141124
  * LIN et al. (2021) BO LIN, SHANGWEN WANG, MING WEN, and XIAOGUANG MAO. 2021\.  Context-Aware Code Change Embedding for Better Patch Correctness Assessment.  _J. ACM_ 1, 1 (2021).
  * Lin et al. (2021) Chen Lin, Zhichao Ouyang, Junqing Zhuang, Jianqiang Chen, Hui Li, and Rongxin Wu. 2021\.  Improving code summarization with block-wise abstract syntax tree splitting. In _2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_. IEEE, 184–195.
  * Lin et al. (2020) Guanjun Lin, Wei Xiao, Jun Zhang, and Yang Xiang. 2020\.  Deep Learning-Based Vulnerable Function Detection: A Benchmark. In _Information and Communications Security_ _(Lecture Notes in Computer Science)_ , Jianying Zhou, Xiapu Luo, Qingni Shen, and Zhen Xu (Eds.). Springer International Publishing, Cham, 219–232.  https://doi.org/10.1007/978-3-030-41579-2_13
  * Lin et al. (2018a) Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Yang Xiang, Olivier De Vel, and Paul Montague. 2018a.  Cross-Project Transfer Representation Learning for Vulnerable Function Discovery.  _IEEE Transactions on Industrial Informatics_ 14, 7 (2018), 3289–3297.  https://doi.org/10.1109/TII.2018.2821768
  * Lin et al. (2018b) Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Yang Xiang, Olivier De Vel, and Paul Montague. 2018b.  Cross-Project Transfer Representation Learning for Vulnerable Function Discovery.  _IEEE Transactions on Industrial Informatics_ 14, 7 (2018), 3289–3297.  https://doi.org/10.1109/TII.2018.2821768
  * Lin and Lu (2021) Junhao Lin and Lu Lu. 2021\.  Semantic feature learning via dual sequences for defect prediction.  _IEEE Access_ 9 (2021), 13112–13124.
  * Ling et al. (2020) Chunyang Ling, Zeqi Lin, Yanzhen Zou, and Bing Xie. 2020\.  Adaptive Deep Code Search. In _Proceedings of the 28th International Conference on Program Comprehension_ _(ICPC ’20)_. Association for Computing Machinery, 48––59.  https://doi.org/10.1145/3387904.3389278
  * Linstead et al. (2008) E. Linstead, C. Lopes, and P. Baldi. 2008\.  An Application of Latent Dirichlet Allocation to Analyzing Software Evolution. In _2008 Seventh International Conference on Machine Learning and Applications_. 813–818.  https://doi.org/10.1109/ICMLA.2008.47
  * Liu et al. (2019c) Bohong Liu, Tao Wang, Xunhui Zhang, Qiang Fan, Gang Yin, and Jinsheng Deng. 2019c.  A Neural-Network Based Code Summarization Approach by Using Source Code and Its Call Dependencies. In _Proceedings of the 11th Asia-Pacific Symposium on Internetware_ (Fukuoka, Japan) _(Internetware ’19)_. Article 12, 10 pages.  https://doi.org/10.1145/3361242.3362774
  * Liu et al. (2020b) Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John Grundy, and Xiaohu Yang. 2020b.  On the Replicability and Reproducibility of Deep Learning in Software Engineering.  arXiv:2006.14244 [cs.SE]
  * Liu et al. (2013) Chen Liu, Jinqiu Yang, Lin Tan, and Munawar Hafiz. 2013\.  R2Fix: Automatically generating bug fixes from bug reports. In _2013 IEEE Sixth international conference on software testing, verification and validation_. IEEE, 282–291.
  * Liu et al. (2020c) Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. 2020c.  A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning. In _Proceedings of the 28th International Conference on Program Comprehension_ (Seoul, Republic of Korea) _(ICPC ’20)_. 37–47.  https://doi.org/10.1145/3387904.3389261
  * Liu et al. (2020) F. Liu, G. Li, Y. Zhao, and Z. Jin. 2020\.  Multi-task Learning based Pre-trained Language Model for Code Completion. In _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 473–485.
  * Liu et al. (2019a) Hui Liu, Jiahao Jin, Zhifeng Xu, Yifan Bu, Yanzhen Zou, and Lu Zhang. 2019a.  Deep learning based code smell detection.  _IEEE Transactions on Software Engineering_ (2019).
  * Liu et al. (2020) Kui Liu, Shangwen Wang, Anil Koyuncu, Kisub Kim, Tegawendé F. Bissyandé, Dongsun Kim, Peng Wu, Jacques Klein, Xiaoguang Mao, and Yves Le Traon. 2020\.  On the Efficiency of Test Suite Based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. 615?–627.  https://doi.org/10.1145/3377811.3380338
  * Liu et al. (2020a) Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, and Yang Liu. 2020a.  ATOM: Commit message generation based on abstract syntax tree and hybrid ranking.  _IEEE Transactions on Software Engineering_ (2020).
  * Liu et al. (2019b) Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. 2019b.  DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 01 (Jul. 2019), 1044–1051.  https://doi.org/10.1609/aaai.v33i01.33011044
  * Liu et al. (2018) Zhongxin Liu, Xin Xia, Ahmed E. Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018\.  Neural-Machine-Translation-Based Commit Message Generation: How Far Are We?. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE 2018)_. 373–384.  https://doi.org/10.1145/3238147.3238190
  * Liu et al. (2019d) Zhongxin Liu, Xin Xia, Christoph Treude, David Lo, and Shanping Li. 2019d.  Automatic generation of pull request descriptions. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 176–188.
  * Long and Rinard (2016) Fan Long and Martin Rinard. 2016.  Automatic Patch Generation by Learning Correct Code. In _Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages_ (St. Petersburg, FL, USA) _(POPL ’16)_. 298–312.  https://doi.org/10.1145/2837614.2837617
  * Lopes et al. (2010) C. Lopes, S. Bajracharya, J. Ossher, and P. Baldi. 2010\.  UCI Source Code Data Sets.  http://www.ics.uci.edu/$∼$lopes/datasets/
  * Lou et al. (2020) Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020.  Can automated program repair refine fault localization? a unified debugging approach. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 75–87.
  * Lu et al. (2017) Yangyang Lu, Zelong Zhao, Ge Li, and Zhi Jin. 2017\.  Learning to generate comments for api-based code snippets.  In _Software Engineering and Methodology for Emerging Domains_. Springer, 3–14.
  * Luiz et al. (2019) Frederico Caram Luiz, Bruno Rafael de Oliveira Rodrigues, and Fernando Silva Parreiras. 2019.  Machine Learning Techniques for Code Smells Detection: An Empirical Experiment on a Highly Imbalanced Setup. In _Proceedings of the XV Brazilian Symposium on Information Systems_ (Aracaju, Brazil) _(SBSI’19)_. Article 65, 8 pages.  https://doi.org/10.1145/3330204.3330275
  * Lujan et al. (2020) Savanna Lujan, Fabiano Pecorelli, Fabio Palomba, Andrea De Lucia, and Valentina Lenarduzzi. 2020\.  A Preliminary Study on the Adequacy of Static Analysis Warnings with Respect to Code Smell Prediction. In _Proceedings of the 4th ACM SIGSOFT International Workshop on Machine-Learning Techniques for Software-Quality Evaluation_ (Virtual, USA) _(MaLTeSQuE 2020)_. 1–6.  https://doi.org/10.1145/3416505.3423559
  * Luong et al. (2017) Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. 2017.  Neural Machine Translation (seq2seq) Tutorial.  _https://github.com/tensorflow/nmt_ (2017).
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  CoCoNuT: Combining Context-Aware Neural Translation Models Using Ensemble for Program Repair. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual Event, USA) _(ISSTA 2020)_. 101–114.  https://doi.org/10.1145/3395363.3397369
  * Lyu (1996) Michael R. Lyu (Ed.). 1996\.  _Handbook of Software Reliability Engineering_.  McGraw-Hill, Inc., USA.
  * Ma et al. (2018) Yuzhan Ma, Sarah Fakhoury, Michael Christensen, Venera Arnaoudova, Waleed Zogaan, and Mehdi Mirakhorli. 2018.  Automatic Classification of Software Artifacts in Open-Source Applications. In _Proceedings of the 15th International Conference on Mining Software Repositories_ (Gothenburg, Sweden) _(MSR ’18)_. 414–425.  https://doi.org/10.1145/3196398.3196446
  * Ma et al. (2012) Ying Ma, Guangchun Luo, Xue Zeng, and Aiguo Chen. 2012\.  Transfer learning for cross-company software defect prediction.  _Information and Software Technology_ 54, 3 (2012), 248 – 256.  https://doi.org/10.1016/j.infsof.2011.09.007
  * Ma et al. (2019) Z. Ma, H. Ge, Y. Liu, M. Zhao, and J. Ma. 2019.  A Combination Method for Android Malware Detection Based on Control Flow Graphs and Machine Learning Algorithms.  _IEEE Access_ 7 (2019), 21235–21245.  https://doi.org/10.1109/ACCESS.2019.2896003
  * Madhavan and Whitehead (2007) Janaki T. Madhavan and E. James Whitehead. 2007.  Predicting Buggy Changes inside an Integrated Development Environment. In _Proceedings of the 2007 OOPSLA Workshop on Eclipse Technology EXchange_ (Montreal, Quebec, Canada) _(eclipse ’07)_. 36–40.  https://doi.org/10.1145/1328279.1328287
  * Majd et al. (2020) Amirabbas Majd, Mojtaba Vahidi-Asl, Alireza Khalilian, Pooria Poorsarvi-Tehrani, and Hassan Haghighi. 2020\.  SLDeep: Statement-level software defect prediction using deep-learning model on static code features.  _Expert Systems with Applications_ 147 (2020), 113156.  https://doi.org/10.1016/j.eswa.2019.113156
  * Malhotra (2014) Ruchika Malhotra. 2014\.  Comparative analysis of statistical and machine learning methods for predicting faulty modules.  _Applied Soft Computing_ 21 (2014), 286 – 297.  https://doi.org/10.1016/j.asoc.2014.03.032
  * Malhotra et al. (2017) R. Malhotra, L. Bahl, S. Sehgal, and P. Priya. 2017\.  Empirical comparison of machine learning algorithms for bug prediction in open source software. In _2017 International Conference on Big Data Analytics and Computational Intelligence (ICBDAC)_. 40–45.  https://doi.org/10.1109/ICBDACI.2017.8070806