# Detecting Security-Relevant Methods using Multi-label Machine Learning

Oshando Johnson<EMAIL_ADDRESS>Fraunhofer IEMZukunftsmeile
1PaderbornGermany33102 ,  Goran Piskachev<EMAIL_ADDRESS>Amazon Web
Services1 Thørväld CircleBerlinGermany ,  Ranjith Krishnamurthy
<EMAIL_ADDRESS>Fraunhofer IEMZukunftsmeile
1PaderbornGermany33102 and  Eric Bodden<EMAIL_ADDRESS>Paderborn University and Fraunhofer IEMFürstenallee 11PaderbornGermany

(2024)

###### Abstract.

To detect security vulnerabilities, static analysis tools need to be
configured with security-relevant methods. Current approaches can
automatically identify such methods using binary relevance machine learning
approaches. However, they ignore dependencies among security-relevant methods,
over-generalize and perform poorly in practice. Additionally, users have to
nevertheless manually configure static analysis tools using the detected
methods. Based on feedback from users and our observations, the excessive
manual steps can often be tedious, error-prone and counter-intuitive.

In this paper, we present Dev-Assist, an IntelliJ IDEA plugin that detects
security-relevant methods using a multi-label machine learning approach that
considers dependencies among labels. The plugin can automatically generate
configurations for static analysis tools, run the static analysis, and show
the results in IntelliJ IDEA. Our experiments reveal that Dev-Assist’s machine
learning approach has a higher F1-Measure than related approaches. Moreover,
the plugin reduces and simplifies the manual effort required when configuring
and using static analysis tools.

Static Analysis, Software Security, Machine Learning, Vulnerability Detection,
Multi-label learning, IntelliJ Plugin development

††journalyear: 2024††copyright: rightsretained††conference: 2024 First IDE
Workshop; April 20, 2024; Lisbon, Portugal††booktitle: 2024 First IDE Workshop
(IDE ’24), April 20, 2024, Lisbon, Portugal††doi:
10.1145/3643796.3648464††isbn: 979-8-4007-0580-9/24/04††ccs: Security and
privacy Software security engineering††ccs: Software and its engineering
Software verification and validation††ccs: Security and privacy Software
security engineering††ccs: Security and privacy Systems security††ccs:
Security and privacy Vulnerability scanners

##  1\. Introduction

With the continued rise in the number of reported software vulnerabilities
(hackerone, 6) such as those in the 2023 Common Weakness Enumeration (CWE) Top
25 Most Dangerous Software Weaknesses list (top25, 5), more and more companies
resort to Static Application Security Testing (SAST) tools to detect
vulnerabilities. Yet, to be able to detect security vulnerabilities
effectively, experts currently need to correctly configure and adapt the SAST
tools. One required configuration comprises security-relevant methods (SRM),
which are critical points in a program that have an impact on the analysis
(swan, 19). SRMs relevant for detecting taint-style vulnerabilities include
sources (methods that create data that the analysis should track), sinks
(methods at which the analysis might need to raise an alarm), and sanitizers
(methods that perform data sanitization).

For Java, several machine learning (ML) approaches have been proposed to
replace the time-consuming and error-prone manual approach of detecting SRMs
in large codebases (susi, 3, 19, 27). One such approach, SWAN, classifies Java
methods into sources, sinks and sanitizers as well as seven111CWE78 OS Command
Injection, CWE79 Cross-site Scripting, CWE89 SQL Injection, CWE306 Missing
Authentication, CWE601 Open Redirect, CWE862 Missing Authorisation, and CWE863
Incorrect Authorisation Common Weakness Enumeration types—with high precision
and recall (swan, 19). SWAN-Assist is an IntelliJ IDEA plugin built on top of
SWAN that uses active machine learning to detect SRMs (swan-assist, 20). With
minimal user effort, the plugin’s semi-automatic approach improves SWAN’s
precision.

The fundamental machine learning problem that SWAN addresses is a multi-label
learning problem in which real-world objects that have several semantic
meanings (zhang2018binary, 34) are represented by a set of labels
(tsoumakas2007multi, 28, 35). Detecting security-relevant methods in Java
programs is a multi-label machine learning problem in which one assigns a
subset of SRM labels to each method. In the code snippet in Listing 1, three
security-relevant methods are invoked. The source method _getParameter()_
creates the taint, the sanitizer _encodeForSQL()_ sanitizes the user data, and
_executeQuery()_ is the sink method. In this example, the source and sanitizer
methods impact the content of the SQL query that the sink method executes on
the database. These methods are therefore relevant for CWE89 SQL injection. A
method that writes unsanitized user input to a database and also uses the
unsanitized data in a redirect should be labeled as a sink both for SQL
injection (CWE89) and Open Redirect (CWE601) vulnerabilities.

⬇

1protected void doPost(HttpServletRequest req, ...) {

2 String usr = req.getParameter("ID"); //source

3 usr = ESAPI.encoder().encodeForSQL(new MySQLCodec(), usr); //sanitizer

4 String query = "select * from user" +

5 "where username=’" + usr + "’";

6 Statement stmt = conn.createStatement();

7 ResultSet rs = stmt.executeQuery(query); //sink

Listing 1: Code snippet with a potential SQL injection from source (l.2) to
sink (l.7) mitigated using a sanitizer (l.3).

Although the classification of security-relevant methods is a multi-label
problem, SWAN solves it with a binary relevance (BR) approach, which
decomposes the machine learning problem into several independent single-label
classification problems (zhang2018binary, 34). This approach fails to exploit
the correlation among labels and can therefore produce prediction models that
generalize badly (binaryrel, 33). Additionally, in order to use SWAN-Assist’s
results to configure a SAST tool, users have to manually extract the SRM
information and externally verify if the configuration produces the expected
results.

In this paper, we present Dev-Assist, a holistic IntelliJ IDEA plugin that
uses a multi-label machine learning approach that generalizes better in
assigning SRM labels to Java methods. The plugin can use labeled SRMs to
automatically generate the configuration for SAST tools, integrates a static
analysis to detect security vulnerabilities, and shows the analysis results
within IntelliJ. We show that the multi-label approach outperforms existing
binary relevance classification approaches for detecting SRMs and reduces
manual effort when using the detected security-relevant methods.

Dev-Assist’s source code, documentation and datasets are available
online222https://github.com/secure-software-engineering/swan/tree/master/dev-
assist. We next explain the motivation and requirements for the plugin.
Section 3 describes the plugin’s interface, features and back-end. We evaluate
our approach in Section 4, and subsequently discuss the limitations and
conclusions of our work.

##  2\. Motivation and Requirements

SWAN and SWAN-Assist have two main shortcomings, the machine learning solution
for the multi-label classification problem and poor usability due to excessive
manual effort.

It is generally known that, because BR uses an independent method that does
not exploit the correlations between labels, it may produce prediction models
that do not generalize well in practice (zhang2018binary, 34). Moreover,
SWAN’s BR approach has a different feature representation for each SRM label
and uses a two-phase model selection approach. These design and implementation
choices complicate maintenance and evaluation of the approach.

We identified the need to improve SWAN-Assist’s usability and reduce the
required manual effort at a Security Testing and Verification Summer School in
Belgium that was attended by more than 80 PhD students and researchers
(summerschool, 7). At the summer school, we organized a hands-on workshop
session focused on detecting SRMs and using them to detect security
vulnerabilities in the WebGoat (webgoat, 18) project. Participants were
required to use SWAN-Assist to detect and update the list of SRMs. To detect
the vulnerabilities, participants had to manually create the specifications
(fluenttql, 22) necessary to configure and run the open-source taint analysis
tool SecuCheck (secucheck, 21). Although many of the participants found the
tools useful, one resounding feedback was that using the SRM list to create
specifications to run SecuCheck, was complex, error-prone, counter-intuitive,
and required too much manual work. Many participants did not complete these
tasks within the allotted time. Additionally, the participants’ feedback and
our observations revealed numerous bugs, errors, and improvements for SWAN-
Assist and SecuCheck.

Given the limitations of SWAN-Assist and SWAN’s machine learning approach, a
new plugin that meets the following requirements is necessary:

  * •

_R1 : The plugin shall assign SRM labels using an ML approach that considers
dependencies among SRM labels_: Given that dependencies may exist among SRM
categories, the model should use this information so that it can generalize
better.

  * •

_R2 : The plugin shall use detected SRMs to automatically generate taint-flow
query specifications_: Instead of requiring users to export the detected SRMs
and manually create taint-flow query specifications, the plugin should use the
SRM list to automatically generate specifications for SecuCheck.

  * •

_R3 : The plugin shall run a taint analysis and report vulnerabilities in the
IDE_: Using automatically generated or external taint-flow specifications, the
plugin should be able to run a taint analysis and display the analysis results
in the IDE.

##  3\. Dev-Assist Plugin: Automated Configuration for Static Analysis

To address the limitations of SWAN and SWAN-Assist, we present a holistic
plugin, Dev-Assist, that automatically detects SRMs using a multi-label
approach, generates taint-flow specifications, and also runs a static code
analysis. Figure 1 shows the architecture of the Dev-Assist plugin which
consists of a core module and analysis pipeline that are built on top of the
IntelliJ platform.

Figure 1. Dev-Assist’s architecture showing the interaction of the core
module, analysis pipeline, and IntelliJ platform.

The core module contains the plugin’s user interface (UI) and an application
programming interface (API) that coordinates the communication with the other
modules. The API interacts with the analysis pipeline by configuring,
executing, and processing its results. The API also interacts with the
IntelliJ platform (intellij-platform, 14) and uses the Program Structure
Interface (PSI) (psi, 11), which provides a syntactic and semantic code model,
the Action System (actionsystem, 9) to add items to IntelliJ’s menus and
toolbars, and the IntelliJ IDEA UI.

The plugin was developed using the Gradle IntelliJ Plugin (1.16.1) and Java
17. In line with IntelliJ’s recommendation (support, 12) to support the latest
major release and at least two previous releases, the plugin supports IntelliJ
Platform version 2022.1 to 2023.3.

###  3.1. Plugin Interface and Features

To support the new analysis pipeline and enhance the plugin’s user interface,
we had to implement several back-end and front-end changes. Figure 2 shows
Dev-Assist’s tool window (1), dialog for updating SRMs (2), and a sample
vulnerability report (3).

Figure 2. IntelliJ window screenshot showing the Dev-Assist plugin tool window
(1), method dialog (2), and analysis results (3).

####  3.1.1. Tool Window (1)

The toolbar in the plugin’s tool window contains buttons to run the analysis
modules, as well as buttons to filter, expand/collapse the expandable list,
and to also import/export the SRM list. The expandable list view below the
toolbar shows the SRM list for methods detected in the active project as well
as SRMs present in SWAN’s training dataset. We have updated the expandable
list to provide more information to the user without having to expand all
nodes in the list.

We added a new action to the plugin’s toolbar to run SecuCheck with the
generated specifications, and another action to manage the plugin settings.
Previously, a user would always see a dialog when running the analysis. Now we
show the dialog only when the user runs the analysis the first time and, if
they wish, they can update the settings using the new toolbar action.

####  3.1.2. Method Dialog (2)

When a user selects a method in the list view or editor, the edit method
dialog appears. Using this dialog, a user can view and update the method’s
classification, taint-flow information as well as other properties.

Instead of two dialogs to update the classification of the SRMs and method
properties, we have merged the dialogs into one dialog that uses a tabbed
pane. We have replaced the double list interface for updating SRM labels with
checkboxes to reduce the effort necessary to change the categories. A new tab
with UI components to update the data flow properties of method parameters is
also now present. The two properties, data-in, and data-out, are particularly
important when generating taint-flow specifications as they indicate which
method parameters are included in the data flow operations.

####  3.1.3. Analysis Results (3)

Instead of building a new interface to show the analysis pipeline results, we
opted to use the existing Problems tool window, which displays problems
detected by IntelliJ IDEA using several tabs (problems, 10). The Server-Side
Analysis tab of the Problems tool window displays problems detected with the
code quality platform Qodana (qodana, 8) and also provides navigation to the
code fragments that contain the reported problems. Using Qodana, the Static
Analysis Results Interchange Format (SARIF) (sarif, 4) analysis results from
Dev-Assist can be opened as local reports.

###  3.2. AI Supported Analysis Pipeline

The analysis pipeline detects security-relevant methods using the multi-label
implementation of SWAN, automatically generates fluentTQL specifications and
finds security vulnerability using SecuCheck. The analysis pipeline module is
integrated into the plugin’s core module as a Gradle dependency. Next, the
three components of the analysis pipeline are discussed.

####  3.2.1. Multi-Label SRM Detection

To address SWAN’s binary relevance limitations and fulfill R1, we extended
SWAN to use MEKA (v1.9.7) (meka, 25), a multi-label extension of the ML Java
Library WEKA(weka, 32). The framework has been designed specifically for the
implications of multi-label approaches such as modeling dependencies between
variables and also the evaluation of such models which involves scoring
multiple target classifications for each instance (meka, 25).

The multi-label ML approach is shown in Figure 3. In the training phase, we
use the static program analysis framework Soot (cetus11soot, 15) to extract
information from the programs in the training dataset to create the feature
vector. The feature vector and labeled methods are provided as inputs to MEKA
to train and identify the best model. In the testing phase, the feature vector
is provided to the trained model and the detected SRMs are exported to a JSON
file.

Figure 3. Diagram of Dev-Assist’s multi-label machine learning approach for
detecting security-relevant methods.

We updated SWAN’s original dataset, which contained 235 labeled Java methods,
with 1417 new methods that we extracted from public SRM lists
(findsecbugs2017, 2, 26, 29) and their respective Java archive (JAR) files.
The updated dataset now contains 1,625 labeled methods that are stored in a
JSON file in the project repository.

Instead of generating 10 feature representations, one for each of the SRM
types, for SWAN’s two-phase approach, all features have been merged into one
feature representation. Dev-Assist reuses all of the features from SWAN,
however, we have consolidated the features so that they can be used for multi-
label learning. The majority of the features use a bag-of-words (bow, 23)
approach to evaluate the presence of pre-defined tokens in the signatures of
methods defined or invoked in a program. Some examples of these tokens are:
”database”, ”delete”, ”replac”, ”encod”, ”redirect”, and others that can be
used to infer what a method or class does. The remaining features are
categorical features that evaluate various properties such as the modifiers
and types of classes and methods.

We have implemented 12 new features that count various properties of the
methods and classes after processing the programs with Soot. For a given
method, we count its code lines, methods invoked, branching statements,
iteration statements, exception handlers, parameters, and variables
defined/used. For the class that the method belongs to, we count its code
lines, functions, and tokens. With the consolidation of the existing features
and addition of the new features, Dev-assist’s feature vector contains 119
features (13 numeric, 99 binary and 7 categorical).

In the training phase, the feature extraction module uses the labels and
feature vector to create a dataset that can be used to train the model. MEKA
uses the Attribute-Relation File Format (ARFF) to specify the labels as
multiple binary attributes and the list of instances (meka, 25). Dev-Assist’s
training ARFF file contains 129 attributes (10 SRM categories and 186
features) and 1625 instances.

Using an ARFF file that contains the training data, we can run experiments to
identify a machine learning model using MEKA’s command line or graphical user
interface. However, instead of manually specifying a list of machine learning
algorithms for our experiments, as was done for SWAN, we opted to use
automated machine learning (AutoML). This approach uses a larger search space
of machine learning algorithm combinations in order to identify the multi-
label classifier as well as its hyperparameters (wever2021automl, 31).

We use the AutoML framework ML2-Plan (v0.2.7) to recursively select, combine,
and parameterize MEKA’s multi-label algorithms in order to identify the
machine learning pipeline for predicting SRMs (ml2plan, 30, 31). We configure
ML2-Plan to use 8 CPUs, 1 hour timeout, 70:30 train-test split, and the
default settings recommended in the Git
repository333https://github.com/starlibs/AILibs/tree/master. The experiments
were executed on a Lenovo T14 Gen 1 20UD (Ryzen 7 Pro 4750U, 1.7 GHz) laptop.
ML2-Plan selected a multi-label machine learning model using the Ensembles of
Pruned Set (4781214, 24) method with logistic model trees
(landwehr2005logistic, 16) as the base classifier.

####  3.2.2. Automatically Generated Specifications

Many widely used SAST tools use taint analysis to detect security
vulnerabilities. SecuCheck (secucheck, 21) is one such tool that provides a
configurable analysis and can be seamlessly integrated into various Integrated
Development Environments (IDEs). With SecuCheck, users can create their own
taint-flow specifications using the Java internal Domain Specific Language
(DSL) fluentTQL (secucheck, 21, 22). To create the taint-flow specifications
with fluentTQL, users must explicitly and manually specify SRMs such as
source, sink, sanitizer, and propagator as well as information from the method
signatures. For example, the user would need to enter the fully qualified
class names for the SRMs as well as other properties necessary for tracking
data flow.

The current approach of exporting the SRM list and requiring a user to
manually create the taint-flow specifications is a tedious and error-prone
process. To improve this process and meet R2, we integrate the fluentTQL API
into the analysis pipeline to generate the taint-flow specifications from a
list of SRMs. The approach provides a central way to manage the information
necessary for creating the specifications, namely the SRM list which can be
stored as a JSON file. The taint-flow specifications can then be used to
configure SecuCheck in order to detect security vulnerabilities.

####  3.2.3. Detecting Vulnerabilities with SecuCheck

To run SecuCheck (secucheck, 21), the path of the project to be analyzed,
taint-flow specifications, and other analysis options can be provided via its
command line interface. The analysis results are exported to a SARIF (sarif,
4) file.

To fulfill R3, we integrated SecuCheck in the analysis pipeline so that we can
use the generated taint-flow specifications described in the previous section.
To make this possible, we refactored the tool’s implementation and
architecture to expose integration points. We also updated SecuCheck to ensure
that it correctly follows the SARIF 2.1.0 (sarif, 4) specification and
generates a valid SARIF file.

##  4\. Evaluation

Dev-Assist’s main motivations were to improve the machine learning approach
for detecting security-relevant methods and to reduce the manual effort when
using SRMs to configure static analysis tools. We assess these contributions
in the sections that follow.

###  4.1. Multi-label Machine Learning Approach

We compare Dev-Assist’s multi-label approach with SWAN-Assist’s binary
relevance approach using the cross-validation results and also their
performance on a real-world project.

To assess the in-sample performance, we did a 10-fold cross-validation with a
70:30 train-test split of the new training dataset for both approaches. Table
1 shows the average precision, recall, and F1-score (harmonic mean of the
precision and recall) over the cross-validation folds. The F1-Score is often
used to estimate the performance of machine learning models, therefore, a
higher F1-Score indicates better performance. Using SWAN’s F1-scores as the
baseline, we see that Dev-Assist outperforms SWAN-Assist for 9 SRM labels
(sinks, sources, CWE78, CWE79, CWE89, CWE306, CWE601, CWE862, and CWE863).
Moreover, for 3 of these SRMs (CWE78, CWE79 and CWE601), Dev-Assist’s
F1-Scores are 1.5-2 times higher than those of SWAN which were all lower than
0.35. For sanitizers, SWAN-Assist (0.89) performs minimally better than Dev-
Assist (0.87). The low performance for CWE78 (0.55) and CWE602 (0.37) can be
attributed to the imbalanced training dataset, where these CWEs have the
lowest number of examples compared to the other CWE labels. Imbalanced
datasets make it difficult for ML models to identify the minority class which
results in lower scores (krawczyk2016learning, 13).

|  |  |  |  | Common Weakness Enumeration
---|---|---|---|---|---
|  | Sa | Si | So | 78 | 79 | 89 | 306 | 601 | 862 | 863
Dev | P | 0.90 | 0.75 | 0.70 | 0.75 | 0.60 | 0.88 | 0.91 | 0.44 | 0.84 | 0.84
R | 0.85 | 0.80 | 0.63 | 0.43 | 0.57 | 0.75 | 0.81 | 0.32 | 0.78 | 0.79
F | 0.87 | 0.78 | 0.66 | 0.55 | 0.58 | 0.81 | 0.86 | 0.37 | 0.81 | 0.81
SWAN | P | 0.89 | 0.79 | 0.66 | 0.87 | 0.41 | 0.86 | 0.87 | 0.41 | 0.89 | 0.83
R | 0.90 | 0.72 | 0.57 | 0.22 | 0.24 | 0.68 | 0.75 | 0.18 | 0.65 | 0.66
F | 0.89 | 0.75 | 0.61 | 0.34 | 0.30 | 0.75 | 0.81 | 0.24 | 0.74 | 0.73
Table 1. Dev-Assist (Dev) and SWAN-Assist (SWAN) 10-fold cross-validation
precision (P), recall (R), and F1-Score (F) results for sanitizers (Sa), sinks
(Si), sources (So), and CWEs.

To evaluate how well Dev-Assist’s generalizes on real-world projects, out-of-
sample, we used Dev-Assist to detect SRMs in the Android 13 (API level 33)
(android13, 17) project. Then, we randomly selected 20 methods predicted for
each SRM label and manually verified the predictions. Dev-Assist’s precision
for the selected methods is shown in Table 2. The average precision was 0.72
and CWE862/CWE863 have the highest precision (0.95). CWE89 had the lowest
precision (0.35) and is consistent with the cross-validation results. Despite
low precision on the random samples for CWE89, Dev-Assist correctly labeled 74
sinks and CWE89 methods in the android.database.sqlite.SQLiteDatabase class.
In comparison to SWAN-Assist, more than 80% of these methods are not
classified as security-relevant which indicates that Dev-Assist is more
precise in practice.

|  |  |  | Common Weakness Enumeration |
---|---|---|---|---|---
| Sa | Si | So | 78 | 79 | 89 | 306 | 601 | 862 | 863 | $\overline{P}$
P | 0.9 | 0.65 | 0.85 | 0.75 | 0.5 | 0.35 | 0.6 | 0.6 | 0.95 | 0.95 | 0.72
Table 2. Dev-Assist’s Precision (P) on 200 randomly selected methods from the
Android 13 library.

###  4.2. Security Vulnerability Specifications and Detection

To reduce the manual effort required to use SRM lists exported by SWAN-Assist
to create security specifications that can be used by taint analysis tools to
detect security vulnerabilities, several changes were made to Dev-Assist. We
will assess the new approach by evaluating the number of steps and the average
time required to accomplish the same task using SWAN-Assist and Dev-Assist.

For our evaluation, we use the tasks that the participants of the Summer
School mentioned in Section 2 were required to perform in order to use SWAN-
Assist and SecuCheck to find vulnerabilities in the WebGoat project. The tasks
are listed in Table 3 along with the average duration for each task that were
estimated by two of the workshop organizers that supported the participants
during the hands-on session. The duration only considers the time when the
user actually performs an activity and therefore excludes the run and/or wait
time.

No. |  Task |  Duration
---|---|---
1 |  Run analysis to detect SRMs |  0.5 Min.
2 |  Update detected SRMs and add new ones |  5 Min.
3 |  Export SRMs to JSON file |  0.5 Min.
4 |  Create fluentTQL specifications using SRM information (for e.g., method signature) in the JSON file |  7 Min.
5 |  Compile and export fluentTQL specifications |  0.5 Min.
6 |  Configure SecuCheck with the exported specifications and set analysis options |  4 Min.
7 |  Run SecuCheck and export analysis results to a SARIF file |  0.5 Min.
8 |  Interpret taint analysis results and validate them in the source code |  7 Min.
|  |  25 Min
Table 3. Workflow showing the steps and average duration when using SWAN to
manually configure SecuCheck.

Based on Table 3, the participants spent 76% of their time using the exported
SRMs to create the security specifications (Tasks 4-5), set up and run
SecuCheck (Tasks 6-7), and to interpret the results (Task 8). Dev-Assist
consolidates and automates five tasks (Task 3-7) such that with the click of a
button all of these steps will be performed and the analysis results will be
available in Qodana. This reduces the number of tasks and the estimated time
to perform the tasks by 50%. Consolidating these steps also means that a user
can obtain results quicker without the need for much manual intervention.

Given that the results are visible within the IDE and intuitively shown, the
effort to interpret and action the results (Task 7) is also reduced. A
developer needs not to switch to an external tool or manually search for the
results within the SARIF file.

##  5\. Limitations and Threats to Validity

Despite our efforts to avoid known pitfalls (dosanddonts, 1) when using ML for
computer security, there are some limitations in our approach. The training
examples are collected from major Java libraries, which have higher code
quality in comparison to real-world projects. As in all related work, most
features in our approach use information from method signatures, therefore,
poorly named classes and methods pose a problem. Additionally, our training
data is imbalanced and is not representative, as the proportion of SRM and
non-SRM methods does not conform to the real world.

##  6\. Conclusion

In this paper, we present a multi-label machine learning approach for
detecting security-relevant methods in Java programs using the Dev-Assist
IntelliJ IDEA plugin. The plugin’s ML approach considers correlation among SRM
types, automatically generates specifications, and also runs a code analysis.
We described Dev-Assist’s features, interface, architecture, and its
underlying multi-label machine learning model. Using the plugin, developers
and SAST experts can create and verify SRM lists, which are required to
correctly configure static analysis tools, with less manual effort and more
precision.

###### Acknowledgements.

We thank Rohith Shanmuganathan for helping with the implementation of the
analysis pipeline and IntelliJ plugin. We acknowledge support by the German
Federal Ministry of Education and Research (BMBF) under grant number 16KIS1257
(AI-DevAssist).

## References

  * (1) Daniel Arp et al.  “Dos and Don’ts of Machine Learning in Computer Security” 46.23.01; LK 01  In _Proc. of the USENIX Security Symposium 2022_ , 2022
  * (2) Philippe Arteau, David Formánek and Tomáš Polešovský  “Find-sec-bugs Resources”, https://github.com/find-sec-bugs/find-sec-bugs/tree/master/findsecbugs-plugin/src/main/resources/injection-sinks, 2020
  * (3) Steven Arzt, Siegfried Rasthofer and Eric Bodden  “SuSi: A Tool for the Fully Automated Classification and Categorization of Android Sources and Sinks”  In _Network and Distributed System Security Symposium 2013_ , NDSS’13, 2013
  * (4) Edited Michael C. and Laurence J. Golding  “Static Analysis Results Interchange Format (SARIF) Version 2.1.0 Plus Errata 01”  In _2019 International Engineering Conference (IEC)_ , 2023  URL: https://docs.oasis-open.org/sarif/sarif/v2.1.0/errata01/os/sarif-v2.1.0-errata01-os-complete.html.%20Latest%20stage:%20https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html.
  * (5) The MITRE Corporation  “CWE Top 25 Most Dangerous Software Weaknesses” Accessed on November 14, 2023, 2023  URL: https://cwe.mitre.org/top25/
  * (6) Hackerone  “Hacker-Powered Security Report: Industry Insights” Accessed on November 14, 2023, 2022  URL: https://www.hackerone.com/reports/6th-annual-hacker-powered-security-report
  * (7) Bart Jacobs and Coen De Roover  “Summer School on Security Testing and Verification”, 2022  URL: https://cybersecurity-research.be/summer-school-security-testing-and-verification-2022
  * (8) JetBrains  “About Qodana”, 2023  URL: https://www.jetbrains.com/help/qodana/about-qodana.html
  * (9) JetBrains  “Actions”, 2023  URL: https://plugins.jetbrains.com/docs/intellij/basic-action-system.html
  * (10) JetBrains  “Problems tool window”, 2023  URL: https://www.jetbrains.com/help/idea/problems-tool-window.htmll
  * (11) JetBrains  “Program Structure Interface (PSI)”, 2022  URL: https://plugins.jetbrains.com/docs/intellij/psi.html
  * (12) JetBrains  “Statistics: Product Versions in Use”, 2023  URL: https://plugins.jetbrains.com/docs/marketplace/product-versions-in-use-statistics.html
  * (13) Bartosz Krawczyk  “Learning from imbalanced data: open challenges and future directions”  In _Progress in Artificial Intelligence_ 5.4 Springer, 2016, pp. 221–232
  * (14) Zarina Kurbatova, Yaroslav Golubev, Vladimir Kovalenko and Timofey Bryksin  “The IntelliJ Platform: A Framework for Building Plugins and Mining Software Data”  In _2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)_ , 2021, pp. 14–17  DOI: 10.1109/ASEW52652.2021.00016
  * (15) P. Lam, E. Bodden, O. Lhoták and L. Hendren  “The Soot framework for Java program analysis: a retrospective”  In _Cetus Users and Compiler Infrastructure Workshop (CETUS 2011)_ , 2011
  * (16) Niels Landwehr, Mark Hall and Eibe Frank  “Logistic model trees”  In _Machine learning_ 59 Springer, 2005, pp. 161–205
  * (17) OWASP  “Andoid 13” Online; accessed December 2023, https://developer.android.com/about/versions/13, 2023
  * (18) OWASP  “WebGoat” Online; accessed January 2020, https://github.com/WebGoat/WebGoat, 2020
  * (19) Goran Piskachev, Lisa Nguyen Quang Do and Eric Bodden  “Codebase-Adaptive Detection of Security-Relevant Methods”  In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ISSTA 2019  Beijing, China: Association for Computing Machinery, 2019, pp. 181–191
  * (20) Goran Piskachev, Lisa Nguyen Quang Do, Oshando Johnson and Eric Bodden  “SWANAssist: Semi-Automated Detection of Code-Specific, Security-Relevant Methods”  In _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering_ , ASE ’19  San Diego, California: IEEE Press, 2020, pp. 1094–1097  DOI: 10.1109/ASE.2019.00110
  * (21) Goran Piskachev, Ranjith Krishnamurthy and Eric Bodden  “SecuCheck: Engineering configurable taint analysis for software developers”  In _2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2021, pp. 24–29  DOI: 10.1109/SCAM52516.2021.00012
  * (22) Goran Piskachev, Johannes Späth, Ingo Budde and Eric Bodden  “Fluently Specifying Taint-Flow Queries with FluentTQL”  In _Empirical Softw. Engg._ 27.5 USA: Kluwer Academic Publishers, 2022  DOI: 10.1007/s10664-022-10165-y
  * (23) Wisam A. Qader, Musa M. Ameen and Bilal I. Ahmed  “An Overview of Bag of Words;Importance, Implementation, Applications, and Challenges”  In _2019 International Engineering Conference (IEC)_ , 2019, pp. 200–204  DOI: 10.1109/IEC47844.2019.8950616
  * (24) Jesse Read, Bernhard Pfahringer and Geoff Holmes  “Multi-label Classification Using Ensembles of Pruned Sets”  In _2008 Eighth IEEE International Conference on Data Mining_ , 2008, pp. 995–1000  DOI: 10.1109/ICDM.2008.74
  * (25) Jesse Read, Peter Reutemann, Bernhard Pfahringer and Geoff Holmes  “MEKA: A Multi-label/Multi-target Extension to Weka”  In _Journal of Machine Learning Research_ 17.21, 2016, pp. 1–5  URL: http://jmlr.org/papers/v17/12-164.html
  * (26) L Sampaio  “Which methods should be considered “Sources”, “Sinks” or “Sanitization”?” Accessed 05.03.2020, https://thecodemaster.net/methods-considered-sources-sinks-sanitization/, 2014
  * (27) Darius Sas, Marco Bessi and Francesca A. Fontana  “Automatic Detection of Sources and Sinks in Arbitrary Java Libraries”  In _2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2018, pp. 103–112
  * (28) Grigorios Tsoumakas and Ioannis Katakis  “Multi-label classification: An overview”  In _International Journal of Data Warehousing and Mining (IJDWM)_ 3.3 IGI Global, 2007, pp. 1–13
  * (29) A Van der Stock et al.  “OWASP code review guide v1. 1”  In _The OWASP Foundation Guidelines_ , 2008
  * (30) Marcel Wever, Felix Mohr and Eyke Hüllermeier  “Automated Multi-Label Classification based on ML-Plan”, 2018  arXiv:1811.04060 [cs.LG]
  * (31) Marcel Wever, Alexander Tornede, Felix Mohr and Eyke Hüllermeier  “AutoML for multi-label classification: Overview and empirical evaluation”  In _IEEE transactions on pattern analysis and machine intelligence_ 43.9 IEEE, 2021, pp. 3037–3054
  * (32) Ian H. Witten, Eibe Frank, Mark A. Hall and Christopher J. Pal  “Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques”  San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2016
  * (33) Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu and Xin Geng  “Binary Relevance for Multi-Label Learning: An Overview”  In _Front. Comput. Sci._ 12.2 Berlin, Heidelberg: Springer-Verlag, 2018, pp. 191–202  DOI: 10.1007/s11704-017-7031-7
  * (34) Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu and Xin Geng  “Binary relevance for multi-label learning: an overview”  In _Frontiers of Computer Science_ 12 Springer, 2018, pp. 191–202
  * (35) Min-Ling Zhang and Zhi-Hua Zhou  “A review on multi-label learning algorithms”  In _IEEE transactions on knowledge and data engineering_ 26.8 IEEE, 2013, pp. 1819–1837