# An Empirical Study on the Use of Static Analysis Tools in Open Source
Embedded Software

Mingjie Shen  Purdue UniversityWest LafayetteUSA<EMAIL_ADDRESS>,  Akul
Pillai  Purdue UniversityWest LafayetteUSA<EMAIL_ADDRESS>,  Brian A.
Yuan  Purdue UniversityWest LafayetteUSA<EMAIL_ADDRESS>,  James C. Davis
Purdue UniversityWest LafayetteUSA<EMAIL_ADDRESS>and  Aravind Machiry
Purdue UniversityWest LafayetteUSA<EMAIL_ADDRESS>
###### Abstract.

Embedded software is used in safety-critical systems such as medical devices
and autonomous vehicles, where software defects comprising security
vulnerabilities have severe consequences. Many embedded software products
incorporate Open-Source Embedded Software (EMBOSS), so it is important for
EMBOSS engineers to use appropriate mechanisms to avoid security
vulnerabilities. One common defense against security vulnerabilities is the
use of static analysis, which can offer sound guarantees. While researchers
have examined the practices, challenges, and potential benefits of static
analysis for many kinds of open-source software, these observations have not
been made for EMBOSS. There is little data to guide open-source software
engineers and regulators on the cost-benefit tradeoffs of applying (or
mandating) static analysis in this context.

This paper performs the first study to understand the prevalence, challenges,
and effectiveness of using Static Application Security Testing (SAST) tools on
EMBOSS repositories. We collect a corpus of 258 of the most popular EMBOSS
projects, representing 13 distinct categories such as real-time operating
systems, network stacks, and applications. To understand the current use of
SAST tools on EMBOSS, we measured this corpus and surveyed developers. To
understand the challenges and effectiveness of using SAST tools on EMBOSS
projects, we applied these tools to the projects in our corpus. We report that
almost none of these projects (just 3%) use SAST tools beyond those baked into
the compiler, and developers give rationales such as ineffectiveness and false
positives. In applying SAST tools ourselves, we show that minimal engineering
effort and project expertise are needed to apply many tools to a given EMBOSS
project. GitHub’s CodeQL was the most effective SAST tool — using its built-in
security checks we found a total of 540 defects (with a false positive rate of
23%) across the 258 projects, with 399 (74%) likely security vulnerabilities,
including in projects maintained by Microsoft, Amazon, and the Apache
Foundation. EMBOSS engineers have confirmed 273 (51%) of these defects, mainly
by accepting our pull requests. Two CVEs were issued. In summary, we urge
EMBOSS engineers to adopt the current generation of SAST tools, which offer
low false positive rates and are effective at finding security-relevant
defects.

##  1\. Introduction

Our dependence on embedded devices (e.g., IoT devices) and consequently
embedded software has significantly increased. Embedded devices control many
aspects of our lives, including our homes (Alrawi2019SoK:Deployments),
transportation (al2020intelligence), traffic management (soni2017review), and
the distribution of vital resources like food (prapti2022internet) and power
(o2013industrial). The adoption of these devices has seen rapid and extensive
growth, with an estimated count of over 50 billion devices (al-
garadi_survey_2020). Vulnerabilities in Embedded Software (EmS) (enabling
these devices) have far-reaching consequences (antonakakis2017understanding;
writer_5_2020) due to the pervasive and interconnected nature of these
devices, as exemplified by the infamous Mirai botnet (margolis2017depth).

Open-source Software (OSS) plays an important role in EmS development
(10.1145/2737182.2737190; lundellPractitionerPerceptionsOpen2011;
amiri2017survey). Various popular Open-Source Embedded Software (EMBOSS)
exist, such as libraries and Real Time Operating Systems (osrtosweb). For
instance, FreeRTOS (noauthor_freertos_nodate) and Zephyr
(the_linux_foundation_zephyr_2023), two of the most popular and industry-
endorsed RTOSes, are open-source. It is important to ensure that EMBOSS do not
contain any vulnerabilities and that suitable vulnerability detection
techniques are used to secure them.

The diversity of hardware platforms (muench2018you; white_making_2011), input
mechanisms, and the lack of support for sanitizers (abbasi2019challenges;
yu2022building) make it hard to apply dynamic analysis based vulnerability
detection techniques, such as fuzzing (manes2019art), to EMBOSS. On the other
hand, static analysis vulnerability detection techniques, i.e., Static
Application Security Testing (SAST) tools, do not have such requirements.
Moreover, the latest State of The Practice (SoTP) tools, such as CodeQL
(avgustinov2016ql) are shown to be effective and can find serious security
vulnerabilities in complex codebases (codeqlwof). Furthermore, many of these
SoTP tools can be easily used in the software engineering pipeline by
integrating into Continuous Integration (CI) Workflows, e.g., GitHub Workflows
(koishybayev2022characterizing). As we show in Section 3, many critical OSS
effectively use SAST tools in their CI Workflows. Furthermore, Chelf et al.
(chelf_ensuring_2009) showed that embedded software can greatly benefit from
using SAST tools. However, the use and effectiveness of SAST tools in EMBOSS
is unknown.

In this paper, we perform the first systematic study on the use of SAST tools
to detect security vulnerabilities in EMBOSS. Specifically, we investigate
three research questions:

  * •

RQ1-Prevalence: Are SAST tools currently used in EMBOSS?

  * •

RQ2-Challenges: Is there any difficulty in configuring SAST tools for EMBOSS?

  * •

RQ3-Effectiveness: Can EMBOSS benefit from using SAST tools?

To answer these questions, we curated a corpus of 258 popular EMBOSS projects
from GitHub. We also identified a set of 12 SAST tools that can be readily
integrated into these projects. We used a combination of automated analysis of
CI Workflows and developer surveys to understand the prevalence of SAST tools
usage. We used manual analysis and developer surveys to understand the
challenges in using SAST tools. Finally, we manually created exemplary SAST CI
Workflows for all EMBOSS projects. We used these Workflows to run a modified
version of CodeQL (one of the most effective SAST tools) on each project and
analyzed the corresponding results.

To summarize our results: available SAST tools — specifically, CodeQL — are
easy to configure and substantially outperform EMBOSS developers’ common
practice, which is the compiler’s warnings. We identified that only 10 (4%)
projects use SAST tools as part of their CI Workflows. Furthermore, our
developer survey indicates that despite developers being aware of SAST tools,
most developers do not use them on EMBOSS projects. Most developers claim to
use strict compiler warnings (i.e.,-Wextra and SoTP SAST tools, as we show in
Section 7.4. We identified that CodeQL, one of the most effective SoTP SAST
tools, cannot handle diverse build systems of EMBOSS repositories and
consequently fails to run on many of them. However, we were able to fix this
with minimal engineering effort and created CI Workflows enabling the
execution of CodeQL on EMBOSS repositories. _We found a total of 540 defects,
with 399 (74%) being security vulnerabilities, demonstrating the need to use
SAST tools on EMBOSS projects._ Our contributions are:

  * •

We performed the first study to understand the prevalence and benefit of using
SAST tools in EMBOSS. We observed that current techniques used by developers
(compiler warnings) are ineffective compared to a SoTP SAST tool.

  * •

As part of our study, we curated a list of 258 EMBOSS projects and created
exemplary GitHub Workflows – encapsulating all the necessary compilation steps
– enabling execution of SAST tools. This is the first large-scale embedded
software dataset with the necessary compilation infrastructure.

  * •

We executed CodeQL on our EMBOSS dataset using the created Workflows. We
identified a total of 540 defects (399 (74%) security vulnerabilities) across
all projects, including projects maintained by reputed groups such as Apache,
Microsoft, and Amazon. We have reported all these defects and raised pull
requests. The developers have already confirmed 273 (51%) of these defects,
mainly by accepting our pull requests.

  * •

We open-source all our datasets and GitHub Workflows, enabling future science.

Significance for software engineering: Empirical software engineering research
has a substantial body of knowledge on open-source software, but has focused
on IT or general-purpose software (amusuo_reflections_2022). We present a
large-scale evaluation of embedded open-source software, reporting on both the
effectiveness of SAST tools and on developers’ perceptions. Across 258 EMBOSS,
the CodeQL SAST tool finds hundreds of defects with modest per-repository
configuration and a low false positive rate. EMBOSS software developers can
use this tool to easily improve software quality.

##  2\. Background

###  2.1. Embedded Software, RTOSes, and Open-Source

Embedded software is designed to run on embedded systems, ranging from
industrial controllers (bhamare2020cybersecurity) to resource-constrained
microcontroller-based IoT devices (Alrawi2019SoK:Deployments). As mentioned in
Section 1, these devices control various aspects of our daily lives. Unlike
regular computers, embedded devices use specialized Real Time Operating
Systems (RTOSes) designed for reduced-resource environments (e.g., real-time
scheduling, low power consumption, low memory overhead). There are 31
different RTOSes (osrtosweb), with the majority (26) of them being open-source
and developed in unsafe languages such as C/C++. Examples of RTOSes include
RIOT, Contiki, FreeRTOS, and Azure RTOS. Open-source Software (OSS) is an
essential part of the software supply chain of embedded systems. The inherent
advantages of open-source software, such as long-term sustainability and
accessibility to source code for debugging purposes, have been acknowledged
and appreciated in the embedded software industry
(lundellPractitionerPerceptionsOpen2011; lundellOpenSourceSwedish2010). Most
of Open-Source Embedded Software (EMBOSS) is in C/C++, and studies
(alnaeli_vulnerable_2016; alnaeli_source_2017) show that many EMBOSS use many
unsafe statements.

###  2.2. SAST Tools

Embedded software, with its hardware-coupled nature (muench2018you) and the
lack of necessary emulation support (fasano2021sok), makes it hard to use
dynamic analysis techniques in a scalable manner. Static Application Security
Testing (SAST) tools are specially designed static analysis techniques to find
security vulnerabilities effectively. They do not require execution support,
making them attractive to use on embedded software. Furthermore, as we show in
Table 2, many SoTP SAST tools have the necessary plugins to be easily
integrated into CI pipelines.

####  2.2.1. Availability

There are many open-source and commercial SAST tools. The open-source tools
vary in the underlying techniques and corresponding guarantees. There are
high-assurance tools, such as IKOS (brat2014ikos), that use abstract
interpretation and provide soundness guarantees. However, these tools must be
properly configured with suitable abstract domains to avoid false positives —
a cumbersome process requiring a formal background. On the other hand, there
are best-effort pattern-based tools, such as marjamaki2013cppcheck and
wheeler2006flawfinder, which can be readily used but do not provide any
guarantees. Several works (gentsch2020evaluation; chatzieleftheriou2011test;
moerman2018evaluating; lu2018evaluating) evaluate these tools on non-embedded
software and show that they vary in precision, recall, and usability. There
are also many commercial SAST tools. Coverity is considered state-of-the-art
and allows developers to customize the tool to reduce false positives
(imtiaz_how_2019). Other notable tools include Fortify
(noauthor_fortify_nodate), Checkmarx (noauthor_checkmarx_nodate), and Veracode
(noauthor_veracode_nodate).

####  2.2.2. CodeQL

This is a recent SAST tool created and maintained by Microsoft. CodeQL
represents code as a relational database and uses relational queries to find
defects in the given codebase. It has several static analysis capabilities,
such as control flow analysis, data flow analysis, and taint tracking to
detect security issues (avgustinov2016ql). Furthermore, CodeQL has built-in
queries for common security issues (i.e., Common Weakness Enumerations).
Security analysts and developers have used CodeQL to find thousands of
security vulnerabilities in large codebases, including the Linux kernel
(codeqlwof; codeqlwof2; codeqlwof3).

###  2.3.  Continuous Integration (CI) Workflows

Continuous Integration (CI) pipelines or Workflows (humble2010continuous) have
become ubiquitous in the software development lifecycle. They automate various
software development processes, such as building, testing, and deploying code.
The GitHub CI with its close integration with GitHub infrastructure, is the
most popular CI framework for the projects hosted on GitHub (9825792). GitHub
CI supports  _Actions_ , i.e., modules or plugins that enable easy development
of Workflows. More than 19K actions are available on GitHub Marketplace
(githubmarketplace). For instance, one can use cmakeaction to build a
bajpai2022secure,mangla2023securing,mansfield2018devops show that CI Workflows
provide a perfect place to run SAST tools as they can be easily integrated
into the development pipeline. Furthermore, there are several GitHub Actions
(Table 2) that enable running various SAST tools, including CodeQL, as part of
a GitHub Workflow.

##  3\. Motivation

Most embedded software are developed in unsafe languages, i.e., C/C++. Many
works (nguyen2021adoption; santoso2021implementation; phan2023challenges;
imtiaz2019synopsis) emphasize the importance of using SAST tools on software
projects, especially those using unsafe languages such as C/C++. Many security
and government organizations (cernsecurity; gsasecurity) also recommend the
use of SAST tools. Many software engineering tasks are being automated in CI
pipelines, which provide an ideal place to use SAST tools. Our analysis shows
that 958 (19%) of top 5K critical and extremely critical OSS projects
(according to OSSF criticality score explained in Section 4.1.3) on GitHub use
one or more SAST tools as part of their CI pipelines, i.e., GitHub Workflows.
It is important to ensure that SAST tools are also used in EMBOSS. Many
effective SAST tools require compilation of the underlying software and assume
certain coding idioms (e.g., use of standard libraries). These tools are often
evaluated (gentsch2020evaluation; chatzieleftheriou2011test;
moerman2018evaluating; lu2018evaluating) on traditional or non-embedded
software. However, embedded software differs (white_making_2011) from
traditional software in design, library usage, organization, build system, and
toolchains. _It is unclear how challenging it is to use existing SAST tools
and their effectiveness on EMBOSS_. A decade ago, Torri et al.
(torri_evaluation_2010) surveyed ten different free/open-source SAST tools and
evaluated their use on five embedded applications. Their results found that
the tools present widely different results, and most are not ready to be
applied to embedded systems. Notably, numerous modifications were required to
the source code and needed to deal with diverse build processes. However,  _no
work exists to understand the prevalence and effectiveness of SAST tools in
EMBOSS at scale_.

##  4\. Study Methodology

Our study addresses three research questions:

  * •

RQ1-Prevalence (Section 5): Are SAST tools currently used in EMBOSS?

  * •

RQ2-Challenges (Section 6): Is there any difficulty in configuring SAST for
EMBOSS?

  * •

RQ3-Effectiveness (Section 7): Can EMBOSS benefit from using SAST tools?

We use mixed-methods study, systematic analysis, and large-scale evaluation to
investigate these questions. We start with details of our data collection
methodology and tool selection. Then, we present the methodology, results, and
analysis for each research question in turn.

###  4.1. Embedded Software Dataset

We aim to collect a set of representative and popular EMBOSS. Embedded systems
are usually powered by an RTOS, which provides the necessary library and
scheduling support for various application components. To collect a
representative EMBOSS dataset, we use a two-pronged approach as shown in
Figure 1.  Figure 1. Our two-pronged approach to collecting embedded software
dataset.

####  4.1.1. GitHub Crawling

We searched (on Apr 8, 2023) for popular embedded software on GitHub.
Specifically, we collected original (i.e., non-forked), active (i.e., non-
archived) C/C++ embedded software. Figure 1 shows the exact filters for our
search. We sorted the resulting repositories according to their popularity
(i.e., number of stars) and collected the top 250. Also, all these
repositories have more than 100 stars, indicating their popularity. We
manually checked each repository to filter false positives by removing non-
embedded repositories. For instance, we filtered out a machine learning
project that also contained the word “embedded” in its keywords. In total,
this resulted in 238 repositories.

####  4.1.2. Well-Known Sources

We collected RTOSes from osrtos.com (osrtosweb), which maintains the list of
all open-source RTOSes released to date. Specifically, we selected those
available on GitHub and with more than 100 stars. This resulted in a total of
32 repositories.

####  4.1.3. Summary

We combined the repositories and de-duplicated them, resulting in a total of
258 unique EMBOSS repositories. Table 4.1.3 shows the summary of all
repositories along with their fine-grained categorization (performed
manually). Appendix E in our extended report (extendedreport) shows Source
Lines of Code (SLOC) statistics of these repositories. Most repositories are
reasonably large, with a median ranging from 10K - 100K SLOC, which agrees
with numbers published in other studies (10.1145/3589610.3596271). Measuring
Importance of Software in the Dataset Using the OSSF Criticality Score The
Open Source Security Foundation (OSSF) organization (ossforganization) created
a mechanism to compute a criticality score for GitHub repositories. This score
is not commonly used in the empirical software engineering literature, but it
is receiving attention in practice (mawanted; 10.1145/3543873.3587336) because
it is useful for security analysts scanning large datasets to triage the
security vulnerabilities. Since SAST tools are often used for this purpose, we
define the OSSF criticality score here. The OSSF criticality score
(ossfcriticalityscore) of a repository is a number ranging from 0.0 - 1.0
intended to measure the importance of a repository. The score is computed
through a formula that incorporates measures of popularity, dependents,
activity, and other attributes. A score $s$ from $0.0\leq s<0.2$, $0.2\leq
s<0.4$, $0.4\leq s<0.6$, $0.6\leq s<0.9$, $0.9\leq s$ indicates low, medium,
high, critical, extremely critical severities, respectively. For instance, The
Swift language frontend (with 2.4K stars) (swiftsyntax) and contiki-os (an
RTOS with 3.6K stars) (contikios) have criticality scores of 0.51, indicating
high severity projects. The Linux kernel (with 157K stars) (torvaldslinux) has
a criticality score of 0.88, indicating a critical project. The Node.js
runtime has (with 97.6K stars) (nodejsnode) has a score of 0.99, indicating an
extremely critical project. In our EMBOSS dataset, most of the 13 categories
have projects with a median criticality score ranging from 0.4-0.5, indicating
projects of considerable importance. The box plot of OSSF criticality score
(Section 4.1.3) for EMBOSS in each category is in Appendix E of our extended
report (extendedreport).

Table 1.  Categorization of repositories in our EMBOSS dataset. _Total_ :
Medians across corpus, not by category.

|  |  | Median | Median | Median
---|---|---|---|---|---
Category | # Repos | Example Repo | GH stars | SLOC | Crit. Score
Hardware access library (HAL) | 18 | grbl | 303.5 | 98,502 | 0.44
Device drivers (DD) | 10 | TinyUSB | 452 | 20,078 | 0.41
Network (NET) | 54 | contik-ng | 314 | 36,345 | 0.46
Database access libraries (DAL) | 8 | tiny SQL | 659 | 26,977 | 0.39
File systems (FS) | 5 | littlefs | 401 | 11,195 | 0.49
Parsing utilities (PAR) | 10 | json library, nanopd | 313.5 | 2,547 | 0.41
Language support (LS) | 33 | micropython | 479 | 33,389 | 0.42
UI utilities (UI) | 14 | flutterpi | 584.5 | 56,712 | 0.46
Embedded applications (APP) | 32 | Infinitime | 508 | 22,662.5 | 0.39
OSes (OS) | 42 | FreeRTOS, Zephyr | 727.5 | 409,667.5 | 0.47
Memory Management Library (MML) | 4 | tinyobjloader-c | 242.5 | 6,205.5 | 0.34
Other General Purpose Library |  |  |  |  |
for Embedded Use (GPL) | 22 | tinyprintf | 391 | 12,742.5 | 0.35
Other (OT) | 6 |  | 368.5 | 94,805 | 0.43
Total | 258 |  | 406.5 | 33545 | 0.43

###  4.2. Usable SAST

Our goal is to find SAST tools that can be readily used on the collected
GitHub repositories. Given that GitHub Actions are expected to be stable, easy
to use, and can be seamlessly integrated into repositories, we used GitHub
Marketplace and found GitHub Actions designed for SAST purposes. 111The query
is category=security&type=actions&query=“C C++” and category=code-
quality&type=actions&query=“C C++”. We manually filtered out pre-release
Actions due to their instability and/or lack of documentation. There were 6
commercial SAST tools, which we omitted as they require purchases of licenses
or subscriptions and place restrictions on scientific publications.
Furthermore, for individuals, small teams, or organizations with limited
financial resources, the cost of commercial tools may be prohibitive, making
them less feasible compared to free or open-source alternatives.

####  4.2.1. Summary

This resulted in a total of 12 GitHub Actions using various SAST tools as
shown in Table 2. Nine of these Actions are plug-and-play, meaning they do not
need any repository-specific configuration. In other words, the steps to use
the Action do not vary with the underlying repository. For instance, the
CodeQL Action (i.e., github_inc_codeql_2023) is a plug-and-play Action because
every repository uses the same steps to use the Action, whereas the frama-
c_frama-ceva_2023 requires developers to create a special Frama-C Makefile and
provide the path to it. Effectiveness on C/C++ Juliet Test Suite
(black2018juliet): Recent work (ampn24) shows developers prefer plug-and-play
SAST tools as they do not need project-specific configuration. To measure
baseline effectiveness, we tested all plug-and-play tools on the Juliet Test
Suite, a labeled dataset commonly used to test SAST tools
(julietusedtoevalsasttools). The last column of Table 2 shows the results.
Except for CodeQL, all other tools either failed or did not finish within 6
hours, which exceeds the maximum time allowed for a job by many CI platforms,
such as GitHub CI (githubusagelimits). CodeQL took 40 minutes. The reasons for
failure include 1) the Static Analysis Results Interchange Format (SARIF) file
produced by the tool is invalid, and 2) the argument list given to the SAST
tool is too long. We checked the results of CodeQL and found that it raised
11,101 warnings with a precision of 71% (7,904/11,101).222We count a reported
flaw as a true positive if the reported location matches that of a ground
truth bug.

Table 2.  List of “usable” SAST GitHub Actions. These are GitHub Actions that perform SAST on C/C++ repositories, not including pre-release or commercial tools.  Name of GitHub Action | Plug-and-play? | Underlying tool(s) | Juliet Test Suite results
---|---|---|---
From Well-established Organizations
github/codeql-action (github_inc_codeql_2023) | Yes | CodeQL | 7,904 true positives
cpp-linter/cpp-linter-action (cpp-linter_cc_2023) | Yes | clang-format, clang-tidy | Not finished in 6 hours
trunk-io/trunk-action (trunk-io_trunkio_2023) |  | No
---
(Bazel/CMake projects required)
| clang-format, clang-tidy,
---
include-what-you-use,
pragma-once
N/A
Frama-C/github-action-eva-sarif (frama-c_frama-ceva_2023) |  | No
---
(Frama-C Makefile required)
Frama-C | N/A
From Independent Developers
IvanKuchin/SAST (ivankuchin_cc_2021) |  |  |
deep5050/flawfinder-action (pal_flawfinder-action_2022) |  |  |
david-a-wheeler/flawfinder (wheeler2006flawfinder) | Yes | flawfinder | Error
Syndelis/cpp-linter-cached-action (lemos_cc_2023) | Yes | clang-format, clang-tidy | N/A
deep5050/cppcheck-action (pal_deep5050cppcheck-action_2023) |  |  |
Konstantin343/cppcheck-annotation-action (konstantin343_cppcheck_2022) | Yes | cppcheck | Not finished in 6 hours
JacobDomagala/StaticAnalysis (domagala_static_2023) | Yes | cppcheck, clang-tidy | Error
whisperity/codechecker-analysis-action (whisperity_codechecker_2023) |  | No
---
(Compilation DB required)
clang | N/A

##  5\. RQ1: Prevalence of SAST Tools

###  5.1. Method

In this research question, we plan to understand the prevalence of SAST tool
usage in EMBOSS repositories. In general, it is impossible to automatically
identify whether an EMBOSS project uses SAST tools because developers might
use them out-of-band with no hints/indications in the corresponding
repository. e.g., developer might manually run a SAST tool before every
release. We perform a mixed-methods study by Workflow analysis and developer
surveys to investigate this research question.

###  5.2. Workflow Analysis

We noticed that 42% (109/258) of the EMBOSS repositories use GitHub Workflows
to automate building and testing the underlying codebase. As mentioned in
Section 2.3, these Workflows are an ideal place to use SAST tools. We
performed an automated analysis of Workflows in each repository to detect the
usage of SAST tools. Specifically, for each Action used in a Workflow, we
check if it is a SAST tool by checking its category (Section 4.2). Next, we
manually check every matching Action to validate that it is indeed a SAST
tool. We found that 248 of the repositories  _do not use_ any SAST tool, 10 of
them use free SAST tools, specifically, CodeQL, and none of them use
commercial SAST tools. Upon further analysis, we found that of the 10
repositories that use CodeQL, 7 are mis-configured (use a deprecated Action
version) while the other 3 configure it correctly. Given the large number of
repositories (248) and complexity of Workflows, it is impractical to manually
verify all the negative results, i.e., Workflows missing SAST tools. Instead,
we performed a random sampling of 20 Workflows and manually checked them. We
found only two false negatives, i.e., 10% false negative rate. Specifically,
InfiniTimeOrg/InfiniTime runs Developer Survey We did an anonymous online
survey with repositories’ maintainers to identify out-of-band usage of SAST
tools. First, we identified all the repositories for which we did not find any
SAST tool usage in their Workflows. For each of these repositories, we
collected emails (from the public GitHub profile) of users who merged recent
pull requests and/or made commits directly to the repository and sent them an
email with the link to our survey. 333This is consistent with GitHub’s
Acceptable Use policy (github_inc_github_acceptable_use_2023): profile emails
are public information and the number of emails sent was small enough to not
qualify as mass spam. We were able to find the maintainers’ email only for 104
(out of 258) projects. Our study is approved by the Institutional Review Board
(IRB) under #2023-1062. The exact questions of the survey are listed in
Appendix F of our extended report (extendedreport). We got responses for 14
projects (a 13% response rate, comparable with other works that survey
developers from GitHub). Figure 2 shows the summary of responses.

Figure 2. Summary of our developer survey on the use of SAST tools.

Use of SAST Tools. 71% (10/14) of the projects claim to use SAST tools, out of
which 80% (8/10) claim to be using them as part of their GitHub Workflows.
This shows that developers are aware of the possibility of using SAST tools in
GitHub Workflows. However, only one project uses an explicit SAST tool (i.e.,
clang static-analyzer). Our automated analysis (Section 5.2) did not find this
as the tool might be used as part of a script (e.g., SAST tools. Our automated
analysis did not check for these flags, which could be used as part of the
build scripts. As we show in Section 7, stringent compiler warnings are
ineffective at detecting security vulnerabilities. Only  _40% (4/10) projects
claim to use SAST tools out-of-band (i.e., outside of Workflows)_. Only 2 of
those 4 projects use SAST tools exclusively out-of-band (i.e., undetectable by
looking at Workflows). This suggests that analyzing Workflows is effective to
identify the use of SAST tools. Not Using SAST Tools. 29% (4/14) of the
projects do not use any SAST tools. Most of these engineers (3 of the 4
projects) believe the security vulnerabilities in the corresponding projects
have a low impact. However, these projects have OSSF criticality scores of
$\sim$0.3 (medium), $\sim$0.4 (high), and $\sim$0.6 (critical), respectively.
These developers may underestimate the severity of security issues, confirming
previous studies (xie2011programmers; 8816991). The developers of another
project reported insufficient resources (e.g., time). Unfortunately, this
project is one of the most popular (7.3K stars) open-source C++ library suites
for embedded systems and is used in many critical embedded projects with OSSF
score of 0.67 (a critical project). Comments on Using SAST Tools. A few (5)
developers mentioned that the effectiveness of SAST tools on embedded software
is questionable and might result in many false positives.

###  5.3. Results

Our methods show that most of EMBOSS repositories do not use SAST tools.
Specifically, our Workflow analysis show that only 10 (4%) repositories use
SAST tools as part of their CI Workflows. Merely 3 (1%) repositories correctly
use SAST tools, specifically CodeQL, a small number compared to non-embedded
repositories, where many similarly-popular repositories (958 of $\sim$5000 or
19%) correctly use it (Section 3). Our developer survey also revealed similar
results, wherein most repositories do not use any explicit SAST tool.
Furthermore, developers mentioned using strict compiler warnings.
tcbcolupperFinding 1: Most (97%) of the EMBOSS repositories  _do not use SAST
tools_. Finding 2: Our survey indicates that many EMBOSS repositories rely on
compiler warnings instead of dedicated SAST tools. Finding 3: Our survey shows
that most developers are aware of CI Workflows and use them to run their SAST
tools.

##  6\. RQ2: Challenges in Effectively Using SAST Tools

In this research question, we want to know how challenging it is to
effectively use SAST tools on EMBOSS repositories. We focus on SoTP tools as
listed in Table 2. In particular, we focus on plug-and-play tools, which have
a fixed set of steps for any repository.

###  6.1. Method

For each SAST tool, we picked the most popular Action implementing it. For
instance, for david-a-wheeler/flawfinder — this repository has the most stars
among the Actions offering this tool. Table 3 shows Actions selected for each
SAST tool. For each selected Action, we reviewed the documentation in order to
create a Workflow for the Action. We created a Workflow for each Action and
executed it on each repository.

###  6.2. Results

Table 3 shows the results of this experiment. Table 3. Results of SAST tools
on EMBOSS repositories.  Action Result format # Success Repo # Failure Repo
Reasons for failure Total # warn Median # warn Precision david-a-
wheeler/flawfinder SARIF 176 82 Invalid SARIF, Python Error 4,637 12 20%
(64/316) cpp-linter/cpp-linter-action GCC error msg 230 28 Timeout, Python
Error 212,228 111 0% (0/213) deep5050/cppcheck-action GCC error msg 256 2
Timeout 31,873 19 58% (116/200) CodeQL Autobuild SARIF 74 184 Autobuild
failure 471 0 96% (154/160)

####  6.2.1. Tool Execution

The encapsulation of all the steps to set up (e.g., dependency installing) a
tool in GitHub Action makes it straightforward to run each tool on all the
repositories in the EMBOSS dataset. More concretely, it requires placing the
tool Workflow file in Tools Failures Except for CodeQL, all Actions ran
successfully on most of EMBOSS repositories. Nonetheless, there are many
repositories where Actions failed, and Table 3 also shows the common reasons
for the failure. Despite the success of CodeQL on Juliet Test Suite (Table 2),
CodeQL failed on the majority of EMBOSS repositories. The reason for this is
the failure of the make, ARMmbed/DAPLink uses a customized Python script to
build the project. Such custom build processes cannot be handled by Tools
Effectiveness As shown in Table 3, produce results in SARIF format, which
makes it easy to navigate to the source location corresponding to a warning.
However, the other tools, i.e., cpp-linter, and cppcheck, produce a non-
standard textual message, making triaging warnings tedious. This is in line
with the observation made by a recent study (ampn24), where developers were
concerned about ineffective error reporting mechanisms of SAST tools. Table 3
also shows the total and median number of warnings across all repositories on
which the corresponding tool ran successfully. At a high level, except for
cpp-linter (with 111), all other tools have a moderate number (¡ 20) of median
warnings. The median warnings are 0 for CodeQL because it did not produce
warnings on more than 50% of repositories. The lack of ground truth data on
our EMBOSS repositories makes it challenging to evaluate the precision/recall
of each tool. Given the large number of warnings, it is infeasible to check
them all. Therefore, we randomly sampled warnings to determine if they were
legitimate. Specifically, for each tool, we randomly selected 30 repositories
with fewer than 20 warnings and manually checked each warning for these
repositories. The last column in Table 3 shows the precision. CodeQL has the
highest precision of 96% — this is unsurprising given its effectiveness on
Juliet Test Suite (Table 2). At the other end of the spectrum, cpp-linter had
the least precision, 0%!. Most of cpp-linter’s warnings were related to
missing header files, pre-processor directives, and other compile-time errors.
cpp-linter uses a fixed mechanism for compilation and cannot handle custom
build setup in each repository, resulting in false positives.
tcbcolupperFinding 4: Few SAST tools produce warnings in a non-standard text
format, hindering their usability. This aligns with a recent study (ampn24),
where developers raised this as a concern for not using SAST tools. Finding 5:
Except for CodeQL, all other SAST tools executed successfully on the majority
of EMBOSS repositories. CodeQL tries to compile the target repository
automatically but fails to handle the diverse build infrastructure of the
majority (184 (71%)) repositories. Finding 6: Our preliminary evaluation based
on random sampling shows that CodeQL has the highest precision on EMBOSS
repositories.

##  7\. RQ3: Effectiveness of SAST Tools on EMBOSS

We want to understand if EMBOSS can benefit from using an effective SAST tool.
Specifically, we want to validate the developers’ perspective that SAST tools
may not be effective on embedded software (Section 5.2). In Section 6, we ran
SoTP SAST tools on EMBOSS repositories. However, given the large number of
warnings, it is infeasible to verify all of them manually. Therefore, in this
section we focus on the best tool identified so far.

###  7.1. Method

We select the most effective SAST tool among those we considered so far. We
configure it for each repository in our corpus. Then, we manually analyze all
the alerts and warnings to understand its effectiveness specifically on
EMBOSS.

####  7.1.1. Tool Selection: CodeQL

We picked CodeQL for our experiment as it satisfies all our requirements.
First, it is the most effective open-source tool for bug finding based on a
previous study (lipp_empirical_2022) and also based on our evaluation of
CodeQL on the Juliet Test Suite (Table 2) and random sampling (Section 6.2.1).
Second, it is maintained by Microsoft and has an active and responsive
community. Finally, several reports (codeqlwof; codeqlwof2; codeqlwof3) show
that CodeQL was able to find several high-impact security vulnerabilities in
large and well-tested codebases. Furthermore, CodeQL’s extensive documentation
(codeqldocumentation), automated scanning (codeqltools), and various other
support tools (codeqlmicrosoft) make it one of the easiest tools to integrate
in CI Workflows. We used CodeQL command-line toolchain release 2.13.1 (May 3,
2023) and the query repository based on the commit 202037e925 (May 12, 2023).

####  7.1.2. Build Scripts Creation

As described in Section 6.2.1, CodeQL failed to compile various (184) EMBOSS
repositories because of using non-standard build setups. We tried to manually
create build scripts for these repositories by referring to their
documentation and CI scripts. We also made the build scripts cover as much
part of the codebase as possible (e.g., by compiling all example applications
and all supported architecture and boards whenever possible). Even for the
repositories where CodeQL’s We were able to successfully create build scripts
for 154 (60%) repositories. For the other 104 repositories, the build
instructions were either missing (17), too complex (i.e., unavailable
toolchains or dependencies) (51), or did not work (36). _On average, it took
45-60 minutes to create self-contained build scripts for each repository_.

####  7.1.3. Configuring CodeQL

CodeQL supports many suites (i.e., collections of queries). We choose Excluded
Queries: We omitted 9 queries from the CodeQL suite for three reasons: (1)
they identify code smells but not necessarily defects; (2) the potential risk
of the corresponding defects is relatively low (e.g., converting the result of
an integer multiplication to a larger type is an issue only when the result is
too large to fit in the smaller type); or (3) they are not applicable to
embedded software. Appendix G in our extended report (extendedreport) shows
the complete list of excluded queries and our detailed rationale. Modified
Queries: We modified 3 queries to improve their precision and ignore certain
restrictions. Appendix H in our extended report (extendedreport) describes all
3 changes. For example, we modified accepted one of our query modifications
into their main repository (codeqlpullrequest).

####  7.1.4. Workflow Creation

We created GitHub Workflows for all the successfully built repositories (154).
These Workflows, when triggered, invoke the necessary build scripts and run
CodeQL with the required configuration.

###  7.2. Results

The scrutiny of the results demanded the dedicated efforts of two authors
spanning 25 days. We analyzed the results of 143 (out of 154) repositories.
The remaining repositories have a substantial number of defects, and we did
not have sufficient time to analyze them thoroughly. In total, across all the
143 analyzed repositories CodeQL reported 578 errors (potential defects) and
2,294 warnings (code-smells, undefined behaviors, etc.). Table 4 summarizes
the results. Table 4. Summary of CodeQL results and their analysis. Number of
… Value Setup Repos in dataset 258 Repos built 154 Repos analyzed 143 CodeQL
Results Errors reported 578 Warnings reported 2,294 Manual Analysis Defects
discovered 540 Repos where defects were discovered 83 (60%) Security defects
discovered 399 Repos where security defects were discovered 71 (51%)
Responsible Disclosure Defects confirmed 273 Security defects confirmed 219
Pull requests raised 139 Pull requests merged 81 CVEs issued 2

####  7.2.1. Defects Discovered

We discovered 540 defects (spanning 83 repositories), of which 399 are security vulnerabilities (spanning 71 repositories). Note that multiple errors or warnings might be raised for a single defect. Figure 3 shows the number of defects found across various repositories according to their categories. _At a high level, across all categories, the number of security defects is more than that of the number of non-security defects._ Furthermore, the number of defects is proportional to the number of repositories of the particular category (Table 4.1.3). For instance, Network (NET), Operating Systems (OS), and Applications (APP) are the top three categories containing the highest number of repositories (128 (50%)), and they also contain the highest number of defects (344 (64%)). The Memory management libraries with the least number (4) of repositories also have the least defects (1). Also, the number of defects is proportional to the size of the codebase. For instance, although NET has more repositories than OS, i.e.,. The median SLOC of OS, i.e., 409K, is higher than NET, i.e., 36K. Consequently, the number of defects in OS repositories is larger than NET, i.e., 166 v/s 96. _In summary, the density of defects remains constant across various EMBOSS repositories_. HAL(4)DD(2)NET(18)DAL(3)PAR(3)LS(11)UI(7)APP(10)OS(15)MML(1)GPL(5)OT(4)$0$$50$$100$$150$$1$$15$$13$$4$$10$$4$$32$$47$$10$$5$$13$$3$$81$$12$$3$$45$$40$$50$$119$$1$$8$$24$Number of defects |  | Security |  |  |  | Non-security |
---|---|---|---|---|---|---|---
Figure 3. Number of defects of each type in EMBOSS of various categories
(Table 4.1.3). The number next to the category indicates the number of
repositories containing at least one defect. Defects in Each Repository.
Figure 5 shows CDFs of the number of all defects and security defects in each
repository. A point $(x,y)$ on a line indicates that $y$% of repositories
contain less than $x$ corresponding type of defects. The left-most point on
both the lines indicates that there are 60% (83) repositories with at least
one defect, and 51% (71) repositories with at least one security defect. The
security defects line has almost the same trend as total defects, indicating
that most defects in all repositories are security-relevant. Although 90% of
the repositories have less than ten total defects, there are still a
considerable number (8) of repositories with a large number of defects. Table
5 shows the top 5 repositories with the highest total defects, security
defects, and their OSSF criticality score. Table 5.  EMBOSS repositories with
the top 5 highest total defects and security defects.  Total Defects Security
Defects Repo Criticality Score Num Repo Criticality Score Num contiki-
ng/contiki-ng 0.67 32 openlgtv/epk2extract 0.45 27 openlgtv/epk2extract 0.45
29 gozfree/gear-lib 0.43 24 ARMmbed/mbed-os 0.72 27 raysan5/raylib 0.70 23
introlab/odas 0.46 25 contiki-ng/contiki-ng 0.67 22 gozfree/gear-lib 0.43 24
jnz/q3vm 0.34 18

$0$$10$$20$$30$$0$%$20$%$40$%$60$%$80$%$100$%# defects in a repo% of repositories |  | All defects |
---|---|---|---
|  | Security defects |
Figure 4.  CDFs of # of all and security-relevant defects in a repository.

$0$$0.2$$0.4$$0.6$$0.8$$1$$0$%$20$%$40$%$60$%$80$%$100$%criticality score% of
security defects Figure 5. CDF of the severity of security defects

Common Types of Security Defects. We found several classes of security defects
across all repositories. Figure 7 shows the top 10 major types of security
defects (i.e., vulnerabilities) (github_inc_codeql_cpp_doc_2023) found along
with the corresponding number of defects. We will discuss the top three types
of defects and corresponding rules: • NULL , while most other calls to the
same function check the return for NULL to prevent subsequent null pointer
dereference. This rule detected 122 such instances. Appendix A in our extended
report (extendedreport) shows an instance of this issue from the
github_inc_overflow_2023 malloc) is a multiplication of operands derived from
potentially untrusted input (e.g., user input). When the operands hold a
really large value, an integer overflow (dietz2015understanding) might occur
and yield a significantly smaller value than intended. Hence, the size of the
allocated memory may be considerably less than expected. Subsequent attempts
to access the allocated buffer would lead to buffer overflows. This rule
detected 42 instances. Appendix A in our extended report (extendedreport)
shows an instance of this defect in the github_inc_unbounded_2023
pincus_beyond_2004. Specifically, this includes analysis of potentially
dangerous function calls (e.g., sscanf) to check whether these are used
properly with valid arguments. This rule detected 36 vulnerabilities of
potential buffer overflow. Appendix A in our extended report (extendedreport)
shows an instance of this vulnerability in the aws-iot-device-sdk-embedded-C
repository.

⬇ // apache/nuttx/drivers/sensors/apds9960.c ret = register_driver(devpath,
&g_apds9960_fops, 0666, priv); if (ret < 0) { snerr(”ERROR: Failed to register
driver: kmm_free(priv)@\textcolor{purple}{\faWarning}@; }
@\textcolor{red}{\faBomb}@priv->config->irq_attach(priv->config,
apds9960_int_handler, priv);

Figure 6.  The memory pointed by `c`an be freed () inside the `c`ondition but
will be accessed later outside, resulting in use-after-free ().

$0$$50$$100$$150$cpp/inconsistent-null-checkcpp/uncontrolled-allocation-
sizecpp/wrong-type-format-argumentcpp/unbounded-writecpp/missing-check-
scanfcpp/offset-use-before-range-checkcpp/incorrect-allocation-error-
handlingcpp/overflowing-snprintfcpp/constant-comparisoncpp/toctou-race-
conditioncpp/overrunning-write122 (36.2%)42 (12.5%)38 (11.3%)36 (10.7%)24
(7.1%)20 (5.9%)15 (4.5%)12 (3.5%)12 (3.5%)8 (2.4%)8 (2.4%)# security-relevant
defects Figure 7.  Top 10 CodeQL queries by # of security-relevant defects
found. (Counts and Percentages)

Severity of Security Defects. The severity of a security bug depends on its
exploitability and the criticality of the underlying software
(10.1145/2989238.2989239; 9825784). Given the large number of defects,
manually assessing exploitability is intractable. Instead, we use the OSSF
criticality score (Section 4.1.3) of the target repository to assess the
severity of a bug. Figure 5 shows the CDF of the severity of security defects.
Specifically, a point $(x,y)$ on the line indicates $y$% of the defects have
severity less than or equal to $x$. 50% ($\sim$ 100) bugs have a severity
score of more than 0.5, which represents high-severity repositories (Section
4.1.3). Specifically, 40% of bugs have a score of more than 0.6, representing
vulnerabilities in critical repositories. For instance, we found an off-by-one
error in lst:off-by-one-bug) and a use-after-free in
lst:useafterfreeinsensor), a RTOS with a score of 0.69 – both of these are
critical projects. Common Types of Non-Security Defects. These defects may not
lead to security vulnerabilities but can cause functionality issues, undefined
behavior, and compilation issues. For instance, the rule return detects non-
void functions with no explicit return statement. This may result in undefined
behavior during runtime (linearity_omitting_2010). Similarly, the rule virtual

carnegie_mellon_university_oop50-cpp_2020. We provide more details about the
major types of non-security defects in Appendix C of our extended report
(extendedreport).

####  7.2.2. Precision

To study the precision of CodeQL results, we sampled 37 repositories and manually categorized all errors and warnings into true and false positives. A false positive means that the result does not match what the rule intends to detect, e.g., an error for an uninitialized variable when it is actually initialized. True positives are further classified as true defects and harmless reports. A harmless report means that although the result matches the rule’s intention, it is not a defect in the specific context of the code. e.g., NULL check of its return value. Although the warning is true, it might be infeasible in the specific context. The overall percentages of true and false positives are 77% (362/468) and 23% (106/468), respectively. Out of 362 true positives, there were 158 harmless reports. Figure 8 shows the CDF of the percentage of rules and their contribution to different types of results. Specifically, a point $(x,y)$ on a line indicates $y$% of the rules have resulted in less than or equal to $x$% of the corresponding results. Approximately 60% rules had no false positives, and 22% had no true positives. This indicates that false positives are polarized, and a few rules contribute to the majority of false positives. We provide a detailed discussion on rules contributing to the false positives in Appendix B of our extended report (extendedreport).  $0$%$20$%$40$%$60$%$80$%$100$%$0$%$20$%$40$%$60$%$80$%$100$%% of defect types% of Rules |  | True defect% |
---|---|---|---
|  | Harmless report% |
|  | False positive% |
Figure 8. CDFs of the percentages of rules and corresponding defect types. //
micropython/extmod/vfs_lfsx.c size_t from = 1; char *cwd = vstr_str
&self-¿cur_dir); while (from ¡ CWD_LEN) for (; cwd[from]@@ == ’/’ from ¡
CWD_LEN@@; ++from) // Scan for the start … The offset `i`s used before the
range check (), leading to an off-by-one error ().

###  7.3. Responsible Disclosure and Developer Response

We responsibly disclosed all the identified defects either by raising issues
or pull requests with appropriate patches (where possible). The bottom part of
Table 4 shows the summary of our responsible disclosure. In total, 51%
(273/540) of defects have been confirmed by developers (via merging our pull
requests or expressing confirmation in replies to issues). Most of the patches
were readily accepted by the developers. In a few cases, developers were even
interested in knowing the techniques we used to find the defects. For
instance, developers of `s`aid  _‘‘I’m curious how you stumbled across this
--- Was there some sort of test you ran or was this something that came up
during your development? I’m hoping we can duplicate your method of discovery
to add some sort of check/test to the repo.’’_ We are in the process of
raising pull requests to integrate our CodeQL Workflows into corresponding
repositories. There were two pull requests where the developers did not choose
to fix potential security issues. They stated that although code robustness is
important, they deemed reduced code size and RAM usage to be a higher priority
in their embedded software. These observations support the conventional wisdom
that software engineers (and especially engineers in embedded systems) trade-
off between security and performance (fujdiak_seeking_2018;
gopalakrishnaIfSecurityRequired2022).

####  7.3.1. CVE Assignment

Our research confirmed the observation of prior work (9152613), that security
defects are often fixed ‘‘silently’’, without tracking via a Common
Vulnerability Enumeration (CVE). When we disclosed the security-relevant
defects, we did not explicitly ask the engineering teams to issue CVEs. Of the
94 repositories against which we opened at least one security-relevant defect,
only two issued CVEs for these defects: `i`ssued CVE-2023-BLINDED, and
`i`ssued CVE-2023-BLINDED. We eventually followed up on our 77 reports of
defects to the 10 most popular repositories (by GitHub stars) to inquire
whether CVEs were being prepared. Two of the engineering teams replied
suggesting that we email their security teams --- we did so, but received no
response. The other eight teams did not respond.

###  7.4. Effectiveness of Stringent Compiler Flags

As reported in Section 5.2, EMBOSS developers often use strict compiler
flags/warnings instead of SAST tools. We evaluated the effectiveness of these
flags in finding the defects detected by CodeQL. We used security bug test
case files from the CodeQL repository for this experiment. These are simple
test cases (< 10 lines), each containing an obvious security issue, e.g.,
passing an invalid pointer types to a function call. We selected test cases to
cover all 82 of the identified defect types and compiled them using the latest
version of `,` i.e., 11.4.0, with strict warnings ( `)`. This configuration of
`f`ound issues in only 17 (21%) defect types (details in our extended report
(extendedreport)). `w`as able to find certain simple security issues, such as
direct use of `.` However, it did not find more complex ones related to code
quality, such as inconsistent null check. Our results indicate that the
current EMBOSS practice of reliance on `w`arnings is inadequate.

  *[EMBOSS]: Open-Source Embedded Software
  *[SAST]: Static Application Security Testing
  *[EmS]: Embedded Software
  *[OSS]: Open-source Software
  *[RTOSes]: Real Time Operating System
  *[SoTP]: State of The Practice
  *[CI]: Continuous Integration
  *[ SAST]: Static Application Security Testing
  *[RTOS]: Real Time Operating System
  *[SLOC]: Source Lines of Code
  *[OSSF]: Open Source Security Foundation
  *[SARIF]: Static Analysis Results Interchange Format
  *[ EMBOSS]: Open-Source Embedded Software
  *[CVE]: Common Vulnerability Enumeration