∎

11institutetext: Fang Hou 22institutetext: Department of Information and
Computer Science, Utrecht University, the Netherlands
22email<EMAIL_ADDRESS>33institutetext: Slinger Jansen 44institutetext:
Department of Information and Computer Science, Utrecht University, the
Netherlands
School of Engineering Science, Lappeenranta University of Technology, Finland
44email<EMAIL_ADDRESS>
# A Systematic Literature Review on Trust in the Software Ecosystem

Fang Hou  Slinger Jansen

(Received: date / Accepted: date)

###### Abstract

Background: The worldwide software ecosystem is a trust-rich part of the
world. Throughout the software life cycle, software engineers, software end-
users, and other stakeholders collaboratively place their trust in major hubs
in the ecosystem, such as package managers, repository services, and software
components. However, as our reliance on software grows, this trust is
frequently violated by bad actors and crippling vulnerabilities in the
software supply chain, endangering our livelihoods and the resilience of our
society.

Purpose: The purpose of this study is to define trust in the worldwide
software ecosystem, that is, to determine what signifies a trustworthy system
or actor. By improving our understanding of trust, we can create novel
mechanisms for protecting the software ecosystem from vulnerabilities and
attacks.

Methods: We conduct a systematic literature review on the concept of trust in
the worldwide software ecosystem. We acknowledge that trust is something
between two actors in the software ecosystem, and we examine what role trust
plays in the relationships between end-users and (1) software products, (2)
package managers, (3) software producing organizations, and (4) software
engineers.

Results: Two major findings emerged from the systematic literature review. To
begin, we provide a definition of trust in the software ecosystem, including a
theoretical framework that decomposes and signifies a theoretical
understanding of trust. Second, we provide a list of trust factors that can be
used to assemble an overview of software trust.

Conclusion: Trust is critical in the communication between actors in the
worldwide software ecosystem, particularly regarding software selection and
evaluation, and can affect the outcome either positively or negatively.
According to the frequency of the trust factors in the literature, the most
significant determinants of trust are software quality, code and structure,
and the reputation of software producing organizations. With this
comprehensive overview of trust, a new foundation is laid for software
engineering researchers to understand and use trust to create a reliable
software ecosystem.

Statements and Declarations

The authors have no relevant financial or non-financial interests to disclose.

No funding was received to assist with the preparation of this manuscript.

Author Slinger Jansen is an Editorial Board Member of Empirical Software
Engineering.

###### Keywords:

Software ecosystem Software trust Software security Software package
evaluation

##  1 Introduction

When software engineers are building software, for example, in-house software,
they are constantly being confronted with a choice: build it themselves or
integrate a component from another software producing organization. If the
choice is made to integrate a component, the second choice is a selection
problem: there are typically many options to select from and it depends on the
software engineer’s perspective which factors heaviest in the selection
process. Factors such as feature completeness, software component quality, and
technology can all determine whether the software engineer selects a
component, while researchers have called for more structured approaches to
software selection (farshidi2021decision (48)), the selection process is
typically completely dependent on the context of the software engineer.

The software ecosystem is a trust-rich part of the world. End-users expect a
certain trustworthiness when downloading apps from their favorite application
store. For instance, when the appstore provides checksums, a technology to
determine the authenticity of the received data, for each deliverable, the
end-user can ensure that the data she receives is the same data that came from
the software engineer. Nevertheless, it is more often than not the case that
trust is put into Software Ecosystem (SECO) hubs such as package managers and
appstores, without there actually being a good basis for this trust. Trust can
be considered as founded and unfounded. There are also more soft ways to
create founded trust, e.g., ensuring that the software engineer has been a
productive member of the SECO for a long time, or ensuring that the software
developed achieves established quality levels. Both soft and hard ways do not
provide watertight guarantees, as the external environment can also have an
impact on trust, e.g., new attacks from the external world. According to
Sonatype2021OSS (131), the number of supply chain attacks is dramatically
increasing, posing significant security risks to both software producing
organizations and end-users. For example, bad actors shift their attacks
“upstream” by inserting malicious code to the source code repositories to
acquire a critical advantage of time, allowing malware to spread throughout
the supply chain.

Since we have already mentioned trustworthiness and trust, it is necessary to
explain their differences, as we found that trust and trustworthiness are used
interchangeably, and often trustworthiness is mistakenly interpreted as trust
(wright2010trust (144)). Typically, trust is from the trustor’s perspective,
it is a belief that the trusted person will actually do what is expected of
her (jadhav2011framework (81, 64, 106, 9)), while the concept of
trustworthiness can be viewed as a probability, i.e., the probability that a
trustee will act as expected by the trustor (bauer2019conceptualizing (9)).
Hence, a trustee’s trustworthiness and a trustor’s trust may differ. In
addition, it is found that trustworthiness can be seen as a characteristic of
trust (becerra2008trustworthiness (11)), which means trust cannot exist
without a high level of trustworthiness (heyns2015dimensionality (72)).

Extensive research has shown that trust is crucial in software selection and
evaluation (bugiel2011scalable (17)). Typically, end-users trust in software
is built upon the two following aspects: one is the product should perform its
desired functions and meet their specific needs; the other is the software
producers should provide reliable services. Software trust differs from
software ecosystem trust in that it concentrates on the software products and
their producers. More information on software trust and software ecosystem
trust can be found in Section 4.1.3.

Currently, standards and models have been proposed to assess software trust,
for instance, the Trusted Computer System Evaluation Criteria (10.5555/1593439
(49)) and the Reliability Growth Model (littlewood1994learning (99)). However,
they are not widely used. The primary reasons are :

  * •

In light of the differences in the needs of end-users for various software
products, these models lack comprehensive and uniform criteria for measuring
the trust in all software products (yan2008comprehensive (147));

  * •

Most of the factors are abstract and lack clarity or guides that can be used
to quantify trust factors (li2021exploring (97));

  * •

There is a lack of breadth of perspective among the criteria for judging the
trustworthiness of software that practitioners would expect. For example, we
rarely receive ratings for various package versions and a few comparable
software producer ratings. Yet, this is precisely what software practitioners
are concerned about when assessing software quality (goode2015rethinking
(59)).

In this context, we conduct a systematic review of the literature (SLR) to
better understand software trust in the software ecosystem and summarize the
key impact factors based on the frequency with which they were mentioned in
the studies. The SLR presented here is part of a larger study of software
trust that breaks a lance by giving trust a proper role in this selection
process. By analyzing the perception and trust of various stakeholders, we
provide a comprehensive overview of the relevant trust concepts. The purpose
is to lay the groundwork for research on the design of trust assessment
mechanisms that will place trust at the center of the selection process for
software engineers.

To help understand this study, we develop a model in Figure 1 to depict the
relationship between significant trust impact factors and actors in the
software ecosystem. It can serve as a brief overview of the results of this
study, a detailed discussion of which is provided in Section 4.

This research begins with the following outline:

$\bullet$

Section 2 introduces the actors and hubs involved in software selection and
evaluation in the software ecosystem, as well as the workflow of the software
packages management. The purpose is to identify relevant trust impacted
elements in the software ecosystem;

$\bullet$

Section 3 presents the research method we used in this literature review,
which works as an acquisition process to collect the software trust concept
and trust factors considered by the software end-users during the software
selection;

$\bullet$

Section 4 provides a definition of software trust as well as a discussion on
the intrinsic and extrinsic trust factors in terms of software components,
package administrators, software production organization, and engineers,
respectively. The results show that quality is the decisive factor for
software trust, followed by software producers and code & structure.
Additionally, trust assessment models typically focus on security, such as
data access or user entitlement, rather than vulnerabilities and attacks;

$\bullet$

Section 5 highlights the effect of trust factors on the software selection,
validity, consideration, and challenges of this work, as well as our future
work. It is difficult to develop a consistent set of criteria for assessing
software trust; therefore, our view is that trust assessment should be
synthetic and drawn on from different perspectives and channels. When
collecting and sharing information, we need to exercise caution to protect
both the privacy of individuals and the objectivity of the data. Additionally,
collecting data on the trust of closed software may be challenging.

Finally, in Section 6, we conclude that current research on software trust is
limited to software quality, ignoring the impact of other hubs or actors in
SECO, e.g., software packages, package managers, or software producers the
future research should expand on this topic.

Figure 1: The meta-model can be used as a reading guide for this article, as
it mentions all the relationships between the main entities in our research.
For instance, the relationships between trust types, trust sources, major
trust impact factors, hubs, and stakeholders in the software ecosystem.

##  2 Background of Software Ecosystem

To assist in understanding what aspects of SECO can affect software trust, we
present here an overview of the package management workflow, as well as the
concepts used in the study. We sketch the primary entities and stakeholders in
the software ecosystem in Figure 2, along with their simplified relationships.

###  2.1 Actors in the Software Ecosystem

Software Product \- The definition of the software product is based on Xu and
Brinkkemper’s concept of a software product. It is defined as “a packaged
configuration of software components or a software-based service, with
auxiliary materials released for and traded in a specific market”
(xu2007concepts (146)). The definition emphasizes four elements: “packaged
components” refer to software, packages, components, libraries, or codes;
“software-based services” refer to the independent, small piece of
functionality accessed through internet, “auxiliary materials” refer to
documentation, user manual or other materials which need to be implemented
with the software or delivery to the end-users , and “release and trading”
refer the activities that release the software to the market and related
commercial value. To this, we add another independent element -“distribution
tools”, emphasize channels that facilitate distribution downstream of the
software supply chain, meaning the systems or tools that assist in the
processing or distribution of the packaged components that comprise a software
product, e.g., package managers.

Component \- Based on Software Engineering Body Of Knowledge (SWEBOK), “a
software component is an independent unit, having well-defined interfaces and
dependencies that can be composed and deployed independently”
(Bourque2014swebok (15)). The significant characteristic of a component is
that it can be reused, can interact with and interact with other objects, and
can be combined with other components to form a system or application. For
instance, a menu class or a button class.

Library \- A library is a collection of prewritten functions or data
structures that are organized to perform the same technically essential
functions, such as functions that handle compatibility issues
(bauer2012structured (10)). Sometimes a library is referred to as a package in
programming language directives, such as npm, RubyGems, and Maven
(zerouali2018empirical (148)).

Package \- A package is simply a collection that contains software, libraries
and metadata. The metadata includes, for example, the name of the software, a
description of its purpose, version number, providers, checksum, and a list of
dependencies for the software or library version. Packages are in general
versioned, which frequently adhere to de facto conventions, such as semantic
versioning (hanus2018semantic (67)). A distribution of packages can be
considered as being a software ecosystem, with a collection of interdependent
software projects that are developed and maintained within the same
environment (decan2019package (36)).

Package Manager \- A package manager provides a privileged, central mechanism
for managing the installation and upgrade of packages on a computer’s
operating system automatically (Cappos2008alook (20)). Sometimes it is
referred to as Software Manager or Application Manager, depending on the
context. The preferred term is Package Manager in programming languages since
the installed software is often just a set of libraries rather than a directly
executable application. Package managers have a major contribution in that
they retrieve the specific versions of the libraries that are required to
build client applications and the libraries they depend on, and install the
required libraries based on their dependencies (hejderup2018software (71)).

Package Repository \- Typically, a package repository is just a web server
that hosts packages and their associated metadata (cappos2008package (21)).
Package managers rely on package repositories to install packages and resolve
any dependency requirements.

End-user \- A software end-user is an individual or organization that adopts
or intends to adopt a software product into their applications. Usually, the
majority of end-users are software engineers, who are concerned with the
functionality of a package and its integration into their existing projects
(chinthanet2021makes (28)). Alternatively, they could be individuals who lack
development experience and rely on a software component to fulfill a specific
requirement. The distinction between these two types of end-users is not
emphasized in our study. However, in our literature review, we found that most
of the interview or survey participants are experienced software engineers.

Software Producer \- The term software producer refers to both software
producing organizations (SPOs) and software engineers who are involved in its
development. An SPO is an organization that builds and maintains software with
a view to getting the software created, adopted as widely as possible
(jansen2012shades (84)). As software engineers, whether they are employed by
an SPO or as independent contractors, they contribute to the development and
maintenance of software products.

Additionally, package maintainers are a subset of software producers. Package
maintainers are responsible for the development and maintenance of software
packages and their frameworks. They develop and maintain source code, package
builders, and installers, including bug fixes and patch releases.
Additionally, they configure package builds to ensure that packages can be
sourced from the distribution by extracting the source code from the
collection of binaries in the distribution (Duan2020Measuring (43)).

###  2.2 Workflow of Package Management

In Figure 2 we describe the package management process. When an end-user uses
a package manager to download a library, first, the package manager searches
the relevant configuration file maintained by itself for files that point to
the location of the repository, e.g., Apple Store or SourceForge, to retrieve
the package. Subsequently, the package manager downloads the relevant packages
from the repository. When the end-user confirms the installation of the
selected package, the package manager installs the relevant packages according
to the package dependency order. After a successful package installation, the
package installation information is stored in the metadata of the local
package database and managed by the package manager to maintain software
dependency and version information. End-users can modify the configuration
file to retrieve packages from other repositories. Alternatively, they can
search and download packages directly from repositories, such as Github.

A trustworthy program or application cannot depend on untrustworthy components
or services (Bourque2014swebok (15)). To this end, in addition to the
trustworthiness of the software product itself, we should also be concerned
about the reliability of the reusable components, such as components and
libraries, as well as the the trustworthiness of the tools that distribute or
store them, such as packages and package managers, and repositories that host
these reusable components. These elements are taken into account in the study
of software trust.

Figure 2: The model shows the package management process, and it includes
relevant stakeholders, software end-users and package maintainer, essential
entities, package manager, package, local database, and repository.

##  3 Research Method

Systematic Literature Review (SLR) is a term used to describe the process of
collecting, reading, analysing, refining, and organizing of data in the
existing literature to provide a comprehensive introduction, elaboration, and
evaluation on a specific research topic or phenomenon of interest
(keele2007guidelines (87)). We performed an SLR following the guidelines and
steps of (kitchenham2004procedures (89)) to gain the knowledge in the existing
literature on the concepts and influences of trust within the software
ecosystem domain to understand and reveal the current relationships between
software end-users and software producers. It followed the outline of the
following six activities in this review:

  * •

Defining the research question;

  * •

Searching for relevant studies;

  * •

Applying inclusion and exclusion criteria;

  * •

Assessing study quality;

  * •

Extracting and analyzing data;

  * •

Coding scheme.

###  3.1 Research Questions

We construct the following research questions (RQ) to get an overview of
software trust in worldwide SECO:

RQ1: How is the concept of software trust defined in literature?

  * We define the concept of software trust as the first object of this study. This is achieved by reviewing and comparing the definitions of software trust in the literature and examining the sources, types, and attributions of software trust. In addition, we extend the definition of software trust based on the relationships between software end-users and producers in the software ecosystem.

RQ2: What trust factors do end-user organizations consider when selecting
software products?

  * RQ2 takes into account end-user organizations’ trust factors when making software product selection decisions, which can be regarded in the following four sub-research questions. Their impact on software trust may be positive or negative, in varying degrees.

SubRQ2.1: What trust factors do end-user organizations consider when selecting
software products and versions?

  * RQ2.1 addresses the factors in software and software packages that affect trust. These factors are comprehensive in that part of them can originate from the software and software packages themselves, primarily the quality of the source code (e.g., integrity, re-usability), known defects and vulnerabilities, packages’ versions and dependencies, and the development process that affects the quality and security of these source codes; another part of them can come from the outside world, such as the reputation, popularity, and structural assurance of the software.

SubRQ2.2: What trust factors do end-user organizations consider when selecting
software package managers?

  * RQ2.2 is concerned with package managers. Since the major role of package managers is to manage dependencies in order to ensure that packages are installed with all dependencies they require, the ability to manage package dependencies becomes a critical factor affecting the trust in package managers. Additionally, the trust in package managers is also affected by security risks, such as consistently detected attacks on metadata, dependencies, and repositories, as well as known design flaws, such as the lack of effective protection mechanisms to detect and undo suspicious package installations.

SubRQ2.3: What trust factors do end-user organizations consider when selecting
software producing organizations?

  * RQ2.3 focuses on how software producers, organizations/ communities build trust in software and software packages. It focuses primarily on their management and strategy, such as follow-up of their products, reputation, and popularity.

SubRQ2.4: What trust factors do end-user organizations consider when selecting
software engineers?

  * RQ2.4 is concerned with the professionalism of software engineers, especially their knowledge, experience, and skills, as these factors all contribute to their ability to develop high-quality software, effectively resolve unexpected problems, and provide follow-up services. Engineers’ satisfaction and happiness are also beneficial for developing quality software.

###  3.2 Search Strategy

Literature review articles are identified by using search strings from
scientific libraries. First, we define the object of study, including software
and related concepts, such as software packages, components, and dependencies.
Then, in conjunction with software supply chain processes, we include software
management, provenance, and engineers, considering that software trust may be
influenced by project management and other human factors. Finally, after
examining a few definitions of trust, we include concepts related or similar
to trust based on the understanding of the nature of trust, such as
credibility, reputation, and uncertainty. This resulted in the following
block.

software AND (package OR component OR dependency OR management OR provenance
OR engineer) AND (trust OR credibility OR reputation OR uncertainty)

Software trust has been studied within software engineering, and related
research can be found in any software engineering-related digital library. To
ensure that no studies were missed, we adopted both manual search, automatic
search, and snowballing technique. We began by manually searching for primary
and secondary studies from the Journal of Systems and Software and Information
and Software Technology. However, it was found that the articles we searched
manually were included in the automatic search results, and they also
incorporated a wider range of articles. Thus, we decided to use automatic
search as the main search method, kept automatic search results from
IEEExplore and ScienceDirect as the academic sources, as well as Google
Scholar as a secondary source. Then, the first author obtained the initial
list of articles from the search engine and eliminated duplicates. 592
articles were chosen based on their titles, excluding articles that clearly do
not belong to software engineering, for example, we encountered titles
containing community environmental health, fish, and patients. In the
following process, five members of our research team applied inclusion and
exclusion criteria to each article, and if all the criteria were met, the
article was then included. After that, the five researchers scanned the
remaining articles to exclude more articles, then read the articles
completely, and performed the quality assessment. Upon reading the articles in
their entirety, we used a snowballing technique to include another seven
articles related to our topic, such as supplements and prior research,
ensuring that we did not miss any extensive versions of the articles. The
steps of the search process are depicted in Figure 3 along with the number of
primary and secondary studies conducted at each step. Note that there is
considerable overlap between articles in different digital libraries,
resulting in the inclusion of numerous irrelevant studies.

Figure 3: This figure indicates the stages of the search process and the
number of publications at each stage.

###  3.3 Inclusion and Exclusion Criteria

The inclusion and exclusion criteria ensure that relevant studies are
included, and studies that do not answer research questions are excluded. If
the inclusion criteria are too broad, poor-quality studies may be included,
lowering the overall quality of the study’s results. If the inclusion criteria
are too stringent, the resulting studies are likely to be small and therefore
not generalizable(meline2006selecting (107)).

The given inclusion criteria adopted in this research are:

  * •

Studies published since 1990s;

  * •

Studies that addressed at least one research question;

  * •

When studies reported the same research, only the most recent one was
included;

  * •

Studies that appropriately addressed software trust in software ecosystems.

The given exclusion criteria selected in this research are:

  * •

Studies that published in a language other than English;

  * •

Studies that incomplete or only provided literature in the form of abstracts
or presentation slides;

  * •

Studies that concentrated on the development of a single tool for a particular
domain;

  * •

Studies that lacked an adequate description of their objectives or context;

  * •

Studies that were not specifically related to software engineering or software
ecosystem.

We include books, articles, and a variety of grey literature, e.g., working
papers or dissertations/theses, on software trust, software selection, and
software evaluation. These search results were published after the 1990s, as
the Internet revolutionized the relationship between people, as well as the
relationship between people and software since then. Meanwhile, to avoid
results from other disciplines interfering with this study, we excluded search
results that did not pertain to software engineering or software ecosystems,
such as software packages that are explicitly applied to medical or
mathematical sciences. Additionally, we excluded search results that were
either incomplete or lacked detailed descriptions.

###  3.4 Quality Assessment

The quality assessment guides the interpretation of the findings
(keele2007guidelines (87)). In this study, five researchers were involved in
the quality assessment by adopting the double data extraction method
buscemi2006single (18), which means the quality of each article was assessed
by two researchers. We organized discussions to reach a consensus on quality
assessment and read three articles to cross-check and compare the results to
use this mechanism to eliminate any doubts about the selection process. Our
quality assessment focuses on the following questions:

QA1. Does the study address at least one research question?

QA2. Is the study based on research or expert opinion?

QA3. Is there a clear statement of the purpose of the study?

QA4. Was the data collection rigorous?

QA5. Are the findings of the study clear?

Each member of the research team was then randomly assigned a subset of these
592 articles. The first author then conducted a quality assessment of all
researchers’ derived results, discovering that 28 articles received varying
quality assessment scores. These 28 articles included six Google Patents, four
articles about specific software systems, such as mobile applications or
enterprise resource planning, and 19 articles about topics other than software
trust, such as hardware and internet trust, or how to estimate maintenance.
Following a discussion of these distinctions, we arrived at 112.

###  3.5 Data Extraction and Synthesis

The research methods were classified according to the categories listed in
Table 2. The majority of the search results are empirical research. Theories,
models, or frameworks that analyze or evaluate software trust or specific
aspects of software products contributed most of the trust factors to our
study. The remaining factors came from theoretical studies and exploratory
research, e.g., literature review. The results can be seen in Table 2.

Two data sets have been created to compile the impact factors found in the
literature.

The qualitative data set \- The analysis of the literature or its findings can
be used to deduce the factors affecting software trust. The majority of
significant factors are listed in tables or explicit categories, while others
were analyzed and extracted from the literature’s context, findings, and
conclusions.

The quantitative data set \- The impact factors for software trust are
determined from the models, frameworks, formulas, or test results shown in the
search results. The quantitative data set was composed of works that employed
experimental and statistical data collection methods. Typically, the impact
factors are presented graphically in tables, or as formulas, or they are
classified within the discussion and context of such literature.

Table 1: Summary of types of the search results Research types | Total
---|---
Empirical Research | 83
Theoretical Research | 18
Exploratory Research | 10

Table 2: Methods of data collection used in the empirical research Data collection methods | Total
---|---
Experiment | 42
Case Study | 19
Survey/Interview | 18
Statistic | 4
User study | 1

Data synthesis was performed using a frequency analysis technique to aggregate
the extracted data. All included articles were analyzed and extracted. It
should be noted that because some models or criteria have been proposed based
on the results of previous research, several trust factors are duplicated. In
the data synthesis, we still counted and accumulated these trust factors. The
reason is that we believe the trust factors that are repeatedly referred to
must be significant. Section 4 presents the synthesis results of the data
extracted from the selected studies and answers the research questions.

###  3.6 Coding Scheme

To analyse what trust factors end-user organizations consider when selecting
software products, we classified the impact factors and tried to find the
relationships between them. We adopted inductive coding method to analyze the
data from each study. Inductive coding is a type of data analysis in which the
researcher reads and interprets raw textual data to develop concepts, themes,
or a process model based on the data chandra2019inductive (25). In addition,
considering our topic, software trust, used combining Cue Utilization Theory
(CUT) to categorize trust factors midha2012factors (109). As CUT is well-known
as a framework for understanding and analyzing the different factors that
influence the subject matter as well as evaluating products. We classified the
cues we selected from the search results using CUT and divided the impact
factors into intrinsic and extrinsic trust factors for the previously proposed
research questions. Intrinsic trust factors have been used to represent a
product’s physical attributes, which refer to source code or architecture-
related attributes or factors, such as technical factors or vulnerabilities;
and extrinsic trust factors have been used to represent external attributes,
such as the product’s reputation, cost, licenses, or capability of the
software producer. In addition, we focus on each actor in the SECO and their
relationships, for example, the reputation or popularity of software
producers, which are also categorized in extrinsic trust factors.

After conducting a systematic review of all impact factors, we assigned them
to our CUT categories as well as to the research questions based on the themes
of each study. We then labeled the codes using the words from both datasets,
as well as their CUT and semantic counterparts and then adapted and refined
them. Additionally, we determined the relative importance of each category by
counting the frequency of the words. If a theme appears frequently, it
indicates that it is significant in the research on software trust and may
have a significant impact on software trust. In Figure 7 we show the major
trust impact factor according to the word frequency. The coding scheme of
trust impact factors is available at this link.

###  3.7 Biases in the SLR Process

Bias is one of the biggest challenges faced by all researchers. For example,
researchers may be unaware of previous research on a subject or maybe aware
but unable to access the findings, or the report may be missing critical data
points. Thus, when data are missing, not only does the size of the collected
sample decrease, but the sample’s validity gets smaller as well
(cooper2008research (32)). In this study, we faced two sampling biases:
Publication bias refers the problem that positive results tend to be published
more frequently than negative results, if negative results are not published
widely enough, false conclusions could be attributed to them as true
(keele2007guidelines (87)), and Retrieval bias, Typically, retrieval bias
refers to the risk that the sample used in the synthesis does not
appropriately represent the literature available (Dur2017bias (44)). In order
to mitigate them we did the following:

  * •

Conduct both automatic search and snowballing technique. The automated search
we conducted includes a broader search of articles about software trust, which
is supplemented by snowballing techniques to make the search more
comprehensive in content and scope.

  * •

Obtain articles from published and unpublished articles. In this study, along
with articles from peer-reviewed journals, we included grey literature, such
as pre-prints and post-prints of articles, theses, and dissertations.

##  4 Results

This section reports our findings of software trust. Figure 4 illustrates the
structure and relationships of our research questions, thereby contributing to
our overall understanding of our study. It also shows the order we answer the
RQs. First, we provide the definition and attributes of software trust (RQ1)
to understand its underlying constructs (RQ2), which are software products and
software producers. Following this, we examine the software products and their
producers to discover the factors that may influence software trust in the
SECO during the selection process. In terms of software products and software
producers, we analyze the trust factors affecting software and software
packages (RQ2.1) and package administrators (RQ2.2), as well as software
production organizations (RQ2.3) and engineers (RQ2.4). Numbers of search
results for each research question are given in Table 3.

Figure 4: This model shows the structure of this study, including the relationship between the research questions. Table 3: Overview of selected studies per research question. It is possible for a publication to address more than one research question. Research Question |  | Total
---|---|---
RQ1:How is the concept of software trust defined in literature? | 39
RQ2.1:What trust factors do end-user organizations consider when selecting software products and versions? |  Software development governance | 40
Software | 62
Package and version | 7
RQ2.2:What trust factors do end-user organizations consider when selecting software package managers? | 4
RQ2.3:What trust factors do end-user organizations consider when selecting software producing organizations? | 16
RQ2.4:What trust factors do end-user organizations consider when selecting software engineers? | 17

###  4.1 Concept of Software Trust in Literature

In this subsection, we focus on answering RQ1: How is the concept of software
trust defined in literature?

As we introduced in the introduction section, the common perception of trust
is that it is a belief that the trusted person will actually do what is
expected by trustor. In fact, researchers have interpreted trust differently
in various domains. In software engineering alone, trust is understood
differently. For example, jadhav2011framework (81, 51, 63) emphasize that
trust involves risks, defects, and vulnerabilities; wang2019updating (138,
117, 51, 53) argue that trust is viewed as a function of quality of service;
jackson2009direct (78, 2, 135) state that software trust encompasses trust in
software producers. Table 4 gives the attribution of software trust we
collected from the literature. This table only contains several high frequency
attributions, understandings of trust can be found more in the literature.

Table 4: Attribution of software trust based on the frequency of metrics in the definitions of software trust from the literature. Attribution |  Definition |  Source
---|---|---
Availability |  Trust is A capability of delivering requested functionality or services. |  boyes2014trustworthy (16)
Boundary |  It is important to identify the critical properties and determine what level of confidence is required in addition to software, physical equipment and humans. |  ghebremedhin2012combining (54, 78)
Calculated, measured, evaluated |  Though software cannot be seen or touched, there must be an evaluation and measurement of its trustworthiness in order to ensure the success of the software. |  rosenberg1998software (123)
Confidence |  Trustworthiness of software is determined by the degree of confidence that exists that it meets certain requirements. |  del2011survey (39)
Credibility |  A comprehensive qualitative description of the software’s availability, stability, performance, security, durability, and supportability under specified conditions and specified time. |  hong2011research (73)
Dependability |  A system is dependable means it can be relied upon or trusted to perform a certain task. In other word, it only be justified if there is evidence that the system has the capability to function without certain failures. |  jackson2009direct (78)
Evidence, Observation |  Trust refers to an aspect that is based on evidence or observation. Alternatively, it includes subjective aspects of an entity’s opinion, such as the presence of human information that indicates that such incidents will not occur. |  jackson2009direct (78)
Weakness |  Trust is the ability of a system to perform its required functions despite disruptions brought about by natural or human-made events, hostile attacks by internal or external parties, and/or system errors caused by hardware and software issues. |  jadhav2011framework (81)
Human, vendor |  One trust is about trust in technological artifacts, while the other is about trusting the vendor’s capability to fulfill the outsourced task and provide quality services. |  goode2015rethinking (59)
Reputation |  Reputation complements trust and can be seen as public perception of trust. |  nunes2019explaining (117)
Quality |  The notion of trust represents the overall system quality indicator, which should reﬂect the quality of performance and security, reﬂecting their static and dynamic nature. |  jadhav2011framework (81)
Structural Assurance |  No matter the specific characteristics of the technology, there are structural conditions such as contracts, guarantees, support, and other safeguards that should guarantee the success of the technology in general. |  guo2014ratings (64)
Subjectivity |  Software trust depends on the individual’s past experiences, their biases, their education, and so forth. It is only through the provision of specific, detailed guidelines that this subjective emphasis can be removed. |  guo2014ratings (64)
Temporal, Dynamicity |  Software trust is built continuously and gradually, and changed over time as more evidence or experience arrives. More evidence is needed to form a high trust. |  guo2014ratings (64)
Transitivity |  If A trusts B and B trusts C, then it may be concluded that A trusts C to a certain degree. Transitivity is widely used in trust-based recommender systems and systems authorized by certificate authorities. |  guo2014ratings (64)
Transparency |  A computer is inherently opaque, and one must determine whether the level of transparency provided is sufficient for the system to be trusted. Therefore stakeholders have to communicate the truth, keep their commitments , and consider the interests of each other. |  mercuri2005trusting (108)
Uncertainty |  Trust is essential for almost every situation where either uncertainty exists, or undesirable outcomes are possible, meaning that the end-user is exposed to the risk that the software might fail to fulfill its prescribed responsibilities whether intentionally or unintentionally. |  mcknight2011trust (106)
Vulnerability |  As a result, end-users accept any vulnerabilities as a part of the software. An alternative understanding is that a system is trustworthy when it functions as intended without unnecessary problems. |  rosenberg1998software (123);jadhav2011framework (81)

Software trust is inherently predictive based on historical data, regardless
of its focus (jadhav2011framework (81); kula2015trusting (93);
jackson2009direct (78); alarcon2020trust (2); trvcek2018brief (135);
mcknight2011trust (106)). According to this interpretation, we need to pay
attention to two aspects: the source of historical data and which aspects will
have an impact on trust, which we are basing our judgments and predictions
about trust. In Section 4.1.1, we discuss the source of interaction histories,
in Section 4.1.2, we explain three aspects that require prediction, i.e.,
software, human and structural assurance (the third column in Figure 6).

####  4.1.1 Source of Software Trust

Software trust is established through the software producer’s and the
software’s characteristics, e.g., how the first impressions of the software
are, which features of the software are more important, or which are regarded
as low priority or irrelevant. Sometimes, end-user first impression may
influence the final choice. For instance, while a project with spelling errors
in the README.md file does not necessarily indicate poor quality, an observer
may conclude that there is insufficient attention given to the project by
software producers.

Figure 5: In this model, we outline the main sources of trust, split into
evidence, e.g., portals of software products or software producing
organizations, and propensity, e.g., software end-users’ knowledge or
experience (amoroso1991toward (4)).

As shown in Figure 5, trust is based on either evidence or propensity.
Evidence includes information from the outside world, such as vulnerability
reports from various trusted repositories, for example, Common Vulnerabilities
and Exposures database (CVE) and National Vulnerability Database (NVD);
portals for software reviews, or communities. However, for proprietary
products, since such data is not open to the public, the way for end-users to
get relevant information is from the software or organization portals, or from
other users’ experience with the software. The issue then is that information
from the software producers directly could be one-sided and unverified,
insufficiently reflecting the reliability of the software product. In
contrast, open-source software products provide evidence in the form of
project data, source code, and bug reports, that are easily accessible since
the repositories, e.g., Github, GitLab, or BitBucket are public and end-users
do not require authorization to access them (manikas2013software (103)). End-
users can analyze this “raw data” to determine the trust of the product
easily. Additionally, evidence may come from stakeholders, such as internal
experts or external consultants for advice or evaluation, or experienced end-
users or experts, such as knowledge sharing or questions and answers from
stack exchange or stack overflow. Besides, various analytics tools and
recommendation systems are sources of trust. For example, the social network-
based recommendation system, which aims to make personalized recommendations
by providing users with the most relevant data from friends
(krishnan2008predicts (91)).

The other part is the propensity of software end-users, shaped by their
experience, education (amoroso1991toward (4)), cognition, and rationality.
Rationality stems from the collection of information, nevertheless, there are
situations, such as when faced with a high level of uncertainty or a time
constraint, in which rationality may not lead to the optimal decisions
(evans1996rationality (46)). Similarly, in the presence of factors such as
preconceptions, memory, or responsibility, cognition may influence the
probability of making an incorrect decision by the end-user
(chattopadhyay2020tale (26)). However, cognition is not always wrong
(decan2018impact (38); jadhav2011framework (81)). For example, a knowledgeable
end-user can determine whether a software package is trustworthy and
appropriate for the current project based on his previous experience.
Correspondingly, a person without any IT background and experience in using
software packages cannot be expected to find a trustworthy one. An interesting
fact about propensity is that when end-users are gathering information, they
tend to be biased and suspicious of positive information, perhaps because
producers eliminate negative information on a large scale, causing end-users
to distrust them (tavakolifard2012taxonomy (134)).

Preference is another kind of propensity. All attributes should be weighed
according to stakeholders’ value propositions (li2012integrated (96)). A
software product may not give the same level of confidence across all
functions but still remain dependable (jackson2009direct (78)). For example, a
bank system may incorrectly calculate the expiration date of a credit card but
must not disclose the customer’s information; or a cellphone may be unable to
change the ringtone, but it can definitely make a call. Software trust has
boundaries, which encompass the specific functionality, timing, user roles,
experience, and requirements of the software (jackson2009direct (78)).
Defining the boundaries of software trust facilitates the focus of the
analysis on specified areas (ghebremedhin2012combining (54)), as well as
clarifying the priority requirements and what level of trust. All software
trust we discuss should be predicated on the boundaries of software.

####  4.1.2 Types of Trust

There is a large volume of publications describing trust. das2001trust (35)
describe trust as goodwill trust and competence trust; sheppard1996network
(130) define trust as deterrence-based trust, knowledge-based trust, and
identification-based trust; and sabherwal1999role (125) specifies trust as
calculus-based, knowledge-based, identification-based, and performance-based
trust. Till now, there is still no consensus on the definition, as trust is
different in different contexts (heiskanen2008control (70)). When extended to
software trust, we find that identification-based trust proposed by
sheppard1996network (130) coincides with traditional social science trust,
which is stakeholder-oriented trust, i.e., trust related to software
engineers, organizations, and communities, referred to software producing
organizations in this study. Another one, knowledge-based trust, is in line
with the trust in software artifacts and products, which is based on knowledge
of a technology or product to predict whether it will function as intended
(jadhav2011framework (81, 93, 78, 2, 135, 106). The subject of this study is
software trust in the context of SECO, hence, as shown in the Figure 6, we
refer to the classification proposed by sheppard1996network (130). These three
types of trust correspond to trust in software product, structural assurance,
and trust in human, respectively. The correspondence is described in the
following section. We highlight the major factors as boldface text (the fourth
column in Figure 6), sub-factors as italics text (the fifth column in Figure
6).

Figure 6: This model depicts the three types of software trust: knowledge-
based trust, deterrence-based trust, and identification-based trust
(sabherwal1999role (125)), as well as the corresponding factors that affect
software trust, i.e., software products, structural assurance, and human.

Knowledge-based Trust - Trust in Software

  * In Figure 6, trust in software is a kind of knowledge-based trust. Based on the amount of citations, it primarily arises from the quality, reputation, and weakness of software products.

Quality is crucial to knowledge-based trust. In software evaluation, it refers
to the functionality, reliability, and helpfulness (mcknight2011trust (106)).
Functionality is the most basic condition of a trustworthy software product,
moumane2016usability (113) defines it as the purpose of the system and what it
does. Thus, the software product should provide confidence to the end-users
that this product is what the end-users want. According to IEEE, reliability
refers to the capacity of a system or component to perform specified functions
for a specified period of time under specified conditions
(rosenberg1998software (123)). Most software development managers equate
reliability with correctness, they use frequency of failure, fault tolerance
and recoverability to gauge how reliable a software product is
(berander2005software (13)). Helpfulness refers to the software product
providing adequate and responsive guidance to users, for instance, if a help
function supports users independently finding their way in the product
(mcknight2011trust (106); lankton2015technology (95)). These three are the
most mentioned subfactors in the literature, some attributes, such as
security, resilience, and safety, are also mentioned in the literature
(jadhav2011framework (81); del2011survey (39); lai2011trust (94)). Although
they are not included in the Figure 6, we believe that they are critical to
ensuring software trust as well.

Reputation is the average belief of the relevant audience that you do or do
not possess a particular trait, and it is built gradually over time
(barclay2015reputation (8)). Therefore, reputation is believed by the majority
of people. (nunes2019explaining (117)) believe “reputation complements trust,
and can be seen as a public perception of trustworthiness”, hence, software
trust is built on a public foundation, accumulating a reputation for
continuous, high-quality performance (mailath2001wants (102)). This
performance is not only about the high quality of the software itself, but
also encompasses whether the software producer can deliver high-quality
services.

Weakness refers to risk or uncertainty, anything predictable or unpredictable
that would cause problems or failure of the software or software producer.
Some studies have argued that ‘‘trust is the willingness to take risks under
uncertainty” (cho2015survey (29)), it describes “the probability of a
particular piece of software to elicit a security failure within a certain
time frame” (bugiel2011scalable (17)). Trust constitutes a distributed,
flexible concept that improves the decision-making process without the need
for heavy management, given the various risks and uncertainties from the
software itself or from the software vendor (moyano2016model (114)). These
uncertainties can entail requirements, technology, cost, schedule, support
resources, project management, and external attacks. We look at separately the
relationship between risk and software trust and the relationship between
uncertainty and software trust. Risk is the situation where the decision-maker
knows the possible outcome of the decision and the probability of its
occurrence (Park2017 (119)). Therefore, risk is predictable. Trust comes with
risk, and the greater the risk, the greater the trust required. There is no
doubt that there is risk in using software (song2010entropy (132)). Therefore,
we should focus on how software producers control risks and how software end-
users predict risks and understand how software producers control risks. For
example, it is common that expecting zero-defect software is unrealistic, so
instead of being concerned about the presence of vulnerabilities in software,
end users are more interested in when the vulnerabilities found can be
resolved in a timely manner. Uncertainty has been mentioned in several studies
on software trust. The sources of uncertainty are diverse and can be either
the quality of the software or the reputation of the software vendor. Although
it is unpredictable (Park2017 (119)), a large part of this uncertainty is not
caused by the internal problems of the software but by the external
environments in which it is deployed. This is because, a part of the
uncertainty can be exposed during the software testing phase or in the
interaction with the external environment that may produce unexpected
disturbances (muller2021trust (115)), timely fixes and remedies can eliminate
this part of the uncertainty. In addition, modeling and predicting disruptions
or impacts from unknown external environments can be difficult, such as the
multi-vector attacks on the software supply chain. Therefore,more uncertainty
comes from these external environments. However, the significance of
uncertainty for trust is that if there was no uncertainty, all information
would be objectively available, and end-users could make judgments
independently and with confidence, then trust is not needed.

Identification-based Trust - Trust in Human

  * In the model in Figure 6, trust in human is a type of identification-based trust. The identification-based trust exists because when all stakeholders understand each other’s intentions and expectations, a connection is built (sheppard1996network (130)). Considering the connection between software end-users and software producers, software producers need to understand end-users need and their expectations; in turn, end-users need to understand the performance of software producers. That is, knowledge of software producers’ intentions and expectations is based on a good understanding of their characteristics, such as prior performance, experience, skills, and team size. Therefore, it can be seen that identification-based trust is built on knowledge-based trust (sabherwal1999role (125)), based on a full understanding between the software producers and end-users.

Trust in software producers is discussed in the literature by examining
whether they are considered professional and reliable and can produce high-
quality software products and provide quality services (jadhav2011framework
(81)). In addition, how software producers remediate and recover from risks
and problems is also considered to be a factor that has an impact on the
reputation of software producers as it affects the end-user’s willingness to
maintain a long-term service relationship with the software producer
(bennett2000service (12)).

Deterrence-based Trust - Trust in Structural Assurance

  * Deterrence-based trust is built by the trustee’s fear of the consequences of breaking a rule, law, or contract. That is, there are licenses, regulations, or contracts in place in order to prevent someone from harming another, so the trustor can trust that the trustee will not do anything that could cause her harm (sheppard1996network (130)). Therefore, to ensure that all stakeholders’ interests are not compromised, clear premises or assumptions, clear responsibilities to be fulfilled by stakeholders, and contracts or clauses with clear penalties are required. When end-users encounter an unknown software producer or software, the emergence of such assurance protects the expectations and interests of end-users (wright2010trust (144)). Here we call them structural assurance (SA), defined as “the degree to which consumers believe that institutional structures like guarantees, regulations, promises, legal recourse, or other procedures are in place to promote success” (sha2009types (129)).

####  4.1.3 Definition of Software Trust

Software ecosystems are complex networks of overlapping supply chains of
software, data, and services. These supply chains have upstream flows from
end-users to software engineers, that for instance contain money and data, and
downstream flows, that for instance contain source code and packages. Software
ecosystem trust constitutes all upstream and downstream trust that is present
between the stakeholders that make up the SECO. This includes, for instance,
the trust of end-users in software engineers to build solutions for their
(business) problems, and the trust of software engineers in software package
management systems to correctly transfer their software to end-users. The
concept of software ecosystem trust is relevant, because without software
ecosystem trust, it becomes impossible for the SECO to fluently deliver value
to its stakeholders. Software trust is different from software ecosystem
trust. The definition of software trust is based on the complex relationships
between software end-users and producers in the software ecosystem. To define
software trust in this context, we refer to the definitions and attributions
of trust, and define software trust as follows:

Software trust refers to the willingness of software ecosystem actors to
accept risks based on subjective beliefs. It is essentially an upstream trust
in that actors expect assurance that other actors on top of a common
technology platform can exhibit reliable behavior and provide them valuable
software products. Software products are expected to be able to carry out
their intended functions in the presence of uncertainty, and run consistently
and reliably without interruption.

###  4.2 Trust Factors of Software Products

This subsection argues RQ2.1 - What trust factors do end-user organizations
consider when selecting software products and versions? As discussed in
Section 3.6, based on CUT, we classify the trust factors as intrinsic trust
factors and extrinsic trust factors. Intrinsic trust factors refer to the
features, code, and architecture of the software product, such as software
quality, code quality, and known vulnerabilities. And extrinsic trust factors
refer to attributes bestowed on the software product by the outside world,
such as popularity or reputation, as well as software-based services and
auxiliary materials that need to be delivered to the end-users. We highlight
the major factors as boldface, as we do in the subsequent sections.

####  4.2.1 Intrinsic Trust Factor

Intrinsic trust factors represent internal physical attributes, rather than
being determined by the environment. The following factors are most frequently
discussed in the literature:

Security, Vulnerability, and Attack Proneness \- Security is one of the main
challenges in software. With the continuous emergence of new types of cyber-
attacks, software attacks show a gradual trend of a large-scale, multi-vector,
designed to infect multiple components of the SECO. According to Sonatype, in
2021 the world witnessed a 650% increase in software supply chain attacks,
aimed at exploiting weaknesses in upstream open source ecosystems
(Sonatype2021OSS (131)).

Typically, software engineers are not qualified to spot vulnerabilities in
code, community knowledge, such as publicly accessible vulnerability
databases, can help (viertel2019detecting (137)). Literature shows that there
are three sources of vulnerability data; the first is public databases, such
as CVE, NVD, or snyk.io; the second is software producers’ databases, such as
Microsoft security bulletins; the third is specialized databases
(massacci2010right (104)). Yet, the quality of the data in the vulnerability
database, in particular the timeliness, consistency, and integrity, is an
issue that requires immediate attention (anwar2021cleaning (5)).

Vulnerabilities are not addressed in a timely manner. The results of a study
of Node Package Manager (npm) showed that most of the reported vulnerabilities
were moderately or highly critical, and while the time required to fix them
varied by severity, fixing them all took a long time. The probability of a
vulnerability being fixed within one month of discovery is 50%, while the
probability of a fix after only 6 months is 74%. Furthermore, more than half
of all dependency packages are affected by vulnerabilities in upstream
packages. One interesting aspect is that most of the affected dependency
packages are not automatically fixed with the fix of an upstream package. The
reasons typically are the lack of proper maintenance of the package or strict
dependencies (decan2018impact (38)). However, not all vulnerable packages or
libraries will propagate vulnerabilities since not all code in the package or
library will actually be used (hejderup2018software (71)). In addition, a
number of organizations defense approaches are more oriented towards
preventing access to sensitive data or finding bugs in software and software
packages than malicious packages (Duan2020Measuring (43)).

Unfortunately, there is still no uniform standard to measure software security
(wang2015trustie (140)). This may be because software security is influenced
by too many factors, such as environmental factors, complex software supply
chains, project management, or the adoption of programming languages. For the
software itself, the complexity of the requirements, the complexity of the
code, the technology used, and even the skills of the engineers all have an
impact on software security, with the problem being that these factors are
hard to quantify. Meanwhile, it is unrealistic to request software testers to
cover all security requirements perfectly. In addition, different stakeholders
have different perceptions of software security. Although organizations and
governments tend to place a high priority on software security
(wang2015trustie (140)), typically, software engineers do not it put in the
first place (vargas2020selecting (136)). In particular, software security
requires significant resources to develop, evaluate, and defend, small and
medium-sized companies may not be able to afford this investment.

Quality and Development Process \- Factors regarding quality revolve around
the quality characteristics of the software itself and about the activities of
the software during its life cycle. Base on ISO/IEC 9126:2001, software
quality has been defined as “the totality of characteristics of a software
product that satisfy stated or implied needs” (coallier2001software (31)),
which means the software product and process should satisfy (1) specific
functional requirements and; (2) end-user needs or expectations
(miguel2014review (110)).

There are various well-known software quality models. McCall’s Quality Model
proposes 11 factors that describe the external view of the software (user
view) and 23 factors that describe the internal view of the software
(developer view) (al2011software (7)). Similarly, Boehm’s Quality Model uses a
hierarchical quality model structure but focuses on a wider range of
characteristics, i.e., as-is utility, maintainability, and portability
(berander2005software (13)). FURPS & FURPS+ are developed by Hewlett-Packard,
identifying software quality factors through functional requirements and non-
functional requirements (khosravi2004quality (88)). Dromey’s Quality Model
recognizes that quality is assessed differently for each product and therefore
requires a combination of product characteristics and quality attributes to
determine how each attribute affects quality attributes and identify software
shortcomings (berander2005software (13)). ISO/IEC 9126:2001 incorporates these
models, it has four parts (quality model, external and internal metrics, and
quality metrics) covering six significant areas of software evaluation:
functionality, reliability, efficiency, maintainability, portability, and
usability (berander2005software (13)). However, there are no indications on
how to quantify these factors and subfactors, and this model may reflect the
perception of software engineers instead of end-users (challa2011integrated
(24); cote2005evolution (33)). It has been replaced by ISO/IEC 25010, which
focuses on the characteristics of software internal quality, software external
quality, and software usage quality. The most important point is that ISO/IEC
25010 proposes trust as “the degree to which a user or other stakeholder has
confidence that a product or system will behave as intended” in quality of
use. Several studies have extended based on these well-known models, such as
the Fuzzy Software Quality Quantification Tool (FSQQT), it takes as input
several real-time values of metrics and uses as output quantified software
quality based on end-user, and engineers’ perspectives. This study shows that
end-users care more about reliability and usability, while engineers focus
more on functionality, efficiency, portability, and maintainability
(challa2011integrated (24)). This is probably because end-users do not know
the software’s code, architecture, or efficiency, so they are more interested
in the maturity of the software, fault tolerance, and ease of operation and
learning.

Table 5 shows the number of primary factors and subfactors stated in the
models of McCall’s Model, Boehm’s Quality Model, Dromey’s Quality Model,
ISO/IEC 9126:2001, ISO/IEC 25010, FURPS & FURPS+, and FSQQT Model. We found
that reliability is the most critical primary factor in quality models, the
following are efficiency, portability, and usability, accuracy, testability,
and adaptability are the most stated subfactors.

Table 5: The table shows the primary factors and counts base on McCall’s
Model, Boehm’s Quality Model, Dromey’s Quality Model, ISO/IEC 9126:2001,
ISO/IEC 25010, FURPS & FURPS+, and FSQQT Model.

Primary Factor | Total
---|---
Reliability | 6
Efficiency | 5
Portability | 5
Usability | 5
Functionality | 4
Maintainability | 4
Correctness | 2
Testability | 2

Subfactor | Total
---|---
Accuracy | 5
Testability | 4
Adaptability | 4
Interoperability | 3
Security | 3
Maturity | 3
Fault tolerance | 3
Operability | 3
Installability | 3
Replaceability | 3
Reusability | 3
Consistency | 3
Completeness | 2

Software quality is not accidental; it is achieved through management and must
be assured throughout the entire product life cycle. Reliability is an
attribute of quality, the quality attributes applied at each stage of the SDLC
determine the software reliability, the focus is on error prevention,
especially the early stages of the lifecycle (rosenberg1998software (123)).
The advantage of the OSS project is that the project data is public, so there
are “many eyeballs” to help find and fix defects (mohagheghi2007quality
(112)), thus ensuring more credible software quality.

It is suggested by models or standards that evidence of software quality or
software trust should cover the information obtained at each stage of the
development process. For instance, IEEE Std 982.2-1988 indicates the
relationship between software reliability and each life cycle phase, including
concept, requirement, design, implementation, test, installation and checkout,
operation and maintenance, and retirement. In spite of the fact that this
standard was published in 1988, it does, to some extent, reflect the
information at each stage of the software development life cycle that
influences the measurement of quality. Additionally, a number of Capability
Maturity Models (CMM) address quality issues from five maturity levels of
software development processes, i.e., initial level, repeatable level, defined
level, managed level, and optimizing level (ellison2016measuring (45);
berander2005software (13)). Repeatable level focuses on project management,
the defined level focuses on engineering process, managed level focuses on
product and process quality, and the optimizing level focuses on continuous
improvement. he2009reference (69) argue that trusted components is affected by
the development process, therefore, the trustworthy proof should be collected
through three phases, i.e., development phase, submission phase, application
phase. Submissions and applications are primarily concerned with the quality
factors in the quality models, for instance, submission phase includes the
proof of functionality, reliability, and usability, application phase
emphasizes the proof of effectivity, security, and efficiency. Similiarly,
wang2011trustie (139) states that trustworthiness evidence includes three
aspects, development-stage evidences, delivery-stage evidences, and
application-stage evidences.

Additionally, with the introduction of third-party components, their features
cannot be assessed using traditional quality attributes. Additional concepts
of software quality arise, such as configurability, customizability,
reusability, scalability, availability, trackability, and compatibility to the
quality requirements for software products produced by traditional software
development processes (miguel2014review (110); chau1994selection (27);
challa2011integrated (24)). These concepts emphasize the need for third-party
components to be flexible in design and application, easy to reuse, change,
and extend. The object is not only making the third-party components reusable,
but also making them adapt components to end-users’ systems and requirements
(wang2019updating (138)).

Source Code Quality and Architecture \- Several researchers calculate software
quality as the ratio of the total number of defects identified in a certain
period of time to the total number of code lines (liu2007design (100)). Thus,
it is not hard to find that software quality, especially for open-source
software, depends on the quality source code (ghebremedhin2012combining (54))
and its attributes, for instance, availability, understandability,
completeness, conciseness, portability, consistency, maintainability,
testability, usability, reliability, structuredness, and efficiency
(crowston2003defining (34); jadhav2009evaluating (80); wheeler2011free (143)).
The most frequently stated impact aspect concerning source code in the
literature is availability. Reasons abound for this, one of which may be that
end-users or downstream engineers need to inspect or analyze source code in
order to determine code quality as well as software quality.

Programming languages may not directly affect software trust. Still, the
experts believed that 68% percent of the programming language features, for
instance, application domain, platforms, maturity level, and
comprehensiveness, have impacts on the aspects of usability, cost, stability,
ownership, guarantee, reputation, support, and maintainability
(farshidi2021decision (48)). Projects written in more popular programming
languages, are more successful in market penetration and attracting human
resources (ghapanchi2015longitudinal (53)), which indirectly gives end-users
greater confidence to choose their software. In addition, several aspects need
to be considered: (1) the project itself, for instance, as a back-end server,
it is not possible to consider a low performance interpreted language like
Python; (2) support of programming languages; whether in-house developers are
proficient in the language; (3) whether subsequent software packages need to
be purchased or used, and the interfaces.

We try to collect a set of indicators for source code analysis, result can be
found in this link, which are also the most commonly used fields in various
source code analysis tools. Examples include STRAM (Security, Trust,
Resilience, and Agility Metrics), a system-level trustworthiness metric
framework (cho2019stram (30)), and a framework proposed by Jadhav and Sonar
for the evaluation and selection of software packages (jadhav2011framework
(81)), and an updating model based on Meyer’s ABCDE(Acceptance, Behavior,
Constraints, Design, Extension) model of software component trustworthiness
per users feedback (wang2019updating (138)). In forming this set of
indicators, we considered, as much as possible, both the goal of the analysis
(i.e., to be able to describe the quality aspects of the software process and
the resulting product) and the availability of metrics in the data
(koch2008exploring (90)), assessing process and product evidence
(donohue2005my (42)), and product metrics (gopalakrishna2005vulnerability
(60)).

The architecture comprises “a collection of software and system components,
connections, and constraints; a collection of system stakeholders’ need
statements; and a rationale that demonstrates that the components,
connections, and constraints define a system that, if implemented, would
satisfy the collection of system stakeholders’ need statements”
(gacek1995definition (50)). It holds a great deal of importance to ensure
software quality. It is reported that“engineers will dismiss an unfamiliar
framework and library if they cannot make it work within an hour”
(haenni2014quantitative (66); haenni2013categorizing (65)). Sometimes, end-
users just select a package or a library to implement a single function
instead of a large system or application, for instance, for brownfield
development, the existing software architecture of the project imposes
constraints on the selection of the subsequent software package or library,
the selected package or library must fit into the existing technology stack of
the project (vargas2020selecting (136)). In addition, as software packages are
hosted off-premises, they are difficult to integrate with other systems or
software packages. End-users and software producers should pay additional
attention to the architecture’s integration, scalability, and reliability.
Software should have the ability to remain available to users within a given
time window and maintain a reasonable response time for users even during peak
periods (godse2009approach (56)).

Versions and Dependencies \- The distribution, installation, and updating of
software packages or components also have an impact on the integrity and
reliability of the software. Distribution requires that the package or
component is in the format provided by the software producer, does not contain
any malformations or corruptions, and that the installation and updates must
also be in compliance with the requirements set forth by the software producer
(catuogno2017secure (23)). However, dependencies complicate the software
components distribution. Dependencies connect various system components, thus
forming a highly interconnected ecosystem (hejderup2018software (71)).
However, the version of the package it depends on is specific, hence, a
critical activity that must be carried out at each stage of the SDLC is
ensuring the integrity and coherence of dependencies with deployment tasks, as
a failure in this process can result in disastrous consequences
(catuogno2017secure (23)).

Considerable security vulnerabilities are caused by aging libraries
(kula2017modeling (92)), however, it does not mean a new version does not
guarantee bug-free software. In addition, the most recent release may break
dependencies and result in incompatibility (decan2016github (37)). Thus,
considerable organizations choose the stable version over the latest version,
they prohibit updates in case of dependency updates (pashchenko2020qualitative
(120)). The dependency hell occurs when several packages have dependencies on
the same package or library, relying on different incompatible versions.
Dependency hell can cause a lot of trouble for installation and subsequent
maintenance, and can create security risks (abate2011mpm (1)). Still, numerous
software engineers are unwilling to update to newer versions, they downgrade
package versions to avoid dependencies and conflicts introduced by the latest
versions (chinthanet2021makes (28)), even at the expense of some software
security, because they are overwhelmed by the burden of maintaining
dependencies (GKORTZIS2021110653 (55)). Therefore, the requirements for a
package or component are: (1) it has to be authentic; (2) unmodified; (3)
fresh; (4) all required dependencies are properly satisfied and ensure
dependency, integrity, and consistency; (5) unauthorized installation is
prohibited (catuogno2017secure (23)).

####  4.2.2 Extrinsic Trust Factors

Extrinsic trust factors refer to the external environment characteristics that
influence software selection. Although the majority of articles discuss
intrinsic trust factors, numerous studies have concluded that end-users
prioritize extrinsic trust factors during the software selection process. In
comparison to intrinsic trust factors, which lack quantitative indicators or
detailed data, extrinsic trust factors have a plethora of data sources on
openly accessible portals.

Structural Assurance \- In Section 4.1 we underlined that structural assurance
is believed to increase software trust. It includes licenses (e.g., GNU
General Public License 2.0, and Apache License 2.0), contracts, regulations,
or other safeguards (e.g., package promises and legal considerations)
(guo2014ratings (64)). End-users may perceive that they are using software in
a secure, protected environment as a result of compliance, policies or laws.
Contracts define the rights and obligations of the software product, including
what each stakeholder expects to accomplish, the feasibility of the goals and
respective roles, the actions that must be taken, as well as the
responsibilities of the software producer and the solutions to anticipated
risks or problems. License is the most important type of structural assurance,
it is based on copyright law and confers certain exclusive rights on the
software producer of a work, including the right to use, reproduce, modify,
merge, publish, distribute, sub-licenses, and sell copies (alspaugh2009role
(3)). This means that the license restricts the end-user’s ability to use the
software. For instance, the General Public License (GPL) specifies what end
users may do with the software, such as modify the program and distribute the
modified version, which protects not only the copyright of software producers
but also the interests of both software producers and end-users, thereby
reducing the perceived risk between end-users. Hence, it is reported that end-
users are more likely to trust a licensed product (osterloh2016trust (118);
chinthanet2021makes (28)). Besides, different versions of the same software
may have different license, which end-users should be aware of during the
software selection.

Documentation \- Documentation provides the first impression to end-users of a
product’s quality (vargas2020selecting (136)). It includes the official
project documentation, e.g., requirement descriptions, code comments, or
testing results (national2007software (79)), and broader forms of
documentation, e.g., documentation translation, user comments, or discussion
in the forums (ayala2011selection (6); chau1994selection (27)). A survey of
npm developers initiated by chinthanet2021makes (28) revealed that 73% of
developers believe that the documentation is highly relevant to package
quality assessment, with the existence of a run example, and the existence of
the README.md file in the root directory of the package considering the top
two documentation features. Similarly, as part of our previous study, we
interviewed 12 participants, including software engineers, DevOps, architects,
and other experts, and ten of them considered documentation crucial to package
selection. Typically, they examine data structures, the functionality of
features, and the package’s compatibility with the project via documentation,
one of them confirmed that packages requiring 2 days of reverse engineering
due to unclear documentation would be eliminated (jansen2021trustseco (82)).

However, there is a problem with both official documentation and broader forms
of documentation. In terms of official documentation, not all types of
documentation are available to the public due to software producers do not
share project data and source code for free commercial software. In terms of
broader forms of documentation, although the documentation for open-source
software or packages is free and open, much of the technical information and
discussion occurs on mailing lists or forums, making it difficult to collect
and maintain valid information, let alone version control of the
documentation.

Popularity \- Popularity is defined as the perceived frequency of use of a
tool among members of the community (xie2021popularity (145)). It represents a
software product’s popularity and is the most straightforward way to determine
how widely used and popular the product is in the public. Popularity can be
measured as the number of likes or downloads of the software product, and the
number of subscribers or views of the information page. Additionally, the
number of dependencies reflects the popularity of the software, as a component
or library, with more dependencies means more people are using it and it is
more stable (vargas2020selecting (136)).

Reputation \- Reputation is an estimation of the nature and values of a
person, a group, an organization, an object, an event, or an activity that is
widely shared in a group (watson2005reputation (142)). It is a judgment about
the quality of a product and future behavior (farooq2016multi (47)).
Reputation can be at the software level or human level. Human-level reputation
is discussed in Section 4.4, The focus here is on the reputation of software.
The most direct reputation indicator is the average software product ratings
(alarcon2020trust (2)). The reputation of a software product may include the
number of releases or versions, product history, and open/closed
issues(alarcon2020trust (2)), programming language used, clarity of comments,
number of bugs reported, tests performed, number of versions, and efficacy
(seigneur2011trust (61)) from users who used the software or stakeholders who
participated in the development or testing. Reputation has a positive impact
on software trust. A good reputation represents a good rating by a certain
size of users over a given period of time, and it represents a lower risk of
choosing the software. Therefore, in similar software quality, the end-user
will choose the software with a higher reputation. However, almost all
reputation information is based on a web source, and one of the significant
issues with single-source reputation is the vulnerability to falsified
information (farooq2016multi (47)), which means that software producers are
likely to falsify information or provide only one-sided information to achieve
a higher reputation. Besides, numerous proprietary product licenses contain
clauses prohibiting public criticism of the product without the software
producers’ prior permission (wheeler2011free (143)). As a result, it becomes
more difficult for end-users to obtain real feedback from other users, making
it difficult to be able to assess the trustworthiness of proprietary products
difficult.

Cost Factors \- Although it is widely accepted that cost is associated with
software selection because solution selection frequently involves a trade-off
between quality and cost (limam2010assessing (98)), they do not believe that
cost is associated with software trust. This is most evident in the growing
awareness of software, particularly open-source software, across a range of
industries. However, for several specialized industries, such as banking, they
resisted adopting free/open source software despite the high cost of
customizing and maintaining the software. This is because they are concerned
about the lack of support in the event of a failure or additional damage to
their core systems due to a security breach. Even if they adopt free/open
source software, it is more likely that they will use it for visualization
than for core banking modules.

Many IT companies and open-source platforms currently use Return on investment
(ROI) to evaluate related cost factors, divided into cost of investment, cost
of reuse assets, and rework savings. The cost of the investment includes
defense cost, which refers to the cost of deploying various alternative
solutions; decision cost, which refers to the administrative delays,
complexity, and effort incurred in the decision process; recovery cost, which
refers to the costs associated with the recovery process to back to a normal
operating system state; and defense costs which refers to the cost of
defending the system against attackers (cho2019stram (30)). Defense cost is
the critical one, it includes, for instance, license cost, training cost,
installation and implementation cost, maintenance cost, mitigation cost, and
upgrading cost (jadhav2009evaluating (80); jadhav2011framework (81)). Cost of
reuse assets and rework savings easily reflect whether the software is
trustworthy or not, they measure the cost savings when adopting a trusted
library or package, as well as the rework cost savings due to reduced system
errors (mohagheghi2007quality (112)). Usually, the two factors are as
difficult to gather before the end-users select a software or component. The
possibility is that they are predicted or calculated after the software has
been in use for a time.

In addition to the financial cost, the time cost and potential cost risks that
can impact the cost also need to be considered (vargas2020selecting (136)).
For instance, the software will decommission in one year or if the supporting
team will be available in a long time.

Figure 7: This sunburst chart shows the major impact factors for RQ2 in
hierarchies. Product segments are intrinsic trust factors, Software Product,
Human, Structural Assurance, and Cost are extrinsic trust factors. The size of
each segment in this sunburst chart indicates the frequency with which this
factor appears in the 112 articles selected.

###  4.3 Trust Factors of Software Package Managers

This subsection focuses on addressing RQ2.2: What trust factors do end-user
organizations consider when selecting software package managers?

A package manager is a software tool used to search, install, track, upgrade,
and delete packages on the operating system. It automatically connects to a
repository and downloads the requested and additional packages needed to build
the client application (Cappos2008alook (20)). During the study, we found the
current research on software packages is inadequate. Of the 112 articles, only
four are about package managers, explicitly refer to Table 3 for details.
These articles on the impact of software trust focus on three aspects: (1)
dependency hell; (2) Security vulnerability; (3) Vulnerability Detection in
the package manager. All three aspects are related to the design of the
package manager and the vulnerabilities of the package manager, they are all
intrinsic trust factors.

Dependency Hell \- As previously stated, installations are currently
experiencing dependency hell, end-users become trapped in a dead-end loop that
state-of-the-art package managers are unable to resolve (abate2011mpm (1)).
Several researchers argue that a trusted package manager should avoid the
dependency hell by selecting the right package sources and versions,
especially guaranteeing confidentiality, integrity, freshness, and
authenticity of the package and dependency (catuogno2017secure (23)).
Currently, attempts have been made, for instance, Nix, a Linux distribution,
installs software packages in a purely functional way to avoid the dependency
hell associated with the traditional upgrading of
/usr/bin/perl(dolstra2010nixos (40)).

Security Vulnerability \- Security vulnerability includes two aspects, bugs
and attacks on the metadata and on the repository. In terms of metadata, as
explained Section 2, metadata includes the version of a package and a set of
dependencies. An attack on metadata may result in an attacker changing the
package’s metadata to indicate that it depends on a package containing
malicious code formulated by the attacker, or the attacker, introducing
unsolvable dependencies or a large number of virtual dependencies that make
the installation impossible (ayala2011selection (6)). In terms of the
repository, if an online repository contains malicious code when downloading
and upgrading packages, it will be imported into the end-user’s computer
through transitive dependencies (hejderup2018software (71)). For example,
GitHub, an online service based on the code version control system,
experienced a malware attack in 2020. The malware identifies the NetBeans
project file, copies the malicious payload to the project file, executes the
code once the NetBeans project is built (gonzalez2021anomalicious (58)). Once
the GitHub repository is successfully compromised, the cybercriminals behind
the malware have full control over the target open-source project, ensuring
that any new build project cannot replace the infected project, and malicious
build artifacts remain the same.

Vulnerability Detection \- The major problem of the package manager design is
the lack of vulnerability detection and protection mechanisms. Vulnerability
detection tools are generally based on known vulnerability disclosure scanning
software components to find vulnerabilities (kaminsky2002hack (86)). However,
those tools do not work efficiently, detecting a limited number of possible
errors based on predefined rules (russell2018automated (124)). In addition,
they also lack oversight of the packages, as well as a mechanism to undo
malicious packages after they were installed. For example, when an attacker
wants to prevent the package manager, APT or YUM, updating a package from
another repository, he refuses to send data or sends data at a very slow rate
to the client’s open connection, but the package manager lacks monitoring and
screening of such situations, does not have any output to indicate problems or
forcibly closes the connection (ayala2011selection (6)).

###  4.4 Trust Factors of Software Producing Organizations

This subsection focuses on addressing RQ2.3. What trust factors do end-user
organizations consider when selecting software producing organizations?

If a product is managed by an organization, it will be more popular
(jarczyk2014github (85), as it gives the end-users the confidence that this
product will be produced and maintained by professional staff. However, it is
often difficult to determine if an organization is a good self-promoter or a
truly reputable organization. Next, we discuss the role that software
producing organizations (SPOs) play in software trust assessment.

Reputation \- Reputation means a publicly recognized judgment about the
character or status of a person or thing (immonen2007trustworthiness (77)). It
is an asset for software producing organizations and gives the organization a
competitive edge over others (pollock2007technology (122);
hoxmeier2000software (75)). Studies have shown that software quality has a
positive relationship with an SPO’s reputation (cai2016reputation (19)), while
an organization’s reputation depends not only on the quality of the products
it produces, but also on other aspects (vargas2020selecting (136)), for
instance, the services they provide (jadhav2011framework (81)), end-user’s
feedback, and their rating values (seigneur2011trust (61)), positive and
timely response, compliance with contracts, and length of experience
(chau1994selection (27); vargas2020selecting (136); jadhav2009evaluating (80);
challa2011integrated (24); capra2011firms (22)). Length of experience, from
the point of view of the SPO, means that the company or organization has
experience in product manufacturing, project management, business operations,
industry, and experienced in-house experts and engineers (chau1994selection
(27)). This extensive experience results in the company or organization
developing higher quality and trustworthy software, especially when risks and
uncertainties come. They can deal with problems or avoid them more
effectively, increasing end-user confidence in their products and services.

SPO’s Capability \- SPO’s capability is the degree of proficiency in adapting
skills to a wide range of activities (goles2008information (57)). It arises
from different factors, including technology skills, firm’s strategy,
maturity, and scale. Technology is undoubtedly the capital of enterprises to
achieve success and is the core competence.

Technology skills emphasize technical knowledge in software project
management, including programming and system development knowledge, R&D
capability, training, and support. Trustworthy software cannot be developed
without programming and system development knowledge, which is the physical
guarantee of software production (lai2011trust (94); cho2019stram (30);
sarrab2014empirical (126)). This means that the organization must have
qualified employees who are proficient in programming, system analysis and
design, project management, and other soft skills in order to deliver software
on time, on quality, and on schedule, and to effectively control and resolve
all risks and issues in the SDLC (amoroso1991toward (4)). It is however not
directly measurable, we need to examine the software products they produce and
the services they provide instead. R&D capability shows whether the company
can develop and maintain current projects and has the potential and innovation
to develop new products. It keeps the organization ahead of the competition by
continuously innovating and introducing new products and services or improving
its existing offerings. End-users prioritize getting training and guidance in
the assistance process (vargas2020selecting (136); jadhav2011framework (81);
del2011survey (39); sarrab2014empirical (126)), which include: (1) training
and documentation, for example, user manual, tutorials, troubleshooting guide,
and training; (2) maintenance and up-gradation, such as technical support and
consultancy, communication with end-users, on-site demo and free-trial
version, rapid responsiveness, business skills (jadhav2009evaluating (80);
jadhav2011framework (81)), management, and operation (hunter2013rise (76));
(3) social network-continued activity; (4) knowledge-sharing user-base
(wang2015comparative (141)), e.g., Discussion forums, signs of a functional
community (10.5555/1593439 (49)). Especially for OSS, a guaranteed, long-term
technical support is an important indicator for selecting an OSS software
(alarcon2020trust (2)).

Current software products rely to some extent on third-party components, or
many large IT companies outsource all or part of their projects to third-party
companies, hence, SPO’s technology skills should consider the skills to manage
suppliers. For instance, how to select third-part components, how to manage
third-part producers, or how to maintain the sourcing strategy. A firm’s
strategy includes the management and strategy of the company, the
organizations’ culture, and the type of industry. Most organizations and end-
users consider influencing the selection process, but they are not the major
factors (vargas2020selecting (136)).

Organizational maturity shows that SPOs have the capability to manage the
product well, which is essential to ensure product quality. It is abstract,
including, for instance, content management systems, partnership models,
rules, and regulations (jansen2014measuring (83)). Although there is no
consensus on how to measure it, from the perspective of software end-users,
formal procedures and standard security policies will help them evaluate and
select software products that are suited for them (vargas2020selecting (136)).

The scale of the organization is presented by, such as how many people write
the code, report problems, and repair defects (mockus2002two (111)), number of
members contributing (bogart2016break (14)), and average quality of members
(challa2011integrated (24)). The number of engineers working on a project is
the most frequently cited factor (crowston2003defining (34);
jansen2014measuring (83); sen2012open (128); mcclean2020social (105);
midha2012factors (109); koch2008exploring (90); scacchi2007free (127)). These
studies show that engineers positively impact the success of a project,
especially for open source projects. In particular, by modeling both the
number of engineers and subscribers, the results suggest that more engineers
in a project may increase the project activity and thus increase the number of
subscribers. Similarly, an increase in subscribers may affect the number of
engineers by attracting new engineers and contributing new engineers from the
subscriber base (sen2012open (128)). Hence, along with the program activity,
the number of subscribers, the engineer base is considered by sources of trust
information as an indicator of the success of the OSS project
(subramaniam2009determinants (133)), as it is obvious that they will increase
the popularity of the project. Most studies do not explicitly categorize the
engineering base as an attribute of software trust. However, it is often
mentioned in studies measuring project success and studies of software trust
regarding organizations and communities.

###  4.5 Trust Factors of Engineers

This subsection answers RQ2.4. What trust factors do end-user organizations
consider when selecting software engineers?.

17 selected studies discuss how engineers influence software trust. The
identified impact factors are extrinsic trust factors, which are focusing on
the following points.

Engineer Abilities and Skills \- Engineer abilities and skills are also
related to software trust. Engineering abilities and skills refer to the
knowledge and abilities that an individual engineer or a group of engineers
should possess throughout the product’s lifecycle, they stem from a good
working knowledge and experience of different types of operating systems and
the ability to learn from others (sarrab2014empirical (126)) These skills and
abilities are used to develop and integrate new and stable modules that meet
end-user requirements, which includes, for instance, deploying and maintaining
software, monitoring news posts and messages in user forums and mailing lists
on a regular basis and promptly responding to bug reports (norris2004mission
(116)). Several researchers believe that they have a significant impact on the
quality of the product they create (mcknight2011trust (106, 116);
sarrab2014empirical (126); donohue2003modeling (41)). Engineer responsiveness,
in particular, is viewed as critical regardless of whether end-users trust the
software or its output (gefen1996developer (52)). If software engineers are
unable to respond promptly to end-users, they will be viewed negatively by
end-users and their products, resulting in distrust. Despite the fact that the
response time is contingent on the availability of engineers in the project
and the priority of end-user problems (mockus2002two (111)). In most cases,
end-users are unconcerned with the severity of defects, what matters is that
they can be resolved in a short time. In this context, their soft skills are
particularly important.

Numerous researchers disagree on the importance of end-users knowing who
developed the software. According to a set of interviews with 75 engineers
that had been conducted in (haenni2013categorizing (65);
haenni2014quantitative (66)), when the end-users select a component or
library, they are not concerned with the engineer’s identity but with the
code’s quality and level of maintenance, e.g., the engineer’s responsibilities
and their response time. The opposite opinion is that the names and numbers of
people working on code development and testing must be public, as reputable
engineers contribute to the code being more reliable and less defective
(madanmohan2004notice (101)). Both views imply that how engineers maintain the
project is crucial for software selection.

Engineers’ Satisfaction and Happiness \- Besides that, several researchers
believe that engineers’ satisfaction and happiness with working significantly
affect the product quality of their products and the trust that end-users
place in them (crowston2003defining (34); scacchi2007free (127)). Engineer
satisfaction and happiness have a positive effect on software quality
(crowston2003defining (34)). The most obvious benefit is increased
productivity, they will stay on tasks longer and concentrate more intently,
resulting in significantly more efficient coding and problem-solving
(graziotin2018happens (62)). As a result, both the software’s quality and the
project’s success will be significantly enhanced. Engineer satisfaction and
happiness are highly subjective factors because each individual’s satisfaction
and happiness are unique, they cannot be collected and analyzed via specific
documents or fields, as test results are. The feasible method is to conduct
sentiment analysis on engineers’ feelings, for example, through their comments
on project portals or websites, to ascertain their level of satisfaction or
happiness.

##  5 Discussion

###  5.1 Impact on the Software Selection

The factors discussed in this study may have a positive or negative effect on
the software selection outcome. No factor, on the other hand, can have an
absolute positive or negative effect. For instance, vulnerabilities and
attacks appear to be detrimental to software trust, as their presence has a
significant impact on the quality and credibility of software (cho2019stram
(30)) and the reputations of software producers (vargas2020selecting (136)).
With the advancement of attack techniques and the expansion of scale and
scope, no software can now claim to be secure enough on its own. However, if
the software can employ an effective defense to ensure a lower number of
attacks over time, and if it is capable of employing more rapid means of
resolving the situation following an attack, we can assert that although
vulnerabilities and attacks undermine the software, their effect on selection
and trust is actually positive (gefen1996developer (52)). Similarly, factors,
such as the software’s quality, reputation, or the skills of the software
producer can have a positive or negative effect, i.e., if the software is of
high quality, has a positive reputation, and is provided by a competent
provider, they will have a beneficial effect on the software, and vice versa.

The negative or positive impact is determined by how the end-user perceives
those factors for a variety of reasons. For instance, several end-users choose
low-cost software, whereas several industries, such as banking, in order to
get enough resources to develop and maintain the software, choose those with
high costs.

There is one situation in which the impact is neither positive nor negative:
end-user propensity, as it affects the end-user to conduct the trust process
professionally and rationally. The resulting effect on decision-making should
be beneficial. However, if the technology of the software evaluation exceeds,
such as end-user experience, it may have no effect on decision-making
(moyano2016model (114)).

###  5.2 Validity Consideration

We considered validity from the following points. First, our selection of
articles was inevitably subjective due to bias and understanding of the
articles and several concepts, especially the large number of different
similar names for the same concept in the literature, so we could not include
all relevant studies included in this SLR. Second, most of the studies and
models that we know focus on software quality. There is relatively little
research on other hubs in the software ecosystem, stakeholders, and their
relationships. The search results are more relevant to FOSS-based discussions
because FOSS provides public access to technical data, whereas proprietaries
have restricted access. Hence, the analysis of the results may not cover
proprietary SECO. Additionally, a considerable number of studies are based on
interviews or questionnaires, which may cause the results they present to be
not generalized. Moreover, most models are not yet widely adopted in the the
open source ecosystem, which means that although the results of several
current studies and models come from questionnaires or interviews or are
further validated by them, they still lack validation and application in real-
world projects, which also further suggests that the results based on
questionnaires or interviews may be problematic in terms of generalizability.
Finally, we categorized software trust as intrinsic trust factor and extrinsic
trust factor, and several subfactors, while this allows for a clarified and
focused presentation of the results, we may have inevitably removed several
factors that could not be categorized and may have overlooked several specific
scenarios that end-users trust.

Next, we will further discuss the role and severity of these factors in the
SECO with different stakeholders and experts to ensure that our understanding
of these factors is correct and will continue to focus on and supplement with
additional impact factors.

###  5.3 Lessons and Research Challenges

Software trust is comprehensive and its factors cover all actors in the
software ecosystem. In this subsection, we identify several challenges that
make the research more complicated. The first complicating factor is that end-
user’s propensity to trust is subjective, and even when different end-users
are given the same information, the trust they place will be different, and
this trust can change over time, the judgment depends on the information they
access. Therefore, it is impossible to assign a fixed priority or weight to
the factors.

Second, it is difficult, if not impossible, to find a comprehensive set of
models to measure software trust objectively and fairly. The measurement and
interpretation of software trust must be dynamic. Third, the interpretation of
trust factors should be multi-faceted. It may be difficult to obtain or apply
project data as the factor determining trust in closed software, but these
data can be used to calculate trust in open source software.

The digitization of society has provided us many affordances and the
penetration of IT in it should be considered an unprecedented development in
history. However, we are standing at the advent of a new period in history,
where societal trust in IT is dropping at an alarming rate. We can say the
hay-day of society’s willingness to adopt new IT solutions and trust the older
ones are over (hayes2020algorithms (68)). With this problem statement in mind,
we have set ourselves a research challenge.

The ambitious goal that we have set is to raise the accessibility,
reliability, and use of trust data in the worldwide software ecosystem
(hou2021trustseco (74)). We envision a future where software can be rapidly
and even automatically assessed in a fair and equal manner, using a shared
understanding of trust that is collaboratively created and maintained.
However, before creating such a free and common infrastructure, several
challenges need to be tackled. We define a set of research challenges for the
future in bold typeface.

The first challenge is that we are collecting and sharing software engineer
data, i.e., data that could reveal their identities and break their privacy
(van2020core (121)). This would directly contradict the goals of this project,
so we need to find mechanisms for protecting personal data long ethical and
legal boundaries, such as the General Data Protection Regulation (GDRP).

There is an inherent problem with sharing information with a community, as
actors could misjudge or misuse the information. It must be noted that
observed vulnerabilities are not a sign of untrustworthy software and that if
these vulnerabilities are rapidly identified and eliminated, they actually
might be a sign of trustworthy software. Furthermore, we need to ensure that
vulnerabilities can be shared openly within the community without them falling
into the ‘wrong’ hands.

We have performed two preliminary interview studies to assess how software
engineers and end-user organizations would use trust data in their work. These
interview studies showed that while trust facts may be irrefutable, the
perception of trust is a moving target for different actors. One actor may be
satisfied with using a bleeding-edge version of a software package that
updates often, while another wants an old version of a patched but reliable
package. We are currently exploring if different categorizations of trust
ratings are necessary, such as “project evolution speed”, “vulnerability fix
time”, and “team liveliness”.

While we respect tools that centrally collect trust data, we believe that
trust data should be part of a zero-trust community. We observe that it is
impossible to observe correct trust data as one central party objectively and
consensus mechanisms are necessary to observe that a software package version
is trustworthy objectively.

The data collection process itself is challenging. We should be cognizant that
a set of impact factors under security in Figure 7, for instance, will be
challenging to operationalize. Once we have identified the values (i.e., trust
facts) necessary to collect, we will have to tackle the challenge of the
availability of these data. Data needed to judge the software trust can come
from usage data, source code, build systems, test systems, etc. It will
thereby be challenging to judge the trust of closed software. Furthermore, it
is hard, if not impossible, to get data from closed reputation databases, such
as app store ratings. Finally, the operationalization of the impact factors is
a major challenge in this work. We plan to perform a large-scale survey study
with software engineers as an approach to this problem to identify a minimal
collectible set of trust facts useful for software engineers.

###  5.4 Future Work

Our near-future works consist roughly of two parts.

First, we plan to launch a software engineer survey based on the outcomes of
this SLR. The developer survey intends to determine a prioritized and
categorized list of trust criteria. Furthermore, we want to use the survey to
gauge and create awareness amongst software engineers about the role of trust
in their software engineering processes.

Secondly, using the list of prioritized trust factors, we develop a tool for
automatically gathering trust scores for different trust impact categories
(hou2021trustseco (74)). Software engineers must evaluate the tool to decide
whether it effectively supports them in downloading and depending on a
particular software package. Finally, we aim to store these trust scores in a
distributed ledger, to make trust common for all software engineers. The
community that uses the ledger can contribute openly observable facts to the
ledger, to increase or decrease trust scores for software packages.

##  6 Conclusion

Although prior work has developed a number of models for quantifying software
trust or other related attributes, only a few have been widely used in real-
world projects. We discovered that the majority of them are oriented around
software quality measurement, ignoring the impact of related actors in SECO on
trust, and omitting the role of attacks and vulnerabilities in measuring
trust.

In this paper, we examined the role of trust in SECO by starting with the
selection and measurement of software products. Our goal is to devise a
strategy for resolving the trust erosion issue raised in the introduction,
specifically the harm caused to SECO’s health by vulnerabilities and attacks.
This study adopted a systematic literature review approach to review trust in
SECO from a selected 112 articles. On this basis, we discussed the definition,
types, and sources of software trust and proposed a definition for software
trust in the context of SECO. Additionally, we classified the impact factors
on software trust from the standpoints of software and packages, software
package managers, software producers, and software end-users. We analyzed the
existing literature for relevant impact factors and counted their frequencies.
Software quality, SPO, code & structure, documentation, and structural
assurance are the top five impact factors. Furthermore, based on these
studies, we compiled a comprehensive table of software trust impact factors
and identified as many practical metrics as possible for each factor to be
measured.

Although the majority of the trust factors we collected are based on empirical
research, and only a few models or metrics are widely used in real-world
projects, they reflect, to some extent, the concerns expressed by software
engineers or end-users during the software selection or evaluation process.

According to the literature, software trust serves as the foundation for
cooperation between hubs and stakeholders in SECO, while software trust is
critical for cooperation between actors, current research on software trust
focuses exclusively on software and components, particularly software quality.
Limited measures of trust assessment cannot be the final word to success, it
should be expanded to include other entities to analyze trust in the software
ecosystem in a multi-level, multi-perspective manner. Next, we will conduct a
survey of software engineers to ascertain their perceptions of software trust
in the future, paving the way for the design of a community-managed
infrastructure that serves as a trust layer for SECO.

###### Acknowledgements.

We thank the SecureSECO team for their constructive feedback and help in the
creation of this article. In particular, we thank Donny Groeneveld, Venja
Beck, and Floris Jansen for their diligence in performing their part of the
literature review, i.e., filtering, double-checking, and processing the
articles. Furthermore, we thank them for the lengthy and productive
discussions that led to an inter-rater agreement between the SLR team members.
Finally, we thank Hidde Reeskamp and Joost Gadellaa for their excellent
comments on the early versions of this article.

## References

  * (1) Pietro Abate, Roberto DiCosmo, Ralf Treinen and Stefano Zacchiroli  “MPM: a modular package manager”  In _Proceedings of the 14th international ACM Sigsoft symposium on Component based software engineering_ , 2011, pp. 179–188
  * (2) Gene M Alarcon et al.  “Trust Perceptions of Metadata in Open-Source Software: The Role of Performance and Reputation”  In _Systems_ 8.3 Multidisciplinary Digital Publishing Institute, 2020, pp. 28
  * (3) TA Alspaugh, HU Asuncion and W Scacchi  “The role of software licenses in open architecture ecosystems. 1st Int’l Work. Soft”  In _Ecosystems (IWSECO-2009)_ , 2009, pp. 4–18
  * (4) Ed Amoroso et al.  “Toward an approach to measuring software trust”  In _Proceedings. 1991 IEEE Computer Society Symposium on Research in Security and Privacy_ , 1991, pp. 198–198  IEEE Computer Society
  * (5) Afsah Anwar et al.  “Cleaning the NVD: Comprehensive quality assessment, improvements, and analyses”  In _IEEE Transactions on Dependable and Secure Computing_ IEEE, 2021
  * (6) Claudia Ayala et al.  “Selection of third party software in Off-The-Shelf-based software development—An interview study with industrial practitioners”  In _Journal of Systems and Software_ 84.4 Elsevier, 2011, pp. 620–637
  * (7) Anas Bassam Al-Badareen et al.  “Software quality models: A comparative study”  In _International Conference on Software Engineering and Computer Systems_ , 2011, pp. 46–55  Springer
  * (8) Pat Barclay  “Reputation”  In _The handbook of evolutionary psychology_ Wiley Online Library, 2015, pp. 1–19
  * (9) Paul C Bauer  “Conceptualizing trust and trustworthiness”  In _Political Concepts Working Paper Series_ Working paper published in: Political Concepts Working Paper Series, 2019
  * (10) Veronika Bauer, Lars Heinemann and Florian Deissenboeck  “A structured approach to assess third-party library usage”  In _2012 28th IEEE International Conference on Software Maintenance (ICSM)_ , 2012, pp. 483–492  IEEE
  * (11) Manuel Becerra, Randi Lunnan and Lars Huemer  “Trustworthiness, risk, and the transfer of tacit and explicit knowledge between alliance partners”  In _Journal of Management Studies_ 45.4 Wiley Online Library, 2008, pp. 691–713
  * (12) Keith Bennett et al.  “Service-based software: the future for flexible software”  In _Proceedings Seventh Asia-Pacific Software Engeering Conference. APSEC 2000_ , 2000, pp. 214–221  IEEE
  * (13) Patrik Berander et al.  “Software quality attributes and trade-offs”  In _Blekinge Institute of Technology_ 97.98, 2005, pp. 19
  * (14) Christopher Bogart, Christian Kästner, James Herbsleb and Ferdian Thung  “How to break an API: cost negotiation and community values in three software ecosystems”  In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2016, pp. 109–120
  * (15) Pierre Bourque and eds R.E.  “Guide to the software engineering body of knowledge”  IEEE Computer Society, 2014
  * (16) HA Boyes, P Norris, I Bryant and T Watson  “Trustworthy Software: lessons fromgoto fail’& Heartbleed bugs”  In _9th IET International Conference on System Safety and Cyber Security_ IET, 2014, pp. 1–7
  * (17) Sven Bugiel, Lucas Vincenzo Davi and Steffen Schulz  “Scalable trust establishment with software reputation”  In _Proceedings of the sixth ACM workshop on Scalable trusted computing_ , 2011, pp. 15–24
  * (18) Nina Buscemi et al.  “Single data extraction generated more errors than double data extraction in systematic reviews”  In _Journal of clinical epidemiology_ 59.7 Elsevier, 2006, pp. 697–703
  * (19) Yuanfeng Cai and Dan Zhu  “Reputation in an open source software community: Antecedents and impacts”  In _Decision Support Systems_ 91 Elsevier, 2016, pp. 103–112  DOI: 10.1016/j.dss.2016.08.004
  * (20) Justin Cappos, Justin Samuel, Scott Baker and John H Hartman  “A look in the mirror: Attacks on package managers”  In _Proceedings of the 15th ACM conference on Computer and communications security_ , 2008, pp. 565–574
  * (21) Justin Cappos, Justin Samuel, Scott Baker and John H Hartman  “Package management security”  In _University of Arizona Technical Report_ , 2008, pp. 08–02
  * (22) Eugenio Capra, Chiara Francalanci, Francesco Merlo and Cristina Rossi-Lamastra  “Firms’ involvement in Open Source projects: A trade-off between software structural quality and popularity”  In _Journal of Systems and Software_ 84.1 Elsevier, 2011, pp. 144–161
  * (23) Luigi Catuogno, Clemente Galdi and Giuseppe Persiano  “Secure dependency enforcement in package management systems”  In _IEEE Transactions on Dependable and Secure Computing_ 17.2 IEEE, 2017, pp. 377–390
  * (24) Jagat Sesh Challa et al.  “Integrated software quality evaluation: a fuzzy multi-criteria approach”  In _Journal of Information Processing Systems_ 7.3 Korea Information Processing Society, 2011, pp. 473–518
  * (25) Yanto Chandra and Liang Shang  “Qualitative research using R: A systematic approach”  Springer, 2019
  * (26) Souti Chattopadhyay et al.  “A tale from the trenches: cognitive biases and software development”  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , 2020, pp. 654–665
  * (27) PYC Chau  “Selection of packaged software in small businesses”  In _European Journal of Information Systems_ 3.4 Taylor & Francis, 1994, pp. 292–302
  * (28) Bodin Chinthanet et al.  “What makes a good Node. js package? Investigating Users, Contributors, and Runnability”  In _arXiv preprint arXiv:2106.12239_ , 2021
  * (29) Jin-Hee Cho, Kevin Chan and Sibel Adali  “A survey on trust modeling”  In _ACM Computing Surveys (CSUR)_ 48.2 ACM New York, NY, USA, 2015, pp. 1–40
  * (30) Jin-Hee Cho et al.  “Stram: Measuring the trustworthiness of computer-based systems”  In _ACM Computing Surveys (CSUR)_ 51.6 ACM New York, NY, USA, 2019, pp. 1–47
  * (31) Francois Coallier  “Software engineering–product quality–part 1: quality model”  In _International Organization for Standardization: Geneva, Switzerland_ , 2001
  * (32) Harris M Cooper, James L Lindsay and Erika A Patall  “Research synthesis and meta-analysis”  In _Applied Methods_ , 2008, pp. 344
  * (33) Marc-Alexis Côté, Witold Suryn, Claude Y Laporte and Robert A Martin  “The evolution path for industrial software quality evaluation methods applying ISO/IEC 9126: 2001 quality model: example of MITRE’s SQAE method”  In _Software Quality Journal_ 13.1 Springer, 2005, pp. 17–30
  * (34) Kevin Crowston, Hala Annabi and James Howison  “Defining open source software project success”  In _ICIS 2003 Proceedings_ , 2003, pp. 28
  * (35) Tushar Kanti Das and Bing-Sheng Teng  “Trust, control, and risk in strategic alliances: An integrated framework”  In _Organization studies_ 22.2 Walter de Gruyter Berlin/New York, 2001, pp. 251–283
  * (36) Alexandre Decan and Tom Mens  “What do package dependencies tell us about semantic versioning?”  In _IEEE Transactions on Software Engineering_ IEEE, 2019
  * (37) Alexandre Decan, Tom Mens, Maëlick Claes and Philippe Grosjean  “When GitHub meets CRAN: An analysis of inter-repository package dependency problems”  In _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ 1, 2016, pp. 493–504  IEEE  DOI: 10.1109/SANER.2016.12
  * (38) Alexandre Decan, Tom Mens and Eleni Constantinou  “On the impact of security vulnerabilities in the npm package dependency network”  In _Proceedings of the 15th International Conference on Mining Software Repositories_ , 2018, pp. 181–191
  * (39) Vieri Del Bianco, Luigi Lavazza, Sandro Morasca and Davide Taibi  “A survey on open source software trustworthiness”  In _IEEE software_ 28.5 IEEE, 2011, pp. 67–75
  * (40) Eelco Dolstra, Andres Löh and Nicolas Pierron  “NixOS: A purely functional Linux distribution”  In _Journal of Functional Programming_ 20.5-6 Cambridge University Press, 2010, pp. 577–615
  * (41) Susan K Donohue and Joanne Bechta Dugan  “Modeling the” good enough to release” decision using ${V}$&${V}$ preference structures and Bayesian belief networks”  In _Annual Reliability and Maintainability Symposium, 2003._ , 2003, pp. 568–573  IEEE
  * (42) Susan K Donohue, Joanne Bechta Dugan and Charles L Brown  “Is My Software” Good Enough” to Release?-${A}$ Probabilistic Assessment”  In _29th annual IEEE/NASA Software engineering workshop_ , 2005, pp. 5–13  IEEE
  * (43) Ruian Duan et al.  “Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”  In _Network and Distributed Systems Security (NDSS) Symposium 2021_ , 2021
  * (44) Christian Durach, Joakim Kembro and Andreas Wieland  “A New Paradigm for Systematic Literature Reviews in Supply Chain Management”  In _Journal of Supply Chain Management_ 53, 2017  DOI: 10.1111/jscm.12145
  * (45) Robert Ellison, William Nichols and Carol Woody  “Measuring software assurance”  In _2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC)_ 2, 2016, pp. 359–364  IEEE
  * (46) Jonathan St BT Evans and David E Over  “Rationality in the selection task: Epistemic utility versus uncertainty reduction”  In _Psychological Review_ 103.2 American Psychological Association, 1996, pp. 356–363
  * (47) Umar Farooq, Antoine Nongaillard, Yacine Ouzrout and Muhammad Abdul Qadir  “A multi source product reputation model”  In _Computers in Industry_ 83 Elsevier, 2016, pp. 55–67
  * (48) Siamak Farshidi, Slinger Jansen and Mahdi Deldar  “A decision model for programming language ecosystem selection: Seven industry case studies”  In _Information and Software Technology_ Elsevier, 2021, pp. 106640
  * (49) Karl Fogel  “How To Run A Successful Free Software Project - Producing Open Source Software”  Scotts Valley, CA: CreateSpace, 2009
  * (50) Cristina Gacek, Ahmed Abd-Allah, Bradford Clark and Barry Boehm  “On the definition of software system architecture”  In _Proceedings of the First International Workshop on Architectures for Software Systems_ , 1995, pp. 85–94  Seattle, WA
  * (51) C Garcı́a  “Reputation management of an Open Source Software system based on the trustworthiness of its contributions”, 2015
  * (52) David Gefen and Mark Keil  “DEVELOPER RESPONSIVENESS AND PERCEIVED USEFULNESS.”  In _Academy of Management Proceedings_ 1996, 1996, pp. 313–317  Academy of Management Briarcliff Manor, NY 10510
  * (53) Amir Hossein Ghapanchi and Madjid Tavana  “A longitudinal study of the impact of open source software project characteristics on positive outcomes”  In _Information Systems Management_ 32.4 Taylor & Francis, 2015, pp. 285–298
  * (54) Abraham Ghebrehiwet Ghebremedhin  “Combining static source code analysis and threat assessment modeling for testing open source software security”, 2012
  * (55) Antonios Gkortzis, Daniel Feitosa and Diomidis Spinellis  “Software reuse cuts both ways: An empirical analysis of its relationship with security vulnerabilities”  In _Journal of Systems and Software_ 172, 2021, pp. 110653  DOI: https://doi.org/10.1016/j.jss.2020.110653
  * (56) Manish Godse and Shrikant Mulik  “An approach for selecting software-as-a-service (SaaS) product”  In _2009 IEEE International Conference on Cloud Computing_ , 2009, pp. 155–158  IEEE
  * (57) Tim Goles, Stephen Hawk and Kate M Kaiser  “Information technology workforce skills: The software and IT services provider perspective”  In _Information Systems Frontiers_ 10.2 Springer, 2008, pp. 179–194  DOI: 10.1007/s10796-008-9072-9
  * (58) Danielle Gonzalez, Thomas Zimmermann, Patrice Godefroid and Max Schäfer  “Anomalicious: Automated Detection of Anomalous and Potentially Malicious Commits on GitHub”  In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_ , 2021, pp. 258–267  IEEE