# An Empirical Study of Blockchain Repositories in GitHub

Ajoy Das<EMAIL_ADDRESS>DISA Lab, University of CalgaryCalgaryCanada ,
Gias Uddin<EMAIL_ADDRESS>DISA Lab, University of
CalgaryCalgaryCanada and  Guenther Ruhe<EMAIL_ADDRESS>University of
Calgary, CanadaCalgaryCanada

(2022)

###### Abstract.

Blockchain is a distributed ledger technique that guarantees the traceability
of transactions. Blockchain is adopted in multiple domains like finance (e.g.,
cryptocurrency), healthcare, security, and supply chain. In the open-source
software (OSS) portal GitHub, we observe a growing adoption of Blockchain-
based solutions. Given the rapid emergence of Blockchain-based solutions in
our daily life and the evolving cryptocurrency market, it is important to know
the status quo, how developers generally interact in those repos, and how much
freedom they have in applying code changes. We report an empirical study of
3,664 Blockchain software repositories from GitHub. We divide the Blockchain
repositories into two categories: Tool (e.g., SDKs) and Applications (e.g.,
service/solutions developed using SDKs). The Application category is further
divided into two sub-categories: Crypto and Non-Crypto applications. In all
Blockchain repository categories, the contribution interactions on commits are
the most common interaction type. We found that more organizations
contributing to the Blockchain repos than individual users. The median numbers
of internal and external users in tools are higher than the application repos.
We observed a higher degree of collaboration (e.g., for maintenance efforts)
among users in Blockchain tools than those in the application repos. Among the
artifacts, issues have a greater number of interactions than commits and pull
requests. Related to autonomy we found that less than half of total project
contributions are autonomous. Our findings offer implications to Blockchain
stakeholders, like developers to stay aware of OSS practices around Blockchain
software.

Blockchain, GitHub, Repositories, Bitcoin, Cryptocurrency

††journalyear: 2022††copyright: acmcopyright††conference: The International
Conference on Evaluation and Assessment in Software Engineering 2022; June
13–15, 2022; Gothenburg, Sweden††booktitle: The International Conference on
Evaluation and Assessment in Software Engineering 2022 (EASE 2022), June
13–15, 2022, Gothenburg, Sweden††price: 15.00††doi:
10.1145/3530019.3530041††isbn: 978-1-4503-9613-4/22/06††ccs: Software and its
engineering Software libraries and repositories

##  1\. Introduction

> Ginni Rometty, CEO, 2019, IBM _“What the internet did for communications,
> Blockchain will do for trusted transactions.”_

Blockchain is a distributed ledger technology that stores records as identical
copies on multiple computers to ensure security and transaction traceability
(The Economist, 2015). It was invented in 2008 to serve the public transaction
ledger of the Bitcoin cryptocurrency. In 2014, Marc Andersson wrote a widely
cited article in New York Times by comparing the influence of bitcoin with
those of internet during 1993 (Andreesen, 2014). He predicted that prominent
industry vendors would soon develop critical solutions using Blockchain.
Indeed, by 2016, Blockchain funding overtook Bitcoin, i.e., Blockchain-based
solutions became more diverse than simply offering cryptocurrency-based
solutions (CoinDesk, 2016).

Many Blockchain projects are open-sourced to promote rapid growth and
adoption. As such, we find an increasing number of Blockchain-based software
repositories (denoted as ‘repos’ from hereon) in the open-source software
(OSS) portal GitHub. As of July 2021, GitHub hosts more than 200M repos from
over 65M developers worldwide. Several studies investigated the source code of
the Blockchain software in GitHub, e.g. (Asher Trockman et al., 2019). The
studies so far focused on a specific and small number of Blockchain repos like
healthcare operationalization of three cryptocurrencies (Osman, 2020), or the
analysis of 481 Bitcoin repos to learn the Bitcoin-based cryptocurrency
aspects (Berkhout et al., 2018). However, we are aware of no studies that
analyzed the diverse states and user interactions in the GitHub Blockchain
software ecosystem. Such insights can complement existing studies with
insights into the overall Blockchain software ecosystem.

This paper reports the results of an empirical study to learn about the
Blockchain OSS ecosystem in GitHub. An open-source Blockchain project tends to
be more secure as many people can verify security. People can trust such OSS
projects more, which is a crucial factor to ensure the success of a
Blockchain-based solution. Given that Github is the biggest platform to host
OSS projects, the Blockchain repos from Github can offer a comprehensive
overview of the states of the Blockchain OSS ecosystem. In our empirical
study, we analyze total 3,664 Blockchain repos from GitHub. Our goal is to
learn about the states of entities (e.g., users) and interactions among the
users as observed in the GitHub Blockchain repos. Our empirical study has two
major phases:

  * •

Phase 1. We manually label the 3,664 Blockchain repos under three categories:
Tool, Application (Cryptocurrency vs. non Cryptocurrency-based). This
categorization ensures that we have a better understanding on the type of
Blockchain OSS software systems being developed in GitHub.

  * •

Phase 2. We use the categories and combine those with the GitHub repo-based
metrics (e.g., star rating, forks, interactions, users) to conduct our
empirical study with a focus to understand the status quo of user engagement
and interactions for this special class of software systems.

Our study offers several findings as summarized below:

  * •

We find that Ethereum and Bitcoin Blockchain platforms have the highest number
of projects. The Blockchain tools show more development activities than the
Blockchain applications. Even though crypto applications are more popular, the
development activity of the crypto and non-crypto applications are mostly
similar. This finding corroborates with a sustained focus on cryptocurrencies
and the use of Blockchain to create those. Similar development activities
across crypto and non-crypto indicate the growing emphasis on applying
Blockchain-based solutions across diverse domains (details in Section 3.1).

  * •

Organizations are contributing more to the Blockchain repos than the
individual users. Many organizations (like Ethereum with mostly tool-based
repos and IBM with mostly application-based) own multiple Blockchain repos
(see Section 3.2).

  * •

We see higher degrees of collaboration among the users in Blockchain tool
repos than those in the application repos. The interactions of commit
contribution are higher in number than almost all other types of interactions.
Developers interact more via issues than via the other artifacts (commits and
pull requests). This finding indicates that Blockchain OSS repos hosted in
GitHub are being used by users and the users report their issues of using the
OSS by logging or commenting on issues (see Section 3.3).

  * •

We also find that the contributing users of the Blockchain repos are not
autonomous. This means that many internal developers generally face
restrictions in merging code changes. Therefore, future measures can be taken
to improve the autonomy of the internal developers in these projects (see
Section 3.4).

Our study findings offer implications to diverse stakeholders in the
Blockchain ecosystem:  (1) Blockchain vendors to improve autonomy in their
Blockchain repositories,  (2) Blockchain developers to learn about current
trends in Blockchain ecosystem,  (3) Blockchain researchers to offer new tools
and techniques for quality assurance of the Blockchain software and the
interactions among users.

Replication Package https://github.com/disa-lab/BlockchainEmpiricalEASE2022

##  2\. Study Data Collection

###  2.1. Project Selection

Picking Blockchain OSS repos from GitHub is a non-trivial task due to the
following reasons:  (1) There is no standard label/tag/identifier in a GitHub
repo that can be used to decide whether it is a Blockchain repo or not.  (2)
GitHub topic (which is introduced as a tag) is not universally applied to tag
each GitHub repo.  (3) A GitHub topic may use any Blockchain-related label
other than simply the keyword ‘Blockchain’.  (4) Searching by the keyword
‘Blockchain’ using GitHub search API will not provide all GitHub repo (e.g.,
when the description does not have Blockchain as the keyword) or may contain
false positives (e.g., when the repo simply mentions Blockchain while
referring to a non-Blockchain feature).  We, therefore, adopted the following
process (see Figure 1) to pick a list of Blockchain repos. We discuss the
steps below.

Figure 1. Steps in Blockchain repos selection process

First, we collect a set of 86 keywords that we can use to search diverse
Blockchain repos. We produce the keywords using the following sources:  (1)
Top 100 Blockchains-based cryptocurrencies from CoinMarketCap (Cry, 2021)
(Bitcoin, Ethereum, Polkadot, etc), to include top Blockchain platforms in
terms of cryptocurrency market capitalization;  (2) Top permissioned
Blockchain names (Hyperledger Fabric, Corda, etc.), as permissioned
Blockchains generally do not have cryptocurrencies, we need to add these names
so that we don’t miss them while searching for the Blockchain projects;  (3)
Popular Blockchain keywords (smart contract, solidity, dapp, etc.), so that we
can include Blockchain-related tools and applications that do not have the
Blockchain names directly in their descriptions.  However, we needed to
exclude some Blockchain names from the keyword list because they were very
common words (e.g., icon, waves), and they produced a substantial amount of
false positives in the project selection step. But, as we also included
popular Blockchain-related keywords (e.g., blockchain, dapp, smart contract),
the projects related to these Blockchains came up in the project selection
even though the Blockchain platform names were filtered out. Second, we
searched the GHTorrent dataset (Gousios, 2013) with the above keywords.
GHTorrent is an offline database that provides a snapshot of the GitHub
database. We used the latest GHTorrent database that was available during the
time of our analysis (January 2021). For each repo in GHTorrent, we searched
the keywords in the project name, description, and topic fields. This returned
802K GitHub repos.

###  2.2. Project Filtering

We then filtered the 802K repos to find relevant and informative Blockchain
repos. Specifically, we followed the same sampling criteria used by Gonzalez
et al. (Gonzalez et al., 2020), who previously investigated the ecosystem of
Machine Learning (ML) repositories in GitHub. To filter out unused, inactive,
and non-source code repositories, these criteria have been selected following
the best practices (Gousios and Spinellis, 2017; Kalliamvakou et al., 2016;
Munaiah et al., 2017; Neto et al., 2020). The filtering criteria are as
follows:  (1) Size: A repo must have a size greater than 0 (KB)  (2)
Popularity: Must have $\geq$ 5 stars  (3) Activity: The last commit of the
repo must be on and after January 2019  (4) Data Availability: Project data
must be accessible by the GitHub API and GHTorrent  (5) Content: Must be a
software project and not a tutorial, homework assignment, coding challenge, or
‘resource’ storage.  This step returned around 5200 GitHub repos. Not all the
repos returned after the third step are Blockchain-related projects because
few non-Blockchain projects also matched with some keywords that we used to
select the projects initially, we also found non-source code repositories
(such as course lectures) after the above filtering. So, we manually analyzed
each of the 5200 repos to discard irrelevant repositories. This step resulted
in our final list of 3,664 Blockchain repos.

##  3\. Empirical Study

In this section, we answer four research questions (RQ) by analyzing the data
of Blockchain software repositories collected in Section 2.

  1. RQ1.

How do the popularity and activity across the Blockchain repos vary? (Section
3.1)

  2. RQ2.

How do different users in the Blockchain repositories vary? (Section 3.2)

  3. RQ3.

How do the different users collaborate in the Blockchain repositories?
(Section 3.3)

  4. RQ4.

How autonomous are the internal users in the Blockchain repositories? (Section
3.4)

Like any OSS project in GitHub, a Blockchain project in GitHub can have two
major entities:  (1) Repository (repo),  (2) User.  The two entities can show
diverse status (e.g., user growth). Therefore, our RQ1 attempts to offer an
understanding of the popularity of the Blockchain repos in GitHub based on
their growth and user activity. Given the Blockchain stakeholders can be
diverse in GitHub, RQ2 offers an empirical evidence of the distribution of the
different users observed in the GitHub Blockchain repos and RQ3 investigates
how the users collaborate while RQ4 offers insights into how autonomous the
internal users to a repo are, i.e., whether they can make progress without
much help from external users to a repo.

###  3.1. RQ1. How do the popularity and activity across the Blockchain repos
vary?

####  3.1.1. Approach

First, we divide the Blockchain repos into two categories:  (1) Tool (i.e.,
offering a stand-alone API/SDK with Blockchain algorithm/model), and  (2)
Application (i.e., offering software to address specific use case scenario).
The definition of the categories is taken from Gonzalez et al. (Gonzalez et
al., 2020). Such categorization can also offer deeper insights into our
Blockchain projects. We thus attempted to categorize our Blockchain repos into
two categories (i.e., Tools and Applications). We can divide the blockchain
applications into two subcategories. Applications that focus on
cryptocurrencies that have direct monetary value, applications that focus on
encompassing blockchain features in other various domains (health, business,
law, entertainment, etc). Given the current focus on Blockchain-based
cryptocurrencies, we divided the ‘Application’ category into two sub-
categories:  (1) Application Crypto, and  (2) Application Non-Crypto/Others  ,
to find out whether these application categories are in fact separate in terms
of different metrics (e.g., development activities), or we can consider them
as a single group in future studies.

Table 1. Keywords to categorize the Blockchain repos Category | Keywords
---|---
Tool | tool, library, stack, client, node, protocol,
| helper, utility, , evm, scripts, package
| framework, sdk, service, ide, api, miner
Applications | exchanges, trading platform, trading bot,
(Crypto) | arbitrage, mining pool, payment channels,
| crypto/coin/currency/token related,
| wallet, wallet extensions
Applications | not categorized in tool or crypto application
(Others) |
Table 2. Agreements between coders Iteration ID | # of Repositories | Cohen $\kappa$
---|---|---
1 | 40 | 0.63
2 | 40 | 0.80

We labeled each Blockchain repo to one of the three categories in two steps.
(1) Automatic categorization: We attempted to label a repo automatically using
a suite of keywords (see Table 1). The list of keywords is picked based on our
manual analysis and by consulting the keywords for the ‘Tool’ category from
Gonzalez et al. (Gonzalez et al., 2020). The automated approach was able to
categorize about only 25% of the dataset.  (2) Manual labeling: Even though we
used an automated approach, we checked each repo manually to determine its
label. This helped us to fix any misclassification that happened at the
automatic categorization step as well as classify the repos that the automated
approach was not able to classify.  A total of three human coders participated
in the labeling process. First, two of the authors consulted in two iterations
to create a labeling guide. In the first iteration, both authors separately
labeled 40 repos. They then computed the agreement between them using Cohen
$\kappa$ value and resolved disagreements by discussing them together. The
Cohen $\kappa$ value was 0.63 which is considered substantial (see Viera et
al. (Viera and Garrett, 2005)). This process also helped them update the
labeling guide. In the second phase, both authors again separately labeled 40
repos and repeated the process of agreement calculation and disagreement
resolution. The Cohen $\kappa$ value was 0.80. Therefore, after this stage,
any of the authors could continue with the manual labeling of the rest of the
repos without introducing any individual bias. The first author then completed
the rest of the manual labels. Nevertheless, we further validated the manual
labels by consulting with a third coder, who is not an author of this paper.

Table 3. Summary of studied 3,664 repos in our dataset Project Type | #Repos | # Orgs | # Users
---|---|---|---
Blockchain Tools | 2,470 | 1,536 | 934
Applications (Crypto) | 546 | 253 | 293
Applications (Others) | 648 | 358 | 290
Total Blockchain Projects | 3,664 | 2,147 | 1,517

In a Github repository, the owner of the repository can set multiple words
describing the summary of a repository, these are called topics of the
repository. Topics generally mean what the repository is about. For example,
Blockchain-based projects use blockchain, cryptocurrency, etc. as their
topics. We also find these topics in the GHTorrent (Gousios, 2013) dataset
that we used to calculate the most common topics across different repository
types.

####  3.1.2. Results

During the manual labeling step (described in Section 2.1), we categorized
each repository in the dataset into the Blockchain platform it belongs to. We
found in total 71 Blockchain platforms in our 3,664 repos (e.g., Ethereum).
Figure 2 shows the top 10 of these platforms based on the total number of
repos. Ethereum wins the race, followed by Bitcoin, then follows the “others”
category (which denotes the rest of the Blockchain-based projects that we have
not explicitly monitored during the classification of Blockchain platforms).
Then comes the multi-Blockchain projects; these projects are those projects
that used or targeted more than one Blockchain platforms in their development
phase.

Figure 2. Top Blockchain platforms by # of projects

Top four topics per project category based on all the repos are:  (a) Tools:
blockchain, ethereum, bitcoin, solidity;  (b) Applications (Crypto): bitcoin,
ethereum, cryptocurrency, wallet;  (c) Applications (Others): ethereum,
blockchain, bitcoin, smart-contracts  . Overall, we can see some cross-cutting
topics like Ethereum and Bitcoin which are mainly Blockchain platform names
and these platforms are used in Blockchain-based application development. We
also see specific topics like ‘wallet’ in Application Crypto, which is to
develop a virtual wallet for the cryptocurrencies.

In Figure 3, we show the violin plot distribution of two popularity metrics (#
of stars and # of forks) per category. Overall, the median values are similar
across the three categories. Still, the differences in quartiles are greater
between Tools and Applications Others, which are also two categories with the
most repos.

Figure 3. Distribution of the repos by # of stars and # of forks

In Figure 4, we show the distribution of the commits, issues, and pull
requests per repository type. We can see that, just like the above popularity
measures, tool repos have a higher number of commits, issues, and pull
requests than the application repos.

We performed Kruskal-Wallis tests on both the popularity and activity measures
and found that in both cases distribution of the measures is not the same
across the repository types (p $<$ 0.001). But from the Dunn’s tests
(Bonferroni adjustment), we see both for the # of stars and # of forks, the
distributions of Tool repositories are not significantly different than the
Application Crypto repositories (p $>$ 0.05). On the other hand, we found from
the Dunn’s tests that the distribution of the activity measures (all of the
commits, issues, and pull requests) of the crypto applications are not
different from the non-crypto ones (p $>$ 0.05). The rest of the distributions
are significantly different from each other. Hence, we can say, though the
popularity of the crypto applications is similar to the popularity of tools
repos, the development activities of the crypto applications are primarily
identical to non-crypto applications.

Figure 4. Violin plots for # of Commits, # of Issues, and # of Pull Requests
per repo category

RQ1. How do the popularity and activity across the Blockchain repos vary? (1)
Ethereum (31.7%) and Bitcoin (15.9%) projects occupy a good portion of the
total repositories. (2) The tools based Blockchain repos show more dev
activities (e.g., issue creation) than the application based Blockchain repos.
(3) Though the crypto applications are more popular, the development activity
of the non-crypto applications are mostly similar to crypto applications.

###  3.2. RQ2. How do different users in the Blockchain repositories vary?

####  3.2.1. Approach

We got the ownership data of the repositories from the dataset itself. But
finding out the internal and external users was not a straightforward task.
Following the steps discussed by Gonzalez et al. (Gonzalez et al., 2020), we
calculated and summarized each repository user’s contribution such as how many
issues, pull requests, or commits the user created, how many of them are
closed/merged by the user himself/herself, how many of them have been
processed by other users instead, etc. After summarizing the degree of a
user’s contribution in a repository, we categorized them into internal and
external users. We marked the internal users with significant participation in
processing other users’ issues or pull requests as the maintainers of the
repository.

####  3.2.2. Results

The distributions of project owners by three project categories are shown on
Table 3. We also indicate the number of total repos, the total number of
contributing organizations, and the total number of users that created those
repositories for each project category. The number of contributing
organizations is 2,147 among the total 3664 repositories demonstrating a
significant focus on Blockchain OSS in GitHub from the software vendors.

As we have shown in Table 3, organizations own most of the Blockchain projects
of our analysis (58.6%), the rest are owned by individual user accounts. In
Blockchain repositories, 501 accounts own more than one repositories (20.9% of
the dataset), and 86 accounts own at least 5 repositories (3.6% of the
dataset). In fact, the majority of GitHub accounts with more than five
Blockchain repos are organizational accounts. The top 3 accounts with most
Blockchain repositories are Ethereum (46 repos), IBM (32 repos), and input-
output-hk (30 repos). Here we see, Ethereum is the topmost contributor,
followed by IBM. The organizations are working on multiple Blockchain-based
solutions in GitHub, which signifies the significant focus on the Blockchain
OSS ecosystem from the industry. Some of these organizations (e.g., Ethereum,
Stellar) are dedicated to support specific Blockchain platforms, while others
offer Blockchain-based solutions (e.g., IBM). The top organization Ethereum,
has mostly tool-based repositories as it works on supporting the
infrastructure of the Ethereum Blockchain platform. On the other hand, IBM
works on multiple Blockchains with most of its repositories are non-crypto
applications (56.3%).

Figures 5a and 5b show the distribution of internal and external users across
Blockchain project categories. We see that tool repositories has higher number
of both external and internal users than the application repositories.

(a) Distribution of internal users

(b) Distribution of external users

Figure 5. Distribution of (a) Internal users and (b) External users (outliers
omitted)

Among the Blockchain tool repos, bitcoin (Bit, 2021) has the most internal
users (614) and go-ethereum (Eth, 2021) has the most external users (4402).
Among the crypto application repos, gekko (van Rossum, 2021) has the most
internal users (199) and metamask-extension (Met, 2021) has the most external
users (3013). Among non-crypto application repos, condenser (Ste, 2021) has
the most internal users (82) and monero-gui (Mon, 2021) has the most external
users (560).

The performed Kruskal-Wallis tests denote that the distribution of the users
is significantly different across repo types. But the Dunn’s tests with
Bonferroni adjustment show that the distribution of the internal users for
crypto applications are not significantly different than the non-crypto repos
(p $>$ 0.05). On the other hand, for the tools and crypto applications, the
distributions of the external users do not differ significantly (p $>$ 0.05).

RQ2. How do different users in the Blockchain repositories vary? (1) We find
more organizations contributing to the Blockchain repos than individual users.
(2) Organizations like Ethereum and IBM own multiple Blockchain repos. While
Ethereum repos are mostly tool-based, IBM repos are mostly application-based.
(3) The median numbers of internal and external users in tools are higher than
the application repos.

###  3.3. RQ3. How do the different users collaborate in the Blockchain
repositories?

####  3.3.1. Approach

We quantify the collaboration among users in a repo based on five types of
interactions defined by Gonzalez et al. (Gonzalez et al., 2020):  (1)
Contribution,  (2) Maintenance,  (3) Process,  (4) Review, and  (5)
Discussion.  Description of each of these interaction types have been provided
in Table 4. The goal then is to compute how frequently users show the above
interaction types when interacting with each other in a repository.

Table 4. Interaction types used to measure Collaboration Type | Description
---|---
Contribution | Interaction that happens between the author
| and the committer of a commit.
Maintenance | Interaction that happens between two users that
| initiate any event (i.e. approve, except
| commenting) for the same artifact (issue/
| pull request), and any user who is not the
| creator (opener/reporter) of the artifact.
Process | Interaction that happens between the creator
| (opener/reporter) of the artifact and another
| user who initiates a maintenance event.
Review | Interaction that happens between a user
| who comments on an artifact and it’s creator
| (author/reporter/opener).
Discussion | Interaction that happens between two users
| who comments on an artifact and the users
| who are not the creator of the artifact.

####  3.3.2. Results

We find that median number of the user per artifact per repository for  (a)
Tools are Commits (15.5), Issues (10), Pull Requests (7);  (b) Applications
(Crypto) are Commits (10), Issues (8), Pull Requests (4);  (c) Applications
(Others) are Commits (12), Issues (9), Pull Requests (7)  . Overall, users’
median per artifact is higher in the Blockchain tools (across all three
categories) than the Blockchain application repos.

In Figures 6, 7, and 8, we quantify the collaboration among different users in
a repo based on the five interaction types defined in Table 4:  (a) Code
contribution (blue bar) denotes the frequency of interaction between the
author of a commit and the committer of that commit. We see that this
interaction is the most prevalent in both tools and non-crypto applications.
(b) Maintenance (brown bar) denotes the frequency of interaction between two
users to approve/comment artifacts like pull requests. We see issue
maintenance interactions are higher than the pull request maintenance in all
repo types.  (c) Process (green bar) denotes the frequency of interactions
between the creator and a maintainer of an artifact. In all repo types,
interaction in issue processing is higher than the pull request processing.
(d) Review (red bar) denotes the frequency of interactions between the creator
and a commenter of an artifact. For tools and non-crypto applications, reviews
in pull requests are higher than the reviews in issues.  (e) Discussion
(purple bar) denotes the frequency of interactions between two users who are
not the artifact’s creator but who nonetheless commented on the artifact.
Discussion is the most prevalent interaction for crypto applications.  We see
that almost all of the collaborative interaction types are higher in tool
repositories than the application repos (note that the scale is different in
each graph).

Figure 6. Collaborative interactions per artifact in Blockchain Tool
repositories (outliers omitted) Figure 7. Collaborative interactions per
artifact in Blockchain Crypto Applications repositories (outliers omitted)
Figure 8. Collaborative interactions per artifact in Blockchain Others (Non-
Crypto) Applications repositories (outliers omitted)

RQ3. How do the different users collaborate in the Blockchain repositories?
(1) We observe higher degrees of collaboration (e.g., for maintenance efforts)
among users in Blockchain tools than those in the application repos. (2) The
amount of interactions that happened in commit contribution are higher in
number than almost all other types of interactions. (3) Among the artifacts,
issues has overall greater number of interactions than commits and pull
requests.

###  3.4. RQ4. How autonomous are the internal users in the Blockchain
repositories?

####  3.4.1. Approach

To measure the autonomy of a repository, Gonzalez et al. (Gonzalez et al.,
2020) divided the users of a repo into  (1) Maintainer,  (2) Autonomous
contributor, and  (3) Dependent contributor.  The definition of these user
types is provided in Table 5. The goal is to find the distribution of the
above user types in a repository. Based on the percentage of users that belong
to each group, the autonomy of the whole project is calculated.

Table 5. User types used for Autonomy calculation Type | Description
---|---
Maintainer | A user who has merged or closed artifacts
| (issues and/or pull requests) opened by others.
Autonomous | A user who committed majority of his/her
Contributor | commits and also and/or self-merged
| majority of his/her pull requests.
Dependent | A user whose majority of the commits were
Contributor | not committed by others, and/or other users also
| merged/closed majority of his/her pull requests.

####  3.4.2. Results

In Figure 9, we show the proportion of three user types (maintainer,
autonomous and dependent contributors as defined in Table 5) across the
Blockchain repo categories. We see that:  (1) The proportion of maintainers in
a repo is higher in crypto applications than both tools and non-crypto
applications.  (2) The proportion of autonomous users are the same across all
of the repo types.  (3) The proportion of dependent contributors are also the
same across all types of repositories.  Overall, we see a lower proportion of
autonomous users but a higher proportion of dependent contributors across all
Blockchain repo categories. This makes the Blockchain repos less autonomous,
indicating more restriction in the Blockchain repos to make quick code
changes. On the other hand, we can find some inherent reasons for this less
autonomy in the Blockchain repositories. Blockchain networks are generally
distributed networks, and no central authority has control over them. So,
updating the network for fixing a bug becomes difficult because all the
relevant parties need to update the software simultaneously across the
network. So making the systems fault-tolerant and bug-free early on is very
important for Blockchain-based projects, as merging buggy code into the
codebase can be fatal. Hence multiple reviews are generally performed before
the code changes are approved and merged, which in turn lessen the autonomy of
the developers.

RQ4. How autonomous are the internal users in the Blockchain repositories? (1)
There are no significant differences in the autonomy of the teams across all
repo types. Less than half of total project contributions are autonomous, that
means that the development teams in Blockchain repositories are not autonomous
in terms of pushing changes to the code base. (2) The proportion of autonomous
and dependent contributors are equal.

Figure 9. Distribution of user types per project type

##  4\. Discussions

###  4.1. Blockchain Vs Non-Blockchain Repositories

Blockchain is a relatively new technology; it is important to know whether and
how the findings we observed could differ from the non-Blockchain projects.
We, therefore, randomly pick an equal number of non-Blockchain repos (i.e.,
3664) from GitHub to compare these two categories. In this subsection, we
compare both the states and the interactions of different entities in
Blockchain vs. non-Blockchain projects using total 7,328 GitHub repos.

Data Collection. At first, we sampled 10k repositories from Github using
Github API after applying the same filtering criteria as of those applied for
the Blockchain repositories (described in Section 2.2). Then we randomly
selected an equal number of non-Blockchain repos (i.e., 3664) for the non-
Blockchain GitHub repository set from the filtered set of 10k repos after
removing any Blockchain repositories from this list.

Table 6. Summary of Blockchain (BC) vs Non-Blockchain (Non-BC) Metrics in the studied 7,328 repos in our dataset Metrics | BC | Non-BC
---|---|---
Avg | Median | Avg | Median
Commits | 107 | 53 | 32 | 18
Issues | 30 | 12 | 9 | 4
Pull Requests | 19 | 8 | 5 | 3
Internal Users | 5 | 4 | 3 | 3
External Users | 3 | 1 | 3 | 1
Maintainers (%) | 43 | 44 | 54 | 50
Autonomous Contrib (%) | 32 | 33 | 35 | 40
Dependent Contrib (%) | 34 | 33 | 24 | 25

We show the summary of the calculated metrics of the Blockchain vs. non-
Blockchain repos in Table 6. We can see in Table 6, the Blockchain repos have
a higher number of commits, issues, and pull requests than the non-Blockchain
repos in our dataset, denoting that Blockchain-based repos in our database are
more actively developed than the non-Blockchain repos. There could be several
reasons for this:

  * •

Larger investment in Blockchain projects in terms of both time and resources;

  * •

The domain is still actively growing, so rapid development is going on;

  * •

Blockchain industry’s one of the key selling points is fault tolerance of the
systems, and to catch the bugs in the development phase, the artifacts (such
as commits and pull requests) go through a lot of processing;

Users. We find that organizations own 2,147 (58.6%) repos among 3664
Blockchain repos. In comparison, only 913 (24.9%) repos of 3664 non-Blockchain
repos are owned by organizations which denote that organizations have a
significant focus on the Blockchain software ecosystem. From Table 6 we see,
the number of both internal and external users is higher in the Blockchain
repos.

Figure 10. Collaborative interactions per artifact in Non-Blockchain
repositories (outliers omitted)

Interactions. Comparing Figure 10 with Figures 6, 7, and 8 we see that:  (1)
The median of Contribution (blue bar) frequencies for Blockchain repos are
between 15-20, but it is around 4 for Blockchain repos.  (2) The median of
Maintenance (brown bar) is around 5-10 for Blockchain repos and 1-2 for non-
Blockchain repos. This indicates that more non-authors collaborate in the
Blockchain artifact maintenance process.  (3) The median for Process (green
bar) is around 5 for Blockchain repos and 1-2 for non-Blockchain repos.  (4)
The median of Review (red bar) is around 5-10 for Blockchain repos and 1-2 for
non-Blockchain repos.  (5) The median for Discussion (purple bar) is around 10
for Blockchain repos and 2-3 for non-Blockchain repos.  Thus, across all the
five interaction types, we observe higher degrees of collaboration among the
users in Blockchain repos than in non-Blockchain repos.

Autonomy. In Table 6 we see that:  (1) The proportion of maintainers in a repo
is similar between Blockchain and non-Blockchain repos.  (2) The proportion of
autonomous users is more in non-Block-chain repos (40% vs. 33% in Blockchain
repos).  (3) The proportion of dependent contributors is more in Blockchain
repos (33% vs. 25% in non-Blockchain repos).  Therefore, the lower proportion
of autonomous users but a higher proportion of dependent contributors make the
Blockchain repos less autonomous, which indicates more restriction in the
Blockchain repos to make quick code changes.

###  4.2. The Case of Archived Blockchain Repos

In software teams, obsolete projects are generally archived. Given Blockchain
is a rapidly evolving paradigm, it is expected that innovative ideas are
rapidly implemented in the Blockchain software ecosystem, with little or no
guidance and previous expertise of developing such solutions. As such, it may
happen that some ideas are less successful than others, resulting in their
archival. We investigated whether a repo is archived using two approaches:

(1) We used Github API to search for the repositories that are marked as
archived  (2) We searched for relevant keywords (e.g., deprecated, obsolete,
archived, no longer maintained, etc) in the Readme file of the repo.

We find total 332 archived Blockchain repos and 316 archived non-Blockchain
repos. Thus, we find a slightly higher proportion of archived repos in
Blockchain than in non-Blockchain repos (9.1% vs 8.6%).

In Figure 11a, we show the distribution of the archived repos over time based
on their creation time. For the archived Blockchain repos, we see a notable
upswing between 2017-2018. No such upswing is visible for the non-Blockchain
archived repos. To investigate the reasons of the upswing in 2017-2018 for the
Blockchain repos, we checked the evolution of all the repos based on their
creation time (see Figure 11b). For both the Blockchain and non-Blockchain
repos (archived + non-archived), we see an upswing between 2017 - 2018, where
the upswing is more prominent for Blockchain repos. This denotes that the more
Blockchain repos got created in 2017 - 2018, the more Blockchain repos also
got archived at the time. This indicates rapid evolution and implementation of
new ideas in the Blockchain GitHub repos, which led to the creation of more
new repos as well as the closure (i.e., archival) of old repos compared to the
Blockchain repos.

(a) Trend of project archival

(b) Trend of project creation

Figure 11. Trend of (1) Archiving Projects (2) New projects

We analyze the reasons of archival in the repos using 200 randomly sampled
repos, 100 each from Blockchain and non-Blockchain types. The sample size for
each type is statistically significant with a 95% confidence level and 10
confidence interval. We then checked the description (i.e., Readme) file of
each sampled repo, where the repo owners could provide their archival reasons.
The first two authors analyzed the reasons together. We observed total of four
archival reasons:  (1) New repo migrationdenotes when a project development
has been moved to a new repo due to change of ownership/name, rebuilding of
the project from scratch, etc.  (2) Resource unavailabilitydenotes the lack of
resources (e.g., developers, funding) to continue a repo.  (3) Obsolete
dependencydenotes that an essential dependency (e.g., a Blockchain platform or
an API/SDK) for a repo became obsolete, which led the project owner closing
the project.  (4) Obsolete usecasesdenotes a project is no longer needed
because its use cases are no longer relevant in the evolving Blockchain
ecosystem.  In Table 7, we present the distribution of five archival reasons
categories across the 200 sampled repos. The archival reasons for more than
50% repos are not provided (see ‘Not Mentioned’ in Table 7). Except that, the
two other major archival reasons are ‘Obsolete usecases’ and ‘New repo
migration’, both of which are observed more in the Blockchain repos.

Table 7. Distribution of the archival reasons (in %) of repos Reason | #Blockchain | #Non Blockchain
---|---|---
New repo migration | 16 | 10
Resource unavailability | 3 | 3
Obsolete dependency | 4 | 4
Obsolete usecases | 26 | 24
Not mentioned | 51 | 59

###  4.3. Implications of Findings

Our study findings can be instrumental to following stakeholders.

(1) Blockchain Vendors.We find that majority of Blockchain repos in our
dataset are owned by organizations (e.g., IBM, Ethereum), which is more than
the user-owned repos (58.6% vs. 41.4%). Compared to the non-Blockchain repos,
users in Blockchain repos show less autonomy while pushing changes in the
codebase. This could indicate that the Blockchain OSS ecosystem is closely
guarded by few users and industry vendors, which then can create a bottleneck
while making rapid progress. Therefore, Blockchain repo owners in GitHub can
create an environment with less restriction so that users can enjoy more
autonomy while creating features.
(2) Blockchain Developersmay expect more development activities and
collaborative interaction in tool repos than the application repos. As the
development teams of the repos are not autonomous, the developers may expect
some restrictions (i.e., multiple review cycles) in applying code changes to
the repos.
(3) Blockchain Career Enthusiastscan take note of the growth of Blockchain
repos to stay aware of this rapidly evolving ecosystem. Blockchain repo
creation peaked in 2016 - 2018, while we a high growth rate in the Blockchain
repos in 2017 - 2018 (Figure 11b). Such insights can help career enthusiasts
to make decisions like when to make a career change and when to invest in the
ecosystem.  (4) Blockchain Researchers.We observed much more interactions of
Blockchain users around issues in GitHub (see Figures 6, 7, and 8). This
denotes that quality assurance of Blockchain repos is getting attention from
Blockchain users. While significant research in Blockchain-based research has
focused on improving security (e.g., smart-contracts (Brent et al., 2018;
Colombo et al., 2018)), our results show that research can develop tools to
improve the autonomy in the repos. Autonomy can be improved by improved
collaboration and the generation of automatic documentation for the Blockchain
repos/APIs, e.g., by automatically expanding the documentation from online
resources (Uddin et al., 2020; Uddin and Khomh, 2019), where discussions and
reviews about new and emerging techniques are readily available (Chakraborty
et al., 2021; Uddin and Khomh, 2017)).

##  5\. Threats to Validity

Internal validity threats relate to the author bias while conducting the
analysis. We mitigated the bias in our manual labeling of Blockchain repos by
using three human coders. We report the Cohen $\kappa$ agreement between the
first two coders in two iterations, which show the substantial to a perfect
agreement (according to Viera et al. (Viera and Garrett, 2005)) between the
coders. Construct Validity threats related to the potential errors in the
study methodologies. To identify the Blockchain-related projects, we used both
topics(Gonzalez et al., 2020) and the project description metadata and
followed best practices to analyze GitHub repos (e.g., (Gonzalez et al., 2020;
Gousios and Spinellis, 2017; Neto et al., 2020)). External Validity threats
relate to the generalizability of the findings. We study Blockchain projects
that are hosted publicly in GitHub. We followed an extensive sampling process
to collect the Blockchain and non-Blockchain repos (see Section 2.1). As we
noted in Section 2, we picked our Blockchain repos based on a suite of 86
Blockchain-specific keywords. This approach was necessary because there is no
other way to find the list of Blockchain repos in GitHub. Although GitHub
offers the ‘topic’ feature to tag a repo, not all repos are tagged by the
topics. Many of the Blockchain repos we found do not have any topic tag. Our
keyword-based approach returned 802K GitHub repos. This large number of repos
indicates that our keyword-based approach is sufficient to find a good number
of Blockchain repos.

##  6\. Related Work

Related work can broadly be divided into two types: Studies to understand the
trends in Blockchain software and Techniques developed to address problems in
Blockchain-based solutions.

Studies. In this research, we study the evolution of Blockchain technology
from the perspective of OSS development in GitHub. Several recent empirical
studies on Blockchain used GitHub data. There have been several systematic
literature reviews that discussed the ongoing software engineering-related
research regarding blockchain-based software development (Vacca et al., 2021;
Demi et al., 2021). This includes testing and analysis of smart contracts,
performance and security of DApp, analyzing architecture types used during the
blockchain-based software development, etc. However, no previous studies
performed an empirical analysis of blockchain-based repos in the wild.
Trockman et al. (Asher Trockman et al., 2019) report one year of development
activity of around two hundred cryptocurrencies and observe that the
popularity of the repos is associated with a higher market capitalization of
the corresponding platforms. Zheng et al. (Zheng et al., 2018) conducted an
extensive survey on Blockchain taxonomy, consensus algorithms, and technical
challenges. Reyna et al. (Reyna et al., 2018) investigate the challenges and
potential applications for block-chain related IoT applications but do not
analyze the development activity of the Blockchain projects.

Techniques. Analysing and finding vulnerabilities in Blockchain is an active
research area (Li et al., 2020; Park and Park, 2017). There are also research
opportunities in Blockchain scalability, i.e., improving throughput and
reducing latency for cryptocurrencies (Zhou et al., 2020; Xie et al., 2019).
Researchers have also presented tools such as Vandal (Brent et al., 2018),
ContractFuzzer (Jiang et al., 2018), TEETHER (Krupp and Rossow, 2018) to
automatically detect security vulnerabilities in Blockchain smart contacts.

While the above studies limit to either cryptocurrencies or specific domains
(e.g., healthcare), we study the overall Blockchain OSS ecosystem in GitHub
based on a large sample of Blockchain repos. Similar to our study, many of the
recent studies (Gonzalez et al., 2020; Dabbish et al., 2012; Cataldo et al.,
2008) study Collaboration among developers. Some recent analysis by Google
(Murphy-Hill et al., 2019) and Microsoft (Storey et al., 2019) showed that
Autonomy is a crucial factor for productivity. We thus study both
collaboration and autonomy of the users in the Blockchain repos. While they
studied the states on the ML universe in GitHub, we study the Blockchain OSS
ecosystem in GitHub.

##  7\. Conclusions

The promise of complete transparency, traceability and data immutability has
made Blockchain a promising architecture to use in software for diverse
domains like finance (e.g., cryptocurrency), supply chain management,
security, etc. Given that this is a new but rapidly evolving ecosystem, it is
important to understand the states of the Blockchain ecosystem. We conduct an
empirical study by analyzing 3,664 Blockchain repositories in GitHub. We find
that this domain is growing rapidly just after the release of the first
Blockchain project in 2010 that peaked up in 2016 - 2018, when Blockchain-
based solutions are adopted in diverse domains besides cryptocurrency. We find
significantly more presence of large organizations in the Blockchain repos,
who own multiple Blockchain repos in GitHub. Moreover, we also see less
autonomy of users in the Blockchain repos. With a view to foster further
innovation in Blockchain OSS ecosystem, in our future work, we aim to study
more the causes of low autonomy in the Blockchain repo and develop guidelines
and tools to improve the user autonomy.

###### Acknowledgements.

Gias Uddin and Ajoy Das were supported by NSERC Grant (RGPIN-2021-02575).

## References

  * (1)
  * Bit (2021) 2021\.  bitcoin/bitcoin, Bitcoin Core integration/staging tree.  https://github.com/bitcoin/bitcoin.
  * Cry (2021) 2021\.  Cryptocurrency Prices, Charts And Market Capitalizations.  https://coinmarketcap.com/.
  * Eth (2021) 2021\.  ethereum/go-ethereum, Official Go implementation of the Ethereum protocol.  ethereum.
  * Met (2021) 2021\.  MetaMask/metamask-extension, The MetaMask browser extension enables browsing Ethereum blockchain enabled websites.  MetaMask.
  * Mon (2021) 2021\.  monero-project/monero-gui, Ui for Monero: the secure, private, untraceable cryptocurrency.  The Monero Project.
  * Ste (2021) 2021\.  steemit/condenser, The greatest application front-end to the Steem Blockchain.  steemit.
  * Andreesen (2014) Mard Andreesen. 2014\.  Why Bitcoin Matters.  https://dealbook.nytimes.com/2014/01/21/why-bitcoin-matters.
  * Asher Trockman et al. (2019) Asher Trockman, Rijnard van Tonder, and Bogdan Vasilescu. 2019\.  Striking Gold in Software Repositories? An Econometric Study of Cryptocurrencies on GitHub. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, Montreal, QC, Canada, 181–185.
  * Berkhout et al. (2018) Matthijs Berkhout, Fons van den Brink, Mart van Zwienen, Paul van Vulpen, and Slinger Jansen. 2018\.  Software Ecosystem Health of Cryptocurrencies. In _Software Business_ _(Lecture Notes in Business Information Processing)_. Springer International Publishing, Cham, 27–42.
  * Brent et al. (2018) Lexi Brent, Anton Jurisevic, Michael Kong, Eric Liu, Francois Gauthier, Vincent Gramoli, Ralph Holz, and Bernhard Scholz. 2018\.  Vandal: A Scalable Security Analysis Framework for Smart Contracts.  _arXiv:1809.03981 [cs]_ (2018).
  * Cataldo et al. (2008) Marcelo Cataldo, James D Herbsleb, and Kathleen M Carley. 2008\.  Socio-technical congruence: a framework for assessing the impact of technical and work dependencies on software development productivity. In _Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement_. 2–11.
  * Chakraborty et al. (2021) Partha Chakraborty, Rifat Shahriyar, Anindya Iqbal, and Gias Uddin. 2021.  How Do Developers Discuss and Support New Programming Languages in Technical Q&A Site? An Empirical Study of Go, Swift, and Rust in Stack Overflow.  _Information and Software Technology (IST)_ (2021), 19.
  * CoinDesk (2016) CoinDesk. 2016.  State of Blockchain Q1 2016.  https://www.coindesk.com/state-of-blockchain-q1-2016.
  * Colombo et al. (2018) Christian Colombo, Joshua Ellul, and Gordon J. Pace. 2018\.  Contracts over Smart Contracts: Recovering from Violations Dynamically.  _Leveraging Applications of Formal Methods, Verification and Validation. Industrial Practice_ (2018), 300–315.
  * Dabbish et al. (2012) Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herbsleb. 2012\.  Social coding in GitHub: transparency and collaboration in an open software repository. In _ACM 2012 conference on computer supported cooperative work_. 1277–1286.
  * Demi et al. (2021) Selina Demi, Ricardo Colomo-Palacios, and Mary Sánchez-Gordón. 2021\.  Software Engineering Applications Enabled by Blockchain Technology: A Systematic Mapping Study.  _Applied Sciences_ 11, 7 (Jan. 2021), 2960\.
  * Gonzalez et al. (2020) Danielle Gonzalez, Thomas Zimmermann, and Nachiappan Nagappan. 2020\.  The State of the ML-universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub. In _Proc. MSR_. ACM, Seoul Republic of Korea, 431–442.
  * Gousios (2013) Georgios Gousios. 2013\.  The GHTorent dataset and tool suite. In _Proc. MSR_ _(MSR ’13)_. IEEE Press, 233–236.
  * Gousios and Spinellis (2017) Georgios Gousios and Diomidis Spinellis. 2017.  Mining Software Engineering Data from GitHub. In _Proc. ICSE_. IEEE, 501–502.
  * Jiang et al. (2018) Bo Jiang, Ye Liu, and W. K. Chan. 2018.  ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection. In _33rd ACM/IEEE International Conference on Automated Software Engineering_. 259–269.
  * Kalliamvakou et al. (2016) Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2016.  An in-depth study of the promises and perils of mining GitHub.  _Empirical Software Engineering_ 21, 5 (Oct. 2016), 2035–2071.
  * Krupp and Rossow (2018) Johannes Krupp and Christian Rossow. 2018.  TEETHER: Gnawing at Ethereum to Automatically Exploit Smart Contracts. In _27th USENIX Conference on Security Symposium_. 1317–1333.
  * Li et al. (2020) Xiaoqi Li, Peng Jiang, Ting Chen, Xiapu Luo, and Qiaoyan Wen. 2020.  A survey on the security of blockchain systems.  _Future Generation Computer Systems_ 107 (2020), 841–853.
  * Munaiah et al. (2017) Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017.  Curating GitHub for engineered software projects.  _Empirical Software Engineering_ 22, 6 (Dec. 2017), 3219–3253.
  * Murphy-Hill et al. (2019) Emerson Murphy-Hill, Ciera Jaspan, Caitlin Sadowski, David Shepherd, Michael Phillips, Collin Winter, Andrea Knight, Edward Smith, and Matt Jorde. 2019.  What Predicts Software Developers’ Productivity?  _IEEE Transactions on Software Engineering_ (2019).
  * Neto et al. (2020) Paulo Anselmo da Mota Silveira Neto, Umme Ayda Mannan, Eduardo Santana de Almeida, Nachiappan Nagappan, David Lo, Pavneet Singh Kochhar, Cuiyun Gao, and Iftekhar Ahmed. 2020.  A Deep Dive on the Impact of COVID-19 in Software Development.  _arXiv:2008.07048 [cs]_ (Aug. 2020).  arXiv:2008.07048 [cs]
  * Osman (2020) Khadija Osman. 2020\.  Studying the Health of the Bitcoin Ecosystem in GitHub.  (2020), 137.
  * Park and Park (2017) Jin Ho Park and Jong Hyuk Park. 2017.  Blockchain security in cloud computing: Use cases, challenges, and solutions.  _Symmetry_ 9, 8 (2017), 164.
  * Reyna et al. (2018) Ana Reyna, Cristian Martín, Jaime Chen, Enrique Soler, and Manuel Díaz. 2018.  On blockchain and its integration with IoT. Challenges and opportunities.  _Future generation computer systems_ 88 (2018), 173–190.
  * Storey et al. (2019) Margaret-Anne Storey, Thomas Zimmermann, Christian Bird, Jacek Czerwonka, Brendan Murphy, and Eirini Kalliamvakou. 2019.  Towards a theory of software developer job satisfaction and perceived productivity.  _IEEE Transactions on Software Engineering_ (2019).
  * The Economist (2015) The Economist. 2015\.  Blockchains: The great chain of being sure about things.  https://www.economist.com/briefing/2015/10/31/the-great-chain-of-being-sure-about-things.
  * Uddin and Khomh (2017) Gias Uddin and Foutse Khomh. 2017.  Automatic Summarization of API Reviews. In _Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering_. 12.
  * Uddin and Khomh (2019) Gias Uddin and Foutse Khomh. 2019.  Automatic opinion mining from API reviews from stack overflow.  _IEEE Transactions on Software Engineering_ (2019), 35.
  * Uddin et al. (2020) Gias Uddin, Foutse Khomh, and Chanchal K Roy. 2020.  Automatic Mining of API Usage Scenarios from Stack Overflow.  _Information and Software Technology (IST)_ (2020), 16.
  * Vacca et al. (2021) Anna Vacca, Andrea Di Sorbo, Corrado A. Visaggio, and Gerardo Canfora. 2021.  A Systematic Literature Review of Blockchain and Smart Contract Development: Techniques, Tools, and Open Challenges.  _Journal of Systems and Software_ 174 (April 2021), 110891\.
  * van Rossum (2021) Mike van Rossum. 2021\.  askmike/gekko, A bitcoin trading bot written in node.  https://github.com/askmike/gekko.
  * Viera and Garrett (2005) Anthony J. Viera and Joanne M. Garrett. 2005.  Understanding interobserver agreement: The kappa statistic.  _Family medicine_ 37, 4 (2005), 360–363.
  * Xie et al. (2019) Junfeng Xie, F Richard Yu, Tao Huang, Renchao Xie, Jiang Liu, and Yunjie Liu. 2019\.  A survey on the scalability of blockchain systems.  _IEEE Network_ 33, 5 (2019), 166–173.
  * Zheng et al. (2018) Zibin Zheng, Shaoan Xie, Hong-Ning Dai, Xiangping Chen, and Huaimin Wang. 2018.  Blockchain challenges and opportunities: A survey.  _International Journal of Web and Grid Services_ 14, 4 (2018), 352–375.
  * Zhou et al. (2020) Qiheng Zhou, Huawei Huang, Zibin Zheng, and Jing Bian. 2020\.  Solutions to scalability of blockchain: A survey.  _IEEE Access_ 8 (2020), 16440–16455.


# Computationally efficient neural network classifiers for next generation
closed loop neuromodulation therapy – a case study in epilepsy

Ali Kavoosi${}^{1\,{\dagger}\,\text{\normalsize\Letter}}$, Robert
Toth${}^{2\,{\dagger}}$, Moaad Benjaber1, Mayela Zamora1,
Antonio Valentín3, Andrew Sharott2 and Timothy
Denison${}^{1,2\,\text{\normalsize\Letter}}$ This work was supported by the
John Fell Fund of the University of Oxford, the UK Medical Research Council
(MC_UU_00003/3, MC_UU_00003/6) and the Royal Academy of
Engineering.${\dagger}$ Ali Kavoosi and Robert Toth contributed equally to
this work and share first authorship.1 Brain Network Dynamics Unit, Department
of Pharmacology, University of Oxford, Oxford OX1 3TH, United Kingdom2
Institute of Biomedical Engineering, Old Road Campus Research Building,
Department of Engineering Sciences, University of Oxford, Oxford OX3 7DQ,
United Kingdom3 Department of Basic and Clinical Neuroscience, King’s College
London, London SE5 9RT, United Kingdom${}^{\text{\normalsize\Letter}}$
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This work explores the potential utility of neural network classifiers for
real-time classification of field-potential based biomarkers in next-
generation responsive neuromodulation systems. Compared to classical filter-
based classifiers, neural networks offer an ease of patient-specific parameter
tuning, promising to reduce the burden of programming on clinicians. The paper
explores a compact, feed-forward neural network architecture of only dozens of
units for seizure-state classification in refractory epilepsy. The proposed
classifier offers comparable accuracy to filter-classifiers on clinician-
labeled data, while reducing detection latency. As a trade-off to classical
methods, the paper focuses on keeping the complexity of the architecture
minimal, to accommodate the on-board computational constraints of implantable
pulse generator systems.

Clinical relevance—A neural network-based classifier is presented for
responsive neurostimulation, with comparable accuracy to classical methods at
reduced latency.

##  I INTRODUCTION

Deep brain stimulation (DBS) first received approval for the symptomatic
treatment of Parkinson’s disease in 1997. While similar in design to cardiac
pacemakers, the implantable pulse generators (IPG) of the time offered only an
open-loop form of therapy, with typically a single stimulation pattern, set by
a clinician for each patient. Real-time seizure detection and responsive
neurostimulation (RNS) was first attempted using a computer-in-the-loop system
by Gotman et al. in 1976 [1], it was Osorio et al. in 1998 [2, 3, 4] who
introduced the more widely studied filter-based spectral biomarker detectors
to the field of epilepsy research. With the continued development of IPGs and
the maturation of low-power microprocessor technology, the first RNS system
for epilepsy received approval for pre-clinical use in 2014. This system from
Neuropace had the capabilities to sense bioelectric signals, and choose
stimulation programs based on clinician-configured classification state [5].

Filter-based spectral-feature detectors have since been used successfully in
other conditions, most notably for tremor suppression in Parkinson’s disease,
through the discovery of beta oscillations as a correlate of disease state
[6]. However, the smaller
($1\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{V}\mathrm{r}\mathrm{m}\mathrm{s}$)
signal size of beta oscillations, compared to epileptiform activity
($10\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{V}\mathrm{r}\mathrm{m}\mathrm{s}$),
made deploying the detector algorithm in IPGs challenging due to the presence
of stimulation and other artifacts. Contemporary work focuses on improving the
robustness of the signal chains to enable simultaneous sensing and
stimulation, thus true closed-loop operation across targeted diseases [7, 8].
Examples include the Medtronic Percept [9] and the Picostim-DyNeuMo research
systems [10, 11]. A complementary avenue of refinement is the use of feed-
forward predictors for adapting stimulation based on periodicities of disease
state and patient needs, such the circadian scheduling of the SenTiva system
from LivaNova or the Picostim-DyNeuMo [12]. Taking advantage of more long
term, weekly or even monthly rhythms are being investigated for epilepsy
management [13].

Patient-specific filter design, while possible to aid with software, can be a
complex problem, likely to limit both clinician-capacity and patient-
throughput. Establishing and validating a neural-network (NN) training
pipeline based on clinician-labeled data could offer a systematic classifier
tuning process. Networks could be pre-trained on aggregate data from multiple
patients, and refined based on individual labeled data at the point of
deployment [14]. Of course, due to the black-box nature of neural network
classifiers, extensive validation work will be required to establish safety
before first-in-human studies. Advances in interpretable deep learning could
facilitate building trust in NN–classifiers for medical use [15].

Liu et al. [16] demonstrated the feasibility of deploying high accuracy
classifiers for seizure detection on modern microprocessors (ARM® Cortex-M4),
through model compression and quantization techniques, showcasing several
advanced NN topologies.

This paper is meant as an initial study to bring focus to the fundamental
challenge of NN classifiers: computational cost. As state-of-the-art deep
neural networks reach ever increasing model sizes [14, 17], we aim to explore
whether lean NNs of only dozens of units could in fact compete in accuracy
with classical, filter-based systems for bioelectric signal classification.

Figure 1: Classifier architectures. Top: classical filter-based spectral power
detector [6]. Bottom: the multi-layer perceptron architecture evaluated in
this paper. Through training on labeled data, the neural network is expected
to assume an overall transfer function similar to the hand-crafted filter
topology.

##  II Design

###  II-A Baseline Method

To establish a baseline for performance as well as for computational cost, we
used a classical band-power estimation filter chain to detect epileptiform
discharges [3, 6], that we have previously deployed with success in the
Picostim-DyNeuMo experimental IPG system [11]. The processing steps of this
method are shown in the top panel of Fig. 1. While this algorithm is
computationally efficient and has a very favorable memory footprint (refer to
Table I), the demodulated envelope signal, thus the detector output, will
always lag the input signal to reduce output ripple – irrespective of
processing speed. This trade-off arises from the very nature of causal
filtering, and is necessary to prevent rapid switching of the detector output
for input signals near the classification threshold. The reference classifier
was configured as follows. Our band-pass stage was an $8-22$ Hz, 4th order
Butterworth filter, with a Direct Form I IIR implementation (16-bit
coefficients, 32-bit accumulators). Envelope demodulation was achieved using
an exponential moving average filter with a decay-coefficient of 32 samples.
The filter chain, and all other classifiers were designed to operate at a
sampling rate of 256 Hz.

###  II-B New neural-based methods

In our search for a low-complexity classifier for time series input, we
explored two main NN families. (1) Multi-Layer Perceptrons (MLP) are the
simplest, and oldest family or of artificial neural networks [18], where the
input vector is connected to ‘hidden’ layers of feedforward units, condensing
information into an output unit. This architecture is shown in the bottom
panel of Fig. 1. (2) As a step up in complexity, Convolutional Neural Network
(CNN) introduce a convolution layer, also known as a filter bank, between the
input vector and the neural layers as an initial extra step [14]. The input to
our networks is formed by a windowed set of past time samples of the local
field potential (LFP) signal. The output signal, calculated once for each
complete window of samples, is thresholded into a binary label. We denote this
classifier the ‘standalone MLP’ model.

Recurrent neural networks, an otherwise natural choice for processing time
series data, were dismissed from consideration as recurrence necessitates the
introduction of dynamic state variables, which significantly increases memory
footprint [14]. Without recurrence, we introduced coherence into our
classifier in a different way. We settled on requiring a consensus of three
subsequent outputs from the NN to define our final output label, providing the
‘adjusted MLP’ model.

###  II-C Training and data

Our raw dataset consisted of LFP signals recorded from two patients, for a
combined 24 hours, with 30 professionally labeled events of clinical
significance. The recordings were resampled to a 256 Hz sampling frequency for
uniformity.

As seizures are comparatively rare events scattered among very long periods of
normal activity, we decided to introduce class imbalance into our training
sets to best prepare the NNs for real-life use. The training set was biased
towards negative samples in a 3:1 ratio, based on clinician annotations. The
dataset was split in the common 70:30 ratio between a training and a
validation set. Network weights and biases were quantized to 8-bit integers.

###  II-D Technical equipment used

Neural networks were modeled and trained in Tensorflow Lite version 2.7.0,
using an Intel Core i7 CPU with 16 GB of RAM. Embedded performance was tested
on an Arduino Nano 33 BLE Sense evaluation board for the nRF52840 ARM
Cortex-M4F microprocessor.

Figure 2: ROCs for different classifiers. Note that performance converges
towards high TPR and FPR, which is the desirable operating point of seizure
detectors as FNs pose significantly greater risk of harm to patients than FPs.

###  II-E Comparison of CNN and MLP models

Fig. 2 shows the performance of our CNN and MLP classifiers. While the CNN
outperforms the standalone MLP, it performs with similar accuracy to the
adjusted MLP model above 80% true positive rate (TPR). For safe use, the
operating point of a seizure detection system should be biased towards high
TPR – missed seizures (false negatives) pose significantly more risk to the
patient than false positives, which merely result in unnecessary stimulation.
Overall, in targeting resource constrained IPGs, we judged the minor edge of
the CNN insufficient to justify the added computational burden of the
convolutional layer.

###  II-F Tuning the MLP classifier

The performance of the MLP model, when trained on a given dataset, is
primarily determined by two hyperparameters: the number of timepoints in the
input window, and the hidden layer’s size. We found that varying the number of
hidden layers had very modest effects on accuracy (not shown in this paper).
Fig. 3 systematically explores the effect of the two key hyperparameters on
the classification error of a single output, single hidden layer MLP model. As
expected, the network requires a certain size and complexity to encode a
feature space sufficient for reliable classification, though increasing the
number of units in either layer beyond a certain point leads to diminishing
returns. To select one of the possible models from the error surface, one
could define a scoring scheme including network size, computational time and
the loss itself, to make an educated choice, however, this is beyond the scope
of this paper. Favoring low complexity, we settled on using a 20-point input
window and 8 hidden neurons, in the ‘transition zone’ of the error surface.

Figure 3: Tuning the MLP classifier. Grid search on the two main model
hyperparameters: the input vector length and the number of hidden layer units.
Loss is represented as binary cross entropy across all samples. Figure 4:
Detailed performance of classifiers. Top: histogram of classification latency.
Bottom: histogram showing the percentage of overlap between positive
classifier output and clinician-labeled event.

##  III Model Performance and Interpretation

The next step is to compare our best MLP result to the baseline classifier.
The ROC of Fig. 2 reveals that a well-tuned filter chain outperforms the small
MLP model below 60% false positive rate, beyond which they converge in
accuracy. Identifying a seizure does not present a holistic view of
performance though. In Fig. 4 we highlight two additional characteristics to
consider in classifier evaluation: (1) latency at event onset, and (2) the
overlap between classifier and clinician labels. As shown, the MLP responds on
average more rapidly to a commencing seizure (mean latency of 0.6 sec vs 1.7
sec), and tracks the clinician label more closely overall, compared to the
baseline method.

Figure 5: Frequency response of classifiers to input signals of different
magnitudes. Left: filter chain classifier. Right: MLP classifier with average
of 3 windows. Accuracy is presented as the mean classifier output for a 1
second sinusoidal test tone over 10 repeats.

To explore the MLP classifier’s internal representation of a seizure, we
present a small interpretation experiment in Fig. 5. We presented the
classifiers with second-long sinusoidal bursts of activity, performing
parameter sweeps along both test frequency and test amplitudes. As seen, the
MLP model (right) was successful in internalizing a notion of the spectral
characteristics of epileptiform activity (low frequency lobe), that
encompasses the pass-band of the filter classifier (left). The greater
effective bandwidth of the MLP could explain the lower false positive rates
seen on the ROC of Fig. 2. The activation lobes at higher frequencies likely
represent a process analogous to aliasing, and we expect this periodicity to
be a correlate of the input window size, which should be investigated further.

TABLE I: Resource Usage

Classifier |  Execution time | Code | Memory
---|---|---|---
(Cycles / Sample) | (bytes) | (bytes)
Filter (custom) |  750 | 675 | 100
MLP (TF Lite) |  2250 | 31k | 8k
CNN (TF Lite) |  3200 | 31K | 13k
MLP (custom) |  *2250* |  *2.0k* |  *300*

  * •

*predicted

##  IV DISCUSSION

The example design of the MLP classifier demonstrates that even tiny neural
networks can be effective at simple signal processing tasks. As the final
step, we should reflect on the embedded resource usage achieved, so we refer
the reader to Table I. Importantly, the network achieved sufficiently low
complexity for real time use. Note that NN execution times are reported per
sample, though the output only changes at the end of a window of 20 samples.
Notably, the true memory footprint of the classifier could not be determined
with this evaluation system – Tensorflow Lite does not generate network code,
rather it provides a network description file, to be run by a relatively
large, general purpose interpreter library in the embedded system. For a more
realistic, yet conservative outlook, we present estimates for the memory usage
of the same network deployed using customized library, trimmed down to the
features used in our design. In summary, the MLP could provide an alternative
to existing tuned filter methods used in commercial devices.

## DISCLOSURES

The University of Oxford has research agreements with Bioinduction Ltd. Tim
Denison also has business relationships with Bioinduction for research tool
design and deployment, and stock ownership ($<$1\text{\,}\mathrm{\char
37\relax}$$).

## ACKNOWLEDGMENT

The authors would like to thank Bence Mark Halpern for reviewing the
manuscript, and Tom Gillbe at Bioinduction Ltd. for feedback on the filter-
classifier design.

## References

  * [1] J. Gotman and P. Gloor, “Automatic recognition and quantification of interictal epileptic activity in the human scalp EEG,” _Clin. Neurophysiol_ , vol. 41, no. 5, pp. 513–529, Nov. 1976.
  * [2] I. Osorio, M. G. Frei, and S. B. Wilkinson, “Real-time automated detection and quantitative analysis of seizures and short-term prediction of clinical onset,” _Epilepsia_ , vol. 39, no. 6, pp. 615–627, Jun. 1998.
  * [3] I. Osorio _et al._ , “Performance reassessment of a real‐time seizure‐detection algorithm on long ECoG series,” _Epilepsia_ , vol. 43, no. 12, pp. 1522–1535, Dec. 2002.
  * [4] N. Bhavaraju, M. Frei, and I. Osorio, “Analog seizure detection and performance evaluation,” _IEEE Trans. Biomed. Eng_ , vol. 53, no. 2, pp. 238–245, Feb. 2006.
  * [5] F. T. Sun and M. J. Morrell, “The RNS system: responsive cortical stimulation for the treatment of refractory partial epilepsy,” _Expert Rev. Med. Devices_ , vol. 11, no. 6, pp. 563–572, Nov. 2014.
  * [6] S. Little _et al._ , “Adaptive deep brain stimulation in advanced parkinson disease: Adaptive DBS in PD,” _Ann. Neurol_ , vol. 74, no. 3, pp. 449–457, Sep. 2013.
  * [7] M. M. Sorkhabi, M. Benjaber, P. Brown, and T. Denison, “Physiological artifacts and the implications for brain-machine-interface design,” _In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , pp. 1498–1498, 2020.
  * [8] J. Ansó _et al._ , “Concurrent stimulation and sensing in bi-directional brain interfaces: a multi-site translational experience,” _J. Neural Eng_ , vol. 19, no. 2, p. 026025, Mar. 2022.
  * [9] W.-J. Neumann _et al._ , “The sensitivity of ECG contamination to surgical implantation site in brain computer interfaces,” _Brain Stimul_ , vol. 14, no. 5, pp. 1301–1306, Sep. 2021.
  * [10] M. Zamora _et al._ , “DyNeuMo Mk-1: Design and pilot validation of an investigational motion-adaptive neurostimulator with integrated chronotherapy,” _Exp. Neurol_ , vol. 351, p. 113977, May 2022.
  * [11] R. Toth _et al._ , “DyNeuMo Mk-2: an investigational circadian-locked neuromodulator with responsive stimulation for applied chronobiology,” _In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , pp. 3433–3440, 2020.
  * [12] M. Zamora _et al._ , “Case report: Embedding “digital chronotherapy” into medical devices — a canine validation for controlling status epilepticus through multi-scale rhythmic brain stimulation,” _Front. Neurosci_ , vol. 15, p. 734265, 2021.
  * [13] M. O. Baud _et al._ , “Multi-day rhythms modulate seizure risk in epilepsy,” _Nature Commun_ , vol. 9, no. 1, p. 88, Jan. 2018.
  * [14] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, May 2015.
  * [15] S. Lapuschkin _et al._ , “Unmasking Clever Hans predictors and assessing what machines really learn,” _Nat. Commun_ , vol. 10, no. 1, p. 1096, Dec. 2019.
  * [16] X. Liu, and A. G. Richardson, “Edge deep learning for neural implants: a case study of seizure detection and prediction,” _J. Neural Eng_ , vol. 18, no. 4, p. 046034, Apr. 2021.
  * [17] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” _In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT)_ , pp. 610–623, Mar. 2021.
  * [18] F. Rosenblatt, “The Perceptron: A probabilistic model for information storage and organization in the brain,” _Psychol. Rev_ , vol. 65, no. 6, pp. 386–408, 1958.