# Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural
Language Processing

Minh Van Nguyen, Viet Lai, Amir Pouran Ben Veyseh, Thien Huu Nguyen
Department of Computer and Information Science
University of Oregon, Eugene, Oregon, USA
<EMAIL_ADDRESS>
###### Abstract

We introduce Trankit, a light-weight Transformer-based Toolkit for
multilingual Natural Language Processing (NLP). It provides a trainable
pipeline for fundamental NLP tasks over 100 languages, and 90 pretrained
pipelines for 56 languages. Built on a state-of-the-art pretrained language
model, Trankit significantly outperforms prior multilingual NLP pipelines over
sentence segmentation, part-of-speech tagging, morphological feature tagging,
and dependency parsing while maintaining competitive performance for
tokenization, multi-word token expansion, and lemmatization over 90 Universal
Dependencies treebanks. Despite the use of a large pretrained transformer, our
toolkit is still efficient in memory usage and speed. This is achieved by our
novel plug-and-play mechanism with Adapters where a multilingual pretrained
transformer is shared across pipelines for different languages. Our toolkit
along with pretrained models and code are publicly available at:
https://github.com/nlp-uoregon/trankit. A demo website for our toolkit is also
available at: http://nlp.uoregon.edu/trankit. Finally, we create a demo video
for Trankit at: https://youtu.be/q0KGP3zGjGc.

##  1 Introduction

Many efforts have been devoted to developing multilingual NLP systems to
overcome language barriers (Aharoni et al., 2019; Liu et al., 2019a;
Taghizadeh and Faili, 2020; Zhu, 2020; Kanayama and Iwamoto, 2020; Nguyen and
Nguyen, 2021). A large portion of existing multilingual systems has focused on
downstream NLP tasks that critically depend on upstream linguistic features,
ranging from basic information such as token and sentence boundaries for raw
text to more sophisticated structures such as part-of-speech tags,
morphological features, and dependency trees of sentences (called fundamental
NLP tasks). As such, building effective multilingual systems/pipelines for
fundamental upstream NLP tasks to produce such information has the potentials
to transform multilingual downstream systems.

There have been several NLP toolkits that concerns multilingualism for
fundamental NLP tasks, featuring spaCy111https://spacy.io/, UDify (Kondratyuk
and Straka, 2019), Flair (Akbik et al., 2019), CoreNLP (Manning et al., 2014),
UDPipe (Straka, 2018), and Stanza (Qi et al., 2020). However, these toolkits
have their own limitations. spaCy is designed to focus on speed, thus it needs
to sacrifice the performance. UDify and Flair cannot process raw text as they
depend on external tokenizers. CoreNLP supports raw text, but it does not
offer state-of-the-art performance. UDPipe and Stanza are the recent toolkits
that leverage word embeddings, i.e., word2vec (Mikolov et al., 2013) and
fastText (Bojanowski et al., 2017), to deliver current state-of-the-art
performance for many languages. However, Stanza and UDPipe’s pipelines for
different languages are trained separately and do not share any component,
especially the embedding layers that account for most of the model size. This
makes their memory usage grow aggressively as pipelines for more languages are
simultaneously needed and loaded into the memory (e.g., for language learning
apps). Most importantly, none of such toolkits have explored contextualized
embeddings from pretrained transformer-based language models that have the
potentials to significantly improve the performance of the NLP tasks, as
demonstrated in many prior works (Devlin et al., 2019; Liu et al., 2019b;
Conneau et al., 2020).

Figure 1: Overall architecture of Trankit. A single multilingual pretrained
transformer is shared across three components (pointed by the red arrows) of
the pipeline for different languages.

In this paper, we introduce Trankit, a multilingual Transformer-based NLP
Toolkit that overcomes such limitations. Our toolkit can process raw text for
fundamental NLP tasks, supporting $56$ languages with $90$ pre-trained
pipelines on $90$ treebanks of the Universal Dependency v2.5 (Zeman et al.,
2019). By utilizing the state-of-the-art multilingual pretrained transformer
XLM-Roberta (Conneau et al., 2020), Trankit advances state-of-the-art
performance for sentence segmentation, part-of-speech (POS) tagging,
morphological feature tagging, and dependency parsing while achieving
competitive or better performance for tokenization, multi-word token
expansion, and lemmatization over the 90 treebanks. It also obtains
competitive or better performance for named entity recognition (NER) on 11
public datasets.

Unlike previous work, our token and sentence splitter is wordpiece-based
instead of character-based to better exploit contextual information, which are
beneficial in many languages. Considering the following sentence:

> “John Donovan from Argghhh! has put out a excellent slide show on what was
> actually found and fought for in Fallujah.”

As such, Trankit correctly recognizes this as a single sentence while
character-based sentence splitters of Stanza and UDPipe are easily fooled by
the exclamation mark “!”, treating it as two separate sentences. To our
knowledge, this is the first work to successfully build a wordpiece-based
token and sentence splitter that works well for 56 languages.

Figure 1 presents the overall architecture of Trankit pipeline that features
three novel transformer-based components for: (i) the joint token and sentence
splitter, (ii) the joint model for POS tagging, morphological tagging,
dependency parsing, and (iii) the named entity recognizer. One potential
concern for our use of a large pretrained transformer model (i.e., XML-
Roberta) in Trankit involves GPU memory where different transformer-based
components in the pipeline for one or multiple languages must be
simultaneously loaded into the memory to serve multilingual tasks. This could
extensively consume the memory if different versions of the large pre-trained
transformer (finetuned for each component) are employed in the pipeline. As
such, we introduce a novel plug-and-play mechanism with Adapters to address
this memory issue. Adapters are small networks injected inside all layers of
the pretrained transformer model that have shown their effectiveness as a
light-weight alternative for the traditional finetuning of pretrained
transformers (Houlsby et al., 2019; Peters et al., 2019; Pfeiffer et al.,
2020a, b). In Trankit, a set of adapters (for transfomer layers) and task-
specific weights (for final predictions) are created for each transformer-
based component for each language while only one single large multilingual
pretrained transformer is shared across components and languages. Adapters
allow us to learn language-specific features for tasks. During training, the
shared pretrained transformer is fixed while only the adapters and task-
specific weights are updated. At inference time, depending on the language of
the input text and the current active component, the corresponding trained
adapter and task-specific weights are activated and plugged into the pipeline
to process the input. This mechanism not only solves the memory problem but
also substantially reduces the training time.

##  2 Related Work

There have been works using pre-trained transformers to build models for
character-based word segmentation for Chinese (Yang, 2019; Tian et al., 2020;
Che et al., 2020); POS tagging for Dutch, English, Chinese, and Vietnamese (de
Vries et al., 2019; Tenney et al., 2019; Tian et al., 2020; Che et al., 2020;
Nguyen and Nguyen, 2020); morphological feature tagging for Estonian and
Persian (Kittask et al., 2020; Mohseni and Tebbifakhr, 2019); and dependency
parsing for English and Chinese (Tenney et al., 2019; Che et al., 2020).
However, all of these works are only developed for some specific language,
thus potentially unable to support and scale to the multilingual setting.

Some works have designed multilingual transformer-based systems via
multilingual training on the combined data of different languages (Tsai et
al., 2019; Kondratyuk and Straka, 2019; Üstün et al., 2020). However,
multilingual training is suboptimal (see Section 5). Also, these systems still
rely on external resources to perform tokenization and sentence segmentation,
thus unable to consume raw text. To our knowedge, this is the first work to
successfully build a multilingual transformer-based NLP toolkit where
different transformer-based models for many languages can be simultaneously
loaded into GPU memory and process raw text inputs of different languages.

##  3 Design and Architecture

Figure 2: Left: location of an adapter (green box) inside a layer of the
pretrained transformer. Gray boxes represent the original components of a
transformer layer. Right: the network architecture of an adapter.

Adapters. Adapters play a critical role in making Trankit memory- and time-
efficient for training and inference. Figure 2 shows the architecture and the
location of an adapter inside a layer of transformer. We use the adapter
architecture proposed by (Pfeiffer et al., 2020a, b), which consists of two
projection layers Up and Down (feed-forward networks), and a residual
connection.

$c_{i}=\textrm{AddNorm}(r_{i}),h_{i}=\textrm{Up}(\textrm{ReLU}(\textrm{Down}(c_{i})))+r_{i}$
(1)

where $r_{i}$ is the input vector from the transformer layer for the adapter
and $h_{i}$ is the output vector for the transformer layer $i$. During
training, all the weights of the pretrained transformer (i.e., gray boxes) are
fixed and only the adapter weights of two projection layers and the task-
specific weights outside the transformer (for final predictions) are updated.
As demonstrated in Figure 1, Trankit involves six components described as
follows.

Multilingual Encoder with Adapters. This is our core component that is shared
across different transformer-based components for different languages of the
system. Given an input raw text $s$, we first split it into substrings by
spaces. Afterward, Sentence Piece, a multilingual subword tokenizer (Kudo and
Richardson, 2018; Kudo, 2018), is used to further split each substring into
wordpieces. By concatenating wordpiece sequences for substrings, we obtain an
overall sequence of wordpieces $\textbf{w}=[w_{1},w_{2},\ldots,w_{K}]$ for
$s$. In the next step, w is fed into the pretrained transformer, which is
already integrated with adapters, to obtain the wordpiece representations:

$x^{l,m}_{1:K}=\textrm{Transformer}(w_{1:K};\theta_{AD}^{l,m})$ (2)

Here, $\theta_{AD}^{l,m}$ represents the adapter weights for language $l$ and
component $m$ of the system. As such, we have specific adapters in all
transformer layers for each component $m$ and language $l$. Note that if $K$
is larger than the maximum input length of the pretrained transformer (i.e.,
512), we further divide w into consecutive chunks; each has the length less
than or equal to the maximum length. The pretrained transformer is then
applied over each chunk to obtain a representation vector for each wordpiece
in w. Finally, $x^{l,m}_{1:K}$ will be sent to component $m$ to perform the
corresponding task.

Joint Token and Sentence Splitter. Given the wordpiece representations
$x^{l,m}_{1:K}$ for this component, each vector $x^{l,m}_{i}$ for
$w_{i}\in\textbf{w}$ will be consumed by a feed-forward network with softmax
in the end to predict if $w_{i}$ is the end of a single-word token, the end of
a multi-word token, or the end of a sentence. The predictions for all
wordpieces in w will then be aggregated to determine token, multi-word token,
and sentence boundaries for $s$.

Multi-word Token Expander. This component is responsible for expanding each
detected multi-word token (MWT) into multiple syntactic words222For languages
(e.g., English, Chinese) that do not require MWT expansion, tokens and words
are the same concepts.. We follow Stanza to deploy a character-based seq2seq
model for this component. This decision is made based on our observation that
the task is done best at character level, and the character-based model (with
character embeddings) is very small.

Treebank | System | Tokens | Sents. | Words | UPOS | XPOS | UFeats | Lemmas | UAS | LAS
---|---|---|---|---|---|---|---|---|---|---
Overall (90 treebanks) | Trankit | 99.23 | 91.82 | 99.02 | 95.65 | 94.05 | 93.21 | 94.27 | 87.06 | 83.69
Stanza | 99.26 | 88.58 | 98.90 | 94.21 | 92.50 | 91.75 | 94.15 | 83.06 | 78.68
Arabic-PADT | Trankit | 99.93 | 96.59 | 99.22 | 96.31 | 94.08 | 94.28 | 94.65 | 88.39 | 84.68
Stanza | 99.98 | 80.43 | 97.88 | 94.89 | 91.75 | 91.86 | 93.27 | 83.27 | 79.33
UDPipe | 99.98 | 82.09 | 94.58 | 90.36 | 84.00 | 84.16 | 88.46 | 72.67 | 68.14
Chinese-GSD | Trankit | 97.01 | 99.7 | 97.01 | 94.21 | 94.02 | 96.59 | 97.01 | 85.19 | 82.54
Stanza | 92.83 | 98.80 | 92.83 | 89.12 | 88.93 | 92.11 | 92.83 | 72.88 | 69.82
UDPipe | 90.27 | 99.10 | 90.27 | 84.13 | 84.04 | 89.05 | 90.26 | 61.60 | 57.81
English-EWT | Trankit | 98.48 | 88.35 | 98.48 | 95.95 | 95.71 | 96.26 | 96.84 | 90.14 | 87.96
Stanza | 99.01 | 81.13 | 99.01 | 95.40 | 95.12 | 96.11 | 97.21 | 86.22 | 83.59
UDPipe | 98.90 | 77.40 | 98.90 | 93.26 | 92.75 | 94.23 | 95.45 | 80.22 | 77.03
spaCy | 97.44 | 63.16 | 97.44 | 86.99 | 91.05 | - | 87.16 | 55.38 | 37.03
French-GSD | Trankit | 99.7 | 96.63 | 99.66 | 97.85 | - | 97.16 | 97.80 | 94.00 | 92.34
Stanza | 99.68 | 94.92 | 99.48 | 97.30 | - | 96.72 | 97.64 | 91.38 | 89.05
UDPipe | 99.68 | 93.59 | 98.81 | 95.85 | - | 95.55 | 96.61 | 87.14 | 84.26
spaCy | 99.02 | 89.73 | 94.81 | 89.67 | - | - | 88.55 | 75.22 | 66.93
Spanish-Ancora | Trankit | 99.94 | 99.13 | 99.93 | 99.02 | 98.94 | 98.8 | 99.17 | 94.11 | 92.41
Stanza | 99.98 | 99.07 | 99.98 | 98.78 | 98.67 | 98.59 | 99.19 | 92.21 | 90.01
UDPipe | 99.97 | 98.32 | 99.95 | 98.32 | 98.13 | 98.13 | 98.48 | 88.22 | 85.10
spaCy | 99.95 | 97.54 | 99.43 | 93.43 | - | - | 80.02 | 89.35 | 83.81

Table 1: Systems’ performance on test sets of the Universal Dependencies v2.5
treebanks. Performance for Stanza, UDPipe, and spaCy is obtained using their
public pretrained models. The overall performance for Trankit and Stanza is
computed as the macro-averaged F1 over 90 treebanks. Detailed performance of
Trankit for 90 supported treebanks can be found at our documentation page.

Joint Model for POS Tagging, Morphological Tagging and Dependency Parsing. In
Trankit, given the detected sentences and tokens/words, we use a single model
to jointly perform POS tagging, morphological feature tagging and dependency
parsing at sentence level. Joint modeling mitigates error propagation, saves
the memory, and speedups the system. In particular, given a sentence, the
representation for each word is computed as the average of its wordpieces’
transformer-based representations in $x^{l,m}_{1:K}$. Let
$t_{1:N}=[t_{1},t_{2},\ldots,t_{N}]$ be the representations of the words in
the sentence. We compute the following vectors using feed-forward networks
$\textrm{FFN}_{*}$:

$\begin{split}r^{upos}_{1:N}&=\textrm{FFN}_{upos}(t_{1:N}),r^{xpos}_{1:N}=\textrm{FFN}_{xpos}(t_{1:N})\\\
r^{ufeats}_{1:N}&=\textrm{FFN}_{ufeats}(t_{1:N}),r^{dep}_{0:N}=[x_{cls};\textrm{FFN}_{dep}(t_{1:N})]\end{split}$

Vectors for the words in $r^{upos}_{1:N}$, $r^{xpos}_{1:N}$,
$r^{ufeats}_{1:N}$ are then passed to a softmax layer to make predictions for
UPOS, XPOS, and UFeats tags for each word. For dependency parsing, we use the
classification token <s> to represent the root node, and apply Deep Biaffine
Attention (Dozat and Manning, 2017) and the Chu-Liu/Edmonds algorithm (Chu,
1965; Edmonds, 1967) to assign a syntactic head and the associated dependency
relation to each word in the sentence.

Lemmatizer. This component receives sentences and their predicted UPOS tags to
produce the canonical form for each word. We also employ a character-based
seq2seq model for this component as in Stanza.

Named Entity Recognizer. Given a sentence, the named entity recognizer
determines spans of entity names by assigning a BIOES tag to each token in the
sentence. We deploy a standard sequence labeling architecture using
transformer-based representations for tokens, involving a feed-forward network
followed by a Conditional Random Field.

##  4 Usage

Detailed documentation for Trankit can be found at:
https://trankit.readthedocs.io.

Trankit Installation. Trankit is written in Python and available on PyPI:
https://pypi.org/project/trankit/. Users can install our toolkit via pip
using:

> pip install trankit

Initialize a Pipeline. Lines 1-4 in Figure 3 shows how to initialize a
pretrained pipeline for English; it is instructed to run on GPU and store
downloaded pretrained models to the specified cache directory. Trankit will
not download pretrained models if they already exist.

Multilingual Usage. Figure 3 shows how to initialize a multilingual pipeline
and process inputs of different languages in Trankit:

Figure 3: Multilingual pipeline initialization.

Basic Functions. Trankit can process inputs which are untokenized (raw) or
pretokenized strings, at both sentence and document levels. Figure 4
illustrates a simple code to perform all the supported tasks for an input
text. We organize Trankit’s outputs into hierarchical native Python
dictionaries, which can be easily inspected by users. Figure 5 demonstrates
the outputs of the command line $6$ in Figure 4.

Figure 4: A function performing all tasks on the input. Figure 5: Output from
Trankit. Some parts are collapsed to improve visualization.

Training your own Pipelines. Trankit also provides a trainable pipeline for
100 languages via the class TPipeline. This ability is inherited from the XLM-
Roberta encoder which is pretrained on those languages. Figure 6 illustrates
how to train a token and sentence splitter with TPipeline.

Figure 6: Training a token and sentence splitter using the CONLL-U formatted
data Nivre et al. (2020).

Demo Website. A demo website for Trankit to support 90 pretrained pipelines is
hosted at: http://nlp.uoregon.edu/trankit. Figure 7 shows its interface.

Figure 7: Demo website for Trankit.

##  5 System Evaluation

###  5.1 Datasets & Hyper-parameters

To achieve a fair comparison, we follow Stanza Qi et al. (2020) to train and
evaluate all the models on the same canonical data splits of 90 Universal
Dependencies treebanks v2.5 (UD2.5)333We skip 10 treebanks whose languages are
not supported by XLM-Roberta. (Zeman et al., 2019), and 11 public NER datasets
provided in the following corpora: AQMAR (Mohit et al., 2012), CoNLL02 (Tjong
Kim Sang, 2002), CoNLL03 (Tjong Kim Sang and De Meulder, 2003), GermEval14
(Benikova et al., 2014), OntoNotes (Weischedel et al., 2013), and WikiNER
(Nothman et al., 2012). Hyper-parameters for all models and datasets are
selected based on the development data in this work.

System | Tokens | Sents. | Words | UPOS | XPOS | UFeats | Lemmas | UAS | LAS
---|---|---|---|---|---|---|---|---|---
Trankit (plug-and-play with adapters) | 99.05 | 95.12 | 98.96 | 95.43 | 89.02 | 92.69 | 93.46 | 86.20 | 82.51
Multilingual | 96.69 | 88.95 | 96.35 | 91.19 | 84.64 | 88.10 | 90.02 | 72.96 | 68.66
No-adapters | 95.06 | 89.57 | 94.08 | 88.79 | 82.54 | 83.76 | 88.33 | 66.63 | 63.11

Table 2: Model performance on 9 different treebanks (macro-averaged F1 score
over test sets).

###  5.2 Universal Dependencies performance

Table 1 compares the performance of Trankit and the latest available versions
of other popular toolkits, including Stanza (v1.1.1) with current state-of-
the-art performance, UDPipe (v1.2), and spaCy (v2.3) on the UD2.5 test sets.
The performance for all systems is obtained using the official scorer of the
CoNLL 2018 Shared
Task444https://universaldependencies.org/conll18/evaluation.html. On five
illustrated languages, Trankit achieves competitive performance on
tokenization, MWT expansion, and lemmatization. Importantly, Trankit
outperforms other toolkits over all remaining tasks (e.g., POS and
morphological tagging) in which the improvement boost is substantial and
significant for sentence segmentation and dependency parsing. For example,
English enjoys a $7.22\%$ improvement for sentence segmentation, a $3.92\%$
and $4.37\%$ improvement for UAS and LAS in dependency parsing. For Arabic,
Trankit has a remarkable improvement of $16.16\%$ for sentence segmentation
while Chinese observes $12.31\%$ and $12.72\%$ improvement of UAS and LAS for
dependency parsing.

Over all 90 treebanks, Trankit outperforms the previous state-of-the-art
framework Stanza in most of the tasks, particularly for sentence segmentation
($+3.24\%$), POS tagging ($+1.44\%$ for UPOS and $+1.55\%$ for XPOS),
morphological tagging ($+1.46\%$), and dependency parsing ($+4.0\%$ for UAS
and $+5.01\%$ for LAS) while maintaining the competitive performance on
tokenization, multi-word expansion, and lemmatization.

###  5.3 NER results

Table 3 compares Trankit with Stanza (v1.1.1), Flair (v0.7), and spaCy (v2.3)
on the test sets of 11 considered NER datasets. Following Stanza, we report
the performance for other toolkits with their pretrained models on the
canonical data splits if they are available. Otherwise, their best
configurations are used to train the models on the same data splits (inherited
from Stanza). Also, for the Dutch datasets, we retrain the models in Flair as
those models (for Dutch) have been updated in version v0.7. As can be seen,
Trankit obtains competitive or better performance for most of the languages,
clearly demonstrating the benefit of using the pretrained transformer for
multilingual NER.

Language | Corpus | Trankit | Stanza | Flair | spaCy
---|---|---|---|---|---
Arabic | AQMAR | 74.8 | 74.3 | 74.0 | -
Chinese | OntoNotes | 80.0 | 79.2 | - | 69.3
Dutch | CoNLL02 | 91.8 | 89.2 | 91.3 | 73.8
WikiNER | 94.8 | 94.8 | 94.8 | 90.9
English | CoNLL03 | 92.1 | 92.1 | 92.7 | 81.0
OntoNotes | 89.6 | 88.8 | 89.0 | 85.4
French | WikiNER | 92.3 | 92.9 | 92.5 | 88.8
German | CoNLL03 | 84.6 | 81.9 | 82.5 | 63.9
GermEval14 | 86.9 | 85.2 | 85.4 | 68.4
Russian | WikiNER | 92.8 | 92.9 | - | -
Spanish | CoNLL02 | 88.9 | 88.1 | 87.3 | 77.5

Table 3: Performance (F1) on NER test sets.

System | GPU | CPU
---|---|---
UD | NER | UD | NER
Trankit | $4.50\times$ | $1.36\times$ | $19.8\times$ | $31.5\times$
Stanza | $3.22\times$ | $1.08\times$ | $10.3\times$ | $17.7\times$
UDPipe | - | - | $4.30\times$ | -
Flair | - | $1.17\times$ | - | $51.8\times$

Table 4: Run time on processing the English EWT treebank and the English Ontonotes NER dataset. Measurements are done on an NVIDIA Titan RTX card. Model Package | Trankit | Stanza
---|---|---
Multilingual Transformer | 1146.9MB | -
Arabic | 38.6MB | 393.9MB
Chinese | 40.6MB | 225.2MB
English | 47.9MB | 383.5MB
French | 39.6MB | 561.9MB
Spanish | 37.3MB | 556.1MB
Total size | 1350.9MB | 2120.6MB
Table 5: Model sizes for five languages.

###  5.4 Speed and Memory Usage

Table 4 reports the relative processing time for UD and NER of the toolkits
compared to spaCy’s CPU processing time555spaCy can process 8140 tokens and
5912 tokens per second for UD and NER, respectively.. For memory usage
comparison, we show the model sizes of Trankit and Stanza for several
languages in Table 5. As can be seen, besides the multilingual transformer,
model packages in Trankit only take dozens of megabytes while Stanza consumes
hundreds of megabytes for each package. This leads to the Stanza’s usage of
much more memory when the pipelines for these languages are loaded at the same
time. In fact, Trankit only takes 4.9GB to load all the 90 pretrained
pipelines for the 56 supported languages.

###  5.5 Ablation Study

This section compares Trankit with two other possible strategies to build a
multilingual system for fundamental NLP tasks. In the first strategy (called
“Multilingual”), we train a single pipeline where all the components in the
pipeline are trained with the combined training data of all the languages. The
second strategy (called “No-adapters”) involves eliminating adapters from XLM-
Roberta in Trankit. As such, in “No-adapters”, pipelines are still trained
separately for each language; the pretrained transformer is fixed; and only
task-specific weights (for predictions) in components are updated during
training.

For evaluation, we select 9 treebanks for 3 different groups, i.e., high-
resource, medium-resource, and low-resource, depending on the sizes of the
treebanks. In particular, the high-resource group includes Czech, Russian, and
Arabic; the medium-resource group includes French, English, and Chinese; and
the low-resource group involves Belarusian, Telugu, and Lithuanian. Table 2
compares the average performance of Trankit, “Multilingual”, and “No-
adapters”. As can be seen, “Multilingual” and “No-adapters” are significantly
worse than the proposed adapter-based Trankit. We attribute this to the fact
that multilingual training might suffer from unbalanced sizes of treebanks,
causing high-resource languages to dominate others and impairing the overall
performance. For “No-adapters”, fixing pretrained transformer might
significantly limit the models’ capacity for multiple tasks and languages.

##  6 Conclusion and Future Work

We introduce Trankit, a transformer-based multilingual toolkit that
significantly improves the performance for fundamental NLP tasks, including
sentence segmentation, part-of-speech, morphological tagging, and dependency
parsing over 90 Universal Dependencies v2.5 treebanks of 56 different
languages. Our toolkit is fast on GPUs and efficient in memory use, making it
usable for general users. In the future, we plan to improve our toolkit by
investigating different pretrained transformers such as mBERT and
$\textrm{XLM-Roberta}_{large}$. We also plan to provide Named Entity
Recognizers for more languages and add modules to perform more NLP tasks.

## Acknowledgments

This research has been supported by the Office of the Director of National
Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA),
via IARPA Contract No. 2019-19051600006 under the Better Extraction from Text
Towards Enhanced Retrieval (BETTER) Program. The views and conclusions
contained herein are those of the authors and should not be interpreted as
necessarily representing the official policies, either expressed or implied,
of ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for governmental
purposes notwithstanding any copyright annotation therein. This document does
not contain technology or technical data controlled under either the U.S.
International Traffic in Arms Regulations or the U.S. Export Administration
Regulations.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Akbik et al. (2019) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019.  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 54–59, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Benikova et al. (2014) Darina Benikova, Chris Biemann, and Marc Reznicek. 2014.  NoSta-d named entity annotation for German: Guidelines and dataset.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)_ , pages 2524–2531, Reykjavik, Iceland. European Languages Resources Association (ELRA).
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Che et al. (2020) Wanxiang Che, Yunlong Feng, Libo Qin, and Ting Liu. 2020.  N-ltp: A open-source neural chinese language technology platform with pretrained models.  _arXiv preprint arXiv:2009.11616_.
  * Chu (1965) Yoeng-Jin Chu. 1965.  On the shortest arborescence of a directed graph.  _Scientia Sinica_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D Manning. 2017.  Deep biaffine attention for neural dependency parsing.  In _Proceedings of the International Conference on Learning Representations_.
  * Edmonds (1967) Jack Edmonds. 1967.  Optimum branchings.  _Journal of Research of the national Bureau of Standards B_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _Proceedings of the International Conference on Machine Learning_.
  * Kanayama and Iwamoto (2020) Hiroshi Kanayama and Ran Iwamoto. 2020.  How universal are Universal Dependencies? exploiting syntax for multilingual clause-level sentiment detection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4063–4073, Marseille, France. European Language Resources Association.
  * Kittask et al. (2020) Claudia Kittask, Kirill Milintsevich, and Kairit Sirts. 2020.  Evaluating multilingual bert for estonian.  _Volume_ , 328:19–26.
  * Kondratyuk and Straka (2019) Dan Kondratyuk and Milan Straka. 2019.  75 languages, 1 model: Parsing Universal Dependencies universally.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2779–2795, Hong Kong, China. Association for Computational Linguistics.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Liu et al. (2019a) Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2019a.  Neural cross-lingual event detection with minimal parallel resources.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 738–748, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 55–60, Baltimore, Maryland. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the Conference on Neural Information Processing Systems_.
  * Mohit et al. (2012) Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012.  Recall-oriented learning of named entities in Arabic Wikipedia.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 162–173, Avignon, France. Association for Computational Linguistics.
  * Mohseni and Tebbifakhr (2019) Mahdi Mohseni and Amirhossein Tebbifakhr. 2019.  MorphoBERT: a Persian NER system with BERT and morphological analysis.  In _Proceedings of The First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers_ , pages 23–30, Trento, Italy. Association for Computational Linguistics.
  * Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  PhoBERT: Pre-trained language models for Vietnamese.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1037–1042, Online. Association for Computational Linguistics.
  * Nguyen and Nguyen (2021) Minh Van Nguyen and Thien Huu Nguyen. 2021.  Improving cross-lingual transfer for event argument extraction with language-universal sentence structures.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP) at EACL 2021_.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Nothman et al. (2012) Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2012\.  Learning multilingual named entity recognition from Wikipedia.  _Artificial Intelligence_ , 194:151–175.
  * Peters et al. (2019) Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019.  To tune or not to tune? adapting pretrained representations to diverse tasks.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 7–14, Florence, Italy. Association for Computational Linguistics.
  * Pfeiffer et al. (2020a) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a.  AdapterHub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2020b) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Straka (2018) Milan Straka. 2018.  UDPipe 2.0 prototype at CoNLL 2018 UD shared task.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 197–207, Brussels, Belgium. Association for Computational Linguistics.
  * Taghizadeh and Faili (2020) Nasrin Taghizadeh and Heshaam Faili. 2020.  Cross-lingual adaptation using universal dependencies.  _arXiv preprint arXiv:2003.10816_.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Tian et al. (2020) Yuanhe Tian, Yan Song, Xiang Ao, Fei Xia, Xiaojun Quan, Tong Zhang, and Yonggang Wang. 2020.  Joint Chinese word segmentation and part-of-speech tagging via two-way attentions of auto-analyzed knowledge.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8286–8296, Online. Association for Computational Linguistics.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Tsai et al. (2019) Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019.  Small and practical BERT models for sequence labeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3632–3636, Hong Kong, China. Association for Computational Linguistics.
  * Üstün et al. (2020) Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020.  UDapter: Language adaptation for truly Universal Dependency parsing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2302–2315, Online. Association for Computational Linguistics.
  * de Vries et al. (2019) Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019.  Bertje: A dutch bert model.  _arXiv preprint arXiv:1912.09582_.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013.  Ontonotes release 5.0.  _Linguistic Data Consortium_.
  * Yang (2019) Haiqin Yang. 2019.  Bert meets chinese word segmentation.  _arXiv preprint arXiv:1909.09292_.
  * Zeman et al. (2019) Daniel Zeman, Joakim Nivre, Mitchell Abrams, Noëmi Aepli, Željko Agić, Lars Ahrenberg, Gabrielė Aleksandravičiūtė, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Victoria Basmov, Colin Batchelor, John Bauer, Sandra Bellato, Kepa Bengoetxea, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agnė Bielinskienė, Rogier Blokland, Victoria Bobicev, Loïc Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Kristina Brokaitė, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Tatiana Cavalcanti, Gülşen Cebiroğlu Eryiğit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Jayeol Chun, Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Çağrı Çöltekin, Miriam Connor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Elvis de Souza, Arantza Diaz de Ilarraza, Carly Dickerson, Bamba Dione, Peter Dirix, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Hanne Eckhoff, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomaž Erjavec, Aline Etienne, Wograine Evelyn, Richárd Farkas, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarína Gajdošová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Kim Gerdes, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta González Saavedra, Bernadeta Griciūtė, Matias Grioni, Normunds Grūzītis, Bruno Guillaume, Céline Guillot-Barbance, Nizar Habash, Jan Hajič, Jan Hajič jr., Mika Hämäläinen, Linh Hà Mỹ, Na-Rae Han, Kim Harris, Dag Haug, Johannes Heinecke, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Jena Hwang, Takumi Ikeda, Radu Ion, Elena Irimia, Ọlájídé Ishola, Tomáš Jelínek, Anders Johannsen, Fredrik Jørgensen, Markus Juutinen, Hüner Kaşıkara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Elena Klementieva, Arne Köhn, Kamil Kopacewicz, Natalia Kotsyba, Jolanta Kovalevskaitė, Simon Krek, Sookyoung Kwak, Veronika Laippala, Lorenzo Lambertino, Lucia Lam, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phương Lê Hồng, Alessandro Lenci, Saran Lertpradit, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, KyungTae Lim, Maria Liovina, Yuan Li, Nikola Ljubešić, Olga Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Cătălina Mărănduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça, Niko Miekka, Margarita Misirpashayeva, Anna Missilä, Cătălin Mititelu, Maria Mitrofan, Yusuke Miyao, Simonetta Montemagni, Amir More, Laura Moreno Romero, Keiko Sophie Mori, Tomohiko Morioka, Shinsuke Mori, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Juan Ignacio Navarro Horñiacek, Anna Nedoluzhko, Gunta Nešpore-Bērzkalne, Lương Nguyễn Thị, Huyền Nguyễn Thị Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adédayọ Olúòkun, Mai Omura, Petya Osenova, Robert Östling, Lilja Øvrelid, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino-Passos, Angelika Peljak-Łapińska, Siyao Peng, Cenel-Augusto Perez, Guy Perrier, Daria Petrova, Slav Petrov, Jason Phelan, Jussi Piitulainen, Tommi A Pirinen, Emily Pitler, Barbara Plank, Thierry Poibeau, Larisa Ponomareva, Martin Popel, Lauma Pretkalniņa, Sophie Prévost, Prokopis Prokopidis, Adam Przepiórkowski, Tiina Puolakainen, Sampo Pyysalo, Peng Qi, Andriela Rääbis, Alexandre Rademaker, Loganathan Ramasamy, Taraka Rama, Carlos Ramisch, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Ivan Riabov, Michael Rießler, Erika Rimkutė, Larissa Rinaldi, Laura Rituma, Luisa Rocha, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Valentin Rosca, Olga Rudina, Jack Rueter, Shoval Sadde, Benoît Sagot, Shadi Saleh, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Dage Särg, Baiba Saulīte, Yanin Sawanakunanon, Nathan Schneider, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Muh Shohibussirri, Dmitry Sichinava, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Isabela Soares-Bastos, Carolyn Spadine, Antonio Stella, Milan Straka, Jana Strnadová, Alane Suhr, Umut Sulubacak, Shingo Suzuki, Zsolt Szántó, Dima Taji, Yuta Takahashi, Fabio Tamburini, Takaaki Tanaka, Isabelle Tellier, Guillaume Thomas, Liisi Torga, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Zdeňka Urešová, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Sowmya Vajjala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Seyi Williams, Mats Wirén, Christian Wittern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan, Zhuoran Yu, Zdeněk Žabokrtský, Amir Zeldes, Manying Zhang, and Hanzhi Zhu. 2019.  Universal dependencies 2.5.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Zhu (2020) Xingran Zhu. 2020.  Cross-lingual word sense disambiguation using mbert embeddings with syntactic dependencies.  _arXiv preprint arXiv:2012.05300_.


#  Metrics Dashboard: A Hosted Platform for Software Quality Metrics

George K. Thiruvathukal1, Shilpika, Nicholas J. Hayward, and Konstantin Läufer
George K. Thiruvathukal is the corresponding author. Please direct all
feedback to<EMAIL_ADDRESS> Department of Computer Science
Loyola University Chicago
Chicago, Illinois 60611-2196, USA
<EMAIL_ADDRESS>
1and Argonne National Laboratory, Lemont, Illinois 60439, USA

###### Abstract

There is an emerging consensus in the scientific software community that
progress in scientific research is dependent on the “quality and accessibility
of software at all levels” [1]. This progress depends on embracing the best
traditional—and emergent—practices in software engineering, especially agile
practices that intersect with the more formal tradition of software
engineering. As a first step in our larger exploratory project to study in-
process quality metrics for software development projects in Computational
Science and Engineering (CSE), we have developed the _Metrics Dashboard_ , a
platform for producing and observing metrics by mining open-source software
repositories on GitHub. Unlike GitHub and similar systems that provide
_individual_ performance metrics (e.g. commits), the Metrics Dashboard focuses
on metrics indicative of _team_ progress and project health. The Metrics
Dashboard allows the user to submit the URL of a hosted repository for batch
analysis, whose results are then cached. Upon completion, the user can
interactively study various metrics over time (at varying granularity),
numerically and visually. The initial version of the system is up and running
as a public cloud service (SaaS) and supports project size (KLOC), defect
density, defect spoilage, and productivity. While our system is by no means
the first to support software metrics, we believe it may be one of the first
community-focused extensible resources that can be used by any hosted project.

##  I Introduction

Software engineering as practiced today is no longer about the stereotypical
monolithic life cycle processes (e.g., waterfall, spiral, etc.) found in most
software engineering textbooks, aimed primarily at large-scale software teams.
These methods are often met with resistance by small/medium sized development
teams, owing to their inherent complexity and rather limited data collection
strategies that predominated the 1980s (a fervent period for emerging software
engineering research) until relatively recently in the mid-2000s. As is well
reported in the industry, more traditional methodology has given way to agile
methods. We welcome this trend but think there is room to include many of the
contributions of traditional software engineering, especially as agile
projects mature and enter maintenance mode (thereby becoming much more like
traditional SE projects and requiring more careful change management in
particular).

The discipline and practice of software engineering includes the subdomain of
software quality, which has an established theoretical foundation for software
metrics. In our work, we aim to demonstrate how software processes can be
effective for agile software teams by using best features/practices of various
models without impeding developer productivity, especially with a growing
number of cloud-based solutions for hosting projects (the most prominent being
GitHub), where robust data collection is taking place and is available for
anyone to study. Our overarching motivation for focusing on software quality
(and measuring it) is to support software projects in applied computing areas
such as scientific computing (e.g., bioinformatics) and engineering, where
there is a real need to trust the software as building blocks for larger-scale
science/engineering challenges. Co-author Thiruvathukal recently co-edited a
volume [2] that examines current practices used in scientific software
development teams. In this edited volume, many scientific software development
teams report the use of practices such as unit/regression testing and design
methodologies. Although there were few reports on metrics, the subject is one
of interest to scientific software developers, especially when it comes to
performance, algorithmic correctness (related to defect density), and
maintainability (related to code complexity). Co-author Thiruvathukal is
leading a survey to understand the metrics needs of scientific software
developers, which we expect to report separately in mid-2018.

The challenge is to identify the most effective software engineering practices
from a large suite of tools and techniques and incorporate them into existing
workflows. Software development teams today actively resist using a practice
if it incurs any additional workload on already short-staffed teams,
especially if the solution is not integrated into existing infrastructure
(e.g., GitHub, used by many open source, computational science projects
alike). Accordingly, our initial Metrics Dashboard implementation can be
incorporated in any existing GitHub-based workflow without requiring
developers to install any additional software. Architecturally, the Metrics
Dashboard can support any version control system, but we intentionally focused
our efforts on GitHub, where the vast majority of scientific software projects
we are studying host their efforts.

Software metrics are a critical tool that provides continuous insight into
products and processes, and helps build reliable software in mission-critical
environments. Using software metrics, we can perform calculations that help
assess the effectiveness of the underlying software or process.

The two types of metrics relevant to our work are:

  1. 1.

Structural metrics, which tend to focus on intrinsic code properties like code
complexity.

  2. 2.

In-process metrics, which focus on a higher-level view of software quality,
measuring information that can provide insight into the underlying software
development process.

We understand that metrics are often used to evaluate individual developer
productivity rather than overall project quality and progress. For example, a
large number of commits made to a project may or may not have any impact on
the software quality (yet this information is displayed prominently on sites
like GitHub). Optimizing for one metric could result in unintended
consequences for a project. For example, these commits could be overly complex
or introduce defects. Therefore, we seek to identify metrics that will be
useful to the project as a whole.

Our aim has been to develop and evaluate a Metrics Dashboard to support
Computational Science and Engineering (CSE) software development projects. To
this end, we have conducted the following activities:

  1. 1.

Assess how metrics are used, and which general classes/types of metrics will
be useful in CSE projects.

  2. 2.

Develop a Metrics Dashboard that will work for teams using source code
management sites like GitHub, Bitbucket, etc.

  3. 3.

Assess the effectiveness of the Metrics Dashboard in terms of project success
and developer attitude towards metrics and process.

###  I-A Novelty/contribution of this work

The Metrics Dashboard distinguishes itself in various ways:

  1. 1.

It is free/open-source software distributed under the GNU Affero General
Public License (AGPL);

  2. 2.

It has an extensible architecture that makes it easy to support and study
additional metrics;

  3. 3.

It provides both a human-facing web application and a RESTful web service for
consumption by programmatic clients;

  4. 4.

We have implemented the Metrics Dashboard using modern web service/application
technologies and a scalable and sustainable architecture;

  5. 5.

It is running as a publicly available, cloud-based software-as-a-service
(SaaS) instance, and users and contributors can choose to self-host their own
instance;

  6. 6.

It supports batch processing for offloading longer-running analytics jobs to
dedicated resources. (We have developed an Apache Spark version of our
analytics services, which is the subject of a future paper.)

While this work is part of an effort to address sustainable practices in
scientific software development, we believe it to be more broadly applicable
to any interdisciplinary software development community.

###  I-B The rest of the paper

The remainder of the paper is organized as follows. In Section II, we present
a summary of related work. In Section III, we discuss the initial set of _in-
process_ metrics that we are supporting in the current (live) version of
Metrics Dashboard at http://luc-metrics.herokuapp.com. In Section IV, we
describe the overall system architecture and the data visualizations it
supports with respect to the various metrics. In Section V, we present a case
study based on Google’s _Project Go_. In Section VI, we end with a discussion
of our initial conclusions and future plans.

##  II Related Work

Our efforts build upon the main reference works by Fenton and Bieman [3] in
the area of software metrics and by Kan [4] in the more specific area of
software quality metrics. As part of their comprehensive study of this area,
these works stake out the software metrics design space in terms of its
dimensions product metrics, process metrics, and project metrics, and define
software quality metrics as a crosscutting matter. More recent works study the
role of software metrics in agile processes and organizations [5, 6].

There have been various efforts to study the relationship between software
metrics and development processes. Lind and Vairavan [7], in the context of a
real-time medical imaging system, examined the relationship among various
structural software metrics and the relationship between those metrics and the
development process, finding that even simple metrics such as code size
correlate relatively well with development effort. More recently, Gala et al.
[8], studied the ratio of the volume of messages on project developer mailing
lists and issue trackers to commits; these metrics showed some promise as
indicators of project health.

_Defect density_ and related metrics have been receiving considerable
attention. Gupta et al. [9] conducted a longitudinal case study of two related
internal projects—a reusable framework and its application—and two metrics,
defect density and _change density_ , with defect density used as a quality
metric. Shah, Morisio and Torchiano [10] carried out a metastudy that cites 19
other papers studying 109 projects and determined that further work was needed
to identify effective predictors of defect density. In another project
involving defect density, Shah et al. [11] analyzed 61 projects along two
dimensions— _process maturity_ and _process type_ (hybrid vs. waterfall)—using
defect density to measure impact of these two dimensions; they observed that
process maturity has a smaller impact than process type on defect density.

Another line of research has focused on the classification of metrics,
especially on the basis of mining software repositories (MSR). In particular,
Saraiva et al. [12] present the case for a catalog of object-oriented software
maintainability metrics and made an initial attempt to classify a large number
of (primarily structural) maintainability metrics on the basis of 138 primary
studies. Similarly, Bouwers et al. [13] argue that a detailed, formally
structured catalog of metrics will enable users to make an informed decision
as to which metrics to use; in addition, they see value in representing the
relationships among metrics. Ohira et al. [14] argue that it is important to
distinguish low-impact defects from high-impact ones; to facilitate further
research in this area, they used manual classification to produce a dataset of
defects in several major Apache projects. Chaturvedi et al. [15] conducted a
meta-analysis of about 150 papers from all past MSR and other relevant
conferences that used data mining for the experimental analysis of software
engineering projects; they classified the data mining tools used in or
developed for these projects, as well as the data mining tasks for which these
tools were used, finding that about half of these data mining tasks are
classification tasks and about two thirds of the tools used are common
existing statistical tools.

There are few actual tools with a focus on in-process metrics. Sharma and
Kaulgud [16] present PIVoT, a proprietary, in-house tool that supports the
automated, non-invasive collection and analysis of in-process and project
health metrics. Gousios and Spinellis [17, 18] have similar goals as our
effort in terms of valuing both product and process metrics and developing a
working, publicly available, open-source tool or service for software
engineering research. Their earlier tool predates the widespread availability
of GitHub and focuses on open-source project repositories via conventional
version control and issue tracking systems. Their current tool, _GHTorrent_ ,
mines half a year’s worth of GitHub commits from hundreds of thousands of
GitHub repositories and makes the resulting linked data available for ”deep
crawling” in the form of a public MongoDB database.

##  III Supported Metrics

In this section, we describe the metrics supported by the Metrics Dashboard.
We begin with a general discussion of how we sample data (applicable to all of
our metrics), followed by an in-depth discussion of the process-related
metrics we support in the current Metrics Dashboard implementation.

###  III-A Longitudinal metrics and sampling

Any process, including a software development process, occurs over time, and
is therefore _longitudinal_ in nature. In our work, we aim to evaluate the
development process through metrics that must themselves be longitudinal;
i.e., they are functions of time. For example, code size may change over time
whenever a committer inserts or deletes portions of the code. One way of
measuring code size is lines of code (LOC), which is still used today in many
projects (and even on GitHub).

While we can conceive these metrics as continuous functions of time, it is
impractical to treat them as such, and one typically uses _sampling_ to
convert them to discrete functions of time. The choice of sampling rate
(frequency) is a practical one. If one wanted to observe, say, intra-day
phenomena, one would choose a relatively high sampling rate, such as hourly or
even every 15 minutes. Our research, however, focuses on longer-term
phenomena, so the commonly used daily sampling rate will be sufficient. In
practice, daily measurements are taken at midnight local time (00 hours).

Less frequent samples can always be obtained by _downsampling_ (decimation) as
follows. The measurement for a metric $y(d)$ for a calendar interval
$[d_{0};d_{1}]$, where $d_{i}$ are calendar dates such that
$d_{0}\leq{}d_{1}$, is given as the daily average of $y$ over the time
interval:

$y([d_{0};d_{1}])=\frac{\sum_{i=0}^{d_{1}-d_{0}}y(d_{0}+i)}{d_{1}-d_{0}+1}$
(1)

Using this definition, we can obtain measurements by week, month, or other
arbitrary period.

###  III-B Metrics supported in our dashboard

####  III-B1 Project size

Before we can speak about many software metrics, it is important that we
discuss a common metric that is used by other metrics, including issue density
(covered in the next section): the _code size_ , which is focused on using
standard _source lines of code_ (LOC) counting. Our intent is not to debate
the merits of different LOC counting mechanism (physical, logical, or non-
comment LOC), here, because this choice itself is an extensible aspect of our
architecture.

A project consists of one or more files, each with its own commit history. In
this paper, we group longitudinal metrics with granularity of week or month,
and we calculate the project size over a time range $[t;t^{\prime}]$
corresponding to the requested granularity. Accordingly, we calculate the
project size by looking at the commit history of each file in the project and
partitioning the commit history based on the requested granularity (a window
of time).

Figure 1: Calculation of KLOC over the commit history of a project

Figure 1 shows how to calculate _file size_ $s_{f}$ (in LOC) for the file
column.py in a GitHub project named Astropy (chosen from projects we’re
actively tracking). We consider this file’s commit history from 24 Nov 2013 to
15 Dec 2013 with weekly granularity. The _cumulative_ LOC for this file from
the beginning of week one until the first commit on 27 Nov 2013 is $s_{0}=0$.
Week one shows one commit for the file, resulting in the _incremental_ LOC
value ${\Delta s}_{1}=793$ from its first commit date (27 Nov 2013) to its
second commit date (1 Dec 2013). On the second commit, we add the incremental
LOC of ${\Delta s}_{2}=+7$ to the cumulative LOC $s_{1}=793$ of the previous
commit, resulting in $s_{2}=800$ between the second commit date to the third
commit date. The process continues for subsequent commits on the file within
the given time range. Note that the incremental LOC can contain zero or
negative values; we simply add the signed LOC value for each commit to the
previously calculated cumulative LOC. Owing to the longitudinal nature of our
computation, we calculate the elapsed time between consecutive commits in
milliseconds and later convert this to seconds for visualization of results,
giving us a cumulative LOC multiplied with the time range $[t_{i};t_{i+1}]$
from the current commit at $t_{i}$ to the next commit at $t_{i+1}$. The last
size value of the file during time range $[t;t`]$ is $s_{n}$ at the time
$t_{n}$ of the last commit. Finally, by dividing the combined cumulative
result by the time interval corresponding to the requested granularity, we end
up with a project size in LOC instead of seconds-LOC. For _project size_
$s_{p}$, we add the corresponding values for each file in the project, using
same granularity window (week or month).

In other words, we can calculate the size $\bar{s}_{f,[t;t^{\prime}]}$ of a
file $f$ during a time range $[t;t^{\prime}]$ as the weighted mean size of $f$
over that time range:

$\displaystyle\frac{s_{f,t}(t_{1}-t)+(\sum_{i=1}^{n-1}{s_{f,t_{i}}(t_{i+1}-t_{i})})+s_{f,t_{n}}(t^{\prime}-t_{n})}{t^{\prime}-t}$
(2)

where $n$ is the number of commits for $f$ during $[t;t^{\prime}]$ and
$t_{1},\dots,t_{n}$ are the times of these commits.

The size $\bar{s}_{p,[t;t^{\prime}]}$ of a project $p$ during a time range
$[t;t^{\prime}]$ is then given as the sum of the sizes of all source files
$f_{1},\dots,f_{k}$ in $p$ during that time range.

$\displaystyle\bar{s}_{p,[t;t^{\prime}]}=\sum_{j=1}^{k}{\bar{s}_{f_{j},[t;t^{\prime}]}}$
(3)

The calculation of project size is crucial for the metrics calculations of
issue density and team productivity covered in Sections III-B2 and III-B4,
respectively.

####  III-B2 Issue density

Defined as the number of issues in a project during a specific time period of
development divided by project size. At any given time, the issue density
$\rho$ of a project is given as the number of reported issues per project size
$s$ in KLOC.

$\displaystyle\rho=|D|/s$ (4)

where $D$ is the set of reported issues in the project at time $t$ and $|D|$
is its cardinality.

Given our focus on open source projects in GitHub, we actually refer to this
metric as _issue density_ , as opposed to the more specific term _defect
density_ , as the original research applies to any issues, not just bugs.
GitHub provides a feature for tracking tasks, enhancements, and bugs for a
project, which means that not all reported issues are actually defects. Having
high issue density, regardless of confirmed defects, could be a sign of an
unhealthy project. For example, if there are a large number of feature
requests that are never acted upon in any way, it means that the software is
lacking many features the users want. This would be a possible warning sign
for poor customer support and, eventually, would lead to low customer
satisfaction (as argued by Deming [19] to be an overarching indicator of
quality).

####  III-B3 Issue spoilage

We define the _age_ of an issue $d$ at time $t$ as the time period from its
creation time $t_{d}$ to $t$, and the _issue spoilage_ of a project as the
mean age of unresolved/open issues in the project at time $t$.

$\displaystyle\alpha_{d}(t)$ $\displaystyle=$ $\displaystyle t-t_{d}$ (5)
$\displaystyle\sigma(t)$ $\displaystyle=$ $\displaystyle\sum_{d\in
D}{\alpha_{d}(t)}/|D|$ (6)

where $D$ is the set of issues in the project at time $t$ and $|D|$ is its
cardinality.

These metrics are useful in determining project health and team effectiveness.
While we learn a great deal about issue density metrics, not attending to
other non-defect issues (e.g., enhancement requests) is something developers
would like to know, if only to improve customer satisfaction.

####  III-B4 Productivity

The most commonly used model for productivity measurement expresses
productivity as the ratio of _process output influenced by personnel_ divided
by _personnel effort or cost during the process_. We define productivity $P$
in the following, team-oriented way:

$\displaystyle P=\bar{s}/E$ (7)

where project size $\bar{s}$ is defined in equation 3, and team effort $E$ is
calculated as the number of person hours assigned to the project during the
time period of interest, $[t;t^{\prime}]$.

##  IV The Metrics Dashboard System

In this section, we present a detailed description of the system architecture
underlying the Metrics Dashboard system. We have architected the system with
the hope of using best practices in web (services) design, especially when it
comes to extensibility and supporting proper separation of concerns. We also
discuss the data visualizations, which allow one to overlay various metrics to
obtain a long-term perspective for any GitHub project. The web service and the
visualizations support the case study in the next section.

###  IV-A Architectural overview

Figure 2: Metrics Dashboard system architecture

The overall system architecture is shown in Figure 2 and comprises four
general components:

  * •

web tier (upper left)

  * •

user services tier (upper center)

  * •

computational services tier and RESTful API (lower left and center)

  * •

persistence tier (rightmost column)

Each of these tiers is responsible for different aspects of the Metrics
Dashboard. The web tier acts almost entirely as a presentation layer and also
exposes user functionality (optionally) for requesting the addition of
projects to the dashboard. Any project that is both open source and public can
be tracked. Our current implementation only supports login with GitHub, given
that the only projects we’re tracking are hosted on GitHub. Future work will
support other major services, such as Bitbucket and GitLab.

The web tier is separated from a layer (written in Node.js) that interfaces to
the services tier, specifically when it comes to the user-specific requests to
track new projects and to get the (current) results of any given repository.
We provide this separation because there is user-level metadata that is only
needed by the web tier, which we want to keep separate from the computational
tier.

The computational tier is where we compute and update various metrics with
some degree of regularity. The analysis of some large repositories (e.g. the
Linux kernel, which is several GB) simply cannot be done in real time. For
these projects, we run them on dedicated cloud instances and publish the
results (to the persistence layer) when they become ready. This also allows
the web tier to remain responsive at all times without being burdened by long-
running computations.

The persistence layer is responsible for taking the results of a computation
and ensuring it is stored in the underlying database, presently MongoDB. We
decided on MongoDB because our web services—and ultimately the web tier
itself—all work with JSON (JavaScript Object Notation). MongoDB is also
precisely designed for this sort of workflow and allows us to perform
essential querying/transformation using JavaScript-based NoSQL queries.

###  IV-B Data visualizations

A primary focus of this application is naturally the provision of a dashboard
for visualizing a project’s metrics data. As such, the Metrics Dashboard has
been designed for use in different contexts for a public and private user.

####  IV-B1 Public dashboard

A public dashboard has been developed to provide data visualization of key
metrics, as outlined above. This dashboard requests a given project’s data
from the exposed web service API, thereby consuming a project’s metrics data
with real-time updates. A public user is able to select an overview dashboard
for key project metrics, and then refine the dashboard view for a specific
metric.

The dashboard itself is interactive, thereby permitting a user to quickly and
easily filter and discriminate data based on date ranges. For example, a pie
chart of project years enables quick filtering by a single year or multiple
years. Likewise, each dashboard view is predicated on a frequency option for
the metrics’ data, including by week and month. Contextual dashboard links are
offered above the visualizations, which enable quick selection of required
metric and tracked project frequency.

####  IV-B2 Private dashboard

In addition to the above options in the public dashboard, an authenticated
private user is provided with a customizable private dashboard. This
customization includes two primary domains, notably any required dashboard
visualizations and personal projects. A private user is, therefore, able to
select their preferred visualization type for their customized dashboard.

####  IV-B3 Sample visualizations

Each dashboard, both public and private, includes default visualization types,
which are designed to aid in quick and easy comprehension of the selected data
and frequency.

Current default visualization types for chosen metric and frequency include,

  * •

_pie chart_ : presents clear division of years within a tracked project’s
dataset, e.g., Figure 6

  * •

_single line chart_ : initial visualization, e.g., issue density by month as
in Figure 3

  * •

_bar chart_ : alternative visualization type

  * •

_dual-comparison line chart_ , e.g., issue density against KLOC as in Figure 4

  * •

_multi-comparison line chart_ , e.g., issues open, issues closed, open and
closed cumulative issues against issue density as in Figure 5

Each dashboard also features a structured table of metrics data, which
includes comparative data, chosen metric data, date, and year.

For example, we can view this combination of visualization types for “issues
open” per month for a chosen project. We can start with our default line
graph, as shown in Figure 3, then start to compare “issues open” against
“issue density” in Figure 4, and finally view a multi-comparison line graph
for each “set of open issues compared against issue density,” as shown in
Figure 5. A user may also choose to filter the dashboard visualizations by
years, as represented in a pie chart in Figure 6.

Figure 3: Example line chart for issues open by month Figure 4: Example
comparison line chart for issues open against issue density Figure 5: Example
multi-comparison chart for issues open against issue density Figure 6: Example
pie chart for years in dataset

An example dashboard for the ’issues open’ metric with monthly granularity is
available at https://luc-
metrics.herokuapp.com/dash/public/month/issues/open/astropy.

##  V Case Study

In this section, we present a case study based on _Project Go_ , an open-
source programming language developed by Google and the repository is hosted
in GitHub. Figure 7 shows the issue density against KLOC for the project by
month.

Figure 7: Project Go: Line chart for density against KLOC by month

At first glance, we note a steep drop in the issue density. This drop occurs
without any corresponding changes in the KLOC for the code, which suggests
that a large number of issues were resolved in a short period of time. We
decided to explore this further to understand what might have happened in the
project and whether it is a cause for concern. Here is what we did to explore
further:

  * •

Navigate to the GitHub issues section for the project
(https://github.com/golang/go/issues?q=sort:created-asc). Here, we notice that
the first issue for the project was created in October of 2009.

  * •

Check the metrics dashboard service
(https://tirtha.loyolachicagocs.org/metrics/api/density/golang/go/master?groupBy=week),
to identify the window where the drop occurred. The results obtained from the
service will be in JSON format as shown below, which contains the fields open
or close and openCumulative and closeCumulative, which specify the issues
opened or closed in the chosen granularity (by week) and the issues that are
in the open or close state in the current window of chosen granularity (by
week as well).

         {
      {
        "start_date": "2014-11-24T00:00:00Z",
        "end_date": "2014-12-01T23:59:59Z",
        "kloc": 651.0461298714263,
        "issues": {
          "open": 30,
          "closed": 0,
          "openCumulative": 9161,
          "closedCumulative": 0
        }
      },
      {
        "start_date": "2014-12-01T00:00:00Z",
        "end_date": "2014-12-08T23:59:59Z",
        "kloc": 653.9527515172911,
        "issues": {
          "open": 34,
          "closed": 0,
          "openCumulative": 9195,
          "closedCumulative": 0
        }
      },
      {
        "start_date": "2014-12-08T00:00:00Z",
        "end_date": "2014-12-15T23:59:59Z",
        "kloc": 639.6212584045984,
        "issues": {
          "open": 120,
          "closed": 7968,
          "openCumulative": 1347,
          "closedCumulative": 7968
        }
      }
    }

On close inspection, we notice that the date 2014-12-08T23:59:59Z is when the
dip occurs, also notice that issues closed is 0 and closedCumulative is 0. For
the next window, ending on 2014-12-15T23:59:59Z, both closed and
closedCumulative are 7968.

  * •

Navigate to GitHub issues (https://github.com/golang/go/issues) to check if
the values reported by the metrics dashboard service is correct. If we filter
using the criteria closed:<2014-12-08, we see that no issues were closed
before this date (https://github.com/golang/go/issues?q=closed:<2014-12-08),
even though the first issue was opened in October, 2009.

  * •

Change the filter to closed:<2014-12-09
(https://github.com/golang/go/issues?q=closed:<2014-12-09) and we will see
that 7926 issues were closed. Change the date to 2014-12-15
(https://github.com/golang/go/issues?q=closed:<2014-12-15) and you will see
7968 issues were closed, which matches the result obtained through the Metrics
Dashboard service.

So, to summarize, the Project Go team managed to close 7926 issues in one day,
which is not normally considered to be a good practice, considering the fact
that the team hadn’t closed any of the identified issues since October, 2009.
A point to note here is that the status of issues in GitHub can be changed
(even for no apparent reason). In this case, however, were are concerned only
with the open and closed dates of issues. Having an issue open for five years
suggests a common issue anecdotally associated with open source projects: the
project relies heavily on volunteers who might not be able to keep up with
incoming issues and do new development at the same time. For the above
analysis, the Metrics Dashboard’s ability to adjust granularity to week level
helped to narrow down the exact date when the dip occurred.

Figure 8: Project Go: Line chart for density against spoilage by month

Figure 8 shows the issue density against spoilage by month, as one expects the
spoilage value dips at around the same window when the issue density dips.
Observe that spoilage increases until the end of the year 2014 to a peak of
almost 7.0, as the time to fix issues increased. After the dip the spoilage
has remained constant which is a good indicator that the issues are being
closed regularly and newer issues are identified and tracked. For an active
project to be healthy, the spoilage should not drop too low, which could
indicate that the project isn’t being tested and the user community isn’t
actively identifying or reporting issues.

Figure 9: Project Go: Line chart for issues grouped by week

Figure 9, shows the issues for Project Go by week, as one expects the issues
for the project were opened until the end of the year 2014 (cumulative open
issues are shown in red). The closed cumulative issue count, shown in yellow,
shows that the issues for the project were closed beginning the end of the
year 2014. Since 2015, the team or users have continued to open and close
issues at a fairly steady rate and no drastic changes in the values are seen.
This means that the team is working to improve its work flow when it comes to
resolving issues.

##  VI Conclusion and Future Work

This paper reflects our initial effort to design and develop a dashboard that
can be used by project teams to assess the effectiveness of their software
development process. Although at an early stage, we have already been able to
use the dashboard to understand issues in existing scientific software
projects that could be helpful to the project maintainers.

We are presently able to track multiple science-focused software development
projects. It is not within the scope of our current paper to describe our
findings about these projects. Our ultimate goal is to make it possible for
teams to do their own wellness monitoring and improve their own software
quality.

The current prototype at http://luc-metrics.herokuapp.com is fully-functioning
and can be used by the public. Anyone with a GitHub account can log into our
system and suggest new projects for us to track, which will result in them
being queued for analysis and regular updating.

We’re in the midst of analyzing the results from a survey of scientific
software teams to assess understanding and interest in software metrics. Upon
completing the analysis of this survey, we will continue work on the Metrics
Dashboard to support additional metrics and update this draft paper for
conference/journal archival.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
under Grant ACI-1445347 and Amazon Web Services. We also acknowledge the
Argonne Leadership Computing Facility for a special director’s allocation of
75,000 hours of time on the Cooley/Mira supercomputers.

## References

  * [1] “WSSSPE: Working Towards Sustainable Software for Science: Practice and Experience.” [Online]. Available: http://wssspe.researchcomputing.org.uk/
  * [2] J. C. Carver, N. P. C. Hong, and G. K. Thiruvathukal, Eds., _Software Engineering for Science_ , ser. Chapman & Hall/CRC Computational Science. Boca Raton, FL: CRC Press, Oct. 2016.
  * [3] N. Fenton and J. Bieman, _Software Metrics: A Rigorous and Practical Approach_ , 3rd ed., ser. Chapman & Hall/CRC Innovations in Software Engineering and Software Development. Boca Raton, FL: CRC Press, Oct. 2014.
  * [4] S. H. Kan, _Metrics and Models in Software Quality Engineering_ , 2nd ed. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2002.
  * [5] E. Kupiainen, M. V. Mäntylä, and J. Itkonen, “Why Are Industrial Agile Teams Using Metrics and How Do They Use Them?” in _Proceedings of the 5th International Workshop on Emerging Trends in Software Metrics_ , ser. WETSoM 2014. New York, NY, USA: ACM, 2014, pp. 23–29.
  * [6] C. W. H. Davis, _Agile Metrics in Action: How to measure and improve team performance_. Greenwich, CT: Manning Publications, Jul. 2015.
  * [7] R. K. Lind and K. Vairavan, “An experimental investigation of software metrics and their relationship to software development effort,” _IEEE Transactions on Software Engineering_ , vol. 15, no. 5, pp. 649–653, May 1989\.
  * [8] S. Gala-Pérez, G. Robles, J. M. González-Barahona, and I. Herraiz, “Intensive metrics for the study of the evolution of open source projects: Case studies from Apache Software Foundation projects,” in _2013 10th IEEE Working Conference on Mining Software Repositories (MSR)_ , May 2013, pp. 159–168.
  * [9] A. Gupta, O. P. N. Slyngstad, R. Conradi, P. Mohagheghi, H. Ronneberg, and E. Landre, “A Case Study of Defect-Density and Change-Density and their Progress over Time,” in _11th European Conference on Software Maintenance and Reengineering, 2007. CSMR ’07_ , Mar. 2007, pp. 7–16.
  * [10] S. M. A. Shah, M. Morisio, and M. Torchiano, “An Overview of Software Defect Density: A Scoping Study,” in _Software Engineering Conference (APSEC), 2012 19th Asia-Pacific_ , vol. 1, Dec. 2012, pp. 406–415.
  * [11] ——, “The Impact of Process Maturity on Defect Density,” in _Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement_ , ser. ESEM ’12. New York, NY, USA: ACM, 2012, pp. 315–318.
  * [12] J. Saraiva, S. Soares, and F. Castor, “Towards a catalog of Object-Oriented Software Maintainability metrics,” in _2013 4th International Workshop on Emerging Trends in Software Metrics (WETSoM)_ , May 2013, pp. 84–87.
  * [13] E. Bouwers, A. v. Deursen, and J. Visser, “Towards a Catalog Format for Software Metrics,” in _Proceedings of the 5th International Workshop on Emerging Trends in Software Metrics_ , ser. WETSoM 2014\. New York, NY, USA: ACM, 2014, pp. 44–47.
  * [14] M. Ohira, Y. Kashiwa, Y. Yamatani, H. Yoshiyuki, Y. Maeda, N. Limsettho, K. Fujino, H. Hata, A. Ihara, and K. Matsumoto, “A Dataset of High Impact Bugs: Manually-Classified Issue Reports,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR)_ , May 2015, pp. 518–521.
  * [15] K. K. Chaturvedi, V. B. Sing, and P. Singh, “Tools in Mining Software Repositories,” in _2013 13th International Conference on Computational Science and Its Applications (ICCSA)_ , Jun. 2013, pp. 89–98.
  * [16] V. S. Sharma and V. Kaulgud, “PIVoT: Project insights and Visualization Toolkit,” in _2012 3rd International Workshop on Emerging Trends in Software Metrics (WETSoM)_ , Jun. 2012, pp. 63–69.
  * [17] G. Gousios and D. Spinellis, “A platform for software engineering research,” in _Proc. 6th IEEE International Working Conference on Mining Software Repositories (MSR)_. Vancouver, BC, Canada: IEEE, May 2009, pp. 31–40.
  * [18] ——, “GHTorrent: Github’s data from a firehose,” in _2012 9th IEEE Working Conference on Mining Software Repositories (MSR)_ , Jun. 2012, pp. 12–21.
  * [19] W. Scherkenbach, _The Deming Route to Quality and Productivity_. WWS, Inc., 2011.