# Cross-lingual Emotion Detection

###### Abstract

Emotion detection can provide us with a window into understanding human
behavior. Due to the complex dynamics of human emotions, however, constructing
annotated datasets to train automated models can be expensive. Thus, we
explore the efficacy of cross-lingual approaches that would use data from a
source language to build models for emotion detection in a target language. We
compare three approaches, namely: i) using inherently multilingual models; ii)
translating training data into the target language; and iii) using an
automatically tagged parallel corpus. In our study, we consider English as the
source language with Arabic and Spanish as target languages. We study the
effectiveness of different classification models such as BERT and SVMs trained
with different features. Our BERT-based monolingual models that are trained on
target language data surpass state-of-the-art (SOTA) by 4% and 5% absolute
Jaccard score for Arabic and Spanish respectively. Next, we show that using
cross-lingual approaches with English data alone, we can achieve more than 90%
and 80% relative effectiveness of the Arabic and Spanish BERT models
respectively. Lastly, we use LIME to analyze the challenges of training cross-
lingual models for different language pairs.

Keywords: cross-lingual, multi-lingual, knowledge transfer, emotion detection

languageresourceLanguage Resources

Cross-lingual Emotion Detection

Sabit Hassan1, Shaden Shaar2, Kareem Darwish3
---
1University of Pittsburgh, 2Cornell University, 3aiXplain Inc.
1Pittsburgh, PA 15260, 2Ithaca, NY 14850, 3Los Gatos, CA 95032
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Detecting emotions in text has a wide variety of applications ranging from
identifying anger in customer responses to evaluating the emotional well-being
of individuals and societies. Hence, there has been much recent work on
automatically detecting emotions in text, with special emphasis on social
media posts. Despite contention [Ortony and Turner, 1990], much work on
psychology suggests that basic emotions are universal [Ekman, 1999, Scarantino
and Griffiths, 2011]. However, the expressions and perceptions of emotions may
exhibit cross-cultural variations [Hareli et al., 2015], which may complicate
cross-lingual learning. Nonetheless, the ability to perform such cross-lingual
classification of emotions effectively is advantageous as it would avoid the
laborious task of building emotion detection datasets, which involves
collecting representative tweets and properly tagging them.

While cross-lingual methods have shown success for some tasks such as cross-
lingual search [Chin et al., 2014], named entity recognition [Darwish, 2013],
and sentiment analysis [Zhou et al., 2016], lesser success has been observed
for other tasks such as parsing [Guo et al., 2015] and offensive language
detection [Pamungkas and Patti, 2019], due to the divergences in linguistic
and cultural specificities. In this paper, we explore the efficacy of using an
existing tagged emotion detection corpus from one language to another, and we
perform rigorous analysis of results to ascertain the sources of errors.
Specifically, we attempt to extend the SemEval2018 English emotion detection
dataset [Mohammad et al., 2018a] to Arabic and Spanish. Though the
effectiveness of cross-lingual training likely yields lower results compared
to training on tagged corpus in the target language, we strive to achieve the
lowest possible drop in effectiveness, which would imply that emotions may
generalize across linguistic and cultural boundaries. The efficacy of cross-
lingual models depends on whether emotions are expressed in similar ways
across languages and cultures and on the effectiveness of cross-lingual
approaches. In the case of Arabic and English, they are not only different in
script, morphology, syntax, etc., they are typically spoken by people from
different cultures. Conversely, Spanish shares more in common with English in
terms of script and culture.

Given English data, we employ a variety of cross-lingual methods that we
compare to using smaller Arabic or Spanish training sets and to mixtures of
English with either Arabic or Spanish data. Recent development of inherently
multilingual models such as pre-trained multilingual Transformer models, e.g.
multilingual BERT (mBERT) [Devlin et al., 2019], and the Multilingual
Universal Sentence Encoder (mUSE) [Cer et al., 2018] present new opportunities
for cross-lingual approaches. We compare the effectiveness of such models to
more traditional models that involve translating the training set into the
target language and then retraining. We also experiment with using a parallel
corpus, where we tag the text in the source language, and use the
corresponding text in the target language for training. Using the translated
training set and the parallel corpus enables us to use more genre specific
static embeddings, target language contextual embeddings, and linguistic
features to obtain more effective classification, as compared to using mBERT
embeddings. In addition, we study combination of the aforementioned
approaches. To ascertain the efficacy of cross-lingual training we pursued 3
distinct paths as follows:

  1. 1.

using English training data exclusively. To cross the language barrier, we
used: multilingual BERT (mBERT), transformer-based multilingual Universal
Encoder, automatic machine translation of the training set, and automatic
tagging of the English side of a large Arabic-English/Spanish-English parallel
corpora and using the Arabic/Spanish sides for training.

  2. 2.

using Arabic/Spanish training data exclusively.

  3. 3.

using mixtures of Arabic or Spanish in combination with English training data.
In this approach, Arabic or Spanish training sets are augmented with English
data. We crossed the language barrier using the aforementioned methods.

The contributions of this paper are as follows:

  * •

We show that cross-lingual models, with English as the source, can achieve
more than 90% and 80% of the relative effectiveness of training SOTA
monolingual models for Arabic and Spanish respectively. This implies that
emotions are generally expressed in similar ways across languages and
cultures. We show that the effectiveness of different cross-lingual approaches
follow similar trends for both languages.

  * •

As a byproduct of our work, we trained Arabic and Spanish models that notably
beat the best SemEval2018 results (52.9 Jaccard vs. 48.9 for Arabic, and 52.4
vs. 46.9 for Spanish).

  * •

Using LIME [Ribeiro et al., 2016] for interpretability, we compare models
trained on target and source languages. We manually annotate and analyze LIME
output to understand the potential challenges of cross-lingual approaches when
dealing with different language pairs.

In section 2, we discuss theories behind human emotions, their usage in the
field of natural language processing, and previous work that leverages cross-
lingual models. In section 3, we describe the dataset and systems used in this
work and present the results. In section 4, we identify limitations of cross-
lingual models across different cultures and lastly, in section 5, we present
a summary of our findings.

## 2\. Background and Related Work

Although there is no definitive list of all human emotions, ?) and ?)
suggested that there are 6 and 8 basic emotions respectively. ?) further
suggested that aside from the 8 primary emotions, there are secondary emotions
that arise from combinations of the primary emotions. ?) reviewed different
emotion categorization models and listed 65 different emotions that are
mentioned in the literature. The discrepancy between emotion categories is
reflected in tagged corpora where some use the basic 6 suggested by ?) (ex.
Abdul-Mageed et al., [Abdul-Mageed et al., 2016]), others use Plutchik’s 8
(ex. [Alhuzali et al., 2018]), and yet others use Pultchik’s 8 with secondary
emotions, such as love (joy + trust), optimism (anticipation + joy), and
pessimism (anticipation + fear) (ex. [Mohammad et al., 2018b]).

Although there has been much work on sentiment analysis [Mourad and Darwish,
2013, Elmadany et al., 2018, Hassan et al., 2021], work on Arabic emotion
detection has been relatively limited. There are a few dataset for Arabic
emotion detection. The datasets of ?), ?), and ?), which contain 3,000,
10,065, and 11,503 tweets respectively, were tagged using Ekman’s basic 6
emotions. ?) used Plutchik’s 8 basic emotions to label 7,268 tweets. The
construction of most of the aforementioned datasets was done by searching
tweets using words or hashtags that indicate specific emotions and then
manually labeling these tweets. ?) used emoticons to find relevant tweets
instead of indicative words. Similar to Arabic, there has been much work on
Spanish sentiment analysis, but little work Spanish emotion detection [Miranda
and Guzmán, 2017, Segura-Bedmar et al., 2017]. ?) created a multilingual
corpus consisting of 8.4K Spanish and 7.3K English tweets annotated for
Ekman’s 6 basic emotions and neutral emotion. Most relevant to our work is the
SemEval2018 dataset [Mohammad et al., 2018b], a multilingual publicly
available dataset. The data consists of 11K tweets in English, 4.3K in Arabic
and 7K tweets in Spanish tagged with one or more of 11 labels corresponding to
Plutchik’s 8 basic emotions in addition to love, optimism, and pessimism.

Many methods have been used for classifying emotions in text ranging from
classical machine learning approaches such as Naive Bayes and Support Vector
Machine (SVM) classifiers [Al-Khatib and El-Beltagy, 2017, Hussien et al.,
2016, Plaza del Arco et al., 2020a] to deep learning approaches such as
recurrent neural networks [Abdullah and Shaikh, 2018, Alhuzali et al., 2018].
Work on cross-lingual emotion detection is relatively scant. ?) used a biLSTM
model to classify emotions in multilingual text. ?) used a multilingual
embedding space to improve Hindi emotion detection by leveraging an English
dataset. Aside from emotion detection, cross-lingual learning was employed by
many for sentiment analysis. For example, ?) used self-learning with
multilingual BERT for document classification and Chinese sentiment analysis.
Similarly, ?) used a parallel corpus with adversarial feature adaptation to
perform cross-lingual text classification with English as a source language
and German, French, Japan and Chinese as target languages. ?) use multilingual
word embeddings with convolutional neural network for extracting opinion
target expressions. ?) translated Arabic to English to classify Arabic
sentiment using an English model. Further, ?) and ?) used bilingual
dictionaries to translate sentiment words and documents. More recent efforts
used automated Machine Translation (MT) instead of bilingual dictionaries
[Wan, 2009]. ?) used statistical (MT) for sentiment analysis in German/English
pair. Translation has also been applied to other tasks such as semantic role
labeling [Fei et al., 2020] and dependency parsing [Zhang et al., 2019].

## 3\. System Setup

### 3.1. Datasets

We used the SemEval2018 emotion detection dataset (Affect in Tweets: Task E-c)
to evaluate our cross-lingual approaches [Mohammad et al., 2018b]. Table 1
lists distribution of tweets across the different languages and train-dev-test
splits.

Language | Train | Dev | Test
---|---|---|---
English | 6,838 | 886 | 3,259
Arabic | 2,278 | 585 | 1,518
Spanish | 3,561 | 679 | 2,854
Table 1:  Distribution of tweets in SemEval2018 dataset

Each tweet was manually annotated, using crowd-sourcing, for the presence of
11 different emotions: joy, sadness, anger, fear, disgust, surprise, trust,
anticipation, love, optimism, and pessimism. Further, given the English
training and development sets, we translated them into Arabic and Spanish
using Google translation APIs to be used with our translation approach.

For our approach requiring a parallel corpus for Arabic, we used the dataset
of ?), which consists of 166K pairs of Arabic-English parallel tweet—pairs
that contain the same content but written separately in Arabic and English.
The advantage of using parallel tweets instead of translating the SemEval2018
dataset (using MT) is that parallel tweets resulted from human translation,
which would hopefully better transfer meaning and emotions. However, the
parallel tweets are not annotated for emotions. To create the annotations, we
tagged the English side using our best setup, namely BERT-uncased, and given
sentences that were tagged with at least 3 emotions on the English side, we
transferred the tags to the corresponding parallel Arabic sentences. Placing
such conditions would improve our chances of getting more accurately tagged
sentences. We experimented with different filters, and retaining tweets with
more than 3 emotions led to the best results. We omitted these experiments for
conciseness. After applying our filter, we end up with 4,450 pairs of tagged
parallel tweets.

For Spanish, due to lack of a parallel tweets corpus, we opted to use the Open
Subtitles dataset [Tiedemann, 2012], which has millions of parallel movie
subtitles in different languages. However, to be comparable to Arabic, we
randomly pick 166K pairs to have the same starting number. Then we apply the
similar filters as Arabic and ended up with 20K pairs, suggesting that movie
subtitles have more emotions than parallel tweets collected using the method
of ?).

### 3.2. Models

Transformer Model: Transformer-based pre-trained contextual embeddings, such
as BERT [Devlin et al., 2019], ULMFIT [Howard and Ruder, 2018], and OpenAI GPT
[Radford et al., 2018] have led to large improvements in many NLP tasks. We
used such transformer based models, namely bert-base-uncased (BERT-uncased),
trained on lower cased English text, bert-base-multilingual-uncased (mBERT),
which is trained on Wikipedia text from different languages, including Arabic,
Spanish, and English, AraBERT [Antoun et al., 2020], which is trained on a
larger Arabic news corpus that contains roughly 2.5B tokens, and, BetoBERT
[Cañete et al., 2020], which is trained with variety of Spanish corpora
consisting of 3B tokens. All the aforementioned models are pre-trained on
identical architectures, namely an encoder with $12$ Transformer blocks, a
hidden size of $768$, and $12$ self-attention heads. All use SentencePiece
segmentation (BP). We used the HuggingFace
implementation111https://huggingface.co, and we fine-tuned the contextual
embeddings using either English, Arabic, Spanish or a combination of Arabic or
Spanish with English training data for 10 epochs with learning rate of 2e-5
and batch size of 8. We placed an output layer with 11 output nodes
corresponding to the probabilities of each emotion class.
Support Vector Machines (SVMs): SVMs have been shown to be effective for a
variety of text classification tasks [Forman, 2008, Mubarak and Hassan, 2021,
Abdelali et al., 2021]. Due to the fact neither mBERT or AraBERT were trained
on dialectal or social media texts, using SVMs would hopefully allow us to
perform feature engineering to achieve competitive results. We used the libSVM
implementation in scikit learn222https://scikit-learn.org with a linear
kernel. We trained a binary classifier for each of the emotions  in a one vs.
rest setting.
Multi-Layer Perceptron (MLP): We implemented an MLP classifier using Keras
[Chollet and others, 2015] with a tensorflow backend [Abadi et al., 2015]. We
used it only with mUSE embeddings. The network had 512 input nodes, two hidden
layers with 256 and 128 nodes respectively, and 11 output nodes corresponding
to the different emotions. One of the major advantages of this setup is that
we can jointly train using all the labels. We trained the classifier for a
maximum of 50 epochs with early stopping with patience of 3 iterations.

### 3.3. Features

Word unigrams: Training a bag-of-words model often serves as a baseline model
(e.g., [Mohammad et al., 2018a]). We weighed word unigrams using term
frequency-inverse term document frequency (tf-idf weighting) and used with
SVMs only.
Character $n$-gram: Character $n$-grams have been shown to be effective in
representing text in retrieval and classification tasks [Mcnamee and Mayfield,
2004, Mubarak et al., 2020a]. Using them helps overcome some of the effects
of: the frequent creative word spellings on social media; and the complex
derivational morphology of Arabic – availing the need for stemming. Similar to
word unigrams, we used character $n$-grams as features when using an SVM
classifier only, and we used tf-idf weighting.
Word Embeddings: For Arabic, we used Mazajak static word-level skip-gram
embeddings [Abu Farha and Magdy, 2019], which were trained on 250M Arabic
tweets with 300-dimensional vectors. Since they were trained on tweets, they
have led to improved tweet text classification results, compared to using
mBERT, for some tasks [Mubarak et al., 2020c, Hassan et al., 2020b]. Due to
unavailability of Twitter specific embeddings for Spanish333Spanish embeddings
based on tweets https://www.spinningbytes.com/resources/wordembeddings/ are
unavailable, we opted for Spanish Billion Word (SBW) embeddings [Cardellino,
2019], trained with approximately 1.5 Billion words from different corpora
including books, newspapers and Wikipedia dumps.
mUSE: mUSE is a transformer based model that is pre-trained on multiple tasks
simultaneously [Cer et al., 2018]. The models use BP segmented text and
produces a 512 feature vector for each input sentence, and it does not require
fine-tuning.

### 3.4. Experimental Setup

#### Evaluation Metrics:

we compared results using the average Jaccard similarity ($\mathcal{J}$)
between the predicted labels and the ground truth. Jaccard score is defined as
the size of the intersection divided by the size of the union of two label
sets. It is effective in comparing results in a multi-label scenario and is
the official metric for the SemEval2018 emotion detection task. We also report
the macro F-measure across all labels ($\mathcal{F}$) and the average accuracy
($\mathcal{A}$). All results are reported on the official test sets of
SemEval2018 emotion detection task.

#### Baseline Monolingual Models:

Initially, we opted to produce SOTA models trained on Arabic or Spanish data
exclusively to compare cross-lingual models to. We trained several Arabic and
Spanish models using the SemEval2018 emotion detection dataset: 1) SVM models
trained using: word uni-grams (W[1-1]), character $n$-grams (C[1-6]), with $n$
ranging from 1 to 6; Mazajak/SBW embeddings; character $n$-grams +
Mazajak/SBW; and mUSE embeddings; 2) an MLP model trained on mUSE embeddings;
3) fine-tuned mBERT, AraBERT, or BetoBERT models.

Table 2 reports on the Arabic baseline results. As the results show, using
AraBERT yielded the best results with a $\mathcal{J}$ score of 52.9. It is
noteworthy that the best result reported in the SemEval2018 shared task was
48.9 [Mohammad et al., 2018b]. Using character $n$-grams with Mazajak
embeddings performed better than using either mUSE or mBERT. This is
consistent with results obtained for other classifications tasks involving
Arabic tweets [Mubarak et al., 2020c, Hassan et al., 2020a].

Table 3 reports on the Spanish baseline results. Using BetoBERT yielded the
best results with a $\mathcal{J}$ score of 52.4, while the best result
reported in the SemEval2018 shared task was 46.9 [Mohammad et al., 2018b]. In
contrast to Arabic, use of word embeddings (SBW) did not result in better
results compared to mUSE or mBERT. This could be due to the fact that SBW are
not Twitter specific embeddings, as opposed to Mazajak. Due to poor
performances of using word-unigrams, we excluded them from further
experiments.

Model | Features | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$
---|---|---|---|---
SemEval |  | 48.9 | 46.1 | -
SVM | W[1-1] | 35.5 | 34.3 | 83.3
SVM | C[1-6] | 42.0 | 37.9 | 85.3
SVM | Mazajak | 46.3 | 44.3 | 84.7
SVM | C[1-6]+Mazajak | 48.6 | 46.2 | 85.5
SVM | mUSE | 38.9 | 38.5 | 82.9
MLP | mUSE | 40.3 | 37.0 | 83.3
mBERT | - | 45.3 | 41.4 | 84.2
AraBERT | - | 52.9 | 48.9 | 86.6
Table 2:  Models trained on Arabic data. SemEval refers to the highest ranked system in the shared task. Model | Features | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$
---|---|---|---|---
SemEval |  | 46.9 | 40.7 | -
SVM | W[1-1] | 29.7 | 28.0 | 87.1
SVM | C[1-6] | 35.0 | 31.3 | 88.4
SVM | SBW | 22.9 | 20.2 | 85.8
SVM | C[1-6]+SBW | 37.4 | 35.2 | 87.9
SVM | mUSE | 39.4 | 40.0 | 88.2
MLP | mUSE | 42.3 | 41.5 | 88.6
mBERT | - | 43.7 | 35.8 | 88.5
BetoBERT | - | 52.4 | 53.7 | 89.8
Table 3:  Models trained on Spanish data. SemEval refers to the highest ranked system in the shared task. Model | Features | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$
---|---|---|---|---
SemEval | - | 58.8 | 52.1 | -
SVM | W[1-1] | 38.8 | 37.2 | 83.0
SVM | C[1-6] | 43.1 | 39.5 | 84.6
SVM | mUSE | 48.4 | 43.7 | 85.6
MLP | mUSE | 51.3 | 45.4 | 85.7
mBERT | - | 54.6 | 46.1 | 87.0
BERT-uncased | - | 56.4 | 53.9 | 87.3
Table 4:  Models for English emotion analysis. SemEval refers to the highest
ranked system in the shared task.

#### Cross-Lingual Models:

| Cross-lingual | Combined
---|---|---
Approach | Model | Features | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$
Best Arabic baseline result | 52.9 | 48.9 | 86.6 |
M | SVM | mUSE | 32.4 | 31.6 | 79.7 | 37.9 | 37.0 | 83.1
M | MLP | mUSE | 35.8 | 33.4 | 79.9 | 41.8 | 40.5 | 83.0
M | mBERT | - | 20.5 | 19.5 | 76.4 | 46.7 | 44.6 | 83.8
T | SVM | C[1-6] | 28.0 | 27.3 | 81.0 | 44.4 | 41.3 | 85.4
T | SVM | Mzjk | 39.3 | 37.1 | 82.8 | 43.7 | 40.4 | 85.2
T | SVM | C[1-6]+Mzjk | 42.5 | 40.2 | 82.5 | 48.0 | 45.0 | 85.6
T | AraBERT | - | 48.1 | 46.3 | 83.8 | 54.1 | 50.8 | 86.2
P | SVM | C[1-6] | 30.4 | 24.9 | 74.4 | 44.1 | 40.5 | 85.3
P | SVM | Mzjk | 36.5 | 31.9 | 74.6 | 45.1 | 43.3 | 84.3
P | SVM | C[1-6]+Mzjk | 37.5 | 32.4 | 75.2 | 48.4 | 46.2 | 85.3
P | AraBERT | - | 36.7 | 31.6 | 75.2 | 53.1 | 50.5 | 85.9
M + T | SVM | mUSE | 34.3 | 32.5 | 80.5 | 38.4 | 36.3 | 83.3
M + T | MLP | mUSE | 36.8 | 34.6 | 80.7 | 41.5 | 38.6 | 83.0
M + T | mBERT | - | 36.0 | 36.0 | 79.9 | 45.9 | 44.0 | 83.6
P + M | SVM | mUSE | 36.7 | 33.9 | 78.9 | 39.1 | 37.4 | 82.8
P + M | MLP | mUSE | 37.9 | 34.9 | 79.2 | 41.4 | 38.6 | 82.8
P + M | mBERT | - | 30.2 | 26.1 | 70.6 | 46.0 | 44.2 | 83.3
P + T | SVM | C[1-6] | 35.6 | 34.2 | 81.2 | 45.4 | 41.9 | 85.1
P + T | SVM | Mzjk | 40.5 | 37.6 | 81.8 | 43.0 | 40.4 | 84.9
P + T | SVM | C[1-6]+Mzjk | 44.0 | 40.1 | 82.2 | 48.0 | 45.1 | 85.5
P + T | AraBERT | - | 47.7 | 45.0 | 82.4 | 53.1 | 50.4 | 85.8
M + P + T | SVM | mUSE | 37.2 | 34.8 | 80.3 | 38.1 | 36.2 | 82.7
M + P + T | MLP | mUSE | 39.3 | 35.1 | 79.8 | 41.5 | 38.5 | 82.2
M + P + T | mBERT | - | 38.5 | 36.7 | 78.8 | 46.4 | 44.5 | 83.6
Table 5: Arabic cross-lingual and multilingual results. M: Multilingual Model; T: Translation; and P: Parallel corpus; Mzjk: Mazajak embeddings | Cross-lingual | Combined
---|---|---
Approach | Model | Features | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{A}$
Best Spanish baseline result | 52.4 | 53.7 | 89.8 |
M | SVM | mUSE | 36.6 | 36.9 | 85.7 | 38.8 | 37.4 | 88.4
M | MLP | mUSE | 37.4 | 38.7 | 84.2 | 42.5 | 41.4 | 88.4
M | mBERT | - | 26.5 | 23.6 | 83.3 | 42.3 | 32.8 | 88.8
T | SVM | C[1-6] | 30.1 | 28.3 | 85.3 | 38.4 | 37.0 | 88.3
T | SVM | SBW | 23.4 | 21.9 | 83.0 | 22.0 | 19.6 | 85.6
T | SVM | C[1-6]+SBW | 31.5 | 30.3 | 84.1 | 38.9 | 37.7 | 87.9
T | BetoBERT | - | 42.3 | 44.1 | 85.5 | 51.8 | 53.4 | 89.2
P | SVM | C[1-6] | 27.1 | 31.8 | 76.6 | 37.9 | 38.5 | 86.2
P | SVM | SBW | 24.3 | 24.8 | 78.5 | 23.3 | 23.6 | 83.2
P | SVM | C[1-6]+SBW | 27.9 | 32.8 | 76.5 | 38.6 | 39.3 | 86.1
P | BetoBERT | - | 34.1 | 40.5 | 77.4 | 50.5 | 51.7 | 88.2
M + T | SVM | mUSE | 36.8 | 35.9 | 85.3 | 39.6 | 38.2 | 88.2
M + T | MLP | mUSE | 37.6 | 36.4 | 84.8 | 42.0 | 42.0 | 87.8
M + T | mBERT | - | 37.0 | 33.9 | 84.6 | 42.2 | 34.0 | 88.4
P + M | SVM | mUSE | 36.1 | 38.1 | 82.4 | 36.9 | 37.4 | 86.7
P + M | MLP | mUSE | 37.0 | 39.4 | 82.1 | 40.1 | 40.8 | 87.2
P + M | mBERT | - | 32.6 | 38.6 | 77.0 | 47.7 | 47.6 | 87.3
P + T | SVM | C[1-6] | 33.1 | 35.6 | 81.6 | 40.0 | 40.0 | 86.4
P + T | SVM | SBW | 24.0 | 24.3 | 81.3 | 23.4 | 23.3 | 83.6
P + T | SVM | C[1-6]+SBW | 33.6 | 36.2 | 81.6 | 39.7 | 40.4 | 86.3
P + T | BetoBERT | - | 40.7 | 44.9 | 82.3 | 50.2 | 52.5 | 88.0
M + P + T | SVM | mUSE | 35.4 | 37.2 | 83.8 | 37.2 | 37.9 | 86.7
M + P + T | MLP | mUSE | 37.3 | 39.0 | 82.9 | 42.1 | 41.9 | 86.7
M + P + T | mBERT | - | 38.7 | 41.3 | 82.6 | 47.3 | 47.3 | 87.2
Table 6: Spanish cross-lingual and multilingual results. M: Multilingual
Model; T: Translation; and P: Parallel corpus; SBW: Spanish Billion Words

We constructed a large number of experimental setups involving the use of
English training data to tag the Arabic and Spanish test set. For setups
involving translation, we used either the training set after translation using
MT or the Arabic and Spanish sides of the parallel corpora. To use parallel
data as a conduit for translation, we experimented with several setups for
English emotion detection on the SemEval data. They included models that were
similar to the Arabic or Spanish baseline models listed earlier, save ones
with Arabic or Spanish specific static embeddings (Mazajak, SBW) and instead
of AraBERT or BetoBERT, we used BERT-uncased. The results are summarized in
Table 4. BERT-uncased outperformed MLP, SVM and mBERT. Thus, we used it to tag
the English side of the parallel corpus.

As with the baseline setups, after translation we: 1) trained an SVM
classifiers using: character $n$-grams, where $n$ ranged between 1 and 6
(C[1-6]); Mazajak/SBW embdeddings; or a combination of the character $n$-gram
and Mazajak/SBW; and 2) fine-tuned AraBERT/BetoBERT contextual embeddings, as
they performed notably better than mBERT in monolingual Arabic/Spanish
baselines.

We also had two other setups that did not involve translation, namely: using
mUSE, where we produced embedding representations of the English training set
and the Arabic/Spanish test set, and we used MLP or SVM classifiers; and using
the mBERT model, fine-tuned on the English training set, to tag the
Arabic/Spanish test set directly. We conducted experiments involving other
combinations of using multilingual representations (mUSE and mBERT), the
translated training set, and the Arabic/Spanish side of the automatically
tagged parallel corpus using similar set of models.

Table 5 summarizes the results of all cross-lingual approaches using only
English data when Arabic is the target language. We observe that mBERT was
unable to produce good results. On the other hand, using mUSE to train SVM and
MLP classifiers notably outperformed mBERT. Using a parallel corpus allowed us
to use language specific features, such as character $n$-gram, and we noticed
an improvement over using multilingual models. However, we suspect that the
problem with using parallel text is that tweets are tagged using an imperfect
model, and the errors percolate from English to the target language.
Automatically translating the English training set to Arabic led to the best
results, with AraBERT achieving a notably higher $\mathcal{J}$ score of 48.1.
A less notable improvement was noticed when using mUSE, Mazajak embedding, and
character $n$-grams on translations compared to parallel text. Our attempts to
combine translations with parallel text and multilingual representations did
not yield results that were better than using translations alone. The best
overall cross-lingual results that we obtained using AraBERT ($\mathcal{J}$
=48.1) achieved 90.9% of the best results that were obtained when training on
the Arabic data using AraBERT ($\mathcal{J}$ =52.9) and were 98.4% of the best
team’s results at the SemEval2018 Arabic emotion detection shared task
($\mathcal{J}$ =48.9). We can conclude that the use of translation along with
fine-tuned AraBERT model can lead to competitive results availing the need for
target language annotation.

Table 6 summarizes the results of all cross-lingual approaches using only
English data when target language is Spanish. We observe several similar
trends to Arabic results. Among inherently multilingual models, mBERT was
outperformed by SVM and MLP trained on mUSE. Translation with transformer
model, BetoBERT ($\mathcal{J}$ =42.3), once again produced the best results,
and we also see that combination of approaches did not affect results notably.
Similarly, the use of parallel corpora yielded suboptimal results for both
languages. One difference is the lower relative performance of translation
with transformer models for Spanish compared to Arabic. Although it still
produces the best results compared to other approaches, its relative efficacy
is lower than we observed in the case of Arabic (80.7% as opposed to 90.9%).
We examine this in more details in the following section.

#### Combined Models:

We wanted to see if cross-lingual English training data in conjunction with
Arabic/Spanish training data would lead to improved results. To do so, we
followed the same cross-lingual approaches as in the previous subsection. The
only difference is that we trained the models using both English training data
and the Arabic or Spanish data. For combined Arabic+English models (shown in
Table 5), we observed similar trends to those observed for cross-lingual
results. Fine-tuning AraBERT on the translated English set in combination with
the Arabic set led to the best results. Combining translations with parallel
data and/or multilingual models led to lower results. While the best results
are notably better than training using just the English data, with 6.0 points
absolute improvement, the difference compared to using the Arabic training set
alone is less pronounced, with a 1.2 points absolute improvement. As for
Spanish+English models in Table 6, we see similar trends to those for Arabic.
These trends show the translation approach outperforming others and there is
little difference in effectiveness compared to training on Spanish alone.

#### Class Wise Evaluation:

Lastly, Table 7 compares the per emotion F1-measure when using monolingual,
cross-lingual, and combined models. As can be seen, in the case of Arabic, the
monolingual data produced very poor results for some emotions such as
surprise. In the case of Spanish, the cross-lingual data performed poorly for
some classes such as trust. In both cases, combined models can attenuate this
effect to an extent.

Model |  anger |  anticipation |  disgust |  fear |  joy |  love |  optimism |  pessimism |  sadness |  surprise |  trust
---|---|---|---|---|---|---|---|---|---|---|---
ar–Arabert | 77.2 | 3.6 | 49.0 | 71.9 | 79.9 | 71.8 | 71.8 | 37.0 | 70.9 | 0.0 | 4.7
en–Arabert | 67.9 | 5.9 | 50.5 | 66.1 | 74.2 | 56.8 | 67.7 | 23.9 | 69.8 | 16.0 | 10.9
ar+en–Arabert | 74.9 | 16.4 | 47.5 | 70.3 | 80.4 | 70.3 | 72.7 | 31.8 | 73.7 | 9.3 | 8.9
es–BetoBert | 74.8 | 49.5 | 46.8 | 68.6 | 80.6 | 65.4 | 39.1 | 45.1 | 65.8 | 26.3 | 28.2
en–BetoBert | 71.9 | 25.5 | 44.1 | 63.4 | 75.0 | 55.1 | 42.4 | 26.7 | 60.4 | 13.7 | 7.3
es+en–BetoBert | 73.4 | 43.4 | 47.0 | 69.8 | 80.8 | 65.9 | 44.0 | 42.6 | 66.4 | 23.7 | 30.6
Table 7: Comparison of class-level f1 Figure 1: Top error types for Arabic
Figure 2: Top error types for Spanish

## 4\. Interpreting Cross-Lingual Differences

As shown in the experiments in the previous section, the best cross-lingual
model for emotions detection were contextual embedding models that are fine-
tuned on the translated training set. In this section, we analyze how they
compare to monolingual models. To aid our analysis, we used LIME [Ribeiro et
al., 2016] to identify the words and phrases that triggered the models to
choose certain labels. LIME is an algorithm that tries to determine how
specific words affect the results of the classifier by creating variations of
an input sentence through randomly removing words. Since this is a multi-label
task, each label was treated independently, and a word would contribute
differently to each of the labels. To ascertain the effect of each word on a
label, LIME would look at all the variant sentences that contain that word,
then average the product of the weight (percentage of words used from the
input sentence) and the probability (classifier’s score for the variant
sentence for a given label). The higher the score for a word, the higher its
impact on the classifier’s prediction.

We analyzed 100 test examples with the greatest difference in Jaccard score
between ar–Arabert (trained on Arabic data) and en–Arabert (trained on
translated English data), with only one of them producing the correct labels.
Specifically, we looked at 50 examples where ar–Arabert was better than
en–Arabert and 50 for the reverse. We did the same for Spanish also with es-
BetoBERT (trained on Spanish data) and en-BetoBERT (trained on translated
English data). Given the 100 examples that we analyzed, Figures 1 and 2 show
the top error types along with their percentages for Arabic and Spanish
respectively. As the results show, the dominant error types depended greatly
on which model performed better. When ar–Arabert performed better, the most
common 3 en–Arabert error types (accounting for 80% of the errors) were:

  * •

The model missed an important word completely – most likely due to
translation. In the example in Figure 3(a), en–Arabert missed the word
corresponding to “I hate”, which was written in dialectal Egyptian.

  * •

Both models identified critical words, but weighted them differently. As in
Figure 3(b), en–Arabert and ar–Arabert indicated that the word corresponding
to “disturbing” is related to anger (correct tag), but each gave it a
different weight.

  * •

The models had two different cultural interpretation to the same word. As in
Figure 3(c), the word “terrrorist” was associated to disgust by the ar–Arabert
while being associated with fear for by en–Arabert.

Conversely, when en–Arabert performed better, the most common 3 ar–Arabert
error types (accounting for 76% of the errors) were:

  * •

Both models identified critical words, but weighted them differently. As in
Figure 4(a), ar–Arabert and en–Arabert indicated that the word corresponding
to “best” and “laughter” are related to love (incorrect tag), with ar–Arabert
giving them much higher weight.

  * •

The models produced tags that were different, but with matching sentiment
(either positive or negative). As in Figure 4(b), ar–Arabert used “sarcasm” to
assign sadness and pessimism tags, while en–Arabert used the same word to
assign anger and disgust (correct tags).

  * •

The models based their decisions on completely different words. As in Figure
4(c), ar–Arabert used “breakfast” to assign joy and en–Arabert used
“expatriation” to assign sadness (correct tag).

For Spanish, we omit details and figures due to space constraint and discuss
only the important error types. When es–BetoBert outperformed en–BetoBert, the
most prominent errors were somewhat different than those for Arabic. The top
four error types, accounting for 70% of the errors, were: the guesses of both
models were different, but matched in sentiment; the models looked at
different words; both models identified critical words, but weighted them
differently; and some words were completely misclassified. When en–BetoBert
outperformed es–BetoBert, the results were somewhat similar to those for
Arabic, where the top 3 errors, accounting for 68% of the cases, were: both
models identified critical words, but weighted them differently; the models
considered different words; and es–BetoBert completely missed some critical
words. One important difference from Arabic is that errors stemming from
cultural differences were less pronounced.  In situations where a model
completely missed a word or looked at the wrong word(s), such could
potentially stem from translation errors. For other cases where the models
weighed words differently, such could be a result of differences in culture or
language use.

(a) Word completely missed

(b) Same words weighted differently

(c) Missed cultural context

Figure 3: Examples of monolingual ar–Arabert outperforming cross-lingual
en–Arabert

(a) Same words weighted differently

(b) Same polarization but different emotions

(c) Looking at different words

Figure 4: Examples of cross-lingual en–Arabert outperforming monolingual
ar–Arabert

## 5\. Conclusion

In this paper, we investigated the efficacy of cross-lingual models in the
context of emotion detection compared to monolingual models. The effectiveness
of such models are contingent on the similarity of how emotions are expressed
across languages and cultures. We focused on using English emotion detection
training data to train models that can tag Arabic/Spanish tweets using 11
different emotions with minimal drop in effectiveness compared to monolingual
models. We presented three different cross-lingual approaches, namely using:
i) multilingual models, ii) a translated training set, and ii) an
automatically tagged parallel corpus. We experimented with these approaches
individually and in combination with a varied set of classifiers. We also
compared cross-lingual models with combined models that combine source and
target language data. Using the translated English training set with fine-
tuned contextual embeddings led to the best results for both Arabic and
Spanish. Such cross-lingual models can avail the need to annotate language
specific data, and show the transferability of emotions across languages and
cultures. We also interpreted the results of the different models to
understand why a cross-lingual model produced errors, while the monolingual
model did not (or vice a versa). Our analysis shows that while data limitation
is a challenge, translation errors and cultural difference can adversely
affect cross-lingual models.

## 6\. Bibliographical References

## References

  * Abadi et al., 2015 Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X.  (2015).  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Abdelali et al., 2021 Abdelali, A., Mubarak, H., Samih, Y., Hassan, S., and Darwish, K.  (2021).  QADI: Arabic dialect identification in the wild.  In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  * Abdul-Mageed et al., 2016 Abdul-Mageed, M., AlHuzli, H., and Elhija, M. D. D.  (2016).  Dina: A multidialect dataset for arabic emotion analysis.  In The 2nd workshop on Arabic corpora and processing tools, page 29.
  * Abdullah and Shaikh, 2018 Abdullah, M. and Shaikh, S.  (2018).  Teamuncc at semeval-2018 task 1: Emotion detection in english and arabic tweets using deep learning.  In Proceedings of the 12th international workshop on semantic evaluation, pages 350–357.
  * Abdullah et al., 2018 Abdullah, M., Hadzikadicy, M., and Shaikhz, S.  (2018).  Sedat: Sentiment and emotion detection in arabic text using cnn-lstm deep learning.  In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 835–840.
  * Abu Farha and Magdy, 2019 Abu Farha, I. and Magdy, W.  (2019).  Mazajak: An online Arabic sentiment analyser.  In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics.
  * Ahmad et al., 2020 Ahmad, Z., Jindal, R., Ekbal, A., and Bhattachharyya, P.  (2020).  Borrow from rich cousin: transfer learning for emotion detection using cross lingual embedding.  Expert Systems with Applications, 139:112851.
  * Al-Khatib and El-Beltagy, 2017 Al-Khatib, A. and El-Beltagy, S. R.  (2017).  Emotional tone detection in arabic tweets.  In International Conference on Computational Linguistics and Intelligent Text Processing, pages 105–114. Springer.
  * Alhuzali et al., 2018 Alhuzali, H., Abdul-Mageed, M., and Ungar, L.  (2018).  Enabling deep learning of emotion with first-person seed expressions.  In Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, pages 25–35.
  * Antoun et al., 2020 Antoun, W., Baly, F., and Hajj, H.  (2020).  Arabert: Transformer-based model for arabic language understanding.  In Proceedings of The 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 9–15.
  * Badarneh et al., 2018 Badarneh, O., Al-Ayyoub, M., Alhindawi, N., Jararweh, Y., et al.  (2018).  Fine-grained emotion analysis of arabic tweets: A multi-target multi-label approach.  In 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pages 340–345. IEEE.
  * Bel et al., 2003 Bel, N., Koster, C. H. A., and Villegas, M.  (2003).  Cross-lingual text categorization.  In Traugott Koch et al., editors, Research and Advanced Technology for Digital Libraries, pages 126–139, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Cardellino, 2019 Cardellino, C.  (2019).  Spanish Billion Words Corpus and Embeddings.  August.
  * Cañete et al., 2020 Cañete, J., Chaperon, G., Fuentes, R., and Pérez, J.  (2020).  Spanish pre-trained bert model and evaluation data.  In to appear in PML4DC at ICLR 2020.
  * Cer et al., 2018 Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al.  (2018).  Universal sentence encoder.  arXiv preprint arXiv:1803.11175.
  * Chin et al., 2014 Chin, J., Heymans, M., Kojoukhov, A., Lin, J., and Tan, H.  (2014).  Cross-language information retrieval, August 5.  US Patent 8,799,307.
  * Chollet and others, 2015 Chollet, F. et al.  (2015).  Keras.  https://keras.io.
  * Darwish, 2013 Darwish, K.  (2013).  Named entity recognition using cross-lingual resources: Arabic as an example.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567, Sofia, Bulgaria, August. Association for Computational Linguistics.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Dong and de Melo, 2019 Dong, X. and de Melo, G.  (2019).  A robust self-learning framework for cross-lingual text classification.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6306–6310, Hong Kong, China, November. Association for Computational Linguistics.
  * Ekman, 1999 Ekman, P.  (1999).  Basic emotions.  Handbook of cognition and emotion, 98(45-60):16.
  * Elmadany et al., 2018 Elmadany, A., Mubarak, H., and Magdy, W.  (2018).  Arsas: An arabic speech-act and sentiment corpus of tweets.  OSACT, 3:20.
  * Fei et al., 2020 Fei, H., Zhang, M., and Ji, D.  (2020).  Cross-lingual semantic role labeling with high-quality translated training corpus.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7014–7026, Online, July. Association for Computational Linguistics.
  * Forman, 2008 Forman, G.  (2008).  Bns feature scaling: an improved representation over tf-idf for svm text classification.  In Proceedings of the 17th ACM conference on Information and knowledge management, pages 263–270.
  * Guo et al., 2015 Guo, J., Che, W., Yarowsky, D., Wang, H., and Liu, T.  (2015).  Cross-lingual dependency parsing based on distributed representations.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1234–1244.
  * Hareli et al., 2015 Hareli, S., Kafetsios, K., and Hess, U.  (2015).  A cross-cultural study on emotion expression and the learning of social norms.  Frontiers in psychology, 6:1501.
  * Hassan et al., 2020a Hassan, S., Samih, Y., Mubarak, H., and Abdelali, A.  (2020a).  ALT at SemEval-2020 task 12: Arabic and English offensive language identification in social media.  In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1891–1897, Barcelona (online), December. International Committee for Computational Linguistics.
  * Hassan et al., 2020b Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., and Chowdhury, S. A.  (2020b).  ALT submission for OSACT shared task on offensive language detection.  In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 61–65, Marseille, France, May. European Language Resource Association.
  * Hassan et al., 2021 Hassan, S., Mubarak, H., Abdelali, A., and Darwish, K.  (2021).  ASAD: Arabic social media analytics and unDerstanding.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 113–118, Online, April. Association for Computational Linguistics.
  * Howard and Ruder, 2018 Howard, J. and Ruder, S.  (2018).  Universal language model fine-tuning for text classification.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, July. Association for Computational Linguistics.
  * Hussien et al., 2016 Hussien, W. A., Tashtoush, Y. M., Al-Ayyoub, M., and Al-Kabi, M. N.  (2016).  Are emoticons good enough to train emotion classifiers of arabic tweets?  In 2016 7th International Conference on Computer Science and Information Technology (CSIT), pages 1–6. IEEE.
  * Jebbara and Cimiano, 2019 Jebbara, S. and Cimiano, P.  (2019).  Zero-shot cross-lingual opinion target extraction.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2486–2495, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Klinger and Cimiano, 2015 Klinger, R. and Cimiano, P.  (2015).  Instance selection improves cross-lingual model training for fine-grained sentiment analysis.  In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 153–163, Beijing, China, July. Association for Computational Linguistics.
  * Mcnamee and Mayfield, 2004 Mcnamee, P. and Mayfield, J.  (2004).  Character n-gram tokenization for european language text retrieval.  Information retrieval, 7(1-2):73–97.
  * Mihalcea et al., 2007 Mihalcea, R., Banea, C., and Wiebe, J.  (2007).  Learning multilingual subjective language via cross-lingual projections.  In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 976–983, Prague, Czech Republic, June. Association for Computational Linguistics.
  * Miranda and Guzmán, 2017 Miranda, C. H. and Guzmán, J.  (2017).  A Review of Sentiment Analysis in Spanish.  Tecciencia, 12:35 – 48, 06.
  * Mohammad et al., 2018a Mohammad, S., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S.  (2018a).  Semeval-2018 task 1: Affect in tweets.  In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  * Mohammad et al., 2018b Mohammad, S., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S.  (2018b).  SemEval-2018 task 1: Affect in tweets.  In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * Mourad and Darwish, 2013 Mourad, A. and Darwish, K.  (2013).  Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs.  In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 55–64, Atlanta, Georgia, June. Association for Computational Linguistics.
  * Mubarak and Hassan, 2021 Mubarak, H. and Hassan, S.  (2021).  UL2C: Mapping user locations to countries on Arabic Twitter.  In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 145–153, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  * Mubarak et al., 2020a Mubarak, H., Abdelali, A., Hassan, S., and Darwish, K.  (2020a).  Spam detection on arabic twitter.  In Samin Aref, et al., editors, Social Informatics, pages 237–251, Cham. Springer International Publishing.
  * Mubarak et al., 2020b Mubarak, H., Hassan, S., and Abdelali, A.  (2020b).  Constructing a bilingual corpus of parallel tweets.  In Proceedings of the 13th Workshop on Building and Using Comparable Corpora, pages 14–21, Marseille, France, May. European Language Resources Association.
  * Mubarak et al., 2020c Mubarak, H., Rashed, A., Darwish, K., Samih, Y., and Abdelali, A.  (2020c).  Arabic offensive language on twitter: Analysis and experiments.  arXiv preprint arXiv:2004.02192.
  * Ortony and Turner, 1990 Ortony, A. and Turner, T. J.  (1990).  What’s basic about basic emotions?  Psychological review, 97(3):315.
  * Pamungkas and Patti, 2019 Pamungkas, E. W. and Patti, V.  (2019).  Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 363–370.
  * Plaza del Arco et al., 2020a Plaza del Arco, F. M., Martín-Valdivia, M. T., Ureña-López, L. A., and Mitkov, R.  (2020a).  Improved emotion recognition in spanish social media through incorporation of lexical knowledge.  Future Generation Computer Systems, 110:1000 – 1008.
  * Plaza del Arco et al., 2020b Plaza del Arco, F. M., Strapparava, C., Urena Lopez, L. A., and Martin, M.  (2020b).  EmoEvent: A multilingual emotion corpus based on different events.  In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1492–1498, Marseille, France, May. European Language Resources Association.
  * Plutchik, 1991 Plutchik, R.  (1991).  The emotions.  University Press of America.
  * Radford et al., 2018 Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  (2018).  Improving language understanding by generative pre-training.  URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf.
  * Ren et al., 2018 Ren, H., Wan, J., and Ren, Y.  (2018).  Emotion detection in cross-lingual text based on bidirectional lstm.  In International Conference on Security with Intelligent Computing and Big-data Services, pages 838–845. Springer.
  * Ribeiro et al., 2016 Ribeiro, M. T., Singh, S., and Guestrin, C.  (2016).  ”why should I trust you?”: Explaining the predictions of any classifier.  CoRR, abs/1602.04938.
  * Scarantino and Griffiths, 2011 Scarantino, A. and Griffiths, P.  (2011).  Don’t give up on basic emotions.  Emotion Review, 3(4):444–454.
  * Segura-Bedmar et al., 2017 Segura-Bedmar, I., Quirós, A., and Martínez, P.  (2017).  Exploring convolutional neural networks for sentiment analysis of Spanish tweets.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1014–1022, Valencia, Spain, April. Association for Computational Linguistics.
  * Tiedemann, 2012 Tiedemann, J.  (2012).  Parallel data, tools and interfaces in opus.  In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, may. European Language Resources Association (ELRA).
  * Wan, 2009 Wan, X.  (2009).  Co-training for cross-lingual sentiment classification.  In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 235–243, Suntec, Singapore, August. Association for Computational Linguistics.
  * Wang et al., 2020 Wang, Z., Ho, S.-B., and Cambria, E.  (2020).  A review of emotion sensing: Categorization models and algorithms.  Multimedia Tools and Applications, pages 1–30.
  * Xu and Yang, 2017 Xu, R. and Yang, Y.  (2017).  Cross-lingual distillation for text classification.  CoRR, abs/1705.02073.
  * Zhang et al., 2019 Zhang, M., Zhang, Y., and Fu, G.  (2019).  Cross-lingual dependency parsing using code-mixed TreeBank.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 997–1006, Hong Kong, China, November. Association for Computational Linguistics.
  * Zhou et al., 2016 Zhou, X., Wan, X., and Xiao, J.  (2016).  Attention-based lstm network for cross-lingual sentiment classification.  In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 247–256.


D Empirical Software Engineering Syful Islamnnaist Dong Wangnnaist Raula
Gaikovina Kulannaist Takashi Ishiomnaist1301152 Kenichi MATSUMOTOfnaist8925358
[naist]The authors are with the Graduate School of Science and Technology,
Nara Institute of Science and Technology, Ikoma-shi, 630-0192, Japan. 0226 714

# An Exploration of npm Package Co-Usage Examples from Stack Overflow: A Case
Study

(2021; 2021)

###### keywords:

Package managers, npm, Stack Overflow

Third-party package usage has become a common practice in contemporary
software development. Developers often face different challenges, including
choosing the right libraries, installing errors, discrepancies, setting up the
environment, and building failures during software development. The risks of
maintaining a third-party package are well known, but it is unclear how
information from Stack Overflow (SO) can be useful. This paper performed an
empirical study to explore npm package co-usage examples from SO. From over
30,000 SO question posts, we extracted 2,100 posts with package usage
information and matched them against the 217,934 npm library package. We find
that, popular and highly used libraries are not discussed as often in SO.
However, we can see that the accepted answers may prove useful, as we believe
that the usage examples and executable commands could be reused for tool
support.

##  1 Introduction

Usage of third-party packages in contemporary software development has become
a common practice by developers. For example, npm (i.e., Node.js package
manager) is by far the largest package manager, allowing developers to reuse
functionality instead of creating their own, saving both time with minimal
efforts. The npm ecosystem has provided over 800,000 free and reusable
software libraries and is trusted by over 11 million developers around the
world.111https://www.npmjs.com/

Despite these benefits of using packages, developers constantly face a variety
of issues when using them. Dietrich et al. [1] performed a case study and
showed that partial package upgrades have high potential to introduce binary
incompatibility problems at build time. Raemaekers et al. [2, 3] pointed out
that developers are wary of the inherent costs and risks of package
incompatibilities when integrating new and unknown packages into their
systems. Most prior work have explored package usage [4, 5], developing
package recommendation tools [6, 7, 8, 9, 10, 11].

Previous studies reported that question-answering websites such as Stack
Overflow (SO) are useful for communicating developers’ issues. Several studies
have been conducted on SO resources including source code utilization [12],
analogical libraries recommendation [13], fixing run-time exception [14],
improving API documentation [15], API usage scenarios [16] and so forth. Other
studies have focused on more interviews and surveys of developers [17, 18]. We
hypothesize that the library usage information from SO may also be beneficial
for developers. While the risks of maintaining third-party libraries are well
known, it is still unclear that whether the library usage information mined
from question-answering sites are useful or not in maintaining libraries.

To fill this gap, in this paper, we perform an exploratory study on package
usage information from SO in term of co-usage relationship. As defined by
Todorov et. al [19], co-usage is the pattern of package dependencies that are
used together. The rationale behind refining the co-usage relationship is to
study problems caused by npm packages. In particular we investigate (i)
whether we can detect package usage (i.e., co-usage) information from SO and
(ii) what the developers are looking for to solve problems related to the
package. To address these, we study over 2,100 SO posts and matched them to
217,934 npm library packages. We reveal the following valuable lessons along
the way:

Lesson 1: We find that only three out of the top ten of the most used npm
libraries are mentioned in SO. The top-3 discussed npm packages are react,
typescript, and webpack. Again, the top-5 libraries that are less frequently
discussed in SO are mocha, eslint, chai, babel-core, and lodash. One possible
reason is that, well-known libraries are well documented and may have their
own forum, chat tools, etc. For this reason, there is no need to discuss them
in SO. Furthermore, we find that 87.95% of package co-usage mined from SO
exist in the latest npm package release.

Lesson 2: Developers post answers provided with usage example or execute
command. Results do indicate the potential for a recommendation system,
especially with the available execute commands and examples.

Although SO has been a useful resource for finding answers to questions, we
find that popular and highly used libraries are not discussed as often.
However, we can see that the accepted answers may prove useful, as we believe
that the usage examples and executable commands could be reused for tool
support.

The remainder of the paper is organized as follows. Section 2 presents
motivating example and research questions. Section 3 describes the data
preparation. Section 4 presents the analysis approach. Section 5 reports the
results for each research question. Section 6, discusses the implications from
this study. Section 7, presents the related works and the research gap.
Section 8 discloses the threats to validity of our study. Finally, we conclude
the paper in section 9 .

Figure 1: A motivating example of npm package co-usage in Stack Overflow. The
example shows that a developer encounters a issue when installing all
node_modules, due to two dependent packages babel-loader and
webpack.222https://stackoverflow.com/questions/46742824 Figure 2: An overview
of the methodology of our study.

##  2 Motivating Example

Recent studies point out that SO is a useful question-answering site among
developers to communicate various issues [13, 14, 15, 20, 16]. In this paper,
our motivation is to investigate the following assumptions:

  * •

Package usage information mined from SO is useful to solve developers issues
while using libraries.

  * •

Developer responses to package usage information in SO follow some useful
patterns that might be reused by the recommendation tools.

Figure 2 shows an example of a package co-usage related question post from
SO.222https://stackoverflow.com/questions/46742824 As shown in the figure, a
developer posts a question on the error issue of node_module installation,
resulting from two dependent packages babel-loader and webpack. A closer look
into the question description, we observe that the successful installation of
<EMAIL_ADDRESS>requires the package dependency of webpack@2||3. This issue
is solved by a simple installation command (i.e., npm install webpack -g)
mentioned in the accepted answer of the question. Such a motivating example
suggests that package usage information mined from question answering sites
may help solve package-related issues.

Research Questions: Our goal in this paper is to investigate whether package
usage information mined from SO can help maintain the packages. We formulate
two research questions to guide our study.

  * •

RQ1: What proportion of package usage information mined from Stack Overflow
exist in npm packages?
Motivation. Developers often share package usage information to communicate
various software development issues through SO. We would like to understand
what is the difference in the package usage information between SO and npm
projects.

  * •

RQ2: What kinds of answers are posted in response to questions that include
package usage information?
Motivation. This research question investigates the developer’s response to
package usage information discussed in SO posts. We argue that a closer look
at the answers may reveal practical insights to improve real developers’
experience dealing with package co-usage issues.

##  3 Data Preparation

Table 1: Summary of dataset used in the study.

Data Source | Initial dataset | Final dataset
---|---|---
D1: SO (npm question posts) | 30,136 | 2,100
D2: libraries.io (npm projects) | 100,5955 | 217,934

Our study exclusively examines the npm package usage information from Stack
Overflow. Stack Overflow is the largest and most popular question-answering
site among developers, which allows them to ask developers and experts for
development related questions. In addition, to compare with the package usage
from the reality, we collect another dataset from the
libraries.io.333https://libraries.io/ libraries.io is popularly known to
monitor package releases. Below, we describe the details of two studied
datasets.

(D1) from Stack Overflow posts: We rely on the SOTorrent [21] to download the
Stack Overflow data dump. We collect npm related question posts using a semi-
automatic method. To do so, we first extract all tags from the question posts,
and then we manually check whether or not the tags are directly related to the
npm packages. After the examination, a list of eight tags that reflect npm
packages posts are generated, i.e., ‘npm’, ‘npm-install’, ‘npm-script’, ‘npm-
ignore’, ‘pnpm’, ‘npm-shrinkwrap’, ‘npm-start’, ‘npm-build’. We automatically
identify all question posts using the defined tag list, resulting in 30,136
questions related to npm packages.

Next we further extract the npm related questions that contain the package
usage information. We observed that several package names are as common as the
natural language, i.e., i, moment, should, express, etc.). Thus, to reduce the
bias, we only take those question posts that contain package.json files,
resulting in 2,805 question posts. As we focus on the relatively popular
libraries, we extract all packages from these question posts and sort out 628
npm packages whose frequency are greater than ten.

To ensure that our sample dataset contains most npm libraries, we use the
cumulative sampling method to assure that our question posts are saturated to
cover all 628 npm packages. Finally, our Stack Overflow npm package usage
dataset consists of 2,100 question posts, as shown in Table 1.

(D2) from npm packages: To compare the package usage information with SO ones,
we construct a dataset consisting of npm packages from libraries.io. To do so,
we first downloaded the latest history data dump, which is available at
https://libraries.io/data, resulting in a total number of 1,005,955 npm
project release history.

Similar to the (D1), we extract the libraries from these 1,005,955 projects
and sort out 23,870 npm packages whose frequency are greater than ten. In the
end, our libraries.io npm package usage dataset consists of 217,934 npm
projects using the cumulative sampling method, as shown in Table 1.

##  4 Data Analysis

In this section, as shown in Fig. 2, we describe in detail the approaches we
use to answer two research questions.

###  4.1  Approach for RQ1

We perform an exploratory study to understand to which extents do developers
discuss the package usage information from SO. Below, we describe our approach
in detail.

Proportion of popular package usage: To analyze the proportion of package
usage in SO and npm projects, we extracted libraries from SO posts obtained in
datasets D1 and D2, separately. Afterward, we count and sort these packages
based on the frequency.

Co-usage information related to npm package: To analyze the frequency of
package usage information, we first need to identify npm package co-usage. To
do so, we follow the two steps: (I) we extract all the target packages
appearing in the code snippets from 2,100 SO post related to npm package
obtained in Section 3 (D1). Then we generate all possible binary combinations
for co-usage of npm packages. For example, if a project contain three package
dependencies ($A$, $B$, $C$), then the generated list of binary package co-
usage will be: ($A$, $B$), ($A$, $C$), ($B$, $C$). After this step, we were
able to retrieve 68,750 npm package co-usage information from SO. (II) we then
extract the npm package co-usage information based on the latest release,
referring to 217,934 npm projects in Section 3 (D2). Finally, we check the
occurrences of SO npm package co-usage information in the generated package
co-usage from latest npm projects using the following formula:
$\frac{\alpha}{\beta}\times 100$ where $\alpha$=Number of SO npm package co-
usage found in the latest npm project release, and $\beta$=Total npm package
co-usage extracted from SO.

In addition, to understand the issues raised by package related question, we
extract the title information from 2,100 Stack Overflow posts obtained in D1
and automatically extracted the keywords using traditional Nature Language
Processing (NLP) , including stop word removals. The output is a corpus of
title keywords.

Figure 3: Word cloud generated from SO npm posts title that contains package
usage information. The word cloud shows that npm posts are primarily related
to various types of errors.

To visually understand the package related issues asked in the SO, we generate
a word cloud based on the constructed title corpus.

Observation 1- The npm post that discuss package usage information are mostly
relate to different type of errors. Figure. 3 shows the Word cloud based on SO
posts titles. The word cloud shows that npm posts regarding package usage
information are primarily related to various types of errors like installation
error, build failure, etc.

###  4.2  Approach for RQ2

We conduct a qualitative analysis to investigate the accepted answer post from
SO. We analyze the accepted answer since these answers are solutions that work
for developers.444https://stackoverflow.com/tour Below we describe the
representative sample construction and the manual coding process.

Representative sample construction: As the full set of our constructed data is
too large to manually examine their accepted answers, we then draw a
statistically representative sample. The representative sample consists of 286
randomly selected accepted answer, with a confidence level of 95% and a
interval of 5. 555https://www.surveysystem.com/sscalc.htm

Manual coding: We adopt three rounds to do our manual coding. First, the three
authors independently sampled 25 random questions and conducted an open
discussion to establish an initial code taxonomy. Hence, the following three
codes emerged from our manual analysis in the first round.

  * •

Execute command: The accepted answer explicitly mentions executing commands.
In definition, execute commands describe the process of running a computer
software program, script, or command. As shown in Fig. 4, npm install -g
@angular/cli is an execute command to install ‘angular/cli’.

  * •

Step by step Instruction: The accepted answer contains step by step
information to get the solution. In definition, instructions are detailed
(i.e., step by step) information about how something should be done or
operated. As shown in Fig. 4, the accepted answer contains three distinct step
to solve the library usage issue (i.e., Angular CLI installation, generating
and serving an Angular project, and open local host page in the browser.)

  * •

Usage Example: The accepted answer explicitly mentions examples, referred to
external links, source code, configuration files, etc. In definition, usage
examples are defined as models, or typical cases (like external links, source
code, etc.) used to solve a problem. As shown in Fig. 4, the external link
mention in the beginning of the answer is usage example.

In the second round, to assure that there is no new cases, the three authors
coded another 25 samples and we found that the initialized codes can totally
fit. In the third round, to test the comprehension understanding, we coded
another 20 samples and used the Kappa score to measure the agreement. The
score is 0.82, which is implied as nearly perfect [22]. Based on this
encouraging result, the first author then coded the rest of the representative
sample independently.

Figure 4: An example that motivates to classify developers response. In the
answer we observe that, it contains usage examples, execute command, and step
by step instruction.

##  5 Results

In this section, we provide the results for each research question. First, we
describe the result analysis, and then highlight our findings and answer each
question.

###  5.1 Answering RQ1

To show the proportion of popular package usage, we depict tables to
statistically show the package usage between SO and npm projects. Then, to
analyze the frequency of So npm package co-usage in the latest npm projects,
we calculated the ratio using formula (i.e., $\frac{\alpha}{\beta}\times 100$
) discussed in the approach.

Table 2: Top-15 npm packages extracted from SO posts with their proportion and
rank in the latest npm projects. Result shows that Only three out of top-10
npm packages are mostly discussed in SO.

npm packages |  | Count
---
(SO)
| Count
---
(npm
projects)
| Rank
---
(SO)
| Rank
---
(npm
projects)
react | 586 | 42, 591 | 1 | 9
typescript | 548 | 57,864 | 2 | 4
webpack | 489 | 52,453 | 3 | 5
rxjs | 471 | 12,339 | 4 | 69
zone.js | 462 | 8,570 | 5 | 119
react-dom | 461 | 32,941 | 6 | 16
@angular/core | 434 | 10,050 | 7 | 95
@angular/common | 434 | 9,406 | 8 | 103
@angular/compiler | 426 | 9,170 | 9 | 110
@angular/platform-browser | 424 | 8,482 | 10 | 120
@angular/platform-browser-dynamic | 419 | 7,634 | 11 | 132
jquery | 413 | 9,263 | 12 | 108
@angular/forms | 401 | 6,608 | 13 | 147
@angular/http | 388 | 5,433 | 14 | 169
@angular/router | 380 | 5,583 | 15 | 166

Table 3: Top-15 package usage extracted from the latest npm projects with
their proportion and rank in the SO posts. The top package usage patterns from
npm projects shows that application developers top usage packages are
different from SO.

npm package |  | Count
---
(npm projects)
| Count
---
(SO)
| Rank
---
(npm projects)
| Rank
---
(SO)
mocha | 101898 | 126 | 1 | 65
eslint | 81767 | 199 | 2 | 45
chai | 58368 | 114 | 3 | 78
typescript | 57864 | 548 | 4 | 2
webpack | 52453 | 489 | 5 | 3
babel-core | 51351 | 309 | 6 | 26
lodash | 45618 | 348 | 7 | 19
babel-loader | 44398 | 340 | 8 | 22
react | 42591 | 586 | 9 | 1
jest | 41188 | 115 | 10 | 76
babel-eslint | 39336 | 134 | 11 | 68
babel-cli | 38038 | 102 | 12 | 83
eslint-plugin-import | 36019 | 94 | 13 | 90
@types/node | 35197 | 320 | 14 | 24
rimraf | 34466 | 139 | 15 | 63

Table 4: Top-15 package co-usage extracted from SO posts except angular since
rest of the top co-usage are related to angular. The top co-usage patterns and
their rank in SO indicate that developers discuss most package dependency
issues related to angular followed by (‘typescript’, ‘zone.js’).

Package Co-usage | Rank | Count
---|---|---
(‘typescript’, ‘zone.js’) | 9 | 317
(‘zone.js’, ‘rxjs’) | 15 | 290
(‘react-dom’, ‘react’) | 17 | 288
(‘typescript’, ‘rxjs’) | 21 | 283
(‘karma’, ‘karma-jasmine’) | 31 | 258
(‘zone.js’, ‘core-js’) | 33 | 251
(‘core-js’, ‘rxjs’) | 39 | 233
(‘webpack’, ‘babel-loader’) | 40 | 230
(‘typescript’, ‘core-js’) | 40 | 230
(‘jasmine-core’, ‘karma-jasmine’) | 43 | 225
(‘karma-jasmine’, ‘karma-chrome-launcher’) | 44 | 223
(‘babel-core’, ‘babel-loader’) | 45 | 220
(‘typescript’, ‘tslint’) | 46 | 216
(‘typescript’, ‘karma’) | 47 | 210
(‘typescript’, ‘karma-jasmine’) | 48 | 209

Observation 2- Only three out of top-10 npm packages are mostly discussed in
SO. Table 2 shows the top-15 packages discussed in SO with their proportion
and ranks in the latest npm projects. The top-3 discussed npm packages are
react, typescript, and webpack. Again, Table 3 shows the top-15 package usage
extracted from the latest npm projects. We observe that, the top-5 packages in
the latest npm projects which are less frequently discussed in SO are mocha,
eslint, chai, babel-core, and lodash. One possible reason is that, such well-
known libraries are well documented and may have their own forum, chat tools,
etc. For this reason, there is no need to discuss them in SO.

Observation 3- 87.95% of the SO package co-usage information exist in the
latest npm project release. Furthermore, Table 4 shows the top-15 SO package
co-usage mentioned by developers. We observed that most of the package co-
usage mentioned by developers are related to angular followed by
(‘typescript’, ‘zone.js’). The top co-usage patterns from SO and their rank
hints that developers face most error type issues when they use angular
packages.

RQ1 Summary: Our analysis result shows that, only three out of top-10 npm
packages are mostly discussed in SO. In addition, 87.95% of the SO npm package
co-usage information exist in the latest npm project release.

###  5.2 Answering RQ2

To show the useful accepted answer attributes pattern in response to the
package usage question by developers, we prepare all possible combinations for
three manually curated attributes: Execute command, Step by step instruction,
and Usage example, respectively. Thus, we obtain eight distinct combinations
(i.e., subsets), including the others (i.e., empty set). Finally, we calculate
the percentage of each variety in the manually analyzed representative sample.

Observation 4- Our results show that, accepted answers posted by developers in
response to questions that include package usage information mostly contain
usage examples (i.e., includes real-life examples, external links, source
code, build configuration files, etc.). Figure 5 shows the analysis result of
accepted answers posted by developers in response to questions that include
package usage information. We observe that usage example (36.76%) is most
dominant attribute in accepted answer, followed by execute command (19.58%).
The third most frequent (15.03%) attribute combination in the accepted answer
is execute command and usage example. These findings hint that application
developers are suffering from a lack of technical depth in managing third-
party libraries in their applications.

Figure 5: Analysis of accepted answers posted in response to questions that
include package usage information. Result shows that 37.76% accepted answers
contain usage example followed by execute command 19.58%.

RQ2 Summary: Result shows that 37.76% accepted answers posted by developers in
response to questions that include package usage information contain usage
example followed by execute command 19.58%.

##  6 Discussion

In order to aid application developers faced with package usage issues, we
conducted an empirical study to understand the usefulness of package usage
information mined from SO. We learned two lessons along the way:

Lesson 1- Many of the library usage information on SO is not from the popular
npm package. We find that only three out of the top ten of the most used npm
libraries are mentioned in SO. The top-3 discussed npm packages are react,
typescript, and webpack. Again, the top-5 libraries that are less frequently
discussed in SO are mocha, eslint, chai, babel-core, and lodash. Furthermore,
we find that 87.95% package co-usage mined from SO exist in the latest npm
package release. The npm post that discuss package usage information are
mostly relate to different type of errors.

Lesson 2- Developers tend to post answers that are usage example or execute
command. The good news is that maybe the answers can be useful for any npm
developer that suffers from similar issues. There is potential for a
recommendation system, especially with the available execute commands and
examples available.

##  7 Related Work

In this section, we discuss the related works. First, we discuss on third-
party package usage issues faced by developers. Second, we discuss on mining
useful information from question-answering sites.

Third-party libraries usage issues: software package is a reusable component
developed by a body other than the original vendor of the development
platform. The usage of third-party libraries provides developers with unique
opportunity to integrate pre-tested, reusable software that saves development
time and cost.666https://tinyurl.com/y5b2fajq Recent empirical studies have
found that 93.30% of modern OSS project use third-party libraries, with an
average of 28 libraries per project [6].

In recent years, analyzing the characteristics of software ecosystem has
gained much attention. Decan et al. [23] investigated package dependency
network for seven software ecosystems. Their findings reveals that, software
ecosystems grew over time in term of number of published libraries. Bogart et
al. [24] investigate the challenges of reusing libraries from software
ecosystem. They reported that developers struggle with changing versions of
the libraries as the changes might potentially break dependent codes. Bavota
et al. [25] examine the evolution of dependencies in Apache ecosystem and
found that developers were reluctant to upgrade their dependencies considering
that changes of a package might break its dependent libraries. Xavier et al.
[26] performed a large scale study on 317 real-life Java libraries with 9K
releases and 260K projects. Their analysis results show that 14.78% of API
changes are incompatible with previous versions. Kula et al. [27] also
reported that, developers do not update their dependencies even though the
updates are related to new features, fix vulnerabilities. Several research was
done on package recommendation tools like LibRec [6], LibCup [7], CrossRec
[11]. In this research, we empirically investigate usefulness of npm package
usage information mined from question-answering site.

Mining SO: Recent studies point out that SO is a useful source for developers
to meet their information needs. For instance, Chen et al. [13] reported that
SO is useful in recommending analogical libraries across different programming
languages. Mahajan et al. [14] proposed a recommendation tool to fix Run-time
Exceptions based on knowledge from SO posts. Similarly, Treude et al. [15],
Rubei et al. [20], and Uddin et al. [16] showed that SO posts are useful
knowledge source to support software developers. Previous studies suggest that
SO can be useful to solve package usage related issues. There is no existing
work that studies package usage information mined from SO help to improve
developers’ experience.

##  8 Threats to validity

In this section, we discuss threats to validity that might influence our
study.

Internal Validity: Threats to internal validity refer to experimental bias. In
this study, we found two main internal threads that could affect our results.
First, is the pre-processing of the dataset we decide the number of posts
(2100) and npm packages (217,934) based on cumulative extraction of npm
libraries and the generated co-usages. We continue the cumulative extraction
until all the libraries and the co-usage cover. Second, in RQ2 we perform
manual analysis on random sample since the dataset size is large. To mitigate
this challenge, we prepare representative sample consists of 286 randomly
selected accepted answer, with a confidence level of 95% and a interval of 5.

External validity: Threats to external validity refer to the generalizability
of our findings. Our datasets consist of npm packages from libraries.io and SO
posts. SO is a popular platform for question and answers from developers with
various domains and experts. Hence, our observations and results can not be
generalized for other package managers like Maven, NuGet, and others. Besides,
we consider only those SO posts that contain package.json file. Selecting more
question posts may cause variation of top package co-usage results.

Construct validity: Threats to construct validity refers to the suitability of
our evaluation measure. In our qualitative analysis of classifying accepted
answers (RQ2), the answer patterns may be miscoded due to the subjective
nature of our coding approach. To mitigate this threat, we took a systematic
approach to validate the taxonomy and the comprehension understanding by the
three authors in several rounds. Only until the Kappa score reaches 0.82,
indicating that the agreement is almost perfect (0.81-1.00), we were able to
complete the rest of the sample dataset.

##  9 Conclusion

In this paper, we examine the usefulness of package usage information mined
from SO. We perform a case study on npm package co-usage information from SO
question posts (2100) and libraries.io (217,934 npm projects) dataset.
Although SO has been a useful resource for finding answers to questions, we
find that unfortunately popular and highly used libraries are not discussed as
often. However, we can see that the accepted answers may prove useful, as we
believe that the usage examples and executable commands could be reused or be
used for tool support. In our future work, we will develop tool support that
will utilize SO usage examples and executable commands extracted from accepted
answers to assist npm application developers.

## Acknowledgment

This work has been supported by JSPS KAKENHI Grant Numbers JP8H04094,
JP20K19774, and JP20H05706.

## References

  * [1] J. Dietrich, K. Jezek, and P. Brada, “Broken promises: An empirical study into evolution problems in java programs caused by library upgrades,” 2014 Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), pp.64–73, IEEE, 2014.
  * [2] S. Raemaekers, A. van Deursen, and J. Visser, “Semantic versioning and impact of breaking changes in the maven repository,” Journal of Systems and Software, vol.129, pp.140–158, 2017.
  * [3] S. Raemaekers, A. Van Deursen, and J. Visser, “Semantic versioning versus breaking changes: A study of the maven repository,” 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation, pp.215–224, IEEE, 2014.
  * [4] F.L. De La Mora and S. Nadi, “Which library should i use?: a metric-based comparison of software libraries,” 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE- NIER), pp.37–40, IEEE, 2018.
  * [5] J. Dietrich, D. Pearce, J. Stringer, A. Tahir, and K. Blincoe, “Dependency versioning in the wild,” 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp.349– 359, IEEE, 2019.
  * [6] F. Thung, D. Lo, and J. Lawall, “Automated library recommendation,” 2013 20th Working conference on reverse engineering (WCRE), pp.182– 191, IEEE, 2013.
  * [7] A. Ouni, R.G. Kula, M. Kessentini, T. Ishio, D.M. German, and K. Inoue, “Search-based software library recommendation using multi-objective optimization,” Information and Software Technology, vol.83, pp.55–75, 2017.
  * [8] M.A. Saied, A. Ouni, H. Sahraoui, R.G. Kula, K. Inoue, and D. Lo, “Improving reusability of software libraries through usage pattern mining,” Journal of Systems and Software, vol.145, pp.164– 179, 2018.
  * [9] H. Alrubaye, M.W. Mkaouer, I. Khokhlov, L. Reznik, A. Ouni, and J. Mcgoff, “Learning to recommend third-party library migration opportunities at the api level,” Applied Soft Computing, vol.90, p.106140, 2020.
  * [10] H. Yu, X. Xia, X. Zhao, and W. Qiu, “Combining collaborative filtering and topic modeling for more accurate android mobile app library recommendation,” Proceedings of the 9th Asia-Pacific Symposium on Internetware, pp.1–6, 2017.
  * [11] P.T. Nguyen, J. Di Rocco, D. Di Ruscio, and M. Di Penta, “Crossrec: Supporting software developers by recommending third-party libraries,” Journal of Systems and Software, vol.161, p.110460, 2020.
  * [12] Y. Wu, S. Wang, C.P. Bezemer, and K. Inoue, “How do developers utilize source code from stack overflow?,” Empirical Software Engineering, vol.24, no.2, pp.637–673, 2019.
  * [13] C. Chen and Z. Xing, “Similartech: automatically recommend analogical libraries across different programming languages,” Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pp.834–839, 2016.
  * [14] S. Mahajan, N. Abolhassani, and M.R. Prasad, “Recommending stack overflow posts for fixing runtime exceptions using failure scenario matching,” Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp.1052–1064, 2020.
  * [15] C. Treude and M.P. Robillard, “Augmenting api documentation with insights from stack overflow,” 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp.392–403, IEEE, 2016.
  * [16] G. Uddin, F. Khomh, and C.K. Roy, “Mining api usage scenarios from stack overflow,” Information and Software Technology, vol.122, p.106277, 2020.
  * [17] E. Larios Vargas, M. Aniche, C. Treude, M. Bruntink, and G. Gousios, “Selecting Third-Party Libraries: The Practitioners’ Perspective’ perspective,” Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, New York, NY, USA, p.245–256, Association for Computing Machinery, 2020.
  * [18] L. Xavier, A. Hora, and M.T. Valente, “Why do we break apis? first answers from developers,” 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp.392–396, IEEE, 2017.
  * [19] B. Todorov, R.G. Kula, T. Ishio, and K. Inoue,“Sol mantra: Visualizing update opportunities based on library coexistence,” 2017 IEEE Working Conference on Software Visualization (VISSOFT),pp.129–133, IEEE, 2017.
  * [20] R. Rubei, C. Di Sipio, P.T. Nguyen, J. Di Rocco,and D. Di Ruscio, “PostFinder: Mining Stack Overflow posts to support software developers,”Information and Software Technology, vol.127,p.106367, 2020.
  * [21] S. Baltes, L. Dumani, C. Treude, and S. Diehl, “Sotorrent: reconstructing and analyzing the evolution of stack overflow posts,” Proceedings of the 15th International Conference on Mining Software Repositories (MSR) 2018, ed. A. Zaidman, Y. Kamei, and E. Hill, pp.319–330, ACM, 2018.
  * [22] A.J. Viera, J.M. Garrett, et al., “Understanding interobserver agreement: the kappa statistic,” Fam med, vol.37, no.5, pp.360–363, 2005.
  * [23] A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” Empirical Software Engineering, vol.24, no.1, pp.381–416, 2019.
  * [24] C. Bogart, C. Kästner, J. Herbsleb, and F. Thung, “How to break an api: cost negotiation and community values in three software ecosystems,” Pro- ceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp.109–120, 2016.
  * [25] G. Bavota, G. Canfora, M. Di Penta, R. Oliveto, and S. Panichella, “How the apache community upgrades dependencies: an evolutionary study,” Empirical Software Engineering, vol.20, no.5, pp.1275–1317, 2015.
  * [26] L. Xavier, A. Brito, A. Hora, and M.T. Valente, “Historical and impact analysis of api breaking changes: A large-scale study,” 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp.138–147, IEEE, 2017.
  * [27] R.G. Kula, D.M. German, A. Ouni, T. Ishio, and K. Inoue, “Do developers update their library dependencies?,” Empirical Software Engineering, vol.23, no.1, pp.384–417, 2018.

[Authors_Photo/Syful.JPG]Syful IslamHe received the M.E. degree in Information
Science from Nara Institute of Science and Technology, Japan. He is currently
working toward the P.hD degree in the same institute. At present, he is on
study leave from Noakhali Science and Technology University, Bangladesh. His
research interests include software ecosystem, mining Stack Overflow, etc.
[Authors_Photo/WANG.JPG]Dong WangHe received the M.E. degree in Information
Science from Nara Institute of Science and Technology, Japan. He is currently
working toward the Doctor degree in Nara Institute of Science and Technology,
Japan. His research interests include code review and mining software
repositories. [Authors_Photo/Raula.png]Raula Gaikovina Kulais currently an
assistant professor at Nara Institute of Science and technology. In 2013, he
graduated with a PhD. from Nara Institute of Science and Technology, Japan. He
is currently an active member of the IEEE Computer Society and ACM. His
research interests include repository mining, code review, software libraries
and visualizations.

[Authors_Photo/ishio.jpg]Takashi Ishioreceived the Ph.D degree in information
science and technology from Osaka University in 2006. He was a JSPS Research
Fellow from 2006-2007. He was an assistant professor at Osaka University from
2007-2017. He is now an associate professor of Nara Institute of Science and
Technology. His research interests include program analysis, program
comprehension, and software reuse. He is a member of the IEEE, ACM, IPSJ and
JSSST.

[Authors_Photo/matsumoto.png]Kenichi Matsumotoreceived the B.E., M.E., and PhD
degrees in Engineering from Osaka University, Japan, in 1985, 1987, 1990,
respectively. Dr. Matsumoto is currently a professor in the Graduate School of
Information Science at Nara Institute Science and Technology, Japan. His
research interests include software measurement and software process. He is a
senior member of the IEEE and a member of the IPSJ and SPM.