# Textless direct speech-to-speech translation
with discrete speech representation

###### Abstract

Research on speech-to-speech translation (S2ST) has progressed rapidly in
recent years. Many end-to-end systems have been proposed and show advantages
over conventional cascade systems, which are often composed of recognition,
translation and synthesis sub-systems. However, most of end-to-end systems
still rely on intermediate textual supervision during training, which makes it
infeasible to work for languages without written forms. In this work, we
propose a novel model, _Textless Translatotron_ , which is based on
Translatotron 2 [1], for training an end-to-end direct S2ST model without any
textual supervision. Instead of jointly training with an auxiliary task
predicting target phonemes as in Translatotron 2, the proposed model uses an
auxiliary task predicting discrete speech representations which are obtained
from learned or random speech quantizers. When a speech encoder pre-trained
with unsupervised speech data is used for both models, the proposed model
obtains translation quality nearly on-par with Translatotron 2 on the
multilingual CVSS-C corpus [2] as well as the bilingual Fisher Spanish-English
corpus [3]. On the latter, it outperforms the prior state-of-the-art textless
model by $+$18.5 BLEU.

Index Terms—  speech-to-speech translation, discrete speech representation,
speech quantization

##  1 Introduction

Speech-to-speech translation (S2ST) helps oral communication between people
speaking different languages and aims to break such communication barriers.
Conventionally, S2ST systems are built with a cascade of automatic speech
recognition (ASR), text-to-text machine translation (MT), and text-to-speech
synthesis (TTS) sub-systems, all of which rely on intermediate text
representations. However, the vast majority of the approximately 7,000
languages in the world do not have speech recognition systems or even
acknowledged written forms [4, 5, 6]. For several widely spoken languages,
there are also many regional dialects used for everyday oral communication
that differ significantly from formal or standard written forms, such as
colloquial Arabic and regional Chinese. S2ST systems relying on intermediate
text representations cannot well support such languages. Additionally,
conventional S2ST systems often rely on phoneme representation in TTS and/or
ASR. However, for many low resource languages with written forms, there lacks
accurate grapheme-to-phoneme conversion tools for building such systems [7].

Recently, there has been progress towards developing S2ST systems without
relying on intermediate text representations. Such approaches can be put into
two categories: 1) End-to-end direct S2ST models, which use a single model to
directly translate speech from one language to another [8, 9, 1, 2, 10, 11,
12]; 2) Cascaded S2ST based on discrete speech representation instead of text
[13, 14, 15, 16, 17, 18, 19]. Although these approaches do not rely on textual
representation at inference time, many of them still need textual supervision
at training time for obtaining the best performance. A few works have
demonstrated the feasibility of training S2ST system without textual
supervision, they significantly underperform similar approaches when textual
supervision is used [13, 14, 15].

In this paper, we propose a novel approach for training end-to-end direct S2ST
model without textual supervision. The proposed model, _Textless
Translatotron_ , is based on Translatotron 2 [1], which is an end-to-end
direct S2ST model composed of a speech encoder, a linguistic decoder, and an
acoustic synthesizer. Instead of predicting the target phonemes as an
auxiliary task from the linguistic decoder in Translatotron 2, the linguistic
decoder in the proposed model predicts discrete representations of the target
speech, which are obtained from a speech quantizer based on VQ-VAE [20]. Such
discrete speech representations are expected to capture phoneme-like
information but without explicitly depending on it. Unlike previous works
using discrete representations [17, 14], which require multiple separately
trained models (e.g: translation model and vocoder) to be cascaded, our
proposed model can be trained end-to-end.

Experiments on two datasets, including a bilingual dataset and a multilingual
dataset, show that our proposed model obtained a very close translation
quality compared with the original Translatotron 2, despite of not using
textual supervision. Such results significantly outperform the prior state-of-
the-art textless model [17] on the Fisher Spanish-English dataset by $+$18.5
BLEU (or, $+$58% relatively).

##  2 Related works

Conventional cascade S2ST systems relying on intermediate text representation
are unable to support languages without written forms, or when textual labels
are missing from datasets for written languages. The recently emerging
research on S2ST without going through intermediate text representation
started to explore such scenarios.

The first proposed direct S2ST model, Translatotron [8], used a sequence-to-
sequence model with attention to directly translate speech spectrogram in one
language to speech spectrogram in a different language. Although it did not
rely on textual intermediate representation at inference time, it required
auxiliary objectives based on text at training time to obtain reasonable
quality.

Tjandra et al. [13] first demonstrated non-trivial results on training S2ST
models without any textual supervision at training time, by using a learned
discrete speech representation instead of text in a cascade system. It first
trained a speech quantizer based on VQ-VAE with a speech spectrogram
reconstruction task in a self-supervised manner. The trained VQ-VAE encoder
was used for converting the S2ST target speech into a discrete representation.
It then trained a second model for translating the S2ST source speech into the
discrete representation corresponding to the target speech. The VQ-VAE decoder
was used for converting the predicted discrete speech representation into
speech spectrograms. The resulting system showed reasonable translation
quality, but significantly underperformed baseline systems using text as the
intermediate representation.

Improvements on top of [13] have been primarily focused on learning better
discrete speech representations, such as utilizing more training data
including labeled data [14], different learning objective [15], and adopting
data augmentation for removing non-linguistic variance such as speaker
identity from the learned representation [17, 18].

Besides discrete speech representation, continuous speech representations
learned on unsupervised data [21, 22, 23, 24, 25] have also been shown
effective in S2ST [10, 19]. Such approaches can be naturally adopted for
textless S2ST as well.

Our work combines end-to-end direct S2ST with discrete speech representation
to benefit from both approaches, and utilizes self-supervised learned
continuous speech representation for obtaining best performance.

##  3 Textless Translatotron

The proposed model, Textless Translatotron, follows the architecture of
Translatotron 2 [1], which is an end-to-end direct S2ST model. The main
components of Translatotron 2 are a speech encoder, a linguistic decoder and
an acoustic synthesizer. In addition to them, we introduce a speech quantizer
based on VQ-VAE, to extract discrete representation from the target speech,
which is used for guiding the training of the linguistic decoder.

Fig. 1: Architecture of our proposed model. The left-side illustrates the
speech-to-speech translation architecture: it uses a pre-trained speech
encoder and linguistic decoder to output quantized speech ids, which are
obtained from a separately pre-trained VQ-VAE model shown on the right side.

We improve the Translatotron 2 model by addressing two data scarcity issues:
1) Both the encoder and the decoder were learned from scratch, which is
ineffective when training data is scarce. 2) The written form of the target
language might be unavailable, therefore supervision using text information
becomes impossible. To alleviate those scarcity issues, we improve each of the
three components and further add a discrete speech quantizer.

###  3.1 Speech encoder

Instead of training a speech encoder from scratch, we initialize our encoder
from a pre-trained multilingual w2v-BERT model [24] as described in [26].
w2v-BERT is a self-supervised model which combines both contrastive learning
and masked language model (MLM) [24]. The w2v-BERT model is pre-trained on a
large collection of multilingual speech datasets. It is used to initialize our
encoder to extract continuous speech representations and gets fine-tuned
during training process.

###  3.2 Speech quantizer

To explore discrete speech representations, we consider using VQ-VAE-based
speech quantizers as shown on the right-side in Figure 1. The motivation of
choosing VQ-VAE as the quantizer is that discrete representation learned from
VQ-VAE are directly optimized for speech spectrogram reconstruction, which
matches how such discrete presentation is used for generating translation
speech in S2ST models. In contrast, discrete representation obtained from
other models (e.g: HuBERT [23]) may not be optimized for spectrogram
reconstruction. Additionally, because of such matching, the decoder of the VQ-
VAE quantizer can be directly used as the synthesizer in the S2ST models.

The speech quantizers are pre-trained only using the S2ST speech data of the
target language (e.g: English only). Let $x$ denote the speech input, the
encoder projects it into a latent space $\mathrm{enc}(x)$. Each encoder has a
stride hyperparameter which controls the number of frames encoded into each
latent vector. We vary the stride from 2 to 16 in our experiment. The model
then maps the latent vector to discrete ids through finding a nearest vector
in a codebook $C=\\{c_{1},c_{2},...,c_{n}\\}$ where $n$ is the codebook size:

$c_{y}=\mathrm{argmin}_{i}\left\|L^{2}(c_{i})-L^{2}(\mathrm{enc}(x))\right\|_{2}$
(1)

Both the codebook and the projected vector are $L^{2}$ normalized. The decoder
takes the discretized representations $c_{y}$ and attempts to reconstruct the
speech inputs $\mathrm{dec}(c_{y})$. The reconstruction loss is the absolute
difference between the speech input and reconstructed input. Combined with the
quantization loss, the total training objective is defined as follows:

$L=\left\|x-\mathrm{dec}(c_{y})\right\|_{1}+\alpha\left\|\mathrm{sg}[\mathrm{enc}(x)]-c_{y}\right\|_{2}+\beta\left\|\mathrm{enc}(x)-\mathrm{sg}[c_{y}]\right\|_{2},$
(2)

where $\mathrm{sg}$ denotes the stop-gradient operator, $\alpha$ and $\beta$
are hyperparameters which we fix to 1.0 and 0.25, respectively. In our
experiments, we adopt a stack of non-causal transformer layers as the decoder
following [27]. However, we consider two different groups of quantizers as
described in the following subsections.

####  3.2.1 Random quantizer

The random quantization is inspired by the BEST-RQ work [25]. Suppose
$[x_{1},x_{2},...,x_{T}]$ denotes the speech input frames where
$x_{i}\in\mathbb{R}^{d}$ represents the input feature of frame $i$. A stacking
process is first applied by combining $s$ frames together without overlapping
where $s$ is the stride hyperparameter. The speech input becomes
$[x^{\prime}_{1},x^{\prime}_{2},...,x^{\prime}_{\lfloor T/s\rfloor}]$ where
$x^{\prime}_{i}\in{\mathbb{R}}^{ds}$. The stacked spectrogram is then mapped
into a latent space with a projection matrix $A$, i.e.
$\mathrm{enc}(x^{\prime};A)=Ax^{\prime}$. Both the projection matrix and the
codebook are randomly initialized and fixed during training, only the decoder
is optimized. The speech spectrogram is channel-normalized into Gaussian
distribution based on global statistics. The projection matrix $A$ is Xavier
[28] initialized, and the codebook $C$ is Gaussian initialized. Such
initialization ensures uniform distribution of the projected code IDs.

####  3.2.2 Learned quantizer

To compare the random projection encoder with learned encoders, we explore two
more quantizers: a linear quantizer and a Transformer quantizer. The linear
quantizer shares the same encoder architecture with the random quantizer,
except that the projection matrix $A$ and the codebook $C$ are learned. The
transformer encoder is similar to the transformer decoder, which has a stack
of non-causal transformer layers. It also stacks $s$ frames before the
transformer layers. These learned quantizers are pre-trained on the target
speech in the S2ST datasets, and frozen during the training of the translation
models.

###  3.3 Linguistic decoder

Instead of using textual supervision, we use the discrete speech
representations from the speech quantizer to guide the training of the
linguistic decoder. The linguistic decoder autoregressively predicts the
discrete code IDs generated from the speech quantizer.

###  3.4 Acoustic synthesizer

The synthesizer of Textless Translatotron also gets simplified compared to
Translatotron 2. In Translatotron 2, a duration predictor and a Gaussian-
weighted upsampler are used to augment the time rate of linguistic
representation from the linguistic decoder to match the same of the target
speech spectrogram. In Textless Translatotron, because the discrete speech
representations have a fixed time rate, a duration predictor is no longer
needed, and a simple transposed convolution is used to match the length of the
linguistic representation sequence and the spectrogram frame sequence.
Additionally, the autoregressive LSTM stack in the synthesizer of
Translatotron 2 is replaced by a non-autoregressive Transformer stack,
optionally initialized from the decoder of the learned VQ-VAE quantizers (Sec.
3.2).

##  4 Experiments

To evaluate the effectiveness of the proposed model and the variations
described in Sec. 3, we conducted comparative experiments on the multilingual
CVSS-C corpus [2] and the bilingual Fisher Spanish-English corpus [3]. The
CVSS-C corpus contains sentence-level paired S2ST data in 21 X$\to$English
language pairs. The source speech in the corpus is 1,153 hours of human read
speech collected via crowdsourcing; the target speech in the corpus is 719
hours of high-quality TTS synthetic speech in a single speaker’s voice, with
speech naturalness on-par with human recordings. The target speech is shorter
than the source speech because of better fluency in the TTS synthetic speech.
The Fisher Spanish-English corpus contains 127 hours of Spanish telephone
conversations and 96 hours of synthetic English translation speech in a single
speaker’s voice.

All the models are implemented in the Lingvo framework [29]. Unless specified
otherwise, all the S2ST models followed the hyper-parameters from [10]. The
speech quantizer used a 64 dimensional latent space with a codebook size of
512. A 256$\times$12 non-causal Transformer stack is used as the VQ-VAE
decoder and the S2ST acoustic synthesizer. The linear-based speech quantizer
used a single-layer linear projection, and the Transformer-based speech
quantizer used the same 256$\times$12 Transformer stack analogous to its
decoder. The pre-trained speech encoder is the same 0.6B-parameter w2v-BERT
model from [10, 26], which was pre-trained on 492k hours of unlabeled speech
in 51 languages.

Following [8, 2], the translation quality of S2ST is evaluated by BLEU on ASR
transcription from the translation speech (in lowercase, excluding punctuation
marks). We used an ASR model from [30] for evaluation, which is the same as
used in [2, 10, 1], therefore the results are comparable to these works. The
results on CVSS-C are grouped into high/mid/low-resource language pair groups
based on the amount of data available in the CVSS-C corpus, following [31,
26].

Two groups of baseline models are used for comparison: text-supervised models
and textless models. For the text-supervised models, we refer to the
Translatotron 2 models described in [1, 2] and its improved version with the
pre-trained encoder [10], which are the state-of-the-art models. For the
textless models, we compare our results with UWSpeech [14] and the prior
state-of-the-art in [15].

###  4.1 Fisher Spanish-English

Table 1: Translation quality on Fisher Spanish-English, measured by 4-reference BLEU on the test set. | BLEU
---|---
_Textless models_ |
UWSpeech VQ-VAE [14] | 3.4
UWSpeech XL-VAE [14] | 9.4
S2U + U2S [15] | 31.8
Textless Translatotron (this work) | 50.3
_Text-supervised models_ |
Translatotron [8] | 26.9
S2U + U2S [15] | 39.9
Translatotron 2 [1] | 42.4
Translatotron 2 w/ pre-trained encoder | 52.2
Reference | 88.6

The experimental results on the Fisher Spanish-English corpus is shown in
Table 1, compared to multiple baseline models. Textless Translatotron obtained
translation quality approaching the state-of-the-art text-supervised S2ST
model Translatotron 2, with a difference of merely 1.9 BLEU. It outperformed
the prior state-of-the-art textless S2ST model [15] by $+$18.5 BLEU (or $+$58%
relatively).

###  4.2 CVSS-C

Table 2: Translation quality on CVSS-C, measured by average BLEU on 21 X$\to$En language pairs, grouped into high/mid/low-resource language pairs. | All | High | Mid | Low
---|---|---|---|---
_Textless models_ |  |  |  |
Textless Translatotron (this work) | 17.7 | 33.5 | 22.8 | 10.2
_Text-supervised models_ |  |  |  |
Translatotron 2 [10] | 10.1 | 26.9 | 14.2 | 2.8
w/ pre-trained encoder [10] | 17.9 | 32.5 | 22.9 | 10.9
Reference | 91.1 | 88.4 | 89.2 | 92.8

The Fisher Spanish-English corpus contains translation between two close
languages and is unable to assess more complicate translation scenarios such
as involving heavy re-orderings between languages. To further evaluate the
performance of the proposed model, we conducted experiments on the
multilingual CVSS-C corpus, as shown in Table 2. Textless Translatotron
obtained translation quality nearly on-par with the Translatotron 2 model with
a pre-trained encoder, with merely 0.2 BLEU difference.

###  4.3 Ablation studies

To understand the impact of the designing and hyperparameter choices in the
proposed model, we conducted multiple ablation studies on the CVSS-C corpus,
as shown in Table 3, 4 and 5.

####  4.3.1 Linguistic training targets

Table 3 shows the impact of the different training target choices for the
linguistic decoder. When the Textless Translatotron is trained without using a
pre-trained speech encoder, it underperformed Translatotron 2 signicantly,
which used phoneme-based textual supervision. There was no significant
performance difference among the quantizer choices, including random quantizer
and learned quantizer. However, when a powerful large pre-trained speech
encoder was used, using a learned quantizer, especially one with a larger
capacity, showed significantly advantages over a random quantizer or a tiny
learned quantizer. With a relatively small Transformer quantizer
(256$\times$12), the performance of Textless Translatotron is nearly on-par
with Translatotron 2. It is important to note that no extra data other than
CVSS-C was used for training the speech quantizer of Textless Translatotron.
These results suggest that one major difficulty on training end-to-end direct
S2ST models lies in speech understanding, which can be overcome by either
introducing extra explicit supervision as in Translatotron 1 & 2, or by
leveraging self-supervised speech representation learning, as in Textless
Translatotron.

Table 3: Ablation studies on the training targets of the linguistic decoder, with or without a pre-trained speech encoder. Reporting average BLEU on all the 21 X$\to$En language pairs in CVSS-C. (Random: random speech quantizer; Linear/Transformer: learned linear or Transformer speech quantizer.) Encoder | Textless Translatotron | Translatotron 2
---|---|---
Random | Linear | Transformer | Phoneme
From scratch | 6.4 | 7.2 | 5.5 | 10.1
Pre-trained | 13.9 | 16.3 | 17.7 | 17.9

####  4.3.2 Speech quantizer stride and codebook size

Table 4: Ablation study on speech quantizer stride. Reporting average BLEU on all the 21 X$\to$En language pairs in CVSS-C. Stride | 2 | 4 | 8 | 16
---|---|---|---|---
BLEU | 17.3 | 17.5 | 17.1 | 13.8
Table 5: Ablation study on speech quantizer codebook size. Reporting average BLEU on all the 21 X$\to$En language pairs in CVSS-C. Codebook | 128 | 512 | 1024 | 8192
---|---|---|---|---
BLEU | 17.5 | 17.7 | 17.4 | 16.5

The stride and the codebook size are two critical hyperparameters of the
speech quantizer. Using a smaller codebook brings stronger supervision to the
linguistic decoder, but suffers from larger information loss. Similarly, using
a larger stride makes the discrete representation shorter and the training and
inference faster, but also suffers from larger information loss. These two
hyperparameters need to be choosed carefully in balancing quality,
convergency, and efficiency.

We compared different choices on these two hyperparameters in Table 4 and 5.
The impact of them are as expected above. We found that using codebook size
512 and stride 4 performed best in the experiments.

###  4.4 Sample analysis

Table 6: Sample translations from Textless Translatotron on CVSS-C. (REF:
transcripts of the reference translation speech; HYP: transcripts of the
predicted hypothesis translation speech.) _ja $\to$en (low-resource)_
---
REF | everyone knows mount fuji.
HYP | the fuji is long to mina all words.
_pt $\to$en (mid-resource)_
REF | a man and a white dog are looking at a postcard exhibit
HYP | a man in a white dog is looking at a postcards exhibit
_fr $\to$en (high-resource)_
REF | after a year spent in the kibbutz his family arrived in paris
HYP | after a year in the cabot his family arrived in paris

To understand the failure patterns, we manually analyzed samples of failure
cases in the BLEU evaluation. Table 6 cherrypicks a few examples that were
considered as failures in such evaluation. One common pattern is that the
model does not translate part of the source speech but copies the
pronunciation into the prediction without translation. Such direct copying can
be desired for words that do not need to be translated, such as names and
proper nouns (e.g. “kibbutz” in the fr$\to$en example; transcribing to “cabot”
is an ASR error in the evaluation), as also pointed out in [8, 2]. However, on
low-resource languages, such copying are often real failure cases (e.g. in the
ja$\to$en example, “mina” means “everyone” in Japanese). Such failure cases
can likely be improved by having more training data.

##  5 Conclusions

We proposed Textless Translatotron, a novel end-to-end S2ST model that can be
trained without any textual labels, therefore supports languages without
written forms. The proposed model is based on Translatotron 2, but uses
discrete speech representation obtained from a VQ-VAE quantizer instead of
phonemes to guide the training of the linguistic decoder. When a large pre-
trained speech encoder is used in both the proposed model and the baselines,
Textless Translatotron demonstraded performance nearly on-par with the state-
of-the-art direct S2ST model with textual supervision on the bilingual Fisher
Spanish-English corpus and the multilingual CVSS-C corpus, and outperformed
the prior state-of-the-art textless S2ST model on Fisher Spanish-English by
$+$18.5 BLEU (or $+$58% relatively).

##  6 Acknowledgments

The authors thank Ankur Bapna, James Qin and Yonghui Wu for helpful discussion
and feedback.

## References

  * [1] Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz,  “Translatotron 2: High-quality direct speech-to-speech translation with voice preservation,”  in Proc. ICML, 2022.
  * [2] Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen,  “CVSS corpus and massively multilingual speech-to-speech translation,”  in Proc. LREC, 2022.
  * [3] Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur,  “Improved speech-to-text translation with the Fisher and Callhome Spanish–English speech translation corpus,”  in Proc. IWSLT, 2013.
  * [4] Tanja Schultz and Katrin Kirchhoff,  Multilingual speech processing,  Elsevier, 2006.
  * [5] Daniel Nettle, Suzanne Romaine, et al.,  Vanishing voices: The extinction of the world’s languages,  Oxford University Press on Demand, 2000.
  * [6] Xinjian Li, Florian Metze, David R. Mortensen, Alan W Black, and Shinji Watanabe,  “ASR2K: Speech Recognition for Around 2000 Languages without Audio,”  in Proc. Interspeech 2022, 2022, pp. 4885–4889.
  * [7] Xinjian Li, Florian Metze, David R Mortensen, Shinji Watanabe, and Alan W Black,  “Zero-shot learning for grapheme to phoneme conversion with language ensemble,”  in Proc. ACL, 2022.
  * [8] Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu,  “Direct speech-to-speech translation with a sequence-to-sequence model,”  in Proc. Interspeech, 2019.
  * [9] Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura,  “Transformer-based direct speech-to-speech translation with transcoder,”  in Proc. SLT, 2021.
  * [10] Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobuyuki Morioka,  “Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation,”  in Proc. Interspeech, 2022.
  * [11] Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang,  “Leveraging pseudo-labeled data to improve direct speech-to-speech translation,”  in Proc. Interspeech, 2022.
  * [12] Rashmi T Shankarappa and Sourabh Tiwari,  “A faster approach for direct speech to speech translation,”  in IEEE Women in Technology Conference, 2022.
  * [13] Andros Tjandra, Sakriani Sakti, et al.,  “Speech-to-speech translation between untranscribed unknown languages,”  in Proc. ASRU, 2019.
  * [14] Chen Zhang, Xu Tan, Yi Ren, Tao Qin, et al.,  “UWSpeech: Speech to speech translation for unwritten languages,”  in Proc. AAAI, 2021.
  * [15] Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, et al.,  “Direct speech-to-speech translation with discrete units,”  in Proc. ACL, 2022.
  * [16] Xutai Ma, Hongyu Gong, Danni Liu, Ann Lee, Yun Tang, Peng-Jen Chen, et al.,  “Direct simultaneous speech to speech translation,”  arXiv, 2021.
  * [17] Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, et al.,  “Textless speech-to-speech translation on real data,”  in Proc. NAACL, 2022.
  * [18] Rongjie Huang, Zhou Zhao, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, and Jinzheng He,  “TranSpeech: Speech-to-speech translation with bilateral perturbation,”  arXiv preprint arXiv:2205.12523, 2022.
  * [19] Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee,  “Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation,”  in Proc. Interspeech, 2022.
  * [20] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu,  “Neural discrete representation learning,”  in Proc. NeurIPS, 2017.
  * [21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv, 2018.
  * [22] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Proc. NeurIPS, 2020.
  * [23] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, et al.,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  TASLP, 2021.
  * [24] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, et al.,  “w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in Proc. ASRU, 2021.
  * [25] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu,  “Self-supervised learning with random-projection quantizer for speech recognition,”  in Proc. ICML, 2022.
  * [26] Ankur Bapna, Colin Cherry, Yu Zhang, et al.,  “mSLAM: Massively multilingual joint pre-training for speech and text,”  arXiv, 2022.
  * [27] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in Proc. ICLR, 2021.
  * [28] Xavier Glorot and Yoshua Bengio,  “Understanding the difficulty of training deep feedforward neural networks,”  in Proceedings of International Conference on Artificial Intelligence and Statistics, 2010.
  * [29] Jonathan Shen, Patrick Nguyen, Yonghui Wu, et al.,  “Lingvo: A modular and scalable framework for sequence-to-sequence modeling,”  arXiv, 2019.
  * [30] Daniel S Park, Yu Zhang, Ye Jia, et al.,  “Improved noisy student training for automatic speech recognition,”  in Proc. Interspeech, 2020.
  * [31] Arun Babu, Changhan Wang, Andros Tjandra, et al.,  “XLS-R: Self-supervised cross-lingual speech representation learning at scale,”  in Proc. Interspeech, 2022.


# Mining the Temporal Evolution of the Android Bug Reporting Community
via Sliding Windows

Feng Jiang  Jiemin Wang  Abram Hindle  Mario A. Nascimento

September 2013

Department of Computing Science
University of Alberta
Edmonton, Alberta, Canada

© University of Alberta

## Abstract

The open source development community consists of both paid and volunteer
developers as well as new and experienced users. Previous work has applied
social network analysis (SNA) to open source communities and has demonstrated
value in expertise discovery and triaging. One problem with applying SNA
directly to the data of the entire project lifetime is that the impact of
local activities will be drowned out. In this paper we provide a method for
aggregating, analyzing, and visualizing local (small time periods)
interactions of bug reporting participants by using the SNA to measure the
betweeness centrality of these participants. In particular we mined the
Android bug repository by producing social networks from overlapping 30-day
windows of bug reports, each sliding over by day. In this paper we define
three patterns of participant behaviour based on their local centrality. We
propose a method of analyzing the centrality of bug report participants both
locally and globally, then we conduct a thorough case study of the bug
reporters’ activity within the Android bug repository. Furthermore, we
validate the conclusions of our method by mining the Android version control
system and inspecting the Android release history. We found that windowed SNA
analysis elicited local behaviour that were invisible during global analysis.

###### Contents

  1. 1 Introduction
  2. 2 Background
    1. 2.1 Betweenness Centrality
    2. 2.2 Overlapping Time Windowing
    3. 2.3 Clustering
  3. 3 Methodology
    1. 3.1 Data
    2. 3.2 Windowing Bug Reports and Extracting Social Networks  Methodology
    3. 3.3 Validation using the Android Release History and the Git
  4. 4 Results and Analysis
    1. 4.1 Global Analysis
    2. 4.2 Local Analysis
  5. 5 Validation
    1. 5.1 Activity pattern validation
      1. 5.1.1 Participants that appeared only once tend to be pure users
      2. 5.1.2 Participants showed up periodically should be a combination of users and developers
      3. 5.1.3 Participants who were continuously central for a long time period could have multiple areas of expertise
    2. 5.2 Cluster validation
      1. 5.2.1 Participants clustered together share similar areas of expertise and tasks
      2. 5.2.2 Clusters’ working areas of expertise are in accordance with the release contents along the time line
  6. 6 Limitations
  7. 7 Conclusion and Future Work

##  1 Introduction

Global analysis provides us with easy to interpret data that gives us an
overview of the entire system. It simplifies complicated dimensions like time
and provides us with an easy way to explain results. Unfortunately, for tools
like _Social Network Analysis_ (SNA), a global analysis can miss a lot of
important interactions, especially between stakeholders, thus we propose a
method of using SNA to study bug repositories and tease out local
collaborations.

SNA is a powerful tool that helps practitioners and researchers study the
complicated interactions of participants within communities; SNA is well
accepted in the area of software maintenance and mining software repositories
communities [1, 2, 3]. The bug repository records interactions among software
developers and users in a software project’s community. With SNA, we are able
to study the structure of the interactions by analysing the graph constructed
through the interaction of bug reporters in the bug repository. The results
can be used in expertise elicitation and triaging in order to suggest which
participants have expertise relevant to an issue [3]. Usually SNA is run
globally across all day, over a single period, or over an entire project
lifetime. In this paper we argue that using SNA in a more local manner
provides valuable insights into interactions between stakeholders during the
development and maintenance of a software system.

Open-source communities are amenable to social network analysis as they are
open to user interaction and participation. At the same time there is a lack
of imposed organizational structures found within corporate organizations [4].
Because open source projects often lack strict centralized control and
requirements [5], developers often choose their tasks instead of being
assigned one [6]. This fact suggests that local structure of interactions
among users and developers who express an interest in one part of the project
tend to self organize and produce interesting collaboration structures
(networks).

Bug repositories are also amenable to social network analysis as bug
repositories host and record discussions regarding issues or bugs relevant to
the development and the use of a software development project [6, 7]. Bug
repositories are also heavily used by open-source projects. Collaboration
among developers has been studied in various aspects about how the
communication introduces or avoids bugs, and further influences the software
quality, [8], [9], [10], [11]. Besides the collaboration among developers,
collaboration between users and developers is evident in bug reports since the
discussions and communications are recorded as reported bugs, and posted
comments on bug reports. One point here is that, both users and developers are
often periodic, and their activities or collaborations can be local and thus
missed out in global analysis.

In the case of the Android bug repository, provided by the 2012 MSR Mining
Challenge [12], a reporter would report a bug, which might attract comments
from bug commenters; the commenters discuss the reasons and possible
resolution of the bug. The bug reporting community members are usually
comprised of both bug reporters and bug commenters who are either Android
developers or Android users. From the perspective of the bug repository,
unlike the version control system, there is actually no obvious boundary
between a user and a developer. We refer to these different participants as
_bug participants_.

In order to apply SNA to the bug repository, we first create the graph based
on the interactions. We pose that each node of the graph represents one bug
participant and each edge represents the connection between two participants
who have communicated on the same bug. We will introduce the network graphs in
detail in Section 3.

We use betweenness centrality to quantify the importance of a participant in
the community [13] (betweenness centrality will be better explained in Section
3). The betweenness centrality could reveal two aspects of a participant in a
community network: 1) the quantity of bug reports (which attract at least one
comment) or comments they have made and 2) the importance of the content of
their reports or comments. When participants have high betweenness, they might
have: 1) reported quantities of bugs with at least one comment on them, 2)
made lots of comments, 3) reported a very critical bug which attracts
comments, 4) or made a very interesting comment which attracts comments from
other participants.

However, the previous work [2, 6] applied SNA on the entire lifetime of a
project, such that only a single community network was constructed. Some of
collaborations might not be evident if one were to analyze a large single
network. That is because certain structures will not be observable on the
global scale. In order to peer into these local self organized structures
using social network analysis, we felt it is better to choose a _windowed_
approach, [3, 14]. Windowing allows us to look at network during a slice of
time and then relate our measures (betweenness centrality per author) to the
next window and beyond. This _sliding window_ view of centrality allows us to
see those developers and users who are constantly at the forefront of
discussion or those who ebb and flow between issues and tasks. Moreover, by
sliding windows, each pair of adjacent windows would have an overlap, which
results in smoother trends, and more importantly, helps to maintain context.
Other benefits provided by time windowed analysis is that it gives a more
accurate and nuanced view of the data as locally central participants then
will not be “drowned out”.

In summary, we use SNA to study the activities of bug participants based on
the Android bug reports and comments repository. We apply the sliding window
method to observe smooth change trends in the collaboration graph across time.
With these mining results, we seek to analyze bug participants’ interactions,
activity trends and patterns. We then demonstrate our analysis results via
answering the following research questions about local and global behaviours.

Global research questions:

RQ1. How does the number of active bug participants change over time? Why?

RQ2. How does the betweenness centrality of a participant change over time?
What are the reasons when they have a certain activity pattern?

Local research questions:

RQ3. Are there special time ranges during which participants are more/less
active or central than normal? Why?

RQ4. What are the possible scenarios for a very sharp change of the
participants’ centrality? Why?

We also validate if this windowed methodology actually highlights relevant
behaviour by inspecting the Android release history111Android release history:
http://developer.android.com/sdk/index.html and the Android version control
system. The validation would be discussed in Section 5.

The rest of our paper is organized as follows. Section 2 introduces basic
concepts and techniques we used in this study. The specific steps and the
methodology will be discussed in Section 3. Section 4 describes the details of
our mining results. The analysis of the results and its corresponding
validation is provided in Section 5. Section 6 presents the limitations of our
mining process and Section 7 summarizes the paper and discusses the future
work.

Figure 1: An example: bug 14038 is reported by timothyA, and there are five
comments on this bug. When time window applied, comments are plotted into two
windows, and the bug report of this example forms two networks with the weight
noted on their edges

##  2 Background

###  2.1 Betweenness Centrality

The betweenness centrality of a vertex is the number of geodesic paths in a
graph that includes this vertex; the geodesic path is defined as the shortest
path which has the minimum weight between two nodes. Defined by Freeman [15],
the betweenness can be represented as:

$\sum_{i=1}^{j-1}\sum_{j=1}^{n}\frac{g_{ij}(k)}{g_{ij}},i\neq j\neq k$ (1)

where $k$ is a vertex of the graph, $n$ is the total number of vertices, $i$
and $j$ are vertices other than $k$, $g_{ij}$ is the number of geodesic paths
between vertex $i$ and $j$, and $g_{ij}(k)$ is the number of geodesic paths
that include $k$.

It is used as a measurement of a person’s importance in a network. A person
would be regarded as central if he is on the geodesic path between two other
persons. As proposed by Freeman [15], if a person is located on the geodesic
path between two other persons, he becomes one of the key persons who connects
the others. That is, the more a person connects to the other people in a
network, the more important or central he is [16].

In our work, we normalize the betweenness centrality values to eliminate the
effect of different sizes of the networks. The betweenness is normalized as:

$Normalized~{}B=\frac{B}{\frac{(n-1)(n-2)}{2}}$ (2)

where $B$ represents the original betweenness value and $n$ is the number of
nodes in the graph being calculated.

Compared with simply counting the total number of comments or total bug
reports of a participant, betweenness acts better to reflect the interactions
among people. For example, when a person reports lots of bugs but none of them
attract any comment, it is very likely that his bug reports are not
interesting or important. In this case, if we merely counted the number of
their reports or comments, we would possibly increase their importance in the
network artificially. Therefore, we choose to use betweenness centrality to
eliminate this unfair counting [13].

###  2.2 Overlapping Time Windowing

When SNA is applied in other papers [2, 3], it is typically applied to the
entire history or one period of the partial history and all the bug reports
within that period. Windowed analysis instead repeats social network analysis
across 100s of windows (in our case, as many windows as we have days). These
windows overlap and often the analysis of one window results in the same
analysis as the previous window due to the overlap. We slid our windows by 1
day and for two adjacent windows $A$ and $B$, $B$ starts on the second day of
$A$, and they would have an overlap of 29 days, that is each window does some
redundant analysis but produces smoother transitions in analysis between
windows. Thus 1 comment in a bug report will have an effect on the graphs of
30 windows. This is similar to Hindle et al.’s [14] analysis of topics using
windows but they did not use an overlap. We could thus see the changes in the
trend of a participant’s activity.

Moreover, time windowed analysis could give a more accurate and nuanced view
of the data [3, 14], as locally central participants would not be “drowned
out”. For instance, if a bug participant participates in many bug reports and
bug comments during one month, he would be one of the most central
participants with a high betweenness within this window. However, if he
appeared only for that month, globally, he would have low betweenness and
would not show up as central, even though during a shorter period he played a
vital role. As we can see in Figure 2, the left column graph shows the
betweenness values of participants over the entire time period; local details
are missed and we get nothing about the trend, compared to the right part of
the results from overlapping windowing. For example, cluster 8 on Figure 2 is
bright and important at the start of our analysis but does not appear in the
global graph on the left. Also, if there is a very sharp drop of values of a
certain participant, the overlapping windows would give a more nuanced view of
the change and what was happening.

Another point is that, comments on the same bug might not be globally
temporally relevant [17, 18] thus a global time analysis would not make much
sense in this case. This could happen if new changes induce new bugs or modify
the behaviour of a reported bug.

Figure 2: Betweenness centrality along time line: the x-axis represents the
number of time windows and the starting dates are denoted every 100 windows.
The y-axis represents the number of bug participants who have ever been
central in the bug community with betweenness centrality valued greater than 0
for some time period. The color represents the value of betweenness
centrality, with darker colors corresponding to lower betweenness and lighter
colors for higher betweenness. We used K-means clustering with cosine distance
where K = 100

###  2.3 Clustering

Figure 3(1) orders participants by their betweenness values. We can indeed
find that there are participants of low overall betweenness but being very
active (show up bright) at some time points, and this supports the necessity
of windowing, as stated in Section 2.2. However, we need more information
about participants’ working patterns and get an idea about their being
interacting groups.

In order to perceive clusters, that are local groups of interactions, we
clustered the bug participants using K-means by their betweenness centrality
distribution along the time line. K-means is one of the most popular
clustering methods which aims to partition $n$ data items into $k$ clusters
that each data item belongs to the cluster with the nearest mean [19]. We
choose to use K-means with cosine distance. The cosine distance between two
vectors is defined as,

$cosine\\_dist(A,B)=1-\frac{A\cdot B}{\Arrowvert A\Arrowvert\Arrowvert
B\Arrowvert}$ (3)

where A and B are two vectors, $\cdot$ represents the inner dot operation and
$\Arrowvert\cdot\Arrowvert$ indicates the module of the vector. With
clustering, authors with similar temporal centrality would be grouped together
so that bug participants with similar activity patterns are also grouped
together.

In this paper, we choose the K-means with cosine distance because it gives a
better visual clustering result, as compared in the plots of Figure 3. In this
case, cosine distance calculates the similarity between each pair of
participants in terms of temporal centrality whereas Euclidean distance
focuses on the magnitude of data, the size and frequency of centrality.

Moreover, we used $k=100$ for the K-means, to cluster authors. There is a
trade off between the size of clusters and the variance within each cluster.
As we can see from Figure 3(3), $k=10$ also gives good visualization result,
but considering the number of more than 1600 participants, we should get a
larger $k$ to keep the diversity of working groups in similar. We set $k=100$
in this case, since we get the aesthetically best visualization (our
subjective opinion based on visible clusters) of all the data; we had tried
other values of $k$ such as 5, 10, 25, 50, 75, 200, 300.

##  3 Methodology

Our methodology consists of six steps that deal with raw data, construct
graphs and apply social network analysis with sliding windows. We conduct a
thorough case study of the Android bug repository with the proposed method and
validate the conclusions from the results by mining the Android version
control system and inspecting the release history.

###  3.1 Data

With the provided Android bug repository 2012 and the Android version control
system from the MSR Challenge [12], we converted and stored the XML format
data into a database for efficient analysis using Microsoft SQL Server
Business Intelligence. Our analysis focused on the bug records of the previous
two years from January 1st 2010 to December 4th 2011 since during these two
years, there are more records in the repository as we counted that
participants are more active; also, the activities are representative, both
the Android platforms and their developer groups are larger and more diverse
during the latest two years and it was also more relevant to modern Android
handsets. The data we used of these two years covers 14,432 out of 20,169
total bug records and 46,806 out of 67,730 total bug comments from the whole
dataset. Related to these bug and comment records, there are 30,969 people who
have either reported a bug or made comments on a bug.

The bug and comment records are grouped into 30-day windows sliding by 1 day.
We extracted 673 windows in total from the bug reports during year 2010 and
2011.

###  3.2 Windowing Bug Reports and Extracting Social Networks
Methodology

We windowed the data and constructed networks that indicated the relations
among the participants within each specific time window. For each window, we
calculated the betweenness centrality of each participant and we plotted the
centrality values per participant in a visualization. The steps of our
methodology are explained as following:

Step 1: Pruning the data. We pruned the records of the reporters and
commenters into a pure name format, which are originally recorded in semi-
anonymous email formats in the XML repository dump. For example, given the
original email address which is represented by<EMAIL_ADDRESS>we
truncate the string starting from “….” and keep the front part “mathias” at
the beginning as the name of the reporter or commenter. This strategy could
lead to name aliasing problem, especially for common names or email addresses
starting at just a simple letter like<EMAIL_ADDRESS>Although algorithms
have been provided to reduce the extent of the problem, [20], [2], [21], it is
difficult or even impossible to eliminate the influence from this data quality
issue. When applied to other repositories that do not anonymize this would be
less of a problem. Hence, we focus on participants whose names are less common
and less ambiguous in our study.

Step 2: Windowing the records. We windowed the data into periods of 30 days
with a 29-day overlap. 30 days was chosen as a window size because it is
smaller than the periods between a major and minor release, it is similar to a
month of work, but long enough to contain the resolution of multiple bugs. We
have compared sliding by 1 day with our previous result of sliding by 7 days,
1 day sliding produces gradual and smoother transitions of centrality.

Step 3: Establishing the network. We made a tool to perform the SNA with
sliding windows. The tool is implemented in Java and built on top of the JUNG
Graph Framework, that converted bug reports and bug comment records within a
window to a social network graph.

The nodes of these networks represent participants who have either reported
some bugs or made comments on bugs. The edges represent connections between
two nodes. All the edges are weighted. For a bug within a selected time
window, whenever a person makes a comment on this bug, the edge between the
bug commenter and the bug reporter would get weight plus one, as well as the
edges linking to the participants who previously made comments on this bug.
Bug reports or comments in different windows would have separate network
graphs depending on the activity of their reporters or commenters. An example
in Figure 1 indicates how the weighted network graph is built.

Step 4: Calculating the centrality. We calculated the betweenness centrality
using JUNG, and normalized the centrality with the number of node pairs, as in
Equation (2). We then get a list of all the bug participants and their
betweenness centrality values for the total 673 overlapping windows.

Step 5: Removing irrelevant participants. We removed the participants with
betweenness centrality value 0, who might have either reported a bug/bugs with
no comments, or made the only comment on a bug so that no other participants
are related. Afterwards, we get 1654 participants with betweenness centrality
value larger than 0, out of the 30969 in total.

Step 6: Generating the analysis graph. The activity of each bug participant is
represented by a 673 dimensional vector representing their betweenness per
window. Each element of the vector indicates the betweenness centrality value
extracted from the graph, which is generated from the window for that specific
time period (in our case, the specific time period is 30 days starting from
the date of the window start point). Then we clustered all the vectors by
using K-means ($k=100$) with cosine distance to 100 clusters. Finally, we
plotted the results, as shown in Figure 2 to visualize the clustered data so
that we could easily analyze our results.

###  3.3 Validation using the Android Release History and the Git

In addition to the methodology of mining the Android bug repository, we made
use of the git version control repository and inspected the release history
highlights to validate the purpose behind the clusters and patterns we
observed. We looked into the participants who contributed to the git
repository in order to find their areas of expertise and validate our analysis
conclusion about how the community participants act in accordance with the
project development.

The types of files modified and the corresponding projects are highly
correlated with the specialization of those who commit changes. For instance,
if a developer always submits kernel related code files, he is more likely to
be specialized in kernel techniques. Types of files include document files,
test files, source files, etc; dictionary paths of files usually indicate what
projects the files belong to. We manually identified the participants’ areas
of expertise by observing the _project_ and the _target_ for all of their
commits (such as source code or documentation). To give a specific example, if
there were commits from a developer, Mr.Guilfoyle, on the target file
media/java/android/media/Ringtone.java under the project
platform_frameworks_base; then, we would suggest that Mr.Guilfoyle likely has
some specialized knowledge about the platform’s ringtone. Thus this is how we
derive participant expertise [3].

Also, we could further relate their expertise to their centrality patterns.
The Android release history could, on the other hand, help to relate the
release highlights to participants central behaviour during that release.
Further validation is discussed in Section 5.

Figure 3: Betweenness centrality along time line. Participants on the y-axes
are ordered differently by betweenness values or various clusterings.

##  4 Results and Analysis

We study the results shown in Figure 2. Each horizontal line represents the
673 betweenness centrality values for the selected bug participant during year
2010 and 2011\. In total, we have 1654 bug participants. By studying these
results, we answered the following questions:

Figure 4: Number of active participants across time.

Figure 5: Sum of betweenness centrality of participants across time.

###  4.1 Global Analysis

RQ1. How does the number of active bug participants change over time? Why?

To give an overview, we compared the interaction of bug participants between
January, 2010 and December, 2011, and found that the interaction among
participants in the Android bug community in 2011 was similar to the
interaction of participants in 2010 but more frequent, as we can see in Figure
2. One “gap” occurs around window 300, which we will explain in the next Local
Analysis subsection.

Correspondingly in Figure 4, that we counted the number of participants with
betweenness centrality value larger than 0 within each window, the number of
active participants during 2011 is slightly larger than that of 2010. Figure 5
shows the sum of betweenness values along the two years’ time line, we can see
that the trend is very similar to that of the number of active participants in
Figure 4. This also suggests that the betweenness centrality reflects the
interaction among participants.

Moreover, a possible reason for the changes of the number of active bug
participants and the betweenness centrality values is that around major or
minor releases of SDKs, API fixes or improvements, participants seem to become
more active in bug reporting, discussing and fixing activities. Also, during
these time periods, bugs are more likely to be discovered and reported.
Perhaps the pressure of the release is causing developers to address
outstanding bugs more than usual. After a release, users also take part in the
activity of discovering the bugs and problems so that in this case both users
and developers would like to discuss the bugs.

RQ2. How does the betweenness centrality of a participant change over time?
What are the reasons when they have a certain activity pattern?

Observing the continuity of betweenness centrality in Figure 2, some
participants have kept active during the entire two years, and correspondingly
they have a very continuous and bright line. For participants of this type,
there are a few possible explanations. First, our conjecture is that these
participants are professional developers who belong to the core development
team so that what they reported are more important issues which attract more
participants to discuss and fix them. Their identities of being professional
developers will be discussed in Section 5.1.

Second, some of these participants are of high community status or expertise,
and they might supervise and guide the development of the project. For
example, when we validated, we did find one developer, romainguy, who has
experiences on almost every component relevant to platforms so that he can be
considered to be an expert. Developers related to these continuous lines are
listed in Table 1, and we will further discuss and validate on them in Section
5.1.

However, in most cases, participants’ betweenness values are highly variant,
as observed in Figure 2. To investigate the variation in betweenness values
over time, we decided to count the number of times that a user experienced a
range of consecutive windows in which the user had non-zero betweenness.

Participants with a count of distinct ranges greater than 1 would be _phasers_
who periodically participate within the Android bug community. Here phasers
are those who phase into centrality and later out of it. These randomly
phasing participants (phasers) are very likely to acquire less expertise or
have lower community status in their community, than those with continuous
high centrality. Phasers might be interested in limited topics and only
central and active during the appearance of bugs relevant to those topics.
Participants who only had 1 distinct range of betweenness are considered to be
participants who only appeared once, and are probably users. We validate the
roles these participants play in Section 5.1.

To summarize, among the 1654 participants with betweenness values larger than
0, we analyzed their centrality patterns and divide them into three
categories: 1) participants appeared only once with a betweeness greater than
0 (71 out of 1654 participants), 2) participants recurred periodically (1575
participants) and 3) participants who are central along the entire project
history (8 participants).

###  4.2 Local Analysis

RQ3. Are there special time ranges during which participants are more/less
active or central than normal? Why?

By inspecting the Android release history highlights, we found that the v2.1
SDK was released on 12 January 2010, which corresponds to the first peak value
in Figure 5. Android v2.2 SDK was released on 20 May 2010 and this corresponds
to peak 2. From Dec. 2010 to the beginning of Mar. 2011, several minor updates
were released and on 22 Feb. 2011, one major update v3.0 SDK was released.
These releases explain the summit, i.e., peak 3, in Figure 5. This is
correlated with more participation at the same time.

In addition, during the first obvious “gap”, which covers the time from
October 2010 to the end of 2010 (around window 300), the social network during
this time period is almost inactive and even “quiet”. There were fewer
releases during the “gap”.

The other low value showing up in the end of Figure 5 results from the fact
that there are no bug reports recorded (right tail censoring) in the given
dataset. This piece of data is still meaningful because it contains comments
belonging to bug reports several weeks or months before. The betweenness value
is thus simply calculated by the comments here.

RQ4. What are the possible scenarios for a very sharp change of the
participants’ centrality? Why?

Considering individual participants, almost all of them has experienced
centrality oscillations. In addition, some participants tend to become active
and core members during the same time period and then they fade away together.

We suspect that the phasers tend to be interested in one or several categories
of problems so that they appear only along with the occurrence of these
issues. They take part in activities related to the bugs or technical issues
and become inactive after the problems are solved. Or in the case when they
are working on a project, they would become inactive when the projects are
finished. As showed in Figure 2, the participants’ tend to get clustered
together around important releases, which supports that the phasers are
working along with projects or related issues. Meanwhile, by observing the
clustered participants of their activity patterns in Figure 2, we suspect that
the phasers that show up densely together could be interested in similar
categories of topics. This assumption is validated in Section 5.2.

##  5 Validation

We made use of the git repository and inspected the release history to
validate our answers to the research questions in the previous section. For
RQ1, it could only get answered based on assumption and the number of active
participants across time as we counted in Figure 4, but not thoroughly
validated. RQ3 is intuitively answered when we match the betweenness
distribution with the release history by time, and no further validation is
needed. For RQ 2 and RQ4, we have made a detailed validation in this section.

###  5.1 Activity pattern validation

From the mining results, among the 1654 participants with betweenness values
larger than 0, we notice that there is a small group of participants who have
been central for most of our analysis period (8 participants out of the 1654);
another relatively larger group appear without any recurrence (71 participants
out of the 1654); the majority would periodically become central in their
community (1575 participants out of the 1654). Based on the three activity
patterns proposed in RQ2, we confirmed many of our previous suspicions:

####  5.1.1 Participants that appeared only once tend to be pure users

We look into the git repository to find the files submitted by the 71
participants who have appeared only once in the bug community. We found that
only 7 of them have ever committed a change, which means that these 7 are
developers rather than pure Android users. The rest do not have commits in the
version control system. This verifies our assumption that participants
appeared only once in the bug community would more likely to be pure users, as
introduced in RQ2.

####  5.1.2 Participants showed up periodically should be a combination of
users and developers

Periodically appearing participants are the majority and we call them phasers.
Based on the methodology in Section 3, we looked into the commit history in
the git repository in order to verify the expertise of phasers. With as many
as 1575 participants, we sampled 156 participants. $21.8\%$ of the sampled
participants were developer phasers, who have submitted changes. We studied
the expertise of the developer phasers from this sample. All except two of
them have worked on specialized tasks that implied some specific kind of
expertise or specialization. The rest $78.2\%$ have never submitted files to
the development community. They are probably users of Android. Thus, phasers
consist of both users and developers. This answers to our assumption of the
phasers’ role in RQ2.

Table 1: 5 continuously central participants who have submitted changes to the git. Participant |  #Submitted-_changes |  Related Project
---|---|---
fadden |  1259 |  device_samsung_crespo, platform(bionic, build, dalvik, etc.)
xav |  3501 |  platform(frameworks_base, build, external_bouncycastle, etc.), device_sample,
mbligh |  80 |  kernel(common, experimental, linux-2.6, msm, omap, qemu, samsung, tegra)
ralf (Ralf.-Hildebrandt) |  665 |  kernel(common, experimental, linux-2.6, dalvik, external_libpng, sdk,system_core, etc.)
romainguy |  1455 |  device_htc_passion, device_samsung_crespo, platform(build, cts, dalvik, development, external_bouncycastle, libcore, ndk, apps(AccountsAndSyncSettings, AlarmClock, Bluetooth, Browser, Calculator, etc.), inputmethods(LatinIME, iOpenWnn, PinyinIME, CalendarProvider), providers(DownloadProvider, GoogleSubscribedFeedsProvider), wallpapers(Basic, LivePicker, MagicSmoke, MusicVisualization), prebuilt, sdk, system_core)

####  5.1.3 Participants who were continuously central for a long time period
could have multiple areas of expertise

5 out of the 8 participants in this group have submissions in the git. We
extracted the projects these 5 participants have submitted changes to, as
listed in Table 1. (On the forth row, ralf and Ralf.Hildebrandt are email
alias of the same person, as we observed that the author_name attributes are
the same for the two email alias.)

Firstly, considering the number of changes they made, all of them except
mbligh have more than 500 commits within the git, which means that they are
quite active in Android development community. This supports that they are
experts or advanced developers since more submissions indicates a broader
range knowledge about the related techniques.

Moreover, fadden, xav, and romainguy are all working on the platform layer,
which includes build, dalvik, development, framework base, libcore, sdk, etc.
All of their areas of expertise are related to the platform layer or system
core layer.

The participant romainguy has experiences modifying almost every component
relevant to platforms, including both the apps and the core, and hence should
be considered as Android platform development leader.

Furthermore, when investigating these continuous lines we found some
participants were Google employees, for example, two developers with alias
mbligh and romainguy. Their email account recorded in the git repository is
from the “google.com” domain, and moreover, when we googled them, they are
indeed introduced as software engineers at Google.

To summarize, this subsection demonstrates that three different centrality
patterns correspond to participants of three categories, which supports our
analysis hypothesis about activity patterns in Section 4.

Table 2: 5 clusters we have chosen, out of a total number of 21. Cluster | Time
---|---
1 | May 16, 2010 - Jun. 24, 2010
2 | Jun. 2, 2010 - Jul. 24, 2010
3 | Jan. 13, 2011 - Mar. 3, 2011
4 | Dec. 3, 2010 - Jan. 31, 2010
5 | Feb. 4, 2011 - May 1, 2011
Table 3: Participants and their areas of expertise in cluster No. 4 ID | Name | Areas Of Expertise
---|---|---
1 | charles | kernel - sound, kernel_linux-2.6
2 | jasta00 | ringtone, media
3 | kristoff | driver(net, video, serial, input)
4 | rik(rik.bobbaers) | kernel_linux-2.6(mlock)
5 | rik(rikard.p.olsson) | kernel_linux-2.6(arm)
6 | rik(riku.voipio) | kernel_linux-2.6(arm), driver
7 | snp | platform sdk(eclipse plugin)
Table 4: Participants’ common areas of expertise of each cluster. Participants number is counted as the number of participants within each cluster who has ever submitted a change and appeared in the git, ie., developers. Cluster |  Participants Number |  Areas Of Expertise
---|---|---
1 |  5 |  netfilter, driver(video), tests, MIPS
2 |  13 |  driver(usb, wireless, mouse), sound, net, i386, performance(tools), input methods
3 |  9 |  sound, driver, frameworks_base, tests, platform, kernel
4 |  7 |  sound, media, kernel_linux-2.6, driver, platform sdk, kernel video/serial
5 |  63 |  net(bluethooth, net driver, ipv$x$, kernel_linux-2.6), driver(dvd, media, usb, gpu, net), ia64, sound, tests
Table 5: Highlights of identified clusters from Figure 2 Release |  Time |  Highlights |  Related cluster
---|---|---|---
v2.2 |  May 20, 2010 |  camera and gallery, portable wifi, multiple keyword language, performance(general, browser), media framework, Bluetooth, kernel upgrade, APIs(media, camera, graphis, data backup, device administrator, UI framework) |  1
v2.2.1 |  Jan. 18, 2011 |  bug fixes(one is about root and unroot), security updates, performance improvements |  3
v2.2.2 |  Jan. 22, 2011 |  fixed minor bugs, including SMS routing issues |  3
v2.3 |  Dec. 6, 2010 |  UI refinements, faster text input, power management, NFC, multiple cameras, download management, new multimedia, new developer features(gaming, communication, multimedia, garbage collector, event distribution, video driver, input, native access-audio, graphics, storage, development), linux kernel upgrade to 2.6.36, Dalvik runtime, mixable audio effects |  4
v2.3.3 |  Feb. 9, 2011 |  NFC, Bluetooth, Graphics, media, framework, speech recognition, voice search, API(identifier, build-in app, locales), emulator skins |  5
v3.0 |  Feb. 22, 2011 |  UI design for tables, redesigned keyboard, improved text selection, copy and pase, connectivity options(USB, WIFI, media, keyboard, bluetooth), apps update, browser, camera and gallery, contacts, email, development support |  5

###  5.2 Cluster validation

As we have discussed above, participants are more active around important
releases. Moreover, we can observe from Figure 2 that participants’ centrality
distributions tend to form into groups or clusters, that often are found
around the releases. Participants belonging to the same group become central
during the same time periods and then fade away together.

We labeled 21 visible clusters from Figure 2 and looked into five of them
which are located more around releases. The five clusters we chose are listed
in Table 2.

We extract changes submitted by the members of each cluster from the Android
git. (For those who do not have records in the git, we regard them as pure
users and do not consider them in this case). After inspecting their
submissions, we would get an idea about what kind of tasks they have been
mostly working on. Based on release history and the commit logs we found that
these clusters tend to be coherent efforts undertaken by multiple kinds of
participants.

####  5.2.1 Participants clustered together share similar areas of expertise
and tasks

Our analysis in Section 4 shows that the phasers that show up densely together
could be interested in similar categories of topics or working on tasks
related to the same area.

As described in Section 3, we extract the targets and project names from the
git for each member appeared within the cluster. The areas of expertise could
be inferred by the contents of the targets and the topics of the projects. We
summarized the areas of expertise of participant clusters (from Figure 2) in
Table 4.

Inspecting the areas of expertise, we find that each cluster has their own
topics, which are relatively different from each other. Also, the topics of
each cluster are concentrated to specific layers of Android’s architecture.
For example, cluster No.1 covers techniques about net filters, drivers, tests,
and MIPS, while cluster No.2 is about drivers for connecting devices (usb,
wireless, and mouse), net, processor, and input. It is easy to tell that
participants of these two clusters are working on different tasks. The other
clusters could lead to the same conclusion. Thus we conclude that clusters
often exist around a topic.

Take cluster No.4 as an example. There are 7 developers contained in this
cluster, as listed in Table 3. It can be observed that work of participants in
this cluster could be generally divided into two groups: one is about the
Linux 2.6 based kernel, another is related to multimedia. Charles,
rik.bobbaers, rikard.p.olsson, and riku.voipio (the pruned bug reporter alias
rik is related to three developers in the git and we look them all; this issue
would be discussed in Section 6) are all modifying the Linux 2.6 kernel.
Charles, jasta00, and kristoff are working on multimedia topic, which includes
sound, video drivers, and ringtone.

When we look into other clusters, we get similar conclusions. Thus, from the
observation and analysis above, we can conclude that participants with similar
centrality patterns often share similar areas of expertise and tasks. This
validates our assumption about the phasers being clustered on specific
techniques in RQ4.

####  5.2.2 Clusters’ working areas of expertise are in accordance with the
release contents along the time line

When observing the Android release history, we concluded that the overall
betweenness centrality becomes higher around releases, and more active
participants appear around important releases, at least according to Figure 4
and Figure 5.

In addition, when taking participants’ areas of expertise into consideration,
we find that the release highlights are in accordance with the areas of
expertise for members of each cluster. Table 5 lists releases and their
corresponding clusters together with the highlighted release contents.

Comparing the release contents and the cluster areas of expertise, these two
subjects are mostly matched on release topics and cluster’s working contents.
For example, cluster No.4 covers from December 3, 2010 to January 31, 2011,
which occurs before release v2.3. Participants in cluster No.4 have areas of
expertise relevant to sound, media, and kernel-video, which match the release
contents of new multimedia, APIs for native audio, and mixable audio effects
in v2.3; We can also find that 4 out of 7 developers in cluster No.4 have
worked on the kernel when the linux kernel was upgraded to 2.6.36 in Android
v2.3.

Cluster No.3 was centered around the releases of v2.2.1 and v2.2.2 (January
18, 2011 and January 22, 2011 respectively). Release 2.2.1 contained security
updates and performance improvement; participants in cluster No.3 are
specialized mostly on kernels or platforms. This occurs in cluster No.1 and
its corresponding release v2.2 as well.

Our conclusion is that participants’ work is relevant to areas of expertise
associated with clusters, and at the same time, the clusters and participation
tends to be correlated with releases. This further validates our answer to RQ4
that developers tend to work as groups on specific projects or issues they are
specialized, and their centrality patterns are related to the occurrences of
projects or issues.

##  6 Limitations

In this study we explicitly trust that the same account of email addresses,
i.e., the part before “@”, belongs to the same bug participant. With the given
semi-anonymous email addresses in Android bug repository, we pruned the part
starting from “….” and kept the front part as the names of bug participants.
However, it is possible that some common names share the same start string.
For example, “Benjamin Franzke”, “Benjamin Tissoires” and “Benjamin Romer”
have the same first name. We cannot distinguish these names with the email
address “Benjamin@XXX”. Besides, some of the email addresses start with a
simple letter which is ambiguous identifying a person, while we analyze the
results without excluding such data.

We validate our analysis based on the assumption that the types and projects
of submitted files reflect the areas of expertise that the developers are
specialized in. Hence, we tagged the participants with the techniques
according to their submitted files in the Android git. However, there could be
inconsistency between the techniques and the submitted files.

Our manual inspection increased the validity of the results, but it still
relied on the authors judgment, interpretation and potential bias.

##  7 Conclusion and Future Work

In this paper, we mined the Android bug repository and studied the data of
2010 and 2011. We combined overlapping time windows with social network
analysis in order to analyze the participants interactions within the Android
bug repository, as part of the Android open source community.

We conducted a thorough case study of the bug reporter activity within the
Android bug repository with our method. We analyzed the temporal evolution of
the Android bug reporting community both globally and locally. We found that
most minor or major releases lead to high betweenness centrality in general.
We found and explained sharp changes of participants’ betweenness values and
we inspected three activity patterns for the participants. Also, we found out
that participants tend to get clustered into groups. Then, we validated these
results by manually inspecting the Android version control system (git) and
the Android release history highlights. We validated the three activity
patterns of bug participants as well as their corresponding reasons. For
participants who were clustered in same groups in our plots, they showed
interest in a set of similar topics as we inspected in our validation.

Thus we conclude that by combining the SNA with sliding windows, we were able
to find many local interactions that would be lost in a global analysis. The
sliding windows make these local collaborations more visible, instead of
drowning them out in a global analysis. In this case, we can get a more
accurate knowledge about participants’ working patterns as well as their group
working. Furthermore, we validated our findings by inspecting other
repositories to confirm that the local behaviour occurred and was of
relevance. This work could be used by managers and researchers to produce
project dashboards, and automated project status reports.

Future work includes applying the approach in this paper to other open source
projects’ repositories in order to improve its generality. We want to further
validate if our overlapping time windowing SNA plots are trustworthy enough to
depict the actual develop processes of various projects.

## References

  * [1] K. F. S. Wasserman, Social network analysis : methods and applications.  Cambridge ; New York : Cambridge University Press, 1994.
  * [2] C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan, “Mining email social networks,” in Proceedings of the 2006 international workshop on Mining software repositories, MSR ’06, (New York, NY, USA), pp. 137–143, ACM, 2006.
  * [3] A. Meneely and L. Williams, “Socio-technical developer networks: should we trust our measurements?,” in Proceeding of the 33rd international conference on Software engineering, ICSE ’11, (New York, NY, USA), pp. 281–290, ACM, 2011.
  * [4] C. Bird, D. Pattison, R. D’Souza, V. Filkov, and P. Devanbu, “Latent social structure in open source projects,” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, SIGSOFT ’08/FSE-16, (New York, NY, USA), pp. 24–35, ACM, 2008.
  * [5] G. Madey, V. Freeh, and R. Tynan, “The open source software development phenomenon: An analysis based on social network theory,” in Proceedings of the Americas Conference on Information Systems (AMCIS 2002), (Dallas, Texas), pp. 1806–1813, 2002.
  * [6] A. Sureka, A. Goyal, and A. Rastogi, “Using social network analysis for mining collaboration data in a defect tracking system for risk and vulnerability analysis.,” in ISEC (A. Bahulkar, K. Kesavasamy, T. V. Prabhakar, and G. Shroff, eds.), pp. 195–204, ACM, 2011.
  * [7] Y. Kamei, S. Matsumoto, H. Maeshima, Y. Onishi, M. Ohira, and K. ichi Matsumoto, “Analysis of coordination between developers and users in the apache community.,” in OSS (B. Russo, E. Damiani, S. A. Hissam, B. Lundell, and G. Succi, eds.), vol. 275 of IFIP, pp. 81–92, Springer, 2008.
  * [8] M. L. Bernardi, G. Canfora, G. A. D. Lucca, M. D. Penta, and D. Distante, “Do developers introduce bugs when they do not communicate? the case of eclipse and mozilla.,” in CSMR (T. Mens, A. Cleve, and R. Ferenc, eds.), pp. 139–148, IEEE, 2012.
  * [9] R. Abreu and R. Premraj, “How developer communication frequency relates to bug introducing changes,” in Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops, IWPSE-Evol ’09, (New York, NY, USA), pp. 153–158, ACM, 2009.
  * [10] M. Pinzger, N. Nagappan, and B. Murphy, “Can developer-module networks predict failures?,” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, SIGSOFT ’08/FSE-16, (New York, NY, USA), pp. 2–12, ACM, 2008.
  * [11] N. Bettenburg and A. E. Hassan, “Studying the impact of social structures on software quality,” in Proceedings of the 2010 IEEE 18th International Conference on Program Comprehension, ICPC ’10, (Washington, DC, USA), pp. 124–133, IEEE Computer Society, 2010.
  * [12] Y. K. E. Shihab and P. Bhattacharya, “Mining Challenge 2012: The Android Platform,” 2012.
  * [13] U. Brandes and T. Erlebach, Network Analysis: Methodological Foundations.  Springer, 2005.
  * [14] A. Hindle, M. W. Godfrey, and R. C. Holt, “What’s hot and what’s not: Windowed developer topic analysis.,” in ICSM, pp. 339–348, IEEE, 2009.
  * [15] L. C. Freeman, “A set of measures of centrality based on betweenness,” Sociometry, vol. 40, pp. 35–41, March 1977.
  * [16] R. A. Hanneman and M. Riddle, Introduction to social network methods.  University of California, Riverside, Riverside, CA, 2005.
  * [17] Y. H. Kidane and P. A. Gloor, “Correlating temporal communication patterns of the eclipse open source community with performance and creativity,” Comput. Math. Organ. Theory, vol. 13, pp. 17–27, Mar. 2007.
  * [18] B. P. T. Iba, K. Nemotob and P. Gloor, “Analyzing the Creative Editing Behavior of Wikipedia Editors: Through Dynamic Social Network Analysis,” vol. 2, no. 4, pp. volume = 13, number = 1, month = mar, year = 2007, issn = 1381–298X, pages = 6441–6456, year = 2010.
  * [19] J. B. MacQueen, “Some methods for classification and analysis of multivariate observations,” in Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability (L. M. L. Cam and J. Neyman, eds.), vol. 1, pp. 281–297, University of California Press, 1967.
  * [20] G. Robles and J. M. Gonzalez-Barahona, “Developer identification methods for integrated data from various sources,” in MSR ’05: Proceedings of the 2005 international workshop on Mining software repositories, 2005.
  * [21] M. Goeminne and T. Mens, “A comparison of identity merge algorithms for software repositories,” Science of Computer Programming, Dec. 2011.