#  On the Impact of Language Selection for Training and Evaluating Programming
Language Models

Jonathan Katzy  Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>Maliheh Izadi  Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>Arie van Deursen  Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

The recent advancements in Transformer-based Language Models have demonstrated
significant potential in enhancing the multilingual capabilities of these
models. The remarkable progress made in this domain not only applies to
natural language tasks but also extends to the domain of programming
languages. Despite the ability of these models to learn from multiple
languages, evaluations typically focus on particular combinations of the same
languages. In this study, we evaluate the similarity of programming languages
by analyzing their representations using a CodeBERT-based model. Our
experiments reveal that token representation in languages such as C++, Python,
and Java exhibit proximity to one another, whereas the same tokens in
languages such as Mathematica and R display significant dissimilarity. Our
findings suggest that this phenomenon can potentially result in performance
challenges when dealing with diverse languages. Thus, we recommend using our
similarity measure to select a diverse set of programming languages when
training and evaluating future models.

###### Index Terms:

language model, code representation, programming language, multilingual model,
Transformer, pretrained model, transfer learning

##  I Introduction

Language Models (LMs) have shown great capabilities in both Natural Language
Processing (NLP) and source code domains [1, 2, 3, 4, 5, 6, 7, 8]. Initially,
studies focused on the performance of LMs on a single language, whereas
currently, the emphasis is on achieving optimal performance across various
languages simultaneously [9, 10, 11, 12, 13, 14, 15]. The current direction of
research suggests that pretraining LMs on multiple languages benefits per-
language performance in many cases [16]. However, there is a lack of clear
rationale or justification for the specific selection of languages used in
these studies. Furthermore, research has shown that models perform better on
certain programming languages and worse on others [11, 13, 10].

In the realm of programming, source code tokens are predominantly written in
English. These tokens, especially user-defined ones like function and variable
names, contain critical information about the code’s functionality. Thus, it
is reasonable to anticipate that machine learning models should learn from and
perform equivalently across different languages. However, significant
performance discrepancies persist among LMs working on different languages.
This suggests that these performance variations might originate from differing
token representations learned by the models.

In this investigation, we aim to answer the following question: How similar
are the representations of various programming languages learned by
Programming Language Models (PLMs)? To answer this question, we first identify
a set of prevalent languages typically employed in the training and evaluation
of PLMs, as well as a number of frequently disregarded ones in the same
context. This enables us to include a diverse set of programming languages in
our study. We then leverage a prominent PLM, the CodeBERT model [4], to obtain
representations of code tokens. Next, we calculate language similarity by
examining the set of tokens shared across all evaluated languages. This
approach allows us to establish a quantitative basis for language selection
for PLM research.

Our findings reveal a persistent difference in representations across both the
multilingual pretrained setting and the non-pretrained monolingual setting. To
validate our results, we conduct a comparison between the identified
similarities and the reported performance of other models. We find that models
evaluated on Scala and Ruby perform worse than when they are evaluated on
Java, Python, C, C++, Go, and JavaScript [11]. This difference in performance
overlaps with the relative similarities we have found in these languages.

The implications of this work are twofold. We have shown that there is an
intrinsic difference in the representations of programming languages learned
by CodeBERT implying that there is an inherent difference in how PLMs use
inputs from different languages. This finding can have a large impact on areas
such as multilingual models and transfer learning. Theoretically, we have
proposed a method for validating language selection in the context of
multilingual model outcomes, as well as a foundational understanding to guide
language choice for multilingual investigations. From a practical standpoint,
we provide a list of languages with akin representations, indicating similar
performance potential. This enables researchers to make informed choices in
selecting programming languages aligned with their specific experimental
design, especially when aiming to leverage transfer learning techniques across
languages.

Finally, we plan to extend this investigation to include more languages to
gain a more comprehensive mapping of available languages. We also want to
extend this investigation to other architectures and training goals. In this
work, we used an encoder-based architecture that used the masked language
modeling task [4], we want to extend the work to other tasks such as code
infilling [14, 17, 10] and architectures such as the T5 models [15]. Lastly,
we release our replication package [18].

##  II Approach

TABLE I: Selected languages and inclusion criteria Language | Inclusion criteria | Files | Total Tokens
---|---|---|---
Assembly | Unique syntax with a limited vocabulary | 100,000 | 364,776,405
C | Widely used general-purpose programming language | 100,000 | 326,871,237
COBOL | The language often present in legacy systems, with a very unique syntax | 2,978 | 10,613,233
C++ | Widely used general-purpose programming language, close to Java and C | 100,000 | 368,090,173
Cuda | Domain specific application of C++ | 58,355 | 283,624,967
Emacs Lisp | Domain-specific application of Lisp | 54,768 | 188,661,262
Fortran | Scientific computing language, with similar syntax to Julia and Ruby | 100,000 | 607,478,891
Go | Domain-specific language with elements from C, C++, Python, and Ruby | 100,000 | 232,054,204
HTML | Domain-specific language, with unique syntax | 100,000 | 723,969,345
Java | Widely used general-purpose programming language | 100,000 | 183,040,204
JavaScript | Widely used domain-specific programming language | 100,000 | 325,109,387
Julia | New emerging scientific computing language | 100,000 | 242,836,338
Kotlin | Mixture of Java and JS elements but less verbose | 100,000 | 111,578,961
Lisp | General purpose list-based programming language | 100,000 | 832,184,093
Mathematica | Mathematical computing language with unique features | 26,895 | 1,035,010,885
Python | General purpose programming language, with semantic whitespace | 100,000 | 237,414,388
R | Scientific computing language | 39,194 | 154,180,798
Ruby | General purpose language with syntax similar to Python and Julia | 100,000 | 93,200,451
Scala | JVM-based language with syntactic elements from JavaScript and C++ | 100,000 | 141,672,916
WebAssembly | Domain-specific emerging list-based language | 5,359 | 59,809,452

In our approach, we first train a set of BERT-style models according to the
methodology of CodeBERTScore [19]. These models will be used to calculate the
representations of tokens. Then we calculate the similarity of languages using
the tokens that are present in all languages.

#### Language Selection and the Dataset

The training of our models utilizes “The Stack” [20] dataset comprising $358$
programming languages, with code availability governed by permissive licenses.

We aim to include a diverse set of languages to encompass as many different
scenarios as possible. There are four main criteria for selection. First, we
look at what languages are commonly used in Machine Learning for Software
Engineering research. Then we look for a variety of grammar and programming
paradigms. We also look at the expected use-case of the languages. Finally, we
look at including both high- and low-resource languages. From these selection
criteria, we curated the list of $20$ languages. We include the full list of
selected languages together with the reason for inclusion in Table I. Note
that we do not include some mainstream languages, such as C#, Matlab, and Rust
as their niche within the selection criteria has been sufficiently filled.
Whereas we include some languages that are not often used in the training of
such models, i.e., Emacs Lisp (ELisp), Lisp, WebAssembly, COBOL, and Fortran
because their grammar is very dissimilar to popular languages. This makes them
good subjects to analyze the potential differences in representations.

The dataset that we use for training our models and evaluating the language
similarities consist of the first $100,000$ files from “The Stack” [20] for
each language. Note that for low-resource languages where data is scarce
(fewer than $100,000$ files), we used all available files in The Stack
dataset. This information is included in Table I. When training the CodeBERT
models, we use the first $90\%$ of the files as training data, and the last
$10\%$ as a test set. We calculate the final similarities with representations
of samples from the test set.

Note that when selecting common tokens across languages, we exclude comments
from consideration. This ensures that we only compare tokens integral to the
source code. However, while training the models and inferring the
representation of tokens, we do include comments in the input, as they can
provide valuable information for the model.

#### Language Representation

The individual languages have $47$K, $5$K, and $27$K tokens as maximum,
minimum, and average tokens, respectively. To represent a language, we
identify tokens that are present in all languages (excluding comments). This
generates common $2,718$ tokens for which we generate a vector representation
denoted as $\mathbf{N}$. We select common tokens by tokenizing all files, and
counting the presence of each token in a language.

We only use tokens that are present in all languages when making a comparison.
This is because using a token that is not present in a particular language
would not allow us to calculate a similarity for that token.

We transform each token into a numerical vector that captures its semantic
meaning or contextual information.

For every token, we have a set of samples, $\mathbf{M}$. Samples correspond to
the same token but are obtained from different occurrences within a given
language. This enables us to generate the token’s representation within a
language as a set of encodings, denoted as $T$. Finally, we represent each
language $\mathcal{L}$ as a set of the token set $T$ as shown by Equation 1,
$l$ is the maximum left context, and $r$ is the maximum right context.

$\begin{split}T&=\\{CodeBERT(t_{n-l},...,t^{m}_{n},...,t_{n+r})|m\in\mathbf{M}\\}\\\
\mathcal{L}&=\\{T^{\mathcal{L}}_{N}|N\in 0...\mathbf{N}\\}\end{split}$ (1)

#### Language Similarity

We use a similarity score to quantify the degree of similarity between two
languages that serve as a measure of their relatedness or resemblance. To that
end, we employ the cosine similarity between the embeddings of each token in
the respective languages.

Next, we define a similarity function for two sets of vectors. This allows us
to assess the similarity between the vector representations of tokens in
different languages. We order a vector pair such that the vectors with the
highest cosine similarity are matched up. This will give the highest
similarity score possible for the chosen tokens. Once we have the definition
of the similarity between two sets of vectors we extend this to the similarity
between two languages as shown in Equation 2. We use the mean to aggregate the
similarities of the sets of vectors and arrive at a single value that gives
the similarity between two languages.

$\begin{split}sim(t^{a}_{x},T^{b})&=max\\{cosim(t^{a}_{x},t^{b}_{y})|t^{b}_{y}\in
T^{b}\\}\\\ sim(T^{a},T^{b})&=\sum^{X}_{x=0}\frac{sim(t^{a}_{x},T^{b})}{X}\\\
sim(\mathcal{L}^{a},\mathcal{L}^{b})&=\sum^{\mathbf{N}}_{n=0}\frac{sim(T^{a}_{n},T^{b}_{n})}{\mathbf{N}}\end{split}$
(2)

The ordering of a vector pair is essential to the collected results. First, a
form of ranking is essential as it allows for the reproduction of the results
as the cosine similarity is only defined for two vectors. Furthermore, using
the maximum similarity as an ordering criterion gives us an upper bound on the
similarities of languages. This means there is no combination of tokens that
would give a higher similarity score, so all results shown are the best-case
scenario for similarity. Additionally, it minimizes the effect of potential
synonyms within the data. Assuming there are two distinct representations for
a given token, this ranking compares the most similar representations between
languages.

#### Model Training

All the models we train for the experiments have the same architecture, and
training setup, i.e., CodeBERT base
architecture.111https://huggingface.co/microsoft/codebert-base-mlm We train
the models in two settings; in the first setting, we reinitialize all the
weights from scratch for each language. In the second setting, we use the
provided pretrained model as a starting point and finetune it on target
languages. The pretrained version is trained on Python, Java, JavaScript, PHP,
Ruby, and Go. The models were trained using the CodeBERT training setup [4],
for 100,000 steps.

#### Experiments

To evaluate the similarity of the learned representations by the CodeBERT
model, we conduct two experimental setups. In the first experiment, we employ
a distance metric to calculate the pairwise distance between all languages.
This analysis will enable us to identify the languages that exhibit the
greatest similarity in terms of how they represent the same tokens. Next, we
examine the similarity of a language to itself. This verifies that the
representations are consistent within a language. We use the same metric as
mentioned earlier but exclude the token from being compared to itself in
Equation 2. Computing the similarities among a token representation within one
language and comparing it to the representation of the same token in another
language serves a dual-purpose function. Firstly, this investigates the
consistency of token representations computed by CodebBERT for identical
tokens. Maintaining this internal consistency within a single language is
crucial when evaluating its correlations with other languages. Secondly, this
enables us to distinguish whether there exists a subset of highly similar
tokens responsible for the entirety of the linguistic correlations, or whether
these similarities across languages can be attributed to a more distributed
pattern of resemblance among numerous tokens. Finally, we conduct all
mentioned experiments in both a multilingual pretrained setting, as well as a
non-pretrained (monolingual) setting to ensure that the differences in
representations are consistent.

##  III Results

Initially, we analyze the outcomes obtained from the distance metric. Figure
1(a) presents pairwise similarity of all languages using the non-pretrained
models while Figure 1(b) shows the same metric for the pretrained models. In
both figures, the languages are sorted by average similarity to all other
languages. A darker shade of blue indicates that the languages are more
similar.

###  III-A Cross-lingual Similarity Results

(a) Non-pretrained setting

(b) Pretrained setting

Figure 1: Pairwise similarity between the representation of all languages

Upon examining the non-pretrained representations of common languages such as
C++, Python, Go, Java, and JavaScript, we find that these languages exhibit
the closest proximity to each other in terms of their representations.
Notably, this similarity is observed without any transfer learning from other
languages. When considering derivative languages such as Cuda and Scala, we
notice that their representations remain relatively close, but a slight
increase in dissimilarity becomes apparent. When examining the most dissimilar
languages, we find that both R and Mathematica exhibit significant differences
compared to all other languages. This observation is intriguing since R and
Mathematica are primarily utilized for mathematical and statistical purposes,
which distinguishes them from the more commonly used programming languages.
Upon comparison with other languages, we observe that list-based languages,
namely WebAssembly, Lisp, and ELisp exhibit distinct dissimilarities from
other languages. However, they do not demonstrate a significantly higher
degree of similarity among themselves compared to other languages. This is a
characteristic previously observed in derivative languages like C++ and Cuda.
Additionally, we observe that both COBOL and Assembly (languages with unique
syntax), also stand significantly apart from all other languages.

For pretrained models, we detect the same trends as previously described.
Although the languages, in general, exhibit greater overall similarity, as
evident from the change in the scale of Figure 1(b), there remains a
discernible difference in the representation of tokens. It is worth noting
that in the non-pretrained setting, the difference between the highest and
lowest similarity score is $0.19$, while in the pretrained setting, this
difference is $0.23$. This indicates that although the similarities are closer
in the pretrained setting, the differences in similarities remain comparable.
Note that due to the pretraining on languages such as Python, Java,
JavaScript, PHP, Ruby, and Go, we observe that these languages become more
similar to each other after fine-tuning. However, COBOL, Assembly, ELisp,
Lisp, Mathematica, and WebAssembly still maintain their distinctiveness and
remain distant from all other languages. Another noteworthy observation is
that Go, as a result of the pretraining, exhibits increased proximity to a
greater number of languages compared to the non-pretrained setting. This
indicates that the pretraining process has influenced the language
representation, making Go more similar to other languages in the fine-tuned
model.

###  III-B Self-Similarity Results

(a) Non-pretrained setting

(b) Pretrained setting

Figure 2: Self-Similarity of languages

Next, we examine the self-similarity of tokens within each language. The self-
similarity of non-pretrained representations can be seen in Figure 2(a), while
the self-similarity of pretrained representations is shown in Figure 2(b).
Upon analyzing the results, it is evident that when working with a pretrained
model, the representation of languages becomes more consistent. This
observation is supported by two key indicators: the increase in similarity
scores and the decrease in variance. Both of these trends reflect the improved
consistency and reduced variability in language representations when utilizing
a pretrained model.

Our analysis reveals marginally greater token similarity within individual
languages compared to inter-language similarities. Notably, Mathematica shows
the highest self-similarity in both pretrained and non-pretrained settings. It
also is the least similar to all other languages in the non-pretrained
setting, and second least in the pretrained setting. These observations
emphasize a consistent divergence between Mathematica’s representations and
those of other languages.

When examining the representations of R, Mathematica, and ELisp, we find that
they exhibit consistent patterns similar to other languages in both the
monolingual and pretrained settings. This observation confirms the notion that
these languages’ representations have been learned to be distinct from the
more commonly used languages in large language models.

Additionally, we observe that both COBOL and WebAssembly exhibit inconsistency
in their representations. We attribute this inconsistency to a potential lack
of sufficient training data for these languages, which can result in the
models underfitting their representations and failing to capture their
inherent linguistic patterns effectively.

The underfitting of two languages provides further insight into inter-language
similarities and reinforces the need for meticulous language selection.
Reviewing non-pretrained model outcomes, Mathematica exhibits greater
dissimilarity to other languages compared to both WebAssembly and COBOL, with
R and ELisp displaying more dissimilarity than COBOL. In the pretrained
setting, no language surpasses COBOL in dissimilarity between languages, yet
ELisp, R, Assembly, and Mathematica show greater differences compared to
WebAssembly. This suggests that certain languages produce representations so
different from others, that they surpass representations that have not yet
converged in dissimilarity.

##  IV Discussion

###  IV-A Implications and Future Work

The implications of our findings lie in the wider field of PLMs. We compare
the similarities we found between languages to the multilingual performance of
a set of PLMs evaluated by Xu et al. [11] in their study. We see that the
trends seen in similarities are mirrored in the performance of the models.
Ruby and Scala, the most dissimilar of the languages evaluated by Xu et al.
[11] consistently performed worst, while C++, C, Java, Python, and JavaScript
had closer and better performances. Furthermore, we note that the languages
used in the evaluation of many models are the same languages which we have
found to be the most similar to one another. This can have grave implications
for the research in this field.

First, when looking at the cross-lingual performance of PLMs, the choice of
language will affect the reported performance. Our results help to select
representative languages in future studies. Second, when investigating the
contributions of the language-neutral and language-specific elements of the
representations learned by PLMs it will be beneficial to choose a set of
languages that are most dissimilar. This should show the greatest difference
in the language-specific representations and make the language-neutral
elements more evident.

We have shown that there is an inherent difference in the representations of
programming languages for an encoder model trained on the Masked Language
Model task. In the future, we aim to extend the investigation to other models
such as StarCoder [10], or InCoder [14], which will use a generative training
setup, allowing us to work at a higher granularity than token level, while
also including other architectures than only encoder models. Finally, an
extension to T5[15] would allow us to evaluate tasks focussed more on code
understanding, while maintaining the code generation task also present in
StarCoder [10] and InCoder[14].

###  IV-B Threats to Validity

Concerning the architecture of the models used, prevalent PLMs generally fall
into different categories. Recent research has indicated the significance of
incorporating future context, particularly in models that employ span masking
techniques [14, 15, 21]. In our study, we focus on single token
representations, which restricts models that mask spans to operate as BERT-
style Masked LMs. Concerning the size of the model, using large models is
prohibitively expensive as we need to train two models for every language. The
current training procedure for $20$ languages took $300$ hours of computing
time. This also limits the number of languages analyzed.

Lastly, the selection of the similarity metric potentially affects our
findings as well. We employed the cosine similarity, a widely accepted metric
in token representations [22, 19], however, other choices can be investigated
as well.

##  V Related Work

NLP researchers have extensively investigated the multilingual and knowledge
transfer performance of encoder models such as Multilingual-BERT [23]. As for
the similarity measurement, the CodeBERTScore model [19] has demonstrated the
potential of using a language model for accuracy assessment and code
translations, proposing a new accuracy metric to address limitations in the
existing BLEU and ROUGE metrics. The authors fine-tune a CodeBERT model [4] to
represent code, enabling the use of the representations as an accuracy metric
for code-to-code tasks. Additionally, CodeBERTScore aligns better with human
annotators’ ratings of code quality, and high-scoring snippets are more likely
to be correct programmatically.

##  VI Conclusion

In this study, we showed that there is a consistent difference in the
representations of the same tokens learned in different languages for BERT-
style models. The difference existed in both monolingual and multilingual
(finetuned) settings. We identified several languages that show a high degree
of similarity, namely, C, C++, Java, and Python. These languages also
correspond to languages commonly used when evaluating code PLMs [11].
Furthermore, we specified the languages that learn a consistently different
representation for the same tokens, namely, Mathematica and R. This raises
inquiries into the performance of PLMs on dissimilar languages, and it calls
into question how much these dissimilar languages could benefit from transfer
learning. In future, we will investigate the effects of these differing
representations on downstream tasks, as well as investigate the
representations found by different model architectures.

## References

  * [1] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [2] OpenAI, “Introducing chatgpt,” 2022. [Online]. Available: https://openai.com/blog/chatgpt
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [4] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–1547. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.139
  * [5] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “UniXcoder: Unified cross-modal pre-training for code representation,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7212–7225. [Online]. Available: https://aclanthology.org/2022.acl-long.499
  * [6] M. Izadi, R. Gismondi, and G. Gousios, “Codefill: Multi-token code completion by jointly learning from structure and naming sequences,” in _Proceedings of the 44th International Conference on Software Engineering (ICSE)_. ACM, 2022, p. 401–412.
  * [7] T. van Dam, M. Izadi, and A. van Deursen, “Enriching source code with contextual data for code completion models: An empirical study,” in _Proceedings of the 19th International Conference on Mining Software Repositories (MSR)_ , 2023.
  * [8] A. Al-Kaswan, T. Ahmed, M. Izadi, A. A. Sawant, P. Devanbu, and A. van Deursen, “Extending source code pre-trained language models to summarise decompiled binaries,” in _Proceedings of the 30th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2023.
  * [9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374
  * [10] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. Stillerman, S. S. Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, N. Fahmy, U. Bhattacharyya, W. Yu, S. Singh, S. Luccioni, P. Villegas, M. Kunakov, F. Zhdanov, M. Romero, T. Lee, N. Timor, J. Ding, C. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra, A. Gu, J. Robinson, C. J. Anderson, B. Dolan-Gavitt, D. Contractor, S. Reddy, D. Fried, D. Bahdanau, Y. Jernite, C. M. Ferrandis, S. Hughes, T. Wolf, A. Guha, L. von Werra, and H. de Vries, “Starcoder: may the source be with you!” 2023. [Online]. Available: https://arxiv.org/abs/2305.06161
  * [11] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ , ser. MAPS 2022\. New York, NY, USA: Association for Computing Machinery, 2022, p. 1–10. [Online]. Available: https://doi.org/10.1145/3520312.3534862
  * [12] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU, “CodeXGLUE: A machine learning benchmark dataset for code understanding and generation,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021. [Online]. Available: https://openreview.net/forum?id=6lE4dQXaUcb
  * [13] A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 1433–1443.
  * [14] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” in _The Eleventh International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=hQwb-lbM6EL
  * [15] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 8696–8708. [Online]. Available: https://aclanthology.org/2021.emnlp-main.685
  * [16] F. Chen, F. H. Fard, D. Lo, and T. Bryksin, “On the transferability of pre-trained language models for low-resource programming languages,” in _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension_ , ser. ICPC ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 401–412. [Online]. Available: https://doi.org/10.1145/3524610.3527917
  * [17] L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey _et al._ , “Santacoder: don’t reach for the stars!” _arXiv preprint arXiv:2301.03988_ , 2023.
  * [18] J. Katzy, M. Izadi, and A. van Deursen, “On the Impact of Language Selection for Training and Evaluating Programming Language Models,” 2023. [Online]. Available: https://doi.org/10.5281/zenodo.8282909
  * [19] S. Zhou, U. Alon, S. Agarwal, and G. Neubig, “Codebertscore: Evaluating code generation with pretrained models of code,” 2023. [Online]. Available: https://arxiv.org/abs/2302.05527
  * [20] D. Kocetkov, R. Li, L. B. allal, J. LI, C. Mou, Y. Jernite, M. Mitchell, C. M. Ferrandis, S. Hughes, T. Wolf, D. Bahdanau, L. V. Werra, and H. de Vries, “The stack: 3 TB of permissively licensed source code,” _Transactions on Machine Learning Research_ , 2023. [Online]. Available: https://openreview.net/forum?id=pxpbTdUEpD
  * [21] A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer, “Cm3: A causal masked multimodal model of the internet,” _ArXiv_ , vol. abs/2201.07520, 2022. [Online]. Available: https://arxiv.org/abs/2201.07520
  * [22] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in Neural Information Processing Systems_ , C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds., vol. 26. Curran Associates, Inc., 2013. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf
  * [23] T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4996–5001. [Online]. Available: https://aclanthology.org/P19-1493


Journal of Data Science Vol.

A Method for Evaluating Options for Motif Detection in Electricity Meter Data

Ian Dent1, Tony Craig2, Uwe Aickelin1, Tom Rodden1

1School of Computer Science Nottingham University

2The James Hutton Institute

> Abstract: Investigation of household electricity usage patterns, and
> matching the patterns to behaviours, is an important area of research given
> the centrality of such patterns in addressing the needs of the electricity
> industry. Additional knowledge of household behaviours will allow more
> effective targeting of demand side management (DSM) techniques.
>
> This paper addresses the question as to whether a reasonable number of
> meaningful motifs, that each represent a regular activity within a domestic
> household, can be identified solely using the household level electricity
> meter data.
>
> Using UK data collected from several hundred households in Spring 2011
> monitored at a frequency of five minutes, a process for finding repeating
> short patterns (motifs) is defined. Different ways of representing the
> motifs exist and a qualitative approach is presented that allows for
> choosing between the options based on the number of regular behaviours
> detected (neither too few nor too many).
>
> Key words: Motif detection, Clustering, Electricity Usage

##  1 Introduction

The investigation of household electricity usage patterns is an important area
of research given the centrality of such patterns in directly addressing the
needs of the electricity industry, both now and in the future. Being able to
identify patterns is essential for effective implementation of demand side
management (DSM) techniques which itself is necessary to allow the electricity
industry to meet the upcoming challenges.

This paper addresses the question as to whether a reasonable number of
meaningful motifs, that represent regular activities within a domestic
household, can be identified solely using the electricity meter data. The
intention is to identify activities that households may do regularly (e.g. the
washing of clothes or cooking of the evening meal) and not to identify the
usage of particular appliances (e.g. toaster, kettle). The actual physical
activity (e.g. washing) will not be determined from the electricity data but
the research will determine whether repeating activities can be identified).

The paper adopts an approach of symbolising the electricity meter data and
then finding repeating patterns within the symbolised data. Various ways of
calculating the symbolised data and various parameter settings are explored to
identify the most appropriate parameters for consistently finding repeating
activities within the household. The goal is to, firstly, determine whether
the data sampled at five minute frequency is sufficient to discover meaningful
motifs, and, secondly, to determine the parameter settings that provide the
most useful motifs.

The remainder of this paper is structured as follows. Section 2 provides
information on the need for changes in analysing electricity usage, how demand
side management may address some of the needs, and current work on using
motifs to find particular activities. Section 3 provides the details on the
data used for analysis, the method of finding motifs, and an approach to
evaluating whether “sufficient” motifs have been found. Section 4 provides the
analysis results and Section 5 draws conclusions from the analysis and
suggests future work.

##  2 Background

The electricity market in the UK is under various pressures. Some are due to
the history and current design of the National Grid and others are arising
from worldwide trends, such as the need to reduce carbon emissions and the
declining sources of hydro-carbon fuels Rhodes (2010). New technologies, such
as electric cars needing household charging facilities, will be much more
common Poyry (2010). The information monitored in the home will grow rapidly,
particularly with the roll-out of Smart Meters planned for UK completion by
2020 DECC (2011). In addition, the drive to change the mix of electricity
generation technologies to reduce greenhouse gas emissions, the desire to
reduce carbon dioxide by changing non-electric demand such as gas central
heating to the electricity network, and the impact of climate change on
altering electricity demand and the greater occurrence of extreme weather
events will increase the difficulties in providing a stable and cost effective
supply without better modelling of the patterns of consumption within the
grid.

###  2.1 Demand Side Management

The electricity market in the UK is undergoing dramatic changes. Legal, social
and political drivers require a transformation of existing practices. In
particular, the change of sampling of electricity usage from a 3 monthly
billing cycle to a 30 minute sampling period using smart meters, alters the
degree of understanding of households’ behaviour that is possible DECC (2009).

Historically, electricity supply in the UK has been driven by a desire to
provide sufficient supply to match the predicted demand, and to avoid
shortages and blackouts. The restrictions on supply due to cost, changing
political opinions regarding generation technologies, and international
obligations to meet carbon reduction targets, means that, in the future, the
emphasis will need to change to demand more closely matching the available
supply. One approach to addressing this issue is the application of demand
side management (DSM) techniques to achieve changes in consumer behaviour.
River (2005) defines DSM as “systematic utility and government activities
designed to change the amount and/or timing of the customer’s use of
electricity” for the collective benefit of society, the utility company, and
its customers. As domestic usage represents 30% of the total electricity usage
MacLeay et al. (2013), understanding and influencing domestic usage will be a
significant part of the solution.

Research in the area of demand side responses has been ongoing for a number of
years. For example, Newborough and Augood (1999), in 1999, demonstrated the
ability to reduce UK household peak usage of electricity by up to 60% by an
assortment of interventions including the replacement of some appliances by
gas powered equivalents. Chamberlin (1993) also showed that demand side
management was being seriously investigated over 20 years ago.

The UK electricity market will allow for more targeted and more complicated
tariff offers for customers to provide many benefits including maximising the
efficiency of the supply process. DECC (2009) shows that the provision of
Smart Meters will allow greatly increased analysis of a customer’s electricity
usage and provide the ability to make customised offers on pricing and
availability to change customer behaviour (for example, to minimise usage
during peak periods) or to increase efficiencies in the electricity supply
chain in meeting the predicted demand Ofgem (2010).

###  2.2 Motifs

The ability to use the meter data (as collected by smart meters) to discern
particular activities can be useful to the electricity supplier in
implementing demand side management techniques. If a particular pattern can be
interpreted as a particular activity such as cooking the evening meal, then
appropriate incentives or penalties can be offered/imposed on the particular
household to gain a change in behaviour.

A motif is defined as a previously unknown, frequently occurring pattern
within a stream of data. It is distinguished from known patterns which can be
found within a data set by a number of well researched methods Lin et al.
(2002). In many cases the repeating patterns are not exactly the same and the
power of a motif finding technique is in its ability to find similar patterns
that, in real life, represent the same thing. For example, when considering
electricity meter data, similar patterns may be discernible that represent the
electricity usage during cooking.

The electricity meter data stream from a household can be considered as a
graph of usage against time and regular activities (e.g. cooking) can be seen
as similar shaped usage patterns. Short patterns that repeat within the data
are defined as “motifs” and detection of these motifs, and their timing, can
lead to understanding of behaviour within the household.

The pattern finding problem was initially stated by Lin et al. (2002) who
distinguished between the problem of efficiently finding defined patterns
within a dataset (which the authors felt was generally solved) and the problem
of finding repeating patterns within the data which were not previously known
and which the authors named as motifs.

Previous work from Das et al. (1998) had addressed a related problem of
finding rules that relate patterns in one time series to patterns in another.
Höppner (2001) considers the problem of finding temporal relationships between
primitive patterns in time series in a generalised way. The term “primitive
patterns” can be defined as motifs.

A significant amount of work has been done in the area of DNA pattern
detection and within textual analysis which can be applied to the motif
finding problem. If real valued data (as from electricity meters) can be
represented by a series of discrete characters (letters) then the techniques
of DNA and textual analysis can be applied Castro (2010).

Meo et al. (2012) is an example of using the frequency of patterns within a
sample to derive classification rules. The regular patterns are used to
generate a probabilistic model which provides a probability that a particular
pattern will occur within a random example of the given class. This approach
allows for all patterns to be used at the same time in the classification.

Zeifman and Roth (2011) reviews the current state of Non Intrusive Appliance
Load Modelling (NIALM) which is the technology to identify individual
appliance usage from the overall electricity usage. The conclusion from this
review is that, currently, no approaches are suitable for detecting all kinds
of appliances. The focus in this paper is on finding interesting, repeating
patterns of behaviour (e.g. cooking or washing) rather than identifying the
usage of individual appliances.

Chang et al. (2010) is a typical study investigating the detection of
appliances making use of the overall power usage only - without the need for
intrusive appliance level monitoring. This study focuses on the signature of
the turning on of the appliance rather than on the total appliance usage load.
Like a lot of NIALM studies, some success has been demonstrated in detecting
particular appliances but often only in a laboratory environment where the
researchers know the actual appliances they are searching for from a short
list and/or by making use of intensive monitoring. This monitoring may be at a
very high frequency (sub-second) or by measuring multiple items (such as
reactive power). This detailed information will not be available from smart
meters rolled out across the UK and, while interesting, are seen as of little
benefit when considering the wide population.

Eagle and Pentland (2006) has introduced the concept of “reality mining” which
is defined as the sensing of complex social systems and using various
monitoring tools to detect social patterns within routines that are detected.
These apply at various time-scales including daily (e.g. getting up, eating
lunch), weekly (Saturday sports) and annual (Christmas holiday family visits).
While Eagle and Pentland (2006) makes use of mobile phones as sensors to
detect the routine activities, the concept equally applies to making use of
the household meter data readings to detect routines within the household. The
paper discusses the amount of entropy in people’s lives where “people who live
disorganised lives tend to be more variable and harder to predict”.

If a particular pattern is seen to recur then the characteristics of the
recurrence can be used to drive demand management interventions. For example,
if it is detected that a particular household undertakes the same activity at
very different times of day, it can be assumed that there may be little or no
requirement within that house to do the activity at a particular time and the
household may be open to an incentive to change to a time that is more
efficient for the overall network. However, a household that shows very
regular behaviours (e.g. cooking the evening meal at the same time every day)
may be less open to the incentive and the information can be used to help the
utility company best target their interventions.

As an example of what can be interpreted from the shapes of the electricity
usage, Figures 1 and 2 show two days of meter readings for a single house from
March 2011.

Figure 1: Example day of meter readings (household 5 on 1-Mar-2011) Figure 2:
Example day of meter readings (household 5 on 2-Mar-2011)

Certain aspects of the household behaviour can be assumed as likely from
inspection of the graphs:

  * •

The household members seem to rise from bed at the same time each day

  * •

It is likely that the household members are out of the house during the
working day returning at about 5:00 to 5:30pm

  * •

The members seem to retire to bed at about 10pm each day

  * •

There is an underlying repeating pattern throughout the day and night which is
likely to be a “always-on” device such as a fridge or freezer

Note that these assumptions are made from examining only two days of readings
but are included to give an example of the types of conclusions that the
detection of motifs can help people implementing demand response programs to
reach.

The early morning (between 6am and 8am) pattern is repeated approximately on
each day and may be a good motif that a larger scale analysis could use to
detect the activity of “rising from bed / breakfast”. Once the motif can be
detected on a significant number of days, the variation in timing of the motif
can be used to investigate variability of behaviour.

The challenge is for the motifs that the human eye can see as “similar” to be
detected automatically by a suitable process. The early morning motif can be
seen to be slightly different on each of the example days but an observer
would use their common sense and knowledge of the types of things people may
be doing in the morning (washing, making toast, making tea/coffee) to assume
that the patterns show the same type of behaviour. The challenge to be
addressed is to find a motif detection method that can automatically match the
“common sense observer”.

Some houses have very high usage for short periods which can distort the shape
of the whole day usage (e.g. see Figure 3 which suggests the use of a powered
shower). This issue can be addressed by splitting the motifs found within a
household into a number of bands (e.g. low range, medium range, high range).
If the definition of low and high ranges is taken from the range shown by the
household (i.e., 8kW in this case) then only the shower motif will be classed
as high range with any other motifs likely to be classed as low range. Instead
of taking the definition of ranges from the household minimum and maximum
usage, information on typical appliance energy requirements can be used to
define low, medium and high ranges and this will allow some of the other
motifs within the day (currently dwarfed by the shower) to be identified.

Figure 3: Example of large usage for a short period - probably a powered
shower

Other work has considered the creation of derived datasets based on the
presence (or absence) of particular patterns within the data. For example, Gay
et al. (2012) propose a method suitable for classification of data with binary
valued attributes. Their work differs from the method proposed in this paper
which deals with real valued meter readings and includes a symbolisation step
to handle approximate matching of patterns.

The detection of motifs within time series data has been applied to many
different application domains but there has been little application of motif
detection to electricity meter data from domestic households. What work that
has been done has concentrated on activities in the electricity network at a
higher level than that of domestic households (e.g. at the local substation
level).

##  3 Methodology

###  3.1 Data selection

The ongoing North East Scotland energy monitoring project (NESEMP) is
examining the relationship between different types of energy feedback and
psycho-social measures including individual environmental attitudes, household
characteristics, and everyday behaviours. As part of this ongoing project,
several hundred households are being monitored with the electricity usage
recorded every five minutes using CurrentCost monitors Craig et al. (2014).

After removing data for households with insufficient readings, the data is
loaded into a MySQL database and the readings are aligned with exact 5 minute
boundaries (e.g., 1pm, 1.05pm, etc.) by interpolation between the actual
readings. A reading at an exact 5 minute point (e.g. 1.05pm) is defined by
considering the actual readings before and after that time and by calculating
the interpolated reading such that the total usage over a longer period (e.g.,
an hour) is the same whether the interpolated readings or the original actual
readings are used Dent et al. (2012). This results in a set of 288 readings
(one for every 5 minute period in the day) for each day and for each of the
households in the database. Each day of sampling is labelled in a number of
ways such as “working day” or “summer”.

The analysis uses data collected at 5 minute intervals for 123 households from
the Spring 2011 period filtered to include just working days (weekdays which
are not public holidays). This results in 1,703,232 readings providing a total
of 5914 days of readings across all households (not all households have the
same number of days of data).

###  3.2 Approach

Different parameter settings for finding the motifs are used and the results
compared using a graphical approach that judges between sets of results
(arising from modifying one parameter) and selects the results that would be
most useful for a electricity industry professional intending to implement a
DSM intervention based around the knowledge of the motif. For example, finding
a motif that represents an activity that occurs once or twice a day within a
household and then implementing a DSM programme to modify behaviour relating
to the activity (e.g. changing the time of occurrence) would be more useful
than finding a motif that recurs many time a day (e.g. that arising from a
fridge) and which cannot be easily addressed by DSM techniques.

###  3.3 Motif Finding Method

Various motif detection algorithms are available and this work uses the SAX
(Symbolic Aggregate approXimation). Other motif finding algorithms could also
be incorporated into the proposed approach (e.g. Mueen et al. (2009)).

The SAX (Symbolic Aggregate approXimation) technique provides for symbolic
representation of time series data and thus provides access to bioinformatics
and text mining techniques Lin et al. (2007). The original work has been
extended to iSAX Shieh and Keogh (2008) which provides support for large
volumes of data. One of the techniques available through SAX is the detection
of motifs in a data stream.

No applications of the SAX technique to domestic electricity data have been
found in analysis of the literature although work on wholesale energy price
time series has been published by Mori and Umezawa (n.d.).

To find the motifs within the data, each period of interest within the day for
each household is examined by taking a moving window over the period. The
subset of the meter readings within the moving window is then converted into a
string using the SAX method and the resulting string is stored in a MySQL
database. Next, the window moves on by one reading and the conversion into a
SAX representation string is repeated. Thus, using a SAX alphabet size of six
and analysing the 4pm to 8pm period (a total of 48 x five minute readings)
will result in 43 motifs stored for each day for each household. See Figure 4
for an example of how the SAX representations are defined. The top graph shows
the five minute readings for a four hour peak period. A sliding window of six
readings (30 minutes) is taken across the peak period with the first two
windows and the last one shown. Each window is normalised within the values in
the window and then translated into the SAX alphabet representations (using an
alphabet size of five) as shown at the bottom of the diagram. The example
shows a 4 hour peak period but the approach applies to any size for the period
of analysis. The analysis in this paper uses the full day as the period of
interest.

Figure 4: Process for creating SAX representative strings

The initial step in the motif finding process is to symbolise the data. This
translates the real valued numbers obtained from the electricity meter into a
restricted number of symbolic representations (generally represented as
letters). This translation into a set number of symbols (e.g. the letters “a”
to “e” for an alphabet size of five) necessarily incorporates some degree of
approximation. As the alphabet size is restricted, the effect is to translate
readings within a particular range into a given SAX letter and thus similar,
although not identical, readings are translated into the same SAX letter. The
resulting SAX string for two motifs may be identical whereas the original
readings may only be approximately similar.

Alphabet sizes of 5, 7 and 9 are selected for analysis. An odd number is
selected so that no change in meter readings between time points (or only
slight changes due to background noise) will map to the “middle” character in
the alphabet. An even alphabet size would provide two alternatives for the
centre symbol and the selection will be driven by noise which is not the
desired behaviour.

Too large an alphabet size will lead to very few repeating motifs being found
whereas, similarly, too small an alphabet will allow very different activities
to be mapped to the same symbolised representation. The selected sizes
(between 5 and 9) are analysed further to ensure that a “reasonable” number of
motifs can be found.

The base motif size used for analysis is 6 which corresponds to a 30 minute
(i.e. 6 x five minutes) period. This figure was selected as the UK electricity
settlement market uses a 30 minute period and 30 minutes is also a reasonable
estimate for a time period that will include most household activities such as
breakfast, showering, etc..

To explore the setting for motif size that results in a reasonable number of
motifs being found, additional motif sizes of 4 (20 minutes), 9 (45 minutes)
and 12 (1 hour) are considered to find the most effective setting. Settings
for the alphabet size and motif size are considered together to find a
reasonable number of motifs.

The motifs found are compared on the basis of similar shape without regard to
absolute value of the data. A possible effect of this is to find motifs within
what is the general noise associated with the meter readings. This is avoided
by ignoring any motifs within a window which has a range of readings of less
than a given size (100W for this analysis).

Various different motifs are defined based on differing ways of considering
the data and on how the data is normalised and converted to a symbolic
representation. The different motifs created are:

  * •

That created from considering the actual meter readings, normalised within the
moving window being considered, and then converted to a symbolic
representation. This process is graphically shown in Figure 4.

  * •

The difference motif is created by taking the same data from the moving window
and then calculating the differences between adjacent meter readings. This
provides a series showing the relative increase or decrease in usage from the
previous time point. This data is then treated as above (normalisation and
symbolisation) to produce the difference motif.

  * •

The overall normalised motif is created by considering all the data for each
household and then normalising the data within the overall maximum range. The
moving window is then taken across this normalised data and each window is
then symbolised. Note that the normalisation within the window step is not
executed.

  * •

Compressed motifs are also created from each of the above approaches by
removing any repeating letters within the symbolised motif. For example, a
motif of “abcccb” will be stored in compressed form as “abcb”.

Values for the overall range of the meter readings within the motif window,
and the range of the incremental change data (the differences between adjacent
meter readings), is stored alongside the motifs.

The final result is that tables specific for each setting of the alphabet size
and motif size are created providing for retrieval of the differing types of
motifs together with details on the time and date of the motif, the household
being considered, and the range information as described above.

The symbolisation step within the motif discovery process is important in
allowing for approximate matching. As real valued meter readings are mapped to
a few characters, as defined by the alphabet size used, various ranges of
value are mapped to the same character and this process has the effect of
compensating for some of the noise that is present in meter data collected
from households. For example, doing the same activity on different days (e.g.
making toast) would be expected to map to the same motif. However, other
activities happening at the same time in the house (e.g. the fridge
automatically running) or slight differences in the toaster settings (e.g.
different level of brownness selected) may lead to slightly different meter
readings. With the right symbolisation settings, it is more likely that the
activity will map to the same symbolised representation.

It is necessary to detect the repeating motifs that signify particular
activities (e.g. cooking the evening meal). These are generally of a similar
shape on different days but will show some differences due to noise.

One issue arising is that normalisation means that the shape of the motif is
matched without regard to the absolute values of the readings and this might
not be valid. While small ranges of less than 100W are excluded, the actual
size of the increase or decrease is arguably more useful when identifying
particular activities (based on combinations of appliances) rather than the
base shape. This can be addressed by normalising over the whole day of a
household’s readings rather then within the motif window and is the basis for
including the normalised difference motif within the data collected.

While the motif matching approach based on shapes can, after normalisation
within the analysis window, lead to two motifs being seen as identical when,
in fact, one has a much larger range of power readings, this can be addressed
by only considering motifs as similar when they fall into the same ranges of
power usage.

As the analysis is to find activities during the day, these generally
correspond to the use of certain individual appliances (or a combination of
appliances) and thus the goal is to find motifs that map onto particular
appliance combinations. The power usage characteristics of appliances in
common use in the house are given by Kato et al. (2009) (although for Japan
rather than the UK).

The Centre for Sustainable Energy for Sustainable Energy (2017, last accessed
June 2017) publishes a list of typical power usage figures for household
appliances. This table can be simplified into the following groups:

  * •

High usage appliances such as power showers, over 5KW

  * •

Appliances generally used for heating - water or space - 3KW-5KW

  * •

Laundry and larger cooking appliances 1KW-3KW

  * •

Smaller appliances, gardening. Multiple lighting within rooms. 300W-1KW

  * •

TV, lighting and audio equipment - under 300W

The ranges to be used can be set by selecting a reasonable number (e.g. five)
and then splitting the range between the lowest and highest reading for a
given house into the five ranges. Alternatively, range cut off points can be
defined based on general industry standard values for power ranges of common
appliances.

The suggested cut-off points in the list above can be used to group the motifs
found into similar classes of appliance usage to avoid matching similar shaped
patterns that actually cover much different ranges of usage.

###  3.4 Identifying “Interesting” Motifs

Motifs arising from the small changes in the underlying electricity usage (the
“noise”) are excluded by setting a minimum range of 100W for a motif to be
considered of interest. Any motif representing a behaviour which influences
the electricity usage by less than 100W over the period of the motif is
unlikely to be of interest to professionals designing DSM programmes.

As the motifs are created by shifting a moving window over the stream of data,
overlapping periods are considered and a long period with no activity, except
for one change in meter reading, will lead to a series of motifs that are
similar. For example, when using an alphabet of five, a long period of no
activity except for a jump of +200W will lead to motifs such as ccccca,
ccccac, cccacc, etc. As only one of these is interesting for further analysis,
the others are excluded by omitting any motifs that start with two or more
“c”. With differently sized alphabets the same approach is adopted by
excluding the motifs that start with multiple “middle” letters.

If a motif consists solely of increases in electricity usage then it is
considered to be of no interest. The analysis is attempting to find motifs
that reflect complete behaviours (e.g. cooking) and a motif showing only
increases in usage would only represent the beginning of the activity. A
useful motif will consist of both starting (turning appliances on) and
finishing (turning appliances off) an activity. Similarly, a motif which
consists solely of decreasing meter readings is also designated as
uninteresting.

Finally, any motifs that span midnight are excluded. While this can be
criticised as a limitation, it is unlikely that electricity professionals will
be considering programmes to change household behaviour around midnight and
this limitation is unlikely to affect the usefulness of the results.

###  3.5 Evaluation of Different Techniques

The approach to finding motifs detailed above makes use of a number of
parameters such as the type of motif to use (compressed, normalised, etc.),
the size of alphabet to use, and the length of the motif. To choose between
the various parameters a method is needed to select which set is most
appropriate.

A useful behaviour to focus on for DSM would be one that repeats within a
household on the majority of days under consideration and which is of a
significant size in terms of power used. It should also be one that only
occurs once or a few times per day. An example would be cooking which may only
be done once during the conventional time period for the particular meal (e.g.
5pm to 8pm for the evening meal). However, with multiple members of the
household, activities may repeat on the same day - e.g. when different groups
of the household do their cooking separately.

To detect a repeating activity a reasonable number of motifs need to be found.
Some houses will have no repeating activity (their behaviour shows no
regularity) and thus no useful motifs may be found. Other houses need to be
evaluated in such a way as to give a reasonable number of motifs. The question
as to “what is reasonable” needs to be addressed. Activity should happen on a
sizeable proportion of the days under consideration to be seen as a regular
activity.

One approach to evaluation is to consider the x most popular motifs within a
household (i.e. the most popular is the one occurring most often, the second
most popular is the one occurring second most often, etc.) and then see how
the number of occurrences vary between the x motifs. For example, considering
the 10 most common motifs, each can be assessed by a useful measure (e.g.
number of occurrences) and a graph similar to that of Figure 5 can be
produced. The points marked as X, Y, and Z need to be defined in terms of
reasonable numbers and then the results from the two sets of parameters
displayed on the graph can be assessed against each other.

Figure 5: Example of motif evaluation: The shaded yellow area is the area of
interest - thus the red line is preferred because it falls more into the
interesting area than the green line.

The point Z sets the number of popular motifs that are to be considered. While
the graph shows the results for the top ten motifs, it may be considered that
concentrating on the top Z motifs is sufficient and any results to the right
of Z can be ignored.

The point X sets the value at which the measure (e.g. number of occurrences
per day) is judged to be too high. Finding motifs that occur many times a day
(say, ten times) is unlikely to be useful for finding behaviours that occur
occasionally and can be influenced.

The point Y gives a value for the low point of the measure below which motifs
are too infrequent to be useful.

These three points define the shaded area which is of “interest” and where the
useful motif results will lie within. Whichever of the lines on the graphs
falls more completely within the shaded area is considered to be the better
set of parameters to use for future analysis. In the example, the red line
would be judged to give better results as although the green line shows higher
values, part of the line falls outside the preferred shaded area and can be
considered to give too high results (e.g. too many occurrences per day).

There are many different measures that could be used to distinguish between
the motif finding techniques. This analysis makes use of the following:

  * •

The number of times that the motif occurs within the data period. This should
not be too high (point X in the example) nor too low (point Y).

  * •

The number of days that include the motif. There is no maximum value (point X)
for this measure as a motif falling on every day would be useful. The value
for a minimum number of days (point Y) needs to be selected.

  * •

The percentage of days that include the motif. This differs from the second
measure as it incorporates the number of days of readings for the household
into the calculation. Not all households have a set of readings for every day
within the period of analysis.

The relevant values for X, Y and Z need to be set for each of the measures
being considered. The values chosen are detailed below. The analysis set out
in the remainder of the paper can easily be adjusted for differing values of
X, Y and Z and could be explored in future work. For this analysis, values are
selected (with justification below) and other values could also be valid.

For each measure, the top three motifs only are considered (i.e. point Z). The
value of three is selected as a value between concentrating on too few motifs,
and allowing too much influence to motifs that only occur occasionally for a
household (e.g. if set as four or more).

When considering the average occurrences per day for each motif, a value for X
needs to be selected that exclude motifs that happen too frequently during the
day and which may reflect regular behaviours which the household members may
have little opportunity to influence. This could include regular automatic
activities (e.g. fridge turning on and off) or behaviours which are necessary
for efficient household functioning (e.g. turning lights on and off). A value
for X of 2.0 is selected.

In a similar way, motifs that occur infrequently would be of little interest
as they do not reflect a regular behaviour and hence Y should not be set too
low. A value of Y of 0.3 is selected, reflecting that motifs occurring on
average on 30% of the days is of interest.

Some motifs may occur many times on a few days but rarely on other days and
the analysis of the number of unique days that contain the motif is used to
identify this situation. The date of occurrence of each motif is found and the
number of unique days for that motif is found and then averaged across all the
households. In this situation, a motif that occurs on every day under analysis
would be of interest and hence X should be set to a maximum value. The value
of Y depends on the number of days in the period of analysis and, when
considering a calendar quarter of working days (i.e. a maximum number of days
of 65), a value for Y of 10 is selected.

Some households do not have readings for every day within the period of
analysis and the percentage number of unique days with a given motif provides
a measure that takes into account the number of days of analysis. In this
case, a figure for X is selected of 90% which excludes the motifs that occur
on nearly every day and may be considered to be too frequent and likely to
reflect regular activities which the household has no control over (e.g.
fridges). The value for Y is set at 20%.

The values selected for each measure are summarised in Table 1.

Table 1: Values selected for each measure Measure | X | Y | Z
---|---|---|---
Motifs per day | 2 | 0.3 | 3
Total days with motif | 65 | 10 | 3
Percent of days with motif | 90 | 20 | 3

##  4 Results

The research uses R Studio version 0.97.551 using R 3.0.0 R Core Team (2012)
running on a Windows 7 64 bit system and accessing the data stored within a
MySQL v5.5.31 database.

###  4.1 Exploring the Different Ways of Treating the Motif Range

Three approaches to handling the range of the motif (the difference between
highest and lowest meter readings during the period of the motif) are
considered. These are:

  * •

No consideration of range. Each motif with a similar shape, irrespective of
actual values of the meter readings, is considered as the same motif. This
approach matches similar shapes but has the possible downside of equating a
motif with a relatively small range of readings (e.g. 200W) with the same
shape over a relatively large range (e.g. 2KW). It is unlikely that these
correspond to the same activity which would generally relate to the use of
particular appliances and which would have particular electricity usage
characteristics (e.g. Wattage ratings).

  * •

Using the range of meter readings within each house under consideration. For
each house, the maximum and minimum values for the range of each motif are
found. 5 ranges are selected and these are set as being equally spread between
the low and high motif range values. For example, for a house with motifs with
a maximum range of 1100W and a minimum range of 100W, the ranges are set as
100-300W, 300-500W, 500-700W, 700-900W and 900W to 1100W. Motifs are only
considered to be the same if, as well as having the same shape, the range of
the motifs match.

  * •

Based on typical appliance usage characteristics, the range boundaries are set
as 300, 1000, 3000, 5000, and 60000W and the motifs are treated as above such
that motifs are only judged to match when their shape is similar and they fall
within the same range.

To assess the different approaches to dealing with ranges, the motifs using
each approach are found for various values of alphabet size (5,7,9) and motif
length (4,6,9,12). The results for each of the motifs detected (using the base
data, using differences between readings and the compressed version of each)
are plotted with the results shown at Figures 6, 7, and 8 for the percentage
of days where the motif is found. The other measures (motifs per day and
number of unique days) are also considered and the results are similar (graphs
not included for space reasons). The graphs show the results for each value of
alphabet size (5,7,9) down the vertical and each value of motif size
(4,6,9,12) across the horizontal.

Figure 6: Percentage of days; Results with no range adjustment Figure 7:
Percentage of days; Results with within house range adjustment Figure 8:
Percentage of days; Results with appliance range adjustment

Except for the largest values for motif size and alphabet size (where,
intuitively, fewer motifs would be expected) all approaches to dealing with
the range provide results within the acceptable and useful area (the shaded
areas on the graphs). As the approach of making use of the ranges of possible
appliances (the third approach) is intuitively more appealing, this option is
taken for future analysis.

###  4.2 Evaluating Normalisation Method

Options considered for normalisation of the data prior to conversion to
symbolised representation are:

  * •

Each motif is considered and the values within the motif are normalised before
conversion to a symbolised string.

  * •

Each household is considered and all the readings for that household are
normalised prior to consideration of motifs.

The results from considering the two approaches to dealing with normalisation
are shown in Figures 9 and 10. Again the results for the number of unique days
is omitted for space reasons.

Figure 9: Unique days (%) for different normalisations Figure 10: Motifs per
day using different normalisations

The motifs per day results show that the normalisation over the whole period
for each household produces a lot of motifs per day and, in fact, more than
the maximum 2 per day that is considered useful. For that reason, the
normalisation within a motif is considered the better approach and is selected
for further analysis.

###  4.3 Selecting the Motif Size

As the motif size increases, the number of possible combinations of letters
within the motifs increases and the number of motifs found reduces. The
requirement for a useful size of motif is to provide useful numbers of motifs
(but not too many). To analyse possible settings, results for motifs of size
4, 6, 9, and 12 (corresponding to 20 minutes, 30 minutes, 45 minutes and an
hour) are each used with alphabet sizes of 5, 7 and 9. The results are shown
in Figure 11.

Figure 11: Different motif sizes

For smaller values of alphabet size, the 30 minute motifs give more useful
results. However, for larger alphabet sizes the 20 minute motifs are better.
To investigate this further, both the 20 minute and 30 minute motifs will be
considered when evaluating the alphabet size.

###  4.4 Selecting the Alphabet Size

As the alphabet size increases, the number of possible combinations of letters
increases and the number of times motifs repeat will be expected to decrease.
To evaluate the best settings for alphabet size, each of 5, 7, and 9 letters
are considered with the motif sizes of both 4 and 6 (20 minutes and 30
minutes). The results are shown in Figure 12.

Figure 12: Different alphabet sizes

The results show that two combinations of settings are likely to provide
useful motifs for further analysis. When using a small number of letters (5)
in the alphabet when symbolising motifs, the best motif size to use is 6
equating to 30 minutes. However, an alphabet size of 7 is best used in
combination with a motif size of 4 (20 minutes).

##  5 Conclusions and Further Work

Based on the above analysis, a set of parameters is selected as the ones that
are most likely to provide motifs that will be useful. The parameter settings
are shown in Table 2.

Table 2: Parameter Settings selected for future analysis Parameter |  Setting |  Meaning
---|---|---
Range |  Using appliance characteristics |  The typical ranges of appliances can be used to distinguish between similar shaped motifs.
Data |  Differences between consecutive readings |  The differences between readings are changes in electricity usage.
Compression |  Compressed data used |  Repeating letters within the motif are removed.
Normalisation |  Within motif |  Values within the motif are normalised before being symbolised.
Motif size |  4 (20 minutes) or 6 (30 minutes) |  The length of each motif.
Alphabet size |  5 or 7 letters (depending on motif size) |  The number of letters used for the motif symbolisation step.

The combination of parameters has been shown to produce a usable number of
motifs. The conclusion is therefore drawn that using a sampling frequency of
five minutes is sufficient for repeating patterns to be detected within the
data streams and that sufficient meaningful patterns can be found for the
motif analysis to be useful.

An example showing the motif found for house 485 (where there is also a large
range for the readings within the motif) can be seen at Figure 13. This shows
the daily usage pattern for each day (each dotted line) for the selected house
within the Spring 2011 period (note that this house has data for about 350
days out of 365 days of posible readings) with the particular motif picked out
in bold. The motif broadly represents a large spike in usage preceded and
superseded by a period of relatively little change in activity. It can be seen
that the motif is found even though the spike in usage may be a slightly
different height on each day (so long as the overall range fits within the
1000-3000W range). It can also be seen that the motif at about 7am is detected
even though there is other electrical activity on that day providing a base
load onto which the motif usage is added. Other spikes in usage that have not
been found to match the motif (i.e. not represented in bold) generally show a
gradual tail-off in usage rather than a sharp reduction to the original level
of usage and probably represent different household activities.

Figure 13: Example of motifs found for house 485

This paper has addressed the question as to whether a reasonable number of
meaningful motifs, that represent regular activities within a domestic
household, can be identified solely using the electricity meter data. The
analysis has shown that data sampled at a five minute interval is sufficient
to provide a usable number of motifs which can be assumed to represent
repeating activities within the household.

The symbolisation process is a suitable method for converting the real valued
meter readings into a restricted set of characters with each representing a
certain range of values. This provides for matching of data samples which,
while not identical, map to the same symbols and can be assumed to be
approximately the same.

The results show that motif finding using the symbolisation technique and the
five minute sampling period is possible. A series of different parameter
settings have been explored and a set of parameters selected.

The method of representing the motif in an alphabetised form has been explored
with possible approaches of using the base meter data or differenced data. In
addition, compression of repeating characters is considered.

Motifs may match but actually represent similar patterns within very
differently sized data and different approaches to splitting the motifs into
various bands of usage over the range of the motif have been explored with a
solution based on a split into various appliance ranges (five ranges from
small to large usage) selected as the best approach.

Various motifs that show uninteresting behaviour can be found (e.g. no change
in usage over the time of the motif such as when the house is unoccupied) and
an automatic method of rejecting some of the uninteresting motifs has been
developed.

An effective method of identifying meaningful motifs opens up opportunities
for electricity industry professionals to gain greater understanding of
activities within given households. The paper has not tested the
implementation of initiatives to change behaviour but has investigated whether
reasonable numbers of motifs can be identified and has established the most
appropriate parameter settings for obtaining reasonable numbers. Future work
can make us of the derived parameters settings to explore the detection of
behaviours and then test interventions to change these behaviours. This would
be a very valuable step along the road of effectively deploying effective DSM
initiatives.

## References

  * (1)
  * Castro (2010) Castro, N. (2010), Time Series Data Mining,, in Semana de Engenharia.
  * Chamberlin (1993) Chamberlin, A. F. J. (1993), Principles and Practice of Demand Side Management,, Technical report, Barakat & Chamberlin Inc.
  * Chang et al. (2010) Chang, H., Lin, C., and Lee, J. (2010), Load identification in nonintrusive load monitoring using steady-state and turn-on transient energy algorithms,, in Computer Supported Cooperative Work in Design (CSCWD), 2010 14th International Conference on, IEEE, pp. 27–32.
  * Craig et al. (2014) Craig, T., Polhill, J. G., Dent, I., Galan-Diaz, C., and Heslop, S. (2014), “The North East Scotland Energy Monitoring Project: Exploring relationships between household occupants and energy usage,” Energy and Buildings, .
  * Das et al. (1998) Das, G., Lin, K.-I., Mannila, H., Renganathan, G., and Smyth, P. (1998), Rule Discovery from Time Series.,, in KDD, Vol. 98, pp. 16–22.
  * DECC (2009) DECC (2009), Towards a Smarter Future, Government Response to the Consultation on Electricity and Gas Smart Metering,, Technical report, Department of Energy and Climate Change.
  * DECC (2011) DECC (2011), Implementation Programme: Response to Prospectus Consultation: Overview Document,, Technical report, Department of Energy and Climate Change.
  * Dent et al. (2012) Dent, I., Craig, T., Aickelin, U., and Rodden, T. (2012), A Method for Cleaning and Storing Electricity Meter Data for Flexible Analysis,, in BeHave 2012, Helsinki.
  * Eagle and Pentland (2006) Eagle, N., and Pentland, A. (2006), “Reality mining: sensing complex social systems,” Personal and Ubiquitous Computing, 10(4), 255–268.
  * for Sustainable Energy (2017, last accessed June 2017) for Sustainable Energy, C. (2017, last accessed June 2017), How much electricity am I using.
http://cse.org.uk/advice/advice-and-support/how-much-electricity-am-i-using

  * Gay et al. (2012) Gay, D., Selmaoui-Folcher, N., and Boulicaut, J.-F. (2012), “Application-independent feature construction based on almost-closedness properties,” Knowledge and information systems, 30(1), 87–111.
  * Höppner (2001) Höppner, F. (2001), “Discovery of temporal patterns,” in Principles of Data Mining and Knowledge Discovery, Berlin: Springer, pp. 192–203.
  * Kato et al. (2009) Kato, T., Cho, H., Lee, D., Toyomura, T., and Yamazaki, T. (2009), “Appliance Recognition from Electric Current Signals for Information-Energy Integrated Network in Home Environments,” in Ambient Assistive Health and Wellness Management in the Heart of the City, eds. M. Mokhtari, I. Khalil, J. Bauchet, D. Zhang, and C. Nugent, Vol. 5597 of Lecture Notes in Computer Science, Berlin: Springer Berlin / Heidelberg, pp. 150–157.
  * Lin et al. (2002) Lin, J., Keogh, E., Lonardi, S., and Patel, P. (2002), Finding motifs in time series,, in Proc. of the 2nd Workshop on Temporal Data Mining, pp. 53–68.
  * Lin et al. (2007) Lin, J., Keogh, E., Wei, L., and Lonardi, S. (2007), “Experiencing SAX: a novel symbolic representation of time series,” Data Mining and Knowledge Discovery, 15(2), 107–144.
  * MacLeay et al. (2013) MacLeay, I., Harris, K., and Annut, A. (2013), Digest of United Kingdom Energy Statistics 2013,, Technical report, Department for Business, Energy and Industrial Strategy.
  * Meo et al. (2012) Meo, R., Bachar, D., and Ienco, D. (2012), “LODE: A distance-based classifier built on ensembles of positive and negative observations,” Pattern Recognition, 45(4), 1409–1425.
  * Mori and Umezawa (n.d.) Mori, H., and Umezawa, Y. (n.d.), A SAX-based method for extracting features of electricity price in power markets,, in Transmission & Distribution Conference & Exposition: Asia and Pacific, 2009, IEEE, pp. 1–4.
  * Mueen et al. (2009) Mueen, A., Keogh, E., Zhu, Q., Cash, S., and Westover, B. (2009), Exact discovery of time series motifs,, in Proc. of 2009 SIAM International Conference on Data Mining, pp. 1–12.
  * Newborough and Augood (1999) Newborough, M., and Augood, P. (1999), Demand-side management opportunities for the UK domestic sector,, in Generation, Transmission and Distribution, IEE Proceedings-, Vol. 146, IET, pp. 283–293.
  * Ofgem (2010) Ofgem (2010), RIIO: A new way to regulate electricity networks,, Technical report, Ofgem.
  * Poyry (2010) Poyry (2010), Options for Low-Carbon Power Sector Flexibility to 2050,, Technical report, The Committee on Climate Change.
  * R Core Team (2012) R Core Team (2012), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria.  ISBN 3-900051-07-0.
  * Rhodes (2010) Rhodes, A. (2010), Smart Grids: Commercial Opportunities and Challenges for the UK.,, Technical report, Energy Generation and Supply Knowledge Transfer Network.
  * River (2005) River (2005), “Primer on demand-side management with an emphasis on price-responsive programs,” prepared for The World Bank by Charles River Associates, Tech. Rep, .
  * Shieh and Keogh (2008) Shieh, J., and Keogh, E. (2008), i SAX: indexing and mining terabyte sized time series,, in Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp. 623–631.
  * Zeifman and Roth (2011) Zeifman, M., and Roth, K. (2011), “Nonintrusive appliance load monitoring: Review and outlook,” Consumer Electronics, IEEE Transactions on, 57(1), 76–84.