# Filling time-series gaps using image techniques: Multidimensional context
autoencoder approach for building energy data imputation

Chun Fu  Matias Quintana  Zoltan Nagy  Clayton Miller<EMAIL_ADDRESS>
###### Abstract

Building energy prediction and management has become increasingly important in
recent decades, driven by the growth of Internet of Things (IoT) devices and
the availability of more energy data. However, energy data is often collected
from multiple sources and can be incomplete or inconsistent, which can hinder
accurate predictions and management of energy systems and limit the usefulness
of the data for decision-making and research. To address this issue, past
studies have focused on imputing missing gaps in energy data, including random
and continuous gaps. One of the main challenges in this area is the lack of
validation on a benchmark dataset with various building and meter types,
making it difficult to accurately evaluate the performance of different
imputation methods. Another challenge is the lack of application of state-of-
the-art imputation methods for missing gaps in energy data. Contemporary
image-inpainting methods, such as Partial Convolution (PConv), have been
widely used in the computer vision domain and have demonstrated their
effectiveness in dealing with complex missing patterns. Given that energy data
often exhibits regular daily or weekly patterns, such methods could be
leveraged to exploit the regularity of the data to learn underlying patterns
and generate more accurate predictions for missing values. To study whether
energy data imputation can benefit from the image-based deep learning method,
this study compared PConv, Convolutional neural networks (CNNs), and weekly
persistence method using one of the biggest publicly available whole building
energy datasets, consisting of 1479 power meters worldwide, as the benchmark.
The results show that, compared to the CNN with the raw time series (1D-CNN)
and the weekly persistence method, neural network models with reshaped energy
data with two dimensions reduced the Mean Squared Error (MSE) by 10% to 30%.
The advanced deep learning method, Partial convolution (PConv), has further
reduced the MSE by 20-30% than 2D-CNN and stands out among all models. Based
on these results, this study demonstrates the potential applicability of time-
series imaging in imputing energy data. The proposed imputation model has also
been tested on a benchmark dataset with a range of meter types and sources,
demonstrating its generalizability to include additional accessible energy
datasets. This offers a scalable and effective solution for filling in missing
energy data in both academic and industrial contexts.

###### keywords:

Missing data , Data reconstruction , Data preprocessing , Deep learning ,
Computer vision

††journal: Applied Thermal Engineering

[inst1]organization=Building and Urban Data Science (BUDS) Lab, Department of
the Built Environment, College of Design and Engineering, National University
of Singapore (NUS), country=Singapore [inst2]organization=Intelligent
Environments Lab (IEL), Department of Civil, Architectural and Environmental
Engineering, Cockrell School of Engineering, The University of Texas at
Austin, state=TX, country=USA

##  1 Introduction

Machine learning (ML) has recently demonstrated great
promise for energy forecasting by providing accurate and reliable predictions
of future energy demand and supply. As detailed by Kazmi et al. [1], the field
is evolving, driven by increased computing power, advancements in data
handling, and innovative algorithmic methods, which together have significant
potential to transform energy demand forecasting at both building and urban
scales. The ML algorithms have shown remarkable potential in predicting energy
consumption in buildings [2, 3, 4], forecasting renewable energy output [5, 6,
7], and estimating grid electricity demand [8, 9, 10].

By outperforming traditional statistical methods, ML models have demonstrated
significant improvements in terms of accuracy and reliability [11, 12].
Moreover, ML also has been applied in building operations and energy
management, such as cooling load forecasting-based optimization for chiller
control [13] and ML-based methods for optimally managing energy sources with
power grids [14]. However, with these advancements brought by ML, practical
applications often face challenges due to instability in data quality of
building energy consumption, which can result in problems with forecasting
performance. Among the issues with data quality, missing data is a primary one
that hinders the efficacy of ML models. These missing data can originate from
a spectrum of causes, ranging from terminal equipment and operator error to
malfunctions of sensors [15, 16]. The missing data can have cascading
consequences, negatively impacting stability, performance, and failure
prevention [17]. In certain commercial buildings, it can result in energy
waste ranging from 15 and 30 percent [18, 19]. Given the criticality of the
issue, our study seeks to utilize state-of-the-art deep learning methods to
bridge the missing data gaps, thereby contributing a new perspective to the
building energy data imputation field.

###  1.1 Data mining and modeling techniques for addressing data quality
issues

The presence of abnormal data in energy forecasting is a significant issue, as
emphasized in the ASHRAE Great Energy Predictor III (GEPIII) competition on
Kaggle [20]. Additionally, the abnormal energy-consumption behavior may also
limit the predictability of ML on energy forecasting [21]. To address these
issues, several initiatives, such as the Large-scale Energy Anomaly Detection
(LEAD) competition, provides a benchmark dataset for energy anomaly detection
in commercial buildings [22], and a tree-based ensemble model was proposed for
scalable anomaly detection [23]. Notable advancements in research include the
development of a model for multivariate time series anomaly detection in
energy consumption based on a graph attention mechanism [24] and an enhanced
fault detection method for building energy systems in the presence of missing
data using expectation-maximization and Bayesian network [25]. Concurrently,
unsupervised techniques have also been extensively utilized to discern
patterns in energy usage through clustering and to identify anomalies in daily
profiles of energy data [26, 27, 28].

The applications of ML in the building life cycle are primarily focused on the
operation and maintenance phase [29]. In particular, specific use cases need
more data in quantity and diversity and thus opted for data-driven methods to
tackle this. As for data quantity, generative models, specifically Generative
Adversarial Networks (GANs), have been used for scenarios without sufficient
historical data, requiring synthetic data. These GAN-based models were first
proposed as a way to learn the distribution of the input data and generate
synthetic data that resembles the input data [30]. In the built environment,
these models have been used in power demand prediction [31], building load
profile [32], fault detection and diagnosis (FDD) [33], and indoor thermal
comfort [34] to generate synthetic complementary data for model training.
Another approach is to augment energy time-series data using masking noise
injection for more efficient imputation of missing values [35]. Several data
augmentation methods that manipulate time series in terms of magnitude and
temporal order have also been discussed [36, 37]. These papers emphasize the
importance of data augmentation in time series analysis and energy modeling,
especially in the absence of sufficient data.

Other approaches circumvent the lack of available and diverse data by
incorporating external data streams to aid in the data mining task. Fu et al.
[38] used Google searches, in the form of Google trends, as additional input
data for energy consumption forecasting. Moreover, Sun et al. [39] used
building facade images to complement energy consumption and building metadata
for energy efficiency rating prediction. These models showcase the
incorporation of external data streams of different sensing modalities and
their potential aid. However, while these approaches increase the performance
of the downstream task, they do not address the data availability issues.
Incorporating new data streams may potentially introduce additional
availability issues.

###  1.2 Imputation models in energy and building fields

To enhance data availability, a plethora of methods have been proposed for
data imputation, ranging from traditional statistical methods, regression
models, and deep learning. The types of missing data can be categorized as
short-term missing (or random missing) and long-term missing (or continuous
missing). Short-term missing (or random missing) refers to missing data points
that cover a relatively small amount of time (e.g., hourly or daily missing),
while long-term missing (or continuous missing) refers to data gaps that cover
a longer period of time (e.g., weekly or monthly missing in time series).
Traditional data reconstruction methods can be traced back to linear or
polynomial regression methods, which have proven effective and efficient in
data imputation, especially for short-term missing data [40, 41]. Some other
supervised-learning regression methods based on lag features were often used
to impute missing energy data gaps. For example, linear regression, weighted
K-nearest neighbors (kNN), support vector machines (SVM), and mean imputation
was implemented for random missing values varying from 5% to 20% missing rate,
demonstrating superior performance than zero or mean imputation [40]. Another
comparative study by Wang et al. stated that both statistical methods and
machine learning regression models exhibit good performance for imputing
short-term missing gaps within a one-day period in time series data[41].
However, these methods have demonstrated limited performance in imputing long-
term missing data, which is commonly encountered in real-world scenarios. This
limitation can be attributed to the reliance of statistical and regression
models on nearby values, which are unavailable for long-term missing data.

To address the long-term missing data mentioned above, neural networks have
been utilized for data imputation due to their advantage of learning context
in unstructured datasets. For example, a method using an improved
Convolutional Neural Network (CNN) that considers the correlation of power
data from dimensions of time and space was proposed for filling missing data
[42]. Recurrent Neural Network (RNN) and Bi-directional Long Short Term Memory
model (BI-LSTM) were proposed to impute energy data missing due to the
temporal aspects of energy data [43, 44]. More recently, autoencoder has
received growing attention among various frameworks of neural networks because
of its self-reconstruction and feature extraction capabilities. For example, a
study applied an autoencoder to reconstruct missing gaps in indoor
environmental quality data, showcasing improved performance compared to
classic polynomial interpolations [45]. In another study, a 2D Convolutional
Neural Network (2D-CNN) autoencoder was employed to address missing energy
data considering the weekly periodicity of the data. The energy data was
reshaped into a two-dimensional format, and the proposed autoencoder
effectively imputed both random and continuous missing data [46]. The studies
above demonstrate how autoencoders can be applied to data imputation in the
built environment for both random and continuous missing. However, most of the
studies have been conducted in the context of a limited number of buildings or
a single power grid, leaving their applicability to a broader set of scenarios
(e.g., unseen buildings or different sites) untested. Moreover, these neural
networks belong to earlier or outdated frameworks, and some modern deep
learning techniques that could yield even more accurate predictions have yet
to be thoroughly investigated for data imputation in this field. In light of
these observations, the development of modern deep learning frameworks
considering generalizability in application holds great potential for
advancing data imputation methods in the built environment.

###  1.3 Image-based data imputation

Missing data is a ubiquitous problem in various fields, such as audio signals
repair [47, 48], and image inpainting in the computer vision field [49, 50].
Modern frameworks developed within these fields to address missing data issues
hold the potential to revolutionize the way missing data is handled in the
field of building energy. One such framework is partial convolutions, which
was proven effective in image inpainting tasks involving regular (e.g.,
circles and rectangles) and irregular holes in images [51]. Another promising
framework is the diffusion model, which employs iterative diffusion processes
to fill in missing data while preserving the smoothness and continuity of the
original signal [52]. However, these modern image-based techniques have yet to
be widely applied to building energy data. There are a few reasons for this.
The first reason is that energy data has traditionally been treated and
imputed as time series. Consequently, past research has primarily relied on
statistical methods for time series or RNNs to handle missing gaps. This means
that the structured context and two-dimensional representation of energy data
that inpainting methods can leverage have yet to be fully utilized. Another
factor may be the lack of sufficient energy data for training. Past studies
often utilized a limited number of meters or grids for developing and
evaluating the imputation model. In contrast, fields like image or audio
processing typically have access to larger datasets, allowing for more
extensive training and evaluation to verify the generalizability and
effectiveness of the proposed methods. For instance, benchmark image datasets
often consist of thousands to millions of image samples [53], which far exceed
the scale of typical energy datasets that may have, at most, thousands of
energy meters. In response to these challenges, this research aims to
transform conventional one-dimensional time-series representation of meter
data into two-dimensional images (i.e., time of week by week). We propose to
implement an advanced image inpainting framework to impute missing gaps with
assistance from the structured context inherent in energy data. In addition,
this research will harness the largest publicly available building energy
dataset, encompassing more than one thousand power meters, for comprehensive
model training and validation.

###  1.4 Research objectives and novelty

To fill the aforementioned gaps, this study aims to investigate how the
context of energy data affects data imputation performance. It achieves this
through the use of a global dataset with diverse meters while setting
different rates of both short-term and long-term missing data. Leveraging deep
learning techniques from the field of computer vision, this research reshapes
time-series energy data to harness the cyclical nature of such data and
applies state-of-the-art image-inpainting models for energy data imputation
(as illustrated in Figure 1). The research objectives of this study are to:

  1. 1.

Present a novel viewpoint in imputing energy data by converting time series
into images and investigating image-based algorithms.

  2. 2.

Explore the use of modern deep learning frameworks, such as partial
convolutions (PConv) and 2D-CNN, for imputing missing data in energy time
series.

  3. 3.

Evaluate these frameworks’ performance in imputing both random and continuous
missing data with varying missing rates and test their generalizability to
different sites.

  4. 4.

Benchmark the performance of different image-based frameworks (i.e., 2D-CNN
and PConv) against baselines (i.e., 1D-CNN and weekly persistence method).

Figure 1: The one-dimensional time series of energy data can be reshaped into
a two-dimensional heatmap image for testing different imputation methods.
Comparisons are made between the one- and two-dimensional imputation,
including baseline model, 1D- and 2D-CNN, and advanced deep learning
techniques used for image inpainting.

This study’s novelty lies in applying image-inpainting models, which have been
widely used in the field of computer vision, to the imputation of energy data.
This presents a new and innovative approach to the challenge of missing data
in energy time series and offers the potential to further improve the accuracy
of data imputation through the contextual learning of the underlying energy
data structure. Consequently, it could enhance the data quality used for
energy-related downstream tasks such as energy forecasting and building energy
modeling. Furthermore, this study verifies the models’ generalizability across
various buildings and meter types and their effectiveness in addressing long-
term missing data. By demonstrating the applicability and effectiveness of
these models in the field of building energy, this study opens up new avenues
for future research at the intersection of image-based techniques in the built
environment.

##  2 Methodology

The proposed methodology for this study consists of three phases, as
illustrated in Figure 2. In the first phase, training models for imputation
will be developed using energy data with synthetic missing values as input and
the corresponding raw data as output. The objective is to train a model that
can accurately impute the synthetic missing values and produce results that
closely resemble the raw data. In the second phase, the performance of these
trained models will be evaluated and quantified on a test dataset with varying
settings of missing and various meter types. Finally, in the third phase, a
comprehensive comparison and discussion of the results obtained from all
models, including the baseline method, will be conducted to draw conclusions.

Figure 2: Overview of the proposed framework and different phases in the
working pipeline.

###  2.1 Dataset: Building Data Genome 2.0 (BDG2)

This study used the time-series hourly data from power meters in the Building
Data Genome 2 (BDG2) project dataset for modeling. These data were also
employed in the Great Energy Predictor III competition hosted on the Kaggle
platform [54]. The Building Data Genome 2.0 (BDG2) is an open dataset
containing hourly meter readings and metadata of 3,053 power meters over two
years. Each building within the dataset is associated with metadata such as
floor area, weather, and primary use type. Due to the variability of meter
characteristics observed across thousands of meters worldwide, this dataset is
an ideal benchmark dataset for comparing different machine learning algorithms
while testing generalizability. Table 1 outlines the metadata variables
available within the dataset. It’s worth noting that only the historical
energy consumption data was used, and other potential data sources, such as
weather and calendar data, were not incorporated in the imputation process.
That’s because the focus of this research was to impute missing data in energy
time series by leveraging nearby values of the data.

Category | Variable | Unit | Content
---|---|---|---
| Power
---
meter
Meter type | - |  | Type of meter: electricity,
---
chilled water, steam, or
hotwater
| Meter readings | kWh | Energy consumption
Building | Primary use | - | Primary category of activities
metadata | Year built | - | Year building was opened
| Floors | - | Number of floors
| Floor area | Sq foot | Gross floor area
Weather | Temperature | °C | Outdoor temperature
data | Cloud cover | Oktas | Portion of the sky covered
| Dew point | °C | Outdoor dew temperature
| Precipitation | Millimeter | Precipitation depth
| Pressure | Millibar | Sea level pressure
| Wind speed | m/s | Wind speed
| Wind bearing | Degree | Wind direction

Table 1: Building metadata available in the Building Data Genome 2 (BDG2)
project.

###  2.2 Data preprocessing

The energy dataset will be subjected to several preprocessing steps to prepare
it for model training. These steps encompass data cleaning to remove
inconsistencies or errors, data normalization to standardize the data across a
common range, data splitting into training, validation, and test sets, and
data augmentation, such as flipping and shifting the time series to expand the
dataset. By following these commonly used preprocessing steps in the domain of
building energy data, the resulting models are equipped to deliver accurate
and reliable predictions.

####  2.2.1 Data cleaning

Since this dataset is sourced from power meters installed in real-world
buildings, it is expected to contain missing data resulting from system errors
or equipment failures. Cleaning the dataset by removing anomalies or errors is
necessary to ensure the quality of the data used for training the imputation
model. To accomplish this, we utilized the data cleaning results provided by
the winning team in the GEPIII Kaggle competition, which has also been used as
a benchmark for detecting anomalies in prior research [55]. This process
involves identifying and removing long streaks of constant values, large
positive/negative spikes, and other anomalies determined through visual
inspection. After cleaning the dataset, we chose the power meters with low
missing rates (less than 5%) for our study, resulting in a total of 1479
meters (approximately 50% of the entire BDG2 dataset).

####  2.2.2 Data normalization

To enhance the training process, the numerical values of the power meters were
normalized to ensure consistent data distributions. Normalization is
particularly essential when preparing the data for neural networks, as these
models rely on the gradient descent algorithm to optimize their parameters.
This algorithm can be affected by the scale of the data; thus, normalization
helps achieve convergence. In this study, we applied min-max normalization,
which scales the data to a range of $[0,1]$. This is achieved by subtracting
the minimum value in each series from the data and dividing by the difference
between the maximum and minimum values (as shown in Equation 1).

$X_{norm}=\frac{X-X_{min}}{X_{max}-X_{min}}$ (1)

####  2.2.3 Data splitting

This study employed a 5-fold cross-validation technique to divide the 1479
power meter data into training, validation, and testing sets. Each round of
modeling utilized 60% of the data for training, 20% for validation to improve
training efficiency, and 20% for testing to evaluate model performance. To
ensure the model’s generalizability, the data was divided by site ID to
prevent the model from imputing missing data based on similar meters within
the same site. This means the test dataset for evaluating the model’s
performance was from unseen sites for the trained model. This allows us to
check if the model is able to accurately impute data for meters from new
sources, which is crucial in real-world scenarios.

####  2.2.4 Data augmentation

Data augmentation was employed in this study due to the small dataset size
(1479 time series or images) available for developing machine learning models
for meter-wise imputation, particularly for image-based Convolutional Neural
Networks (CNNs) and modern image inpainting methods. To address this issue,
the study utilized two specific data augmentation techniques for time-series
data: shifting and flipping, as illustrated in Figure 4. Shifting moves the
time series forward by a certain number of steps. This can be beneficial for
tasks where the sequence of data points is critical as it enables the
generation of new data points, similar to the original data but with a unique
temporal arrangement.

Flipping, on the other hand, comes in two forms - horizontal and vertical.
Horizontal flipping, which reverses the time series sequence, is inappropriate
for our study as the temporal order is of utmost importance. Hence, horizontal
flipping has been excluded from our data augmentation strategies. Vertical
flipping doesn’t disrupt the chronological order; rather, it inverts the
values, akin to reflecting the time series along the horizontal axis. This
strategy can be pertinent in the context of building energy data. For example,
chilled water and hot water meter readings are often inversely correlated due
to their opposing operational nature; the flipped representation of such time
series may provide additional context for model training. Therefore, our study
used vertical flipping as a data augmentation method.

Despite the fact that the existence of other data augmentation methods, such
as those outlined in relevant work [36, 37], this study specifically opted for
flipping and shifting as they effectively preserve the inherent structure and
temporal sequence of the original data. By applying these techniques, the
initial dataset was expanded fourfold, resulting in approximately three times
more data, equivalent to over 5000 time series.

Figure 3: Data augmentation: (a) Shifting: shifting time series by a certain
number of timesteps (b) Flipping: flipping time series in the vertical
direction.

###  2.3 Missing masks

Missing masks are synthetic filters used to train imputation models to predict
missing gaps in energy data. There are two common types of missing gaps in
energy data: random missing, resulting from occasional events, and continuous
missing, caused by system shutdown or sensor failure (as shown in Figure 4).
Both types of missing data are included in the imputation tasks of this study
and are given a missing rate range of 5% to 50%.

Figure 4: Missing type: (a) Random missing: randomly selected days of missing
data (b) Continuous missing: consecutive days of missing data

####  2.3.1 Random missing

Random missing (also known as short-term missing), which refers to gaps in
time series data that are randomly scattered over time, has garnered attention
in previous studies. The granularity of the missing data, such as hourly or
daily intervals, can vary to assess a model’s ability to handle different time
scales. In this study, daily granularity is utilized for the missing gaps.
Prior research has primarily focused on imputing hourly intervals, and their
results indicate that this task is relatively simple [44, 45]. Hourly missing
data can be accurately estimated by nearby values, and even classic
statistical methods can achieve satisfactory results [56, 41]. In contrast,
missing entire days at random poses a more significant challenge for an
imputation model. To evaluate the performance of an imputation model under
various missing levels, a missing rate from 5% to 50% is employed.

####  2.3.2 Continuous missing

Continuous or consecutive missing (also known as long-term missing) refers to
gaps in time series data that occur over a prolonged period, such as missing
data that lasts for a week or a month. Imputing this type of missing data can
be more challenging as there are fewer nearby data points for reference.
Nonetheless, deep learning architectures, particularly those used in computer
vision, have successfully addressed this issue by utilizing the data structure
to impute the missing values. Similar to the setting of random missing data, a
range of 5% to 50% continuous missing rate is chosen to evaluate the model’s
performance.

###  2.4 Modeling

This study explored several modeling approaches for imputing missing energy
data, including a weekly persistence model as a naïve baseline, CNN-based
models, and PConv. Table 2) provides a summary of the models used and their
respective settings.

Model | Hyperparameters |  | Dimensions of data
---
(samples, dim1, dim2)
| Weekly
---
persistence
None | (1479, 8736, 1)
1D-CNN |  | encoder = 3 layers; decoder = 3 layers;
---
fully-connected layer at bottleneck
(1479, 8736, 1)
2D-CNN |  | encoder = 2 layers; decoder = 2 layers;
---
fully-connected layer at bottleneck
(1479, 168, 52)
PConv | encoder = 4 layers; decoder = 4 layers | (1479, 168, 52)

Table 2: Overview of models and naïve baseline regarding hyperparameters and
dimensions of data.

####  2.4.1 Weekly persistence model

The weekly persistence model, a naïve method, was implemented in this study as
a baseline for comparison with other imputation approaches [57, 58]. Unlike
sophisticated algorithms like deep neural networks or regression methods, the
weekly persistence model operates on the assumption that energy consumption
patterns persist over time. Specifically, it predicts the energy consumption
at a given time in the current week by referencing the energy consumption at
the same time in the previous week. For instance, the consumption value at
8:00 AM on a Wednesday is used to predict the energy usage for the subsequent
Wednesday at 8:00 AM. Despite its simplicity, the weekly persistence model has
been effectively used in energy modeling as a benchmark and has, in some
scenarios, outperformed classical statistical methods [59]. Its widespread use
and demonstrated effectiveness make it a valuable comparison point for
evaluating the performance of more advanced imputation methods.

####  2.4.2 Convolutional Neural Networks (CNNs)

Convolutional Neural Networks (CNNs) are a type of neural network that is
particularly well-suited for processing data with a spatial structure, such as
images or time series. Compared to Fully-Connected Neural Networks (FCNNs),
CNNs differ in that they learn local patterns in their input feature space,
while FCNNs learn global patterns. This means that after a CNN has
successfully learned a pattern in one location, it can recognize it in another
location with minimal effort. In contrast, an FCNN would have to learn the
pattern again if it appeared in a different location. As a result, CNNs are
more efficient when dealing with multidimensional data, as they require fewer
training samples to learn representations with high generalization power.
Thus, we used two CNN frameworks for data imputation: a one-dimensional CNN
(1D-CNN) for time series data and a two-dimensional CNN (2D-CNN) for reshaped
data. To fit energy data to a 2D-CNN, the data is reshaped with the time of
the week and week number serving as the two dimensions. For example, suppose
the energy data from a power meter consists of hourly readings over a year. In
that case, it is reshaped into a matrix where rows represent the time of the
week (e.g., hour 1, hour 2, …, hour 168) and columns represent the week number
(e.g., week 1, week 2, …, week 52). The reshaped data with the dimensions
($168\times 52$) is then fed into the 2D-CNN for processing and imputation of
missing values.

The CNNs in this study were implemented using an autoencoder architecture,
consisting of two components: an encoder and a decoder. The encoder converts
the input data into a compressed representation with reduced dimensions. This
compression process could capture and extract the essential features of the
input data in a more compact form, enabling efficient representation learning.
Subsequently, the decoder component receives the compressed representation
generated by the encoder and reconstructs the original data from this
compressed representation. The decoder’s role is to transform the compressed
representation back into its original format, facilitating the recovery of the
complete information. During the training process, the autoencoder aims to
minimize the difference between the original data and the reconstructed data,
encouraging the model to learn an effective compression and reconstruction
process. In the context of imputing missing values in energy data, the
autoencoder is provided with data containing synthetic missing gaps, aiming to
predict the original complete data as its output.

####  2.4.3 Partial Convolution (PConv)

In addition to utilizing traditional CNN models for imputing missing energy
data, an advanced deep learning framework called Partial Convolution (PConv)
was also tested to explore the potential of a computer-vision inpainting
technique in the energy domain. One of the main advantages of PConv is its
automatic transmission mask mechanism, which allows the model to identify the
areas that need to be repaired and use this information to achieve state-of-
the-art results. The PConv model was implemented using the U-Net architecture,
modified to use partial convolutions instead of regular convolutions. The
final architecture is shown in Figure 5. Notably, PConv excels in handling
irregularly shaped missing areas, as opposed to traditional image repair
techniques that typically only deal with regular gaps (e.g., rectangular and
circle shapes). This makes PConv particularly robust and well-suited for a
wide range of missing data scenarios.

While there are other advanced deep learning frameworks for image inpainting,
such as Generative Adversarial Networks (GANs) and diffusion models, they can
be difficult to implement and train. GANs, in particular, can be challenging
to train due to the need to balance the generator and discriminator and the
instability of the minimax loss function [60, 61]. In contrast, PConv does not
require a discriminator to be trained and has a more simplified model
architecture, making it more suitable for small datasets like the building
energy dataset used in this study. Regarding the input data for training
PConv, the energy time series data was preprocessed into a two-dimensional
matrix, identical to the input format for 2D-CNN models.

Figure 5: U-net architecture of Pconv for imputing two-dimensional energy
data.

####  2.4.4 Model hyperparameters setting and training process

All of the models compared in this study, except for the weekly persistence
model, are based on neural networks. The performance of these models is
significantly influenced by their hyperparameters, such as the number of
layers and channels. For the 1D- and 2D-CNN models, the hyperparameters were
based on the autoencoder example provided on the Keras official website
111https://blog.keras.io/building-autoencoders-in-keras.html. Keras is a
popular high-level API for deep learning, built on top of TensorFlow, one of
the most widely used deep learning frameworks. To expand the model’s capacity
for learning, a fully-connected layer is added at the bottleneck, which
enables the model to learn complex relationships between the input and output
data by allowing information to flow freely between all neurons in the layer.

The hyperparameters for the PConv model were adopted from the original paper
by Liu et al. [51]. However, due to memory constraints in the Google Colab
platform222https://colab.research.google.com/ used in this study, the image
size was adjusted to 192$\times$192 pixels from the original 256$\times$256
pixels. It’s worth noting, however, that the resolution of 192$\times$192
pixels is sufficiently large to accommodate the original dimensions of
168$\times$52, thus ensuring that no significant loss of data or features
occurs in the resizing process. The dimensions of the reshaped time series
were resized from 168$\times$52 to 192$\times$192 using interpolation. To
maintain data integrity, the dimensions were resized to 192$\times$192 using
interpolation, a method that retains the key structure and trends of the data.
An overview of the chosen hyperparameters for the models can be found in Table
2.

In this study, we employed several strategies to enhance model training and
ensure the robustness of the results. To facilitate model training, both
random and continuous synthetic missing masks with varying rates were randomly
applied to the training dataset, along with additional irregular missing
masks, which were suggested in the original paper[51]. Furthermore, an early
stopping mechanism was implemented in the PConv and CNN models to monitor the
convergence of the model training process. This mechanism observes the model’s
performance on a validation set comprising 20% of the total dataset and stops
the training process when the validation loss fails to improve over five
consecutive epochs. The training process is allowed to continue for a total of
50 epochs unless halted earlier by the early stopping mechanism. Figure 6
provides an example of the loss over epochs for the PConv model, demonstrating
the convergence of training and validation loss and the triggering of the
early stop when the patience of 5 epochs was reached due to the lack of
improvement in the validation loss.

Figure 6: Training and validation loss over epochs for the PConv model.

####  2.4.5 Evalaution metrics

To assess the model’s performance for imputing missing gaps, we employed the
Mean Squared Error (MSE) and R-squared metric, both of which are widely
accepted measures for assessing model performance in time series prediction
tasks, particularly in energy prediction. MSE is a commonly used measure of
prediction error that calculates the average squared difference between the
predicted and actual values. This metric is used to quantify the prediction
error between the predicted results and the ground truth on synthetic missing
energy data. R-squared, also known as the coefficient of determination,
quantifies the extent to which the trend of the predicted values aligns with
the actual values. In energy time series data, it is crucial to predict trends
and patterns accurately, and R-squared can help determine whether the model
can capture these trends and patterns. Though Normalized Root Mean Squared
Error (NRMSE) is another commonly used metric in this field, we opted not to
incorporate it into our evaluation. This decision was due to the pre-
processing step where all energy meter data were normalized via a max-min
normalization technique prior to the application of the inpainting algorithm,
thereby limiting the added value that the NRMSE metric could offer in this
context.

###  2.5 Experiment design

Table 3 presents the settings for the experiment, which are based on the data
preprocessing, missing masks, and modeling techniques described in previous
sections. This experimental design will allow us to evaluate the model’s
generalization performance and assess its ability to handle missing data of
various degrees under different scenarios.

| Setting of experiments
---|---
| Models and data
---
dimensions
| \- Weekly persistence model (1D)
---
\- 1D-CNN (1D)
\- 2D-CNN (2D)
\- PConv (2D)
Missing masks |  | Random and continuous days with
---
missing rates between 5 and 50%
Cross-validation | Five folds split by site ID
Data augmentation | Shifting and flipping time series

Table 3: Overview of modeling and experiment settings.

##  3 Results

In the result section, we will discuss and interpret our findings pertaining
to the prediction error values between the models and explore how the type of
meter (e.g., electricity and hot water meters) affects the predictability of
missing values. To further demonstrate the effectiveness of image imputation
in energy data, examples in the form of trend plots (1D) and heatmaps (2D)
from meters will also be presented to show how context can assist in the data
imputation process for each model.

###  3.1 Random and continuous missing

To investigate whether the extra dimension from reshaped time series impacts
the imputation of missing values, the prediction errors of the models under
different missing categories are comprehensively compared in Figure 7.
Overall, continuous missing data poses a higher challenge for imputation in
comparison to random missing data. This is due to the fact that imputing
random missing gaps can leverage more context from neighboring values. When
comparing different models’ performance in imputing continuous missing data,
we observe that the 1D-CNN model significantly underperforms in comparison to
two-dimensional models (i.e., 2D-CNN and PConv) and the weekly persistence
method. In contrast, PConv, as an image-inpainting framework, demonstrates
superior results with an MSE of 0.017, significantly lower than other models
by 27 to 61%. As for data imputation results for random missing on the right
boxplots, each model performs much better than continuous missing data, and
PConv overwhelmingly outperforms other models, with an average MSE 50% lower
than the next best model (i.e., 2D-CNN). The results show that two-dimensional
methods with reshaped energy data considerably outperform the one-dimensional
and na”ive methods. Especially, PConv has better performance than 2D-CNN,
showcasing the efficacy of image-based techniques in missing energy data
imputation. However, it is essential to note that a higher missing rate for
continuous data could significantly impact the performance of the models. For
instance, when the missing rate for continuous data rises from 5% to 10%, the
average R2 of PConv significantly decreases from 0.75 to less than 0.7 (as
shown in Figure 8). This highlights the fundamental difficulty of imputing
data when the missing rate exceeds 10% of the annual data, roughly equivalent
to a two-week absence of data.

Figure 7: Comparison between models in different imputation tasks with varying
missing rates and types. Figure 8: Zooming in on R2 comparison to observe the
decline of imputation performance as the missing rate increases

###  3.2 Breakdown by meter types

The effect of different meter types on imputation results is further explored
and shown in Figure 9. Generally, PConv offers the most accurate predictions,
followed by 2D-CNN, the weekly persistence method, and finally, 1D-CNN.
Interestingly, the weekly persistence method performs comparably to the 2D-CNN
in predicting electricity meters. This could be attributed to the more regular
patterns present in electricity meter data, which allows the weekly
persistence method to predict values accurately by mimicking values from the
previous week. In contrast, meters highly influenced by weather conditions,
such as chiller water, steam, and hot water meters, present greater prediction
difficulties for the weekly persistence method due to their inherent
irregularity. Despite PConv’s superior performance across other models, it
still struggles with predicting the continuous missing data for weather-
dependent meters. For example, as demonstrated by the bottom-left boxplots,
PConv’s imputation on steam and hot water meters cannot even reach an R
squared of 0.6 for continuous missing data. This underscores the challenge
that stems from not only the continuity of missing data and its rate but also
the dependencies of the meters on weather conditions.

Figure 9: Breakdown of model performance according to meter types.

###  3.3 Zoom-in of imputation results from visualized examples

In the time-series plots of Figure 10 and 11, we look closely at a few
examples and compare prediction results across models and missing settings. In
the case of random missing data shown in plots 10(a) and 10(b), we found that
PConv and 2D-CNN, both employing the two-dimensional context, perform
remarkably well in the prediction of time-series profiles. Conversely, the
1D-CNN fails to capture energy profiles, and the weekly persistence method
only offers moderate predictive outcomes. This discrepancy is particularly
noticeable in the case of the hot water meter due to its weather dependence.
In 11(a) and 11(b), prediction results with more challenging continuous
missing gaps are shown. Although both PConv and 2D-CNN predictions exhibit
stable periodic patterns closely aligned with the original data, the 2D-CNN
model struggles to capture sudden shifts in overall trends. In contrast, PConv
and the weekly persistence method are able to account for such trend changes
in their imputation predictions. As for the prediction of hot water usage with
continuous missing data, both PConv and 2D-CNN outperform other methods,
benefiting from the two-dimensional context. The weekly persistence method and
1D-CNN perform poorly in predicting weather-dependent hot water meter, owing
to the irregularity in its usage patterns. Figure 12 illustrates the same
results in the form of heatmaps, emphasizing the benefits of a two-dimensional
context in the imputation process.

(a) 10% Random missing (electricity meter).

(b) 10% Random missing (hot water meter).

Figure 10: Time series plots of power meters for comparing imputation results
from different methods for 10% random missing data.

(a) 10% Continuous missing (electricity meter).

(b) 10% Continuous missing (hot water meter).

Figure 11: Time series plots of power meters for comparing imputation results
from different methods for 10% continuous missing data.

(a) 10% Continuous missing (electricity meter).

(b) 10% Continuous missing (hot water meter).

(c) 10% Random missing (electricity meter)

(d) 10% Random missing (hot water meter)

Figure 12:  Visual comparisons of models in 2D heatmaps. Each example from
left to right: input image, PConv, and ground truth. All images have a
$192\times 192$ pixel dimension.

Despite the promising results shown previously, there were instances where the
predicted values deviated significantly from the actual values, particularly
when the missing rate escalates to 30%, as illustrated in Figure 13. In Figure
13(a), 1D-CNN and the weekly persistence method fail to capture the weekly
regularity of energy data, resulting in constant values appearing uniformly
colored. Meanwhile, the predictions generated by the 2D-CNN appear to be
noisy. Even though PConv outperforms the other models considerably, it still
shows unclear areas, likely resulting from the lack of additional weather
information when attempting to impute such long-term missing data. The
difficulties become even more pronounced in Figure 13(b), which features 30%
continuous missing data in the hot water meter. None of the models are able to
generate results of satisfactory quality. The absence of weather information,
which is particularly crucial for weather-dependent meters, significantly
hampers the predictive capabilities of the models. These failures demonstrate
the limitations of imputation models when confronted with long-term continuous
missing gaps.

(a)

(b)

Figure 13: Some examples of failed prediction with high continuous missing
rate.

Overall, the order of performance from best to worst is PConv, 2D-CNN, weekly
persistence, and 1D-CNN. PConv, a modern framework for image inpainting with
deep learning, consistently performs significantly better than the other
models in any task of missing imputation. Among the other models, 2D-CNN
marginally outperforms the weekly persistence model, whereas 1D-CNN
significantly lags behind. However, the effectiveness of 2D-CNN and weekly
persistence can fluctuate depending on the specific circumstances. In
situations where there’s a trend shift within the missing gaps, weekly
persistence might perform slightly better than 2D-CNN as it can incorporate
this trend shift into its predictions by replicating the values from the
adjacent week. Furthermore, when the prediction target is to predict missing
gaps for weather-dependent meters, such as hot water meters, 2D-CNN performs
better than the weekly persistence model. The latter primarily depends on the
regularity of the time series, which can be a limiting factor in such cases.

##  4 Discussion

In this study, we explored the use of various machine learning models,
including advanced image inpainting techniques, to impute missing data from
energy meters at different missing rates in both random and continuous
fashion.

###  4.1 Evaluation and comparison of imputation models

This study evaluated a range of imputation models, from a naïve baseline
(i.g., weekly persistence model) to advanced ones (2D-CNN and PConv). Notably,
this study introduced a unique aspect to the field by incorporating advanced
image inpainting techniques, specifically PConv, for imputing missing data.
Contrary to the majority of previous works that concentrated on imputing
short-term missing data (e.g., hourly or daily), our study broadened its scope
to cover longer-term periods, such as weekly or even monthly missing data. The
relative ease of imputing short-term missing data based on nearby values was
affirmed in the study, thus differentiating our work owing to its long-term
focus. Moreover, the superior performance of PConv over other methods
underlined its efficacy, especially when faced with the challenging task of
long-term imputation.

Furthermore, most past studies were benchmarked on a restricted number of
meters within specific buildings or sites, while the proposed method in this
study was comprehensively validated on a benchmark dataset with thousands of
meters worldwide. Consequently, the distinct nature of this study’s tasks
involving long-term missing data and the inherent complexities of the dataset
make it less directly comparable to the imputation results of earlier studies.
Nonetheless, to maintain methodological continuity with past studies, our
research incorporated some common deep learning frameworks, such as the CNN-
based autoencoder and naïve weekly persistence model, as benchmarking
references. The inclusion of these model frameworks, various missing periods,
and our benchmark dataset ensures that our deep learning method based on
reshaped data contributes to advancing data imputation in the field.

###  4.2 Exploring the maximum time period for imputing continuously missing
data

This study assessed the feasibility and limitation of employing energy time
series as images for imputing missing data. Our results demonstrate that,
while the proposed approach PConv outperformed other models via learning
context within images, it struggled with the prediction of extended periods of
continuous missing values. Specifically, when the missing rate of continuous
data exceeded 10%, approximating to about a month of missing, the average R2
of PConv dropped below 0.7. Additionally, the imputation performance of PConv
significantly deteriorated when applied to weather-dependent meters, primarily
due to less predictable regularity compared to electricity meters. These
findings reveal that, even though image-based deep learning models may improve
the performance of imputation, their capability to accurately forecast missing
values over extended periods remains restricted.

###  4.3 Challenges and future direction in imputing data missing

Imputing continuous missing data, especially with increasing missing rates,
poses significant challenges due to the lack of adjacent context values
required for long-term prediction. This study demonstrates that the advanced
deep learning frameworks in computer vision, such as 2D-CNN and PConv, exhibit
superior performance via learning context from time-series data represented as
images. However, even with these advanced methods, predicting long-term
continuous missing values remains problematic. This is evident in the
performance of PConv, which noticeably deteriorates when the proportion of
continuous missing data exceeds 10%.

This research emphasizes the necessity for comprehensive context information,
such as weather and occupant-related data, to enhance the performance of the
imputation model. Without these contextual data, the model may struggle to
generate accurate predictions for long-term missing values, given that such
predictions are typically influenced by weather patterns and human activity.
Encouragingly, future work in this area could benefit from strategies employed
in image inpainting studies that integrated additional layers of data to
improve accuracy. For instance, one study integrated a texture information
layer into the imputation process, successfully enhancing the performance by
including detailed outlines of human faces or the sketch drawings of buildings
[62]. Applying the same concept to building energy data imputation, weather
and occupant behavior data could serve as valuable additional data layers to
improve predictions. As an example, the utility of Google Trends data as a
human behavior proxy to enhance energy prediction has been proven [38]. Thus,
incorporating such additional data layers, human behavior data and weather
data, into data imputation for building energy data presents a promising
direction for future research. This approach could potentially address the
current challenges in imputing long-term missing data and contribute to more
accurate and reliable predictions.

##  5 Conclusion

This study aimed to investigate whether the missing imputation of energy data
could be improved by integrating an extra dimension through reshaping energy
time series. The prediction performance was extensively evaluated by testing
various model frameworks and diverse missing rates and types. The results
revealed the beneficial role of the two-dimensional energy data structure in
data imputation, particularly when used with advanced image-based deep
learning models like PConv, leading to better imputation performance than
classical CNN models and the weekly persistence method. Even when applied to
subsets of different meter types, PConv is able to maintain consistent high
prediction accuracy across different meter types. However, a challenge arose
when PConv attempted to accurately impute long-term continuous missing values
beyond a 10% missing data rate. This situation caused the average R2 of PConv
to decrease below 0.7, thereby revealing a limitation of the proposed
imputation model and reshaped energy data.

To the best of our knowledge, this is the first study to extensively verify
this imputation method using more than one thousand power meters in the field
of building energy. This research demonstrates the potential of employing two-
dimensional reshaped energy data in conjunction with image-based deep learning
techniques for imputing missing energy data. The proposed method, which has
been thoroughly tested and validated on a wide-ranging benchmark dataset with
diverse meter types and sources, is scalable and generalizable. With the
future availability of more energy datasets, our approach holds promise as a
potent tool for automatically imputing missing data on a significant scale.
This could offer substantial benefits to the industry by facilitating enhanced
energy predictions and management based on the imputed data. Furthermore, the
proposed method could find wider application across other time series data in
the built environment, such as HVAC, lighting, and appliances data, offering a
holistic benefit to the building industry.

##  6 Limitations

While this study yielded exciting results and made significant contributions,
there are several limitations that should be acknowledged and to be addressed
in the future. The first limitation is that our focus was on imputing
continuous and random missing gaps in separate tasks, neglecting other forms
of missing data patterns encountered in the real world. For example, multiple
scattered continuous missing gaps or a combination of both random and
continuous missing were not specifically considered and assessed in this
study. Secondly, the study did not consider the integration of weather data,
despite its well-established impact on energy consumption. Their inclusion in
the imputation models could provide valuable contextual information for more
accurate predictions. Similarly, the integration of calendar data, which
captures human events like national holidays or extended vacations that
influence energy consumption, could further enhance the imputation process.
Investigating the integration of these contextual factors would contribute to
more comprehensive and accurate imputation models. These limitations suggest
that further research is needed to fully explore and leverage the potential of
image-based techniques for imputing missing energy data.

##  7 Reproducibility

This analysis can be reproduced using the data and code from the following
GitHub repository: https://github.com/buds-lab/Filling-time-series-gaps-using-
image-techniques.

## CRediT author statement

CF: Conceptualization, Methodology, Software, Formal Analysis, Investigation,
Data Curation, Visualization, Writing - Original Draft; MQ: Methodology,
Writing - Reviewing & Editing; ZN: Methodology, Writing - Reviewing & Editing;
CM: Conceptualization, Methodology, Resources, Writing - Reviewing & Editing,
Supervision, Project administration, Funding acquisition.

## Funding

This research is funded by the NUS-based Singapore MOE Tier 1 Grant titled
Ecological Momentary Assessment (EMA) for Built Environment Research
(A-0008301-01-00).

## Acknowledgements

## References

  * [1] H. Kazmi, C. Fu, C. Miller, Ten questions concerning data-driven modelling and forecasting of operational energy demand at building and urban scale, Building and Environment 239 (2023) 110407.
  * [2] N. Somu, G. R. MR, K. Ramamritham, A deep learning framework for building energy consumption forecast, Renewable and Sustainable Energy Reviews 137 (2021) 110591.
  * [3] A.-D. Pham, N.-T. Ngo, T. T. H. Truong, N.-T. Huynh, N.-S. Truong, Predicting energy consumption in multiple buildings using machine learning for improving energy efficiency and sustainability, Journal of Cleaner Production 260 (2020) 121082.
  * [4] R. Olu-Ajayi, H. Alaka, I. Sulaimon, F. Sunmola, S. Ajayi, Building energy consumption prediction for residential buildings using deep learning and other machine learning techniques, Journal of Building Engineering 45 (2022) 103406\.
  * [5] A. Dolara, A. Gandelli, F. Grimaccia, S. Leva, M. Mussetta, Weather-based machine learning technique for day-ahead wind power forecasting, in: 2017 IEEE 6th international conference on renewable energy research and applications (ICRERA), IEEE, 2017, pp. 206–209.
  * [6] A. Gensler, J. Henze, B. Sick, N. Raabe, Deep learning for solar power forecasting—an approach using autoencoder and lstm neural networks, in: 2016 IEEE international conference on systems, man, and cybernetics (SMC), IEEE, 2016, pp. 002858–002865.
  * [7] C. Tian, T. Niu, W. Wei, Developing a wind power forecasting system based on deep learning with attention mechanism, Energy 257 (2022) 124750.
  * [8] G. Hafeez, K. S. Alimgeer, Z. Wadud, Z. Shafiq, M. U. Ali Khan, I. Khan, F. A. Khan, A. Derhab, A novel accurate and fast converging deep learning-based model for electrical energy consumption forecasting in a smart grid, Energies 13 (9) (2020) 2244.
  * [9] E. E. Avalos, M. R. Licea, H. R. González, A. E. Calderón, A. B. Gutiérrez, F. P. Pinal, Comparative analysis of multivariable deep learning models for forecasting in smart grids, in: 2020 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC), Vol. 4, IEEE, 2020, pp. 1–6.
  * [10] G. Hafeez, K. S. Alimgeer, I. Khan, Electric load forecasting based on deep learning and optimized by heuristic algorithm in smart grid, Applied Energy 269 (2020) 114915.
  * [11] N. G. Paterakis, E. Mocanu, M. Gibescu, B. Stappers, W. van Alst, Deep learning versus traditional machine learning methods for aggregated energy demand prediction, in: 2017 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), IEEE, 2017, pp. 1–6.
  * [12] S. Touzani, J. Granderson, S. Fernandes, Gradient boosting machine for modeling the energy consumption of commercial buildings, Energy and Buildings 158 (2018) 1533–1543.
  * [13] L. Wang, E. W. M. Lee, R. K. Yuen, W. Feng, Cooling load forecasting-based predictive optimisation for chiller plants, Energy and Buildings 198 (2019) 261–274.
  * [14] M. H. Shams, H. Niaz, B. Hashemi, J. J. Liu, P. Siano, A. Anvari-Moghaddam, Artificial intelligence-based prediction and analysis of the oversupply of wind and solar energy in power systems, Energy Conversion and Management 250 (2021) 114892.
  * [15] P. Gao, M. Wang, S. G. Ghiocel, J. H. Chow, B. Fardanesh, G. Stefopoulos, Missing data recovery by exploiting low-dimensionality in power system synchrophasor measurements, IEEE Transactions on Power Systems 31 (2) (2015) 1006–1013.
  * [16] J. Sun, H. Liao, B. R. Upadhyaya, A robust functional-data-analysis method for data recovery in multichannel sensor systems, IEEE Transactions on Cybernetics 44 (8) (2013) 1420–1431.
  * [17] J. Stones, A. Collinson, Power quality, power engineering journal 15 (2) (2001) 58–64.
  * [18] I. Khan, A. Capozzoli, S. P. Corgnati, T. Cerquitelli, Fault detection analysis of building energy consumption using data mining techniques, Energy Procedia 42 (2013) 557–566.
  * [19] Y. Choi, S. Yoon, Autoencoder-driven fault detection and diagnosis in building automation systems: Residual-based and latent space-based approaches, Building and Environment 203 (2021) 108066.
  * [20] C. Miller, P. Arjunan, A. Kathirgamanathan, C. Fu, J. Roth, J. Y. Park, C. Balbach, K. Gowri, Z. Nagy, A. D. Fontanini, et al., The ashrae great energy predictor iii competition: Overview and results, Science and Technology for the Built Environment 26 (10) (2020) 1427–1447.
  * [21] C. Miller, B. Picchetti, C. Fu, J. Pantelic, Limitations of machine learning for building energy prediction: Ashrae great energy predictor iii kaggle competition error analysis, Science and Technology for the Built Environment 28 (5) (2022) 610–627.
  * [22] M. Gulati, P. Arjunan, Lead1. 0: a large-scale annotated dataset for energy anomaly detection in commercial buildings, in: Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, 2022, pp. 485–488.
  * [23] C. Fu, P. Arjunan, C. Miller, Trimming outliers using trees: winning solution of the large-scale energy anomaly detection (lead) competition, in: Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, 2022, pp. 456–461.
  * [24] Z. Zhang, Y. Chen, H. Wang, Y. Wang, Q. Fu, Y. Lu, A multivariate time series anomaly detection model based on graph attention mechanism in energy consumption of intelligent buildings, in: 2022 Tenth International Conference on Advanced Cloud and Big Data (CBD), IEEE, 2022, pp. 122–127.
  * [25] Z. Wang, L. Wang, Y. Tan, J. Yuan, Fault detection based on bayesian network and missing data imputation for building energy systems, Applied Thermal Engineering 182 (2021) 116051.
  * [26] J. Y. Park, X. Yang, C. Miller, P. Arjunan, Z. Nagy, Apples or oranges? identification of fundamental load shape profiles for benchmarking buildings using a large and diverse dataset, Applied energy 236 (2019) 1280–1295.
  * [27] J. Y. Park, E. Wilson, A. Parker, Z. Nagy, The good, the bad, and the ugly: Data-driven load profile discord identification in a large building portfolio, Energy and Buildings 215 (2020) 109892.
  * [28] C. Miller, Z. Nagy, A. Schlueter, Automated daily pattern filtering of measured building performance data, Automation in Construction 49 (2015) 1–17.
  * [29] M. M. Abdelrahman, S. Zhan, C. Miller, A. Chong, Data science for building energy efficiency: A comprehensive text-mining driven review of scientific literature, Energy & Buildings (2021) 110885doi:10.1016/j.enbuild.2021.110885.
  * [30] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Nets, Corrosion (2014) iiiarXiv:1011.1669v3, doi:10.1016/B978-0-408-00109-0.50001-8.
  * [31] Y. Ye, M. Strong, Y. Lou, C. A. Faulkner, W. Zuo, S. Upadhyaya, Energy & Buildings Evaluating performance of different generative adversarial networks for large-scale building power demand prediction, Energy & Buildings 269 (2022) 112247.  doi:10.1016/j.enbuild.2022.112247.
  * [32] Z. Wang, T. Hong, Generating realistic building electrical load profiles through the Generative Adversarial Network (GAN), Energy and Buildings 224 (2020) 110299.  doi:10.1016/j.enbuild.2020.110299.
  * [33] K. Yan, A. Chong, Y. Mo, Generative adversarial network for fault detection diagnosis of chillers, Building and Environment 172 (December 2019) (2020) 106698\.  doi:10.1016/j.buildenv.2020.106698.
  * [34] M. Quintana, S. Schiavon, K. W. Tham, C. Miller, Balancing thermal comfort datasets: We GAN, but should we?, in: Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, Virtual Event, Japan, 2020, pp. 120–129.  arXiv:2009.13154, doi:10.1145/3408308.3427612.
  * [35] A. Liguori, R. Markovic, M. Ferrando, J. Frisch, F. Causone, C. van Treeck, Augmenting energy time-series for data-efficient imputation of missing values, Applied Energy 334 (2023) 120701.
  * [36] B. K. Iwana, S. Uchida, An empirical survey of data augmentation for time series classification with neural networks, Plos one 16 (7) (2021) e0254841.
  * [37] S. Demir, K. Mincev, K. Kok, N. G. Paterakis, Data augmentation for time series regression: Applying transformations, autoencoders and adversarial networks to electricity price forecasting, Applied Energy 304 (2021) 117695.
  * [38] C. Fu, C. Miller, Using google trends as a proxy for occupant behavior to predict building energy consumption, Applied Energy 310 (2022) 118343.
  * [39] M. Sun, C. Han, Q. Nie, J. Xu, F. Zhang, Q. Zhao, Understanding building energy efficiency with administrative and emerging urban big data by deep learning in Glasgow, Energy and Buildings 273 (2022) 112331.  doi:10.1016/j.enbuild.2022.112331.
  * [40] A. Chong, K. P. Lam, W. Xu, O. T. Karaguzel, Y. Mo, Imputation of missing values in building sensor data, ASHRAE and IBPSA-USA SimBuild 6 (2016) 407–14.
  * [41] M.-C. Wang, C.-F. Tsai, W.-C. Lin, Towards missing electric power data imputation for energy management systems, Expert Systems with Applications 174 (2021) 114743.
  * [42] P. Wu, M. Xu, L. Cheng, An improved cnn-based completion method for power grid middle platform data, in: Journal of Physics: Conference Series, Vol. 1815, IOP Publishing, 2021, p. 012034.
  * [43] A. Lucbert, J. van der Niet, A. Corson, M. Weij, R. I. van der Elst, J. M. M. de Juan, T. B. S. Rahola, Time series building energy systems data imputation, in: CLIMA 2022 conference, 2022.
  * [44] J. Ma, J. C. Cheng, F. Jiang, W. Chen, M. Wang, C. Zhai, A bi-directional missing data imputation scheme based on lstm and transfer learning for building energy data, Energy and Buildings 216 (2020) 109941.
  * [45] A. Liguori, R. Markovic, T. T. H. Dam, J. Frisch, C. van Treeck, F. Causone, Indoor environment data time-series reconstruction using autoencoder neural networks, Building and Environment 191 (2021) 107623.
  * [46] X. Wang, Y. Chen, W. Ruan, Q. Gao, G. Ying, L. Dong, Intelligent detection and recovery of missing electric load data based on cascaded convolutional autoencoders, Scientific Programming 2020.
  * [47] P. A. Esquef, L. W. Biscainho, An efficient model-based multirate method for reconstruction of audio signals across long gaps, IEEE transactions on audio, speech, and language processing 14 (4) (2006) 1391–1400.
  * [48] A. Marafioti, N. Perraudin, N. Holighaus, P. Majdak, A context encoder for audio inpainting, IEEE/ACM Transactions on Audio, Speech, and Language Processing 27 (12) (2019) 2362–2372.
  * [49] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A. A. Efros, Context encoders: Feature learning by inpainting, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536–2544.
  * [50] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, T. S. Huang, Generative image inpainting with contextual attention, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5505–5514.
  * [51] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, B. Catanzaro, Image inpainting for irregular holes using partial convolutions, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 85–100.
  * [52] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, L. Van Gool, Repaint: Inpainting using denoising diffusion probabilistic models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11461–11471.
  * [53] J. Jam, C. Kendrick, K. Walker, V. Drouard, J. G.-S. Hsu, M. H. Yap, A comprehensive review of past and present image inpainting methods, Computer vision and image understanding 203 (2021) 103147.
  * [54] C. Miller, P. Arjunan, A. Kathirgamanathan, C. Fu, J. Roth, J. Y. Park, C. Balbach, K. Gowri, Z. Nagy, A. D. Fontanini, et al., The ashrae great energy predictor iii competition: Overview and results, Science and Technology for the Built Environment 26 (10) (2020) 1427–1447.
  * [55] M. Quintana, T. Stoeckmann, J. Y. Park, M. Turowski, V. Hagenmeyer, C. Miller, Aldi++: Automatic and parameter-less discord and outlier detection for building energy load profiles, Energy and Buildings 265 (2022) 112096.
  * [56] L. Zhang, A pattern-recognition-based ensemble data imputation framework for sensors from building energy systems, Sensors 20 (20) (2020) 5947.
  * [57] R. De Coninck, L. Helsen, Practical implementation and evaluation of model predictive control for an office building in brussels, Energy and Buildings 111 (2016) 290–298.
  * [58] P. S. Sauter, P. Karg, M. Kluwe, S. Hohmann, Load forecasting in distribution grids with high renewable energy penetration for predictive energy management systems, in: 2018 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), IEEE, 2018, pp. 1–6.
  * [59] M. Weber, M. Turowski, H. K. Çakmak, R. Mikut, U. Kühnapfel, V. Hagenmeyer, Data-driven copy-paste imputation for energy time series, IEEE Transactions on Smart Grid 12 (6) (2021) 5409–5419.
  * [60] M. Arjovsky, S. Chintala, L. Bottou, Wasserstein generative adversarial networks, in: International conference on machine learning, PMLR, 2017, pp. 214–223.
  * [61] M. Arjovsky, L. Bottou, Towards principled methods for training generative adversarial networks, arXiv preprint arXiv:1701.04862.
  * [62] X. Guo, H. Yang, D. Huang, Image inpainting via conditional texture and structure dual generation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14134–14143.


Sentisead offered improvement over each stand-alone tool based on the five
error categories.

Sentisead infrastructure could be leveraged to add the polarity labels of any
newly introduced SE-specific stand-alone sentiment detection tool. We envision
that this flexibility of Sentisead will allow the co-creation of two new-
breeds of efforts in software engineering research:  (1) More efforts to
create new SE-specific stand-alone sentiment detection tool focusing on
diverse SE datasets, and  (2) Studying  (a) the incorporation of diverse
complementary SE-specific stand-alone sentiment detection tools.  (b) The
effectiveness of diverse features, such as fine-grained emotions from various
tools into Sentistead (or its extension).

As we noted in Section 1, recent advances in deep learning-based SE-specific
sentiment detection shows more promising results than those observed by Lin et
al. (Lin-SentimentDetectionNegativeResults-ICSE2018, ). While within the
limited space of this paper, we attempted to offer a comprehensive analysis by
investigating the feasibility of developing a hybrid tool based on rule-based,
shallow and deep learning stand-alone classifiers, we leave following
extensions to Sentisead for future work:

(1) SentiEmoji (Chen-SentiEmoji-FSE2019, ), a tool developed by Chen et al.
shows that we can use emoji-powered data from Twitter and Facebook to further
improve sentiment classification in SE. SentiEmoji (Chen-SentiEmoji-FSE2019, )
also shows that an embedding based on the emoji information and the textual
contents can be more useful. Sentisead could leverage such emoji-powered
information as features besides word-embedding or bag of words.  (2) Deep
learning models, like PTMs, are found to offer good performance even with low
amount of data due to their design as pre-trained models. However, deep
learning models also benefit from more data. A problem with SE-specific
sentiment detection is the diversity of datasets. With emoji-powered labels
like those used by SentiEmoji (Chen-SentiEmoji-FSE2019, ), we may be able to
get more data with automatic polarity labels, which then can help with domain-
specific nuances more. Therefore, Sentisead could benefit from such large-
scale data, especially for PTMs like a RoBERTa as an ensembler (see RQ6).

We conclude that our empirical study with Sentisead offers valuable insights
to the field of SE-specific sentiment detection research with empirical
insights, actionable results, and a hybrid tool Sentisead that shows marginal
to promising performance over stand-alone SE-specific sentiment detection
tools.

## Acknowledgment

We would like to thank the anonymous reviewers, who helped improve our work
with comments and suggestions. We would like to thank Nibir Mandal, Mohamed
Raed El Aoun, and Houssem Ben Braiek, who helped run the experiments with the
advanced language-based pre-trained models in Section 4.

## References

  * [1] T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi.  Senticr: A customized sentiment analysis tool for code review interactions.  In Proceedings of the 32nd International Conference on Automated Software Engineering, pages 106–111, 2017.
  * [2] I. E. Asri, N. Kerzazi, G. Uddin, F. Khomh, and M. J. Idrissi.  An empirical study of sentiments in code reviews.  Information and Software Technology, 114:37–54, 2019.
  * [3] E. Biswas, M. E. Karabulut, L. Pollock, and K. Vijay-Shanker.  Achieving reliable sentiment analysis in the software engineering domain using bert.  In IEEE International Conference on Software Maintenance and Evolution, pages 162–173, 2020.
  * [4] E. Biswas, K. Vijay-Shanker, and L. Pollock.  Exploring word embedding techniques to improve sentiment analysis of software engineering texts.  In Proceedings of the 16th International Conference on Mining Software Repositories, pages 68–78, 2019.
  * [5] F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli.  Sentiment polarity detection for software development.  Journal Empirical Software Engineering, pages 2543–2584, 2017.
  * [6] F. Calefato, F. Lanubile, and N. Novielli.  Emotxt: A toolkit for emotion recognition from text.  In Proc. 7th Affective Computing and Intelligent Interaction, page 2, 2017.
  * [7] C. Castaldi, A. Blaz, and K. Becker.  Sentiment analysis in tickets for IT support.  In IEEE/ACM 13th Working Conference on Mining Software Repositories, pages 235–246, 2016.
  * [8] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of Artificial Intelligence Research, 16(1):321–357, 2002\.
  * [9] Z. Chen, Y. Cao, X. Lu, Q. Mei, and X. Liu.  Sentimoji: an emoji-powered learning approach for sentiment analysis in software engineering.  In 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 841–852, 2019.
  * [10] J. Cohen, S. G. West, L. Aiken, and P. Cohen.  Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.  Lawrence Erlbaum Associates, 3rd edition, 2002.
  * [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  Technical report, https://arxiv.org/abs/1810.04805, 2018.
  * [12] P. P. B. Filho, L. Avanco, T. A. S. Pardo, and M. G. V. Nunes.  Nilc_usp: An improved hybrid system for sentiment analysis in twitter messages.  In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 428–432, 2014.
  * [13] D. Gachechiladze, F. Lanubile, N. Novielli, and A. Serebrenik.  Anger and its direction in collaborative software development.  In 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, pages 11–14, 2017.
  * [14] B. Ghotra, S. McIntosh, and A. E. Hassan.  Revisiting the impact of classification techniques on the performance of defect prediction models.  In Proceedings of the 37th International Conference on Software Engineering, pages 789–800, 2015.
  * [15] P. Goncalves, M. Araujo, F. Benevenuto, and M. Cha.  Comparing and combining sentiment analysis methods.  In Proceedings of the first ACM conference on Online social networks, pages 27–38, 2013.
  * [16] E. Guzman, R. Alkadhi, , and N. Seyf.  A needle in a haystack: What do twitter users say about software?  In 24th IEEE International Requirements Engineering Conference, pages 96–105, 2016.
  * [17] E. Guzman, D. Azócar, and Y. Li.  Sentiment analysis of commit comments in github: an empirical study.  In Proceedings of the 11th Working Conference on Mining Software Repositories, pages 352–355, 2014.
  * [18] E. Guzman and B. Bruegge.  Towards emotional awareness in software development teams.  In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 671–674, 2013.
  * [19] E. Guzman and B. Bruegge.  Towards emotional awareness in software development teams.  In Proceedings of the 7th Joint Meeting on Foundations of Software Engineering, pages 671–674, 2013.
  * [20] A. E. Hassan.  Predicting faults using the complexity of code changes.  In Proc. 31st International Conference on Software Engineering, pages 78–89, 2009.
  * [21] M. Hu and B. Liu.  Mining and summarizing customer reviews.  In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168–177, 2004.
  * [22] M. R. Islam and M. F. Zibran.  Leveraging automated sentiment analysis in software engineering.  In Proc. 14th International Conference on Mining Software Repositories, pages 203–214, 2017.
  * [23] M. R. Islam and M. F. Zibran.  Deva: Sensing emotions in the valence arousal space in software engineering text.  In 33rd Annual ACM Symposium on Applied Computing, page 1536–1543, 2018.
  * [24] M. R. Islam and M. F. Zibran.  Deva: Sensing emotions in the valence arousal space in software engineering text.  In 33rd Annual ACM Symposium on Applied Computing, page 1536–1543, 2018.
  * [25] R. Jongeling, S. Datta, and A. Serebrenik.  Choosing your weapons: On sentiment analysis tools for software engineering research.  In Proceedings of the 31st International Confernece on Software Maintenance and Evolution, 2015.
  * [26] R. Jongeling, P. Sarkar, S. Datta, and A. Serebrenik.  On negative results when using sentiment analysis tools for software engineering research.  Journal Empirical Software Engineering, 22(5):2543–2584, 2017.
  * [27] F. Khomh, B. Chan, Y. Zou, and A. E. Hassan.  An entropy evaluation approach for triaging field crashes: A case study of mozilla firefox.  In Proceedings of the 2011 18th Working Conference on Reverse Engineering, pages 261–270, 2011.
  * [28] A. J. Ko, B. A. Myers, and D. H. Chau.  A linguistic analysis of how people describe software problems.  In 2005 IEEE Symposium on Visual Languages and Human-Centric Computing, pages 127–134, 2005.
  * [29] D. Lab.  An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets (Online Appendix).  https://github.com/disa-lab/HybridSESentimentTOSEM, 21 April 2021 (last accessed).
  * [30] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut.  ALBERT: A lite bert for self-supervised learning of language representations.  Technical report, https://arxiv.org/abs/1909.11942, 2020.
  * [31] B. Lin, F. Zampetti, G. Bavota, M. D. Penta, and M. Lanza.  Pattern-based mining of opinions in Q&A websites.  In Proc. 41st International Conference on Software Engineering, pages 548–559, 2019.
  * [32] B. Lin, F. Zampetti, G. Bavota, M. D. Penta, M. Lanza, and R. Oliveto.  Sentiment analysis for software engineering: how far can we go?  In Proc. 40th International Conference on Software Engineering, pages 94–104, 2018.
  * [33] B. Liu.  Sentiment Analysis and Opinion Mining.  Morgan & Claypool Publishers, 1st edition, May 2012.
  * [34] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  RoBERTa: A robustly optimized bert pretraining approach.  Technical report, https://arxiv.org/abs/1907.11692, 2019.
  * [35] W. Maalej, Z. Kurtanović, H. Nabil, and C. Stanik.  On the automatic classification of app reviews.  In International Requirements Engineering Conference, page 311–331, 2016.
  * [36] R. Maipradit, H. Hata, and K. Matsumoto.  Sentiment classification using n-gram inverse document frequency and automated machine learning.  IEEE Software, 36(5):65–70, 2019.
  * [37] C. D. Manning, P. Raghavan, and H. Schütze.  An Introduction to Information Retrieval.  Cambridge Uni Press, 2009.
  * [38] M. Mȧntylȧ, B. Adams, G. Destefanis, D. Graziotin, and M. Ortu.  Mining valence, arousal, and dominance – possibilities for detecting burnout and productivity?  In Proceedings of the 13th Working Conference on Mining Software Repositories, pages 247–258, 2016.
  * [39] A. T. Misirli, A. B. Bener, and B. Turhan.  An industrial case study of classifier ensembles for locating software defects.  Software Quality Journal, 19(3):515–536, 2011.
  * [40] A. Murgia, P. Tourani, B. Adams, and M. Ortu.  Do developers feel emotions? an exploratory analysis of emotions in software artifacts.  In Proceedings of the 11th Working Conference on Mining Software Repositories, 2014.
  * [41] N. Novielli, A. Begel, and W. Maalej.  Introduction to the special issue on affect awareness in software engineering.  Journal of Systems and Software, 148:180–182, 2019.
  * [42] N. Novielli, F. Calefato, D. Dongiovanni, D. Girardi, and F. Lanubile.  Can we use se-specific sentiment analysis tools in a cross-platform setting?  In 17th International Conference on Mining Software Repositories, page 11, 2020.
  * [43] N. Novielli, F. Calefato, and F. Lanubile.  The challenges of sentiment detection in the social programmer ecosystem.  In Proceedings of the 7th International Workshop on Social Software Engineering, pages 33–40, 2015.
  * [44] N. Novielli, F. Calefato, and F. Lanubile.  A gold standard for emotion annotation in stack overflow.  In Proceedings of the 15th International Conference on Mining Software Repositories (Data Showcase), page 4, 2018.
  * [45] N. Novielli, D. Girardi, and F. Lanubile.  A benchmark study on sentiment analysis for software engineering research.  In Proceedings of the 15th International Conference on Mining Software Repositories, page 12, 2018.
  * [46] N. Novielli and A. Serebrenik.  Sentiment and emotion in software engineering.  IEEE Software, 36(5):6–23, 2019.
  * [47] D. D. Nucci, F. Palomba, R. Oliveto, and A. D. Lucia.  Dynamic selection of classifiers in bug prediction: An adaptive method.  IEEE Transactions on Emerging Topics in Computational Intelligence, 1(3):202–212, 2017.
  * [48] M. Ortu, B. Adams, G. Destefanis, P. Tourani, M. Marchesi, and R. Tonelli.  Are bullies more productive? empirical study of affectiveness vs. issue fixing time.  In Proceedings of the 12th Working Conference on Mining Software Repositories, pages 303–313, 2015.
  * [49] M. Ortu, B. Adams, G. Destefanis, P. Tourani, M. Marchesi, and R. Tonelli.  Are bullies more productive? empirical study of affectiveness vs. issue fixing time.  In Proceedings of the 12th Working Conference on Mining Software Repositories, 2015.
  * [50] M. Ortu, A. Murgia, G. Destefanis, P. Tourani, R. Tonelli, M. L. Marchesi, and B. Adams.  The emotional side of software developers in jira.  In 13th International Conference on Mining Software Repositories, pages 480–483, 2016.
  * [51] B. Pang, L. Lee, and S. Vaithyanathan.  Thumbs up? sentiment classification using machine learning techniques.  In Conference on Empirical Methods in Natural Language Processing, pages 79–86, 2002.
  * [52] S. Panichella, A. D. Sorbo, E. Guzman, C. A. Visaggio, G. Canfora, and H. C. Gall.  How can i improve my app? classifying user reviews for software maintenance and evolution.  In IEEE International Conf. on Software Maintenance and Evolution, pages 281–290, 2015.
  * [53] J. Petric, D. Bowes, T. Hall, B. Christianson, and N. Baddoo.  Building an ensemble for software defect prediction based on diversity selection.  In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, page Article No. 46, 2016.
  * [54] D. Pletea, B. Vasilescu, and A. Serebrenik.  Security and emotion: sentiment analysis of security discussions on github.  In Proceedings of the 11th Working Conference on Mining Software Repositories, pages 348–351, 2014.
  * [55] scikit learn.  Machine Learning in Python.  http://scikit-learn.org/stable/index.html#, 2017.
  * [56] F. Sebastiani.  Machine learning in automated text categorization.  Journal of ACM Computing Surveys, 34(1):1–47, 2002.
  * [57] C. E. Shannon.  A mathematical theory of communication.  The Bell System Technical Journal, 27(3):379–423, 1948.
  * [58] V. Sinha, A. Lazar, and B. Sharif.  Analyzing developer sentiment in commit logs.  In 13th International Conference on Mining Software Repositories, pages 281–290, 2016.
  * [59] R. Socher, A. Perelygin, J. Wu, C. Manning, A. Ng, and J. Chuang.  Recursive models for semantic compositionality over a sentiment treebank.  In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP), page 12, 2013.
  * [60] M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas.  Sentiment in short strength detection informal text.  Journal of the American Society for Information Science and Technology, 61(12):2544–2558, 2010.
  * [61] P. Thongtanunam, S. Mcintosh, A. E. Hassan, and H. Iida.  Review participation in modern code review: An empirical study of the android, qt, and openstack projects.  Journal of Empirical Software Engineering, 22(2):768–817, 2017\.
  * [62] G. Uddin, O. Baysal, L. Guerroj, and F. Khomh.  Understanding how and why developers seek and analyze api related opinions.  IEEE Transactions on Software Engineering, page 40, 2019.
  * [63] G. Uddin and F. Khomh.  Automatic summarization of API reviews.  In Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering, pages 159–170, 2017.
  * [64] G. Uddin and F. Khomh.  Mining api aspects in api reviews.  Technical report, https://swat.polymtl.ca/data/opinionvalue-technical-report.pdf, 2017.
  * [65] G. Uddin and F. Khomh.  Opiner: A search and summarization engine for API reviews.  In Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering, pages 978–983, 2017.
  * [66] G. Uddin and F. Khomh.  Automatic opinion mining from API reviews from stack overflow.  IEEE Transactions on Software Engineering, page 35, 2019.
  * [67] G. Uddin, F. Khomh, and C. K. Roy.  Automatic api usage scenario documentation from technical q&a sites.  ACM Transactions on Software Engineering and Methodology, page 43, 2020.
  * [68] G. Uddin, F. Khomh, and C. K. Roy.  Automatic mining of api usage scenarios from stack overflow.  Information and Software Technology (IST), page 16, 2020.
  * [69] G. Uddin and M. P. Robillard.  Resolving api mentions in forum texts.  Technical report, McGill University, 2015.
  * [70] M. Valiev, B. Vasilescu, and J. Herbsleb.  Ecosystem-level determinants of sustained activity in open-source projects: a case study of the pypi ecosystem.  In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 644–655, 2018.
  * [71] L. Villarroel, G. Bavota, B. Russo, R. Oliveto, and M. D. Penta.  Release planning of mobile apps based on user reviews.  In Proceedings of the 38th International Conference on Software Engineering, pages 14–24, 2016.
  * [72] S. Wang and C. D. Manning.  Baselines and bigrams: simple, good sentiment and topic classification.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 90–94, 2012.
  * [73] A. B. Warriner, V. Kuperman, and M. Brysbaert.  Norms of valence, arousal, and dominance for 13,915 english lemmas.  Behavior Research Methods, 45(4):1191–1207, 2013.
  * [74] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén.  Experimentation in software engineering: an introduction.  Kluwer Academic Publishers, Norwell, MA, USA, 2000.
  * [75] X. Yang, D. Lo, X. Xia, and J. Sun.  TLEL: A two-layer ensemble learning approach for just-in-time defect prediction.  Information and Software Technology, 87(2):206 – 220, 2017.
  * [76] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le.  XLNet: Generalized autoregressive pretraining for language understanding.  Technical report, https://arxiv.org/abs/1906.08237, 2020.
  * [77] T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, and L. Jiang.  Sentiment analysis for software engineering: How far can pre-trained transformer models go?  In IEEE International Conference on Software Maintenance and Evolution, pages 70–80, 2020.
  * [78] Y. Zhang, D. Lo, X. Xia, and J. Sun.  Combined classifier for cross-project defect prediction: an extended empirical study.  Frontiers of Computer Science, 22(2):280 – 296, 2018.

##  Appendix A Details of Performance Analysis of Tools

Table 22. Performance comparison of majority voting-based hybrid classifiers with the baselines. Extension of Table 5 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
Majority-All | 0.721 | 0.825 | 0.640 | 0.645 | 0.859 | 0.516 | 0.819 | 0.734 | 0.925
Majority-Supervised | 0.706 | 0.818 | 0.621 | 0.645 | 0.879 | 0.510 | 0.825 | 0.735 | 0.939
Majority-Unsupervised | 0.536 | 0.736 | 0.422 | 0.490 | 0.762 | 0.361 | 0.755 | 0.647 | 0.907
Baseline - Senti4SD | 0.731 | 0.773 | 0.692 | 0.690 | 0.748 | 0.640 | 0.822 | 0.783 | 0.865
Baseline - SentiCR | 0.728 | 0.803 | 0.667 | 0.687 | 0.804 | 0.599 | 0.830 | 0.767 | 0.903
Baseline - SentistrengthSE | 0.720 | 0.752 | 0.690 | 0.640 | 0.736 | 0.566 | 0.795 | 0.750 | 0.847
Baseline - Opiner | 0.494 | 0.467 | 0.524 | 0.512 | 0.577 | 0.460 | 0.676 | 0.668 | 0.684
Baseline - POME | 0.127 | 0.389 | 0.076 | 0.065 | 0.415 | 0.035 | 0.697 | 0.550 | 0.950
Table 23. Performance comparison of the developed hybrid classifier (Sentisead) with the baselines. Extension of Table 8 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
SentiseadRF-B | 0.764 | 0.832 | 0.706 | 0.738 | 0.827 | 0.666 | 0.843 | 0.790 | 0.905
SentiseadRF-N | 0.759 | 0.797 | 0.724 | 0.72 | 0.777 | 0.671 | 0.837 | 0.801 | 0.877
Performance Increase of SentiseadB (i.e., Hybrid with Bag of Words)
Over Baseline (SentiSE = SentistrengthSE)
Majority | 6.0% | 0.8% | 10.3% | 14.4% | -3.7% | 29.1% | 2.9% | 7.6% | -2.2%
Senti4SD | 4.5% | 7.6% | 2.0% | 7.0% | 10.6% | 4.1% | 2.6% | 0.9% | 4.6%
SentiCR | 4.9% | 3.6% | 5.8% | 7.4% | 2.9% | 11.2% | 1.6% | 3.0% | 0.2%
SentiSE | 6.1% | 10.6% | 2.3% | 15.3% | 12.4% | 17.7% | 6.0% | 5.3% | 6.8%
Opiner | 54.7% | 78.2% | 34.7% | 44.1% | 43.3% | 44.8% | 24.7% | 18.3% | 32.3%
POME | 501.6% | 113.9% | 828.9% | 1035.4% | 99.3% | 1802.9% | 20.9% | 43.6% | -4.7%
Table 24. Performance comparison of the designed Hybrid detector (Sentisead) with added features from Table 10. Extension of Table 11 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
SentiseadRF-B+ | 0.767 | 0.832 | 0.712 | 0.74 | 0.818 | 0.675 | 0.844 | 0.794 | 0.901
SentiseadRF-BNE+ | 0.761 | 0.831 | 0.702 | 0.738 | 0.832 | 0.663 | 0.843 | 0.788 | 0.906
SentiseadRF-BNP+ | 0.765 | 0.829 | 0.711 | 0.737 | 0.813 | 0.674 | 0.843 | 0.793 | 0.899
SentiSeadRF-N+ | 0.747 | 0.777 | 0.719 | 0.698 | 0.741 | 0.66 | 0.822 | 0.793 | 0.854
SentiseadRF-NNE+ | 0.755 | 0.791 | 0.722 | 0.708 | 0.772 | 0.653 | 0.834 | 0.796 | 0.876
SentiseadRF-NNP+ | 0.746 | 0.773 | 0.721 | 0.698 | 0.741 | 0.66 | 0.821 | 0.793 | 0.852
Sentisead${}_{RF-B}*$: With bag of words. Sentisead${}_{RF-N}*$: No bag of
words.
NP = No partial polarity. NE = No polarity entropy
Table 25. Performance of stand-alone advanced pre-trained advanced language-based deep learning models across the datasets. Extension of Table 13 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
BERT | 0.772 | 0.768 | 0.775 | 0.759 | 0.767 | 0.752 | 0.847 | 0.845 | 0.848
Change Over SentiseadRF | 1.0% | -7.7% | 9.8% | 2.8% | -7.3% | 12.9% | 0.5% | 7.0% | -6.3%
RoBERTa | 0.788 | 0.773 | 0.804 | 0.768 | 0.768 | 0.768 | 0.847 | 0.855 | 0.839
Change Over SentiseadRF | 3.1% | -7.1% | 13.9% | 4.1% | -7.1% | 15.3% | 0.5% | 8.2% | -7.3%
ALBERT | 0.767 | 0.792 | 0.743 | 0.747 | 0.764 | 0.731 | 0.846 | 0.828 | 0.865
Change Over SentiseadRF | 0.4% | -4.8% | 5.2% | 1.2% | -7.6% | 9.8% | 0.4% | 4.8% | -4.4%
XLNet | 0.779 | 0.749 | 0.812 | 0.767 | 0.786 | 0.749 | 0.848 | 0.857 | 0.84
Change Over SentiseadRF | 2.0% | -10.0% | 15.0% | 3.9% | -5.0% | 12.5% | 0.6% | 8.5% | -7.2%
Table 26. Performance of Sentisead using pre-trained langange based advanced deep learning models as ensembler. Extension of Table 15 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
SentiseadBERT | 0.77 | 0.771 | 0.768 | 0.76 | 0.761 | 0.759 | 0.843 | 0.842 | 0.844
Change Over SentiseadRF | 0.8% | -7.3% | 8.8% | 3.0% | -8.0% | 14.0% | 0.0% | 6.6% | -6.7%
Change Over BERT | -0.3% | 0.4% | -0.9% | 0.1% | -0.8% | 0.9% | -0.5% | -0.4% | -0.5%
SentiseadRoBERTa | 0.787 | 0.78 | 0.794 | 0.774 | 0.792 | 0.757 | 0.853 | 0.849 | 0.857
Change Over SentiseadRF | 3.0% | -6.2% | 12.5% | 4.9% | -4.2% | 13.7% | 1.2% | 7.5% | -5.3%
Change Over RoBERTa | -0.1% | 0.9% | -1.2% | 0.8% | 3.1% | -1.4% | 0.7% | -0.7% | 2.1%
SentiseadALBERT | 0.764 | 0.769 | 0.76 | 0.735 | 0.731 | 0.738 | 0.839 | 0.838 | 0.84
Change Over SentiseadRF | 0.0% | -7.6% | 7.6% | -0.4% | -11.6% | 10.8% | -0.5% | 6.1% | -7.2%
Change Over ALBERT | -0.4% | -2.9% | 2.3% | -1.6% | -4.3% | 1.0% | -0.8% | 1.2% | -2.9%
SentiseadXLNet | 0.785 | 0.774 | 0.796 | 0.767 | 0.772 | 0.763 | 0.848 | 0.852 | 0.844
Change Over SentiseadRF | 2.7% | -7.0% | 12.7% | 3.9% | -6.7% | 14.6% | 0.6% | 7.8% | -6.7%
Change Over XLNet | 0.8% | 3.3% | -2.0% | 0.0% | -1.8% | 1.9% | 0.0% | -0.6% | 0.5%
Table 27. Performance of Sentisead using pre-trained advanced deep learning models as ensembler and stand-alone detectors. Extension of Table 17 | Positive | Negative | Neutral
---|---|---|---
| F1 | P | R | F1 | P | R | F1 | P | R
SentiseadBERT+ | 0.769 | 0.766 | 0.771 | 0.757 | 0.76 | 0.755 | 0.844 | 0.844 | 0.844
Change Over SentiseadRF | 0.7% | -7.9% | 9.2% | 2.6% | -8.1% | 13.4% | 0.1% | 6.8% | -6.7%
Change Over SentiseadBERT | -0.1% | -0.6% | 0.4% | -0.4% | -0.1% | -0.5% | 0.1% | 0.2% | 0.0%
Change Over BERT | -0.4% | -0.3% | -0.5% | -0.3% | -0.9% | 0.4% | -0.4% | -0.1% | -0.5%
SentiseadRoBERTa+ | 0.783 | 0.758 | 0.809 | 0.768 | 0.776 | 0.761 | 0.847 | 0.857 | 0.837
Change Over SentiseadRF | 2.5% | -8.9% | 14.6% | 4.1% | -6.2% | 14.3% | 0.5% | 8.5% | -7.5%
Change Over SentiseadRoBERTa | -0.5% | -2.8% | 1.9% | -0.8% | -2.0% | 0.5% | -0.7% | 0.9% | -2.3%
Change Over RoBERTa | -0.6% | -1.9% | 0.6% | 0.0% | 1.0% | -0.9% | 0.0% | 0.2% | -0.2%
SentiseadALBERT+ | 0.766 | 0.763 | 0.768 | 0.744 | 0.752 | 0.736 | 0.84 | 0.838 | 0.843
Change Over SentiseadRF | 0.3% | -8.3% | 8.8% | 0.8% | -9.1% | 10.5% | -0.4% | 6.1% | -6.9%
Change Over SentiseadALBERT | 0.3% | -0.8% | 1.1% | 1.2% | 2.9% | -0.3% | 0.1% | 0.0% | 0.4%
Change Over Albert | -0.1% | -3.7% | 3.4% | -0.4% | -1.6% | 0.7% | -0.7% | 1.2% | -2.5%
SentiseadXLNet+ | 0.777 | 0.752 | 0.803 | 0.768 | 0.781 | 0.756 | 0.845 | 0.852 | 0.837
Change Over SentiseadRF | 1.7% | -9.6% | 13.7% | 4.1% | -5.6% | 13.5% | 0.2% | 7.8% | -7.5%
Change Over SentiseadXLNet | -1.0% | -2.8% | 0.9% | 0.1% | 1.2% | -0.9% | -0.4% | 0.0% | -0.8%
Change Over XLNet | -0.3% | 0.4% | -1.1% | 0.1% | -0.6% | 0.9% | -0.4% | -0.6% | -0.4%