# On the Energy Consumption of Different Dataframe Processing Libraries - An
Exploratory Study

Shriram Shanbhag, Sridhar Chimalakonda  Research in Intelligent Software &
Human Analytics (RISHA) Lab
Department of Computer Science & Engineering
Indian Institute of Technology Tirupati India
cs20s503<EMAIL_ADDRESS>
###### Abstract.

Background: The energy consumption of machine learning and its impact on the
environment has made energy efficient ML an emerging area of research.
However, most of the attention stays focused on the model creation and the
training and inferencing phase. Data oriented stages like preprocessing,
cleaning and exploratory analysis form a critical part of the machine learning
workflow. However, the energy efficiency of these stages have gained little
attention from the researchers. Aim: Our study aims to explore the energy
consumption of different dataframe processing libraries as a first step
towards studying the energy efficiency of the data oriented stages of the
machine learning pipeline. Method: We measure the energy consumption of 3
popular libraries used to work with dataframes, namely Pandas, Vaex and Dask
for 21 different operations grouped under 4 categories on 2 datasets. Results:
The results of our analysis show that for a given dataframe processing
operation, the choice of library can indeed influence the energy consumption
with some libraries consuming 202 times lesser energy over others. Conclusion:
The results of our study indicates that there is a potential for optimizing
the energy consumption of the data oriented stages of the machine learning
pipeline and further research is needed in the direction.

dataframe, data preprocessing, data cleaning, energy efficiency

††copyright: none††copyright: none

##  1\. Introduction

Machine learning is a branch of artificial intelligence that focuses on using
data and algorithms to make the systems learn. The problems addressed by
machine learning applications include computer vision (Krizhevsky et al.,
2012), speech recognition (Amodei et al., 2016), natural language processing
(Cai et al., 2017), machine translation (Wu et al., 2016) and software
engineering (Lee et al., 2017a) among others. Data plays a critical role in
machine learning as it is required to train the models to make predictions. In
recent years, there is a drastic improvement in performance of machine
learning models driven by new architectures like neural networks, availability
of GPU based powerful hardware infrastructure and huge amounts of data to
train the machine learning models.However, there is also a critical and
growing concern of energy consumption of these models. A study from 2019 by
Strubell et al. (Strubell et al., 2019) revealed that a single neural network
model with 626,155 parameters has a carbon footprint equivalent to lifetime
carbon footprint of 5 cars.

The machine learning workflow (Amershi et al., 2019) typically contains “data
oriented” stages like data preprocessing, cleaning and exploratory data
analysis. The use of relational databases for these stages come with a lot of
limitations. Oftentimes, the data collected would not be well structured
making it difficult to work with database queries (Wu, 2020). Writing complex
queries would require a strong familiarity with the schema which is not easy
in case of wide tables with many columns (Gathani et al., 2020). Dataframes
have been introduced as an alternative to overcome some of these limitations.
Dataframe is a data structure that organizes data into rows and columns like a
spreadsheet. Over the years, the use of dataframes has become widely popular
as they provide an intuitive way of storing and working with data (Wu, 2020;
Petersohn, 2021). They are convenient for use in REPL style imperative
interfaces and data science notebooks (Perez and Granger, 2015). Several
libraries (McKinney et al., 2011; Rocklin, 2015; Breddels and Veljanoski,
2018) have been developed to enable the handling and manipulation of
dataframes. These libraries provide a convenient interface to work with
dataframes and are widely used for exploratory analysis, data ingest, data
preparation and feature engineering. Dataframes offer several features
including implicit ordering on both rows and columns by treating them
symmetrically. The dataframe manipulation libraries offer data analysis
modalities that include relational operators like filter, join, transpose and
spreadsheet-like operators such as pivot.

The dataframe manipulation libraries are used in the data oriented stages of
the machine learning workflow. Pandas, the most popular dataframe processing
library for instance, has been downloaded over 300 million
times111https://pypistats.org/packages/pandas with its Github repository
starred over 33k times as of April 2022222https://github.com/pandas-
dev/pandas. Machine learning however, has become very data intensive in recent
years. As the size of the data gets bigger, it cannot fit in the system RAM
leading to the need for more efficient, out-of-core dataframe processing
methods.This has prompted the need for development of faster and scalable
dataframe processing methods. This has led to the development of scalable ways
of dataframe processing using the libraries such as Dask, Vaex, RAPIDS etc.
These libraries enable scaling either through parallelization or through the
use of efficient computation techniques and memory usage. While the emphasis
in these libraries remains focused on achieving faster processing and
scalability, energy consumption aspect of these libraries is typically
ignored.

In the recent years, energy consumption of the software systems have gained a
major attention from the researchers. Several works have observed a link
between design of a software and its energy consumption (Hindle et al., 2014;
Cruz and Abreu, 2019a). The lack of knowledge about the energy implications of
poor design choices have also been observed in a study (Pinto and Castor,
2017). In the domain of machine learning, the computation and energy demands
have been increasing exponentially which would require a larger energy
production. This has the potential to cause a significant portion of the
carbon emissions (Edenhofer, 2015). Owing to this many researchers have
focused on energy efficiency of machine learning. There has been a focus on
measuring the energy consumption of machine learning models (Lee et al.,
2017b; Sanh et al., 2019), reducing the energy requirements of training (Jiao
et al., 2018; Sarwar et al., 2018) and inferencing (Sanh et al., 2019).
However, the work on energy efficient machine learning have largely ignored
the data oriented stages of the machine learning pipeline. To fill this gap,
as a first step, we perform an exploratory study on energy consumption of
dataframe libraries.

In this paper, we perform an empirical evaluation of the energy consumption of
three popular libraries namely Pandas333https://pandas.pydata.org/,
Vaex444https://vaex.io and
Dask555https://docs.dask.org/en/stable/dataframe.html for four categories of
tasks. Although Dask is not a library specifically mean for working with
dataframes, it does contain a dataframe module that can be perform operations
on dataframes efficiently using computation graphs (Rocklin, 2015). The four
types of tasks include input-output operations, handling missing data,
row/column operations and statistical aggregation.

The rest of the paper is organized as follows. We provide a brief description
about the libraries used in our study in Section 2. The tasks addressed in the
study are discussed in Section 3. The datasets used in the study are presented
in Section 4 The experimental settings and the procedure followed are
described in Section 5. The results of the experiments are presented in
Section 6. Section 7 discusses the potential threats that may impact the
validity of our study. The discussion, related work and conclusion are
presented in Section 8, Section 9 and Section 10 respectively.

Figure 1. Flowchart describing the experimental procedure followed in the
study based on Georgiou et al. (2022)

##  2\. Dataframe Processing Libraries

The data used in machine learning is normally structured in tabular form
called dataframes. Dataframe is a 2 dimensional labeled data structure with
columns of potentially different types. Dataframes provide a common structure
to work with data from domains such as finance, statistics, social sciences
among many others. Libraries like Pandas and Vaex provide a rich set of
functionalities to manipulate the dataframes that enable easier data cleaning
and preparation.

Pandas (McKinney et al., 2011) is an open source Python library for data
manipulation developed in the year 2008. It provides data structures and
operations for manipulation of tabular data. It was originally developed by
Wes McKinney for working with financial data. In addition to data
manipulation, it can also be used for data analysis and visualization. Vaex
(Breddels and Veljanoski, 2018) on the other hand is a Python library for
working with lazy out of core dataframes on huge amounts of tabular data. The
project has over 7K stars and over 500 forks on GitHub as of March
2022666https://github.com/vaexio/vaex. Vaex claims to be an alternative to
Pandas that has a more efficient and faster implementation of data
manipulation functions using memory mapping, a zero memory copy policy, and
lazy computations.

Dask is an opens source library for parallel computing in Python. Dask
provides support for dataframe processing through a module that has an
interface similar to Pandas. The module implements a large dataframe out of
many Pandas dataframes (Rocklin, 2015). It uses a threaded scheduler that
enables efficient computation on partitioned datasets (Rocklin, 2015). Dask
minimizes the amount of data held in memory while parallelizing the tasks
using computation graphs (Rocklin, 2015).

##  3\. Addressed Tasks

In order to compare the energy consumption of these libraries while working
with the dataframes, we used different libraries to perform the same set of
operations on three different datasets. We picked some of the commonly
performed operations on dataframes as tasks for our evaluation. The operations
included loading of the csv file, dropping columns, conatenatination of
dataframes, duplicate removal, merging dataframes, subset, sampling, and
aggregation operations such as sum, median, min, max, unique. A brief
description of each of these tasks is given in Table 1.

Dataframe processing libraries can be used to perform several operations on
dataframes including read/write operations, visualization, relational and
statistical operations. In order to compare the energy consumption of these
libraries while working with the dataframes, we used different libraries to
perform the same set of operations on three different datasets. we address a
few frequently performed tasks input/output operations, relational operation
and statistical operations.

###  3.1. Input/Output Operations

Dataframes can be read and stored in several formats including CSV, HDF5,
Parquet, JSON etc. The choice of the format is typically decided based on the
type of data being stored and the type of operations being performed on the
data. As an example, storing dataframes in a format like Parquet can be
benifitial when used in case of big data systems like Hadoop or Spark.
However, unlike CSV, the storage format is also not human readable. Since our
work focuses on energy, we compare the energy consumption of input and output
operations on different file formats used to represent the dataframes.
Specifically we compare the energy consumption values of reading and writing
the dataframes in three popular formats namely CSV, JSON and HDF5 using
Pandas, Vaex and Dask libraries.

CSV stands for comma-separated values. CSVs store data as plain text where
each row is a line of columns separated by ”,”. It is very widely used due to
the fact that its human readable and can be read from and written to by most
data software. JSON stands for JavaScript Object Notation is another human
readable format that uses text to store data using attribute-value pairs and
arrays. Like CSV, JSON can also be readily displayed and edited in simple
editors. It also has an advantage of being a preferred format for transmitting
data in web applications. HDF5 stands for Hierarchical Data Format 5 suited
for storing and organizing large amounts of heterogeneous data. The data is
stored as an internal file-like structure which provides the ability to
randomly access different parts of the data. HDF5 offers efficient storage and
faster access when dealing with huge data.

###  3.2. Handling missing data

Missing data is a very common problem in real-world datasets. They are
generally referred to as NA values in dataframes. Missing data is generally
caused by lack of collection of existing data or because of the non-existence
of the data. Missing data in the dataframes is undesirable as they generally
interfere with the operations and calculations being performed on
rows/columns. This makes handling of missing data a very frequently performed
task in data preprocessing. The missing data handling functions compared in
our study are found in Table 1

###  3.3. Row/Column operations

Dataframes represent the data in the form of a table consisting of rows and
columns. Dataframes support relational operations like join, filter, groupby
etc. They also support reshape and transpose operations on the tables. This
makes it easier to merge data obtained from multiple sources in multiple
dataframes and order them in a desired way to enable a better exploratory
analysis. They also enable the users to apply functions on rows/columns that
allow modification and filtering of the data. The missing data handling
functions compared in our study are found in Table 1

###  3.4. Statistical Aggregation Operations

Statistical aggregation functions enable the users to perform statistical
analysis of their data. They can be used for operations like aggregating
multiple values under a row or column using a single summary value, computing
the correlation and covariance between pairs of rows/columns, finding the
unique occurrences, count of rows/columns etc. These functions are frequently
used in exploratory data analysis to get the summary or the description of the
dataset. The statistical aggregation functions compared in our study are found
in Table 1

Task ID | Task Name | Description | Category
---|---|---|---
csvin | Read CSV | Read a comma-separated values (csv) file into dataframe | I/O operation
jsonin | Read JSON | Read JSON string into a Dataframe | I/O operation
hdfin | Read HDF5 | Read an heirarchical data format 5 (hdf5) file into dataframe | I/O operation
csvout | Write to CSV | Write dataframe into a comma-separated values (csv) file | I/O operation
jsonout | Write to JSON | Write dataframe into a JSON file | I/O operation
hdfout | Write to HDF5 | Write dataframe into a heirarchical data format 5 (hdf5) file | Handling missing data
isna | Detect missing values | Detect missing values in a given series object | Handling missing data
dropna | Drop missing values | Drop rows/columns containing missing values | Handling missing data
fillna | Fill missing values | Fill missing values with a default | Handling missing data
replace | Replace values | Replace a value with a specified value | Handling missing data
drop | Drop rows/columns | Drop specified labels from rows or columns | Row/Column operations
groupby | Groupby Operation | Group rows that have the same values together | Row/Column operations
concat | Concatenation | Concatenate dataframes along a particular axis | Row/Column operations
sort | Sort values | Sort the values along an axis | Row/Column operations
merge | Merge dataframes | Merge dataframes with a database-join style | Row/Column operations
count | Count cells | Count the number of non-NA cells along an axis | Statistical Aggregation
sum | Sum values | Sum values along an axis | Statistical Aggregation
mean | Average values | Average values along an axis | Statistical Aggregation
min | Minimum value | Find the minimum of the values along an axis | Statistical Aggregation
max | Maximum value | Find the maximum of the values along an axis | Statistical Aggregation
unique | Unique values | Find the set of unique values along the axis | Statistical Aggregation
Table 1. Tasks addressed in the study

##  4\. Datasets Used

In order to compare the energy consumption of the dataframe processing
libraries, we have to perform the same set of operations on the same datasets.
Towards this end, we relied on well known standard datasets from UCI Machine
learning repository777https://archive.ics.uci.edu/ml/index.php. The energy
consumption of the libraries may also vary based on the size of the dataframe
they process on. A library that may be more energy efficient on a smaller
dataframe may become less energy efficient as the dataframes get bigger. In
order to account for these variations, we performed our experiments on 2
different datasets of varying sizes from the UCI repository shown in Table 2.
The datasets used are briefly described below.

###  4.1. Adult Dataset

The dataset was extracted based on the 1994 census database. The task
associated with the dataset is the classification task to determine whether a
person makes over 50K a year or not. The dataset contains 48842 data points
with 14 attributes. We would refer to this dataset as D1 in the rest of the
paper.

###  4.2. Drug Review Dataset

This dataset contains patient reviews on specific drugs along with related
health conditions. It also contains the 10 star patient rating of the drug.
The dataset has 215063 data points with 6 attributes. We would refer to this
dataset as D2 in the rest of the paper.

Dataset Name | Characteristics | Associated Tasks | No. of Data Points | No. of Attributes
---|---|---|---|---
Adult | Multivariate | Classification | 48842 | 14
Durg Review |  | Multivariate,
---
Text
| Classification,
---
Regression, Clustering
215063 | 6
Table 2. Datasets used for the study

##  5\. Experimental Procedure

All the experiments regarding energy consumption are run on a system with an
intel i5 4200 four core processor equipped with a frequency of 1.6 GHz. The
system was equipped with a RAM of 8 GB. We used Python 3.8 as the programming
language for all our experiments. The system had Ubuntu 20.04 installed as the
operating system. We used Pandas version 1.2.4, Vaex version 4.8.0 and Dask
version 2022.2.1 for our study.

To measure the energy consumption values, we used Intel’s Running Average
Power Limit (RAPL) interface. Using the RAPL tool is one of the most accurate
ways to measure the global energy consumption of the processor (David et al.,
2010). Several studies related to measuring the energy consumption of the
software systems have relied on the RAPL interface (Pereira et al., 2017;
Ournani et al., 2021; Kumar and Shi, 2019). The tool provides energy
consumption values of various power domains such as package, core, uncore,
power plane, DRAM etc. The energy consumption values of the following domains
were noted down during the experiments

  * •

Package (PKG): This domain provides the energy consumption values of all the
cores, integrated graphics and uncore components such as last level caches and
memory controllers.

  * •

Core: The energy consumption values of all the CPU cores are provided by this
domain.

  * •

Uncore: This domain provides the energy consumption values of all the caches,
integrated graphics and the memory controllers.

  * •

DRAM: The energy consumption values corresponding to the random access memory
attached to the memory controller is provided by this domin.

Since the PKG values are inclusive of core and non-core values, we only report
the PKG values in the results in Section 6.

The experimental procedure using a single library is represented as a
flowchart in Figure 1. At the start of the experiment, we shut down all the
processes that were not required to run the OS. This was done to ensure
minimal variation in the readings due to interference from the processes
running in the background.

Power consumption measurement in the system can be influenced by noise. So one
of the methods used to minimize the effect of the noise on the measurements is
to run the same experiment multiple times and take the mean value of the
measurements for all the experiments as followed in a study by Georgiou et al.
(Georgiou et al., 2022). Based on this, we ran the experiment for each
operation 10 times and recorded the mean value of the energy measurements as
shown in Figure 1. After running each task on a dataset, we let the system
remain idle for 30 seconds using the command “sleep”. This was done in order
to avoid the power tail states (Bornholt et al., 2012) influencing the reading
and allow the system to reach a stable state again before the start of another
task. At the end of each task, the RAPL readings were stored in an excel sheet
for further analysis. The information included the time required to run each
task along with the energy consumption values of the components.

The experiment was performed using all the three libraries considered for the
study. The mean values of all the trials for each task was computed and noted
down using automated python script. The scripts used for the study along with
the detailed energy consumption values of the tasks for each trial is
available here888https://anonymous.4open.science/r/esem2022-5D62.

##  6\. Results

This section discusses the results of the experiments measuring energy
consumption of 3 different libraries on various dataframe operations. The
energy consumption measurements for input/output operations are shown in Table
3 and Table 4. The energy consumption of operations on handling missing data
is shown in Table 5. The energy consumption values of table and statistical
operations are shown in Table 6 and Table 7 respectively. We use the
abbreviation PKG for the energy consumption values of all core and non-core
components of the processor. The RAPL interface also provides the energy
consumption of the core and non-core components separately. Since those are
covered under PKG, we do not report them explicitly. We use RAM for energy
consumption of the main memory. The RAPL interface provides the energy
consumption values in micro-joules (1 micro-joule = $10^{-6}$ Joules).
However, we present them in the results by converting them into milli-joules
(1 mini-joule = $10^{-3}$ Joules) and rounding them off to one decimal. This
is done to ensure better readability. The energy consumption values in all the
tables are mean values from 10 trials of each operation performed as described
in Section 5.

###  6.1. Input/Output Operations

####  6.1.1. Input Operations

The energy consumption of the dataframe input operation in CSV, JSON and HDF5
formats for the datasets D1 and D2 are shown in Table 3. For the dataset D1,
Vaex consumes the highest amount of energy for reading the dataframes in all
three file formats. Dask consumes the least amount of energy for inputs in CSV
and HDF5 formats while Pandas consumed the least energy for reading the
dataframe in JSON format. For the dataset D2, we observe that Vaex consumes
the most energy for inputs in CSV and JSON formats while Pandas consumes the
most energy for input in HDF5. However, unlike D1, Dask consumes lesser energy
than Pandas for input operations in JSON format as well along with CSV and
HDF5.However, we could not get values for HDF5 for input and output operations
for Dask library in the Drugs Review Dataset due to technical glitches because
of the size of the data.

Across all libraries and datasets, we can also observe that the reading the
dataframe stored as JSON is most expensive in terms of energy. For Pandas,
while reading D1, the energy consumption remains comparable for CSV and HDF5
formats. However, as the size of the dataset increases, HDF5 becomes less
expensive. This can be observed from the values for D2 with Pandas library in
Table 3. We can also observe that the energy consumption of CSV inputs for
Dask is not affected much by increase in the dataset size. The reason could be
because of the lazy execution (Rocklin, 2015) using computation graphs where
it only loads information about the header and the datatypes (Rocklin, 2015).

####  6.1.2. Output Operations

The energy consumption of the dataframe output operation in CSV, JSON and HDF5
formats for the datasets D1 and D2 are shown in Table 4. For dataset D1, Dask
consumes the most energy while Pandas consumes the least energy for outputs in
all three formats. A similar observation can be made for the dataset D2. The
output of dataframe into a CSV file consumes the most energy across Pandas and
Vaex. However, for output using Dask library, HDF5 is the most energy-
expensive file format.

The following are some of the key highlights from the observation of the
results of input/output operations.

(1) Overall, Vaex is the most energy-expensive library for input operations.
(2) JSON is the most expensive of the three file formats for dataframe input.
(3) The energy consumption of a CSV input remains unaffected by the size of
the dataset for Dask. (4) Dask consumes the most energy for output operations
while Pandas consumes the least. (5) CSV is the most energy-expensive file
format for outputting a dataframe in Pandas and Vaex. (6) HDF5 is the most
energy-expensive file format for output using Dask library.

Adult Dataset | Drugs Review Dataset
---|---
| Pandas |  | Vaex |  | Dask |  | Pandas | Vaex | Dask
| File
---
format
PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM
CSV | 551.2 | 62.5 | 1030.4 | 95.1 | 62.6 | 6 | 9696.6 | 963.7 | 13042.2 | 1327.6 | 65.1 | 6.1
JSON | 3178.7 | 432.4 | 3697.8 | 480.1 | 3471.9 | 465.2 | 156434.5 | 11384.1 | 160102.2 | 11791.9 | 154388.9 | 11382.7
HDF5 | 565 | 55.7 | 3379.2 | 283.1 | 502.7 | 48.8 | 5568.2 | 804.7 | 4272.1 | 418.2 | NA | NA
Table 3. Mean values for energy consumption for dataframe input in different formats.  Adult Dataset | Drugs Review Dataset
---|---
| Pandas |  | Vaex |  | Dask |  | Pandas | Vaex | Dask
| File
---
format
PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM
CSV | 1673.1 | 113.6 | 2261.6 | 260 | 4473.8 | 460.2 | 28941.1 | 2229 | 29620.6 | 2688.4 | 193428.6 | 14391.7
JSON | 774.1 | 76.5 | 1515.7 | 206.9 | 3683.7 | 416.8 | 9797.8 | 1105.3 | 12402.3 | 1600.6 | 170093.8 | 13343
HDF5 | 288.7 | 34.9 | 1713.6 | 138.8 | 5336.7 | 684.1 | 3993.2 | 523.8 | 4732.7 | 611.1 | NA | NA
Table 4. Mean values for energy consumption for dataframe output in different
formats.

###  6.2. Handling Missing Data

Figure 2. Cumulative energy consumption of missing data operations of the
libraries for both the datasets

The energy consumption values of the missing data handling functions are shown
in Table 5. For the dataset D1, Pandas consumes the highest energy for isna,
dropna, and fillna operations. While Dask consumes the highest energy for
replace operation, it is very close the energy consumption of Pandas with a
difference of just 1.4 milli-joules. Vaex consumes the least energy for isna
and replace while Dask consumes the least energy for dropna and fillna
operations.

For the dataset D2, Pandas consumes the most energy for all operations. Vaex
consumes the lowest energy for alloperations except fillna. Dask is the least
expensive library for fillna operation. It must be noted that for isna and
replace functions for both datasets, the energy values of both Dask and Vaex
are comparable with a maximum difference between them being just 5.6 milli-
joues. The cumulative energy consumption of missing data operations of the
libraries for both the datasets is shown in Figure 2.

The following are some of the key highlights from the results on missing data
handling operations.

(1) Overall, Pandas is the most energy-expensive library for missing data
handling operations. (2) Dask is the most energy-efficient library for fillna
operation on both the datasets. (3) Vaex is the most energy-efficient library
for isna and replace operations on both datasets.

Adult Dataset | Drugs Review Dataset
---|---
| Pandas | Vaex | Dask | Pandas | Vaex | Dask
Operation | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM
isna | 26.2 | 2.3 | 9.5 | 0.9 | 10.6 | 1 | 184.6 | 21.6 | 10 | 0.7 | 10.6 | 0.9
dropna | 392.8 | 26.3 | 137.7 | 13.8 | 39 | 3 | 1329.9 | 111.1 | 77 | 7.9 | 324.5 | 45.3
fillna | 411.5 | 29.3 | 155.7 | 10.4 | 41 | 3 | 1263 | 99.7 | 63.4 | 4.4 | 34.6 | 3.1
replace | 11.2 | 1 | 5.4 | 0.4 | 11.8 | 0.9 | 124.7 | 20 | 4.6 | 0.4 | 9.9 | 0.7
Table 5. Mean values for energy consumption for missing data handling
operations.

###  6.3. Row/Column Operations

The energy consumption values for the row/column operations are shown in Table
6. For dataset D1, Vaex is the least energy-expensive library on all
operations except groupby. Both Pandas and Dask have a significantly lower
energy consumption for groupby operation compared to Vaex. For the dataset D2,
Vaex is the most energy efficient library for drop and merge operations. Dask
consumes the least energy for sort operation. Similar to D1, Vaex is the most
energy-expensive library for groupby operation with both other libraries
having a significantly lower energy consumption.

We can also observe that despite the increase in the number of rows in D2, the
energy consumption of Dask does not change very significantly except in the
case of merge. The size of the dataset also does not seem to have much
influence on gropuby operation on any of the libraries. The energy consumption
values for the remaining operations increases with the increase in the dataset
size for the Pandas library.

The following are the key highlights from the results on row/column operations

(1) Vaex is the most energy-efficient library for drop, concat, and merge
operations. (2) Vaex is the most energy-expensive framework for groupby
operation. (3) The energy consumption of Dask does not change very
significantly despite change in the size of the dataset. (4) For the dataset
D2, Pandas is the most energy-expensive library for all operations other than
groupby.

Adult Dataset | Drugs Review Dataset
---|---
| Pandas | Vaex | Dask | Pandas | Vaex | Dask
Operation | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM
drop | 27.8 | 3.6 | 12 | 0.8 | 44.3 | 3.3 | 93.5 | 20 | 6.3 | 0.6 | 33.6 | 2.5
groupby | 3.3 | 0.5 | 245.7 | 26.9 | 5.1 | 0.4 | 5.5 | 0.5 | 272.2 | 35.3 | 3.5 | 0.3
concat | 102.2 | 15.5 | 27.8 | 2.8 | 139.9 | 9.8 | 462.1 | 92.1 | 16.7 | 1.6 | 105.3 | 7.4
sort | 57.6 | 7.2 | 30.1 | 2.4 | 38.5 | 2.9 | 303.8 | 48.3 | 92.1 | 8.7 | 30.6 | 2.2
merge | 498.5 | 43.9 | 46.5 | 3.4 | 257 | 17.6 | 4249.3 | 534.6 | 24.2 | 1.9 | 154.4 | 10.4
Table 6. Mean values for energy consumption for missing row/column operations.

###  6.4. Statistical Aggregation Operations

Figure 3. Cumulative energy consumption of statistical aggregation operations
of the libraries for both the datasets

The energy consumption values for the statistical aggregation operations
performed on the datasets is shown in Table 7. For the dataset D1, we observe
that Dask consumes the highest energy for all operations. Of all the
operations considered, count is the most energy-expensive operation across all
libraries. The sum, mean, min, max and unique are the cheapest in Pandas
followed by Vaex. For the count operation however, Vaex consumes the least
energy.

For the dataset D2, a similar observation can be made where Dask is the most
energy-expensive library for all the operations. The count operation is the
most expensive one for Pandas and Dask. For Vaex however, unique is the most
energy-expensive operation. Like in case of D1, Pandas is the least expensive
library for all the operations other than count. Vaex is the least expensive
operation for count operation.

The cumulative energy consumption of statistical aggregation operations of the
libraries for both the datasets is shown in Figure 3.

The following are the key highlights from the results on statistical
aggregation operations.

(1) Dask is the most expensive library for statistical aggregation operations
on both datasets. (2) Pandas is the most energy-efficient library for all
operations considered other than count. (3) Vaex is the most energy-efficient
library for count operation.

Adult Dataset | Drugs Review Dataset
---|---
| Pandas | Vaex | Dask | Pandas | Vaex | Dask
Operation | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM | PKG | DRAM
count | 393.7 | 25.8 | 77.2 | 6.2 | 1075 | 100.5 | 1218.5 | 84.4 | 33.4 | 2.6 | 11816.3 | 1224.6
sum | 5.6 | 0.4 | 34.6 | 3.3 | 379.5 | 40.4 | 7.5 | 0.8 | 35.2 | 3.2 | 5222 | 551.4
mean | 2.4 | 0.1 | 54 | 5.3 | 380.5 | 40.4 | 28.7 | 3.9 | 46.2 | 4.4 | 5265.4 | 567.9
min | 3.3 | 0.3 | 40 | 3.6 | 354.9 | 38.9 | 6.8 | 0.6 | 50 | 5.5 | 5185.2 | 551.8
max | 3.4 | 0.3 | 40.2 | 3.4 | 344.8 | 38.2 | 4.7 | 0.6 | 34.5 | 3.3 | 5175.5 | 532.1
unique | 4.2 | 0.2 | 56.2 | 5.9 | 312 | 35.2 | 99.6 | 7.3 | 407.5 | 39.3 | 5312.9 | 562.6
Table 7. Mean values for energy consumption for missing statistical
aggregation operations.

##  7\. Threats to Validity

In this section, we discuss potential systematic errors in our work that may
pose threats to the outcome of our study.

One of the most significant threats to the validity of our study are the
possible impact of noise, voltage spikes, daemons and other background
processes that may cause inaccurate energy measurement We tried to minimize
this issue by repeating the same task 10 times and averaging the recorded
values. We also made sure to have the system idle for 30 seconds before the
start of the next task to avoid the impact of tail states. Despite the best
care taken to ensure minimal impact of these factors, eliminating their effect
completely is very difficult as it would require us to have control over the
background operations that are needed to run the operating system.Due to
hardware constraints, we used a system with basic configuration to run our
experiments. The possibility of obtaining significantly different values on a
system with a different configuration must also be considered.

Another factor that may influence the results of our study is the choice of
datasets. Libraries such as Vaex and Dask are optimized for computations and
memory usage. The optimizations may influence the amount of work done for
performing a certain operation based on the nature of attributes, type and
size of the data in the dataset thereby influencing energy readings. To
minimize its influence, we performed our experiments with two datasets of
different of varying size and attributes.

The results that we obtained were from the time we set up the study. Open
source libraries are often developed by people across the globe. The
”competition” between different libraries would mean that the implementations
would get more efficient over time. Thus, the results of the experiment
obtained in the future for similar experiments may vary based on the
modifications to the libraries. Keeping this in mind, we have provided the
versions of the libraries used in Section 5.

##  8\. Discussion

This paper is our first step towards exploring energy efficiency of the data
oriented stages of the machine learning pipeline. Through this work, we also
aim to call the attention of the research community to work towards energy
efficiency of the data oriented stages of the machine learning pipeline. The
results of the study indicate that the choice of libraries for a given task
and the format of data storage could potentially influence the energy
consumption of the dataframe processing. As an example, based on the results
on statistical aggregation operations, we can see that using Dask can be a
poor choice in terms of energy when aggregation operations are performed very
frequently. Similarly, when we have too many and too large dataframes to read,
Dask can be the most energy-efficient choice.

Although we have explored the energy consumption of different libraries on a
set of tasks, it must be noted that there may exist an alternate ways or
functions to accomplish the same task in a library. As an example, many of the
missing data handling operations could also be performed using the
apply999https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html
function. The energy consumption of these alternate methods may be different
and they may be more energy-efficient or energy-hungry. However, the current
work does not focus on such cases.

We also observe in some cases that the energy consumption values of a task on
a larger dataset (D2) is lesser compared to the values for the smaller dataset
(D1). Few examples of this include the merge operation for Vaex and Dask, drop
operation for Dask, and concat operation for Vaex. Although it may look
counter-intutive, we argue a possible reason for this may be because of other
factors such as the number of attributes or the inner working of these
libraries. The influence of such factors could be explored further.

##  9\. Related Work

In recent years, the energy consumption of software elements have caught the
attention of researchers. This has led the researchers to explore and study
the energy consumption of software components across various domains with the
intention gaining insights that help the developers make energy efficient
choices. Measuring the energy consumption and comparing the variations for
different available options in the is a popular theme among such studies.

Rouvoy et al. (Rouvoy et al., 2021) performed an empirical investigation of
differences in energy consumption of read/write methods of some of the famous
Java libraries. They found some methods to be more efficient than others.
Hasan et al. (Hasan et al., 2016) studied the energy profiles of various Java
collection classes and found that choice of an inefficient collection can cost
as much as 300% more energy. Schuler et al. (Schuler and Anderst-Kotsis, 2020)
studied the relation between System API utilization and energy consumption in
third-party software libraries. Ournani et al. (Ournani et al., 2021) compared
the energy consumption of 27 Java I/O methods for different file sizes and
found that the energy consumption varies across APIs with some of them
consuming about 30% less energy than the others. Pereira et al. (Pereira et
al., 2017) analysed and compared the energy efficiency across 27 different
programming languages. They also explored how energy consumption relates to
speed and memory utilization in these languages (Pereira et al., 2017). Kumar
et al. (Kumar and Shi, 2019) analyzed the energy consumption of Java command
line options and found Oracle JDK to be more energy efficient than Open JDK.
They also found that UseG1GC and Xint were the most and the least energy
efficient command line options respectively (Kumar and Shi, 2019). Maleki et
al. (Maleki et al., 2017) studied the impact of OOP design patterns on energy
efficiency and found that the use of overloading and decorator patterns can
degrade the energy efficiency of the application.

Energy efficiency is of a critical importance in battery powered devices such
as mobile phones. To this end, exploratory and comparative studies directed
towards energy efficiency have also been performed in this area (Gupta et al.,
2011; Aggarwal et al., 2014; Chowdhury et al., 2016, 2018; Cruz and Abreu,
2019b; Sahin et al., 2016). Gupta et al. (Gupta et al., 2011) explored the
energy consumption of software modules and the impact of co-occurrences of
modules on the change in energy consumption values on a windows phone. They
also explored the anomalies in power traces. Aggarwal et al. (Aggarwal et al.,
2014) investigated the relationship between the system call invocation and
energy consumption across multiple versions of Android application. Chowdhury
et al. (Chowdhury et al., 2016) compared the energy consumption of HTTP/2 with
its predecessor HTTP/1.1 for mobile apps found that the former outperforms the
latter in most scenarios. Chowdhury et al. (Chowdhury et al., 2018) studied
the energy impact of logging on Android applications and found that limited
logging has little to no impact on energy consumption. Cruz et al. (Cruz and
Abreu, 2019b) analyzed the energy consumption of eight UI automation
frameworks and found that certain frameworks can increase the energy
consumption by over 2000 percent. Sahin et al. (Sahin et al., 2016) explored
the energy impact of code obfuscation by studying the impact of 18
obfuscations for 21 usage scenarios accross 11 Android applications on four
different mobile phone platforms. They found that obfuscations have a
significant impact on energy consumption and is more likely to increase energy
consumption than decrease it (Sahin et al., 2016).

Energy efficiency is crucial in server systems and cloud applications due to
its economic and ecological costs. Due to this, many of the-efficiency energy
studies have been directed towards this domain (Singh et al., 2015;
Procaccianti et al., 2016; Khomh and Abtahizadeh, 2018). Singh et al. (Singh
et al., 2015) explored the energy costs of running several Java APIs on
servers to accomplish a set of tasks and found that the developers can reduce
the energy costs of server by choosing energy-efficient APIs. Procaccianti et
al. (Procaccianti et al., 2016) performed an empirical investigation of two
energy efficient software practices namely ”sleep” and ”use of efficient
queries” and found that they can reduce energy consumption by upto 25%. Khomh
et al. (Khomh and Abtahizadeh, 2018) explored the energy consumption of 6
cloud patterns and found that they can be energy efficient in some cases.

In the domain of machine learning, several researchers have been working
towards improving the energy efficiency. However, the methods mainly remain
focused on the machine learning model itself. The methods include model
compression techniques like quantization (Lee et al., 2017b; Moons et al.,
2016), pruning (Yang et al., 2017), distillation (Sanh et al., 2019), use of
efficient hardware accelerators (Chen et al., 2016; Park et al., 2018; Ko et
al., 2017) and use of efficient computation methods in the hardware (Sarwar et
al., 2018; Jiao et al., 2018). However, the stages of the machine learning
pipeline that focus on the data preprocessing and cleaning have largely been
ignored. Our exploratory study serves a step in that direction.

##  10\. Conclusion and Future Work

In this paper, we presented an exploratory analysis of energy consumption of
dataframe processing libraries. This work is intended to be our first study
towards the energy efficiency in the data oriented stages of the machine
learning pipeline which involves data preprocessing, data cleaning and
exploratory data analysis.

We analyzed the energy consumption of three different dataframe processing
libraries, namely Pandas, Vaex and Dask for different dataframe processing
tasks. The tasks included i/o operations, handling missing data, row/column
operations and statistical aggregation operations as shown in Table 1. We used
the Intel’s RAPL interface to measure the energy consumption of the operations
using experimental setttings described in Section 5. We used two datasets from
the UCI repository, namely Adult Dataset and the Drug Review dataset to run
our experiments. The results of our experiments contained some interesting
observations summarized in Section 6. The results show us that the choice of
library and the format of the data storage can indeed influence the energy
consumption of the data preprocessing and data cleaning stages. This choice
may be based on factors such as the type of operations performed more
frequently, the format in which data is available and the size of the dataset.

An additional goal of this work is to also call the attention of the research
community to focus on the energy efficiency in the stages of the machine
learning pipeline other than model training and inferencing. A more
comprehensive analysis of energy consumption of the data oriented stages could
potentially lead to the discovery of energy efficient practices in data
preprocessing, data cleaning and exploratory analysis stages of the pipeline.
We have merely scratched the surface of this area and we plan several
extensions to the study. In the future, we plan to further explore the energy
consumption of dataframe processing libraries on multiple systems with
different configurations to get more generalized results. We also plan to
repeat the experiments on several larger datasets to explore trends with
respect to the energy consumption values for different tasks. The influence of
the data types in the dataframe on energy could also be a possible future
extension. The list of tasks considered in the study is limited. Given that
the libraries have many more functionalities, their energy consumption could
also be explored in the future.

## References

  * (1)
  * Aggarwal et al. (2014) Karan Aggarwal, Chenlei Zhang, Joshua Charles Campbell, Abram Hindle, and Eleni Stroulia. 2014\.  The power of system call traces: predicting the software energy consumption impact of changes.. In _CASCON_ , Vol. 14. 219–233.
  * Amershi et al. (2019) Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019.  Software engineering for machine learning: A case study. In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 291–300.
  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016\.  Deep speech 2: End-to-end speech recognition in english and mandarin. In _International conference on machine learning_. PMLR, 173–182.
  * Bornholt et al. (2012) James Bornholt, Todd Mytkowicz, and Kathryn S McKinley. 2012\.  The model is not enough: Understanding energy consumption in mobile devices. In _2012 IEEE Hot Chips 24 Symposium (HCS)_. IEEE, 1–3.
  * Breddels and Veljanoski (2018) Maarten A Breddels and Jovan Veljanoski. 2018.  Vaex: big data exploration in the era of gaia.  _Astronomy & Astrophysics_ 618 (2018), A13.
  * Cai et al. (2017) Ruichu Cai, Boyan Xu, Xiaoyan Yang, Zhenjie Zhang, Zijian Li, and Zhihao Liang. 2017\.  An encoder-decoder framework translating natural language to database queries.  _arXiv preprint arXiv:1711.06061_ (2017).
  * Chen et al. (2016) Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. 2016.  Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks.  _IEEE journal of solid-state circuits_ 52, 1 (2016), 127–138.
  * Chowdhury et al. (2018) Shaiful Chowdhury, Silvia Di Nardo, Abram Hindle, and Zhen Ming Jack Jiang. 2018.  An exploratory study on assessing the energy impact of logging on android applications.  _Empirical Software Engineering_ 23, 3 (2018), 1422–1456.
  * Chowdhury et al. (2016) Shaiful Alam Chowdhury, Varun Sapra, and Abram Hindle. 2016\.  Client-side energy efficiency of HTTP/2 for web and mobile app developers. In _2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER)_ , Vol. 1. IEEE, 529–540.
  * Cruz and Abreu (2019a) Luis Cruz and Rui Abreu. 2019a.  Catalog of energy patterns for mobile applications.  _Empirical Software Engineering_ 24, 4 (2019), 2209–2235.
  * Cruz and Abreu (2019b) Luis Cruz and Rui Abreu. 2019b.  On the energy footprint of mobile testing frameworks.  _IEEE Transactions on Software Engineering_ 47, 10 (2019), 2260–2271.
  * David et al. (2010) Howard David, Eugene Gorbatov, Ulf R Hanebutte, Rahul Khanna, and Christian Le. 2010.  RAPL: Memory power estimation and capping. In _2010 ACM/IEEE International Symposium on Low-Power Electronics and Design (ISLPED)_. IEEE, 189–194.
  * Edenhofer (2015) Ottmar Edenhofer. 2015\.  _Climate change 2014: mitigation of climate change_. Vol. 3.  Cambridge University Press.
  * Gathani et al. (2020) Sneha Gathani, Peter Lim, and Leilani Battle. 2020.  Debugging database queries: A survey of tools, techniques, and users. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Georgiou et al. (2022) Stefanos Georgiou, Maria Kechagia, Tushar Sharma, Federica Sarro, and Ying Zou. 2022.  Green AI: Do Deep Learning Frameworks Have Different Costs? ACM: Association for Computing Machinery.
  * Gupta et al. (2011) Ashish Gupta, Thomas Zimmermann, Christian Bird, Nachiappan Nagappan, Thirumalesh Bhat, and Syed Emran. 2011.  Detecting energy patterns in software development.  _Microsoft Research Microsoft Corporation One Microsoft Way Redmond, WA_ 98052 (2011).
  * Hasan et al. (2016) Samir Hasan, Zachary King, Munawar Hafiz, Mohammed Sayagh, Bram Adams, and Abram Hindle. 2016\.  Energy profiles of java collections classes. In _Proceedings of the 38th International Conference on Software Engineering_. 225–236.
  * Hindle et al. (2014) Abram Hindle, Alex Wilson, Kent Rasmussen, E Jed Barlow, Joshua Charles Campbell, and Stephen Romansky. 2014.  Greenminer: A hardware based mining software repositories software energy consumption framework. In _Proceedings of the 11th working conference on mining software repositories_. 12–21.
  * Jiao et al. (2018) Xun Jiao, Vahideh Akhlaghi, Yu Jiang, and Rajesh K Gupta. 2018.  Energy-efficient neural networks using approximate computation reuse. In _2018 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 1223–1228.
  * Khomh and Abtahizadeh (2018) Foutse Khomh and S Amirhossein Abtahizadeh. 2018\.  Understanding the impact of cloud patterns on performance and energy consumption.  _Journal of Systems and Software_ 141 (2018), 151–170.
  * Ko et al. (2017) Jong Hwan Ko, Burhan Mudassar, Taesik Na, and Saibal Mukhopadhyay. 2017.  Design of an energy-efficient accelerator for training of convolutional neural networks using frequency-domain computation. In _2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ 25 (2012).
  * Kumar and Shi (2019) Mohit Kumar and Weisong Shi. 2019.  Energy consumption analysis of java command-line options. In _2019 Tenth International Green and Sustainable Computing Conference (IGSC)_. IEEE, 1–8.
  * Lee et al. (2017b) Edward H Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S Simon Wong. 2017b.  Lognet: Energy-efficient neural networks using logarithmic computation. In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5900–5904.
  * Lee et al. (2017a) Sun-Ro Lee, Min-Jae Heo, Chan-Gun Lee, Milhan Kim, and Gaeul Jeong. 2017a.  Applying deep learning based automatic bug triager to industrial projects. In _Proceedings of the 2017 11th Joint Meeting on foundations of software engineering_. 926–931.
  * Maleki et al. (2017) Sepideh Maleki, Cuijiao Fu, Arun Banotra, and Ziliang Zong. 2017\.  Understanding the impact of object oriented programming and design patterns on energy efficiency. In _2017 Eighth International Green and Sustainable Computing Conference (IGSC)_. IEEE, 1–6.
  * McKinney et al. (2011) Wes McKinney et al. 2011\.  pandas: a foundational Python library for data analysis and statistics.  _Python for high performance and scientific computing_ 14, 9 (2011), 1–9.
  * Moons et al. (2016) Bert Moons, Bert De Brabandere, Luc Van Gool, and Marian Verhelst. 2016.  Energy-efficient convnets through approximate computing. In _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1–8.
  * Ournani et al. (2021) Zakaria Ournani, Romain Rouvoy, Pierre Rust, and Joel Penhoat. 2021.  Comparing the Energy Consumption of Java I/O Libraries and Methods. In _37th International Conference on Software Maintenance and Evolution (ICSME)_.
  * Park et al. (2018) Eunhyeok Park, Dongyoung Kim, and Sungjoo Yoo. 2018.  Energy-efficient neural network accelerator based on outlier-aware low-precision computation. In _2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 688–698.
  * Pereira et al. (2017) Rui Pereira, Marco Couto, Francisco Ribeiro, Rui Rua, Jácome Cunha, João Paulo Fernandes, and João Saraiva. 2017\.  Energy efficiency across programming languages: how do energy, time, and memory relate?. In _Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering_. 256–267.
  * Perez and Granger (2015) Fernando Perez and Brian E Granger. 2015.  Project Jupyter: Computational narratives as the engine of collaborative data science.  _Retrieved September_ 11, 207 (2015), 108.
  * Petersohn (2021) Devin Petersohn. 2021\.  _Dataframe Systems: Theory, Architecture, and Implementation_.  Ph. D. Dissertation. Ph. D. Dissertation. EECS Department, University of California, Berkeley ….
  * Pinto and Castor (2017) Gustavo Pinto and Fernando Castor. 2017.  Energy efficiency: a new concern for application software developers.  _Commun. ACM_ 60, 12 (2017), 68–75.
  * Procaccianti et al. (2016) Giuseppe Procaccianti, Héctor Fernández, and Patricia Lago. 2016.  Empirical evaluation of two best practices for energy-efficient software development.  _Journal of Systems and Software_ 117 (2016), 185–198.
  * Rocklin (2015) Matthew Rocklin. 2015\.  Dask: Parallel computation with blocked algorithms and task scheduling. In _Proceedings of the 14th python in science conference_ , Vol. 130. Citeseer, 136\.
  * Rouvoy et al. (2021) Romain Rouvoy, Pierre Rust, and Joel Penhoat. 2021\.  Comparing the Energy Consumption of Java I/O Libraries and Methods. In _ICSME 2021-37th International Conference on Software Maintenance and Evolution_.
  * Sahin et al. (2016) Cagri Sahin, Mian Wan, Philip Tornquist, Ryan McKenna, Zachary Pearson, William GJ Halfond, and James Clause. 2016.  How does code obfuscation impact energy usage?  _Journal of Software: Evolution and Process_ 28, 7 (2016), 565–588.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Sarwar et al. (2018) Syed Shakib Sarwar, Swagath Venkataramani, Aayush Ankit, Anand Raghunathan, and Kaushik Roy. 2018\.  Energy-efficient neural computing with approximate multipliers.  _ACM Journal on Emerging Technologies in Computing Systems (JETC)_ 14, 2 (2018), 1–23.
  * Schuler and Anderst-Kotsis (2020) Andreas Schuler and Gabriele Anderst-Kotsis. 2020.  Characterizing energy consumption of third-party API libraries using API utilization profiles. In _Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_. 1–11.
  * Singh et al. (2015) Jasmeet Singh, Kshirasagar Naik, and Veluppillai Mahinthan. 2015\.  Impact of developer choices on energy consumption of software on servers.  _Procedia Computer Science_ 62 (2015), 385–394.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019\.  Energy and Policy Considerations for Deep Learning in NLP. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 3645–3650.
  * Wu (2020) Yifan Wu. 2020.  Is a dataframe just a table?. In _10th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU 2019)_. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Yang et al. (2017) Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. 2017\.  Designing energy-efficient convolutional neural networks using energy-aware pruning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5687–5695.


# Efficient Multi-Resolution Fusion
for Remote Sensing Data with Label Uncertainty

###### Abstract

Multi-modal sensor data fusion takes advantage of complementary or reinforcing
information from each sensor and can boost overall performance in applications
such as scene classification and target detection. This paper presents a new
method for fusing multi-modal and multi-resolution remote sensor data without
requiring pixel-level training labels, which can be difficult to obtain.
Previously, we developed a Multiple Instance Multi-Resolution Fusion (MIMRF)
framework that addresses label uncertainty for fusion, but it can be slow to
train due to the large search space for the fuzzy measures used to integrate
sensor data sources. We propose a new method based on binary fuzzy measures,
which reduces the search space and significantly improves the efficiency of
the MIMRF framework. We present experimental results on synthetic data and a
real-world remote sensing detection task and show that the proposed MIMRF-BFM
algorithm can effectively and efficiently perform multi-resolution fusion
given remote sensing data with uncertainty.

Index Terms—  multi-resolution sensor fusion, multi-modal, Choquet integral,
binary fuzzy measure, hyperspectral, label uncertainty, efficiency

##  1 INTRODUCTION

Two fundamental challenges exist with remote sensor data integration. First,
existing optical sensors operate on various spatial, spectral, or temporal
resolutions. They may also produce heterogeneous data representations, such as
hyperspectral imagery on a pixel grid or LiDAR (Light Detection and Ranging)
point clouds with geometric measurements [1]. It is not always feasible to
convert all data to the same resolution or map to the same grid for fusion
without introducing misalignment errors or losing accuracy. Second, standard
supervised learning methods generally require accurate labels for each data
point, which can be difficult to obtain in high volume. Moreover, raw sensor
data often contains missing and imprecise measurements due to occlusion,
environmental factors, and sensor noise, making uncertainties unavoidable when
fusing real-world remote sensing data.

To address these challenges, we previously proposed a Multiple Instance Multi-
Resolution Fusion (MIMRF) method [2] for integrating multi-modal and
imprecisely labeled remote sensing data. It formulated the label uncertainty
problem using Multiple Instance Learning (MIL) [3] and relied on the Choquet
integral [4], a powerful non-linear aggregation tool to perform multi-
resolution fusion. However, the computational complexity of the original MIMRF
algorithm grows exponentially with the number of sensor sources to be fused,
which limits its efficiency. In this work, we propose an efficient alternative
to the original MIMRF algorithm by incorporating binary fuzzy measures (BFMs)
with the MIL framework to improve the efficiency while maintaining the
effectiveness of multi-resolution fusion. Results are presented on real-world
remote sensing data to demonstrate the efficiency and effectiveness of the
proposed fusion algorithm.

The remainder of the paper is organized as follows. Section 2 describes the
MIMRF framework. Section 3 presents our proposed efficient MIMRF with binary
fuzzy measures. Section 4 presents experimental results. Section 5 discusses
the proposed method and conclusions.

##  2 The MIMRF Framework

The MIMRF framework [2] takes heterogeneous sensor data as inputs (e.g.,
hyperspectral imagery and LiDAR point cloud) and learns a set of real-valued
variables (“fuzzy measures” [5]) that reflects the interactions among input
sensor sources. Assume there are $S$ sources to be fused, the size of the
fuzzy measure is $2^{S}$ as each fuzzy measure element corresponds to a subset
of the sensor combinations. A monotonic and normalized (non-binary) fuzzy
measure can take any real value between 0 and 1. The higher the fuzzy measure
value, the more weighting it is placing on the combination of sensor sources.
The MIMRF algorithm uses the Choquet integral [4] to perform fusion based on
the learned fuzzy measures. The advantage of the MIMRF framework is that it
can work with uncertain and imprecise labels, where training labels are
associated with groups of data points (called “bags” or superpixels [6])
instead of each pixel, which greatly reduces the need to individually label
every data point during training.

##  3 Efficient MIMRF with BFM (Proposed)

###  3.1 Binary Fuzzy Measures (BFMs)

In contrast to MIMRF, our proposed efficient MIMRF-BFM algorithm utilizes
binary fuzzy measures (BFMs) to reduce the search space and greatly improve
the efficiency. A BFM is defined as a real valued function that maps
$2^{S}\rightarrow\\{0,1\\}$. It satisfies $\mathscr{G}(\emptyset)=0$ (empty
set rule); $\mathscr{G}(S)=1$ (normalization); and
$\mathscr{G}(A)\leq\mathscr{G}(B)$ if $A\subseteq B$ and $A,B\subseteq S$
(monotonicity). Different from a standard real-valued (non-binary) normalized
fuzzy measure used in the previous work, the BFM only take values of $0$ or
$1$, instead of $[0,1]$. Thus, for $S$ sensor sources to be fused, BFM only
needs to search and optimize over $\\{0,1\\}^{2^{S}}$ instead of
$[0,1]^{2^{S}}$ for the real-valued fuzzy measures, which leads to a simpler
representation, a finite search space and more efficient computation.

###  3.2 Objective Function and Model Learning

As discussed in the Introduction, one challenge with fusing remote sensor data
is to accommodate the difference in resolution and modality among sensor
inputs. Assume we are fusing hyperspectral imagery (HSI) with LiDAR point
cloud data. Each pixel in HSI may correspond to multiple data points in the
LiDAR point cloud (see Figure 3 in [2] for an illustration). Additionally, we
assume pixel-level labels are not accurate enough during training and we can
only leverage bag-level labels with uncertainty (i.e., training labels are
provided per superpixel but not on a pixel-level, which is very common in
remote sensing data due to ground sample distance, sensor accuracy, etc.).
Thus, we account for these two levels of fusion uncertainty by writing the
objective function:

$\displaystyle\min_{\mathscr{G}}\,J$
$\displaystyle=\sum_{a=1}^{B^{-}}\max_{\mathcal{S}_{ai}^{-}\in\mathscr{B}_{a}^{-}}\left(\boxed{\min_{\mathbf{x}_{k}^{-}\in\mathcal{S}_{ai}^{-}}C_{\mathscr{G}}(\mathbf{x}_{k}^{-})}-0\right)^{2}$
(1)
$\displaystyle+\sum_{b=1}^{B^{+}}\min_{\mathcal{S}_{bj}^{+}\in\mathscr{B}_{b}^{+}}\left(\boxed{\max_{\mathbf{x}_{l}^{+}\in\mathcal{S}_{bj}^{+}}C_{\mathscr{G}}(\mathbf{x}_{l}^{+})}-1\right)^{2},$

where $B^{+}$ is the total number of positive bags containing the target
object or material we wish to detect (label $1$), $B^{-}$ is the total number
of negative bags containing only non-target background information (label
$0$), $\mathbf{x}_{k}^{-}$ is the $i^{th}$ instance in the $a^{th}$ negative
bag and $\mathbf{x}_{l}^{+}$ is the $j^{th}$ instance in the $b^{th}$ positive
bag, $C_{\mathscr{G}}$ is the Choquet integral (CI) fusion output computed
based on the binary fuzzy measure $\mathscr{G}$, $\mathscr{B}_{a}^{-}$ is the
$a^{th}$ negative bag, and $\mathscr{B}_{b}^{+}$ is the $b^{th}$ positive bag,
and $\mathcal{S}$ denotes the set of all possible matching combinations of the
multi-resolution/multi-modal sensor outputs. By minimizing this objective
function, we seek the unknown binary fuzzy measure $\mathscr{G}$ given all the
training data points ($\mathbf{x}$) and bag-level labels. The first term
encourages all instances in negative bags to produce a fusion result of “0”
(non-target) and the second term encourages at least one set of data instance
in positive bags to have label “1” (target), which satisfies the MIL
assumption accounting for label uncertainty given multi-resolution data. The
structure of our objective function is similar to the standard MIMRF method
(Eq.(7) in [2]), but we introduce the novel use of BFMs to compress the search
space of the fuzzy measure for enhanced efficiency. An evolutionary algorithm
[7] was used to train and optimize the BFMs. The proposed MIMRF-BFM algorithm
automatically learns the non-linear interactions and relationships (as
represented by the BFM values) among the input sensor data sources to produce
an optimized fusion result.

##  4 Experimental Results

###  4.1 MUUFL Gulfport Hyperspectral and LiDAR Fusion

Our proposed MIMRF-BFM algorithm was tested on a real-world remote sensing
fusion task on the MUULF Gulfport dataset [8]. The data set contains
hyperspectral imagery and 3-D LiDAR point clouds collected during two aerial
flights over the University of Southern Mississippi – Gulfpark campus. We
follow a similar setup as in [2] and perform building detection by fusing
three types of sensor inputs, one based on the adaptive coherence estimator
(ACE) to detect building spectral signatures and two based on geometric
measurements of building elevation from LiDAR point cloud data. These three
sensor sources are multi-modal and multi-resolutional (multiple LiDAR points
correspond to each HSI pixel due to the difference in sensor resolution and
measurement inaccuracy). Figure 1 shows the MUUFL Gulfport data containing
hyperspectral imagery, the LiDAR point clouds, and the ground truth map and
superpixel-level labels for fusion task.

Fig. 1: MUUFL Gulfport data. (a) Hyperspectral imagery. (b) Ground truth map
for building detection (yellow represents the target buildings that we wish to
detect); (c) Bag-level label map, where the superpixels containing target
buildings are labeled positive (red) and the rest are labeled negative (blue).
The superpixels are generated by the simple linear iterative clustering (SLIC)
algorithm [9]. (d) 3-D LiDAR point cloud. Fig. 2: Confidence maps of the three
individual sensor sources generated from the raw hyperspectral and LiDAR
sensor data and visual results of various fusion techniques. The color bar is
between 0 and 1 (blue is low and yellow is high confidence). (a) ACE detector
confidence based on the asphalt material in the hyperspectral imagery; (b)(c)
Two LiDAR height maps that highlight different building structures; (d)(e)(f)
Min, Max, and Mean of the three input sensor sources; (g) SVM; (h) mi-SVM; (i)
KNN; (j) MICI [10] (non-multi-res); (k) MIMRF (multi-res, regular FM) [2]; and
(l) Proposed MIMRF-BFM method (multi-res, with BFM).

The proposed MIMRF-BFM algorithm was compared with the detection results from
the three individual sensor input sources before fusion as well as a variety
of other fusion methods to analyze its effectiveness and efficiency. The
comparison methods include (i) non-multi-resolution fusion methods that
requires pixel-perfect labels, such as taking the min/max/mean of the three
input sources and using the support vector machine (SVM) and k-nearest
neighbor (KNN) for classification; (ii) mi-SVM [11], which is an MIL extension
to SVM that works with bag-level labels; (iii) the Multiple Instance Choquet
Integral (MICI) method with a noisy-or objective function [12, 10], which is a
non-multiresolution MIL fusion method based on the Choquet integral; (iv)
previous MIMRF method with regular (non-binary) fuzzy measures; and (v) our
proposed MIMRF-BFM method, which incorporates BFMs for added efficiency and
works with both multi-resolution remote sensing data and bag-level training
labels with uncertainty.

Table 1: The AUC and RMSE results of building detection using the MUUFL
Gulfport data fusion (Best, Second Best).

Fusion Method | AUC $\uparrow$ / RMSE $\downarrow$ / PSNR $\uparrow$
---|---
Train 1 Test 2 | Train 2 Test 1
ACE | 0.906/0.362/8.839 | 0.952/0.346/9.214
LiDAR1 | 0.888/0.267/11.497 | 0.880/0.272/11.319
LiDAR2 | 0.850/0.273/11.243 | 0.839/0.280/11.053
Min | 0.877/0.255/12.262 | 0.867/0.261/11.673
Max | 0.916/0.434/7.333 | 0.932/0.422/7.501
Mean | 0.941/0.310/10.492 | 0.953/0.302/10.400
SVM | 0.892/0.415/7.637 | 0.958/0.285/7.637
mi-SVM | 0.951/0.226/12.379 | 0.972/0.203/13.863
KNN | 0.954/0.237/12.437 | 0.952/0.243/12.279
MICI Noisyor | 0.943/0.377/8.621 | 0.946/0.326/9.030
MIMRF |  0.976/0.310/10.314 |  0.989/0.254/10.635
MIMRF-BFM |  0.974/0.131/17.661 |  0.973/0.128/17.859

Figure 3 shows a ROC (receiver operating characteristic) curve result and
Table 1 shows the AUC (area under curve), RMSE (root mean square error from
ground truth), and PSNR (peak signal-to-noise ratio from ground truth) results
across all methods for quantitative comparison. Higher AUC and PSNR and lower
RMSE indicate better detection results after fusion. As shown, our proposed
MIMRF-BFM achieved high detection performance, low error, and low noise
overall.

MIMRF-BFM produced an AUC score second to non-binary MIMRF, but outperformed
other fusion techniques. MIMRF-BFM also produced the lowest RMSE and highest
PSNR, indicating that the BFMs allow background noise to be eliminated more
effectively.

Figure 2 shows visual results of the sensor inputs and the fusion maps.
Compared to the non-binary MIMRF results (subfigure k), our proposed MIMRF-BFM
(subfigure l) suppresses the background pixels much better and only assigns a
high confidence score to the desired building pixels. This demonstrated that
the BFM actually helps improve the contrast between positive and negative
classes and benefits detection, as the learned BFM only takes values in
$\\{0,1\\}$.

We also investigated the actual BFM values learned during the fusion process.
The final BFM learned by the proposed MIMRF-BFM algorithm is
$\mathscr{G}_{12}=\mathscr{G}_{13}=1$, and other measure elements
($\mathscr{G}_{1}$, $\mathscr{G}_{2}$, $\mathscr{G}_{3}$, $\mathscr{G}_{23}$)
were zero. This means that the MIMRF-BFM algorithm correctly identified that
the intersection between source 1&2 and source 1&3 contributed most to the
detection of all four buildings. This makes sense, as the ACE detector (source
1) highlighted all asphalt materials including building rooftops as well as
roads, whereas source 2 and 3 are LiDAR height maps which highlighted partial
buildings but also tree canopies. By learning the combination among these
sources, the MIMRF-BFM was able to suppress false detections of other
materials such as roads and trees and instead only place high confidence on
the targets (buildings). In terms of computation time, the MIMRF takes on
average $30$min to train, whereas the MIMRF can complete the search and finish
training in $10$s (over 180 times faster).

Fig. 3: ROC curve results of building detection accuracy on the MUUFL Gulfport
dataset (cross validated over two flights). Orange box: zoomed-in view. Best
viewed in color.

###  4.2 Multi-Source BFM Efficiency Analysis

To further analyze the efficiency of the proposed MIMRF-BFM approach, we
generated an additional synthetic multi-resolution dataset with incrementally
growing (6, 8, 10, and 12) number of sensor sources and compared the
computation efficiency of the proposed MIMRF-BFM to the original MIMRF (non-
BFM) algorithm. The computation time (the time it took to converge on an
optimal FM/BFM) was recorded and presented in Table 2. It is clear that the
proposed MIMRF-BFM algorithm is capable of learning how to fuse a high number
of sources significantly faster than the original (non-binary) MIMRF. When the
input sensor sources to be fused increased to over ten, the MIMRF without BFMs
consistently require more than five hours to train without a guarantee to
converge, whereas the MIMRF-BFM can finish significantly faster due to the
reduction in search space.

Table 2: Computation time comparison between MIMRF and MIMRF-BFM.
Mean(Standard deviation) across 5 runs. Computation was capped at 5 hours.

Fusion Method | Computation Time (s)
---|---
#6 | #8 | #10 | #12
MIMRF | 149.5(148.0) | 772.1(442.1) | >$5$ hrs. | >$5$ hrs.
MIMRF-BFM | 17.6(1.1) | 92.1(5.1) | 120.3(5.0) | 772.4(15.9)

##  5 Discussion and Conclusions

This paper presents MIMRF-BFM, an effective and efficient extension to the
previously developed MIMRF framework that incorporates binary fuzzy measures.

The use of BFMs drastically reduced the search space during model learning and
resulted in significant improvements in efficiency, which was particularly
useful when scaling up the number of fusion sources. Additionally, the learned
BFM measure values provide a clear and explainable representation
corresponding to the combination and (non-linear) interactions of the sensor
input sources, which allows humans to interpret and gain insights on the
fusion process.

In addition to the HSI and LiDAR fusion task presented in this paper, the
proposed MIMRF-BFM can be extended to fusing alternative sensor modalities
accounting for varying resolution and label uncertainties. Future work
includes adapting the MIMRF-BFM to multi-view, multi-temporal and multi-
spatial sensor data with uncertainty and incorporating generalized fuzzy
measures, such as fuzzy measures on a bipolar scale (e.g., when FMs map to
$[-1,1]$) [13].

## References

  * [1] Danfeng Hong, Jocelyn Chanussot, and Xiao Xiang Zhu,  “An overview of multimodal remote sensing data fusion: From image to feature, from shallow to deep,”  in IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), 2021, pp. 1245–1248.
  * [2] Xiaoxiao Du and Alina Zare,  “Multiresolution multimodal sensor fusion for remote sensing data with label uncertainty,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 4, pp. 2755–2769, 2020.
  * [3] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez,  “Solving the multiple instance problem with axis-parallel rectangles,”  Artif. Intell., vol. 89, no. 1-2, pp. 31–71, Jan. 1997.
  * [4] Gustave Choquet,  “Theory of capacities,”  in Annales de l’institut Fourier, 1954, vol. 5, pp. 131–295.
  * [5] James M. Keller, Derong Liu, and David B. Fogel,  Fundamentals of computational intelligence: Neural networks, fuzzy systems and evolutionary computation,  IEEE Press Series on Computational Intelligence. John Wiley & Sons, Inc., 1st edition, 2016.
  * [6] Murong Wang, Xiabi Liu, Yixuan Gao, Xiao Ma, and Nouman Q Soomro,  “Superpixel segmentation: A benchmark,”  Signal Process.: Image Communication, vol. 56, pp. 28–39, 2017.
  * [7] Xiaoxiao Du, Alina Zare, and Derek T Anderson,  “Multiple instance choquet integral with binary fuzzy measures for remote sensing classifier fusion with imprecise labels,”  in IEEE Symp. Comput. Intell., 2019, pp. 1154–1162.
  * [8] Alina Zare, Paul Gader, Jen Aitken, Ryan Close, Grady Tuell, Taylor Glenn, Dmitri Dranishnikov, and Xiaoxiao Du,  “GatorSense/MUUFLGulfport: Release 01 (Version v0.1) [Data set],” https://github.com/GatorSense/MUUFLGulfport/tree/v0.1, 2018,  DOI: https://doi.org/10.5281/zenodo.1186326.
  * [9] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk,  “Slic superpixels compared to state-of-the-art superpixel methods,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 11, pp. 2274–2282, 2012.
  * [10] Xiaoxiao Du and Alina Zare,  “Multiple instance choquet integral classifier fusion and regression for remote sensing applications,”  IEEE Trans. Geosci. Remote Sens., vol. 57, no. 5, pp. 2741–2753, 2019.
  * [11] S. Andrews,  “Support vector machines for mulitple-instance learning,”  in Advances in Neural Info. Process. Syst., 2002.
  * [12] Xiaoxiao Du, Alina Zare, James M Keller, and Derek T Anderson,  “Multiple instance choquet integral for classifier fusion,”  in IEEE Congr. Evol. Comput., 2016, pp. 1054–1061.
  * [13] Michel Grabisch and Christophe Labreuche,  “Bi-capacities–II: the choquet integral,”  Fuzzy sets and systems, vol. 151, no. 2, pp. 237–259, 2005.