# Storage and Memory Characterization of Data Intensive Workloads for Bare
Metal Cloud

Hosein Mohammadi Makrani1
1George Mason University
Submission Type: Research Draft

###### Abstract

As the cost-per-byte of storage systems dramatically decreases, SSDs are
finding their ways in emerging cloud infrastructure. Similar trend is
happening for main memory subsystem, as advanced DRAM technologies with higher
capacity, frequency and number of channels are deploying for cloud-scale
solutions specially for non-virtualized environment where cloud subscribers
can exactly specify the configuration of underling hardware. Given the
performance sensitivity of standard workloads to the memory hierarchy
parameters, it is important to understand the role of memory and storage for
data intensive workloads. In this paper, we investigate how the choice of DRAM
(high-end vs low-end) impacts the performance of Hadoop, Spark, and MPI based
Big Data workloads in the presence of different storage types on bare metal
cloud. Through a methodical experimental setup, we have analyzed the impact of
DRAM capacity, operating frequency, the number of channels, storage type, and
scale-out factors on the performance of these popular frameworks. Based on
micro-architectural analysis, we classified data-intensive workloads into
three groups namely I/O bound, compute bound, and memory bound. The
characterization results show that neither DRAM capacity, frequency, nor the
number of channels play a significant role on the performance of all studied
Hadoop workloads as they are mostly I/O bound. On the other hand, our results
reveal that iterative tasks (e.g. machine learning) in Spark and MPI are
benefiting from a high-end DRAM in particular high frequency and large number
of channels, as they are memory or compute bound. Our results show that using
SSD PCIe cannot shift the bottleneck from storage to memory, while it can
change the workload behavior from I/O bound to compute bound.

###### Index Terms:

memory; storage; performance; cloud

##  I Introduction

Advances in various branches of technology – data sensing, data communication,
data computation, and data storage – are driving an era of unprecedented
innovation for information retrieval. The world of big data is constantly
changing and producing substantial amounts of data that creates challenges to
process it using existing solutions. To address this challenge, several
frameworks such as Hadoop and Spark, based on cluster computing, have been
proposed. The main characteristic of these new frameworks is their ability to
process large-scale data-intensive workloads on commodity hardware [1].

Data analytics heavily rely on deep machine learning and data mining
algorithms, and are running complex database software stack with significant
interaction with I/O and OS, and exhibit high computational intensity and I/O
intensity. In addition, unlike conventional CPU workloads, these workloads
combine a high data rate requirement with high computational power
requirement, in particular for real-time and near-time performance
constraints.

Three well-known parallel programming frameworks used by community are Hadoop,
Spark, and MPI. Hadoop and Spark are two prominent frameworks for big data
analytics. Spark has been developed to overcome the limitation of Hadoop on
efficiently utilizing main memory. Both Hadoop and Spark use clusters of
commodity hardware to process large datasets. MPI, a de facto industry
standard for parallel programming on distributed memory systems, is also used
for data analytics [2].

In the era of big data, it is important to evaluate the effect of memory and
storage parameters on the performance of data intensive workloads. While there
are literatures on understanding the behavior of those workloads, most of
prior works have focused on the CPU parameters such as core counts, core
frequency, cache parameters, and network configuration or I/O implication with
the assumption of the demand for using the fastest and largest main memory in
the commodity hardware [3, 4, 5, 6, 7, 8].

Since the storage is a well-known bottleneck in data intensive workloads,
several studies have been investigating the benefits of using Solid State
Drives as a replacement for traditional Hard Disk Drive [9, 10, 11, 12, 13,
14, 15, 16]. Solid-state storage offers several advantages over hard disks
such as lower access latencies for random requests, higher bandwidth, and
lower power consumption. Using SSD as storage raises the important question of
whether the bottleneck has shifted to memory subsystem or not. To answer this
question, we need to effectively characterize the performance of such
workloads with respect to memory parameters. However, none of the previous
works have studied the impact of main memory subsystem and storage at the same
time to characterize data intensive workloads and the underlying frameworks.

The objective of this paper is to perform joint analysis of the impact of
memory and storage parameters on the performance of data intensive workloads
on bare metal cloud such as IBM/SoftLayer. To perform the memory subsystem
analysis, we have investigated three configurable memory parameters including
memory capacity, memory frequency, and number of memory channels, three types
of storage (HDD, SSD SATA, and SSD PCIe), and scaling factors such as number
of cores, core frequency, and number of nodes to determine how these
parameters affect the performance of such workloads. This analysis helps in
making architectural decision such as what memory architecture to use to build
a server for data intensive workloads.

To the best of our knowledge this is the first work that looks beyond just the
memory capacity to understand Hadoop, Spark and MPI based big data workloads’
memory and storage behavior by analyzing the effect of memory frequency as
well as number of memory channels on the performance.

The remainder of this paper is organized as follows: Section 2 provides
technical overview of the investigated workloads and the experimental setup.
Results are presented in Section 3. Section 4 presents a discussion on the
results. Section 5 describes related works. Finally, Section 6 concludes the
paper.

##  II Experimental Setup

###  II-A Workloads

For this study we target various domains of workloads namely that of
microkernels, graph analytics, machine learning, E-commerce, social networks,
search engines, and multimedia. We used BigDataBench [2] and HiBench [17]. We
selected a diverse set of workloads and frameworks to be representative of
data intensive workloads. More details of these workloads are provided in
Table I. The selected workloads have different characteristics such as high
level data graph and different input/output ratios. Some of them have
unstructured data type and some others are graph based. Also these workloads
are popular in academia and are widely used in various studies. In our study,
we used Hadoop MapReduce version 2.7.1, Spark version 2.1.0 in conjunction
with Scala 2.11, and MPICH2 version 3.2 installed on Linux Ubuntu 14.04. Our
JVM version is 1.8.

TABLE I: Studied workloads

Workload | Domain | Input type | Input size (huge) | Framework | Suite
---|---|---|---|---|---
Wordcount | micro-kernel | text | 1.1 TB |  |
Sort | micro-kernel | data | 178.8 GB | Hadoop, Spark, MPI |
Grep | micro-kernel | text | 1.1 TB |  | BigData Bench
Terasort | micro-kernel | data | 834 GB | Hadoop, Spark |
Naive Bayes | E-commerce | Data | 306 GB | Hadoop, Spark, MPI |
Page Rank | E-commerce | Data | 306 GB | Hadoop, Spark |
Bayes | E-commerce | Data | 306 GB | Hadoop, Spark | HiBench
k-means | Machine learning | Graph | 112.2 GB | Hadoop, Spark, MPI | BigDataBench
nweight | Graph analytics | Graph | 176 GB | Spark | HiBench
Aggregation | Analytical query | Data | 1.08 TB | Hadoop
Join | Analytical query | Data | 1.08 TB
Scan | Analytical query | Data | 1.08 TB
B.MPEG | Multimedia | DVD stream | 437 GB | MPI | BigDataBench
DBN | Multimedia | Images | MNIST Dataset
Speech recognition | Multimedia | Audio | 252 GB
Image segmentation | Multimedia | Images | 162 GB
SIFT | Multimedia | Images | 162 GB
Face detection | Multimedia | Images | 162 GB

###  II-B Hardware platform

We carefully selected our experimental platform to investigate the micro
architectural effect on the performance of data intensive workloads to
understand whether our observations remain valid for future architectures with
enhanced micro architecture parameters or not. This includes analyzing the
results when increasing the core count, cache size and processor operating
frequency. This is important, as the results will shed light on whether in
future architectures larger number of cores, higher cache capacity and higher
operating frequency change memory behavior of data intensive workloads or not.
Using the data collected from our experimental test setup, we will drive
architectural conclusion on how these micro architecture parameters are
changing DRAM memory behavior and therefore impacting performance of
workloads. While network overhead in general is influencing the performance of
studied applications and therefore the characterization results, for data
intensive applications, as shown in a recent work [18], a modern high speed
network introduces only a small 2% performance benefit. We therefore used a
high speed 1 Gbit/s network to avoid making it a performance bottleneck for
this study. Our NICs have two ports and we used one of them per node for this
study. For running the workloads and monitoring statistics, we used a six-node
standalone cluster with detailed characteristics presented in Table II. We
used single socket servers in this study, in order to hide the NUMA effect (to
understand DRAM-only impact). To have a comprehensive experiment we used
different SDRAM memory modules and all modules are provided from the same
vendor.

TABLE II: Hardware Platform

Hardware | Parameter | Value
---|---|---
type
CPU | Model | Intel Xeon
E5-2683 V4
# Core | 16 (32 thread)
Base Frequency | 2.1 GHz
Turbo Frequency | 3.0 GHz
TDP | 120
L3 Cache | 40 MB
Memory Type | DDR4
Support | 1866/2133/2400
Maximum Memory | 76.8 GB/S
Bandwidth
Max Memory | 4
Channels supported
Disk | Model | Samsung 960 PRO M.2
Capacity | 512 GB
(SSD PCIE) | Speed | Max 3.5 GB/S
Disk | Model | HyperX FURY
Capacity | 480 GB
(SSD SATA) | Speed | 500 MB/S
Disk | Model | Seagate
Capacity | 500 GB
(HDD) | Speed | 7200 RPM
Network | Model | ST1000SPEXD4
Interface card | Speed | 1000 Mbps

###  II-C Methodology

The experimental methodology of this paper is focused on understanding how
studied frameworks are utilizing main memory and storage.

1) Data collection: We used Intel Performance Counter Monitor tool (PCM) [19]
to understand hardware (memory and processor) behavior. Several works used
performance counters to estimate the performance and power consumption of
processors [20, 21, 22] or employed performance counter for enhancing the
security [23, 24, 25]. In this work we use performance counters to study the
memory behavior. The performance counter data are collected for the entire run
of each workload. We collect OS-level performance information with DSTAT
tool—a profiling tool for Linux based systems by specifying the event under
study. Some of the metrics that we used for this study are memory footprint,
L2, and Last Level Cache (LLC) hits ratio, instruction per cycle (IPC), core
C0 state residency, and power consumption.

2) Parameter tuning: For both Hadoop and Spark frameworks, it is important to
set the number of mapper and reducer slots appropriately to maximize the
performance. Based on the result of [26], the maximum number of mappers
running concurrently on the system to maximize performance should be equal to
the total number of available CPU cores in the system. Therefore, for each
experiment, we set the number of mappers equal to the total number of cores.
We also follow same approach for the number of parallel tasks in MPI.
Adjusting default memory parameters of Hadoop and Spark also is important.
Hence, we tuned Hadoop and Spark memory related configuration parameters.
Followings are two most important memory related parameters that we tuned for
all experiments:

mapreduce.map.memory.mb: is the upper memory limit that Hadoop allows to be
allocated to a mapper, in megabytes. spark.executor.memory: Amount of memory
to use per executor process in Spark (e.g. 2g, 8g).

We set those values according to the following (we reserved 20% of DRAM
capacity for OS):

$\displaystyle mapreduce.map.memory.mb=$ (1) $\displaystyle(DRAMcapacity\times
0.8)/$ $\displaystyle NumberofConcurrentMappersperNode$

$\displaystyle spark.executor.memory=$ (2) $\displaystyle((DRAMcapacity-
spark.driver.memory)\times 0.8)/$ $\displaystyle NumberofExecutorperNode$

A recent work has shown that among all tuning parameters in a MapReduce
framework, HDFS block size is most influential on the performance [27]. HDFS
Block size has a direct relation to the number of parallel tasks (in Spark and
Hadoop), as shown in EQ. (3).

$\displaystyle\begin{aligned} \@add@centering\centering
NumberofTasks=InputSize/BlockSize\end{aligned}$ (3)

In the above equation, the input size is the size of data that is distributed
among nodes. The block size is the amount of data that is transferred among
nodes. Hence, block size has impact on the network traffic and its usage.
Therefore, we first evaluate how changing this parameter affects the
performance of the system. We studied a broad range of HDFS block sizes
varying from 32 MB to 1GB when the main memory capacity is 64 GB and it has
the highest frequency and number of channels. Table 5 demonstrates the best
HDFS configuration for maximizing the performance in both Hadoop and Spark
frameworks based on the ratio of Input data size to the total number of
available processing cores, and the workload class. The rest of the
experiments presented in this paper are based on Table III configuration. Our
tuning methodology guarantees to put the highest pressure on memory subsystem.

TABLE III: HDFS block size tuning

Application |  $Input\ size/(\\#\ nodes\times\\#cores\ per\ node$)
---|---
class |  $<$64 MB |  $<$512 MB |  $<$4 GB |  $>$ 4 GB
CPU | 32 MB | 64 MB | 128 MB | 256 MB
intensive
I/O | 64 MB | 256 MB | 512 MB | 1 GB
intensive
Iterative | 64 MB | 128 MB | 256 MB | 512 MB
tasks

Figure 1: Impact of CPU frequency on execution time and C0 state residency

##  III Results

Our experimental results are presented in this section. First, we present the
classification of workloads into memory bound, compute bound, and I/O bound,
which helps to accurately present the relation of performance and workload
characteristics. Then, we present the memory analysis of the studied
workloads. Then, we provide results of architectural implication of processor
parameters on data intensive workloads. All performance metrics such as
execution time, CPU active state residency, LLC and L2 hit ratio are discussed
in this section. We also discuss the impact of storage system, and cluster
size on memory subsystem.

###  III-A Classification of workloads

As the main goal of this paper is to study the combined impact of node
architecture and data intensive workload’s characteristics, it is important to
classify those workloads first. To this goal, we have explored the micro
architectural behavior of studied workloads to classify them and find more
insights. Throughout this section we will present the results based on high
speed SSD disk.

1) Core frequency implication: Figure 1 shows that studied workloads behave in
two distinct ways. The execution time of the first group is decreased linearly
by increasing the core frequency. The second group’s execution time does not
drop significantly by increasing the CPU frequency, particularly when changing
frequency from 1.9 GHz to 2.6 GHz. These two trends indicate that studied
workloads have distinct behaviors of being either CPU bound or I/O bound. This
conclusion further advocated by C0 state residency of processor. This proves
sort, grep, PageRank, and scan from Hadoop, wordcount, grep, PageRank, Bayes,
and nBayes from Spark, and sort, BasicMPEG, and grep from MPI to be Disk-
intensive while others to be CPU-intensive. This can be explained as follows:
If increasing the processor’s frequency reduces the active state residency
(C0) of the processor, the workload is I/O bound, as when a core is waiting
for I/O, the core changes its state to save power. Similarly, if active state
residency does not change the workload is CPU bound.

2) Cache implication: Modern processor has a 3-level cache hierarchy. Figure 2
shows cache hit ratio of level 2 (L2) and last level cache (LLC). The results
reveal an important characteristic of data intensive workloads. Our
experimental results show most of the studied workloads (particularly
MapReduce workloads) have a much higher cache hit ratio, which helps reducing
the number of accesses to the main memory. Based on simulation as well as
real-system experiment results in recent works, it is reported that these
applications’ cache hit rate is too low (under 10%) [3, 4] for a system with
10 MB of LLC and for having LLC hit rate of 40%, the system should have around
100 MB of LLC. However, our real system experimental results show that most of
data intensive workloads have much higher LLC hit rate (more than 50%) with
only 40 MB LLC. The reason of high cache hit ratio is that each parallel task
of MapReduce framework processes data in a sequential manner. This behavior
increases the cache hits; therefore it prevents excessive access to DRAM.
Hence, based on the cache hit ratio of CPU intensive workloads and the
intensity of accesses to memory, we can classify CPU bound workloads into two
more groups namely compute bound and memory bound. If the cache hit ratio is
low and the workload is an iterative task, it is classified as memory
intensive. Our characterization showed that N-weight and Kmeans from Spark,
and Image Segmentation from MPI are memory intensive. Therefore, we divided
our workloads into three major groups of I/O bound, compute bound, and memory
bound. Based on this classification we present our result in the following
sections.

Figure 2: LLC and L2 hit rate

###  III-B Memory analysis

In this section, we present a comprehensive discussion on memory analysis
results to help better understanding the memory requirements of data intensive
Workloads.

1) Memory channels implication: The off-chip peak memory bandwidth equation is
shown in EQ. (4).

$\displaystyle\begin{aligned} \@add@centering\centering
Bandwidth=Channels\times Frequency\times Width\end{aligned}$ (4)

We observe in Figure 3 that increasing the number of channels have significant
effect on the execution time of memory bound workloads (All of them are
iterative tasks). Figure 4 provides more insights to explain this exceptional
behavior. This Figure demonstrates the memory bandwidth utilization of each
group. Bandwidth utilization of memory bound workloads is shown to be
substantially higher than other workloads. Hence providing more bandwidth
decreases their execution time. By increasing the number of channels from 1 to
4, the gain is found to be 28%.

Figure 3: Effect of channel on the execution time (Normalized to 1CH)
Figure 4: Average memory bandwidth utilization

2) Memory frequency implication: As results in Figure 5 shows, similarly we do
not observe significant improvement of bandwidth utilization or execution time
by increasing memory frequency (from 1866 MHz to 2400 MHz) for none memory
bound workloads. This finding may mislead to use the lowest memory frequency
for other workloads. Based on EQ. (5), read latency of DRAM depends on the
memory frequency.

$\displaystyle\begin{aligned} \@add@centering\centering
Readlatency=2\times(CL/Frequency)\end{aligned}$ (5)

Figure 5: Effect of memory frequency on the execution time (Normalized to
1866M)

However, for DDRx (e.g. DDR3), this latency is set fixed by the manufacturer
with controlling CAS latency (CL). This means two memory modules with
different frequency (1333 MHz and 1866 MHz) and different CAS Latency (9 and
13) can have the same read latency of 13.5 ns, but provide different bandwidth
per channel (10.66 GB/s and 14.93 GB/s). Hence, as along as reduction of
frequency does not change the read latency, it is recommended to reduce DRAM
frequency for most of data intensive applications unless the application is
memory intensive. Later in this paper we will discuss memory sensitivity of
studied applications.

3) DRAM capacity implication: To investigate the effect of memory capacity on
the performance of data intensive workloads, we run all workloads with 7
different memory capacities per node. During our experiments, Spark workloads
encountered an error when running on a 4GB memory capacity per node due to
lack of memory space for the Java heap. Hence, the experiment of Spark
workloads is performed with at least 8 GB of memory. Based on our observation,
we found that only MPI workloads have an unpredictable memory capacity usage.
In fact, a large memory capacity has no significant effect on the performance
of studied Hadoop and Spark workloads. Hadoop workloads do not require high
capacity memory because Hadoop stores all intermediate values generated by map
tasks on the storage. Hence, regardless of the number of map tasks or input
size, the memory usage remains almost the same. Spark uses RDD to cache
intermediate values in memory. Hence, by increasing the number of map tasks to
run on a node, the memory usage increases. Therefore, by knowing the number of
map tasks assigned to a nodes and the amount of intermediate values generated
by each task, the maximum memory usage of Spark workloads is predictable per
node. To better understand the impact of memory capacity on the performance,
we have provided the average normalized execution time of these three
frameworks in Figure 6 (Normalized to 64 GB). To illustrate how these
frameworks utilize DRAM capacity we present K-means memory usage on 3
different frameworks in Figure 7.

Figure 6: Impact of memory capacity per node on performance
Figure 7: K-means memory usage on various frameworks

###  III-C Performance analysis

1) Disk implication: To show how choice of storage can change the performance
while using different memory configuration, we performed several experiments
using three types of storage (HDD, SSD SATA, and SSD PCIe). Figure 8 shows
that changing the disk from HDD to SSD PCIe improves the performance of Spark,
Hadoop, and MPI by 1.6x, 2.4x, and 3.3x respectively. The reason that MPI
workloads take more advantage from faster disk is that these workloads are
written in C++. However, Hadoop and Spark are Java based frameworks and they
use HDFS as an intermediate layer to access and manage storage. Our results
show a high bandwidth DRAM is not required to accelerate the performance of
MapReduce frameworks in presence of a slow HDD. However, MPI based workloads
has the potential to benefit from high-end DRAM.

Figure 8: Effect of memory and storage configuration on the performance

Another point regarding the storage is to use multiple disks per node to
alleviate IO bottleneck. We performed a new set of experiments with two SSD
storages per node. We found that HDFS is not aware of multiple disks on the
node and all data is written or read from one disk. Therefore, using multiple
disks per node does not guarantee the parallel access to the data blocks of
HDFS to reduce the IO bottleneck. Another point is to use RAID. Since HDFS is
taking care of fault-tolerance and ”striped” reading, there is no need to use
RAID underneath an HDFS. Using RAID will only be more expensive, offer less
storage, and also be slower (depending on the concrete RAID configuration).
Since the Namenode is a single-point-of-failure in HDFS, it requires a more
reliable hardware setup. Therefore, the use of RAID is recommended only on
Namenodes. In our experiments, we also have increased aggregate storage by
increasing the number of nodes (from 6 nodes to 12 nodes). We present the
results in section III.C.3. The result reveals that increasing the number of
nodes not only reduces IO requests’ pressure on each node, but also it reduces
pressure on memory subsystem of each node.

It is important to note that SSD increases the read and write bandwidth of
disk and substantially reduces the latency of access to disk compared to HDD.
However, accessing to I/O means loosing of millions of CPU cycles, which is
large enough to vanish any noticeable advantage of using a high-performance
DRAM. On the other hand, the only way to take advantage of a SSD is to read or
write a big file (hundreds of megabyte) at once but our result shows that HDFS
reads the data in much smaller blocks, regardless of HDFS block size.

2) Core count implication: In the previous section, we classified workloads
into two groups of CPU intensive and I/O intensive. Figure 9 demonstrates the
effect of increasing the number of cores per node on the performance of these
two groups. The expectation is that performance of the system improves
linearly by adding cores because big data workloads are heavily parallel.
However, we observe a different trend. For CPU intensive workloads and when
the core count is less than 6 cores per node, the performance improvement is
close to the ideal case. The interesting trend is that increasing the number
of cores per node does not improve the performance of data intensive workloads
noticeably beyond 6 cores. As the increase in the number of cores increases
the number of accesses to the disk, the disk becomes the bottleneck of the
system. At 8 cores, the CPU utilization is dropped to 44% for I/O intensive
workloads, on average.

Figure 9: Effect of core count on the performance

Based on these observations, we develop Eq. (6) to find the number of cores
for which further increase does not noticeably enhance the performance of
system:

$\displaystyle\begin{aligned} \@add@centering\centering Max(cores)=((BW\times
Nd))/((Nsc\times Fr\times\lambda))\end{aligned}$ (6)

We define the variables used in this equation as follow: BW is the nominal
bandwidth of each disk. Nd is the number of disk installed on the server. Nsc
is the number of sockets. Fr is CPU core frequency and Lambda is a constant,
which we found through our real-system experiments. As the effective I/O
bandwidth depends on the block size and I/O request size, we have used fio
[28] to calculate Lambda for different block size requests, presented in
figure 10. Designers can use this equation to select an optimum configuration,
such as the number of cores, core frequency, disk type, and number of disks
per node. As an example, the number of cores beyond which there is no
noticeable performance gain on a server with one socket, one SSD storage with
nominal 400 MBpS bandwidth, 16KB IO request size, running at 2.8 GHz is 8
based on the above equation. We have validated equation 1 for different
classes of workload and the result is presented in Figure 11.

Figure 10: Lambda value for different request size and storage type
Figure 11: Average error of optimum core count prediction

3) Cluster size implication: Our cluster size (6 nodes) is small compared to a
real-world server cluster. It is therefore important to understand the impact
of cluster size on the memory characterization results. To achieve this, we
performed three additional experiments with a single node, a three-node, as
well as a twelve-node cluster to study the effect of cluster size on memory
subsystem performance. Figure 12 shows the result of our experiments. These
results show that increasing the size of cluster from 1 to 12 changes the
memory behavior; it slightly reduces the pressure on memory subsystem.
Increasing the cluster size reduces both memory usage and memory bandwidth
utilization on each node, on average. Based on these results we anticipate
that the memory subsystem mostly will not be under pressure in large scale
server cluster. Therefore, we anticipate no urgent need to over provision
memory subsystem in such environment for the studied workloads.

Figure 12: Node count implication on memory usage

##  IV Discussion

The contribution of this paper is to give an insight on the role the memory
and storage subsystem in the overall performance of the bare metal servers
when running data intensive workloads. Our experimental results illustrate
that data intensive workloads show three distinct behaviors of being I/O
bound, compute bound, and memory bound. Based on the results presented in this
paper, we observed that Hadoop framework is not memory intensive. This means
Hadoop does not require high frequency, and large number of channels memory
for higher performance. Moreover, Spark and Hadoop frameworks’ memory usage is
bounded because they are MapReduce based frameworks and use HDFS block-based
file system for storage. On the other hand, MPI framework’s memory usage is
not predictable and different workload may have different demands for memory
capacity. Our results show MPI and Spark based iterative tasks benefit from
high frequency memory as they are memory bound workloads. Increasing the
number of memory channels beyond two channels does not enhance the performance
of those workloads. This is an indication for lack of efficient memory
allocation and management in both hardware (memory controller) and software
stacks. It is important to use our finding to help decision making in
budgeting server infrastructure when building cloud or even when scheduling
workloads. For instance, with a given budget to build a cluster, our results
suggest allocating more budget to acquire more nodes rather than expensive
high performance DRAM subsystems. Just as an example, the price of a server
with two Sockets and 8 cores per socket with 128 GB memory running at 1333 MHz
is equal to a server with one Socket and 8 cores with 256 GB memory running at
1866MHz. For a designer who wants to select a server for MapReduce clusters,
our results suggest the former one to be a better option.

##  V Related Work

Latest works on memory characterization of big data applications are [29, 30,
31]. However, these works did not study the impact of storage. Another recent
work studied the effect of memory bandwidth on the performance of MapReduce
frameworks and presented a memory navigator for modern hardware [32]. A recent
work on big data [3] profiles the memory access patterns of Hadoop and noSQL
workloads by collecting memory DIMM traces using special hardware. This study
does not examine the effects of memory frequency and number of channels on the
performance of the system. A more recent work [33] provides a performance
model that considers the impact of memory bandwidth and latency for big data,
high performance, and enterprise workloads. The work in [4] shows how Hadoop
workload demands different hardware resources. This work also studies the
memory capacity as a parameter that impacts the performance. However, as we
showed in this work, their finding is in contrast with ours. In [34] the
authors evaluate contemporary multi-channel DDR SDRAM and Rambus DRAM systems
in SMT architectures. The work in [35] mainly focuses on page table and
virtual memory optimization of big data and [5] presents the characterization
of cache hierarchy for a Hadoop cluster. Few works [36, 37, 38] studied the
impact of fault tolerant techniques on the performance and memory usage of
embedded system. These works do not analyze the memory subsystem.

A recent work on big data benchmarking [39] analyzes the redundancy among
different big data benchmarks such as ICTBench, HiBench and traditional CPU
workloads and introduces a new big data benchmark suite for spatio-temporal
data. The work in [6] selects four big data workloads from the BigDataBench
[2] to study I/O characteristics, such as disk read/write bandwidth, I/O
devices utilization, average waiting time of I/O requests, and average size of
I/O requests. Another work [7] studies the performance characterization of
Hadoop and DataMPI, using Amdahl’s second law. This study shows that a DataMPI
is more balanced than a Hadoop system. In a more recent work [40] the authors
analyze three SPEC CPU2006 benchmarks (libquantum, h264ref, and hmmer) to
determine their potential as big data computation workloads. The work in [41]
examines the performance characteristics of three high performance graph
analytics. One of their findings is that graph workloads fail to fully utilize
the platform’s memory bandwidth. The work in [8] performs performance analysis
and characterizations for Hadoop K-means iterations. Moreover, there are
studies on hardware acceleration of Hadoop applications that do not analyze
the impact of memory and storage on the performance [42, 43].

##  VI Conclusions

Characterizing the behavior of data intensive workloads is important as it
helps guiding scheduling decision in cloud scale architectures as well as
helping making decisions in designing server cluster for big data computing.
This work performs a comprehensive analysis of memory requirements through an
experimental evaluation setup. We study diverse domains of workloads from
microkernels, graph analytics, machine learning, E-commerce, social networks,
search engines, and multimedia in Hadoop, Spark, and MPI. This gives us
several insights into understanding the memory and storage role for these
important frameworks. We observe that most of studied workloads in MapReduce
based frameworks such as Hadoop and Spark do not require a high-end memory. On
the other hand MPI workloads, as well as iterative tasks in Spark (e.g.
machine learning) benefit from a high-end memory. Moreover, our result shows
that changing the disk from HDD to SSD improves the performance of Spark,
Hadoop, and MPI by 1.6x, 2.4x, and 3.3x respectively. However, I/O bandwidth
caps the performance benefit of multicore CPU. Therefore, we developed an
experimental equation to help designers to find the number of cores for which
further increase does not enhance system performance noticeably. Moreover, we
found that the current storage systems are the main bottleneck for the studied
applications hence any further improvement of memory and CPU architecture
without addressing the storage problem is a waste of money and energy. To the
best of our knowledge this is the first work that looks beyond just the memory
capacity to understand memory behavior by analyzing the effect of memory
frequency as well as the number of memory channels on the performance of
Hadoop, Spark, and MPI based big data workloads.

## References

  * [1] E. Bertino, “Big data-opportunities and challenges,” _IEEE. p479-480_ , 2013\.
  * [2] L. Wang, J. Zhan, C. Luo, Y. Zhu, Q. Yang, Y. He, W. Gao, Z. Jia, Y. Shi, S. Zhang _et al._ , “Bigdatabench: A big data benchmark suite from internet services,” in _High Performance Computer Architecture (HPCA), 2014 IEEE 20th International Symposium on_. IEEE, 2014, pp. 488–499.
  * [3] M. Dimitrov, K. Kumar, P. Lu, V. Viswanathan, and T. Willhalm, “Memory system characterization of big data workloads,” in _Big Data, 2013 IEEE International Conference on_. IEEE, 2013, pp. 15–22.
  * [4] I. Alzuru, K. Long, B. Gowda, D. Zimmerman, and T. Li, “Hadoop characterization,” in _Trustcom/BigDataSE/ISPA, 2015 IEEE_ , vol. 2. IEEE, 2015, pp. 96–103.
  * [5] Z. Jia, L. Wang, J. Zhan, L. Zhang, and C. Luo, “Characterizing data analysis workloads in data centers,” in _Workload Characterization (IISWC), 2013 IEEE International Symposium on_. IEEE, 2013, pp. 66–76.
  * [6] F. Pan, Y. Yue, J. Xiong, and D. Hao, “I/o characterization of big data workloads in data centers,” in _Workshop on Big Data Benchmarks, Performance Optimization, and Emerging Hardware_. Springer, 2014, pp. 85–97.
  * [7] F. Liang, C. Feng, X. Lu, and Z. Xu, “Performance characterization of hadoop and data mpi based on amdahl’s second law,” in _Networking, Architecture, and Storage (NAS), 2014 9th IEEE International Conference on_. IEEE, 2014, pp. 207–215.
  * [8] J. Issa, “Performance characterization and analysis for hadoop k-means iteration,” _Journal of Cloud Computing_ , vol. 5, no. 1, p. 3, 2016.
  * [9] J. Hong, L. Li, C. Han, B. Jin, Q. Yang, and Z. Yang, “Optimizing hadoop framework for solid state drives,” in _Big Data (BigData Congress), 2016 IEEE International Congress on_. IEEE, 2016, pp. 9–17.
  * [10] N. S. Islam, X. Lu, M. Wasi-ur Rahman, D. Shankar, and D. K. Panda, “Triple-h: A hybrid approach to accelerate hdfs on hpc clusters with heterogeneous storage architecture,” in _Cluster, Cloud and Grid Computing (CCGrid), 2015 15th IEEE/ACM International Symposium on_. IEEE, 2015, pp. 101–110.
  * [11] Y. Kim, A. Gupta, B. Urgaonkar, P. Berman, and A. Sivasubramaniam, “Hybridstore: A cost-efficient, high-performance storage system combining ssds and hdds,” in _Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), 2011 IEEE 19th International Symposium on_. IEEE, 2011, pp. 227–236.
  * [12] T. Harter, D. Borthakur, S. Dong, A. S. Aiyer, L. Tang, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau, “Analysis of hdfs under hbase: a facebook messages case study.” in _FAST_ , vol. 14, 2014, p. 12th.
  * [13] K. Kambatla and Y. Chen, “The truth about mapreduce performance on ssds.” in _LISA_ , 2014, pp. 109–118.
  * [14] K. Krish, M. S. Iqbal, and A. R. Butt, “Venu: Orchestrating ssds in hadoop storage,” in _Big Data (Big Data), 2014 IEEE International Conference on_. IEEE, 2014, pp. 207–212.
  * [15] S. Ahn and S. Park, “An analytical approach to evaluation of ssd effects under mapreduce workloads,” _JOURNAL OF SEMICONDUCTOR TECHNOLOGY AND SCIENCE_ , vol. 15, no. 5, pp. 511–518, 2015.
  * [16] I. S. Choi, W. Yang, and Y.-S. Kee, “Early experience with optimizing i/o performance using high-performance ssds for in-memory cluster computing,” in _Big Data (Big Data), 2015 IEEE International Conference on_. IEEE, 2015, pp. 1073–1083.
  * [17] S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, “The hibench benchmark suite: Characterization of the mapreduce-based data analysis,” in _Data Engineering Workshops (ICDEW), 2010 IEEE 26th International Conference on_. IEEE, 2010, pp. 41–51.
  * [18] K. Ousterhout, R. Rasti, S. Ratnasamy, S. Shenker, B.-G. Chun, and V. ICSI, “Making sense of performance in data analytics frameworks.” in _NSDI_ , vol. 15, 2015, pp. 293–307.
  * [19] _Available at: https://software.intel.com/en-us/articles/intel-performance-counter-monito_.
  * [20] H. Sayadi, D. Pathak, I. Savidis, and H. Homayoun, “Power conversion efficiency-aware mapping of multithreaded applications on heterogeneous architectures: A comprehensive parameter tuning,” in _Design Automation Conference (ASP-DAC), 2018 23rd Asia and South Pacific_. IEEE, 2018, pp. 70–75.
  * [21] H. Sayadi and H. Homayoun, “Scheduling multithreaded applications onto heterogeneous composite cores architecture,” in _Green and Sustainable Computing Conference (IGSC), 2017 Eighth International_. IEEE, 2017, pp. 1–8.
  * [22] H. Sayadi, N. Patel, A. Sasan, and H. Homayoun, “Machine learning-based approaches for energy-efficiency prediction and scheduling in composite cores architectures,” in _Computer Design (ICCD), 2017 IEEE International Conference on_. IEEE, 2017, pp. 129–136.
  * [23] H. Sayadi, S. M. P D, A. Houmansadr, S. Rafatirad, and H. Homayoun, “Comprehensive assessment of run-time hardware-supported malware detection using general and ensemble learning,” _ACM International Conference on Computing Frontiers, CF_ , 2018.
  * [24] H. Sayadi, N. Patel, S. M. P D, A. Sasan, S. Rafatirad, and H. Homayoun, “Ensemble learning for effective run-time hardware-based malware detection: A comprehensive analysis and classification,” _ACM/IEEE Design Automation Conference, DAC_ , 2018.
  * [25] H. Sayadi, H. Makrani, O. Randive, S. M. P D, S. Rafatirad, and H. Homayoun, “Customized machine learning-based hardware-assisted malware detection in embedded devices,” _The 17th IEEE International Conference On Trust, Security And Privacy In Computing And Communications (IEEE TrustCom-18)_ , 2018\.
  * [26] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “Clearing the clouds: a study of emerging scale-out workloads on modern hardware,” in _ACM SIGPLAN Notices_ , vol. 47, no. 4. ACM, 2012, pp. 37–48.
  * [27] M. Malik, A. Sasan, R. Joshi, S. Rafatirah, and H. Homayoun, “Characterizing hadoop applications on microservers for performance and energy efficiency optimizations,” in _Performance Analysis of Systems and Software (ISPASS), 2016 IEEE International Symposium on_. IEEE, 2016, pp. 153–154.
  * [28] K. Salah, M. Al-Saba, M. Akhdhor, O. Shaaban, and M. Buhari, “Performance evaluation of popular cloud iaas providers,” in _Internet Technology and Secured Transactions (ICITST), 2011 International Conference for_. IEEE, 2011, pp. 345–349.
  * [29] H. M. Makrani and H. Homayoun, “Memory requirements of hadoop, spark, and mpi based big data applications on commodity server class architectures,” in _Workload Characterization (IISWC), 2017 IEEE International Symposium on_. IEEE, 2017, pp. 112–113.
  * [30] H. M. Makrani, S. Tabatabaei, S. Rafatirad, and H. Homayoun, “Understanding the role of memory subsystem on performance and energy-efficiency of hadoop applications,” in _Green and Sustainable Computing Conference (IGSC), 2017 Eighth International_. IEEE, 2017, pp. 1–6.
  * [31] H. M. Makrani, S. Rafatirad, A. Houmansadr, and H. Homayoun, “Main-memory requirements of big data applications on commodity server platform,” in _IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGRID)_ , 2018.
  * [32] H. M. Makrani and H. Homayoun, “Mena: A memory navigator for modern hardware in a scale-out environment,” in _2017 IEEE International Symposium on Workload Characterization (IISWC)_. IEEE, 2017, pp. 2–11.
  * [33] R. Clapp, M. Dimitrov, K. Kumar, V. Viswanathan, and T. Willhalm, “Quantifying the performance impact of memory latency and bandwidth for big data workloads,” in _Workload Characterization (IISWC), 2015 IEEE International Symposium on_. IEEE, 2015, pp. 213–224.
  * [34] Z. Zhu and Z. Zhang, “A performance comparison of dram memory system optimizations for smt processors,” in _High-Performance Computer Architecture, 2005. HPCA-11. 11th International Symposium on_. IEEE, 2005, pp. 213–224.
  * [35] A. Basu, J. Gandhi, J. Chang, M. D. Hill, and M. M. Swift, “Efficient virtual memory for big memory servers,” in _ACM SIGARCH Computer Architecture News_ , vol. 41, no. 3. ACM, 2013, pp. 237–248.
  * [36] H. M. Makrani, A. Hosseini Monazzah, H. Farbeh, and S. G. Miremadi, “Evaluation of software-based fault-tolerant techniques on embedded os’s components,” in _International Conference on Dependability (DEPEND)_ , 2014, pp. 51–57.
  * [37] H. Sayadi, H. Farbeh, A. M. H. Monazzah, and S. G. Miremadi, “A data recomputation approach for reliability improvement of scratchpad memory in embedded systems,” in _Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT), 2014 IEEE International Symposium on_. IEEE, 2014, pp. 228–233.
  * [38] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, “Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing,” in _Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation_. USENIX Association, 2012, pp. 2–2.
  * [39] W. Xiong, Z. Yu, Z. Bei, J. Zhao, F. Zhang, Y. Zou, X. Bai, Y. Li, and C. Xu, “A characterization of big data benchmarks,” in _Big Data, 2013 IEEE International Conference on_. IEEE, 2013, pp. 118–125.
  * [40] K. Hurt and E. John, “Analysis of memory sensitive spec cpu2006 integer benchmarks for big data benchmarking,” in _Proceedings of the 1st Workshop on Performance Analysis of Big Data Systems_. ACM, 2015, pp. 11–16.
  * [41] S. Beamer, K. Asanovic, and D. Patterson, “Locality exists in graph processing: Workload characterization on an ivy bridge server,” in _Workload Characterization (IISWC), 2015 IEEE International Symposium on_. IEEE, 2015, pp. 56–65.
  * [42] K. Neshatpour, H. M. Makrani, A. Sasan, H. Ghasemzadeh, S. Rafatirad, and H. Homayoun, “Design space exploration for hardware acceleration of machine learning applications in mapreduce,” in _The 26th IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM)_ , 2018.
  * [43] K. Neshatpour, H. Mohammadi Makrani, A. Sasan, H. Ghasemzadeh, S. Rafatirad, and H. Homayoun, “Architectural considerations for fpga acceleration of machine learning applications in mapreduce,” in _SAMOS_. IEEE, 2018.