Vrije Universiteit Amsterdam

Honours Program, Project Report

Operational Characterization of a
Public Scientific Datacenter During and
Beyond the COVID-19 Period

Author: Mehmet Berk Cetin (2644886)
---
1st supervisor: | prof. dr. ir. Alexandru Iosup
---|---
daily supervisor: | ir. Laurens Versluis
2nd reader: | 1st supervisor & daily supervisor

A report submitted in fulfillment of the requirements for the Honours Program,
which is an excellence annotation to the VU Bachelor of Science degree in
Computer Science/Artificial Intelligence/Information Sciences
version 1.0

###### Table of Contents

  1. 1 Introduction
  2. 2 Background Information
  3. 3 Methodology
    1. 3.1 Extracting, processing, and analyzing the data
    2. 3.2 The Scientific Instrument
  4. 4 Approach & Contributions
  5. 5 Results and Analysis
    1. 5.1 General Utilization Levels
    2. 5.2 Daily and Hourly Utilization Levels
    3. 5.3 Rack Utilization levels
    4. 5.4 Overview of the COVID-19 period
    5. 5.5 Custom rack/node utilization levels
  6. 6 Threats to Validity
  7. 7 Related Work
  8. 8 Conclusion and Future Work
  9. A Problem Statement
  10. B Self-Reflection

## Abstract

Datacenters are imperative for the digital society. They offer services such
as computing, telecommunication, media, and entertainment. Datacenters,
however, consume a lot of power. Improving datacenter operations is important
and may result in better services, reduced energy consumption and reduced
costs. To improve datacenters, we must understand what is going on inside
them. Therefore, we use operational traces from a scientific cluster in the
Netherlands to investigate and understand how that cluster operates.

Due to work-from-home circumstance, the covid period might have changed our
daily usage of online applications, such as zoom and google meet. In this
research, we focus on the operations of a scientific cluster (LISA) inside the
SURF datacenter. The global pandemic might have changed how the LISA cluster
operates. To understand the change, we collect, combine, and analyze
operational logs from the LISA cluster. The tool to collect the data that
belongs to the non-covid period was accomplished in previous research.
Nonetheless, both the tool and instrument to combine and analyze the traces
are lacking. This research focuses on designing an instrument that can combine
and analyze the traces during and before the coronavirus period. The
instrument can also produce graphs for customarily selected rack, nodes and
periods. Moreover, we characterize the traces that belong to the coronavirus
period using the scientific instrument and additional tools. The outcome of
this research helps us understand how the operations for a scientific cluster
(LISA) in the Netherlands has changed after the global pandemic.

Keywords Datacenter; cluster; instrument; workload; operation; covid; non-
covid;

##  1 Introduction

Datacenters are enormous systems that are designed to host thousands of
computer servers, systems, and networking equipment that offer reliable and
high speed services [7][4]. The digital society and economy is dependent on
these systems to perform complex tasks and store data for future use.
Datacenters, however, consume lots of energy. According to the research from
Uptime Institute [2], the annual power consumption of datacenters is between
400 terawatt-hours(TWh) and 500 TWh. Thus, for a better world, we must reduce
the energy consumption and improve the services of datacenters. To achieve
this, we must know how datacenters work. Unfortunately, knowing how
datacenters operate isn’t trivial, as they are like black boxes and we must
obtain operational traces from the datacenter and investigate those traces in
order to know how the datacenter operates.

In this research, we design and develop a scientific instrument that can
provide insights on the operations of a scientific cluster (LISA) in the
Netherlands. This instrument can also compare two different periods of the
datacenter so we can understand how the datacenter operates in different time
periods. For our research, the two different time periods is the covid and
non-covid periods. From the LISA cluster, we have low-level operational traces
that span approximately 7.5 months. The scientific instrument combines the
low-level traces, performs basic statistical operations on them, and provides
us visual information, mainly graphs, on how the cluster is operating.
Moreover, this instrument is able to produce graphs for two different
scenarios. First is the covid vs non-covid periods, as the instrument gives us
two different graphs comparing the covid and non-covid periods. The second
scenario is that the user can specify the analysis in a custom manner by
passing arguments to the instrument. For instance, a system administrator can
investigate two specific months for certain nodes and receive graphs and
tables that help understand how those nodes operated, see Section 5.5.

In this research, we address the problem of not knowing how a scientific
cluster in the Netherlands operates during the coronavirus period, and the
lack of tools and instruments to find out the operations of the scientific
cluster. Therefore, we design and develop the scientific instrument that can
analyze and compare two different time periods (mainly covid vs non-covid). In
addition to that, to support data engineers and system administrators, the
instrument can provide graphs and tables for node-level or rack-level
operations, see Section 5.5.

The scientific instrument developed in this research is used to address the
problem described in the previous paragraph. The main research question is how
to design an instrument that can analyze the collected traces and
statistically compare them during and before the coronavirus period. From the
main research question, we derive three more research questions and describe
the approach in section 4. This research has two main contributions. The first
contribution is a scientific instrument that can give insight on how the LISA
cluster operated per rack per node levels. The second contribution helps use
understand how a scientific cluster in the Netherlands has operated during and
before the coronavirus period, see Section 4. Using different parameters, the
instrument can produce graphs and tables for specific use cases. We believe
the instrument can be useful to system administrators for performance
evaluation in the past because the instrument works on data that has already
been collected.

##  2 Background Information

SURF infrastructure SURF is the biggest public scientific datacenter in the
Netherlands. This datacenter is mostly used in scientific areas, such as
physics, chemistry, machine learning, and computer systems. This datacenter
has many clusters and one of them is the LISA cluster. Alex et al. has
collected and open-sourced traces from the LISA cluster and they have the
finest granularity [11]. These traces span 3 months and they contain metrics
that are gathered within a 15 second interval from over 300 computer nodes,
where over 50 of the nodes are primarily used for running ML (GPU nodes)
workloads, which are called ML nodes and they mainly utilize their GPUs. The
other 250+ nodes are used more to run Generic (CPU nodes) workloads, which are
called Generic nodes and mainly utilized their CPUs. In this research, we use
these traces with an addition of 4.5 months. So, the date of the traces we use
start from January 1 2020 and end in August 12 2020.

Terms & definitions A datacenter is a giant building that hosts computer
systems and its components. Inside datacenters, there may contain many
clusters. These clusters contain many computers that work together like a
single computer system to accomplish various tasks. To understand how these
clusters and datacenter work, we must collect operational traces from these
systems. The operational traces are high-level or low-level workload data
collected either from the jobs and tasks run or from the utilized resources to
run the jobs and tasks. High-level traces contain job or task level data. In
these high-level traces, there isn’t much information on how the disk, memory,
CPU, or GPU were utilized. We define low-level traces as data that contains
information on how the CPU, GPU, disk, and memory were utilized while running
jobs and task. These traces also contain information on OS-level metrics such
as network traffic and I/O operations.

Updates to LISA New ML nodes were added at the end of February and beginning
March. We will consider this update while interpreting our graphs in Section
5.

##  3 Methodology

###  3.1 Extracting, processing, and analyzing the data

Figure 1: Data extraction, processing, and analysis.

The data used in this research was extracted from the LISA cluster using
Prometheus as the toolkit. The whole workflow of extracting, processing, and
analyzing the data is shown in Figure 1. Prometheus is an open-source toolkit
that is used for systems monitoring and alerting toolkit originally built at
SoundCloud [8]. Moreover, Prometheus extracts relevant metrics from the system
that it’s monitoring and produces the data as raw data in JSON format.
Afterwards, for faster processing, raw data is filtered and converted to
parquet format. Next, the instrument processes the data according to user
inputs (metric, period, rack, nodes). Last but not least, the instrument
produces graphs and tables from the metrics that were processed earlier. Now,
the user can interpret the results and see if there are any interesting
insights on how the cluster performed.

The only part we did in Figure 1 for this research is step 5, 6 and 7. The
other steps were done by another research [11]. To process and analyze the
parquet data, we use Python 3.8.6, Pandas 1.1.0, Matplotlib 3.3.1, and
Argparse 1.1. We first load the parquet data and process it based on user
input. From the processed data, after the analysis, graphs and tables are
produced. There are seven types of graphs, five of them are based on
timestamps. Four of the graphs are daily and hourly graph. Two of daily and
hourly graphs are daily seasonal and daily monthly graphs. The other two are
hourly seasonal and hourly monthly graphs. The last graph is a curve showing
the utilization of the metric across the whole season. Furthermore, the
instrument produces cumulative distribution function graphs to show the
distribution of the metric. For rack analysis, the instrument produces bar
plots and violin plots to depict the distribution of the data. The tables show
basic statistical values of the metric.

###  3.2 The Scientific Instrument

The instrument can produce graphs per node or per rack. If, however, the user
doesn’t pass a custom period and nodes or racks as arguments, all the nodes
are processed by splitting the Generic and ML nodes and the timestamps are
split into covid vs non-covid periods. Based on the user’s desire, the
instrument can produce various graphs for distinct metrics. For instance, the
users of LISA are complaining to the system administrator and they claim that
a rack or node isn’t performing as expected. If system administrator wants to
know the cause of this anomaly, the administrator, can use our instrument and
pass arguments, one being the rack or node that isn’t performing as expected
and the other being the time period when the rack was down, to obtain graphs
and tables on the performance of that rack or node.

##  4 Approach & Contributions

In this section, we describe the approach for each research question and
mention the conceptual contributions.
To address RQ1 we design and develop an instrument that can statistically
compare the operations of a cluster across different time periods. This
instrument can split all the nodes by Generic and ML nodes. To address RQ2, we
use the method of using the scientific instrument and additional tools to
interpret the covid period of the graphs and compare them to the non-covid
period to understand the differences. We can understand how the covid period
has effected the operations of the LISA cluster. To address RQ3, we use the
method of using the scientific instrument and additional tools to produce
curves and violin plots, especially in rack level, for the covid and non-covid
period. We interpret these curves and plots to characterize the operations for
the covid period. The approaches should work because many other research
papers [11][1][3][6][10] use basic statistical methods together with curves,
violin plots and bar plots.

This research makes two main contributions and one potential contribution. The
first contribution is a scientific instrument that can produce graphs for two
different time periods for nodes that are selected based on the user’s desire.
The second contribution is the analysis for the covid and non-covid period. In
terms of load1, power consumption, temperature, and RAM utilization, we saw
that the covid period has effected how the LISA cluster operates. The third
(potential) contribution is that this research encourages other scientists to
collect, combine, and analyze traces from the LISA cluster for the post-covid
period, which can be compared with before and during covid analysis.

##  5 Results and Analysis

The national Dutch Government declared a pandemic in 27 February 2020 [5].
From 1 January 2020 till the end of 26 February 2020, we refer the period as
non-covid. From the beginning of 27 February 2020 to the end of the dataset,
we refer the period as covid.

In this section, we address all the research questions. Our main concern is to
find out if the operations of the LISA cluster have changed during the covid
period. Hence, we compare the general utilization levels, daily & hourly
utilization levels and rack utilization levels for the covid vs non-covid
periods. Moreover, we showcase that any user of the instrument can select a
custom node or rack and obtain graphs for any given time period.

###  5.1 General Utilization Levels

Figure 2: Load1, power consumption, RAM utilization and ambient temperature
value distribution for the covid and non-covid periods across all the nodes.

####  5.1.1 Power consumption

In Figure 2, there seems to be no significant difference between the power
consumption during and before the covid period. The IQR for both periods are
almost same. Nonetheless, the upper outside points for the covid period are
approximately 50W higher than the non-covid period. Furthermore, Figure 12
indicates that all the ML node racks, except for Rack 33, consume 5W to 150W
more average power during the covid period. Figure 13 depict the increase in
power consumption during the covid period for the ML node racks. There is no
significant difference in power consumption for the generic node racks.
Nonetheless, Rack 23, consumes significantly more power, which peaks to
approximately 1200W during the covid period. In Figure 3, the power
consumption for Node 26 in Rack 23 increases from time to time, and peaks to
approximately 1200W during the covid period.

####  5.1.2 Load1

In Figure 13, almost all of the generic node racks have more instances where
load1 values are higher during the covid period. Perhaps working from home has
caused the user’s of LISA (scientist & engineers) to run intense jobs that
hammer certain generic nodes or racks. These jobs might rapidly utilize the
node’s CPU and cause the load1 values to sky rocket for a certain amount of
time.

In Figure 2, both periods contain nodes that have reached up to 23378 load.
This indicates that there were times when the CPU’s were highly utilized, some
nodes were even hammered (see Section 5.3). The box plots for both periods
have almost the same IQR. The upper whisker for both periods reach up to
approximately 30 load. Nevertheless, the median (the white dot) for the covid
period is higher than the non-covid period.

####  5.1.3 Ambient Temperature

The ambient temperature in Figure2 depicts that the temperature for both
periods are mostly 24°C. Nevertheless, the safe temperature for a server to
work efficiently is 25°C. If servers work for a long time in an environment
where the temperature is higher or lower than 25°C, the average failure
probability for servers will be higher [9]. Moreover, in Figure 12, the
average temperatures during the covid period for all ML node racks is 1°C to
3°C higher during the covid period. Out of all the generic node racks, only
Rack 23 has, on average, higher temperature during the covid period. Figure 3
depicts the 2°C average increase during the covid period for Node 26 in Rack
23.

####  5.1.4 RAM Utilization

According to the RAM utilization violin plot in Figure 2, the IQR is 10%
longer during the non-covid period, indicating a higher RAM utilization.
Moreover, the upper outside points for the non-covid period is larger than the
covid period, especially above 90% RAM utilization. This means that there were
more instances during the non-covid period where 90+% RAM was utilized. Also,
most of the generic node racks utilize more RAM during the non-covid period,
see Section 5.3.

Figure 3: Temperature and power consumption for Rack 23 across the whole
period. The vertical line separates the covid and non-covid periods.

###  5.2 Daily and Hourly Utilization Levels

In this section, we investigate the power consumption and load1 values. Each
graph gets the mean of the daily and hourly values, and aggregates them across
all the Generic or ML nodes.

####  5.2.1 Power Consumption

Figure 4: Daily power consumption for the covid and non-covid periods
aggregated across all ML nodes. The maximum PDF value is annotated in the
violin plot.

In Figure 4, the aggregated power consumption during the covid period for ML
nodes is approximately 1000W higher than the non-covid period. There can be
two reasons for this. The first one is that ML nodes were added to the LISA
cluster at the end of February and beginning of March, see Section 2. The
second reason can be that the cluster users have used LISA more during the
covid period and that can lead to higher power consumption.

Figure 5: Daily power consumption for the covid and non-covid periods
aggregated across all generic nodes. The maximum PDF value is annotated in the
violin plot.

According to Figure 5, during the first 4 days of the week, the aggregated
power consumption for the covid period is about 1500W higher than the non-
covid period. After Friday noon, the gap between the two periods increase up
to 2000W because the aggregated power consumption in the non-covid period
decreases. The reason might be that cluster users no longer run jobs in the
weekend because, during the non-covid period, people didn’t work in the
weekend. Perhaps, working from home in the covid period has caused people to
work also during the weekend.

Figure 6: Hourly power consumption for the covid and non-covid periods
aggregated across all ML nodes. The maximum PDF value is annotated in the
violin plot.

For the ML nodes, the curves for both periods are similar in Figure 6. Both
curves decrease during the night after 00:00 and increase after 9:00. Even if
the curves for both periods are similar, throughout the day, the power
consumption is approximately 1000 watts higher during the covid period. The
cause of this can be the new added ML nodes (see Section 2), or the cluster
users utilized the LISA more during the covid period. Both of the reasons can
be the reason for the higher power consumption during the covid period.

Figure 7: Hourly power consumption for the covid and non-covid periods
aggregated across all generic nodes. The maximum PDF value is annotated in the
violin plot.

The power consumption during the covid period is around 1000W higher than the
non-covid period in Figure 7. Since there were no new generic nodes added to
the cluster, the higher power consumption during the covid period can be
caused by higher utilization levels. Moreover, for both periods, the power
consumption for generic nodes don’t vary throughout the day. Perhaps, the
users of LISA run jobs also in the evening and night, or the users run long
jobs that take days to finish.

####  5.2.2 Load

Figure 8: Daily load1 values for the covid and non-covid periods aggregated
across all generic nodes. The maximum PDF value is annotated in the violin
plot.

In Figure 8, during the covid period, aggregated load1 values are as high as
5000, especially during Thursday and Friday. There is only a moments where
aggregated load1 is significantly higher during the non-covid period: Friday
evening. Other than that, the power consumption is higher during the covid
period. Furthermore, during the covid period, aggregated load1 values for
Monday, Tuesday and Wednesday are almost the same as Saturday and Sunday. This
indicates that cluster users run jobs also on weekends due to work from home
policies. There is almost a 1000 load increase for both periods after Thursday
morning and the values remain same throughout Friday. The reason can be that
cluster users are trying to finish up their remaining tasks before the
weekend.

Figure 9: Daily load1 values for the covid and non-covid periods aggregated
across all ML nodes. The maximum PDF value is annotated in the violin plot.

As mentioned in Section 2, there were new ML nodes added at the end of
February and the beginning of March. This didn’t seem to have an effect to the
covid period because load1 values don’t change and they are close to zero,
perhaps no more than 50. In the non-covid period, however, load1 values vary
immensely.

Figure 10: Hourly load1 values for the covid and non-covid periods aggregated
across all ML nodes. The maximum PDF value is annotated in the violin plot.

In Figure 10, the load1 values for ML nodes during the covid period don’t
change throughout the day. The non-covid period is approximately 300 loads
higher than the covid period. The reason might be that ML nodes don’t utilize
their CPU’s much (because they run mainly ML workloads that utilize GPU) and
that can cause the load1 values to remain low.

Figure 11: Hourly load1 values for the covid and non-covid periods aggregated
across all generic nodes. The maximum PDF value is annotated in the violin
plot.

generic nodes mainly run jobs that require CPU usage. Figure 11 depicts that
the generic nodes were utilized more during the covid period. The aggregated
load1 values during the covid period are approximately 1000 loads higher than
the non-covid period. This was almost the same in the daily load1 values for
the generic nodes in Figure 8. We can see that during the covid period, there
is an increase in load1 values around 9:00, which is when people start
working. After 17:00, the load1 values decrease for the covid period and
increase for the non-covid period. The cause might be that the cluster users
run their last jobs right before they leave the office, and those jobs took
around 3 hours to finish (after 20:00 the load1 values for the non-covid
period decrease).

###  5.3 Rack Utilization levels

In this section, we dive deeper into the cluster, as we investigate the
periods in rack level. We compare all the racks between the covid and non-
covid periods.

Figure 12: Average RAM utilization, load1, power consumption and ambient
temperature values for each rack. The left side of the vertical line are the
generic racks. Figure 13: RAM utilization, load1, power consumption and
ambient temperature value distribution for each rack. The left side of the
green vertical line are the generic racks.

####  5.3.1 Power consumption

In Figure 12, there isn’t a significant difference in power consumption for
the generic node racks. Nonetheless, according to Figure 13, Rack 23 consume
significantly more power during the covid period. The bar plots, however,
don’t depict Rack 23 much higher during the covid period because the average
power consumption for Rack 23 remains the same for both periods. Nonetheless,
during the covid period, the error bar for Rack 23 have a higher value, which
means there are more outliers for that rack. This indicates that there were
instances where up to 1200W power was consumed by Rack 23 during the covid
period. The other generic node racks consume approximately 200W for both
periods.

In Figure 13, the ML node racks 30, 31, 32 and 34 consume more power during
the covid period. Rack 33 consumes up to 1400W for both periods, which also
indicates that ML node racks consume around 1000W-1200W more power than the
generic node racks, where Rack 23 is an exception. In Figure 12, the ML node
racks indicate that, on average, all the ML node racks, excluding r33, consume
more power during the covid period. Moreover, these rack utilization levels
reflect the results in Section 5.1 and 5.2, as the general, daily and hourly
power consumption utilization levels were higher during the covid period,
especially for ML node racks. Therefore, the power consumption during the
covid period is higher than the non-covid period.

####  5.3.2 Load1

Figure 12 depicts that during the covid period, the generic node racks have
higher average load1 values. In Figure 13, Rack 11, 12, 25 and 26 have moments
where their load values were above 19500 during the covid period. This isn’t
same during the non-covid, as there is only Rack 23 that contains at least one
load value over 19500. The bar plots depict that, during the covid period, all
the generic node racks, except for Rack 23, have higher load1 values. During
the non-covid period, the violin plots depict that Rack 23 has the highest
load1 value (27885). The bar plots show that, for the same rack and period,
the average load1 value for Rack 23 is above 20 load and the standard
deviation is 607.2.

The box plots for load1 depict that all the ML node racks, except for Rack 30
which has 7543 load as the highest load for the non-covid period, have higher
load1 values during the covid period. Rack 30, 32 and 33 have moments where
their load values were above 1000 during the covid period. For nodes that run
primarily ML workloads, 1000 load is and extremely high load value because ML
nodes mainly utilize their GPUs. Moreover, the violin plots depict that among
all the racks, only the Racks 23, 27 and 30 have higher load1 values during
the non-covid period. All the other racks have higher load1 values during the
covid period. Perhaps, the reason for this is that users of the LISA cluster
utilize the nodes more during the covid period.

####  5.3.3 Ambient Temperature

In Figure 12, all the generic node racks, except for Rack 23, have an average
temperature close to 25°C for both periods. Rack 23 has 1°C to 2°C higher
average temperature during the covid period. All the ML node racks have 1°C to
3°C higher average temperature during the covid period, especially Rack 31. In
Figure 13, all the ML node racks have significant outliers, which means high
standard deviation, for both periods. Nonetheless, the generic node racks
don’t contain much outliers, indicating a low standard deviation. In general,
the ML node racks have higher temperature from the generic node racks for both
periods. The covid period for the ML nodes racks is hotter than their non-
covid period.

####  5.3.4 RAM Utilization

In Figure 12, during the non-covid period, 8 Generic and 2 ML node racks
approximately have 1% to 5% higher RAM utilization than the covid period. Rack
23 is the only Generic rack that has, on average, utilized (3%) more RAM
during the covid period. Across the whole cluster, the highest average RAM
utilization occurs during the covid period by Rack 32. This rack has utilized
its RAM on average 30%, with a standard deviation exceeding 50%. All the
nodes, except for Rack 32, contain lots of outliers, see Figure 13. This
indicates a high standard deviation. The error bars in Figure 12 depict that
the RAM utilization in the LISA cluster has approximately 20% standard
deviation, meaning that the RAMs were utilized inconsistently high.

###  5.4 Overview of the COVID-19 period

Overall, the covid period has caused some changes in the operations of the
LISA cluster. The load1 values were significantly higher for certain racks
during the covid period. Most of the racks consumed almost the same amount of
power during the covid. Nonetheless, the power consumption during the covid
period was slightly higher. Almost all of the generic node racks and some of
the ML node racks have utilized their RAM more during the non-covid period.
Average RAM utilization for all the racks was 10% to 30% and the standard
deviation was about 20%. The average ambient temperature for all the generic
node racks, except Rack 23, were the same for both periods. All the ML node
racks have 1°C to 3°C higher temperature during the covid period.

###  5.5 Custom rack/node utilization levels

In this section, we explain the custom features of the scientific instrument.
The instrument can produce graphs according to selected custom nodes, racks
and periods, see Section 3.2. The user of the instrument can specify a node or
nodes or a rack, and a custom period. If a custom period isn’t specified, the
instrument will take the default periods (covid and non-covid). A data
engineer or system administrator can use this instrument to delve deeper into
the cluster to see how each rack or node is performing. For instance, if a
system administrator wants to investigate node r30n1 from 1 March 2020 to 1
May 2020 for load1, the administrator can input the parameters and the
instrument produces graphs like Figure 14. If that administrator also wants to
compare load1 for the same node with a different time period, the
administrator can input the required parameters and the instrument produces
graph similar graphs. The administrator can compare the performance of node
r30n1 for two different time periods. The same comparison can be done for
multiple nodes, one rack, or all nodes.

Figure 14: Daily, hourly, and seasonal load1 values for node r30n1.

##  6 Threats to Validity

The extracted dataset has some deficiencies, Prometheus might contain bugs.
Prometheus monitoring system is one of the most popular system-monitoring
toolkit. This system is used by many scientists and engineers and we expect
Prometheus to produce correct results. Prometheus, however, can contain bugs
that affect the data collection mechanism. For instance, in the metric that
contains values for load1, some of the ML nodes are missing. Only %52.7 of the
ML nodes are in the metric. The rest of the nodes and their values are
lacking. There may be more missing nodes in various other metrics that we
haven’t investigated.

The calculations while processing the data is wrong. Even if we have checked
the codes that are used to process the data, miscalculations might exist in
the instrument and other script that we have made.

ML workloads might differ from each other. With the experience of the
engineers at SURF, we label nodes in the LISA cluster having GPUs as machine
learning nodes, because 90+% of the workload belongs to the machine learning
domain. The workload is run by various users, mainly scientists, and the ML
workloads might be different from each other.

##  7 Related Work

There are several related works in the field for understanding how datacenters
and other computer systems work. Although, there isn’t any research on
comparing datacenter operations during and before the coronavirus period. A
similar study to our research is conducted by Uta et al. as the lack of open-
source operational traces of low-level metrics is mitigated. The absence of
such logs force large-scale systems experts and infrastructure developers to
design, and implement their systems using unverified assumptions. The
operational traces mitigates this issue and provides important information to
system administrators and researchers on OS-level operations of datacenters.
Furthermore, the traces enable performance evaluation, together with
information on the meaning of the datacenter operations.

There are several related work conducted by experts in the field of workload
data. Feitelson [3] points out to the importance of workload data for
evaluating performance in computer systems. The PWA contains 22 different
logs, collected since 1993, and they contain accounting data about the jobs
that executed on parallel supercomputers, clusters, and grids. The study also
mentions about the quality issues of logs regarding missing data, inconsistent
data, erroneous data an so on.

Figure 15: Accumulated yearly number of hits received when searching for the
Parallel Workloads Archive (PWA) and the Grid Workloads Archive (GWA) in
Google Scholar as of 28 October 2013 [3]

According to Figure 15, the PWA partially aids the need for workload data in
the sense that the PWA has been cited more than 800 times since it was first
open-sourced. In addition, both the GWA [6] and PWA have been cited more than
1100 times, which further proves the value of such data. However, due to
certain regulations, the amount of workload traces are scarce and to address
this issue, Iosup has published the GWA. The initial user group consists of
researchers, industry, and education. In addition, for non-expert users, GWA
has a mechanism for automated trace ranking and selection system so those
users can also benefit and use the data for various purposes.

Figure 16: An overview of the Grid Workloads Archive design. The arrows
represent the direction of data flows. [6]

In Figure 16, Iosup et al. shows the design requirements of the GWA. Box
number one represents tools for collecting grid workloads, boxes with number 2
represent the tools for processing grid workload, Boxes with number 3
represent the tools for using grid workloads, and boxes with number 4
represent the tools for sharing grid workloads. After sharing grid workloads,
they can be used for performance evaluations in computer systems, which is
represented in box number 5. We can see that the GWA is a massive contribution
to the computer systems community. It enables for unbiased research using
real-world data. The results obtained from the data can be used to enhance
grid computing and computer systems in general.

Amvrosiadis et al. states that the main goal is to point out that most of the
prior work, more than 450 publications, on trace analysis and characterization
were done using the Google cluster traces [12]. To generalize the analysis and
provide new traces, Amvrosiadis makes two contributions. First, he releases
four traces: two from the private cluster Two Sigma, and two from HPC clusters
located at the Los Alamos National Laboratory. Second, he examines the
generality of workload characteristics obtained from the Google cluster trace.
The findings in his research is that conducting new research using the Google
trace alone is insufficient to guarantee generality. Therefore, this research
is imperative in the sense that workload generality is important to understand
how different clusters work.

Figure 17: A dashboard to visualize 10 metrics for a single GPU-enabled server
in the LISA cluster. Each metric is normalized by the maximum value
encountered during the 3 months recorded for this server [11].

Moreover, Uta et al. mentions that many analysis can be made using the open-
sourced low-level traces. For instance, in Figure 17, we can see a typical
sysadmin dashboard from utilization-level metrics. Sysadmins can use these
metrics to pinpoint an error or an abnormal situation in the system. Moreover,
Uta advocates that using the operation logs, we can make systems more
efficient, understand how certain workloads interact with the hardware,
originate the errors, and gain deeper insights on the working of large-scale
systems (datacenters). We can see that low-level operational logs important
for various stakeholders. The main contribution Uta makes is open-sourcing
fine-grained low-level operational logs spanning 3 months. Another
contribution is a schema used for the collection of the datacenter metrics.
This section demonstrates the importance of operational traces. Using the
right traces, we can understand how a cluster or the whole datacenter works.
The operations of a cluster can tell us a lot about what can we improve.

##  8 Conclusion and Future Work

The coronavirus crisis has changed the way we work. Before the covid period,
people mostly went to their offices work. During the global pandemic, people
work from home and use online resources like zoom and google meet more. With
the increased demand for datacenters, the operations of clusters might have
changed during the covid period. In this research, we analyzed traces from a
scientific cluster (LISA) in the SURF datacenter. The process of extracting
and analyzing these traces is explained in Section 3. We designed and
developed a scientific instrument that can statistically compare two different
time periods. The instrument produced graphs for the covid and non-covid
period. We have seen that the power consumption, CPU usage, and temperature
has increased during the covid period. The ram utilization was higher during
the non-covid period. Moreover, the daily and hourly graphs showed us that the
load1 values and power consumption has increased for the Generic nodes. The
graphs that contain rack level utilization depicted that almost all the racks
consumed more power, had higher temperature, and utilized their CPU more
during the covid period. The instrument also produced node level curves with
specified custom periods, which gives system administrators and data engineers
the freedom to investigate any node for any given period.

The traces we used in this research are the most fine-grained traces so far.
Future work can further analyze these traces. Hopefully, after the covid
period ends, traces can be extracted and analyzed comparing the pre-covid,
covid and post-covid periods.

##  Appendix A Problem Statement

Due to coronavirus, many countries have made regulations on traveling and
lockdowns have been issued. After the coronavirus lockdowns, the demand for
services(e.g., video conferencing apps, social media, educational platforms,
online gaming, streaming media) offered by datacenters has arisen and this has
affected the operations of datacenters in an unprecedented way. Consequently,
during the coronavirus period, the datacenter operations may have changed in a
way that pressured datacenters due to intense workloads and understanding the
change is in interest.

In order to understand the change and compare the difference of workloads, we
need empirical, real-world data which is unavailable at the moment. Thus, we
first need a tool to collect such data. Afterwards, we need to combine the
collected data for meaningful analysis. Hence, a well designed instrument is
required to accomplish all of the tasks (collect-combine-analyze workload
data). Luckily, Uta et al. [11] has collected and open-sourced workload data
from SURFsara that has the finest-granularity out of all the other traces
published so far. However, the instrument for combining and analyzing workload
data is lacking. Hence, we need an instrument that can combine and analyze the
collected workload data. The tool and instrument to analyze the datacenter
workload are required to find out how datacenter operations have changed
regarding workloads. To find the meaning of datacenter workloads in certain
time scales(periods), an instrument will be designed that statistically
compares the workload of a datacenter across different time periods. Using the
designed instrument, we will check if there are any significant differences
between the datacenter operations pre and during the coronavirus period.

##  Appendix B Self-Reflection

This research has given me the opportunity learn how datacenters work in terms
of low-level operations. I learned how to process and analyze data. I learned
how to design and develop a scientific instrument. I worked a lot on this
research project, mostly part-time. Most of the time was spent on developing
the instrument. Obtaining meaningful results and comparing graphs, tables were
also challenging. Together with interpreting the selected graphs, selecting
those graphs for different metrics was also time consuming. A significant
amount of time was also spent on reading other related papers on datacenter
workload characterization. In the last phase of this research, I got the
coronavirus and I was sick for 3-4 days. Perhaps, the covid period didn’t like
being analyzed.

## References

  * [1] G. Amvrosiadis, J. W. Park, G. R. Ganger, G. A. Gibson, E. Baseman, and N. DeBardeleben.  On the diversity of cluster workloads and its impact on research results.  In 2018 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 18), pages 533–546, 2018.
  * [2] R. Bashroush and A. Lawrence.  Beyond pue: tackling it’s wasted terawatts.  Uptime Institute Available from: https://uptimeinstitute. com/beyond-pue-tackling-it’s-wasted-terawatts [Accessed 11 June 2020], 2020\.
  * [3] D. G. Feitelson, D. Tsafrir, and D. Krakov.  Experience with using the parallel workloads archive.  Journal of Parallel and Distributed Computing, 74(10):2967–2982, 2014.
  * [4] F. Gens.  Worldwide and regional public it cloud services 2014–2018 forecast.  IDC, October, 2014.
  * [5] D. Government.  Ontwikkeling COVID-19 in grafieken.  https://www.rivm.nl/coronavirus-covid-19/grafieken, 2020.  [Online; accessed 19-December-2020].
  * [6] A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, and D. H. Epema.  The grid workloads archive.  Future Generation Computer Systems, 24(7):672–686, 2008.
  * [7] A. Iosup, A. Uta, L. Versluis, G. Andreadis, E. van Eyk, T. Hegeman, S. Talluri, V. van Beek, and L. Toader.  Massivizing computer systems: a vision to understand, design, and engineer computer ecosystems through and beyond modern distributed systems.  CoRR, abs/1802.05465, 2018.
  * [8] open source.  Prometheus description.  https://prometheus.io/docs/introduction/overview/.  Accessed: 2020-11-26.
  * [9] M. Pedram.  Energy-efficient datacenters.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 31(10):1465–1484, 2012.
  * [10] S. Shen, V. van Beek, and A. Iosup.  Statistical characterization of business-critical workloads hosted in cloud datacenters.  In 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pages 465–474. IEEE, 2015.
  * [11] A. Uta, K. Laursen, A. Uta, A. Iosup, P. Melis, D. Podareanu, and V. Codreanu.  Beneath the surface: An mri-like view into the life of a 21st century datacenter.  USENIX, June 2020.
  * [12] J. Wilkes.  More google cluster data.  Google research blog, Nov, 2011.


# The Fewer Splits are Better: Deconstructing Readability in Sentence
Splitting

Tadashi Nomoto
National Institute of Japanese Literature
Tachikawa, Tokyo 190-0014, Japan
<EMAIL_ADDRESS>

###### Abstract

In this work, we focus on sentence splitting, a subfield of text
simplification, motivated largely by an unproven idea that if you divide a
sentence in pieces, it should become easier to understand. Our primary goal in
this paper is to find out whether this is true. In particular, we ask, does it
matter whether we break a sentence into two or three? We report on our
findings based on Amazon Mechanical Turk.

More specifically, we introduce a Bayesian modeling framework to further
investigate to what degree a particular way of splitting the complex sentence
affects readability, along with a number of other parameters adopted from
diverse perspectives, including clinical linguistics, and cognitive
linguistics. The Bayesian modeling experiment provides clear evidence that
bisecting the sentence leads to enhanced readability to a degree greater than
what we create by trisection.

MnLargeSymbols’164 MnLargeSymbols’171

The Fewer Splits are Better: Deconstructing Readability in Sentence Splitting

Tadashi Nomoto National Institute of Japanese Literature Tachikawa, Tokyo
190-0014, Japan<EMAIL_ADDRESS>

##  1 Introduction

In text simplification, one question people often fail to ask is, whether the
technology they are driving truly helps people better understand texts. This
curious indifference may reflect the tacit recognition of the partiality of
datasets covered by the studies Xu et al. (2015) or some murkiness that
surrounds the goal of text simplification.

As a way to address the situation, we examine a role of simplification in text
readability, with a particular focus on sentence splitting. The goal of
sentence splitting is to break a sentence into small pieces in a way that they
collectively preserve the original meaning. A primary question we ask in this
paper is, does a splitting of text affect readability? In the face of a large
effort spent in the past on sentence splitting, it comes as a surprise that
none of the studies put this question directly to people; in most cases, they
ended up asking whether generated texts ‘looked simpler’ than the original
unmodified versions Zhang and Lapata (2017), which of course does not say much
about their readability. We are not even sure whether there was any agreement
among people on what constituted simplification.

Another related question is, how many pieces should we break a sentence into?
Two, three, or more? In the paper, we focus on a particular setting where we
ask whether there is any difference in readability between two- and three-
sentence splits. We also report on how good or bad sentence splits are that
are generated by a fine-tuned language model, compared to humans’.

A general strategy we follow in the paper is to elicit judgments from people
on whether simplification made a text anyway readable for them (Section 4),
and do a Bayesian analysis of their responses to identify factors that may
have influenced their decisions (Section 5).111We will make available on
GitHub the data we created for the study soon after the paper’s publication
(they should be found under https://github.com/tnomoto).

##  2 Related Work

Historically, there have been extensive efforts in ESL (English as a Second
Language) to explore the use of simplification as a way to improve reading
performance of L2 (second language) students. Crossley et al. (2014) presented
an array of evidence showing that simplifying text did lead to an improved
text comprehension by L2 learners as measured by reading time and and accuracy
of their responses to associated questions. They also noticed that simple
texts had less lexical diversity, greater word overlap, greater semantic
similarity among sentences than more complicated texts. Crossley et al. (2011)
argued for the importance of cohesiveness as a factor to influence the
readability. Meanwhile, an elaborative modification of text was found to play
a role in enhancing readability, which involves adding information to make the
language less ambiguous and rhetorically more explicit. Ross et al. (1991)
reported that despite the fact that it made a text longer, the elaborative
manipulation of a text produced positive results, with L2 students scoring
higher in comprehension questions on modified texts than on the original
unmodified versions.

While there have been concerted efforts in the past in the NLP community to
develop metrics and corpora purported to serve studies in simplification Zhang
and Lapata (2017); Sulem et al. (2018a); Narayan et al. (2017); Botha et al.
(2018); Niklaus et al. (2019); Kim et al. (2021); Xu et al. (2015), they fell
far short of addressing how their work contributes to improving the text
comprehensibility by readers. Part of our goal is to break away from a
prevailing view that relegates the readability to a sideline.

##  3 Method

The data come from two sources, the Split and Rephrase Benchmark (v1.0) (SRB,
henceforth) Narayan et al. (2017) and WikiSplit Botha et al. (2018). SRB
consists of complex sentences aligned with a set of multi-sentence
simplifications varying in size from two to four. WikiSplit follows a similar
format except that each complex sentence is accompanied only by a two-sentence
simiplification.222 We used WikiSplit, together with part of SRB, exclusively
to fine tune BART to give a single split (bipartite) simplification model, and
SRB to develop test data to be administered to humans for linguistic
assessments. SRB was derived from WebNLG Gardent et al. (2017) by making use
of RDFs associated with textual snippets to assemble simplifications.  We
asked Amazon Mechanical Turk workers (Turkers, henceforth) to score
simplifications on linguistic qualities as well as to indicate whether they
have any preference between two-sentence and three-sentence versions in terms
of readability.

We randomly sampled a portion of SRB, creating test data (call it $\cal H$),
which consisted of triplets of the form: $\langle S_{0},A_{0},B_{0}\rangle$,
$\dots$, $\langle S_{i},A_{i},B_{i}\rangle$, $\dots$, $\langle
S_{m},A_{m},B_{m}\rangle$, where $S_{i}$ is a complex sentence, $A_{i}$ a
corresponding two-sentence simplification, and $B_{i}$ its three-sentence
version. While $A$ alternates between versions created by BART and by human,
$B$ deals only with manual simplifications.333 HSplit Sulem et al. (2018a) is
another dataset (based on Zhang and Lapata (2017)) that gives multi-split
simplifications. We did not adopt it here as the data came with only 359
sentences with limited variations in splitting.  See Table 1 for a further
explanation.

Separately, we extracted from WikiSplit and SRB, another dataset $\cal B$
consisting of complex sentences as a source and two-sentence simplifications
as a target (Table 2) i.e. ${\cal B}=\\{\langle
S^{\prime}_{0},A^{\prime}_{0}\rangle$, $\dots$, $\langle
S^{\prime}_{n},A^{\prime}_{n}\rangle\\}$, to use it to fine-tune a language
model (BART-large).444https://huggingface.co/facebook/bart-large The fine-
tuning was done using a code available at GitHub.555
https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/run_translation.py

| bart | hum
---|---|---
a (two-sentence split) | 113 | 108
b (three-sentence split) | $-$ | 221
Table 1: A break down of ${\cal H}$. 113 of them are of type A (bipartite split) generated by BART-large; 108 are of type A created by humans. There were 221 of type B (tripartite split), all of which were produced by humans.  train | dev
---|---
1,135,009 (989,944) | 13,797(5,000)
Table 2: A training setup for BART. The data comes from SRB Narayan et al. (2017) and WikiSplit Botha et al. (2018). The parenthetical numbers indicate amounts of data that originate in WikiSplit Botha et al. (2018). Figure 1: A screen capture of HIT. This is what a Turker would be looking at when taking the test. question | available choices |
---|---|---
| s | bart-a | hum-b | not sure | total
$\llangle$s, bart-a$\rrangle_{|q}$ | 254 (0.32) | 527 (0.67) | – | 10 (0.01) | 791
$\llangle$s, hum-b$\rrangle_{|q}$ | 290 (0.37) | – | 490 (0.62) | 11 (0.01) | 791
| s | hum-a | hum-b | not sure | total
$\llangle$s, hum-a$\rrangle_{|q}$ | 253 (0.33) | 494 (0.65) | – | 9 (0.01) | 756
$\llangle$s, hum-b$\rrangle_{|q}$ | 288 (0.38) | – | 463 (0.61) | 5 (0.01) | 756
Table 3: Results from the Comparison Section. We are showing how many Turkers went with each available choice. S: source. BART-A: BART-generated two-sentence simplification. HUM-A: manual two-sentence simplification. HUM-B: manual three-sentence simplification. $\llangle$S, BART-A$\rrangle_{|q}$ asked Turkers which of S and BART-A they found easier to understand. 67% said they would favor BART-A, and 32% S, with 1% not sure. $\llangle$S, HUM-B$\rrangle_{|q}$ compares S and HUM-B for readability. $\llangle$S, HUM-A$\rrangle_{|q}$ looks at S and HUM-A.  question | available choices |
---|---|---
| bart-a | hum-b | not sure | total
$\llangle$bart-a, hum-b$\rrangle_{|q}$ | 460 (0.58) | 316 (0.40) | 15 (0.02) | 791
| hum-a | hum-b | not sure | total
$\llangle$hum-a, hum-b$\rrangle_{|q}$ | 439 (0.58) | 301 (0.40) | 16 (0.02) | 756
Table 4: Comparison of two- vs three-sentence simplifications. The majority
went with two-sentence simplifications regardless of how they were generated.

A task (or a HIT in Amazon’s parlance) we asked Turkers to do was to work on a
three-part language quiz. The initial problem section introduced a worker to
three short texts, corresponding to a triplet $\langle
S_{i},A_{i},B_{i}\rangle$; the second section asked about linguistic qualities
of $A_{i}$ and $B_{i}$ along three dimensions, meaning, grammar, and fluency;
and in the third, we asked two comparison questions: (1) whether $A_{i}$ and
$B_{i}$ are more readable than $S_{i}$, and (2) which of $A_{i}$ and $B_{i}$
is easier to understand.

Figure 1 gives a screen capture of an initial section of the task. Shown Under
Source is a complex sentence or $S_{i}$ for some $i$ . Text A and Text B
correspond to $A_{i}$ and $B_{i}$, which were displayed in a random order.

In total, there were 221 HITs (Table 1), each administered to seven people.
All of the participants were self-reported native speakers of English with a
degree from college or above. The participation was limited to residents in
US, Canda, UK, Australia, and New Zealand.

##  4 Preliminary Analysis

Table 3 summarizes results from comparison questions. A question, labelled
$\llangle$S, BART-A$\rrangle_{|q}$, asks a Turker, which of Source and BART-A
he or she finds easier to understand, where BART-A is a BART generated two-
sentence simplification. We had 791 (113$\times$7) responses, out of which 32%
said they preferred Source, 67% liked BART better, and 1% replied they were
not sure. Another question, labelled $\llangle$S, HUM-A$\rrangle_{|q}$,
compares Source to HUM-A, a two-sentence split by human. It got 756 responses
(108$\times$7). The result is generally parallel to $\llangle$S,
BART-A$\rrangle_{|q}$. The majority of people favored a two-sentence split
over a complex sentence. The fact that three sentence versions are also
favored over complex sentences suggests that breaking up a complex sentence
improves readability, regardless of how many pieces it ends up with.

category | hum-a | hum-b
---|---|---
*fluency | 4.04 (0.39) | 3.75 (0.38)
grammar | 4.12 (0.32) | 4.10 (0.32)
meaning | 4.31 (0.36) | 4.33 (0.28)
Table 5: Average scores and standard deviations for HUM-A and HUM-B. HUM-A is
more fluent than HUM-B. Note: ** = $p<0.01$.

Table 4 gives a tally of responses to comparison questions on two- and three-
sentence splits. More people voted for bipartite over tripartite
simplifications. Tables 5 and 6 show scores on fluency, grammar, and meaning
retention of simplifications, comparing BART-A and HUM-B,666 As Tables 5 and 6
indicate, BART-A is generally comparable to HUM-A in the quality of its
outputs, suggesting that what it generates is mostly indistinguishable from
those by humans.  on one hand, and HUM-A and HUM-S, on another, on a scale of
1 (poor) to 5 (excellent). In either case, we did not see much divergence
between A and B in grammar and meaning, but they diverged the most in fluency.
A T-test found the divergence statistically significant. Two-sentence
simplifications generally scored higher on fluency (over 4.0) than three
sentence counterparts (below 4.0). Table 7 gives an example showing what
generated texts looked like in BART-A and HUM-A/B.

category | bart-a | hum-b
---|---|---
*fluency | 4.04 (0.37) | 3.72 (0.36)
grammar | 4.07 (0.30) | 4.05 (0.34)
meaning | 4.21 (0.38) | 4.25 (0.35)
Table 6: Average scores and standard deviations of BART-A and the
corresponding HUM-B. BART-A is significantly more fluent than HUM-B. ‘**’
indicates the two groups are distinct at the 0.01 level.

##  5 A Bayesian Perspective

type |  text
---|---
original |  The Alderney Airport serves the island of Alderney and its 1st runway is surfaced with poaceae and has a 497 meters long runway .
bart-a |  Alderney Airport serves the island of Alderney . The 1st runway at Aarney Airport is surfaced with poaceae and has 497 meters long .
hum-a |  The runway length of Alderney Airport is 497.0 and the 1st runway has a poaceae surface . The Alderney Airport serves Alderney .
hum-b |  The surface of the 1st runway at Alderney airport is poaceae . Alderney Airport has a runway length of 497.0 . The Alderney Airport serves Alderney .
Table 7: Original vs. Modified

A question we are curious about at this point is what are the factors that led
Turkers to decisions that they made. We answer the question by way of building
a Bayesian model based on predictors assembled from the past literature on
readability and in related fields.

###  5.1 Model

We consider a Bayesian logistic regression.777 Equally useful in explaining
relationships between potential causes and the outcome are Bayesian tree-based
methods Chipman et al. (2010); Linero (2017); Nuti et al. (2019), which we do
not explore here. The latter could become a viable choice when an extensive
non-linearity exists between predictors and the outcome.

$\begin{split}Y_{j}&\backsim Ber(\lambda),\\\
\text{logit}(\lambda)&=\beta_{0}+\sum_{i}^{m}\beta_{i}X_{i},\\\
\beta_{i}&\backsim{\cal N}(0,\sigma_{i})\;\;(0\leq i\leq m)\\\ \end{split}$
(1)

$Ber(\lambda)$ is a Bernoulli distribution with a parameter $\lambda$.
$\beta_{i}$ represents a coefficient tied to a random variable (predictor)
$X_{i}$, where $\beta_{0}$ is an intercept. We assume that $\beta_{i}$,
including the intercept, follows a normal distribution with the mean at $0$
and the variance at $\sigma_{i}$. $Y_{i}$ takes either 1 or 0. $Y=1$ if a
Turker finds a two-sentence simplification more readable, and $Y=0$ if a
three-sentence version is preferred.

category | var name |  description | value
---|---|---|---
synthetic | bart |  true if the simplification is generated by BART; false otherwise. | categorical
cohesion | ted1 |  the tree edit distance (TED) between a source and its proposed simplification.888https://github.com/jasonyux/FastKASSIM where TED represents the number of editing operations (insert, delete, replace) required to turn one parse tree into another; the greater the number, the less the similarity Boghrati et al. (2018); Zhang and Shasha (1989). | continuous
ted2 |  TED across sentences contained in the simplification. | continuous
subset |  Subset based Tree Kernel Collins and Duffy (2002); Moschitti (2006); Chen et al. (2022)footnote 8 | continuous
subtree |  Subtree based Tree Kernel Collins and Duffy (2002); Moschitti (2006); Chen et al. (2022)footnote 8 | continuous
overlap |  Szymkiewicz-Simpson coefficient, a normalized cardinality of an intersection of two sets of words Vijaymeena and Kavitha (2016).999https://github.com/luozhouyang/python-string-similarity | continuous
cognitive | frazier |  the distance from a terminal to the root or the first ancestor that occurs leftmost Frazier (1985). | continuous
yngve |  per-token count of non-terminals that occur to the right of a word in a derivation tree Yngve (1960). | continuous
dep length |  per-token count of dependencies in a parse Magerman (1995); Roark et al. (2007). | continuous
tnodes |  per-token count of nodes in a parse tree Roark et al. (2007) | continuous
classic | dale |  Dale-Chall readability score Chall and Dale (1995)101010https://github.com/shivam5992/textstat | continuous
ease |  Flesch Reading Ease Flesch (1979)footnote 10 | continuous
fk grade |  Flesch-Kincaid Grade Level Kincaid et al. (1975)footnote 10 | continuous
perception | grammar |  grammatical integrity (manually coded) | continuous
meaning |  semantic fidelity (manually coded) | continuous
fluency |  language naturalness (manually coded) | continuous
structural | split |  true if the sentence is bisected; false otherwise. | categorical
informational | samsa |  measures how much of the original content is preserved in the target Sulem et al. (2018b). | continuous
Table 8: Predictors

###  5.2 Predictors

We use predictors shown in Table 8. They come in six categories: synthetic,
cohesion, cognitive, classic, perception and structural. A synthetic feature
indicates whether the simplification was created with BART or not, taking true
if it was and false otherwise. Those found under cohesion are our adaptions of
SYNSTRUT and CRFCWO, which are among the diverse features McNamara et al.
(2014) created to measure cohesion across sentences. SYSTRUCT gauges the
uniformity and consistency across sentences by looking at their syntactic
similarities, or by counting nodes in a common subgraph shared by neighboring
sentences. We substituted SYSTRUCT with tree edit distance Boghrati et al.
(2018), as it allows us to handle multiple subgraphs, in contrast to SYSTRUCT,
which only looks for a single common subgraph. CRFCWO gives a normalized count
of tokens found in common between two neighboring sentences. We emulated it
here with the Szymkiewicz-Simpson coefficient, given as $O(X,Y)=\frac{\lvert
X\cap Y\rvert}{\min(\lvert X\rvert,\lvert Y\rvert)}$.

Predictors in the cognitive class are taken from works in clinical and
cognitive linguistics Roark et al. (2007); Boghrati et al. (2018). They
reflect various approaches to measuring the cognitive complexity of a
sentence. For example, yngve scoring defines a cognitive demand of a word as
the number of non-terminals to its right in a derivation rule that are yet to
be processed.

####  5.2.1  yngve

Consider Figure 2. yngve gives every edge in the parse a number reflecting its
cognitive cost. NP gets ‘1’ because it has a sister node VP to its right. The
cognitive cost of a word is defined as the sum of numbers on a path from the
root to the word. In Figure 2, ‘Vanya’ would get $1+0+0=1$, whereas ‘home’
$0$. Averaging words’ costs gives us an Yngve complexity.

Figure 2: Yngve scoring

####  5.2.2 frazier

frazier scoring views the syntactic depth of a word (the distance from a leaf
to a first ancestor that occurs leftmost in a derivation rule) as a most
important factor to determining the sentence complexity. If we run frazier on
the sentence in Figure 2, it will get the score like one shown in Figure 3.
‘Vanya’ gets $1+1.5=2.5$, ‘walks’ $1$ and ‘home’ $0$ (which has no leftmost
ancestor).

Figure 3: Frazier scoring

Roark et al. (2007) reported that both yngve and frazier worked well in
discriminating subjects with mild memory impairement.

####  5.2.3 dep length

dep length (dependency length) and tnodes (tree nodes) are also among the
features that Roark et al. (2007) found effective. The former measures the
number of dependencies in a dependency parse, and the latter the number of
nodes in a phrase structure tree.

####  5.2.4 subset and subtree

subset and subtree are both measures based on the idea of Tree Kernel Collins
and Duffy (2002); Moschitti (2006); Chen et al. (2022).111111 Tree Kernel is a
function defined as $K(T_{1},T_{2})=\sum_{n_{1}\in N(T_{1})}\sum_{n_{2}\in
N(T_{2})}\Delta(n_{1},n_{2})$ where
$\Delta(a,b)=\left\\{\begin{array}[]{ll}0&\mbox{if $a\neq b$};\\\ 1&\mbox{if
$a=b$};\\\
\prod^{C(a)}_{i}(\sigma+\Delta(c^{(i)}_{a},c^{(i)}_{b}))&\mbox{otherwise}.\end{array}\right.$
$C(a)$ = the number of children of $a$, $c^{(i)}_{a}$ represents the $i$-th
child of $a$. We let $\sigma>0$.  The former considers how many subgraphs two
parses share, while the latter how many subtrees. Note that subtrees are those
structures that end with terminal nodes.

####  5.2.5 Classic readability features

We also included features that have long been established in the readability
literature as standard, i.e. Dale-Chall Readability, Flesch Reading Ease, and
Flesch-Kincaid Grade Level Chall and Dale (1995); Flesch (1979); Kincaid et
al. (1975).

####  5.2.6 Perceptual features

Those found in the perception category are from judgments Turkers made on the
quality of simplifications we asked them to evaluate. We did not provide any
specific definition or instruction as to what constitutes grammaticality,
meaning, and fluency during the task. So, it is most likely that their
responses were spontaneous and perceptual.

####  5.2.7 split and samsa

Finally, we have split, which records whether or not the simplification is
bipartite: it takes true if it is, and false if not. samsa is a recent
addition to a battery of simplification metrics, which looks at how much of a
propositional content in the source remains after a sentence is split Sulem et
al. (2018b). (The greater, the better.) We standardized all of the features,
except for bart and split, by turning them into z-scores, where
$z=\frac{x-\bar{x}}{\sigma}$.

###  5.3 Evaluation

We trained the model (Eqn. 1) using bambi Capretto et al.
(2020),121212https://bambinos.github.io/bambi/main/index.html with the burn-in
of 50,000 while making draws of 4,000, on 4 MCMC chains (Hamiltonian). As a
way to isolate the effect (or importance) of each predictor, we did two
things: one was to look at a posterior distribution of each factor, i.e. a
coefficient $\beta$ tied with a predictor, and see how far it is removed from
0; another was to conduct an ablation study where we looked at how the absence
of a feature affected the model’s performance, which we measured with a metric
known as ‘Watanabe-Akaike Information Criterion’ (WAIC) Watanabe (2010);
Vehtari et al. (2016), a Bayesian incarnation of AIC Burnham and Anderson
(2003).131313 WAIC is given as follows.
$\text{WAIC}=\sum_{i}^{n}\log\mathbb{E}[p(y_{i}|\theta)]-\sum_{i}^{n}\mathbb{V}[\log
p(y_{i}|\theta)].$ (2) $\mathbb{E}[p(y_{i}|\theta)]$ represents the average
likelihood under the posterior distribution of $\theta$, and
$\mathbb{V}[\alpha]$ represents the sample variance of $\alpha$, i.e.
$\mathbb{V}[\alpha]=\frac{1}{S-1}\sum_{1}^{S}(\alpha_{s}-\bar{\alpha})$, where
$\alpha_{s}$ is a sample draw from $p(\alpha)$. A higher WAIC score indicates
a better model. $n$ is the number of data points.

Figure 4: Posterior distributions of coefficients ($\beta$’s) in the full
model. The further the distribution moves away from 0, the more relevant it
becomes to predicting the outcome.

Figure 4 shows what posterior distributions of parameters associated with
predictors looked like after 4,000 draw iterations with MCMC. None of the
chains associated with the parameters exhibited divergence. We achieved
$\hat{R}$ between 1.0 and 1.02, for all $\beta_{i}$, a fairly solid stability
Gelman and Rubin (1992), indicating that all the relevant parameters had
successfully converged.141414 $\hat{R}=\text{the ratio of within- and between-
chain variances}$, a standard tool to check for convergence Lambert (2018).
The closer the ratio is to the unity, the more likely MCMC chains have
converged.

At a first glance, it is a bit challenging what to make of Figure 4, but a
generally accepted rule of thumb is to assume distributions that center around
0 as of less importance in terms of explaining observations, than those that
appear away from zero. If we go along with the rule, then the most likely
candidates that affected readability are: ease, subset, fk grade, grammar,
meaning, fluency, split, and overlap. What remains unclear is, to what degree
the predictors affected readability.

One good way to find out is to do an ablation study, a method to isolate the
effects of an individual factor by examining how seriously its removal from a
model degrades its performance. The result of the study is shown in Table 9.
Each row represents performance in WAIC of a model with a particular predictor
removed. Thus, ‘ted1’ in Table 9 represents a model that includes all the
predictors in Table 8, except for ted1. A row in blue represents a full model
which had none of the features disabled. Appearing above the base model means
that a removal of a feature had a positive effect, i.e. the feature is
redundant. Appearing below means that the removal had a negative effect,
indicating that we should not forgo the feature. A feature becomes more
relevant as we go down, and becomes less relevant as we go up the table. Thus
the most relevant is fluency, followed by meaning, the least relevant is
subtree, followed by dale, and so forth. We can tell from Table 9 what
predictors we need to keep to explain the readability: they are grammar,
split, fk grade, ease, meaning and fluency (call them ‘select features’). Note
that bart is in the negative realm, meaning that from a perspective of
readability, people did not care about whether the simplification was done by
human or machine. samsa was also found in the negative domain, implying that
for a perspective of information, a two-sentence splitting carries just as
much information as a three way division of a sentence.

To further nail down to what extent they are important, we ran another
ablation experiment involving the select features alone. The result is shown
in Table 10. At the bottom is fluency, the second to the bottom is split,
followed by meaning, and so forth. As we go up the table, a feature becomes
less and less important. The posterior distributions of these features are
shown in Figure 5.151515We found that they had $1.0\leq\hat{R}\leq 1.01$, a
near-perfect stability. Settings for MCMC, i.e. the number of burn-ins and
that of draws, were set to the same as before. Not surprisingly, they are
found away from zero, with fluency furtherest away. The result indicates that
contrary to the popular wisdom that classic readability metrics such as ease,
and fk grade, are of little use, they had a large sway on decisions people
made when they were asked about readability.

effect | predictor | rank$\uparrow$ | waic$\uparrow$ | p_waic$\downarrow$ | d_waic$\downarrow$ | se$\downarrow$ | dse$\downarrow$
---|---|---|---|---|---|---|---
$-$ | subtree | 0 | -1899.249 | 17.797 | 0.000 | 17.787 | 0.000
dale | 1 | -1899.287 | 17.852 | 0.038 | 17.791 | 0.207
dep_length | 2 | -1899.362 | 17.916 | 0.113 | 17.777 | 0.211
yngve | 3 | -1899.406 | 17.904 | 0.157 | 17.777 | 0.464
tnodes | 4 | -1899.414 | 17.898 | 0.165 | 17.797 | 0.408
bart | 5 | -1899.421 | 17.967 | 0.172 | 17.786 | 0.216
samsa | 6 | -1899.450 | 18.018 | 0.201 | 17.776 | 0.315
ted1 | 7 | -1899.557 | 17.996 | 0.308 | 17.771 | 0.575
ted2 | 8 | -1899.632 | 18.019 | 0.383 | 17.782 | 0.624
frazier | 9 | -1899.740 | 18.096 | 0.492 | 17.779 | 0.708
subset | 10 | -1900.069 | 17.811 | 0.820 | 17.741 | 1.282
overlap | 11 | -1900.431 | 17.966 | 1.182 | 17.750 | 1.511
ref. | base | 12 | -1900.532 | 19.089 | 1.283 | 17.787 | 0.208
$+$ | grammar | 13 | -1900.780 | 17.979 | 1.531 | 17.698 | 1.657
split | 14 | -1900.852 | 18.030 | 1.603 | 17.697 | 1.776
ease | 15 | -1901.657 | 17.962 | 2.408 | 17.670 | 2.064
fk_grade | 16 | -1901.710 | 18.030 | 2.462 | 17.685 | 2.049
meaning | 17 | -1903.795 | 17.885 | 4.546 | 17.425 | 3.071
fluency | 18 | -1965.386 | 17.938 | 66.137 | 14.067 | 11.349
Table 9: Comparison in WAIC. p_waic = the effective number of parameters Spiegelhalter et al. (2002), a measure to estimate the complexity of the model: the greater, the more complex. d_waic = the distance in WAIC to the top model. se = standard error of WAIC estimates. dse = standard error of differences in WAIC estimates between the top model and each of the rest. $\uparrow$ means that higher is better. $\downarrow$ indicates the opposite. predictor | rank$\uparrow$ | waic$\uparrow$ | p_waic$\downarrow$ | d_waic$\downarrow$ | se$\downarrow$ | dse$\downarrow$
---|---|---|---|---|---|---
base | 0 | -1891.901 | 7.181 | 0.000 | 17.485 | 0.000
grammar | 1 | -1892.235 | 6.183 | 0.335 | 17.365 | 1.672
ease | 2 | -1893.515 | 6.137 | 1.614 | 17.350 | 2.324
fk_grade | 3 | -1893.626 | 6.161 | 1.726 | 17.366 | 2.358
meaning | 4 | -1895.308 | 6.145 | 3.407 | 17.111 | 3.059
split | 5 | -1900.028 | 6.169 | 8.127 | 17.038 | 4.247
fluency | 6 | -1956.041 | 5.935 | 64.140 | 13.784 | 11.289
Table 10: Results in WAIC for the reduced model Figure 5: Posterior
distributions of the coefficient parameters in the reduced model.

##  6 Conclusions

In this work, we asked two questions: does cutting up a sentence help the
reader better understand the text? and if so, does it matter how many pieces
we break it into? We found that splitting does allow the reader to better
interact with the text (Table 3) and moreover, two-sentence simplifications
are clearly favored over three-sentence simplifications (Tables 3,9,10). Why
two-sentence splits make a better simplification is something of a mystery. A
possible answer may lie in a potential disruption splitting may have caused in
a sentence-level discourse structure, whose integrity Crossley et al. (2011,
2014) argued, constitutes a critical part of simplification, a topic that we
believe is worth a further exploration in the future.

##  7 Limitations

  * •

We did not consider cases where a sentence is split into more than three. This
is mainly due to our failure to find a dataset containing manual
simplifications of length greater than three in a large number. While it is
unlikely that our claim in this work does not hold for cases beyond three,
testing the hypothesis on cases that involve more than three sentences would
be desirable.

  * •

A cohort of people we solicited for the current work are generally well
educated adults who speak English as the first language. Therefore, the
results we found in this work may not necessarily hold for L2-learners,
minors, or those who do not have college level education.

##  8 Acknowledgement

We thank anonymous reviewers for sharing with us their comments and ideas. We
note their effort with much gratitude and appreciation.

## References

  * Boghrati et al. (2018) Reihane Boghrati, Joe Hoover, Kate M. Johnson, Justin Garten, and Morteza Dehghani. 2018.  Conversation level syntax similarity metric.  _Behavior Research Methods_ , 50(3):1055–1073.  [link].
  * Botha et al. (2018) Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018\.  Learning to split and rephrase from Wikipedia edit history.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 732–737, Brussels, Belgium. Association for Computational Linguistics.  [link].
  * Burnham and Anderson (2003) K.P. Burnham and D.R. Anderson. 2003.  _Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach_.  Springer New York.  [link].
  * Capretto et al. (2020) Tomás Capretto, Camen Piho, Ravin Kumar, Jacob Westfall, Tal Yarkoni, and Osvaldo A. Martin. 2020.  Bambi: A simple interface for fitting bayesian linear models in python.  [link].
  * Chall and Dale (1995) J.S. Chall and E. Dale. 1995.  _Readability Revisited: The New Dale-Chall Readability Formula_.  Brookline Books.  [link].
  * Chen et al. (2022) Maximillian Chen, Caitlyn Chen, Xiao Yu, and Zhou Yu. 2022.  Fastkassim: A fast tree kernel-based syntactic similarity metric.  _arXiv preprint arXiv:2203.08299_.
  * Chipman et al. (2010) Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. 2010.  BART: Bayesian additive regression trees.  _The Annals of Applied Statistics_ , 4(1).  [link].
  * Collins and Duffy (2002) Michael Collins and Nigel Duffy. 2002.  New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 263–270, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.  [link].
  * Crossley et al. (2011) Scott A. Crossley, David B. Allen, and Danielle S. McNamara. 2011.  Text readability and intuitive simplification: A comparison of readability formulas.  _Reading in a Foreign Language_ , 23(1):84–101.
  * Crossley et al. (2014) Scott A. Crossley, Hae Sung Yang, and Danielle S. McNamara. 2014.  What’s so simple about simplified texts? A computational and psycholinguistic investigation for text comprehension and text processing.  _Reading in a Foreign Language_ , 26(1):92–113.
  * Flesch (1979) R. Flesch. 1979.  _How to Write Plain English: A Book for Lawyers and Consumers_.  Harper & Row.  [link].
  * Frazier (1985) Lyn Frazier. 1985.  Syntactic complexity.  In David R. Dowty, Lauri Karttunen, and Arnold M.Editors Zwicky, editors, _Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives_ , Studies in Natural Language Processing, pages 129–189. Cambridge University Press.  [link].
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  Creating training corpora for NLG micro-planners.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 179–188, Vancouver, Canada. Association for Computational Linguistics.  [link].
  * Gelman and Rubin (1992) Andrew Gelman and Donald B. Rubin. 1992.  Inference from iterative simulation using multiple sequences.  _Statistical Science_ , 7(4):457–472.  [link].
  * Kim et al. (2021) Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch. 2021\.  BiSECT: Learning to split and rephrase sentences with bitexts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6193–6209, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.  [link].
  * Kincaid et al. (1975) J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command.
  * Lambert (2018) Ben Lambert. 2018.  _A Student’s Guide to Bayesian Statistics_.  SAGE.
  * Linero (2017) Antonio R. Linero. 2017.  A review of tree-based bayesian methods.  _Communications for Statistical Applications and Methods_ , 24:543–559.
  * Magerman (1995) David M. Magerman. 1995.  Statistical decision-tree models for parsing.  In _33rd Annual Meeting of the Association for Computational Linguistics_ , pages 276–283, Cambridge, Massachusetts, USA. Association for Computational Linguistics.  [link].
  * McNamara et al. (2014) Danielle S. McNamara, Arthur C. Graesser, Philip M. McCarthy, and Zhiqiang Cai. 2014\.  _Automated Evaluation of Text and Discourse with Coh-Metrix_.  Cambridge University Press.
  * Moschitti (2006) Alessandro Moschitti. 2006.  Making tree kernels practical for natural language learning.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 113–120, Trento, Italy. Association for Computational Linguistics.  [link].
  * Narayan et al. (2017) Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017.  Split and rephrase.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 606–616, Copenhagen, Denmark. Association for Computational Linguistics.  [link].
  * Niklaus et al. (2019) Christina Niklaus, André Freitas, and Siegfried Handschuh. 2019.  MinWikiSplit: A sentence splitting corpus with minimal propositions.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 118–123, Tokyo, Japan. Association for Computational Linguistics.  [link].
  * Nuti et al. (2019) Giuseppe Nuti, Lluís Antoni Jiménez Rugama, and Andreea-Ingrid Cross. 2019\.  An explanable bayesian decision tree algorithm.  _arXiv:1901.03214v3 [stat.ML]_.  [link].
  * Roark et al. (2007) Brian Roark, Margaret Mitchell, and Kristy Hollingshead. 2007.  Syntactic complexity measures for detecting mild cognitive impairment.  In _Biological, translational, and clinical language processing_ , pages 1–8, Prague, Czech Republic. Association for Computational Linguistics.  [link].
  * Ross et al. (1991) Steven Ross, Michael H. Long, and Yasukata Yano. 1991.  Simplification or elaboration? the effects of two types of text modifications on foreign language reading comprehension.  _University of Hawai’i Working Papers in ESL_ , 10(22):1–32.
  * Spiegelhalter et al. (2002) David J Spiegelhalter, Nicola G Best, Bradley P Carlin, and Angelika Van Der Linde. 2002.  Bayesian measures of model complexity and fit.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 64(4):583–639.
  * Sulem et al. (2018a) Elior Sulem, Omri Abend, and Ari Rappoport. 2018a.  BLEU is not suitable for the evaluation of text simplification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 738–744. Association for Computational Linguistics.  [link].
  * Sulem et al. (2018b) Elior Sulem, Omri Abend, and Ari Rappoport. 2018b.  Semantic structural evaluation for text simplification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 685–696, New Orleans, Louisiana. Association for Computational Linguistics.  [link].
  * Vehtari et al. (2016) Aki Vehtari, Andrew Gelman, and Jonah Gabry. 2016.  Practical bayesian model evaluation using leave-one-out cross-validation and WAIC.  _Statistics and Computing_ , 27(5):1413–1432.  [link].
  * Vijaymeena and Kavitha (2016) M. K. Vijaymeena and K. Kavitha. 2016.  A survey on similarity measures in text mining.  _Machine Learning and Applications: An International Journal_ , 3(1):19–28.  [link].
  * Watanabe (2010) Sumio Watanabe. 2010.  Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory.  _Journal of Machine Learning Research_ , pages 3571–3594.
  * Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015.  Problems in current text simplification research: New data can help.  _Transactions of the Association for Computational Linguistics_ , 3:283–297.  [link].
  * Yngve (1960) Victor H. Yngve. 1960.  A model and an hypothesis for language structure.  _Proceedings of the American Philosophical Society_ , 104(5):444–466.  [link].
  * Zhang and Shasha (1989) Kaizhong Zhang and Dennis Shasha. 1989.  Simple fast algorithms for the editing distance between trees and related problems.  _SIAM Journal on Computing_ , 18(6):1245–1262.  [link].
  * Zhang and Lapata (2017) Xingxing Zhang and Mirella Lapata. 2017.  Sentence simplification with deep reinforcement learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.  [link].