Sandeep Kumar

School of Information Technology
IIT Delhi
New Delhi

Abhisek Panda

Department of Computer Science and Engineering, IIT Delhi
New Delhi

Smruti R. Sarangi

Department of Computer Science and Engineering, IIT Delhi
New Delhi

Trusted execution environments (TEEs) such as facilitate the secure execution of an application on
untrusted machines. Sadly, such environments suffer from serious limitations and performance overheads in terms of writing
back data to the main memory, their interaction
with the OS, and the ability to issue I/O instructions. There is thus a plethora of work that focuses on improving
the performance of such environments – this necessitates the need for a standard, widely accepted benchmark suite (something
similar to SPEC and PARSEC). To the best of our knowledge, such a suite does not exist.

Our suite, , contains a diverse
set of workloads such as blockchain codes, secure machine learning algorithms, lightweight web servers, secure key-value stores, etc.
We thoroughly characterize the behavior of the benchmark suite on a native platform and on a platform
that uses a library OS-based shimming layer (). We observe that the most important metrics of interest are performance counters
related to paging, memory, and TLB accesses. There is an abrupt change in performance when the memory footprint starts to exceed
the size of the EPC size in , and the library OS does not add a significant overhead ($\approx \pm$ 10%).

§ INTRODUCTION

Intel Secure Guard eXtension or  <cit.> has gained popularity in recent years as a way to securely execute an application on a remote, untrusted machine. The security of the application and data within , i.e.,  confidentiality, integrity, and freshness  are guaranteed by the hardware.
The code and data within is even out of the reach of privileged system software such as the operating system and hypervisor. Recently, Microsoft Azure adopted to provide secure computing in their data centers <cit.>.

However, this protection comes at a cost. , to ensure security guarantees, puts certain restrictions on the applications running within it, such as no system calls, as the operating system is not a part of the trusted framework of  <cit.>. Therefore, few additional  and expensive  steps are required to enable system call support, which incurs additional performance overheads <cit.>.
Furthermore, reserves a portion of the main memory for its operations, which is managed by the hardware. However, this reserved memory is limited in size, and any application allocating more than the reserved memory, incurs a significant amount of performance overhead <cit.>.

Researchers have focused on alleviating this problem by proposing different mechanisms and workarounds to reduce the overheads <cit.>.

To show the benefits of their methods, researchers have resorted to manual porting of applications to  <cit.>. However, porting an application requires significant expertise and development effort <cit.>. Also, the decision of which application to port is generally motivated by the ease of porting, and not necessarily by the gains accrued by doing so.
Hence, there is no accepted, standard method for benchmarking -based systems primarily due to the ad hoc nature of
workload creation.

The big picture is as follows. The workloads used to evaluate the efficiency of different methods for improving  vary across different proposals. Hence, it is not possible to compare the performance gains in one work with those of another in any meaningful manner.
Therefore, there is a need for a standard benchmark suite for , much like traditional benchmarks suites such as SPEC <cit.> and PARSEC <cit.>.

A benchmark suite needs to thoroughly evaluate all the critical components of , and enable  performance comparison by setting a common denominator across different works.
Primarily, there are three sources of performance overheads in : encryption/decryption of the data in the reserved secure memory, the cost for accessing operating system services, and the additional time for swapping in data when an application has allocated more memory than the reserved memory <cit.>. Prior works <cit.> in this field propose different benchmark suites <cit.> for evaluating . However, they only focus on the first two costs, ignoring the last one, which accounts for the maximum performance overhead <cit.>.

We present  – a comprehensive benchmark suite for .
contains 10 real-world and synthetic benchmarks from different domains that thoroughly evaluate all the critical components of .

We use to evaluate in two different modes: native mode where we port the benchmarks to , and shim mode where we execute benchmarks in an environment where a  thin system software layer intercepts the system calls and intercedes with the OS on behalf of the application <cit.>. Such
shim layers are also known as library operating systems; they are gaining popularity because they significantly reduce the development time required to run an application on as compared to porting the same application to  <cit.>.

Our precise list of contributions are as follows.

* We present , a benchmark suite for that thoroughly evaluates all of its components.

* We stress test the impact of EPC on the performance of applications — a crucial component that is missing from prior work.

* We thoroughly evaluate the performance overhead incurred while executing an application with a library operating system.

The rest of the paper is organized as follows: we discuss the required background for the paper in Section <ref>. In Section <ref> we discuss related work and the motivation for the paper.

This is followed by a detailed overview of our benchmark suite, ,  in Section <ref>. We discuss the evaluation of the benchmarks in Section <ref>.
We finally conclude in Section <ref>.

§ BACKGROUND

In this section, we discuss the necessary background for the paper.

Intel Secure Guard eXtension or SGX ensures the secure execution of an application either on a local or remote machine. It guarantees confidentiality, integrity, and freshness of the code and data running within it. Even privileged system software such as the operating system and hypervisor cannot affect its execution.

reserves a part of the system memory for its use at boot time. This reserved memory is known as the Processor Reserved Memory or PRM <cit.>.  Our system supports 128 MB of PRM, and the rest of the discussion in the paper is based on this setting. The PRM is split into two regions that are used to store metadata and date/code of user applications, respectively. The latter is called the Enclave Page Cache or EPC. The size of the EPC is 92 MB, although supports applications that require more memory   (details in Section  <ref>).
For every process, creates a trusted execution environment an enclave <cit.>.

The operating system cannot access the data within an enclave. However, an enclave still requires the operating system's support for setting it up, scheduling, context switching, page management, and cleanup. To enable this, the memory management of the enclave is done by the operating system.
Just before launching an enclave, the hardware checks the loaded binary for tampering by securely calculating its signature (hash) and matching it with the signature provided by the enclave's author.

§.§ Enclave Page Cache

The EPC is used to allocate memory for all the applications executing within . The  data in  the EPC is always in an encrypted form to prevent any snooping from   privileged system software such as the operating system. The data is decrypted when brought in to the LLC (last level cache) upon a CPU request. uses a dedicated hardware called the Memory Management Engine or MEE for encrypting and decrypting the data.

Sadly, the size of the EPC is one of the major limitations of  <cit.>.
A typical modern application generally has a working set that is more than 92 MB <cit.>. In such cases, transparently evicts pages from the EPC to the untrusted memory, albeit in an encrypted form, to make space for the new data. When an application tries to access an evicted page, an EPC fault is raised, and brings the page back to the EPC <cit.>.

An EPC fault is an expensive operation. encrypts and calculates the MAC (encrypted hash) of a page prior to an eviction. When the page is brought back, it needs to be decrypted and integrity checked before its use.
Our experiment found that evicting a page from the EPC takes on an average of 12,000 cycles.

§.§ Enclave Transitions: s and s

The security provided by comes at a cost. For security reasons, puts several restrictions on an executing enclave – notably, it cannot make any system calls <cit.>.

In the framework, the operating system is an untrusted entity, and hence, systems calls are restricted.
To make a system call, an enclave first exits the secure region by calling an (outside call) function.
After this, it makes the system call, collects its results, and returns to the secure region.
Similarly, an application from an unsecure region can call a function within an enclave by calling an (enclave call) function.

During a transition from the secure region to the unsecure region, the TLB entries of the enclave are flushed due to security concerns <cit.>. When the enclave returns, the TLB entries have to be populated again.
While adding a TLB entry to the TLB, if the entry points to an EPC page, it is first verified. For this purpose, maintains a table called the Enclave Page Cache Map or EPCM <cit.> in the secure region. The EPCM contains one entry for every page in the EPC. For each EPC page, the EPCM tracks its owner and the corresponding virtual address for which this page was allocated. These values are checked when a TLB entry for the corresponding page is being added to the TLB (see Figure  <ref>).

Frequent enclave transitions affect the performance of an application due to context switches, TLB misses, and cache pollution. hotcalls show that the cost of calling an enclave function typically requires 17,000 cycles.

Figure showing the relation between the address space, the EPC, and the EPCM.

§.§ Library Operating Systems

Intel provides a software development kit (Intel SDK <cit.>) to ease application development for . However, porting or writing an application for still requires an in-depth knowledge of the workings of and significant development effort <cit.>.

Due to this, many researchers have opted for executing an application using a shim layer built on top of – also known as a library operating system or LibOS. A library operating system can execute an unmodified binary on ; thus, saving on the high cost and effort of porting the application.
Scone <cit.>,  <cit.>, and Panoply <cit.> are some examples of such systems. Researchers have reported that it can take up to 3-4 months for porting applications <cit.>; this is also in line with our observations. Moreover, the task of verifying correct execution for all corner cases will take even more time. This is precisely why such shim layers need to be inevitably used and are fast becoming an inseparable part
of the stack. Even though they have their share of performance overheads, the sheer reduction in the development and verification effort makes them a necessary part of many deployments. No benchmark suite for can be oblivious of them.

§ RELATED WORK AND MOTIVATION

In this section, we discuss the related work in this area and the motivation for .

§.§ Related Work

Limited work has been done in this area, mainly due to the limitations of the framework and the engineering effort required to port an application to .

portorshim in their work Port-or-Shim, ported a part of  <cit.> for and compared its performance against a shimmed version running within a library operating system. They also used for their evaluations.
They point out that porting to took months – and that too after removing certain features from it <cit.>. Whereas running a shimmed version of on  <cit.> took a week of effort.
They specifically focus on the cost of the encryption/decryption and enclave transitions. They intentionally avoided EPC faults by ensuring that the amount of memory allocated to the benchmarks is less than the size of the EPC (92 MB).
They report that the performance of   (ported version of ) and the version that runs within the library OS is the same – this raises a question about whether porting was worth the effort.

Apart from this,  sgxlapd_sgxnbench proposed a method to prevent side-channel attacks in . They ported  <cit.> to to evaluate the effectiveness of their solution. However, the working set of the benchmarks was small and limited analyses were performed.

Both and (ported versions) are single-threaded benchmark suites <cit.>. mainly focuses on the memory bandwidth and the system call latencies. mostly contains CPU-intensive workloads and is designed to check the efficiency of integer and floating point operations of a CPU.
Our suite is far more comprehensive in terms of its coverage (evaluated in Section <ref>).

sgxometer also point out the issues with using for evaluation. They propose a CMake based framework to develop and non-applications from the same source.
Apart from this, there are other proposals by  sgxperf,  teemon,  teeperf, and  sgxtop that propose methods to collect statistics about an executing secure application. This information can help a developer  debug a secure application or improve its performance. However, these are not benchmark suites; rather, they focus on the efficient profiling aspect for an executing enclave.

§.§ Motivation

Here, we discuss the motivation for . The system setup for experiments used here is listed in Table <ref>.

§.§.§ Experiment: Stressing the EPC

Allocating beyond the EPC size increases the overhead. The baseline is a (non-) setting with the same input size. For EPC evictions the baseline is the setting.

The limited amount of EPC memory is one of the biggest challenges in  <cit.>.
Due to the small size, EPC faults are a common event. Multiple instances of an enclave with a small memory footprint may also cause a number of EPC faults. This is because an enclave prior to its execution is loaded completely in the EPC to verify its content <cit.>.

As can be seen in Figure <ref>, on crossing the EPC boundary the number of dTLB misses increases by 91$\times$, page walk cycles by more than 124$\times$, and EPC evictions by 100$\times$ as  compared to when the amount of memory is less than the EPC size.
Hence, analyzing the impact of the  EPC size on the performance is crucial – a fact completely ignored by  <cit.> and  <cit.>.

§.§.§ Experiment: Execution of multi-threaded benchmarks

The latency of the server increases with the number of concurrent accesses by up to 7$\times$ while running in and compared to a (non-) execution.

An application leverages the multiple cores on a modern system by using threads to speed up its operation. does not support thread creation inside the secure region; however, numerous threads can do an and execute the same function using the same global enclave ID <cit.>. The overhead due to can change drastically based on the number of threads making the .
As shown in Figure <ref>, the latency of increases with the
number of threads (by 7$\times$).
Hence, it is crucial to capture the executions' characteristics in this setting also.
and  do no contain any multi-threaded benchmarks.

§.§.§ Experiment: Library Operating System

As noted in prior work <cit.> and also by us, shimming an application is much easier than porting an application for – in terms of development and verification effort. We believe that in the future a library operating system will be the primary way to execute an application on . Hence, it is essential to understand the behavioral changes between ported and shimmed applications.
Port-or-Shim <cit.> also focuses on this problem, but with benchmarks that have a small working set (70ṀB). Our
observations are more comprehensive and also differ.
As shown in Figure <ref>, the impact of a library operating system depends on the characteristics of the application
and thus needs to be rigorously studied.

A library operating system may affect the performance of an application in a positive or negative manner, depending on the characteristics of the application.

§.§.§ Experiment: Real-world benchmarks

Real world applications exhibit different phases during their execution. A typical pattern is that an application will read some data from the file system, process it, and then store the results. Micro-benchmarks such as  <cit.> lack this phase change behavior and thus do not represent a real-world scenario (details in Section <ref>).

Let us summarize.

* Existing benchmark suites for  <cit.> are ported version of decade-old benchmarks that were designed to evaluate the CPU performance; they are not well suited to .

* Using multiple threads changes the overheads of , hence it is necessary to also include multi-threaded benchmarks.

* The performance impact of a library operating system depends on the characteristics of the application executing within it. Thus, there is a need for further study.

* Modern applications show different phases during their execution. Therefore, it is necessary to  assess using real-world benchmarks.

In this section, we create a novel set of criteria for evaluating the quality of
a benchmark suite.
The starting point is the set of properties listed qualitatively by the authors of the
PARSEC suite <cit.>.

* Diverse: The benchmark suite should evaluate all the components of the framework that is under consideration.

* Phase changes: Modern applications show different characteristics during their execution. Most synthetic or micro-benchmarks lack this feature. Hence, it is necessary to evaluate the system using real-world benchmarks.

* Coverage: The benchmark suite should evaluate each component for a wide range of values. This analysis reveals how the performance changes with changing values of these components.

We devise the following scores to mathematically capture the aforementioned criteria.

Let us assume that our benchmark suite ($\mathcal{W}$) is a set of n benchmarks, i.e., $\mathcal{W} = \{w_1, w_2,\dots w_n\}$. During the executing of a benchmark ($w_i \in \mathcal{W}$) we collect $m$ execution-related statistics. We store them in an $m$-dimensional vector $x_i$ for benchmark $w_i$. For a single execution of all the benchmarks in the suite, the matrix $\mathcal{X}$ contains all the individual vectors (stored as row vectors). This description is agnostic to the specific parameters captured in an execution.

§.§ Cluster Score

The benchmarks in a suite should be designed or chosen such as they show different properties as compared to each other in terms of execution statistics. If they are all similar to one another, running different benchmarks will reveal no additional insights. In simple terms, the benchmarks should not be clustered.

To capture the diverse nature of a benchmark suite, we define a score called ClusterScore. For this, we first collect the matrix $\mathcal{X}$ to create clusters using K-means clustering <cit.>.  To determine the quality of the clusters, we use a well-known score called the Silhouette score  <cit.>. We want a clustering that is as poor as possible.

§.§.§ Silhouette Score

Let us say we form k clusters. For a point $p \in C_i$, i.e., point $p$ in cluster $C_i$, we first calculate an intra-cluster dissimilarity score for the point $p$ called $\eta(p)$.

\begin{equation}
\eta(p) = \frac{1}{|C_i|-1}\sum_{\forall p'\in C_i} dis(p,p')
\end{equation}

where, $dis(p,p')$ is the Euclidean distance between $p$ and $p'$.

After this, we calculate an inter-cluster dissimilarity score for point $p$, i.e., $\lambda(p)$. We do this for all the clusters except for the cluster containing $p$, and then select the minimum value.

\begin{equation}
Cost(p, j) = min \frac{1}{|C_j|}\sum_{\forall p' \in C_j, C_j \ne C_i} dis(p,p')
\end{equation}

Let us define $\lambda(p)$ as the lowest cost for any $j \ne i$.

Finally, we calculate the silhouette score for the point $p$ ($\mathcal{S}(p)$):

\begin{equation}
\mathcal{S}(p) =
\begin{cases}
\frac{\lambda(p)-\eta(p)}{max\{\lambda(p), \eta(p)\}} & k > 1  \\
0 & \text{ if } k=1
\end{cases}
\end{equation}

The silhouette score of cluster $C_i$ is an average of the silhouette scores of all the points within that cluster.

\begin{equation}
\mathcal{S}(C_i) = \frac{1}{|C_i|}\sum_{\forall p \in C_i} \mathcal{S}(p)
\end{equation}

The silhouette score of a benchmark suite is the average of silhouette scores for all the clusters. Assuming k clusters:

\begin{equation}
\mathcal{S}(\mathcal{W})_k = \frac{1}{k}\sum_{i=1}^{k} \mathcal{S}(C_i)
\end{equation}

To calculate the $ClusterScore$, we calculate $\mathcal{S}(\mathcal{W})_k$ for $k=2\text{ to }|\mathcal{W}|-1$, and take the average. Here, $k$ is the number of clusters.

\begin{equation}
\boxed{ClusterScore = \frac{1}{|\mathcal{W}|-2}\sum_{k=2}^{|\mathcal{W}|-1}\mathcal{S}(\mathcal{W})_k}
\end{equation}

§.§ Trend Score

Here, we define a metric that captures the difference between the execution trends of different workloads.

For a given input to a set of workloads $\mathcal{W}$ ($|\mathcal{W}|=n$), let us say $T_z = \{t_1,t_2,t_3...t_n\}$, is the set of  vectors for a particular hardware performance counter $z\in Z$ (e.g. ) across all the individual workloads. Here, $Z$ is the set of all the hardware performance counters. Each  vector, $t_i$, is a time series (specific to a given workload).

To measure the distance between two time series, we use a well-known  technique called Dynamic Time Warping or DTW <cit.>. DTW aims to minimize the distance between the two time series, which can have different lengths by non-linearly “warping" the time-space to match them as shown in Figure  <ref>.

Figure showing Dynamic Time Warping (DTW)

We calculate the pair-wise DTW distance of all the data points in $T_z$, and report the average for metric $z$ ($TScore_z$).

\begin{align}
TScore_z = \frac{1}{|\mathcal{W}|*(|\mathcal{W}|-1)}\sum_{t_m \in T_z}\sum_{t_n \in T_z, m \ne n}DTW(t_m,t_n)
\end{align}

We then calculate the average DTW score for all the metrics in $Z$, that is the final $TrendScore$.

\begin{equation}
\boxed{TrendScore = \frac{1}{|Z|}\sum_{\forall i \in Z }^{}TScore_i}
\end{equation}

§.§.§ Normalization in  DTW

Note that if a particular $t_m \in T_z$ has unusually high values (see Figure  <ref>), then the $TScore_z$ score will be dominated by this, and eventually the $TrendScore$ <cit.>.

To fix this, we use the CDF (cumulative distribution function) <cit.> of a particular metric instead of its absolute values. The CDF normalizes the Y-axis (see Figure  <ref>). Doing so also bounds the distance between two points to [0,100].

LLC misses as a function of time

Normalizing the y-axis by considering the CDF

Normalizing the x-axis also by considering the percentage of total execution

Normalizing LLC misses for workloads W1 to W5.

However, even now, the x-axis, i.e., the time axis is still not normalized because different workloads might have taken different times to execute.
To fix this, instead of the absolute time, we use the
percentiles of the execution time to define the x-axis. The resultant normalized time series are used to compute the DTW values and ultimately the trend score ($TrendScore$).

§.§ Coverage Score

Here, we define a metric called CoverageScore for a benchmark suite, that captures its spread over the parameter space (hardware counters). All the benchmarks during their execution incur different hardware events, such as TLB or LLC misses. Depending on the characteristics of the benchmarks, the value of the events can be significantly different from one another. Ideally, a benchmark suite should have a high CoverageScore, indicating a good spread over the parameter space.

Let us assume we want to compare the coverage of two benchmarks suites $\mathcal{W}_1$ and $\mathcal{W}_2$, each containing $n$ benchmarks.

For a same (or similar) input, we execute workloads in $\mathcal{W}_1$ and $\mathcal{W}_2$ and collect the hardware counter matrices $\mathcal{X}_1$ $\mathcal{X}_2$,  respectively. All the individual parameters are normalized
and brought into the range $[0,1]$.

Then, we use Principal Component Analysis or PCA <cit.> to reduce the dimensionality of the data, while preserving its variance. PCA will eliminate all the redundant features (events) from the data. We ensure that 98% of the original variance is preserved.

\begin{align}
\langle \mathcal{X}^T_1, d_1 \rangle &= PCA(\mathcal{X}_1, \texttt{variance} =.98) \\
\langle \mathcal{X}^T_2, d_2 \rangle &= PCA(\mathcal{X}_2, \texttt{variance} =.98)
\end{align}

Here, $d_1$ and $d_2$ are the PCA components for the transformed  data, $\mathcal{X}_1$ and $\mathcal{X}_2$, respectively.
We define the CoverageScore as the variance present in the transformed data.

We define the CoverageScore for $\mathcal{W}_1$ as

\begin{align}
\boxed{CoverageScore_{\mathcal{W}_1} = \frac{1}{d_1}\sum_{i=0}^{d_1-1} Variance \left ( comp_i (\mathcal{X}^T_1) \right )}
\end{align}

Here, $comp_i$ is the function that returns the contents of the $i^{th}$ PCA component. Similarly, we can calculate the CoverageScore for $\mathcal{W}_2$ also.
A high CoverageScore indicates a good coverage of the $n$-dimensional parameter space. Prior work <cit.> has also used a simple variance-based metric to define the coverage of PARSEC<cit.> and SPLASH-2 <cit.> benchmark suites.

§ BENCHMARK SUITE

The most important challenge in front of us was to find an appropriate set of workloads that
need to be executed in a secure environment.

This problem
has a degree of subjectivity. Nonetheless, we followed standard practice and restricted ourselves to workloads that have been used by highly cited works on in the recent past. We found the following workloads: blockchain related <cit.>, protecting key-value pairs <cit.>, securing databases <cit.>, protecting keys <cit.>, securing a machine learning models <cit.>, protecting network routing tables <cit.>, securing communication <cit.>, graph traversals <cit.>, protecting web-servers <cit.>, and HPC workloads <cit.>.

The next task was to refine the set of workloads and choose an
appropriate set.

There are three main sources of overheads in : encryption/decryption, enclave transitions, and EPC faults (see Section <ref>).
While selecting workloads for , our primary aim was to ensure complete coverage of all the components.

First, we selected some of the most commonly used workloads  such as  <cit.> and  <cit.>.
We then analyzed them and identified where they lack in terms of stressing the components. For e.g. both and do not stress the CPU much. Hence, to fulfil this criterion, we selected workload which is a CPU-intensive and multi-threaded workload. However, while it stresses the CPU, it does not use a lot of memory. To ensure both the components are stressed, we opted for an HPC workload , that was used by a prior work <cit.> for similar purposes.

For selecting workloads that exclusively stress the EPC, we selected the following from prior work: , , , and . Each of them has different data access patterns. is used commonly in databases for efficient lookups and has been used in setting also <cit.>. is used in protecting the control flow graph of an application <cit.>. While aims to minimize the number of nodes it visits, aims to visit all nodes efficiently. performs a number of hash table probing operations which is at the core of many systems <cit.>.
<cit.> is a widely used workload for link analysis.
is a machine learning (ML) workload that is CPU and memory-intensive. It runs multiple iterations over the same input data, a typical pattern of ML workloads.

We also discarded some workloads such as Redis <cit.>, Fourier transform <cit.>, License Managers <cit.>, GUPS <cit.>, Nginx <cit.>, etc. because they were similar to other workloads that were already chosen.

§.§ Evaluation Modes and Input Settings

We execute the workloads in in different modes and different input settings to gain a better understanding of workings. Table <ref> list the different execution modes and input setting used in the paper.

Conventions used in the paper for discussion

LightCyan 2|c|Execution Modes
An application executing without support.
An application executing within after it is ported to the framework.
An application executing with in shimmed mode – with the support of a LibOS ().
LightCyan 2|c|Input Settings
$Memory(footprint) < Size (EPC)$, wherever applicable

$Memory(footprint) \approx Size (EPC)$, wherever applicable

$Memory(footprint) > Size (EPC)$, wherever applicable

Description of the workloads in along  with the specific settings used in the paper.

S.No.     Workload      mode    mode     mode     Property     Low ($<$  EPC)     Medium ($\approx$ EPC)     High ($>$  EPC)

1.    <cit.>                  CPU/-intensive     Blocks 3     Blocks 5     Blocks 8

2.    OpenSSL <cit.>                 Data-intensive     File Size 76 MB     File Size 88 MB     File Size 151 MB

3.    BTree <cit.>                 Data/CPU-intensive     Elements 1 M    Elements 1.5 M     Elements 2 M

4.    HashJoin <cit.>                 Data/CPU-intensive     Data Table Size 61 MB     Data Table Size 91 MB     Data Table Size 122 MB

5.    BFS <cit.>                 Data-intensive     Nodes 70 K Edges 909 K     Nodes 100 K Edges 1.3 M     Nodes 150 KEdges 1.9 M

6.    Pagerank <cit.>                  Data-intensive     Nodes 4500 Edges 10.1 M     Nodes 4750 Edges 11.2 M     Nodes 5000 Edges 12.5 M

7.     <cit.>                Data/-intensive     Records: 50K Operations:800K     Records: 100K Operations:800K     Records: 200K Operations:800K

8.     <cit.>                CPU-intensive     Points: 53K Lookups: 100    Points: 88K Lookups: 100     Points: 768K Lookups: 100

9.     <cit.>                 -intensive     Requests: 50K Threads: 16     Requests: 60K Threads: 16     Requests: 70K Threads: 16

10.     <cit.>                 Data/CPU-intensive     Rows 4000 Features 128     Rows 6000 Features 128     Rows  10000 Features 128

§.§ Workloads' Description

§.§.§ Blockchain <cit.>:
A blockchain is a distributed ledger that does not require any central authority for its management. A blockchain is essentially a linked list of blocks. A block has a payload portion (contents of the block)
and the hash of contents of the previous block on the chain. Given that the chain is immutable, these hashes have a degree of
finality and permanence.
We vary the number of blocks in the chain to create different inputs for the workload (see Table  <ref>).
The hash computation is the sensitive operation; hence, this operation is offloaded to . This function is called by many threads from the unsecure region resulting in many  s.

§.§.§ OpenSSL <cit.>:

is a library that provides access to cryptographic primitives to developers. Our workload using Intel SGX-SSL <cit.> reads encrypted data from an input file and decrypts it within . Then, it performs a small compute-intensive task based on the content of the decrypted file. Finally, it encrypts the generated output and saves it in the untrusted filesystem.
This workload stresses the mechanisms that copy data from the unsecure memory region to the EPC and the EPC if the input file size is more than the EPC size.

§.§.§  <cit.>
The data structure is used for an efficient organization of  data, specifically in database management systems. This enables efficient lookup in large databases and  applications of a similar nature – a crucial feature in today's big-data world.
This workload creates a consisting of a certain number of elements and performs multiple find operations on a randomly generated set of keys. This workload is also designed to stress the EPC and the paging system.

§.§.§  <cit.>:
The hash-join algorithm is used in modern databases to implement equi-join <cit.>. It has two phases: build and probe. Given two data tables, it first builds a hash table from the rows in the first table, and then probe it using the rows in the second table.

We vary the size of the first table and, in effect, vary the memory and compute-intensive nature of the workload.

§.§.§ Breadth-First Search (BFS) <cit.>:
The workload is a port from the implementation of the well-known breadth-first search algorithm used in the Rodinia benchmark suite <cit.>.
The input to the workload is an undirected graph. It first reads the input graph to the EPC and then traverses all the connected components in the graph.

This is primarily memory and compute-intensive workload. The computation overhead is a function of the number of nodes in the graph; the degree is at least 3.

§.§.§ Page Rank <cit.>:
PageRank is used to rank web pages based on the popularity of pages that point to it.

The input to the workload is a connected directed graph represented in the adjacency list format with an out-degree of at least 1. The workload loads the graph into the EPC and builds an adjacency matrix of pages with a default initial rank for all. The workload then uses the number of out links of the page, previous rank, and the weight of the out neighbor pages to assign a new rank. This is repeated a fixed number of times.

§.§.§ : <cit.>
is an in-memory key-value store. It is used in production servers to cache hot data in memory.

We use the popular YCSB <cit.> workload to evaluate the performance of . YCSB first populates with a specified amount of data and then performs a specified set of (read or write) operations on those key-value pairs.

§.§.§ : <cit.>
is a key computational kernel of the Monte Carlo neutron transport algorithm over a set of nuclides and grid-points <cit.>. We vary the number of grid points to generate different input sizes for the workload  (see Table  <ref>).

§.§.§  <cit.>:
is a light-weight web server that is optimized for concurrent accesses. The server however runs on a single thread.
Our workload hosts a web-page of size 20 KB (similar to <cit.>). We use the ab tool, which is a part of the Apache suite <cit.>, to make a certain number of requests to the server using concurrent threads (see Table <ref>).

§.§.§ Support Vector Machine (SVM) <cit.>
SVM is a popular machine learning technique to classify the input data by projecting it into a higher dimensional space, and then using a linear combination of separating functions.
We implemented SVM using libSVM <cit.>, a library to implement SVM in C/C++ code.

§.§ Porting to

contains 10 benchmarks. We have ported 6 of these to execute natively on (native). The other 4 are real-world benchmarks, which are
evaluated in mode using (details in Section <ref>). For these benchmarks, the engineering and verification
effort in creating a native port was prohibitive, and the benefits were not clear. Table <ref> summarizes this information and also
talks about the three execution settings (with different memory footprints): , , and .

While porting an application to , the ideal case is to run the entire application within an enclave. However, this is not always possible due to the restrictions imposed by . In this case, a crucial function is typically moved to the enclave and is accessed via an . This is the standard practice <cit.>.

We follow this approach while porting the applications. We completely ported , , , , and to . However, uses multiple threads to speed up the hash finding process. does not support the creation of threads within an enclave <cit.>. Nevertheless, multiple threads from the untrusted region can call the same function.
Hence, for , we moved the hash function inside  ; it is called from different threads from the main application which runs in the untrusted region.

§.§ Running on

To execute a binary on , we first need to define a manifest file. The manifest file contains the binary's location, list of libraries required, and the required input files. The parameters such as the enclave size and the threads to be used are also listed here. then processes this file and calculates the hash of all the required input files, which are then verified at the time of the execution.

§ EVALUATION

Here, we discuss the performance of workloads in under different execution modes and with different input settings (see Table <ref>). We focus on the insights that are not listed in prior work <cit.>, or where our observations differ from theirs.

System configuration

LightCyan 5cHardware Settings
2|l|Xeon E-2186G CPU, 3.80 GHz                      3l|Disk: 1 TB (HDD)

5|l|CPUs: 1 Socket, 6 Cores, 2 HT

1|l|DRAM:  32 GB        4l| L1: 384 KB, L2: 1536 KB, L3: 12 MB

LightCyan 5cSystem Settings

2|l|Linux kernel: 5.9      2l|ASLR: Off        1l|GCC: 9.3.0

2|l|DVFS: fixed frequency (performance)          3l|Transparent Huge Pages: never

LightCyan 5cSGX Settings
1|l|PRM: 128 MB      1l|Driver: 2.11                      3l|SDK version: 2.13
LightCyan 5cSettings
1|l|Enclave Size: 4 GB      1c|Threads: 16                      3c|Internal Memory: 64 MB

§.§ Experimental Setup

The details of our evaluated system can be seen in Table <ref>.
We use the library operating system <cit.> for our experiments in mode.
Since is under constant development, we found that the performance of the code in the master branch is significantly better than its official release (v1.1). Hence, we used the code from the master branch of their GitHub repository[Commit ID: adf6269218dfa80aed276d57121a98e7b13b0f4e].

§.§.§ Instrumenting

In order to instrument -related events, we added instrumentation code directly to the driver code. This approach has also been used in prior work <cit.>. We  identified crucial functions within the driver code that are called during different events, such as  (page fault handling function). Note that these functions do not execute within the secure world and thus can be easily instrumented.
We report the latencies of 's crucial functions in Appendix  <ref>.

Overhead in system-related events. Avg. value of EPC evictions is reported when compared with the mode. The overhead refers to the performance overhead (run time).

LightCyan  7cMode w.r.t (6 workloads)
Over- head       dTLB misses     Walk Cycles     Stall cycles     LLC misses     EPC Evictions
Low             2.0$\times$       8.38$\times$          29.7$\times$                   2.5$\times$             1.8$\times$   21.5 K
Medium          3.0$\times$     14.6$\times$          57.0$\times$                  5.3$\times$             2.0$\times$   49.6 K
High            3.4$\times$     17.48$\times$          59.1$\times$                  6.4$\times$             3.0$\times$   79.6 K

LightCyan  7cMode w.r.t (10 workloads)

Low             2.03$\times$       40.6$\times$          517$\times$                   114$\times$            24$\times$         796 K
Medium          3.13$\times$     59.7$\times$          724$\times$                  146$\times$             18.5$\times$         1,792 K
High            3.7$\times$     44.0$\times$          113$\times$                  12.7$\times$             15.5$\times$         2,255 K

LightCyan  7cMode w.r.t Native (6 workloads)

Low             1.03$\times$       3.3$\times$          5.1$\times$                   8.3$\times$             9.3$\times$         75$\times$
Medium          1.03$\times$     2.7$\times$          4.0$\times$                  7.9$\times$             9.2$\times$         68$\times$
High            0.9$\times$     2.0$\times$          3.0$\times$                  5.9$\times$             7.2$\times$         45$\times$

§.§ Evaluation Plan

We take the following approach for evaluation.

* mode performance: We analyze the impact of on the applications executing natively on it for different input sizes.

* mode performance: We study the overheads that are introduced while executing on using a library operating system.

* mode vs. mode: We compare the performance of the and execution modes.

Table  <ref> shows an overview of the evaluation results. The geometric mean value is computed across at least 10 executions (seen to be enough).

Runtime overhead in mode.

EPC evictions in mode.

Performance impact of on applications in mode for different input sizes.

Statistics for for an empty workload.

Figure showing the overhead in the execution for the  setting.

Figure showing total EPC page reloads for the  setting.

The latency of improves when using the switchless mode

Performance impact of on workloads in .

§.§ mode Performance

Here, we evaluate the performance overhead of running an application in the mode as compared to the mode with different input sizes.
As shown in Figure  <ref>, the performance overhead increases by up to $8.8\times$ as we go from the to the setting, and by up to $1.4\times$ from the to the setting.

As shown in Figure  <ref>, the total number of EPC evictions increase by up to $75\times$ when the input size is increased from to . On further increasing the input from to , the total number of EPC evictions increases by up to $2.6\times$.
As already explained in Section  <ref>, the TLB entries of an enclave are flushed before a transition to the unsecure region due to security reasons.

Hence, as we increase the size from to ,  dTLB misses increase by up to $79\times$, and by up to $2.6\times$ as we go
from to .

Due to this, the total walk cycles increase by up to $79\times$ (to ), and by up to $2.6\times$ while going from to .

Consequently, the total stall cycles increase by up to $43\times$ (to ), and by up to $5.4\times$ while going from to .

Summary: As we approach the EPC size (to ), there is a sudden rise in all the paging and TLB-related performance counters. However, going beyond the EPC size (to ) does not affect the performance to that extent.
We present a detailed discussion for all the workloads in Appendix <ref> and impact of the counters on the workloads' performance in Appendix <ref>.

§.§ Mode Performance

Here, we evaluate the performance impact of .

§.§.§ Overhead

We first characterize the overhead of just using an empty () workload.

As shown in Figure  <ref>, in this setup, performs $\approx$300 s, $\approx$1000 s, and $\approx$1000  exits. During this time, total EPC evictions are $\approx$ 1 M. However, out of these 1 M evicted EPC pages, only $\approx$ 700 pages (2 MB) are loaded back.

The reason for the unusually high number of EPC evictions is the enclave size property, which is set to 4 GB.
As prior to executing an enclave, completely loads it in the EPC to calculate its signature <cit.>. Doing so for a 4 GB enclave will cause 1 M EPC faults (1 M*4 KB=4 GB).
Lowering the value of the property enclave-size reduces the EPC evictions but worsens the performance by up to 4$\times$, even for the workloads with a small memory footprint such as Blockchain.

All these EPC evictions are done at the beginning of the execution, i.e., while initializing . We do not count this time in the execution time of a workload running on it (see Appendix  <ref>).

As shown in Figure <ref>, the performance overhead increases by up to $8.7\times$ while going from to , and by up to $2.7\times$ while going from to .

As shown in Figure <ref>, the total number of EPC load-backs (page brought back to the EPC from the untrusted memory) increases by up to $341\times$ when the input size increases from to , and by up to $4.1\times$ on further increasing the input size from to .
The total number of dTLB misses increases by up to $18.6\times$ as we go from to , and by up to $1.5\times$ as we go from to .
Due to this, the total number of walk cycles increases by up to $11\times$ as we go from to , and by up to $2.7\times$ when
we go from to .

Consequently, the total number of stall cycles increases by up to $5.11\times$ and $2.17\times$, respectively.

Summary: Similar to mode, as we approach the EPC size (to ), there is a sudden increase in all the TLB and paging-related performance counter values. However, going beyond the EPC size (to ) does not affect the performance as much. We discuss the I/O related overheads with in Appendix <ref>.

§.§ Mode vs Mode

Here, we compare the performance of the and modes (see Table  <ref>). We observe that as we increase the input size, the performance overhead of as compared to the mode starts decreasing. The overall overhead reduces by $12.6\%$ when we increase the input setting from to or to .
The total number of dTLB misses comes down by $18\%$ and $25\%$, when the input is increased from to and to , respectively.
In the same setting, the total walk cycles comes down by 21% and 25%, stall cycles by 4% and 25%, LLC misses by 1% and 21%, and EPC evictions by 9% and 33% when we increase the input size from to and to , respectively.

Summary: On increasing the workload size, the overhead of starts decreasing, and eventually, approaches that of the mode.

§.§ Switchless Mode

To reduce the cost of an , supports a switchless mode of operation where it leverages multiple cores of a modern system to make an without exiting an enclave – thus preventing a TLB flush. In this case, a set of  threads (proxy threads) on  dedicated cores are used to handle the s. Here, the parameters of an and other relevant data are sent to a proxy thread running on another core using an unsecure shared memory channel. The proxy thread reads the request and performs the operation. Once the operation is finished, the results are written to the shared memory region; these results are subsequently read by the enclave that issued the request. This is a standard pattern and is used to to hide
the overheads of system calls as well in regular operating systems.

We configured to use 8 cores for handling requests from enclaves.
In , this reduces the total number of dTLB misses by 60% thus improving the latency by 30%, as compared to the default implementation of (see Figure <ref>).

§ CONCLUSION

We introduced SGXGauge, a benchmark suite for Intel
SGX that captures a holistic view of the performance of
applications running in such TEEs – this includes the impact
of the EPC memory. SGXGauge contains diverse benchmarks
that affect different components of SGX. We also performed
an evaluation of the performance of SGX in LibOS mode and
showed that there is a marked difference in behavior as the
memory footprint crosses the EPC size limit.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

titleab - Apache HTTP server benchmarking tool - Apache
HTTP Server Version 2.4.
(Accessed on 06/21/2021).

titleIntel Launches Its Most Advanced Performance Data
Center Platform | Business Wire.
(Accessed on 06/15/2021).

titleManifest syntax — Graphene documentation.
(Accessed on 05/26/2021).

titlePageRank - Wikipedia.
(Accessed on 06/24/2021).

titleAcademic Research | Intel Software Guard Extensions |
Intel Software.
(Accessed on 11/18/2019).

titlememcached - a distributed memory object caching
(Accessed on 11/18/2019).

(Accessed on 11/18/2019).

titleRDTSC — Read Time-Stamp Counter.
(Accessed on 05/26/2020).

authorpersonReto Achermann.
titlemitosis-project/mitosis-workload-btree: The BTree
workload used for evaluation.
(Accessed on 10/03/2020).

authorpersonReto Achermann.
titlemitosis-project/mitosis-workload-hashjoin: The
HashJoin workload used for evaluation.
(Accessed on 10/03/2020).

[Ács and ColeşaÁcs and
authorpersonDávid Ács and
personAdrian Coleşa. year2019.
Securely Exposing Machine Learning Models to Web
Clients using Intel SGX.
journal2019 IEEE 15th International Conference on
Intelligent Computer Communication and Processing (ICCP)
(year2019), pages161–168.

[Ayoade, Karande, Khan, and HamlenAyoade
et al2018]
authorpersonGbadebo Ayoade, personVishal
Karande, personLatifur Khan, and personKevin
Hamlen. year2018.
Decentralized IoT Data Management Using BlockChain
and Trusted Execution Environment. In booktitle2018 IEEE
International Conference on Information Reuse and Integration (IRI).

[Bailleu, Dragoti, Bhatotia, and
FetzerBailleu et al2019]
authorpersonMaurice Bailleu, personDonald
Dragoti, personPramod Bhatotia, and personChristof
Fetzer. year2019.
TEE-Perf: A Profiler for Trusted Execution
Environments. In booktitle2019 49th Annual IEEE/IFIP
International Conference on Dependable Systems and Networks (DSN).

[Bao, Wang, Shi, Wang, Lei, and ChenBao
et al2020]
authorpersonZijian Bao, personQinghao Wang,
personWenbo Shi, personL. Wang, personHong
Lei, and personBangdao Chen. year2020.
When Blockchain Meets SGX: An Overview, Challenges,
and Open Issues.
journalIEEE Access  volume8
(year2020), pages170404–170420.

[Bashar, Avila, and DagherBashar
et al2020]
authorpersonG. Bashar,
personAlejandro Anzola Avila, and personGaby G.
Dagher. year2020.
PoQ: A Consensus Protocol for Private Blockchains
Using Intel SGX. In booktitleSecureComm.

[Bienia, Kumar, Singh, and LiBienia
et al2008]
authorpersonChristian Bienia, personSanjeev
Kumar, personJaswinder Pal Singh, and personKai
Li. year2008.
The PARSEC Benchmark Suite: Characterization and
Architectural Implications. In booktitleProceedings of the
17th International Conference on Parallel Architectures and Compilation
Techniques (Toronto, Ontario, Canada) (seriesPACT '08).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages72–81.

authorpersonAndre Bogus.
booktitleLighttpd installing, compiling,
configuring, optimizing, and securing this lightning-fast Web Server / Andre
publisherPackt Publishing, addressBirmingham,

[Brekalo, Strackx, and PiessensBrekalo
et al2016]
authorpersonHelena Brekalo, personRaoul
Strackx, and personF. Piessens.
Mitigating Password Database Breaches with Intel
journalProceedings of the 1st Workshop on System
Software for Trusted Execution (year2016).

authorpersonBYTE. year1995.
howpublished<https://www.math.utah.edu/ mayer/linux/bmark.html>.
(Accessed on 09/23/2019).

[Capps et alCapps
et al2019]
authorpersonDon Capps et al
titleIOzone Filesystem Benchmark.
(Accessed on 11/15/2019).

[Chakrabarti, Baker, and VijChakrabarti
et al2017]
authorpersonSomnath Chakrabarti, personB.
Baker, and personMona Vij. year2017.
Intel SGX Enabled Key Manager Service with
OpenStack Barbican.
journalArXiv  volumeabs/1712.07694

[Chang and LinChang and Lin2011]
authorpersonChih-Chung Chang and
personChih-Jen Lin. year2011.
LIBSVM: A Library for Support Vector Machines.
journalACM Trans. Intell. Syst. Technol.
volume2, number3, Article articleno27
(dateMay year2011), numpages27 pages.

[Che, Boyer, Meng, Tarjan, Sheaffer, Lee, and
SkadronChe et al2009]
authorpersonShuai Che, personMichael Boyer,
personJiayuan Meng, personDavid Tarjan,
personJeremy W. Sheaffer, personSang-Ha Lee, and
personKevin Skadron. year2009.
Rodinia: A Benchmark Suite for Heterogeneous
Computing (seriesIISWC '09). publisherIEEE
Computer Society, addressUSA, pages44–54.

[che Tsai, Porter, and Vijche Tsai
et al2017]
authorpersonChia che Tsai, personDonald E.
Porter, and personMona Vij. year2017.
Graphene-SGX: A Practical Library OS for Unmodified
Applications on SGX. In booktitleUSENIX Annual Technical

[Chen, Li, Ma, Guan, and JacobsenChen
et al2019]
authorpersonLixia Chen, personJian Li,
personRuhui Ma, personHaibing Guan, and
personHans-Arno Jacobsen. year2019.
EnclaveCache: A Secure and Scalable Key-Value Cache
in Multi-Tenant Clouds Using Intel SGX (seriesMiddleware
'19). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages14–27.

[Chen, Ailamaki, Gibbons, and MowryChen
et al2004]
authorpersonS. Chen, personA. Ailamaki,
personP.B. Gibbons, and personT.C. Mowry.
Improving hash join performance through
prefetching. In booktitleProceedings. 20th International
Conference on Data Engineering. pages116–127.

[Choi, Tian, Hernandez, Patton, Mood,
Shrimpton, Butler, and TraynorChoi et al2019]
authorpersonJoseph I. Choi,
personDave (Jing) Tian, personGrant Hernandez,
personChristopher Patton, personBenjamin Mood,
personThomas Shrimpton, personKevin R. B. Butler,
and personPatrick Traynor. year2019.
A Hybrid Approach to Secure Function Evaluation
Using SGX. In booktitleProceedings of the 2019 ACM Asia
Conference on Computer and Communications Security (Auckland, New Zealand)
(seriesAsia CCS '19). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Cooper, Silberstein, Tam, Ramakrishnan, and
SearsCooper et al2010]
authorpersonBrian F. Cooper, personAdam
Silberstein, personErwin Tam, personRaghu
Ramakrishnan, and personRussell Sears.
Benchmarking Cloud Serving Systems with YCSB. In
booktitleProceedings of the 1st ACM Symposium on Cloud
Computing (Indianapolis, Indiana, USA) (seriesSoCC '10).
publisherACM, addressNew York, NY, USA,

[Costan and DevadasCostan and
authorpersonVictor Costan and
personSrinivas Devadas. year2016.
Intel SGX Explained.
journalIACR Cryptology ePrint Archive
volume2016 (year2016), pages86.

[Dinh Ngoc, Bui, Bitchebe, Tchana, Schiavoni,
Felber, and HagimontDinh Ngoc et al2019]
authorpersonTu Dinh Ngoc, personBao Bui,
personStella Bitchebe, personAlain Tchana,
personValerio Schiavoni, personPascal Felber, and
personDaniel Hagimont. year2019.
Everything You Should Know About Intel SGX
Performance on Virtualized Systems.
journalProc. ACM Meas. Anal. Comput. Syst.
volume3, number1, Article articleno5
(dateMarch year2019), numpages21 pages.

[Erick LavoieErick Lavoie[n.d.]]
authorpersonDavid Erick Lavoie.
(Accessed on 06/19/2021).

[Fu, Bauman, Quinonez, and LinFu
et al2017]
authorpersonYangchun Fu, personErick
Bauman, personRaul Quinonez, and personZ. Lin.
Sgx-Lapd: Thwarting Controlled Side Channel Attacks
via Enclave Verifiable Page Faults. In booktitleRAID.

[Gebai and DagenaisGebai and
authorpersonMohamad Gebai and
personMichel R. Dagenais. year2018.
Survey and Analysis of Kernel and Userspace Tracers
on Linux: Design, Implementation, and Overhead.
journalACM Comput. Surv. volume51,
number2, Article articleno26 (dateMarch
year2018), numpages33 pages.

[Gribov, Vinayagamurthy, and GorbunovGribov
et al2019]
authorpersonA. Gribov, personDhinakaran
Vinayagamurthy, and personS. Gorbunov.
StealthDB: a Scalable Encrypted Database with Full
SQL Query Support.
journalProceedings on Privacy Enhancing
Technologies  volume2019 (year2019),
pages370 – 388.

[Harby, Fahmy, and AminHarby
et al2019]
authorpersonAhmed A. Harby, personS. Fahmy,
and personA. F. Amin. year2019.
More Accurate Estimation of Working Set Size in
Virtual Machines.
journalIEEE Access  volume7
(year2019), pages94039–94047.

[Hasan, Riley, and PonomarevHasan
et al2020]
authorpersonAisha Hasan, personRyan Riley,
and personDmitry Ponomarev. year2020.
Port or Shim? Stress Testing Application
Performance on Intel SGX. In booktitle2020 IEEE
International Symposium on Workload Characterization (IISWC).

authorpersonMarti A. Hearst.
Support Vector Machines.
journalIEEE Intelligent Systems
volume13, number4 (dateJuly
year1998), pages18–28.

authorpersonIntel. year[n.d.]a.
titleintel/intel-sgx-ssl: Intel® Software Guard
Extensions SSL.
(Accessed on 06/23/2021).

authorpersonIntel. year[n.d.]b.
titleIntel(R) Software Guard Extensions SDK Developer
Reference for Linux* OS.
(Accessed on 06/28/2021).

authorpersonIntel. year2019.
titleIntel SGX for Linux*.
(Accessed on 09/23/2019).

[Karande, Bauman, Lin, and KhanKarande
et al2017]
authorpersonVishal Karande, personErick
Bauman, personZhiqiang Lin, and personLatifur
Khan. year2017.
SGX-Log: Securing System Logs With SGX
(seriesASIA CCS '17). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Karl, Takeshita, and JungKarl
et al2020]
authorpersonRyan Karl, personJonathan
Takeshita, and personTaeho Jung.
Using Intel SGX to improve private neural network
training and inference.
journalProceedings of the 7th Symposium on Hot
Topics in the Science of Security (year2020).

[Kim, Park, Chang, Jeon, and HuhKim
et al2018]
authorpersonTaehoon Kim, personJ. Park,
personJ. Chang, personSeungheun Jeon, and
personJaehyuk Huh. year2018.
Secure In-memory Key-Value Storage with SGX.
journalProceedings of the ACM Symposium on Cloud
Computing (year2018).

[Kim, Park, Woo, Jeon, and HuhKim
et al2019]
authorpersonTaehoon Kim, personJ. Park,
personJaewook Woo, personSeungheun Jeon, and
personJaehyuk Huh. year2019.
ShieldStore: Shielded In-memory Key-value Storage
with SGX.
journalProceedings of the Fourteenth EuroSys
Conference 2019 (year2019).

[Krahn, Dragoti, Gregor, Quoc, Schiavoni,
Felber, Souza, Brito, and FetzerKrahn et al2020]
authorpersonRobert Krahn, personDonald
Dragoti, personFranz Gregor, personDo Le Quoc,
personValerio Schiavoni, personPascal Felber,
personClenimar Souza, personAndrey Brito, and
personChristof Fetzer. year2020.
TEEMon: A Continuous Performance Monitoring
Framework for TEEs. In booktitleProceedings of the 21st
International Middleware Conference (Delft, Netherlands)
(seriesMiddleware '20). publisherAssociation
for Computing Machinery, addressNew York, NY, USA,

[Kumar, Moolchandani, Ono, and SarangiKumar
et al2019]
authorpersonSandeep Kumar, personDiksha
Moolchandani, personT Ono, and personSmruti R
Sarangi. year2019.
F-LaaS: A Control-Flow-Attack Immune
License-as-a-Service Model. In booktitle2019 IEEE
International Conference on Services Computing (SCC).

[Kunkel, Quoc, Gregor, Arnautov, Bhatotia, and
FetzerKunkel et al2019]
authorpersonRoland Kunkel, personDo Le
Quoc, personFranz Gregor, personSergei Arnautov,
personPramod Bhatotia, and personChristof Fetzer.
TensorSCONE: A Secure TensorFlow Framework using
Intel SGX.

authorpersonKevin Lahey.
titleMonitoring Intel® SGX Enclaves.
(Accessed on 05/12/2021).

[Limaye and AdegbijaLimaye and
authorpersonAnkur Limaye and personTosiron
Adegbija. year2018.
A Workload Characterization of the SPEC CPU2017
Benchmark Suite. In booktitle2018 IEEE International
Symposium on Performance Analysis of Systems and Software (ISPASS).

[Lind, Priebe, Muthukumaran, O'Keeffe, Aublin,
Kelbert, Reiher, Goltzsche, Eyers, Kapitza, Fetzer, and PietzuchLind
et al2017]
authorpersonJoshua Lind, personChristian
Priebe, personDivya Muthukumaran, personDan
O'Keeffe, personPierre-Louis Aublin, personFlorian
Kelbert, personTobias Reiher, personDavid Goltzsche,
personDavid M. Eyers, personRüdiger Kapitza,
personChristof Fetzer, and personPeter R.
Pietzuch. year2017.
Glamdring: Automatic Application Partitioning for
Intel SGX. In booktitleUSENIX Annual Technical

[Liu, Wang, Wang, Gong, Zhao, and YewLiu
et al2020]
authorpersonXiming Liu, personWenwen Wang,
personLizhi Wang, personXiaoli Gong,
personZiyi Zhao, and personPen-Chung Yew.
Regaining Lost Seconds: Efficient Page Preloading
for SGX Enclaves. In booktitleProceedings of the 21st
International Middleware Conference (Delft, Netherlands)
(seriesMiddleware '20). publisherAssociation
for Computing Machinery, addressNew York, NY, USA,

[Lucani, Fehér, Fonseca, Rosa, and
DespotovLucani et al2018]
authorpersonD. Lucani, personMarcell
Fehér, personK. Fonseca, personM. Rosa, and
personBogdan Despotov. year2018.
Secure and Scalable Key Value Storage for Managing
Big Data in Smart Cities Using Intel SGX.
journal2018 IEEE International Conference on Smart
Cloud (SmartCloud) (year2018), pages70–76.

[Mahhouk, Weichbrodt, and KapitzaMahhouk
et al2021]
authorpersonMohammad Mahhouk, personNico
Weichbrodt, and personRüdiger Kapitza.
SGXoMeter: Open and Modular Benchmarking for Intel
SGX. In booktitleProceedings of the 14th European Workshop
on Systems Security (Online, United Kingdom)
(seriesEuroSec '21). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

authorpersonMoxie Marlinspike.
titleSignal >> Blog >> Technology preview: Private contact
discovery for Signal.
(Accessed on 06/24/2021).

[Matetic, Wüst, Schneider, Kostiainen,
Karame, and CapkunMatetic et al2019]
authorpersonSinisa Matetic, personKarl
Wüst, personMoritz Schneider, personKari
Kostiainen, personGhassan Karame, and personSrdjan
Capkun. year2019.
BITE: Bitcoin Lightweight Client Privacy using
Trusted Execution. In booktitle28th USENIX Security
Symposium (USENIX Security 19). publisherUSENIX
Association, addressSanta Clara, CA, pages783–800.

[McVoy and StaelinMcVoy and Staelin1996]
authorpersonLarry McVoy and personCarl
Staelin. year1996.
Lmbench: Portable Tools for Performance Analysis.
In booktitleProceedings of the 1996 Annual Conference on
USENIX Annual Technical Conference (San Diego, CA)
(seriesATEC '96). publisherUSENIX Association,
addressUSA, pages23.

[Melara, Freedman, and BowmanMelara
et al2019]
authorpersonM. Melara, personM. Freedman,
and personM. Bowman. year2019.
EnclaveDom: Privilege Separation for Large-TCB
Applications in Trusted Execution Environments.
journalArXiv  volumeabs/1907.13245

authorpersonAlex Merritt.
titlealexandermerritt/gups: gups mirror.
(Accessed on 06/23/2021).

authorpersonSaurav Mohapatra.
titlemohaps/libcatena: a simple toy blockchain written in
C++ for learning purposes.
(Accessed on 09/23/2019).

[Nitu, Kocharyan, Yaya, Tchana, Hagimont, and
AstsatryanNitu et al2018]
authorpersonVlad Nitu, personA. Kocharyan,
personHannas Yaya, personA. Tchana,
personD. Hagimont, and personH. Astsatryan.
Working Set Size Estimation Techniques in
Virtualized Environments: One Size Does not Fit All.
journalAbstracts of the 2018 ACM International
Conference on Measurement and Modeling of Computer Systems

authorpersonOpenSSL. year2019.
(Accessed on 12/07/2019).

[Orenbach, Lifshits, Minkin, and
SilbersteinOrenbach et al2017]
authorpersonMeni Orenbach, personPavel
Lifshits, personMarina Minkin, and personMark
Silberstein. year2017.
Eleos: ExitLess OS Services for SGX Enclaves
(seriesEuroSys '17). publisherAssociation for
Computing Machinery, addressNew York, NY, USA.

[Paladi, Svenningsson, Medina, and
ArlosPaladi et al2019]
authorpersonNicolae Paladi, personJakob
Svenningsson, personJorge Medina, and personPatrik
Arlos. year2019.
Protecting OpenFlow Flow Tables with Intel SGX
(seriesSIGCOMM Posters and Demos '19).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages146–147.

[Priebe, Vaswani, and CostaPriebe
et al2018]
authorpersonChristian Priebe, personKapil
Vaswani, and personManuel Costa.
EnclaveDB: A Secure Database Using SGX.
journal2018 IEEE Symposium on Security and Privacy
(SP) (year2018), pages264–278.

authorpersonWill Reese.
Nginx: The High-Performance Web Server and Reverse
journalLinux J. volume2008,
number173, Article articleno2 (dateSept.

authorpersonColin Robertson.
title__rdtsc | Microsoft Docs.
(Accessed on 04/02/2021).

authorpersonMark Russinovich.
titleAzure and Intel commit to delivering next generation
confidential computing | Azure Blog and Updates | Microsoft Azure.
(Accessed on 06/15/2021).

[Sergei Arnautov and Bohdan Trach and Franz
Gregor and Thomas Knauth and Andre Martin and Christian Priebe and Joshua
Lind and Divya Muthukumaran and Dan O’Keeffe and Mark L.
Stillwell and David Goltzsche and Dave Eyers and Rüdiger Kapitza and
Peter Pietzuch and Christof FetzerSergei Arnautov and Bohdan Trach and
Franz Gregor and Thomas Knauth and Andre Martin and Christian Priebe and
Joshua Lind and Divya Muthukumaran and Dan O’Keeffe and Mark
L. Stillwell and David Goltzsche and Dave Eyers and Rüdiger Kapitza and
Peter Pietzuch and Christof Fetzer2016]
authorpersonSergei Arnautov and Bohdan Trach and Franz
Gregor and Thomas Knauth and Andre Martin and Christian Priebe and Joshua
Lind and Divya Muthukumaran and Dan O’Keeffe and Mark L.
Stillwell and David Goltzsche and Dave Eyers and Rüdiger Kapitza and
Peter Pietzuch and Christof Fetzer. year2016.
SCONE: Secure Linux Containers with Intel SGX.
In booktitle12th USENIX Symposium on Operating Systems
Design and Implementation, OSDI 16). publisherUSENIX
Association, addressSavannah, GA, pages689–703.

[Shen, Tian, Chen, Chen, Wang, Xu, Xia, and
YanShen et al2020]
authorpersonYouren Shen, personHongliang
Tian, personYu Chen, personKang Chen,
personRunji Wang, personYi Xu, personYubin
Xia, and personShoumeng Yan. year2020.
Occlum: Secure and Efficient Multitasking Inside a
Single Enclave of Intel SGX (seriesASPLOS '20).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA.

[Shinde, Tien, Tople, and SaxenaShinde
et al2017]
authorpersonShweta Shinde, personDat Le
Tien, personShruti Tople, and personPrateek
Saxena. year2017.
Panoply: Low-TCB Linux Applications With SGX
Enclaves. In booktitleNDSS.

[Shun and BlellochShun and Blelloch2013]
authorpersonJulian Shun and personGuy E.
Blelloch. year2013.
Ligra: A Lightweight Graph Processing Framework for
Shared Memory.
journalSIGPLAN Not. volume48,
number8 (dateFeb. year2013),

[Taassori and ShaTaassori and Sha2018]
authorpersonMeysam Taassori and personAli
Sha. year2018.
VAULT : Reducing Paging Overheads in SGX with
Efficient Integrity Verification Structures.
journalProceedings of the Twenty-Third International
Conference on Architectural Support for Programming Languages and Operating
Systems (ASPLOS'18) (year2018), pages665–678.

[Tian, Zhang, Yan, Rudnitsky, Shacham, Yariv,
and MilshtenTian et al2018]
authorpersonHongliang Tian, personQiong
Zhang, personShoumeng Yan, personAlex Rudnitsky,
personLiron Shacham, personRon Yariv, and
personNoam Milshten. year2018.
Switchless Calls Made Practical in Intel SGX. In
booktitleProceedings of the 3rd Workshop on System Software
for Trusted Execution (Toronto, Canada) (seriesSysTEX
'18). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages22–27.

[Tramm, Siegel, Islam, and SchulzTramm
et al2014]
authorpersonJohn R Tramm, personAndrew R
Siegel, personTanzima Islam, and personMartin
Schulz. year2014.
XSBench - The Development and Verification of a
Performance Abstraction for Monte Carlo Reactor Analysis. In
booktitlePHYSOR 2014 - The Role of Reactor Physics toward
a Sustainable Future. addressKyoto.

authorpersonMarios Trivyzas.
titleHow we made joins 23 thousand times faster: part one
| Lab notes.
(Accessed on 06/24/2021).

authorpersonPeter Verhas.
titleLicense3j: Free Licence Management Library.
(Accessed on 11/18/2019).

[Wang, Liu, Su, Ma, Wang, Yang, Shen, Li,
Zhang, and DongWang et al2017]
authorpersonYongzhi Wang, personLingtong
Liu, personCuicui Su, personJiawen Ma,
personLei Wang, personYibo Yang, personY.
Shen, personG. Li, personTao Zhang, and
personXuewen Dong. year2017.
CryptSQLite: Protecting Data Confidentiality of
SQLite with Intel SGX.
journal2017 International Conference on Networking
and Network Applications (NaNA) (year2017),

[Wang, Ma, Zhang, Gao, Ding, and JiangWang
et al2020]
authorpersonYichuan Wang, personBing Ma,
personTong Zhang, personW. Gao,
personYifan Ding, and personXinyu Jiang.
Bottom-up: A SGX-based Blockchain Trusted Startup
and Verification Scheme.
journalProceedings of the 1st ACM International
Workshop on Security and Safety for Intelligent Cyber-Physical Systems

[Weichbrodt, Aublin, and KapitzaWeichbrodt
et al2018]
authorpersonNico Weichbrodt,
personPierre Louis Aublin, and personRüdiger
Kapitza. year2018.
SGX-Perf: A performance analysis tool for Intel
SGX enclaves.
journalProceedings of the 19th International
Middleware Conference, Middleware 2018 (year2018),

[Weisse, Bertacco, and AustinWeisse
et al2017]
authorpersonOfir Weisse, personValeria
Bertacco, and personTodd Austin.
Regaining Lost Cycles with HotCalls: A Fast
Interface for SGX Secure Enclaves.
journal2017 ACM/IEEE 44th Annual International
Symposium on Computer Architecture (ISCA), pages81–93.

[Yadalam, Ganapathy, and BasuYadalam
et al2021]
authorpersonSujay Yadalam, personV.
Ganapathy, and personArkaprava Basu.
SGXL: Security and Performance for Enclaves Using
Large Pages.
journalACM Trans. Archit. Code Optim.
volume18 (year2021), pages12:1–12:25.

[Yan and SuYan and Su2009]
authorpersonXin Yan and personXiao Gang
Su. year2009.
booktitleLinear Regression Analysis: Theory and
publisherWorld Scientific Publishing Co., Inc.,

[Zhao, Saifuding, Tian, Zhang, and XingZhao
et al2016]
authorpersonChongChong Zhao, personDaniyaer
Saifuding, personHongliang Tian, personYong Zhang,
and personChunXiao Xing. year2016.
On the Performance of Intel SGX. In
booktitle2016 13th Web Information Systems and Applications
Conference (WISA). publisherIEEE, pages184–187.

[Zhou, Cai, Peng, Wang, Ma, and LiZhou
et al2021]
authorpersonWenchao Zhou, personYifan Cai,
personYanqing Peng, personSheng Wang,
personKe Ma, and personFeifei Li.
VeriDB: An SGX-based Verifiable Database.
journalProceedings of the 2021 International
Conference on Management of Data (year2021).

§ LATENCIES

Instrumenting an application running in the secure mode within is a non-trivial task due to the constraints imposed by .
instructions <cit.>, which are generally used to measure the cycles taken by an operation, are not allowed in . Fortunately, the driver code does not execute in and thus can be easily instrumented.

We measured the latencies of the core operations: allocating a page (), evicting a page (), loading back a page (), and handling a page fault in (). We use the ftrace tool <cit.> for this purpose.
uses the  instruction to evict a page from the EPC and the  instruction to load it back.
While evicting  an EPC page, first its MAC is calculated and then the page is encrypted. While loading back, the page is decrypted and its integrity is checked using the MAC <cit.>.

These functions are highly optimized with latencies in the range of a few micro-seconds. Figure <ref> reports the  mean of 40K+ samples. The latency of evicting an EPC page is $16\%$ more than loading back an EPC page. evicts pages in a batch that is typically 16 pages. However, during a fault, a single page is loaded back.

Latency of core operations in .

§ MODE PERFORMANCE

Here, we discuss the overheads in the 6 of workloads from while executing in the mode. We use the change in the most relevant hardware performance counters for this discussion. A heat-map of these counters is shown in Figure <ref>.

Overheads for the workloads when executing in the mode w.r.t the mode.

As seen in Figure <ref>, while executing in the mode, the total number of dTLB misses is  $\approx$ 2000$\times$ more than the mode. This is because in our implementation of the hash function is protected inside an enclave. This is called via s from many threads that are executing in the unsecure region. This results in many enclave transitions and thus many TLB flushes. These TLB entries are to be populated for every by a page table walk. Hence, we see a similar increase in the number of walk cycles.

With 16 threads in the setting there are $\approx$ 3,133 K s, for $\approx$ 4,831 K s, and for the setting there are $\approx$ 8,944 K s.

In , the number of EPC evictions increases from 389 K to 433 K to 721 K, as we increase the input size from  to and , respectively. Due to this, in the setting, the total number of enclave exits increases, thus increasing the total number of dTLB misses by 131$\times$ and walk cycles by 196$\times$ w.r.t. the mode.

In , the total number of EPC evictions increases from 79 K to 116 K to 305 K, when we move from the to and then to the input setting, respectively. However, here the total number of dTLB misses increase by only 2.2$\times$ only in the setting.
This is because the total number of dTLB misses is dominated by the number of page faults caused by the workload. To serve a page fault, an enclave performs an asynchronous exit (AEX), which also causes a TLB flush.
The total number of page faults increases from 3$\times$ in the setting to 7.5$\times$ in the setting, and the total LLC misses increase from $\approx$1 in the setting to $\approx$6.4 in the setting.

In , on increasing the input size we observe an increase in almost all of the performance counters. Most notably, the total number of page faults and dTLB misses increase by $\approx$246$\times$ and $\approx$140$\times$ over mode in the input setting, respectively.
This is due to the characteristics of the workload. A typical hashjoin operation incurs many cache misses and stall cycles <cit.>.

In , the total number of page faults increase by 3$\times$ as compared to the mode. However, we do not observe a large impact with the increase in the input size. This is because of the inherent locality in the workload.

In , we observe a decrease in the total number of walk cycles on increasing the input size. This is because dTLB misses also go down with an increasing input size. The main reason for this is the nature of the workload. In the mode (not shown in the Figure), the number of dTLB misses increases by 3.6$\times$ when we increase the input size from the to the setting. Hence, the nature of the workload dominates the total number of misses, hiding the extra misses caused due to .

§ COUNTER IMPACT ON PERFORMANCE

provides a way to execute an application securely on a remote machine, although with some limitations. Researchers are working on developing methods to circumvent these limitations.
Different solutions might affect the components of differently. Here we provide a generic approach for the developer to select correct benchmarks from as per the requirement.

As pointed out in prior work and observed by us in our experiments, when a benchmark reserves more memory than the EPC size it suffers a slowdown. Usage of more memory than the EPC may impact the total number of dTLB misses, LLC misses, walk cycles, stall cycles, and EPC-Evictions.

We rank the metrics in the order of importance for each of the workloads in . We use linear regression <cit.> for this purpose. Linear regression predicts the execution time given these metrics as input. While doing so, it assigns coefficients to these metrics. The magnitude of these coefficients is correlated with the importance of that metric in determining the execution time (see Table <ref>).

Table showing the most important hardware performance counter that determines the performance of each workload (shown in bold). LLC refers to the last level cache.

Workloads          Walk cycles     Stall cycles     Page faults     dTLB misses     LLC misses     EPC evictions
LightCyan 7|c|mode
Blockchain     0.33             0.32              0.01             0               0.32            0
OpenSSL        0.08             0.12              0.17             0               0.21            0.14
BTree          0.27         $-$0.11         0.22         $-$0.05        0.22        0.11
HashJoin       0.21         0.16          0.21         $-$0.03        0.21        0.21
BFS            0.10         0.17          0.18         0.21         0.09        0.22
PageRank       0.44         $-$0.54         0.05         0.33         0.65        0.04
LightCyan 7|c|mode
Memcached      0.03            0.04             0.09            0.15            0.13           0.09
XSBench        0.16            0.17             0.17            0.18            0.16           0.17
Lighttpd       0.18            0.19             0               0.09            0.26           0
SVM            0.09            0.60             0.27            0.09            0.31           $-$0.03

We can conclude that most of the time paging and TLB-related counters are the most correlated with the performance. LLC misses are mostly an important factor in .

§ START-UP OVERHEAD

Here, we discuss the overhead in initializing .

Figure showing the performance counter values for EPC page allocation, eviction, and load-back during the execution phase of in the mode (N-) and mode (G-).

verifies the signature of an enclave prior to its execution. To do so, it loads the entire enclave into the EPC. In v1, a heap size greater than the EPC size was not allowed, as that will not allow to load the complete enclave in the EPC. Since v2, a heap size greater than the EPC is allowed. transparently evicts and loads pages as per its requirement.

Figure <ref> shows the allocation, eviction, and load back of EPC pages in and modes for a representative workload, . This pattern remains the same across other workloads also. first calculates the signature of the enclave that causes the initial EPC evictions. Note that EPC pages are allocated after the verification is done. After that, the EPC pattern of is the same as that of the mode.

recommends setting the enclave size as per the maximum requirement of the application. However, in our experiments we observed additional overheads on setting a lower value of the enclave size in mode. This is related to how initializes the enclave. We thus used an enclave size of 4 GB for all our experiments. We do not count the start up time in the workload executing time while calculating the overheads in Section <ref> mainly because this
is an one-time activity and a workload can run for a very long time after its enclave is initialized. Also note in
Figure <ref> that after the initialization phase the gray () and black () lines
converge (same behavior).

§ WHAT ABOUT  I/O?

As mentioned before, does not support system calls, notably file system calls. An enclave needs to rely on an to read or write a file to the file system.
In this case, by default, the data is transferred in plaintext and it is the responsibility of the developer to protect it via encryption. has a sealing feature, where the data can be encrypted using the sealing enclave <cit.>. The sealing enclave is an Intel-authored enclave that is part of the Intel SDK. It can seal or encrypt data using a platform dependent hardware key. The sealed data can only be unsealed or decrypted on the same platform, and optionally, it can be configured to be decrypted only by the same enclave that encrypted it.

Library operating systems support file system operations by transparently capturing these calls and handling them either via an or via a parallel, proxy thread executing on a different core. However, as this is not covered by constructs, a naive implementation will still write the data in plain text to the  file system, essentially leaking data. supports transparently encrypting files before they are written to the file system. This feature is known as the protected file system or PF <cit.> mode. However, as shown in Figure <ref> this feature is not optimized, and the performance of an I/O intensive application can suffer by up to $98\%$ when PF is used.

Read performance

Write performance

The I/O overhead with (S-G) and with protected files (S-P). : reading and writing 1 GB of data with 4 M blocks.

We use the popular file system benchmark Iozone <cit.> to evaluate the performance of the PF system. We compare this against the mode, and mode without using the protected file setting.
incurs an overhead of 33%  and 36% compared to the mode for read and write operations, respectively. The overhead increases to $98\%$ and $95\%$ for read and write operations, respectively, when the protected files mode is enabled. The main reason for this is the increase in the number of s (see Figure <ref>) and s (see Figure <ref>).

The PF mode needs to be optimized to make it practical for production-quality systems.