# Leveraging the SCION Internet Architecture to Accelerate File Transfers over
BitTorrent

Marten Gartner  Otto-von-Guericke University
Magdeburg, Germany
<EMAIL_ADDRESS>Thorben Krüger  Otto-von-Guericke University
Magdeburg, Germany
<EMAIL_ADDRESS>David Hausheer  Otto-von-Guericke University
Magdeburg, Germany
<EMAIL_ADDRESS>
###### Abstract

As the needs of Internet users and applications significantly changed over the
last decade, inter-domain routing became more important to fulfill these
needs. The ways how data flows over the Internet are still completely in the
hand of network operators, who optimize traffic according to their own, local
view of the network. We observe two potential limitations from this:
Optimizing according to the local view may a) result in unused capacities in
the global network and b) not meet the actual needs of users and applications.
To identify and overcome these limitations, we present our BitTorrent over
SCION approach, which enables multipath communication and intelligent path
selection for endhosts in global torrent networks. We compare our
implementation against BitTorrent over BGP and BGP-M in a small-scale Internet
topology, observing an increase in goodput of 48% through multipathing
compared to BitTorrent over BGP and 33% compared to the BGP-M candidate.
Furthermore, we show that our proposed disjoint path selection algorithm is
able to improve traffic flow in the network with a low number of outgoing
connections to unchoked peers.

###### Index Terms:

Peer-to-Peer, Path-aware networking, Path Selection, SCION, BitTorrent

##  I Introduction

The Internet was designed decades ago, and some of the original design
decisions have had unfortunate side-effects and security implications that
persist to this day. So far, from the perspective of a mere host, there has
been no general mechanism for splitting traffic across multiple different
paths to a specific destination, which would be beneficial bandwidth-intensive
applications.

As the de facto standard protocol for inter-domain routing, BGP [32] is used
to disseminate routes between Autonomous Systems (ASes), forming the Internet
as we know it today. Within each AS on a given path through the Internet,
packet forwarding is affected by (sometimes highly complex) local traffic
engineering preferences and policies. In general however, there is just a
single packet forwarding path between two hosts in a BGP-based Internet. To
also add support for multiple forwarding paths, BGP-M [24] was proposed, which
adds support for load sharing across multiple inter-domain links, subject to
configurable preferences. However, BGP-M is does little to help optimizing
traffic flow in the global Internet: Firstly, it’s impact is necessarily of
limited, local scope. Secondly, BGP-M is not adaptive and can not dynamically
react to changing network conditions. Given these limitations and given the
existence of alternative inter-domain paths in the Internet [2], we
hypothesize that the current Internet still has unused capacities that cannot
be exploited entirely by using BGP or BGP-M as the inter-domain routing
protocol.

Path-aware networking architectures promise to overcome these limitations by
offering deeper insight into and better control over packet forwarding in the
network. Different approaches to enable path-control in networks have been
proposed. Some try to work within the limits of the current Internet
architecture [41, 18], other attempts involve a complete redesign of the
Internet architecture from scratch. While there are a number of path-aware
approaches [42, 34] with their own merits, our work focuses on the (arguably)
most mature and most widely-deployed open-source SCION architecture [7, 44].

To leverage unused capacities in the network, especially in the backbone,
Peer-to-Peer _(P2P)_ applications promise unique opportunities through their
globally distributed nature [36, 8, 39]. As the most well-researched and
understood protocol, we selected BitTorrent as the foundation of our work,
adding support for active endhost-based path-selection via SCION. We compare
our augmented BitTorrent implementation with a non path-aware BitTorrent
implementation in comparable BGP and BGP-M-based inter-domain network
topologies.

We anticipate that any achieved improvements will easily translate to other
P2P networks, e.g. IPFS [38]. While BitTorrent itself is famous for improving
bandwidth utilization on the last mile, we hypothesize that in combination
with SCION, it could also unlock unused capacities in the network at large. To
this end, we contribute the following:

  * •

We discuss the existence of unused capacities in the backbone with BGP or
BGP-M as utilized inter-domain routing protocol

  * •

We show why path-aware networking is able to unlock these capacities and
discuss the impact of host-based path selection

  * •

We present our BitTorrent over SCION design introducing the notion of path-
level peers and propose an algorithm for disjoint path selection

  * •

Finally, we analyze the performance improvements of BitTorrent over SCION
aggregating unused capacities in the network compared to BitTorrent over BGP
and
BGP-M

The remainder of this work is structured as follows: In Section II we provide
background for SCION and BitTorrent, followed by a discussion about
limitations of BGP-based deployments and impacts of host-based path selection
in Section III. We present the design and implementation of multipath support
for BitTorrent over SCION in Section IV, followed by a presentation of our
disjoint path selection algorithm. Afterwards, we show our virtualized
Internet-scale testbed in Section V and the experimental results of comparing
Multipath BitTorrent over SCION against an unmodified implementation in BGP
and BGP-M-based scenarios in Section VI. Finally, we discuss related work in
Section VII, conclude and provide outlook for future work in Section VIII.

##  II Background

In the following, we provide a brief overview on BGP, BGP-M and the SCION
architecture as well as on BitTorrent.

###  II-A BGP and BGP-M

BGP plays a unique role in the current Internet. While it can also enable in
intra-domain routing (iBGP), we will exclusively refer to BGP’s more important
role as the Internet’s predominant inter-domain routing protocol in the rest
of this work. The principle task of a BGP border router is to inform and
update its neighbours about specific IP address ranges (i.e., IP prefixes), to
which the router’s AS is able to forward traffic to. The router announces the
prefixes that its AS has learned from its other neighbors. Routing loops are
prevented by means of the _AS_PATH_ , a list of hops to which each router
prepends its own AS Number (ASN) before sending it on as part of a route
announcement. BGP routers maintain the respective AS_PATHs as well as the
announced prefixes in a routing table. The forwarding destination is
determined by referencing this table based on the destination address of an
incomping packet. By default, BGP selects a single path for each match in the
routing table according to configurable policies. These policies may reflect
e.g., local forwarding preferences, filters constraining the number of
exported routes, etc.

To account for the fact that there often are multiple matching paths to a
particular destination, multipath BGP _(BGP-M)_ [15] was introduced. For
certain cases, BGP-M adds support for traffic load splitting111Load splitting
is based on flow hashes of the 5 tuple _(src address, dst address, src port,
dst port, layer 4 protocol)_. over multiple outgoing AS links. In the most
simple case, if several possible forwarding AS_PATHs are of the same length,
Equal-Cost Multipath Routing _(ECMP)_ can be applied, with a configurable
maximum number of parallel paths222In some border router implementations, the
relevant settings are: a) bgp bestpath as-path multipath-relax to enable load
splitting for equal length paths, and b) maximum-paths to set the maximum
number of paths over which load splitting will be performed.. In Addition,
BGP-M also allows for more advanced policies on how to combine multiple paths,
e.g., some Cisco routers offer Unequal-Cost Load Sharing [21], which allows
load splitting over paths with different length by using a dedicated
configurable weight.

###  II-B SCION

The SCION architecture [44] has been designed to overcome the limitations of
BGP and BGP-M in the future Internet, addressing modern threat models at the
fundamental protocol level and endeavouring to avoid many current issues with
hijacking attacks and single roots of trust. SCION also promises communication
guarantees and path control capabilities, allowing applications to use two or
more paths in parallel to a given destination, generally enabling _multipath
communication_.

For traditional multipath approaches like MPTCP [29] and MPQUIC [9] to work as
intended, a host must provide multiple network interfaces. With these
protocols, multipath communication implies sending data over multiple
interfaces in parallel, beyond which no further influence on the data
forwarding paths is possible. SCION on the other hand is based around ”packet
carried forwarding state” (PCFS), where every packet contains the complete
inter-AS path to the intended destination (in the form of _hop fields_) in its
packet header. Packets can thereby be easily directed via different paths, by
changing the SCION header alone. Our work heavily relies upon this precise
_path awareness_ property of the SCION architecture.

Unlike the flatter organizational hierarchy of today’s BGP-based Internet, in
SCION, collections of ASes are combined into Isolation Domains, _(ISDs)_ ,
which are envisioned to correspond to, e.g., geographical regions or
legislative domains (like a single country), but can also be made up of
company or research networks. One or multiple ASes in an ISD form the _ISD
Core_ , which collectively manages a cryptographic trust root (_TRC_) on
behalf of the other ISD members, enabling service authentication and other
cryptographic functions within the ISD, simultaneously avoiding many of the
notorious problems that plague globally centralized cryptographic trust
systems with single points of failure that are beyond the control of the ISD.
The ISD Core also manages the exchange of path information with other ISDs,
while independent peering is also possible among non-Core ASes of different
ISDs.

###  II-C BitTorrent

The BitTorrent protocol specifies file transfer as a distributed mechanism
between peers without the need for any central coordination. Some initial way
to exchange network addresses among peers is nevertheless required, e.g., by
means of a _tracker_ or, alternatively, a distributed hash table (DHT). In
BitTorrent, files are typically not transferred in the usual form of a single,
continuous byte stream that contains the complete file. Instead, large files
are split into equal-sized _pieces_ (typically with a size between 32KB and
256KB). It is a key feature of BitTorrent that exchange of these pieces can be
easily parallelized. BitTorrent peers that already have a complete local copy
of a file are referred to as _seeders_ , as opposed to _leechers_ , which
still have to obtain some or all of the constituent pieces of the file from
other peers. For each new file uploaded to the BitTorrent network, there must
be at least one seeding peer that initiates the distribution of the file.

##  III Host-Based Path Selection

###  III-A Limitations of the BGP-Based Inter-Domain

Generally, two critical aspects of BGP and BGP-M for inter-domain routing may
lead to unused capacities in the global Internet: Firstly, AS operators
inherently only have limited insight into network conditions beyond their
local network and can also only manage traffic locally. Secondly, BGP and
BGP-M do not provide features to dynamically adapt routing to different needs.
In this section, we further characterize these limitations before discussing
possible improvements that path-aware networking could bring to the table.

Despite the large benefits of traffic engineering of AS-operators on intra-
domain level, the potential of optimizing inter-domain routing is limited in
the current Internet. Each operator can only optimize the traffic flow in
their own AS until it reaches its local destination or the neighbour AS. This
may especially impact performance for flows that traverse multiple hops before
they reach their destination, since each hop performs its own, local
optimization. In case one of the first hops performs a non-optimal routing
decision for the flow (e.g. routing to particular neighbour interfaces that
are already under heavy load), the overall performance of the flow is
affected.

As discussed in Section II, BGP-M provides several options to perform load
sharing on multiple links. However, these options need to be configured
statically in the network. Consequently, the network can not always fulfill
the varying needs of different participants, e.g., endhosts who prefer to
optimize for different criteria. BGP-M reflects the anticipated needs of AS
operators, not the actual needs that endhosts have, which may differ
significantly from those anticipated by the AS operators.

Path-aware networking promises to overcome many of these limitations and their
impacts on performance, and, (in the case of SCION,) gives endhosts the
opportunity to freely choose suitable inter-domain paths for their traffic.

###  III-B Implications of Host-Based Path Selection

Traffic engineering on the Internet is performed by network operators
attempting to locally optimize data flows for various factors. With host-based
path selection, operators hand over this control to endhosts. While this
promises to help endhosts to optimize their traffic, it comes with potential
implications for network operators and may be at odds with their own
interests, especially with respect to the inter-domain. Operators tend to
prefer the use of peering links over that of transit links to avoid costs,
while endhosts do not have such an incentive to avoid transit links.
Additionally, host-based path selection ideally requires up-to-date insight
into network conditions on all hosts. It also needs to ensure that hosts do
not change paths too often, which could result in undesirable oscillation. In
simulation, Scherrer et al. show that the impact of oscillation through host-
based path selection is low[37]. Moreover, in SCION, endhosts can only dictate
the ingress and egress router interfaces of ASes, allowing network operators
to still optimize their internal traffic within these constraints. Overall,
while the benefits, drawbacks and trade-offs of path-awareness are certainly
not yet fully understood, there is significant potential for it in the the
future Internet.

##  IV Design and Implementation of BitTorrent over SCION

In this section, we present our approach of path-level peers to enable
multipath support for BitTorrent over SCION, followed by our disjoint path
selection algorithm.

###  IV-A Multipath through Path-Level Peers

As one of its features, SCION provides path control for inter-AS traffic while
guaranteeing that the traffic flows along the chosen paths. This opens up the
potential to aggregate capacities in the network, enabling applications to
leverage multipath communication and parallel data processing to increase
performance. In this work, we choose BitTorrent as a suitable, already
parallelized application as a foundation for our experiments on bandwidth
aggregation via multiple SCION paths.

By default, a BitTorrent peer is identified solely by its network address and
port. This address could be either an IPv4, IPv6 or, in our case, a SCION
address. We will refer to peers that are only described by their address as
_address-level_ peers for the rest of this work. To distinguish between
multiple SCION paths to a particular peer, we introduce a new representation
that we will refer to as a _path-level_ peer. They are represented by the
tuple $(addr,path)$, consisting of the peer’s SCION address (including the
port), together with one possible path to this address.

Figure 1:  Multipath implementation by downloading a torrent file over
multiple paths (connections).

We introduce this notion of path-level peers to a path-aware BitTorrent
implementation which we will henceforth refer to as _BitTorrent over SCION_ ,
and which treats different paths to the same peer as several distinct, path-
level peers.

Generally, peers that are returned from a tracker or that are added via static
bootstrapping333I.e., providing a list of peer addresses as arguments to the
client binary. to BitTorrent are address-level peers, since they may not be
SCION peers to begin with and may not have path information associated with
them. Thus, to generate path-level peers from a given SCION address, an
additional path lookup is required. For a resulting number of possible paths
that are available to a peer, the same number of corresponding path-level
peers can be generated.

BitTorrent over SCION is configured with an upper bound for the number of
different path-level peers, as the default BitTorrent client is, too. Path-
level peers are obtained from address-level peers through a path selection
algorithm.

After this obtaining of path-level peers, the usual BitTorrent P2P algorithms
operate over each path independently. For each path-level peer, a QUIC444We
choose QUIC, because TCP is currently not yet implemented for SCION.[23]
connection is established to ensure reliable transfer of data. Figure 1 shows
the piece download of a particular file. Each successfully established
connection fetches piece information from a queue and requests the particular
piece by sending request messages and wait for peers sending back the
requested pieces. After each received piece, its integrity is verified. For
this, a hash is computed and verified against the one referenced in the
torrent file for that piece. Retrieved pieces are stored in main memory until
the file is downloaded completely.

Since requests and retrievals of pieces over different connections are handled
in their own dedicated threads, pieces can easily be downloaded concurrently,
which speeds up the process and improves the overall download bandwidth. It is
the responsibility of the main thread to iteratively check the result bitmap
for missing pieces. Once all pieces are retrieved, the main thread closes all
connections to all still connected peers that serve pieces of the current
torrent and assembles the complete file to disk.

###  IV-B Upload-based Disjoint Path Selection

Although BitTorrent over SCION may simply use all available paths to each
connected peer, there are reasons to limit the number of path-level peers.
When this limit is set suitably high, all available paths to each peer are
considered. However, multiple paths may share the same bottleneck, making it
pointless to aggregate them in hope of increasing overall performance. Since
each BitTorrent peer has an upper limit of outgoing connections to neighbours,
a performance increase could also be expected by simply increasing this upper
limit to exchange pieces with more peers. Consequently, using the shortest
path or simply aggregating all paths does not promise to have significant
impacts on BitTorrent over SCION’s performance compared to path-unaware
BitTorrent implementations. To address this, we use built-in SCION
capabilities to implement an improved, _disjoint_ path selection strategy,
aiming to avoid such shared bottlenecks.

In BitTorrent over SCION, our improved path selection strategy relies on two
core assumptions: 1) Different paths that share the same hop may also share a
bottleneck at this hop and 2) Peers that offer pieces to others are in a good
position for path selection decisions with knowledge about all downloading
peers, allowing them to strategically distribute their outgoing traffic via
disjoint paths.

We implement a disjoint path selection strategy by searching for overlaps in
all paths and discard all but one of the paths that share the same hops,
following assumption 1). With respect to assumption 2), we decide to delegate
path selection exclusively to the uploading peer. Both in combination promise
to outperform naïve path selection approaches.

As presented in Section II, a SCION path consists of multiple hops. Each hop
contains ingress and egress interfaces of the respective AS. In SCION, the
interfaces are represented as numeric IDs, that are unique within the given
AS. The combination of the AS identifier with the interface number results in
a globally unique _interface ID_. In Figure 2, we show an intuitive mapping of
a visual representation of hops and their interfaces to a list of such
interface IDs. The interface IDs are used to determine disjointness between
paths by counting the number of identical interface IDs.

Figure 2:  List-based representation of hop interfaces using unique ids to
perform conflict detection.

Data: $peers,maxOutgoingConns$

Result: $pathLevelPeers\leftarrow[\>]$;

$allPaths\leftarrow[\>]$;

for  _$p\in peers$_ do

$paths\leftarrow lookupPaths(peer)$;

$allPaths\leftarrow append(allPaths,paths)$;

end for

for  _$path1\in allPaths$_ do

for  _$path2\in allPaths$_ do

if  _$path1!=path2$_ then

$confs\leftarrow numConflicts(path1,path2)$;

$path.conflicts+=confs$;

end if

end for

end for

$allPaths\leftarrow sortByConflictsAndHops(allPaths)\;$

$i\leftarrow 0$;

while  _$i\leq maxOutgoingConns$_ do

$pathLevelPeer\leftarrow fromPath(allPaths[i])$ $pathLevelPeers\leftarrow
append(pathLevelPeers,pathLevelPeer)$;

$i\leftarrow i+1$

end while

Algorithm 1 Disjoint path selection

Based on the interface IDs in the SCION paths, the uploading peer is able to
perform disjoint path selection to all connected peers. To achieve this, an
interested peer connects over the first available path to the uploading peer
and waits for it to connect back. The uploading peer now applies its disjoint
path selection and connects back to the interested peer over the selected path
set. Algorithm 1 depicts the procedure of finding the least disjoint paths to
all connected address-level peers and returning a proper list of path-level
peers. At first, the paths to each address-level peers are determined in a
loop and aggregated in the _allPaths_ variable. Afterwards, each path in
allPaths is checked against all other paths calculating the number of
conflicts (i.e. conflicting interface IDs), which is saved in the path. Next,
the allPath list is sorted in ascending order by the number of conflicts and
the number of hops. Finally, until the number of _maxOutgoingConns_ is
reached, the algorithm iterates over allPaths and transforms each path into a
path-level peer, which is stored in the return variable _pathLevelPeers_.

##  V Benchmark Setup: A Representative Small-Scale Internet Testbed

To compare our BitTorrent over SCION implementation against a BGP-based setup,
we choose to run a virtualized network of multiple ASes that reflects the
topology of the current Internet in a small-scale setup. Figure 3 shows the
designed topology that we used to evaluate BitTorrent over SCION.

The topology consists of two core layers which represent Tier-1 and Tier-2
ASes in the Internet, connected via peering and transit links. With the
growing popularity of IXPs, the number of Tier-3 ASes decreased significantly
over time. Consequently, our testbed does not contain any Tier-3 ASes. The
topology design follows actual AS and peering data, obtained from CAIDA data
[6, 4, 5] and peeringdb [28], with randomized AS numbers. We limit the network
link capacities to 15 Mbit/s for Tier-1 links and 10 Mbit/s for all remaining
links to make all candidates network bound. Otherwise, the CPU could limit
candidates resulting in potentially biased results. We derive two torrent
networks from our proposed topology: The first network 5ASes consists of the 5
ASes marked dark green (AS102, AS1002, AS1004, AS103, AS1006) and the second
network consists of the 10 ASes marked dark and light green (AS102, AS1002,
AS1004, AS103, AS1006, AS1001, AS101, AS04, AS105, AS1009). ASes are connected
either with transit or peering links. Our topology follows the _valley-free_
routing [30]. Consequently, peering links can only be used by the peering
neighbours and their customers. In BGP and BGP-M, this is implemented with
prefix lists on each AS that has peering links. Since filtering support of
peering links in SCION is currently in progress, we apply a static path filter
to each AS to filter out valley-free violations. Since ISD’s are a concept
exclusively for SCION, we locate all SCION ASes in the same ISD, to achieve
better comparability.

In our evaluations, we run three different BitTorrent candidates: The first
one is BitTorrent over BGP, which is a usual BitTorrent client implemented in
Go based on inter-domain routing performed by BGP. The second candidate is
BitTorrent over BGP-M, which uses the same BitTorrent client but with
configured load splitting in BGP in each AS. Despite other interesting
approaches for load splitting, we apply ECMP-based load splitting for BGP-M in
our testbed, since it is the most deployed approach in the current Internet.
Our third candidate is BitTorrent over SCION, which implements the disjoint
path selection based on the presented idea of path-level peers.

The complete topology is running on a bare-metal server in a virtualized
environment based on Docker. Each AS runs one or more containers (routers,
hosts). Multiple ASes are connected via Docker Bridge Networks [13]. The
server is running an Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz with 36 threads
and 500GB of main memory.

Figure 3:  Virtualized testbed of a representative Internet topology

##  VI Evaluation

After presenting our virtualized Internet topology, we evaluate our BitTorrent
over SCION approach in this section.

###  VI-A Terminology

The following parameters are used to create and evaluate different BitTorrent
experiments:

  * •

_MaxPeers_ : Number of available peers in the torrent

  * •

_OutgoingConns_ : Number of outgoing connections (to peers) that each peer
instantiates

  * •

_NumASes_ : Number of involved ASes in the experiment

To design our experiments, we stick to findings from Hamra et al. [19]
measuring BitTorrent’s performance in real-world torrents. In most cases,
MaxPeers is significantly higher than OutgoingConns, meaning each peer
exchanges pieces with a subset of all availables peers. Hamra et al. show that
setting OutgoingConns to the half of MaxPeers is a good tradeoff. Furthermore,
the MaxPeers parameter changes over time in real-world torrents. This number
increases often at the beginning of the torrent when many peers are interested
in downloading the file and decreases after the majority of peers finished
downloading and leave the torrent. We adapt this behaviour for our
experiments.

In the following, we present the results of our two experiments: At first, we
evaluate how heavy multipathing implemented in BitTorrent over SCION can
aggregate bandwidth in the network that is not available for BitTorrent over
BGP/BGP-M. Afterwards, we compare BitTorrent over SCION against BitTorrent
over BGP/BGP-M with a varying number of OutgoingConns to evaluate the effect
of our disjoint path selection approach.

###  VI-B Bandwidth Aggregation

In our first experiment, we run 20 BitTorrent peers in the torrent networks
5ASes and 10ASes exchanging a 100Mbyte file. By choosing 2 different torrent
network sizes with a fixed-size topology, we can evaluate the impact of
density of peers and the number of additional ASes that are not participating
in the torrent, and consequently may provide additional capacities. We set the
OutgoingConns parameter to infinity in this experiment (in detail it is set to
the maximum number of path-level peers that one peer can connect to) to
evaluate the maximum possible goodput that all candidates can achieve.

Figure 4a) shows the aggregated goodput in percent of all peers for the three
candidates, while BGP serves as baseline with 100%. We decide to compare the
goodput instead of the overall bandwidth, since SCION packets have a larger
header than plain IP packets.

For the torrent network 5ASes, we observe an aggregated goodput of 111% for
BGP-M compared to BGP, while SCION achieves 148% compared to BGP. From these
results, we observe that enabling multipath BGP via ECMP increases the overall
goodput by 11%. We conclude that in the 5ASes torrent network, the number of
equal length BGP paths are comparatively low, leading to a small increase of
goodput. However with BitTorrent over SCION’s approach of path-level peers, a
48% increase of goodput is achieved compared to BGP and a 33% increase
compared to BGP-M. In the 10ASes torrent network, we observe an increase of
goodput for BitTorrent over SCION of still 38% compared to BitTorrent over
BGP, despite that the 10AS torrent network has a smaller number of not
participating ASes that may provide additional network capacities.
Consequently, BitTorrent over SCION is able to aggregate also heterogeneous
paths and we confirm our hypothesis of BitTorrent over SCION’s capability of
aggregating bandwidth in the network that is unused in BGP/BGP-M.

$5$$10$$80$$100$$120$$140$$160$Number of ASesRelative Goodput over all peers
[%]a)BGP (baseline)BGP-MSCION

$5$$10$$40$$60$$80$$100$Number of ASesRelative Download time [%]b)BGP
(baseline)BGP-MSCION

$5$$10$$0$$20$$40$$60$$80$$100$Number of ASesAverage CPU usage per thread
[%]c)BGPBGP-MSCION

Figure 4: Comparison of BitTorrent over BGP/BGP-M and SCION in the 5AS and
10AS torrent network with 4 peers per AS. a) shows the aggregated goodput of
all peers in % with BGP as 100% baseline, b) shows the download time in % with
BGP as 100% and c) the aggregated CPU usage of all threads in %

To verify that increased goodput has a direct impact on the actual download
time of peers, we measure the average download time of all peers in the 5ASes
and 10ASes torrent networks, shown in Figure 4b). Reflected by the lowest
overall goodput, we observe the highest average download time for peers in
BitTorrent over BGP, which is our baseline at 100%. BitTorrent over BGP-M
results in 91% average download time for the 5ASes torrent network. In
BitTorrent over SCION, the average download time is around 69%. Also for the
10ASes torrent network, we observe that the goodput reflects the average
download times, resulting in 73% for BitTorrent over SCION and 88% for
BitTorrent over BGP-M. As expected, the increased goodput through aggregating
unused capacities in the network directly translates into lower download times
for peers, increasing the overall performance of the system.

In addition to the goodput and download times, we also measure the overall CPU
usage of all candidates running the 5ASes and 10ASes torrent network, shown in
Figure 4c). Since we observed an equal distribution of load over all threads,
we present the CPU usage as average CPU usage per thread. While BitTorrent
over BGP and BGP-M only use 5% and 8%, respectively, BitTorrent over SCION
uses around 31% of the available CPU usage for the 5ASes torrent network. We
observe an expectable increase of resource usage of all candidates running
10ASes, with SCION using around 47% of each thread. Despite the higher
throughput, BitTorrent over SCION achieves, the increase of CPU usage is not
inreasing in the same amount. We reason this increase by the open-source SCION
stack [26] (there also exists a closed-source SCION stack optimized for
performance [1]). Especially the SCION Border Router implementation has
potential to be optimized for performance, while the framework to route BGP
and BGP-M (frrouter [25]) is heavily tuned. Consequently, we assume that using
the closed-source SCION stack, we can decrease the CPU usage to a level
comparable to BGP and BGP-M.

From this experiment, we confirm our hypothesis that BitTorrent over SCION can
aggregate otherwise unused capacities in the network through multipath usage.
We observe a significantly higher CPU usage for BitTorrent over SCION, which
can potentially be strongly reduced by using the high-performance SCION stack,
which is closed source.

###  VI-C Peer Selection

Always setting the OutgoingConns parameter sufficiently high may create unfair
advantages for BitTorrent over SCION. Therefore, we compare all 3 candidates
with a varying upper limit of OutgoingConns in this experiment. We again run
20 peers exchanging pieces of a 100Mbyte file in our 5ASes and 10ASes torrent
network. As discussed before, setting OutgoingConns to the half of MaxPeers is
a good tradeoff, we decide to vary OutgoingConns between 3 and 10. We expect
that BitTorrent over BGP and BGP-M stagnates with a low number of
OutgoingConns, meaning simply adding more connected peers to BitTorrent over
BGP and BGP-M does not directly lead to improved performance, while BitTorrent
over SCION handles an increasing number of OutgoingConns better.

Figure 5 shows the average download time of all peers with a varying number of
OutgoingConns for the 5ASes torrent network. We observe that BitTorrent over
BGP and BGP-M start to stagnate after 5 OutgoingConns, while BitTorrent over
SCION results in decreased download times until 8 OutgoingConns. With
OutgoingConns greater than 7, the results are matching the ones presented in
the bandwidth aggregation experiment.

$3$$4$$5$$6$$7$$8$$9$$10$$80$$100$$120$$140$$160$$180$$200$Number of
OutgoingConnsAverage Download Time [s]BGPBGP-MSCION

Figure 5:  Average download time in seconds for BitTorrent over BGP, BGP-M and
SCION in the 5ASes torrent network.

We conclude that between 3 and 7 OutgoingConns, BitTorrent over SCION is still
able to find disjoint paths between peers, while with more than 7
OutgoingConns, the additionally used path-level peers share bottlenecks.
However, we assume that in real-world, Internet-scale torrent networks, the
number of disjoint paths is significantly higher, leading to better results
for higher numbers of OutgoingConns.

In Figure 6, we show the average download time of all peers with a varying
number of OutgoingConns for the 10ASes torrent network. We observe a high
decrease of download times with less than 5 OutgoingConns for all candidates,
while BitTorrent over SCION is still able to decrease the download time for up
to 7 OutgoingConns. All three candidates have reached their minmum download
times for greater than 7 OutgoingConns, in contrast to 5 OutgoingConns for the
5ASes torrent network. This is reasoned by the higher percentage of
participating ASes in the torrent compared to the total number of ASes, and
for BitTorrent over SCION especially by the lower number of additional network
capacities.

$3$$4$$5$$6$$7$$8$$9$$10$$40$$60$$80$$100$$120$Number of OutgoingConnsAverage
Download Time [s]BGPBGP-MSCION

Figure 6:  Average download time in seconds for BitTorrent over BGP, BGP-M and
SCION in the 10ASes torrent network.

From this experiment, we observe that BitTorrent over SCION also outperforms
BitTorrent over BGP and BGP-M with a limited number of connected peers,
disproving the intuitive argument that the path-level peer approach only works
for a sufficiently high number of OutgoingConns.

##  VII Related Work

The mature P2P mechanisms behind BitTorrent have made the protocol an
attractive target for networking research in the past. Ren et al. present
_TopBt_ [33], an adaption of BitTorrent that uses proximities in addition to
transmission rates to detect peers to collaborate with. Catro et al. propose
_BestPeer_ [3], a peer selection algorithm that supports multipath in a multi-
radio, multi-channel wireless mesh network.

Recent works cover analyses of BitTorrent’s locality [36, 8, 39], concluding
that the majority of BitTorrent’s traffic is still running globally.
Furthermore, Decker et al. analyze behavioral patterns and topologies in
existing torrent networks [10], while Cuevas et al. [8] analyze how
BitTorrent’s locality impacts transit costs in existing networks.

A lot of research investigates IP multicast as an efficient way to distribute
content to multiple peers without duplicating the traffic [11]. Since IP
multicast requires expensive dedicated support in network equipment, it has so
far only seen localized deployment[12, 31]. As an alternative to IP multicast,
overlay approaches are considered: Bullet by Kostić et al[22] is an overlay
approach to efficiently distribute files from a single source to a large
number of receivers. Also IPFS [38] shares similarities with BitTorrent
through its P2P based nature. Finally, Fujinoki et al. provide an approach to
unlock private peering links for inter-domain routing in the Internet [16],
which provides interesting potential for our multipath approach.

Next to SCION, several other approaches to enable path control on the host
exist. PathLet Routing by Godfrey et al. [17] is an approach based on
segmentation of inter-domain routes into path fragments. Establishing
multipath data transfer can also be realized completely on the application
level: Yu et al. proposed mpath [41], an algorithm and implementation to
leverage proxies to create multiple paths to a particular end host.

To detect shared bottlenecks, different approaches have been proposed for
MPTCP, some via active measurements [40, 14], others via passive shared
bottleneck detection [20]. These approaches are constrained by an inherent
lack of information about the actual path that the data uses through the
Internet. With Espresso[43], Google presented a BGP-based approach for traffic
distribution and bottleneck avoidance at the edge of their network, rather
than on the endhosts.

Finally, demonstrating SCION’s high-performance capabilities, Neukom et al.
propose Hercules [27], a protocol for very high performance bulk data transfer
over SCION and de Ruiter et al. present a SCION Border Router implementation
in P4 [35].

##  VIII Conclusions and Future Work

In this work, we developed the notion of _path-level peers_ , which allowed us
to enhance BitTorrent with SCION support to add multipath features, with
minimal modifications to the underlying file-sharing algorithms. Furthermore,
we propose an algorithm for disjoint selection of path-level peers to improve
the usage of network capacities. We evaluate BitTorrent over SCION in a
virtualized inter-domain testbed comparing it to BitTorrent over BGP and
BGP-M. We observe a 48% improvement of goodput for BitTorrent over SCION
compared to BGP and 38% compared to BGP-M, which reflects in smaller average
download times for participating peers. Furthermore, we show that our proposed
disjoint path selection algorithm is able to improve traffic flow in the
network with a low number of outgoing connections to unchoked peers.
Consequently, we confirm our hypothesis that BitTorrent over SCION is capable
of aggregating capacities in the network that are unused when BGP or BGP-M is
utilized for inter-domain routing.

As future improvement for BitTorrent over SCION, we plan to extend the tracker
implementation to pre-select particular path-level peers. Peers may actively
communicate their selected path sets to the tracker, which can improve the
location of shared bottlenecks based on the knowledge about path usage of all
known peers.

Furthermore, we plan to evaluate the impact of peers actively communicating
the selected path set to other peers. We assume that peers can improve the
location and avoidance of shared bottlenecks with this approach.

Finally, we plan to extend our disjoint path selection to allow multiple peers
to reuse the same hops without creating shared bottlenecks, by observing
variation in bandwidth to all connected peers when adding new paths containing
shared hops.

## References

  * [1] Anapaya systems, https://www.anapaya.net/, accessed 2022-09-12
  * [2] Bakhshaliyev, K., Canbaz, M.A., Gunes, M.H.: Investigating characteristics of internet paths. ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS) 4(3), 1–24 (2019)
  * [3] Castro, M., Kassler, A., Avallone, S.: Bestpeer-a load-aware multi-path peer selection for wireless mesh networks. In: 2012 IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). pp. 1–6. IEEE (2012)
  * [4] Center for Applied Internet Data Analysis: Caida as-rank, https://asrank.caida.org/, accessed 2022-09-17
  * [5] Center for Applied Internet Data Analysis: Caida as-relationships, https://www.caida.org/data/as-relationships/, accessed 2022-09-17
  * [6] Center for Applied Internet Data Analysis: Caida geolocation data, https://www.caida.org/data/as-relationships-geo/, accessed 2022-09-18
  * [7] Chuat, L., Legner, M., Basin, D.A., Hausheer, D., Hitz, S., Müller, P., Perrig, A.: The complete guide to scion-from design principles to formal verification (2022)
  * [8] Cuevas, R., Laoutaris, N., Yang, X., Siganos, G., Rodriguez, P.: Bittorrent locality and transit trafficreduction: When, why, and at what cost? IEEE Transactions on Parallel and Distributed Systems 25(5), 1177–1189 (2013)
  * [9] De Coninck, Q., Bonaventure, O.: Multipath quic: Design and evaluation. In: Proceedings of the 13th international conference on emerging networking experiments and technologies. pp. 160–166 (2017)
  * [10] Decker, C., Eidenbenz, R., Wattenhofer, R.: Exploring and improving bittorrent topologies. In: IEEE P2P 2013 Proceedings. pp. 1–10. IEEE (2013)
  * [11] Deering, S.E.: Rfc1112: Host extensions for ip multicasting (1989)
  * [12] Diot, C., Levine, B.N., Lyles, B., Kassem, H., Balensiefen, D.: Deployment issues for the ip multicast service and architecture. IEEE network 14(1), 78–88 (2000)
  * [13] Dua, R., Kohli, V., Konduri, S.K.: Learning Docker Networking. Packt Publishing (2016)
  * [14] Ferlin, S., Alay, Ö., Dreibholz, T., Hayes, D.A., Welzl, M.: Revisiting congestion control for multipath tcp with shared bottleneck detection. In: IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. pp. 1–9. IEEE (2016)
  * [15] Fujinoki, H.: Multi-path bgp (mbgp): A solution for improving network bandwidth utilization and defense against link failures in inter-domain routing. In: 2008 16th IEEE International Conference on Networks. pp. 1–6. IEEE (2008)
  * [16] Fujinoki, H.: Beyond-right bgp for utilizing hidden private peering links in the inter-domain routing in the internet. In: 2019 2nd World Symposium on Communication Engineering (WSCE). pp. 68–73. IEEE (2019)
  * [17] Godfrey, P.B., Ganichev, I., Shenker, S., Stoica, I.: Pathlet routing. ACM SIGCOMM Computer Communication Review 39(4), 111–122 (2009)
  * [18] Gvozdiev, N., Vissicchio, S., Karp, B., Handley, M.: On low-latency-capable topologies, and their impact on the design of intra-domain routing. In: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. pp. 88–102 (2018)
  * [19] Hamra, A.A., Legout, A., Barakat, C.: Understanding the properties of the bittorrent overlay. arXiv preprint arXiv:0707.1820 (2007)
  * [20] Hayes, D.A., Ferlin, S., Welzl, M.: Practical passive shared bottleneck detection using shape summary statistics. In: 39th Annual IEEE Conference on Local Computer Networks. pp. 150–158. IEEE (2014)
  * [21] Jerome Tissieres: Bgp load sharing and unequal cost load sharing, https://aboutnetworks.net/bgp-load-sharing/, accessed 2022-09-26
  * [22] Kostić, D., Rodriguez, A., Albrecht, J., Vahdat, A.: Bullet: High bandwidth data dissemination using an overlay mesh. In: Proceedings of the nineteenth ACM symposium on Operating systems principles. pp. 282–297 (2003)
  * [23] Langley, A., Riddoch, A., Wilk, A., Vicente, A., Krasic, C., Zhang, D., Yang, F., Kouranov, F., Swett, I., Iyengar, J., et al.: The quic transport protocol: Design and internet-scale deployment. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication. pp. 183–196 (2017)
  * [24] Li, J., Giotsas, V., Wang, Y., Zhou, S.: Bgp-multipath routing in the internet. IEEE Transactions on Network and Service Management (2022)
  * [25] Linux Foundation: Frrouting project, https://frrouting.org/, accessed 2022-08-30
  * [26] Network Security Group, ETH Zurich, Anapaya Systems: Scion internet architecture, https://github.com/scionproto/scion, accessed 2022-08-24
  * [27] Neukom, C.: High-Performance File Transfer in SCION. Master’s thesis, ETH Zurich (2020)
  * [28] PeeringDB: Peeringdb, https://www.peeringdb.com/, accessed 2022-09-18
  * [29] Peng, Q., Walid, A., Hwang, J., Low, S.H.: Multipath tcp: Analysis, design, and implementation. IEEE/ACM Transactions on networking 24(1), 596–609 (2014)
  * [30] Qiu, S.Y., McDaniel, P.D., Monrose, F.: Toward valley-free inter-domain routing. In: 2007 IEEE International Conference on Communications. pp. 2009–2016. IEEE (2007)
  * [31] Ratnasamy, S., Ermolinskiy, A., Shenker, S.: Revisiting ip multicast. In: Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications. pp. 15–26 (2006)
  * [32] Rekhter, Y., Li, T.: Rfc1771: A border gateway protocol 4 (bgp-4) (1995)
  * [33] Ren, S., Tan, E., Luo, T., Chen, S., Guo, L., Zhang, X.: Topbt: A topology-aware and infrastructure-independent bittorrent client. In: 2010 Proceedings IEEE INFOCOM. pp. 1–9. IEEE (2010)
  * [34] Rothenberger, B., Roos, D., Legner, M., Perrig, A.: Piskes: Pragmatic internet-scale key-establishment system. In: Proceedings of the 15th ACM Asia Conference on Computer and Communications Security (ASIA CCS’20) (2020)
  * [35] de Ruiter, J., Schutijser, C.: Next-generation internet at terabit speed: Scion in p4. In: Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies. pp. 119–125 (2021)
  * [36] Rumin, R.C., Laoutaris, N., Yang, X., Siganos, G., Rodriguez, P.: Deep diving into bittorrent locality. In: 2011 Proceedings IEEE INFOCOM. pp. 963–971. IEEE (2011)
  * [37] Scherrer, S., Legner, M., Perrig, A., Schmid, S.: An axiomatic perspective on the performance effects of end-host path selection. ACM SIGMETRICS Performance Evaluation Review 49(3), 16–17 (2022)
  * [38] Trautwein, D., Raman, A., Tyson, G., Castro, I., Scott, W., Schubotz, M., Gipp, B., Psaras, Y.: Design and evaluation of ipfs: a storage layer for the decentralized web (2022)
  * [39] Wang, H., Liu, J., Xu, K.: On the locality of bittorrent-based video file swarming. In: IPTPS. p. 12 (2009)
  * [40] Wei, W., Wang, Y., Xue, K., Wei, D.S., Han, J., Hong, P.: Shared bottleneck detection based on congestion interval variance measurement. IEEE Communications Letters 22(12), 2467–2470 (2018)
  * [41] Xu, Y., Leong, B., Seah, D., Razeen, A.: mpath: High-bandwidth data transfers with massively multipath source routing. IEEE Transactions on Parallel and Distributed Systems 24(10), 2046–2059 (2012)
  * [42] Yang, X.: Nira: A new internet routing architecture. ACM SIGCOMM Computer Communication Review 33(4), 301–312 (2003)
  * [43] Yap, K.K., Motiwala, M., Rahe, J., Padgett, S., Holliman, M., Baldus, G., Hines, M., Kim, T., Narayanan, A., Jain, A., et al.: Taking the edge off with espresso: Scale, reliability and programmability for global internet peering. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication. pp. 432–445 (2017)
  * [44] Zhang, X., Hsiao, H.C., Hasker, G., Chan, H., Perrig, A., Andersen, D.G.: Scion: Scalability, control, and isolation on next-generation networks. In: 2011 IEEE Symposium on Security and Privacy. pp. 212–227. IEEE (2011)