Further author information: (Send correspondence to E.B.)
E.B.: E-mail<EMAIL_ADDRESS>
# XSLIDE (X-Ray Spectral Line IDentifier and Explorer): a quick-look tool for
XRISM

Efrem Braun  NASA Goddard Space Flight Center, Greenbelt, Maryland, USA  ADNET
Systems, Inc., 6720B Rockledge Drive, Bethesda, MD, USA  Chris Baluta  NASA
Goddard Space Flight Center, Greenbelt, Maryland, USA  KBR, Inc., 601
Jefferson Street, Houston, TX, USA  Trisha F. Doyle  NASA Goddard Space Flight
Center, Greenbelt, Maryland, USA  INNOVIM, LLC, 6401 Golden Triangle Drive,
Greenbelt, MD, USA  Patricia L. Hall  NASA Goddard Space Flight Center,
Greenbelt, Maryland, USA  INNOVIM, LLC, 6401 Golden Triangle Drive, Greenbelt,
MD, USA  Robert S. Hill  NASA Goddard Space Flight Center, Greenbelt,
Maryland, USA  ADNET Systems, Inc., 6720B Rockledge Drive, Bethesda, MD, USA
Matthew P. Holland  NASA Goddard Space Flight Center, Greenbelt, Maryland, USA
Michael Loewenstein  NASA Goddard Space Flight Center, Greenbelt, Maryland,
USA  University of Maryland, Department of Astronomy, College Park, MD, USA
Center for Research and Exploration in Space Science and Technology, NASA
Goddard Space Flight Center, Greenbelt, MD, USA  Eric D. Miller  Massachusetts
Institute of Technology, Kavli Institute for Astrophysics and Space Research,
Cambridge, MA, USA  Michael C. Witthoeft  NASA Goddard Space Flight Center,
Greenbelt, Maryland, USA  ADNET Systems, Inc., 6720B Rockledge Drive,
Bethesda, MD, USA  Tahir Yaqoob  NASA Goddard Space Flight Center, Greenbelt,
Maryland, USA  Center for Research and Exploration in Space Science and
Technology, NASA Goddard Space Flight Center, Greenbelt, MD, USA  University
of Maryland Baltimore County, Center for Space Sciences and Technology,
Baltimore, MD, USA

###### Abstract

We present XSLIDE (X-Ray Spectral Line IDentifier and Explorer), a graphical
user interface that has been designed as a quick-look tool for the upcoming
X-Ray Imaging and Spectroscopy Mission (XRISM). XSLIDE is a simple and user-
friendly application that allows for the interactive plotting of spectra from
XRISM’s Resolve instrument without requiring the selection of models for
forward-fitting. XSLIDE performs common tasks such as rebinning, continuum
fitting, automatically detecting lines, assigning detected lines to known
atomic transitions, spectral diagnostics, and more. It is expected that XSLIDE
will allow XRISM’s scientific investigators to rapidly examine many spectra to
find those which contain spectral lines of particular interest, and it will
also allow astronomers from outside the field of high-resolution X-ray
spectroscopy to easily interact with XRISM data.

###### keywords:

XRISM, X-ray Astronomy, Astronomical Software

##  1 INTRODUCTION

The X-Ray Imaging and Spectroscopy Mission (XRISM) Resolve instrument is
expected to collect high energy X-ray spectra of unprecedented quality[1].
Rigorous analyses of these spectra require iterative forward-fitting, in which
a model spectrum is specified and convolved with the instrument response to
produce predicted photon counts, which are compared with the observed photon
counts, and followed by adjustment of the model parameters to minimize the
difference between the predicted and observed counts[2]. This procedure can be
overly complicated and time-consuming for one who wants to quickly examine a
spectrum, as it requires the user to specify and then refine a model spectrum.

As a precursor to this rigorous analysis, the XRISM Science Data Center (SDC)
has developed XSLIDE (X-Ray Spectral Line IDentifier and Explorer), which
allows scientific investigators to rapidly browse and survey available Resolve
spectra. This will allow for easy access to XRISM data for astronomers from
outside the field of high-resolution X-ray spectroscopy, quick
browsing/surveying of available XRISM data prior to a decision to focus on a
given spectrum, and simple extraction and characterization of XRISM spectra
for non-publishing use such as writing proposals or giving presentations.

XSLIDE has been designed as a simple graphical user interface available as a
desktop or web application (see Fig. 1) which allows users to interactively
plot X-ray spectra. It walks the user through loading and modifying a
spectrum, detecting lines, identifying those lines with known lines from
atomic databases, performing diagnostics, and exporting the results. Following
exploratory work performed with XSLIDE, more rigorous tools such as XSPEC[3]
can be used to do further analyses on spectra of interest.

In this paper, we give a brief overview of XSLIDE. We discuss the methodology
by which it allows users to view a spectrum with no selection of models for
forward-fitting required and we assess the validity of the assumptions
required for this process. We then outline the features of XSLIDE and show how
the user is walked through the application step-by-step. Finally, we discuss
the software architecture of XSLIDE to give an idea of how the program is
built.

---
Figure 1:  Screenshots of the XSLIDE user interfaces, demonstrating the
similarity between the desktop (above) and web (below) versions.

##  2 Methodology

A spectrometer obtains counts ($C$) within particular instrument channels
($I$). This is related to the source spectrum ($S$, in
$\frac{\text{photon}}{\text{cm}^{2}\text{ second keV}}$) by

$C(I)=T\int R_{\text{RMF}}(I,E)R_{\text{ARF}}(E)S(E)dE\,,$ (1)

where $T$ is the total observation time, $R_{\text{RMF}}$ is the unitless
redistribution matrix file (RMF) which gives the probability of a photon of a
given energy ending up as a count in a particular channel, and
$R_{\text{ARF}}$ is the ancillary response file (ARF) which contains the
effective area in $\frac{\text{cm}^{2}\text{ count}}{\text{photon}}$[2].

As Equation (1) cannot be analytically inverted to solve for $S$, it is
typically solved by forward-fitting, whereby a model spectrum $S$ is
specified, and the resulting predicted counts are compared to the observed
counts, adjusting the parameters of the model to minimize some statistical
measure of the error between the predicted counts and the observed counts[2].
This procedure can be quite complicated and time-consuming as it requires the
user to specify and then refine the model.

XSLIDE instead assumes that the RMF is a diagonal matrix (providing an ideal
one-to-one mapping between incident photon energy and detector channel) and
that the ARF is slowly varying such that it is approximately constant between
neighboring instrument channels. This allows for $S$ to be discretely solved
for as

$S(E)=\frac{C(I)}{R_{\text{ARF}}(E)T\Delta E}\,,$ (2)

which can be computed using the ARF to map the instrument channel to the
spectrum energy. This allows a spectrum to be viewed directly, requiring the
user to provide only the spectrum and ARF files.

---
Figure 2:  A sample RMF from the Hitomi SXS instrument, showing a sharp peak
along the diagonal. The RMF shown was generated using the Hitomi “sxsrmf” tool
for a single SXS pixel and for HighRes events, using the “x-large” format to
include full input model resolution with all response components. For testing
and display purposes, the level of the electron continuum component was
increased by a factor of ten over the CalDB value.

###  2.1 Validity

As the RMF is not truly diagonal, the assumption of its being diagonal will
result in an inexact spectrum; hence the need for the use of XSLIDE to be
followed up with more rigorous tools such as XSPEC[3]. However, if the RMF is
strongly peaked along the diagonal without significant off-diagonal
contributions, then the locations of the spectral peaks will be correct, while
only the peak widths will be incorrect. Thus, the approximate spectra gained
using this assumption will be valid for the purpose of line identification and
certain analyses. Although RMFs are not yet available for the XRISM mission,
it is expected that the XRISM Resolve instrument’s RMF will be similar to that
of the Hitomi SXS instrument[1], for which the RMF has been observed to be
strongly peaked along the diagonal (see Fig. 2).

The assumption of the ARF being locally constant can bring about spurious
lines or hide lines at energies where the ARF changes quickly. The user can
verify that the ARF is not changing quickly by overlaying it on the spectrum.
In addition, if any detected lines are found in regions where the ARF is
changing quickly, a warning is shown to the user that these lines may be
artifacts.

As a check on whether these assumptions are sufficiently valid for XSLIDE to
be used reliably with spectra taken by the XRISM Resolve instrument, we
compared the spectrum of the Perseus cluster taken by the Hitomi SXS
instrument as produced rigorously in Tashiro et al.[4] with a spectrum
generated by XSLIDE. As Fig. 3 shows, the spectrum produced by XSLIDE well-
reproduces the more rigorously produced spectrum.

---
Figure 3:  Comparison between spectra of the Perseus cluster taken by the
Hitomi SXS instrument as shown in Tashiro et al.[4] (above) and as produced by
XSLIDE (below). The XSLIDE image used the spectrum in ObsID 100040020,
rebinned to 4.0 eV. See Tashiro et al.[4] for full details of their methods
used. Note that some differences between the spectra are expected due to
Tashiro et al.[4] having combined multiple observations for a longer
cumulative exposure time and having performed additional corrections, which
likely explain the small peaks at 5.6, 6.1, and 6.3 keV that were not observed
with XSLIDE.

##  3 Running XSLIDE

XSLIDE is designed to be simple and easy to use. The user is guided through
ordered steps and substeps, with little need for actions to be taken outside
this guided procedure. In this section, we give a brief overview of these
steps to illustrate XSLIDE’s capabilities.

###  3.1 Step 1: Spectrum

In this step, the spectrum is loaded and modified.

The user can either load the spectrum and ARF files from the local desktop or
perform an online query of NASA’s High Energy Astrophysics Science Archive
Research Center (HEASARC)[5]. After the spectrum is loaded, the user has the
option to clip the edges of the spectrum, which can be used for such purposes
as removing artifacts due to artificially low ARF. The user can then rebin the
spectrum to improve the signal-to-noise ratio.

Fig. 4 shows a sample screenshot of XSLIDE after Step 1 is complete.

---
Figure 4:  Screenshot of the desktop version of XSLIDE after Step 1 is
complete.

###  3.2 Step 2: Detect

In this step, spectral lines are detected.

The first substep in accomplishing this is fitting a continuum, which can help
with the next step of automatically detecting lines when the lines are placed
over a non-flat continuum, such as can occur with bremsstrahlung radiation.
Two methods exist for continuum fitting. First, XSLIDE can automatically fit a
power law function in a piecewise manner to segments of the spectrum. Second,
the user can manually specify points, which XSLIDE will then interpolate to
form the continuum.

XSLIDE then provides two methods for automatic line detection. The piecewise
statistics method works by first calculating the average and standard
deviation statistics of piecewise segments of the spectrum, and then setting
aside points with flux outside a user-specified number of standard deviations
of the segment’s average as outliers. This process is then repeated
iteratively several times, calculating the average and standard deviation
statistics for the segments using only points that are not considered outliers
until the statistics converge. Sets of consecutive outliers are considered
detected lines. The second method works using the continuous wavelet transform
(CWT), convolving the spectrum with Ricker wavelets of user-defined widths to
effectively search the spectrum for regions which match the shape of the
wavelets. The parts of the spectrum which better match the shape yield higher
CWT coefficients, and points with local maximums in these CWT coefficients and
which are above a user-defined signal-to-noise ratio are labeled as peaks.
Each of the two methods has strengths and weaknesses. The piecewise statistics
method can search for lines of varying widths, and it finds both emission and
absorption lines, but it can fail to identify closely-spaced peaks as being
separate lines if the valleys are sufficiently far away from the continuum,
and it requires appropriate binning and continuum fitting to work. The CWT
method is better at separately identifying closely-spaced peaks, and proper
binning and continuum fitting are not critical to its success, but the user
needs to specify the peak widths to search for, and it is mainly only capable
of searching for either emission or absorption lines (but not both) in a given
spectrum.

After automatically detecting lines, the user can then make manual adjustments
as required. Undetected lines can be added, spurious lines can be removed, and
existing lines can be narrowed or broadened as required.

We note that automatic line detection is far from a solved problem; neither of
the methods provided in XSLIDE will be able to find all the peaks and discard
all of the noise that can be readily discerned with a human eye. Rather, the
point of XSLIDE’s automatic line detection is to quickly find a spectrum’s
dominant features to determine if further analysis with a rigorous tool such
as XSPEC[3] is warranted.

###  3.3 Step 3: Identify

In this step, the detected lines are identified using an atomic line database.

The user must first choose a set of atomic lines to use. XSLIDE comes packaged
with AtomDB[6] version 3.0.9, several atomic databases custom-generated with
XSTAR[7, 8, 9] version 2.58, and a neutral species database that was created
using the data in Table 1-3 of the X-Ray Data Booklet[10]. These built-in
databases are expected to meet the requirements of the majority of XSLIDE
users, and XSLIDE also allows users with niche needs to load their own atomic
databases. Atomic line filters are implemented to allow the user to focus on
particular ions or “stronger” lines with cutoffs for oscillator strength or
emissivity.

Before the detected lines can be matched with the atomic lines, the spectrum’s
velocity shift (generally referred to simply as “redshift”) needs to be
accounted for so that the lines are energetically aligned. Several methods are
available to specify the velocity shift. The manual method works by having the
user simply enter a value. With the single line alignment method, the user
selects a detected line and an atomic line, and the velocity shift gets set
such that the energies of these two lines will be equal. This method of
velocity shift determination is often done for intracluster medium spectra
using the prominent Fe24+ line at 6.7 keV, for example. A more experimental
method is the full line search, where a user-specified range of velocity
shifts are generated, and for each velocity shift, a loss function is
determined by calculating the deviation between each detected line and its
nearest atomic line, summing the absolute values of these deviations. Velocity
shifts with lower loss functions are worth further examination and can be
applied by the user. This method is shown in Fig. 1, where it can be observed
that a velocity shift of 0.0178 was found for the Perseus cluster observation
by the Hitomi SXS instrument, quite close to the value of 0.01756 calculated
in Tashiro et al.[4].

Once the detected lines and atomic lines are aligned with each other, it is
simple to identify the detected lines. XSLIDE simply matches each detected
line with all atomic lines that are within the energy range determined by the
edge points of the detected line. To help the user identify detected lines for
which multiple atomic lines are possible, XSLIDE shows the oscillator strength
and emissivities of the atomic lines to inform the user of the strength of the
various lines. The user can choose to display labels next to the lines on the
graph to indicate the identified atomic transitions.

Fig. 1 shows sample screenshots of XSLIDE after Step 3 is complete.

###  3.4 Step 4: Diagnostics

In this step, the user views diagnostics that are computed using the
identified lines. These diagnostics consist of ratios of line fluxes which
have been found to correlate with system properties that may be of interest,
such as temperature or ionization parameter.

The purpose of the XSLIDE diagnostics is only to provide a preliminary
analysis, alerting the user to the possibility of the spectrum’s diagnostic
potential. The diagnostics should not be considered quantitatively reliable,
as the detected line fluxes used for these calculations are only
approximations which are highly dependent on the continuum and the number of
points that define the detected lines. Any overlap between detected lines can
drastically distort these calculated fluxes. To actually compute reliable
diagnostics, a more rigorous tool such as XSPEC[3] is required.

Three diagnostics are available. In the hydrogen diagnostics (see Fig. 5 for a
sample screenshot), ions for which the Lyman lines have been identified in the
spectrum will have the Ly$\beta$/Ly$\alpha$ ratio calculated. This ratio will
also be computed for the selected ion using the data in the atomic database as
a function of temperature or ionization parameter and shown to the user in a
graph; by seeing where the calculated ratio of the detected line fluxes
intersect this graph, the temperature or ionization parameter may be
approximated (as long as the assumptions that were used to create the selected
atomic database are valid).

In the helium diagnostics (see Fig. 5), ions for which the common helium-like
lines (w, x, y, and z; also known as resonance (R), intercombination (I),
intercombination (I), and forbidden (F)) have been identified in the spectrum
will have the G ($\frac{x+y+z}{w}$) and R ($\frac{z}{x+y}$) ratios calculated.
Similarly to the hydrogen diagnostics, the G ratio will also be computed for
the selected ion using the data in the atomic database as a function of
temperature or ionization parameter and shown to the user in a graph for the
purpose of approximating the temperature or ionization parameter. The R ratio
is commonly used as a diagnostic to indicate the density of the observed
source, but assisting the user in approximating this value is beyond the scope
of XSLIDE.

---
Figure 5:  Partial screenshots of the hydrogen (left) and helium (right)
diagnostics from Step 4 of XSLIDE. The hydrogen diagnostics shown indicate a
temperature less than 30 million K (emissivities are not provided in AtomDB
for temperatures low enough to give a Ly$\beta$/Ly$\alpha$ ratio of 0.0619 for
Fe25+), and the helium diagnostics shown indicate a temperature of 16.2
million K.

In the K diagnostics, ions for which K$\beta$ or K$\alpha$ lines have been
identified in the spectrum will have the K$\beta$/K$\alpha$ ratio of the
detected line fluxes calculated. This ratio is also commonly used as a
diagnostic to indicate the temperature or ionization parameter of the observed
source.

###  3.5 Step 5: Export

In this step, the results can be exported to various formats. This allows the
results to be presented or utilized in other programs.

Output formats include a PDF report, PNG or SVG figures, and CSV or Excel
tables of data. In addition, XSLIDE can export a model spectrum in a format
that can be read directly into XSPEC[3]. This model spectrum will consist of
the lines and continuum that have been detected in XSLIDE, giving the XSPEC
user a starting point that can be further modified as desired.

###  3.6 Other Features

A key feature of XSLIDE is the interactive plot of the spectrum. The user can
pan and zoom with intuitive controls to focus on particular features of the
plot as desired. The user can also customize the plot by setting line colors,
styles, widths, and symbols. In addition to being able to view the spectrum
and the fitted continuum, the user can also choose to display the spectrum
minus the continuum, the raw counts, and the ARF.

XSLIDE is available in both English and Japanese. The user can select the
language as a menu option.

An extensive user manual is provided with XSLIDE which documents the various
methods and parameters available in each step. It also provides a walkthrough
of the software, demonstrating how XSLIDE can be used to approximately
reproduce Figure 1 of Tashiro et al.[4].

##  4 Software Architecture

XSLIDE has a Model-View-Controller (MVC) software design, which keeps a level
of separation between the data and the GUI. “Data” refer to the physical
quantities that XSLIDE is operating on, such as spectra, ARFs, continua,
theoretical line lists, emissivity models, etc. The Model performs operations
on the data elements, the View displays the current state of the Model to the
user, and the Control part of the GUI manipulates data in response to user
events by sending commands to operate on the Model.

This separation of programming elements allows for a central underlying Model
to be displayed to multiple Views, allowing for separate desktop and web
applications to be written with minimal reproduction of effort. It further
simplifies development by providing a guided process by which to add a new
feature, where first the Model is updated such that the new feature is
available in the command line, then the View is updated such that new GUI
elements showing the state of the Model are displayed, and finally the Control
is updated so that the Model appropriately responds to user events. The
separation helps with testing as well, as unit tests are written for the Model
independently from functional tests written for the final GUI applications.

There are ultimately three XSLIDE products: a Python command-line-interface
(XSLIDEpy), the desktop GUI (XSLIDE for desktop), and the web GUI (XSLIDE for
web). The Model elements of the MVC are largely contained within XSLIDEpy,
whereas the View and Controller elements are separately written for the
desktop and web GUIs. Most users will only access either the desktop and/or
web GUIs, with XSLIDEpy largely reserved for use in the development process.

XSLIDEpy is a Python package that leverages the standard scientific Python
ecosystem of NumPy[11], SciPy[12], and Astropy[13, 14] to quickly conduct
common astronomical operations. XSLIDE uses PyVO, an Astropy-affiliated
package, to query NASA’s HEASARC[5] archive for spectrum and ARF files from
the XRISM and Hitomi missions. Unit tests cover a large portion of the source
code.

XSLIDE for desktop is built using Qt[15], which allows XSLIDE to be run cross-
platform on Mac and multiple Linux distributions. The Qt for Python (PySide6)
bindings are used to develop the Control, and PyQtGraph[16] is used for the
interactive plot. PyInstaller[17] is used to bundle the application into a
single package that can be installed via simple drag-and-click. Squish[18] is
used to run automated functional tests that cover most of the functionality
available to the end user. Squish runs the application through scripts of user
actions, recognizing GUI objects and interacting with them by performing mouse
and keyboard clicks, and then verifying that these actions produced the
expected results by confirming that GUI objects contain predetermined values
and that screenshots are visually equivalent to prepared standards. A video of
Squish running one of the functional tests can be seen in the link provided in
Fig. 6.

---
Figure 6:  A video of Squish[18] performing one of XSLIDE’s functional tests
can be accessed as an ancillary file.

XSLIDE for web is built using HTML, CSS, and Bokeh[19] to design the View and
interactive plot. JavaScript and a Bokeh server are used to develop the
Control. Selenium[20] is used to run functional tests in a manner similar to
what is done by Squish for the desktop version.

For both the desktop and web versions of XSLIDE, Japanese language
translations are provided using Qt’s QTranslator class and user documentation
is prepared with Sphinx[21].

###### Acknowledgements.

We thank Lorella Angelini for laying the foundational groundwork of XSLIDE,
Tim Kallman for generating the XSTAR atomic database files, Megumi Shidatsu
for assistance in creating Japanese translations, and Edmund Hodges-Kluck and
Kenji Hamaguchi for providing invaluable user feedback that helped guide the
development process.

## References

  * [1] Tashiro, M. et al., “Concept of the X-ray Astronomy Recovery Mission,” in [Space Telescopes and Instrumentation: Ultraviolet to Gamma Ray ], den Herder, J.-W. A., Nikzad, S., and Nakazawa, K., eds., Proc. SPIE 10699, 520–531 (2018).
  * [2] Smith, R. K., Arnaud, K. A., and Siemiginowska, A., “Data analysis,” in [Handbook of X-ray Astronomy ], Arnaud, K. A., Smith, R. K., and Siemiginowska, A., eds., Cambridge University Press (2011).
  * [3] Arnaud, K. A., “XSPEC: The first ten years,” in [Astronomical Data Analysis Software and Systems V ], Jacoby, G. H. and Barnes, J., eds., ASP Conference Series 101, 17 (1996).
  * [4] The Hitomi collaboration, “The quiescent intracluster medium in the core of the Perseus cluster,” Nature 535, 117–121 (2016).
  * [5] “High Energy Astrophysics Science Archive Research Center (HEASARC).” https://heasarc.gsfc.nasa.gov/.  A service of the Astrophysics Science Division at NASA/GSFC and of the Smithsonian Astrophysical Observatory’s High Energy Astrophysics Division (Accessed: June 2022).
  * [6] Smith, R. K., Brickhouse, N. S., Liedahl, D. A., and Raymond, J. C., “Collisional plasma models with APEC/APED: Emission-line diagnostics of hydrogen-like and helium-like ions,” Astrophys. J. 556, L91–L95 (2001).
  * [7] Mendoza, C. et al., “The XSTAR atomic database,” Atoms 9 (2021).
  * [8] Kallman, T. et al., “Photoionization models for high-density gas,” Astrophys. J. 908, 94 (2021).
  * [9] Kallman, T. and Bautista, M., “Photoionization and high-density gas,” Astrophys. J., Suppl. Ser. 133, 221–253 (2001).
  * [10] Kortright, J. B. and Thompson, A. C., “X-ray emission energies,” in [X-Ray Data Booklet ], Thompson, A. C., ed., Lawrence Berkeley National Laboratory, Berkeley, CA, third ed. (2009).  https://xdb.lbl.gov/.
  * [11] Harris, C. R. et al., “Array programming with NumPy,” Nature 585(7825), 357–362 (2020).
  * [12] Virtanen, P. et al., “SciPy 1.0: Fundamental algorithms for scientific computing in Python,” Nat. Methods 17, 261–272 (2020).
  * [13] Astropy Collaboration, “Astropy: A community Python package for astronomy,” Astron. Astrophys. 558, A33 (2013).
  * [14] Astropy Collaboration, “The Astropy project: Building an open-science project and status of the v2.0 core package,” Astron. J. 156, 123 (2018).
  * [15] “Qt.” https://www.qt.io/.  (Accessed: June 2022).
  * [16] “PyQtGraph.” https://www.pyqtgraph.org/.  (Accessed: June 2022).
  * [17] “PyInstaller.” https://pyinstaller.org/.  (Accessed: June 2022).
  * [18] “Squish.” https://www.froglogic.com/squish.  (Accessed: June 2022).
  * [19] Bokeh Development Team, “Bokeh: Python library for interactive visualization.” https://bokeh.pydata.org/en/latest/ (2018).  (Accessed: June 2022).
  * [20] “Selenium.” https://www.selenium.dev/.  (Accessed: June 2022).
  * [21] “Sphinx.” https://www.sphinx-doc.org/.  (Accessed: June 2022).


Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

* Corresponding author: L. Han (e-mail: l.han@mmu.ac.uk).

# A Generic Performance Model for Deep Learning in a Distributed Environment

TULASI KAVARAKUNTLA1  LIANGXIU HAN1*, HUW LLOYD, MIEEE1, ANNABEL LATHAM,
SMIEEE1, ANTHONY KLEEREKOPER1, AND SAMSON B. AKINTOYE1  Department of
Computing and Mathematics, Manchester Metropolitan University, UK (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>)

###### Abstract

Performance modelling of a deep learning application is essential to improve
and quantify the efficiency of the model framework. However, existing
performance models are mostly case-specific, with limited capability for the
new deep learning frameworks/applications. In this paper, we propose a generic
performance model of an application in a distributed environment with a
generic expression of the application execution time that considers the
influence of both intrinsic factors/operations (e.g. algorithmic
parameters/internal operations) and extrinsic scaling factors (e.g. the number
of processors, data chunks and batch size). We formulate it as a global
optimization problem and solve it using regularization on a cost function and
differential evolution algorithm to find the best-fit values of the constants
in the generic expression to match the experimentally determined computation
time. We have evaluated the proposed model on three deep learning frameworks
(i.e., TensorFlow, MXnet, and Pytorch). The experimental results show that the
proposed model can provide accurate performance predictions and
interpretability. In addition, the proposed work can be applied to any
distributed deep neural network without instrumenting the code and provides
insight into the factors affecting performance and scalability.

###### Index Terms:

Deep Learning, Performance modelling, Optimization, Differential evolution.

=-15pt

##  I Introduction

Deep neural networks are effective tools for unsupervised data exploration to
discover correlation structures. As a result, they are widely used in computer
vision, self-driving cars, medical image analysis, video games, and online
self-service applications. However, deep neural network architectures such as
GoogLeNet[1], ResNet[2], VGG Net[3], and Deep Convolutional Neural Networks
(CNN) [4] necessitate the use of high computational resources. Training with a
large amount of data requires a parallelised and distributed environment,
primarily data parallelism, model parallelism, pipeline parallelism, and
hybrid parallelism. Performance modelling is essential in quantifying the
efficiency of large parallel workloads. Performance models are used to obtain
run-time estimates by modelling various aspects of an application on a target
system. However, accurate performance modelling is a challenging task.
Existing performance models are broadly categorised into two catergories:
analytical modelling and empirical modelling. Analytical modelling uses a
transparent approach to convert a model’s or an application’s internal
mechanism into a mathematical model corresponding to the system’s goals, which
can significantly expedite the creation of a performance model for the
intended system. The existing analytical modelling works investigated deep
learning performance modelling and scaling optimization in distributed
environments [5], asynchronous GPU processing based on mini-batch SGD[6],
efficient GPU utilisation in deep learning[7], comprehensive analysis and
comparison of the performance of deep learning frameworks running on GPUs [8,
9]. However, the major limitation of these works is the poor presentation of
the underlying internal operations (i.e., areas of the features’ space or
specific workload conditions) in the distributed environment.
Empirical modelling is a good alternative to analytical models, which predicts
the outcome of an unknown set of system parameters based on observation and
experimentation. It characterises an algorithm’s performance across problem
instances and parameter configurations based on sample data. Existing works
investigated deep convolutional neural networks using asynchronous stochastic
gradient descent techniques in a distributed environment[6]. Nevertheless, the
existing empirical modelling methods are still facing a challenge on how to
provide an unbiased experimental study in a distributed environment using
GPUs.

Thus, inspired by the hybridisation of the analytical and empirical
approaches, this paper proposes a novel generic performance model that
provides a general expression of intrinsic and extrinsic factors of a deep
neural network framework in a distributed environment for accurate performance
prediction and interpretability. Especially, the proposed model is applicable
to any to any distributed deep neural network without instrumenting the code,
which furthermore allows for explaining intrinsic parameters’ performance and
scalability, providing added value in the field. Our contributions include the
following:

  * •

We have developed a generic expression for a performance model considering the
influence of intrinsic parameters and extrinsic scaling factors that affect
computing time in a distributed environment.

  * •

We have formulated the generic expression as a global optimization problem
using regularization on a cost function in terms of the unknown constants in
the generic expression, which we have solved using differential evolution to
find the best fitting values to match experimentally determined computation
times.

  * •

We have evaluated the proposed model in three deep learning frameworks, i.e.,
TensorFlow, MXnet, and Pytorch, to demonstrate its performance efficiency.

The remainder of the paper is organised as follows. Section II discusses
related work of the existing performance models in a distributed environment.
In section III, we discuss research methodology, e.g., problem description,
the proposed performance model. Section IV discusses an experimental
evaluation of the effectiveness of our proposals. Finally, section V concludes
the paper and highlights the future works.

##  II Related Works

This section provides an overview of the existing performance models in a
distributed computing environment and differential evolution as a solution to
the optimization problem.

###  II-A Existing Performance Models

Performance modelling involves prediction of the performance of a system, the
impact of change on an existing system, or the impact of a change in workload
on an existing system [10, 11]. Existing performance modelling of Deep
Learning (DL) frameworks can be broadly divided into two categories: 1)
Analytical modelling and 2) Empirical modelling.

####  II-A1 Analytical Performance Modelling of DL frameworks

Yan et al. [5] developed a performance model to evaluate the effect of the
partitioning and resourcing decisions on the distributed system architectures’
overall performance and scalability using a DL framework Adam[12]. In
addition, the performance model was also used to guide the development of a
scalability optimizer that quickly selects the optimal system configuration
for reducing DNN training time. However, the model can only be applied to
specific DL systems, particularly when it has parameter servers and
synchronous weights between worker nodes dynamically.

Qi et al. [13] developed an analytical performance model named Paleo,
predicting the deep neural network performance by considering communication
schemes, network architecture and parallelization strategies. The results
demonstrated that hybrid parallelism performed much better than data
parallelism while training the Alexnet model. However, the model did not
consider other factors affecting the overall performance of a model, such as
memory usage, data transfer, or communication overhead in distributed
environments.

Heehoon Kim et al. [14] evaluated five popular deep learning frameworks
TensorFlow[15], CNTK[16], Theano[17], Caffe-MPI[18] and Torch[19] in terms of
their performance in both single and multi-GPU environments. In this work,
each framework incorporated and compared different convolution algorithms,
such as Winograd, General Matrix Multiplication (GEMM), Fast Fourier
Transformation (FFT), and direct convolution algorithms, in terms of layered-
wise analysis and execution time. The results have shown that FFT and Winograd
algorithms surpass the GEMM and other convolution algorithms. However, the
convolution algorithms used by the frameworks provided poor explainability
regarding their internal operations.

Shi et al. [9] proposed a performance model to evaluate various distributed
deep learning frameworks’ performance with different convolutional neural
networks in the multi-GPU environment. They measured training time, memory
usage, and GPU utilization and compared the frameworks in terms of training
time and resource utilization. However, they did not provide a breakdown of
the time to divide the mini-batch into smaller batches or measure the load
imbalance factor, which are critical factors that could significantly affect
the training efficiency and performance in a parallel computing environment.
Kavarakuntla et al. [20] extended Shi’s analytical performance model to
evaluate the deep learning frameworks’ run-time performance with the
autoencoder, multilayer perceptron and convolutional neural network models in
the GPU cluster environment. The extended model considered the load imbalance
factor and made a layer-wise analysis of a neural network, providing a more
comprehensive evaluation of the frameworks’ performance. The experimental
results showed that the load balance is an influential factors affecting the
system performance.

However, the models mentioned above have poor explainability and were
developed for specific architectures, which were not generic and couldn’t be
applied to a wide range of neural networks.

####  II-A2 Empirical Modelling of DL frameworks

Empirical modelling builds models through observation and experimentation,
which is antithetical to analytical modelling.

Oyama et al. [6] proposed a performance model for predicting the statistics of
an asynchronous stochastic gradient descent-based deep learning system,
potentially improving the model’s performance by optimizing the
hyperparameters, such as gradient staleness and mini-batch size. They did not
consider parallelization methods and applied direct weights synchronized among
GPUs. The study results showed that the proposed method could predict the
statistics of asynchronous SGD parameters, including mini-batch size, sweeping
dataset time, staleness, and probability distributions of these essential
parameters. However, the work did not address the issue of communication
overhead and network latency, which could significantly affect the performance
of distributed deep learning systems.

Rakshith et al. [21] presented an empirical study of the performance of
Horovod, a distributed deep learning framework, for image classification
tasks. They evaluated the performance of Horovod on two popular image
datasets, CIFAR-10 and ImageNet, using a cluster of machines with varying
numbers of GPUs. They also compared the performance of Horovod with other
distributed deep learning frameworks, such as PyTorch and TensorFlow, and
found that Horovod achieved better performance in certain scenarios. They
provided recommendations for optimizing the performance of Horovod on large-
scale image datasets, such as using efficient data loading and preprocessing
techniques, and optimizing the communication and synchronization between the
machines. However, the experimental configuration utilized in the research
might not accurately reflect real-world situations in which the underlying
hardware and network setups may differ substantially.

Lin et al. [22] considered the network topology and communication patterns to
train deep learning models on GPU clusters. The model included the
communication and computation times for each layer in a deep neural network
and used a prediction model that is more sophisticated than a simple linear
regression approach to predict the total training time. The model was
evaluated on several deep learning benchmarks and showed that it achieved
higher accuracy in predicting training time than existing models. The model
could also be used to optimize the performance of distributed deep learning by
finding the optimal configuration of GPU nodes and reducing the training time.
However, the assessment of the proposed model was confined to three distinct
GPU clusters, potentially limiting its generalizability to other GPU clusters
or distributed DL architectures.

Most recently, inspired by the concept of combining elements of analytical
modelling and empirical modelling for better performance prediction developed
in other fields [23], we [24] developed a generic performance model for deep
learning applications in distributed environments, which offers the advantage
of applicability to various deep learning frameworks. However, its performance
is sub-optimal and lacks comprehensive analysis and experimental evaluation.

To address the above limitations, in this paper, we have implemented the model
that gives insights into the intrinsic parameters’ performance and scalability
of the extrinsic parameters for more accurate performance prediction. Our
proposed model in this paper provides a generic expression applicable to any
distributed deep neural network without instrumenting the code and enabling
functionality such as explaining internal parameters’ performance and
scalability. The detailed method is described in section III below.

###  II-B Differential Evolution

Differential Evolution (DE) was developed by Storn et al.[25] as an algorithm
to solve various complex optimization problems such as motor fault diagnosis
[26], structure prediction of materials [27], automatic clustering techniques
[28], community detection [29], learning applications [30] and so on. The
algorithm achieves optimal solutions by maintaining a population of individual
solutions and employing a distinct process to generate new offspring through
the combination of existing solutions. Those offspring exhibiting superior
objective values are retained in subsequent iterations of the algorithm,
thereby enhancing the individual’s new objective value and subsequently
assimilating them into the population. Conversely, if the newly acquired
objective value fails to surpass existing solutions, it is promptly
disregarded. This iterative process continues until a specific termination
condition is met, ensuring the algorithm’s convergence [31]. It shares
similarities with other evolutionary algorithms, such as the Genetic Algorithm
(GA) [32], wherein mutation, crossover, and selection operators are employed
to steer the population towards increasingly favourable solutions. In contrast
to the genetic algorithm, the differential evolution algorithm imparts
mutation to each individual while transferring them to the next generation. In
its mutation procedure, for each solution, three more individuals are picked
from the population, and as a consequence, a mutated individual is produced.
It is determined based on the fitness value whether or not the first
individual selected will be replaced. In differential evolution, the crossover
is not the primary operation, as it is in the genetic algorithm. In recent
times, several works have been proposed to use DE for neural network
optimization [33], [34], [35].

However, none of the works mentioned above used DE to analyse and evaluate the
performance of deep neural networks with many processes in a distributed
environment with the goal of finding the best-fit values by minimising the
regularised cost function.

##  III Methodology

###  III-A The Generic Performance Model

Given an application consisting of a number of processes in a distributed
environment, the execution time of the application can be considered from two
levels: 1) Execution time of internal processes of the application (for
example, intrinsic parameters of the application); and 2) External scaling
factors that affect the computing efficiency (such as a number of
machines/processors or data chunks or batch size). A generic performance model
for computing total computational time $(t)$ per iteration of an application
can be described as follows:

$\displaystyle t(I,E)=t_{I}(I)f_{E}(E)+C$ (1)

Here, we represent intrinsic parameters as $I$, $E$ represents the extrinsic
parameters, $t_{I}$ represents the computation time of the processes affected
by intrinsic parameters, $f_{E}$ represents extrinsic scaling factors that
affect the computing performance, and $C$ is a constant. In general, $I$ and
$E$ are vectors in which each element is a hyperparameter of the deep learning
model such as a filter size (intrinsic) or batch size (extrinsic).

In our model, we represent the internal time $t_{I}$ as a sum of terms in
powers of the components of $I$:

$\displaystyle t_{I}=\sum_{i=1}^{n}a_{i}I_{i}^{p_{i}}$ (2)

Basically, intrinsic parameters represent model parameters of the deep neural
network, as shown in Fig.1. In equation (2), the coefficients $a_{i}$ relate
to the relative importance of the processes, and the powers $p_{i}$ relate to
the computational complexity. The external factors are related to scaling, and
these appear in the model as multiplicative terms with different powers in the
computation of the external scaling factor $f_{E}$, which is given by:

$\displaystyle f_{E}=\prod_{j=1}^{m}E_{j}^{q_{j}}$ (3)

Here, the powers $q_{j}$ give information about scalability. By substituting
the $t_{I}$ and $f_{E}$ in equation (1), the computational time (t) is given
as follows:

$\displaystyle
t(I,E,x)=\left(\sum_{i=1}^{n}a_{i}I_{i}^{p_{i}}\right)\prod_{j=1}^{m}E_{j}^{q_{j}}+C$
(4)

which we now write as a function of $I$, $E$ and $x$ where
$\boldsymbol{x}=\left\\{{a_{1},...,\,\,a_{n_{I}},p_{1},...\,p_{n_{I}},q_{1},...\,q_{n_{E}},c}\right\\}\in\mathbb{R}^{M}$
is a vector formed by combining $\boldsymbol{a},\boldsymbol{p},\boldsymbol{q}$
and the constant coefficient $\boldsymbol{C}$. In (4), the intrinsic
parameters $I$ and extrinsic parameters $E$ are the known input values.
$\boldsymbol{a},\boldsymbol{p},\boldsymbol{q}$ and coefficient
$\boldsymbol{C}$ are unknown constants. The functional diagram of the proposed
performance model is shown in Fig. 2.

We compute the optimal values of these unknown constants (total:
$M=2n_{I}+n_{E}+1$) using the differential evolution algorithm. The aim is to
find the best-fitting values of these constants, by fitting the model to
experimentally measured execution times obtained with different values of the
internal and external parameters $I$ and $E$. Before going to the cost
function formulation of the differential evolution algorithm [36], we describe
the general methodology for obtaining the experimental data. For every
possible combination of values of intrinsic and extrinsic parameter there
will, in general, be too many combinations for an exhaustive grid search.
Therefore, we have applied random sampling to ensure that every hyperparameter
in the population has an equal opportunity of being selected for obtaining
measured times. Here, the methodology used for measured time is the time taken
for an iteration of an epoch. We compute the iteration time as the difference
between an iteration’s end time and starting time.

The experimental data for fitting the model comprises $N$ measurements with
randomly selected values of the parameters. We denote the values of the
intrinsic parameters by

$I_{i,k},\hskip 5.0pt\hskip 5.0pti\in[1,n_{I}],\hskip 5.0ptk\in[1,N]$ (5)

where $i$ indexes the components of the vector of parameters, and $k$ refers
to a given observation in the experiment. Similarly, the extrinsic parameters
are denoted by

$E_{j,k},\hskip 5.0pt\hskip 5.0ptj\in[1,n_{E}],\hskip 5.0ptk\in[1,N]$ (6)

The measured time for obervation $k$ is

$t_{k},\hskip 5.0ptk\in[1,N].$ (7)

Here, $i,j$ denote the input feature indices. $k\in[1,N]$ indicate the sample
index in dataset $\boldsymbol{D}$. $N$ is the number of input samples in
$\boldsymbol{D}$.

Figure 1: Internal processes involved in a convolutional neural network.
Figure 2: Functional diagram of proposed performance model.

###  III-B Global Optimization Using Differential Evolution

Given the generic expression as shown in equation (4), as mentioned in earlier
sub section, we find the best fit values of
$\boldsymbol{a},\boldsymbol{p},\boldsymbol{q}$ and $C$ by minimizing a cost
function. We formulate the cost function as the mean absolute difference
between the predicted execution time and the actual measured times as follows:

$f(x)=\frac{1}{N}\sum_{k=1}^{N}\left|t_{k}-\hat{t}_{k}(I_{k},E_{k},x)\right|$
(8)

where $N$ as number of data samples, $t_{k}$ is the measured time,
$\hat{t}_{k}$ is the predicted time derived from equation 4.

To solve the above optimization problem, we have used the differential
evolution algorithm (DE) using the cost function in equation 4. The mean
absolute error between the predicted times of the model and the measured times
is minimized, resulting in a value of the vector $x$ which represents the
best-fitting model. Recall that this vector encodes the coefficients and
powers of the terms in the model due to each of the hyperparamters; these can
then be used to make predictions of the execution time for any set of values
of the intrinsic and extrinsic paramters and furthermore, inspection of these
coefficients can provide insight into the relative importance and
computational complexity of the internal processes, as well as the scalability
of the external processes. For this work, we use the DE implementation from
the scipy python package, with default values of the hyperparameters. We
enforce limits of $(0\ldots 1000)$ for constants and coefficients ($a,C$) and
$-5\ldots 5$ for powers ($p,q$).

###  III-C Regularization

A globally optimized, unconstrained model may be prone to overfitting or
producing unstable solutions with high parameter variance. To address these
issues, we introduce a regularization term to the cost function.
Regularization achieves the best fit by introducing a penalizing term in the
cost function, which assigns a higher penalty to complex curves. So, we are
motivated to apply regularization to our performance model. Generally,
regularization can be defined as:

$f_{\mbox{\scriptsize reg}}(x)=f(x)+\lambda.L$ (9)

The parameter $\lambda$ controls the balance between bias and variance in the
model, where L represents the model’s complexity. There are two regularization
techniques: (a) Lasso regression (L1) and (b) Ridge regression (L2). L1
regularization, also known as lasso regression, includes the absolute value of
the coefficient magnitude as a penalty in the loss function. The resulting
solution from L1 regularization is sparse, meaning it tends to eliminate less
important features by setting their coefficients to zero. This is useful for
feature selection when dealing with a large number of features. On the other
hand, L2 regularization, or ridge regression, incorporates the squared
magnitude of the coefficient as a penalty in the loss function. The solution
obtained from L2 regularization is non-sparse and penalizes the model’s
complexity. The regularization parameter $\lambda$ penalizes all parameters
except the intercept, ensuring that the model generalizes the data and avoids
overfitting. Ridge regression uses the squared magnitude of the coefficient as
the penalty term. We have applied both L1 and L2 regularizations to the
performance model. The cost function for optimization with both L1 and L2
regularizations is as follows:

$\displaystyle
f(x)=\frac{1}{N}\sum_{k=1}^{N}\left|t_{k}-\hat{t}_{k}(I_{k},E_{k},x)\right|+\lambda.\sum_{k=1}^{N}\left|x\right|$
(10) $\displaystyle
f(x)=\frac{1}{N}\sum_{k=1}^{N}\left|t_{k}-\hat{t}_{k}(I_{k},E_{k},x)\right|+\lambda.\sum_{k=1}^{N}\left|x\right|^{2}$
(11)

Here, applying the regularization term $\lambda$ reduces the bias-variance
trade-off in the internal processes.

##  IV Experimental Evaluation

To evaluate the performance of the proposed model, we have applied our
approach to three popular deep learning frameworks (TensorFlow, PyTorch and
MxNet) and conducted extensive experiments. We assess the proposed performance
evaluation approach by modelling distributed training of a CNN architecture on
a multi-GPU system. The main goal is to investigate how well the predicted
execution time fits the experimentally measured time.

###  IV-A System Configuration

We implement the experiments on a single node containing three GEFORCE RTX
2080 GPUs, each with 2.60 GHz speed and 16 GB GPU RAM. The node also consists
of a 2.81 GHz speed CPU machine, 25 Gbps network bandwidth and a CUDA-10.2
with a Linux operating system. Furthermore, the node consists of various
software configurations/installations, including PyTorch 1.2.0, Torchvision
0.4.0, Python 3.6, TensorFlow 2.1.0 and MXnet 1.6.0.

###  IV-B Dataset and Model Selection

For three deep learning frameworks, we selected a CNN architecture, LeNet-5,
which Yann LeCun proposed in 1998 as a general common neural network structure
for handwritten font recognition. It consists of two convolutional layers, two
fully-connected layers, pooled layers for cross-combination and an output
layer that predicts values via the fully connected layer. Besides, LeNet-5
works well with handwritten datasets [37], it also reduces the number of
parameters and can automatically learn features from raw pixels [38].

We train LeNet-5 on three popular datasets, MNIST, fashion-MNIST and CIFAR-10,
using TensorFlow, PyTorch and MxNet, in a multi-GPU system. MNIST [39, 40] is
a database of handwritten digits derived by the National Institute of
Standards and Technology (NIST) for learning techniques and pattern
recognition methods with a little effort on pre-processing and formatting. It
contains 60,000 training and 10,000 testing images, divided into four files:
training set images, testing set images, training set labels and testing set
labels. Each image has 28 x 28 pixels. Fashion-MNIST [41] replaces the MNIST,
where each image has a 28x28 grayscale and is associated with a label from 10
classes. The CIFAR-10 dataset [42] comprises 60,000 images, classified into
ten classes - aeroplane, automobile, bird, cat, deer, dog, frog, horse, ship,
and truck. Each image has $32\times 32$ pixels, and each classification has
6,000 images.

###  IV-C Performance Metrics

The scalability and the mean absolute percentage error (MAPE) are selected as
performance metrics for run-time evaluation on three different frameworks.
Scalability is measured in the powers of external parameters as shown in (3).
The MAPE can be defined as:

$\displaystyle
MAPE=\frac{1}{N}\sum_{k=1}^{N}\frac{\left|t_{k}-\hat{t_{k}}\right|}{t_{k}}$
(12)

where $t_{k}$ = the measured value, $\hat{t_{k}}$ = the predicted value, n =
the total number of data points.

The experimental evaluation aims to evaluate the proposed performance model
and find the best-fit values using the differential evolution algorithm. The
MAPE is used to evaluate the closeness of this fit and the quality of the
performance model. The scaling parameters are used in our proposed model to
evaluate the performance of the deep learning frameworks.

###  IV-D Experiments

We have conducted a set of experiments to evaluate the proposed model from the
following aspects:

  1. 1.

Performance evaluation of deep learning frameworks using the proposed
performance model with and without regularization. Specifically, we have
applied the proposed performance model to three deep learning frameworks:
TensorFlow, MXnet, and PyTorch, under two circumstances, with and without
regularization.

  2. 2.

Comparison of the proposed model with the existing black-box machine learning
models. We have also compared our proposed model with two widely used models
including Random Forest Regression [43] and Support Vector Machine [44], and
demonstrated its performance and interpretability.

In our experiment, we have performed the distributed training of LeNet-5 on
MNIST, fashion-MNIST and CIFAR-10 datasets using the three deep learning
frameworks. The values of the experimental training parameters are created by
applying random sampling on a set of intrinsic and extrinsic parameters and
its corresponding average training time taken by a deep CNN architecture per
iteration. Table I shows intrinsic and extrinsic parameters and their possible
values. The intrinsic parameters are the model’s hyperparameters, including
kernel size, pooling size, activation function, etc. The number of GPUs and
the batch size are extrinsic factors since these affect the scaling over
multiple processes.

TABLE I: Parameters of the performance model, with ranges of values sampled in the experiments. Index | Name | Set of possible values considered
---|---|---
Intrinsic parameters
1 | Kernel size | {2,3,4,5}
2 | Pooling size | {2,3,4,5}
3 | Activation function | {Relu, Tanh, Sigmoid}
4 | Optimizer | {Adam, SGD}
5 | Image_dataset_name | {MNIST,Fashion-MNIST,CIFAR-10}
6 | Number of filters | {4,8,16,32,64}
7 | Learning rate | {0.1,0.01,0.001,$10^{-4}$, $10^{-5}$, $10^{-6}$}
8 | Padding_mode | {valid, same}
9 | Stride | {1,2,3}
10 | Dropout probability | {0.2,0.5,0.8}
Extrinsic parameters
11 | Number of GPUs | {1,2,3}
12 | Batchsize | {8,16,32,64,128}

The experiments involve several trials in which we measure the time for a
single training iteration using randomly selected intrinsic and extrinsic
parameter values. We conduct 1500 trials to prepare a dataset of 1500 data
samples. For each sample, we run three iterations with the same parameter
values, and take the median value of the measured time. The experimental data
for 900 trials are used to fit our performance model or train the standard
black-box models for comparison. The remaining 600 are used to test and
validate models. Finally, the experimental parameters are used to build three
performance evaluation models, such as the Differential evolution (DE)
algorithm with and without using regularization models and two standard black-
box models. We run each fit ten times with different random seeds to obtain
the mean and standard deviation for each of our fitted parameters. The
performance of these models and their corresponding results are explained in
the subsequent subsections.

TABLE II: Derived intrinsic and extrinsic parameters from the differential evolution-optimized performance models for the three deep learning frameworks. Parameters are given as the mean and standard deviation over ten fits. $a$ and $p$ represent coefficients and powers, respectively, of a term representing an intrinsic parameter, whereas $q$ is power in a multiplicative term representing an extrinsic (scaling) parameter. | Mxnet | Pytorch | TensorFlow
---|---|---|---
Intrinsic parameters | $a$ | $p$ | $a$ | $p$ | $a$ | $p$
Filter size | 554.87 $\pm$ 311.73 | -4.06 $\pm$ 0.53 | 423.36 $\pm$ 256.88 | -2.88 $\pm$ 1.04 | 346.73 $\pm$ 216.24 | -3.22 $\pm$ 0.78
Kernel size | 10.57 $\pm$ 7.05 | -4.10 $\pm$ 0.70 | 168.54 $\pm$ 123.27 | -2.34 $\pm$ 1.82 | 54.78 $\pm$ 32.91 | -4.00 $\pm$ 1.41
Pool size | 18.08 $\pm$ 5.17 | -4.21 $\pm$ 0.46 | 209.14 $\pm$ 186.87 | -3.31 $\pm$ 0.92 | 79.45 $\pm$ 53.53 | -3.48 $\pm$ 1.33
Learning rate | 459.50 $\pm$ 258.52 | 3.68 $\pm$ 0.62 | 489.52 $\pm$ 221.63 | 3.21 $\pm$ 0.70 | 458.34 $\pm$ 278.03 | 3.26 $\pm$ 0.91
Stride | 17.29 $\pm$ 6.12 | -0.83 $\pm$ 0.23 | 140.64 $\pm$ 138.62 | -0.63 $\pm$ 0.58 | 29.00 $\pm$ 14.54 | -1.85 $\pm$ 0.90
Dropout probability | 1.79 $\pm$ 0.75 | 2.24 $\pm$ 1.62 | 437.06 $\pm$ 184.32 | 1.80 $\pm$ 1.66 | 10.23 $\pm$ 9.51 | 1.87 $\pm$ 1.62
Same | 2.50 $\pm$ 0.97 | - | 11.02 $\pm$ 5.09 | - | 6.14 $\pm$ 1.54 | -
Valid | 1.56 $\pm$ 0.96 | - | 0.77 $\pm$ 1.81 | - | 1.61 $\pm$ 2.24 | -
Sigmoid | 23.25 $\pm$ 10.23 | - | 475.92 $\pm$ 139.65 | - | 251.57 $\pm$ 122.01 | -
Relu | 21.90 $\pm$ 10.40 | - | 475.56 $\pm$ 137.27 | - | 255.93 $\pm$ 122.35 | -
Tanh | 23.14 $\pm$ 10.30 | - | 444.48 $\pm$ 138.25 | - | 254.28 $\pm$ 121.27 | -
MNIST | 35.75 $\pm$ 12.81 | - | 815.62 $\pm$ 69.44 | - | 232.24 $\pm$ 108.77 | -
Fashion-MNIST | 35.94 $\pm$ 12.75 | - | 815.68 $\pm$ 68.39 | - | 231.33 $\pm$ 109.93 | -
CIFAR-10 | 18.57 $\pm$ 12.68 | - | 308.73 $\pm$ 53.32 | - | 124.56 $\pm$ 108.01 | -
SGD | 16.68 $\pm$ 10.10 | - | 361.65 $\pm$ 130.64 | - | 158.74 $\pm$ 109.25 | -
Adam | 16.85 $\pm$ 10.32 | - | 720.15 $\pm$ 123.99 | - | 168.55 $\pm$ 108.67 | -
Extrinsic parameters | $q$ | $q$ | $q$
Batchsize | -0.99 $\pm$ 0.003 | -1.13 $\pm$ 0.01 | -1.35 $\pm$ 0.08
No. of GPUs | -0.99 $\pm$ 0.004 | -1.029 $\pm$ 0.001 | -0.74 $\pm$ 0.001
Constant term | $C$ | $C$ | $C$
| 3.703 $\pm$ 0.017 | 12.677 $\pm$ 0.038 | 1.930 $\pm$ 0.122
TABLE III: Derived intrinsic and extrinsic parameters from the differential evolution-optimized performance models for the three deep learning frameworks using L2 regularization. Parameters are given as the mean and standard deviation over ten fits. $a$ and $p$ represent coefficients and powers, respectively, of a term representing an intrinsic parameter, whereas $q$ is power in a multiplicative term representing an extrinsic (scaling) parameter. | Mxnet | Pytorch | TensorFlow
---|---|---|---
Intrinsic parameters | $a$ | $p$ | $a$ | $p$ | $a$ | $p$
Filter size | 6.27$\pm$ 0.59 | 0.36 $\pm$ 0.01 | 6.07 $\pm$ 1.59 | 0.89 $\pm$ 0.05 | 8.39 $\pm$ 0.37 | 0.77 $\pm$ 0.01
Kernel size | 4.44 $\pm$ 0.65 | 0.50 $\pm$ 0.04 | 4.84 $\pm$ 1.90 | 2.02 $\pm$ 0.24 | 6.59 $\pm$ 0.29 | 2.04 $\pm$ 0.03
Pool size | 4.69 $\pm$ 0.33 | 0.52 $\pm$ 0.03 | 3.23 $\pm$ 0.83 | 1.55 $\pm$ 0.45 | 6.70 $\pm$ 0.67 | 1.98 $\pm$ 0.05
Learning rate | 3.62 $\pm$ 0.41 | -0.04 $\pm$ 0.003 | 3.75 $\pm$ 1.70 | -0.27 $\pm$ 0.02 | 4.40 $\pm$ 0.60 | -0.22 $\pm$ 0.007
Stride | 4.51 $\pm$ 0.40 | -0.99 $\pm$ 0.11 | 2.92 $\pm$ 1.54 | -0.83 $\pm$ 1.42 | 4.13 $\pm$ 0.59 | 2.46 $\pm$ 0.10
Dropout probability | 4.20 $\pm$ 0.67 | -0.35 $\pm$ 0.05 | 35.92 $\pm$ 1.15 | -5.00 $\pm$ 0.00 | 4.46 $\pm$ 0.43 | -1.94 $\pm$ 0.07
Same | 2.66 $\pm$ 0.51 | - | 2.08 $\pm$ 0.84 | - | 1.90 $\pm$ 0.58 | -
Valid | 1.50 $\pm$ 0.43 | - | -0.57 $\pm$ 1.56 | - | 0.49 $\pm$ 0.71 | -
Sigmoid | 2.18 $\pm$ 0.45 | - | 2.32 $\pm$ 1.15 | - | 1.41 $\pm$ 0.41 | -
Relu | 1.52 $\pm$ 0.33 | - | 3.21 $\pm$ 1.35 | - | 1.85 $\pm$ 0.64 | -
Tanh | 2.29 $\pm$ 0.39 | - | 2.93 $\pm$ 1.93 | - | 1.99 $\pm$ 0.71 | -
MNIST | 5.48 $\pm$ 0.52 | - | 3.37 $\pm$ 1.35 | - | 1.99 $\pm$ 0.72 | -
Fashion-MNIST | 7.73 $\pm$ 0.36 | - | 3.42 $\pm$ 1.56 | - | 2.28 $\pm$ 0.72 | -
CIFAR-10 | 1.00 $\pm$ 0.02 | - | 1.89 $\pm$ 1.00 | - | 1.63 $\pm$ 0.69 | -
SGD | 2.31 $\pm$ 0.36 | - | 2.16 $\pm$ 1.00 | - | 1.73 $\pm$ 0.46 | -
Adam | 1.78 $\pm$ 0.41 | - | 3.42 $\pm$ 1.45 | - | 2.01 $\pm$ 0.85 | -
Extrinsic parameters | $q$ | $q$ | $q$
Batchsize | -0.87 $\pm$ 0.005 | -1.00 $\pm$ 0.007 | -1.19 $\pm$ 0.01
No. of GPUs | -1.07 $\pm$ 0.007 | -1.01 $\pm$ 0.004 | -0.74 $\pm$ 0.005
Constant term | $C$ | $C$ | $C$
| 3.45$\pm$ 0.024 | 1.03 $\pm$ 0.07 | 12.62 $\pm$ 0.05

(a) TensorFlow

(b) MXnet

(c) Pytorch

Figure 3: The proposed performance model prediction and the actual measured
times in three deep learning frameworks without regularization

(a) TensorFlow

(b) MXnet

(c) Pytorch

Figure 4: The proposed performance model prediction and the actual measured
time in three deep learning frameworks with regularization

(a) TensorFlow

(b) MXnet

(c) Pytorch

Figure 5: Random forest regressor prediction and the actual measured time in
three deep learning frameworks

(a) TensorFlow

(b) MXnet

(c) Pytorch

Figure 6: Support vector machine prediction and the actual measured times in
three deep learning frameworks

(a) R2 values with different regularization values in three different
frameworks using L1 regularization

(b) R2 values with different regularization values in three different
frameworks using L2 regularization.

Figure 7: Effect of regularization.

(a) TensorFlow

(b) MXnet

(c) Pytorch

(d) Pytorch

Figure 8: Effect of regularization, with model coefficients plotted against regularization parameter. Constant coefficients of intrinsic parameters are plotted in (a), the power coefficients of intrinsic parameters are shown in (b), coefficients of categorical intrinsic parameters in (c), with powers of extrinsic parameters in (d).  TABLE IV: L1 and L2 regularization results in terms of the mean absolute percentage error (MAPE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) | L1 Regularization | L2 Regularization
---|---|---
| MXnet | PYtorch | TF | MXnet | PYtorch | TF
MAPE | 8% | 29% | 13% | 7% | 27% | 10%
MSE | 105.74 | 450.93 | 220.16 | 103.37 | 443.35 | 201.81
RMSE | 10.28 | 17.52 | 14.83 | 10.16 | 17.02 | 14.20

###  IV-E Results and Analysis

This section shows the results of our proposed performance model for three
popular deep neural networks, i.e., TensorFlow, MXnet, and PyTorch. We
evaluate the performance model with and without regularization and compare it
with standard black-box regression models such as Support Vector Machine and
Random Forest Regressor (RF). Tables II and III compare intrinsic parameters
and scalability in various frameworks with and without using regularization.
Table IV shows mean absolute percentage error values on predictions of the
performance models using L1 and L2 regularization.

####  IV-E1 Performance Evaluation of Deep Learning Frameworks using the
Proposed Performance Model without regularization

We have applied the differential evolution algorithm to our proposed model and
evaluated it using the three deep learning frameworks. The actual execution
time for training the model using the three frameworks is recorded, and
predicted execution times are also generated. Fig.3 shows the scatter graph of
the predicted execution times from the proposed model plotted against the
actual execution time for the test dataset. The linear fit to the straight
line determines how well the model can predict unseen configurations. We find
the best fit constant coefficients for all frameworks are shown in Table II.

The results show stable and consistent fits for the extrinsic parameters and
the additive constant $C$, indicating that the scalability results are
accurate. The higher variances in the intrinsic parameters are reduced by
using regularization. Table II shows that the model gives broadly consistent
performance for the constant coefficients, representing the relative
importance of the process controlled by categorical parameters. For instance,
Adam has a large constant for the activation function coefficients and takes
more training time than SGD in Pytorch and TensorFlow frameworks, while SGD
has the highest training time with the MXnet framework. The padding parameter,
which is categorical with two possible values valid and same. same shows
better performance for the valid mode.

####  IV-E2 Performance Evaluation of Deep Learning Frameworks using the
Proposed Performance Model with regularization

We have applied regularization to the cost function to the proposed
performance model to optimize the vector constants and reduce high variance in
intrinsic parameters in three deep learning frameworks. We applied both
regularizations to our model and compared the results of L1 and L2. The MAPE,
MSE, and RMSE results are better with L2 regularization, as shown in Table IV.
We therefore consider L2 regularization appropriate for our performance model
and applied various regularization parameter values in logarithmic scale in L1
and L2 to find the best value of the $\lambda$ parameter. In Figures 7(a) and
7(b), we see that the R2 score deteriorates when the $\lambda$ value is higher
than 0.001. For instance, when $\lambda$ = 0.001, the model fits well, and the
model gives broadly consistent performance for the constant coefficients and
represents the relative importance of the process controlled by categorical
parameters. Furthermore, in Table III, we can see that the model gives
consistent performance for the constant coefficients, representing the
relative importance of the processes controlled by categorical parameters. The
results show that the performance model using regularization is a generalised
model with optimized good fits in all the frameworks. For example, for padding
coefficients, same parameter takes more training time than valid parameter in
all frameworks. For activation function coefficients, Tanh takes more training
time than Relu and Sigmoid in MXnet and TensorFlow frameworks, while Relu
takes maximum time with Pytorch. Also, in terms of dataset coefficients, the
Fashion-MNIST dataset takes more training time than MNIST and CIFAR-10
datasets in all three frameworks.

####  IV-E3 Comparison of the Proposed Performance Model with Black Box Models

We have compared the proposed model with two standard black box models, i.e.,
Random Forest Regressor and Support Vector Machine. Generally, the random
forest regressor has better prediction accuracy due to its ensemble learning
shown in Fig.5. The result shows a good linear fit compared to the
differential evolution algorithm with and without regularization. However, the
drawback of the random forest regressor is that it cannot give any insights
into its internal working mechanism. Support vector machine regression is a
non-parametric technique because it depends on kernel functionality. It is
more productive in high-dimensional spaces. Fig.6 shows the predicted and
measured times of the support vector machine. The result shows a poor fit for
all the deep learning frameworks compared with the random forest regressor and
differential evolution algorithm with and without regularization. We evaluate
the fits using the mean absolute percentage error between predicted execution
time and actual times, as shown in Table V. Note that the performance of our
proposed model is slightly inferior to the random forest. However, the
proposed model can provide insights into the internal behaviour and
scalability, which are impossible with a black box model such as a random
forest.

TABLE V:  Mean Absolute Percentage Error (MAPE) on predictions of the performance models on the 300 instances in the evaluation dataset in seconds. | TensorFlow | MXnet | Pytorch
---|---|---|---
| Differential
---
evolution
5% | 5% | 12%
| Differential
---
evolution
with regularization
10% | 7% | 14%
| Random
---
forest
0.7% | 3% | 23%
| Support
---
vector
machine
16% | 21% | 54%
TABLE VI: nGPUs scaling power in various frameworks, nGPUs represent number of GPUs. Frameworks | nGPUs scaling power
---|---
TensorFlow | -0.74
MXnet | -0.99
Pytorch | -1.02

####  IV-E4 Scalability Analysis

Observing the coefficients $q$ from Table II and Table III, where $q$ is power
in a multiplicative term representing an extrinsic parameter, we see that the
extrinsic parameter coefficients are consistent in the proposed performance
model with and without regularization. As shown in Table VI, -1 indicates
ideal scaling, in which case the time is inversely proportional to the number
of GPUs. The coefficients in Pytorch and MXnet frameworks show better scaling
performance than TensorFlow. In TensorFlow, the value -0.73 is less than -1,
indicating sub-optimal scaling.

##  V Conclusion and Future Works

In this work, we have developed a generic performance model for deep learning
applications in a distributed environment with a generic expression of the
application execution time that considers the influence of both intrinsic and
extrinsic factors. We also formulated the proposed model as a global
optimization problem and solved it using regularization on a cost function and
differential evolution algorithm to find the best-fit values of the constants
in the generic expression. The proposed model has been evaluated on three
popular deep learning frameworks: TensorFlow, MXnet, and Pytorch, and shown to
provide accurate performance predictions and interpretability. Also, the
experimental results show that MXnet and Pytorch exhibit better scalability
performance than TensorFlow. Furthermore, the proposed method with
regularization has been found to optimize the vector constants and reduce high
variance in intrinsic parameters. The model can be applied to any distributed
deep learning framework without requiring any changes to the code and can
offer insight into the factors affecting deep learning application performance
and scalability. Future work may include evaluating the model’s performance on
various deep learning frameworks to assess its generalisation capability.

## References

  * [1] P. Ballester and R. M. Araujo, “On the performance of googlenet and alexnet applied to sketches,” in _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * [2] S. Targ, D. Almeida, and K. Lyman, “Resnet in resnet: Generalizing residual architectures,” _arXiv preprint arXiv:1603.08029_ , 2016.
  * [3] L. Wang, S. Guo, W. Huang, and Y. Qiao, “Places205-vggnet models for scene recognition,” _arXiv preprint arXiv:1508.01667_ , 2015.
  * [4] N. Aloysius and M. Geetha, “A review on deep convolutional neural networks,” in _2017 international conference on communication and signal processing (ICCSP)_. IEEE, 2017, pp. 0588–0592.
  * [5] F. Yan, O. Ruwase, Y. He, and T. Chilimbi, “Performance modeling and scalability optimization of distributed deep learning systems,” in _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2015, pp. 1355–1364.
  * [6] Y. Oyama, A. Nomura, I. Sato, H. Nishimura, Y. Tamatsu, and S. Matsuoka, “Predicting statistics of asynchronous sgd parameters for a large-scale distributed deep learning system on gpu supercomputers,” in _2016 IEEE International Conference on Big Data (Big Data)_. IEEE, 2016, pp. 66–75.
  * [7] M. Song, Y. Hu, H. Chen, and T. Li, “Towards pervasive and user satisfactory cnn across gpu microarchitectures,” in _2017 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2017, pp. 1–12.
  * [8] S. Shi, Q. Wang, P. Xu, and X. Chu, “Benchmarking state-of-the-art deep learning software tools,” in _2016 7th International Conference on Cloud Computing and Big Data (CCBD)_. IEEE, 2016, pp. 99–104.
  * [9] S. Shi, Q. Wang, and X. Chu, “Performance modeling and evaluation of distributed deep learning frameworks on gpus,” in _2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech)_. IEEE, 2018, pp. 949–957.
  * [10] S. Pllana, S. Benkner, F. Xhafa, and L. Barolli, “Hybrid performance modeling and prediction of large-scale computing systems,” in _2008 International Conference on Complex, Intelligent and Software Intensive Systems_ , 2008, pp. 132–138.
  * [11] T. Fahringer, S. Pllana, and J. Testori, “Teuta: Tool support for performance modeling of distributed and parallel applications,” in _International Conference on Computational Science_. Springer, 2004, pp. 456–463.
  * [12] T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, “Project adam: Building an efficient and scalable deep learning training system,” in _11th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 14)_, 2014, pp. 571–582.
  * [13] H. Qi, E. R. Sparks, and A. Talwalkar, “Paleo: A performance model for deep neural networks,” 2016.
  * [14] H. Kim, H. Nam, W. Jung, and J. Lee, “Performance analysis of cnn frameworks for gpus,” in _2017 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 2017, pp. 55–64.
  * [15] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard _et al._ , “$\\{$TensorFlow$\\}$: A system for $\\{$Large-Scale$\\}$ machine learning,” in _12th USENIX symposium on operating systems design and implementation (OSDI 16)_ , 2016, pp. 265–283.
  * [16] F. Seide and A. Agarwal, “Cntk: Microsoft’s open-source deep-learning toolkit,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 2135–2135.
  * [17] R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas, F. Bastien, J. Bayer, A. Belikov, A. Belopolsky _et al._ , “Theano: A python framework for fast computation of mathematical expressions,” _arXiv e-prints_ , pp. arXiv–1605, 2016.
  * [18] A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda, “S-caffe: Co-designing mpi runtimes and caffe for scalable deep learning on modern gpu clusters,” in _Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_ , 2017, pp. 193–205.
  * [19] R. Collobert, S. Bengio, and J. Mariéthoz, “Torch: a modular machine learning software library,” Idiap, Tech. Rep., 2002.
  * [20] T. Kavarakuntla, L. Han, M. Huw Lloyd, S. Annabel Latham, and S. B. Akintoye, “Performance analysis of distributed deep learning frameworks in a multi-gpu environment,” in _2021 IEEE 20th Intl Conf on Ubiquitous computing and communications(IUCC-2021), The 4th Intl Conf on Data science and Computational Intelligence(DSCI-2021)_ , 2021.
  * [21] R. Rakshith, V. Lokur, P. Hongal, V. Janamatti, and S. Chickerur, “Performance analysis of distributed deep learning using horovod for image classification,” in _2022 6th International Conference on Intelligent Computing and Control Systems (ICICCS)_. IEEE, 2022, pp. 1393–1398.
  * [22] Z. Lin, X. Chen, H. Zhao, Y. Luan, Z. Yang, and Y. Dai, “A topology-aware performance prediction model for distributed deep learning on gpu clusters,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 2020, pp. 2795–2801.
  * [23] D. Didona, F. Quaglia, P. Romano, and E. Torre, “Enhancing performance prediction robustness by combining analytical modeling and machine learning,” in _Proceedings of the 6th ACM/SPEC international conference on performance engineering_ , 2015, pp. 145–156.
  * [24] T. Kavarakuntla, L. Han, H. Lloyd, A. Latham, A. Kleerekoper, and S. B. Akintoye, “A Generic Performance Model for Deep Learning in a Distributed Environment, ” in _2022 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2022, pp. 191.
  * [25] R. Storn and K. Price, “Differential evolution – a simple and efficient heuristic for global optimization over continuous spaces,” _J. of Global Optimization_ , vol. 11, no. 4, pp. 341–359, dec 1997. [Online]. Available: https://doi.org/10.1023/A:1008202821328
  * [26] C.-Y. Lee and C.-H. Hung, “Feature ranking and differential evolution for feature selection in brushless dc motor fault diagnosis,” _Symmetry_ , vol. 13, no. 7, 2021. [Online]. Available: https://www.mdpi.com/2073-8994/13/7/1291
  * [27] W. Yang, E. M. D. Siriwardane, R. Dong, Y. Li, and J. Hu, “Crystal structure prediction of materials with high symmetry using differential evolution,” _Journal of Physics: Condensed Matter_ , vol. 33, 2021.
  * [28] S. Saha and R. Das, “Exploring differential evolution and particle swarm optimization to develop some symmetry-based automatic clustering techniques: Application to gene clustering,” _Neural Comput. Appl._ , vol. 30, no. 3, pp. 735–757, aug 2018. [Online]. Available: https://doi.org/10.1007/s00521-016-2710-0
  * [29] Y.-H. Li, J.-Q. Wang, X.-J. Wang, Y.-L. Zhao, X.-H. Lu, and D.-L. Liu, “Community detection based on differential evolution using social spider optimization,” _Symmetry_ , vol. 9, no. 9, 2017. [Online]. Available: https://www.mdpi.com/2073-8994/9/9/183
  * [30] M. Baioletti, A. Milani, and V. Santucci, “Learning bayesian networks with algebraic differential evolution,” in _Parallel Problem Solving from Nature – PPSN XV_. Cham: Springer International Publishing, 2018, pp. 436–448.
  * [31] M. F. Ahmad, N. A. M. Isa, W. H. Lim, and K. M. Ang, “Differential evolution: A recent review based on state-of-the-art works,” _Alexandria Engineering Journal_ , vol. 61, no. 5, pp. 3831–3872, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S111001682100613X
  * [32] D. Liu, D. Hong, S. Wang, and Y. Chen, “Genetic algorithm-based optimization for color point cloud registration,” _Frontiers in Bioengineering and Biotechnology_ , vol. 10, 2022. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fbioe.2022.923736
  * [33] M. Baioletti, G. Di Bari, A. Milani, and V. Poggioni, “Differential evolution for neural networks optimization,” _Mathematics_ , vol. 8, no. 1, 2020. [Online]. Available: https://www.mdpi.com/2227-7390/8/1/69
  * [34] N. Ikushima, K. Ono, Y. Maeda, E. Makihara, and Y. Hanada, “Differential evolution neural network optimization with individual dependent mechanism,” in _2021 IEEE Congress on Evolutionary Computation (CEC)_ , 2021, pp. 2523–2530.
  * [35] R. A. Venkat, Z. Oussalem, and A. K. Bhattacharya, “Training convolutional neural networks with differential evolution using concurrent task apportioning on hybrid cpu-gpu architectures,” in _2021 IEEE Congress on Evolutionary Computation (CEC)_ , 2021, pp. 2567–2576.
  * [36] K. Fleetwood, “An introduction to differential evolution,” in _Proceedings of Mathematics and Statistics of Complex Systems (MASCOS) One Day Symposium, 26th November, Brisbane, Australia_ , 2004, pp. 785–791.
  * [37] S. Park, J. Lee, and H. Kim, “Hardware resource analysis in distributed training with edge devices,” _Electronics_ , vol. 9, no. 1, p. 28, 2020.
  * [38] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the recent architectures of deep convolutional neural networks,” _ArXiv_ , vol. abs/1901.06032, 2019. [Online]. Available: http://arxiv.org/abs/1901.06032
  * [39] Y. LeCun, C. Cortes, and Christopher, “The mnist database of handwritten digits,” 2020. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [40] L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” _IEEE Signal Processing Magazine_ , vol. 29, pp. 141–142, 2012.
  * [41] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” _ArXiv_ , vol. abs/1708.07747, 2017.
  * [42] F. O. Giuste and J. C. Vizcarra, “Cifar-10 image classification using feature ensembles,” _ArXiv_ , vol. abs/2002.03846, 2020.
  * [43] T. K. Ho, “Random decision forests,” in _Proceedings of 3rd international conference on document analysis and recognition_ , vol. 1. IEEE, 1995, pp. 278–282.
  * [44] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.