# Very High-energy Afterglow Emission of GRB 190829A: Evidence for Its
Hadronic Origin?

Sarira Sahu<EMAIL_ADDRESS>Instituto de Ciencias Nucleares,
Universidad Nacional Autónoma de México,
Circuito Exterior, C.U., A. Postal 70-543, 04510 Mexico DF, Mexico  Isabel
Abigail Valadez Polanco<EMAIL_ADDRESS>Facultad de Ingeniería,
Universidad Autónoma de Yucatán,
Industrias No Contaminantes S/N, Sin Nombre de Col 27, Mérida, Yucatán, México
Subhash Rajpoot<EMAIL_ADDRESS>Department of Physics and
Astronomy, California State University,
1250 Bellflower Boulevard, Long Beach, CA 90840, USA

###### Abstract

The detection of multi-TeV gamma-rays from the afterglow phase of GRB 190829A
by High Energy Stereoscopic System (H.E.S.S.) telescope is an addition to the
already existing list of two more GRBs observed in the very high energy (VHE)
gamma-rays in recent years. Jets of blazars and GRBs have many similarities
and the photohadronic model is very successful in explaining the VHE gamma-ray
spectra from the high energy blazars. Recently, the photohadronic model has
been successfully applied to study the sub-TeV gamma-rays from the afterglow
phases of GRB 180720B and GRB 190114C. We employed this model again to explain
the VHE spectra observed for the two consecutive nights from GRB 190829A. We
show that the spectra of GRB 190829A can be due to the interactions of high
energy protons with the synchrotron self-Compton photons in the forward shock
region of the GRB jet, similar to the low emission state of the VHE flaring
events of high energy blazars. We speculate that, if in future, it is possible
to observe the VHE gamma-ray spectra from nearby GRBs in their afterglow
phases, then some of them could only be explained by employing two different
spectral indices. If confirmed, such VHE spectra could be interpreted as a
result of the interactions of the high energy protons with the photons, both
from the synchrotron background and the synchrotron self-Compton background in
the forward shock region.

Particle astrophysics (96), Blazars (164), Gamma-ray bursts (629),
Relativistic jets (1390)

mainReferences appendixReferences (Appendix)

##  1 Introduction

The birth of Gamma-ray bursts (GRBs) signal the formation of stellar-mass
black holes or fast rotating magnetars either from the collapse of massive
stars or from the merger of very compact binaries (Piran (2004); Kumar & Zhang
(2015) and Refs. therein). The afterglow in GRBs takes place soon after the
prompt phase and the standard model to understand them is the external shock
model. According to this model, the relativistic ejecta (also called fireball)
gets decelerated by the circumstellar medium and as a consequence two shocks
are developed. The one which lasts longer is the forward shock that is still
penetrating through the medium while the short-lived second one is the reverse
shock propagating into the fireball. Gamma-rays above $\sim 100$ MeV are
believed to be produced from electron synchrotron in the afterglow shocks
(Sari & Esin, 2001; Kumar & Barniol Duran, 2009; Ghisellini et al., 2010; Wang
et al., 2010). Also, joint observations of X-rays by Swift-X-ray Telescope
(XRT) and gamma-rays by Fermi Large Area Telescope (LAT) from many GRBs are
consistent with a single spectral component (Ajello et al., 2018). However, it
is difficult to produce photons above a few GeV in the context of synchrotron
mechanism unless an unrealistically large bulk Lorentz factor is employed
(Razzaque et al., 2010). The fireball model also predicts the origin of GeV-
TeV photons and the emission can last from minutes to several hours (Piran,
2004; Kumar & Zhang, 2015). The synchrotron self-Compton (SSC) process is the
favored one and widely used mechanism to interpret the very high energy (VHE
$>100$ GeV) photons in the afterglow era, where the relativistic electrons
upscatter their own synchrotron radiation(Waxman, 1997; Zhang & Mészáros,
2001; Sari & Esin, 2001; Kumar & Zhang, 2015). In this scenario, the SSC
emission takes place from a constant density circumburst medium or from the
Comptonization of X-ray photons in the afterglow shock (Wang et al., 2019;
Derishev & Piran, 2019).

Recently, the VHE photons were detected from the afterglow phases of the GRB
180720B by Major Atmospheric Gamma Imaging Cherenkov (M.A.G.I.C.) telescopes
and from the GRB 190114C and GRB 190829A by H.E.S.S. telescope (Abdalla et
al., 2019; Acciari et al., 2019a; Abdalla et al., 2021). The detection of VHE
photons from GRBs by the ground based Imaging Air Cherenkov Telescopes
(IACTs), like the M.A.G.I.C. and the H.E.S.S., is a new development in the
study of GRBs in the VHE $\gamma$-ray regime and will provide vital
information on the physical processes responsible for the radiation mechanisms
and particle acceleration in an extreme environment (Mészáros, 2006; Kumar &
Zhang, 2015; Fraija et al., 2019).

GRB 190829A was first detected by the Gamma-Ray Burst Monitor (GBM) on board
the Fermi gamma-ray Space Telescope on 29 August 2019 at 19:55:53 universal
time (UT, $T_{0}$) (Lesage et al., 2019). Subsequently, it was observed and
followed in the multiwavelength by several other telescopes (Abdalla et al.,
2021). The afterglow was also observed by ground based optical, infra-red and
radio telescopes starting 1318 s after the GBM trigger and observations were
continued for several days afterwards. The measured redshift of the host
galaxy is $z=0.0785\pm 0.0005$ (de Naurois, 2019; Valeev et al., 2019), making
this the nearest GRB detected in VHE so far. The H.E.S.S. telescope observed
multi-TeV gamma-rays from 4.3 to 55.9 hours after the prompt emission with a
direction consistent with the location of the source. GRB 190829A has some
observational peculiarities when compared with the other two GRBs, GRB 180720B
and GRB 190114C, that have been recently observed in VHE. It has two episodes
of prompt emission, separated by a quiescent time gap of about 40 s, and both
of them are of different nature. Also the isotropic equivalent luminosity of
GRB 190829A is $L_{iso}\sim 10^{49}\,\mathrm{erg\,s^{-1}}$ which is smaller
than the typical long GRBs, and possibly belongs to low-luminosity GRB (LLGRB)
category (Chand et al., 2020). Association of the GRB 190829A with the broad-
line type-Ic supernova SN 20190yw has been established (Hu et al., 2021).

Observations of VHE photons at late times ($>100$s) are difficult to
comprehend due to the substantial decrease in the bulk Lorentz factor of the
shock wave. Thus, various radiation mechanisms mostly based on the Inverse
Compton scattering, Proton synchrotron model and their variants are proposed
to explain the origin of VHE $\gamma$-rays during the afterglow phase. The SSC
(Derishev & Piran, 2019; Wang et al., 2019; Vurm & Beloborodov, 2017) and/or
the external inverse-Compton (EIC) (Zhang et al., 2020) scenarios are used to
explain the sub-TeV emission from GRBs. The extremely high energy protons in
the magnetic field of the afterglow shock can produce the observed gamma-ray
spectrum through the synchrotron process(Totani, 1998; Razzaque et al., 2010).
Also, the interaction of these energetic protons with the background photons
in the afterglow shock can produce the observed VHE gamma-rays through the
photopion process (Asano & Meszaros, 2012; Sahu & López Fortín, 2020). The
advantages and disadvantages of leptonic and hadronic models are reviewed in
(Kumar & Zhang, 2015).

Several studies on the emission mechanisms in blazars and GRBs show many
similarities (Nemmen et al., 2012; Wang & Wei, 2011; Wu et al., 2016). It is
observed that the GRB afterglows have the same radiation mechanism as the BL
Lac objects (Wang & Wei, 2011). A similar correlation of the synchrotron
luminosity and Doppler factor between GRBs and AGNs has been found (Wu et al.,
2011). Also, the relativistic jets in AGNs and GRBs have a similar energy
dissipation efficiency (Nemmen et al., 2012). All the above studies show that,
despite order of magnitude differences in their masses and bulk Lorentz
factors, the jets in GRBs and blazars have many characteristics in common.
Thus, it is imperative to study the VHE emission mechanisms in the afterglow
phases of GRBs by using the common mechanisms and processes that are being
used to study the multi-TeV flaring of high energy blazars.

In studying the multi-TeV flaring from high-energy peaked blazars (HBLs) we
have made use of the photohadroinc model (Sahu et al., 2020; Sahu, 2019; Sahu
et al., 2019). In this model, within the blazar jet, the Fermi accelerated
high energy protons interact with the background seed photons through the
process $p\gamma\rightarrow\Delta^{+}$ and the subsequent decay of the
$\Delta$-resonance produces VHE $\gamma$-rays through intermediate neutral
pion decays. This model is very successful in explaining the VHE $\gamma$-ray
spectrum from several HBLs. Recently, keeping in mind the similarity between
the emission mechanisms in GRBs and blazars, we have applied the photohadronic
model to explain the sub-TeV emission from the GRB 190114C and GRB 180720B and
explain the VHE $\gamma$-ray spectra of both these GRBs extremely well (Sahu &
López Fortín, 2020). We have also shown that the VHE spectrum of GRBs 190114C
is due to the interaction of Fermi accelerated high energy protons with the
SSC photons while the spectrum of GRB 180720B is from the interaction of high
energy protons with the synchrotron photons in the external forward shock
region.

In this work we wish to exploit the success of the photohadronic model once
again to explain the VHE afterglow emission from the GRB 190829A.

##  2 Photohadronic scenario

The kinematical condition to produce VHE photons from the $p\gamma$
interaction is given by (Sahu, 2019)

$E_{\gamma}\epsilon_{\gamma}=\frac{0.032\,\Gamma\,{\cal
D}}{(1+z)^{2}}\,\mathrm{{\it GeV}^{2}},$ (1)

where the observed VHE photon energy is $E_{\gamma}$ and $\epsilon_{\gamma}$
is the background seed photon energy. $\Gamma$ and ${\cal D}$ are the bulk
Lorentz factor and the Doppler factor respectively. The observed HBLs and GRBs
have their jets beaming towards the observer on earth, so we have
$\Gamma\simeq{\cal D}$. $z$ is the redshift of the source. In the
photohadronic process, the observed VHE photon carries about 10% of the proton
energy, i.e., $E_{\gamma}\simeq 0.1\,E_{p}$, where $E_{p}$ is the proton
energy.

The VHE photons from extragalactic sources get attenuated due to their
interactions with the extragalactic background light (EBL) by producing
$e^{+}e^{-}$ pairs (Stecker et al., 1992; Ackermann et al., 2012). This
reduces the VHE flux by a factor $e^{-\tau_{\gamma\gamma}}$, where
$\tau_{\gamma\gamma}$ is the optical depth for the lepton pair production
process which depends on $E_{\gamma}$ and $z$. In the photohadronic model, the
observed VHE $\gamma$-ray flux $F_{\gamma}$ is proportional to
$E^{2}_{\gamma}dN/dE_{\gamma}$ and is given by (Sahu (2019) and Refs. therein)

$F_{\gamma}(E_{\gamma})=F_{0}\left(\frac{E_{\gamma}}{{\it
TeV}}\right)^{-\delta+3}e^{-\tau_{\gamma\gamma}}=F_{\gamma,int}\,e^{-\tau_{\gamma\gamma}}.$
(2)

The normalization factor $F_{0}$ can be fixed from the observed spectrum. The
spectral index $\delta=\alpha+\beta$ is the free parameter in the model and
$F_{\gamma,int}$ is the intrinsic flux. The Fermi accelerated protons in the
jet have a power-law profile $dN/dE_{p}\propto E^{-\alpha}_{p}$ and the
spectral index $\alpha\geq 2$. Here we use the generally accepted value of
$\alpha=2.0$ (Dermer & Schlickeiser, 1993). $\beta$ is the spectral index of
the background seed photons and it is further observed that for HBLs the seed
photon flux is also a power-law
$\Phi_{\gamma}\propto\epsilon^{\beta}_{\gamma}\propto E^{-\beta}_{\gamma}$.
However, for GRBs, the $\beta$ value can be either positive or negative as
shown in (Sahu & López Fortín, 2020). Also, the sign of $\beta$ locates the
seed photon background in the synchrotron regime ($\beta<0$) or in the SSC
regime ($\beta>0$). By fitting the observed VHE spectrum, the value of
$\delta$ is fixed and it automatically fixes the value of $\beta$. For HBLs,
we have shown that the value of $\delta$ is always in the range
$2.5\leq\delta\leq 3.0$ (Sahu et al., 2019).

Figure 1:  The multi-TeV $\gamma$-ray spectrum observed by H.E.S.S. telescope
from GRB 190829A on the first night is fitted with the photohadronic model.
The normalization constant $F_{0}$ is expressed in units of
$\mathrm{erg\,cm^{-2}\,s^{-1}}$. The blue shaded region corresponds to
$F_{0}=(1.07^{+0.18}_{-0.11})\times 10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$
and $\delta=3.0^{+0.16}_{-0.10}$. The intrinsic flux for the photohadronic
model is also shown. The photohadoronic fit is compared with the power-law+EBL
fit, where $\gamma^{int}_{VHE}=2.06$, $E_{0}=0.556$ TeV and $F_{PW}=1.12\times
10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ are used. The light grey shaded
region is between $\gamma^{int}_{VHE}=2.42$ (upper edge) and
$\gamma^{int}_{VHE}=1.7$ (lower edge) where the statistical and the systematic
errors are taken into account. (Abdalla et al., 2021).  Figure 2:  The VHE
$\gamma$-ray spectrum observed by H.E.S.S. telescope from GRB 190829A (second
night) is fitted with the photohadronic model and compared with the power-
law+EBL fit ($\gamma^{int}_{VHE}=1.86$, $E_{0}=0.741$ TeV and
$F_{PW}=2.03\times 10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ are used ). The
light grey shaded region is between $\gamma^{int}_{VHE}=2.29$ (upper edge) and
$\gamma^{int}_{VHE}=1.43$ (lower edge) when the statistical and the systematic
errors are taken into account (Abdalla et al., 2021). The intrinsic flux for
the photohadronic model is a flat curve. The normalization constant $F_{0}$ is
expressed in units of $\mathrm{erg\,cm^{-2}\,s^{-1}}$. The blue shaded region
corresponds to $F_{0}=(1.99^{+0.70}_{-0.60})\times
10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ and $\delta=3.0^{+0.33}_{-0.40}$.

##  3 results

The H.E.S.S. collaboration observed the afterglow of the GRB 190829A for three
consecutive nights (Abdalla et al., 2021). In the first night it was observed,
starting at $T_{0}+4.3$ hours, for 3.6 hours. In the second night the
observation started at $T_{0}+27.2$ hours and continued for 4.7 hours. In the
third night the beginning time was at $T_{0}+51.2$ hours and lasted for 4.7
hours. Although H.E.S.S. telescope observed gamma-rays in all the nights, the
signal in the last night was too weak to construct the spectrum. In the first
night the observed spectrum was in the energy range $0.18\,{\it TeV}\leq
E_{\gamma}\leq 3.3\,{\it TeV}$ and in the second night the spectrum was in the
energy range $0.18\,{\it TeV}\leq E_{\gamma}\leq 1.4\,{\it TeV}$. The
differential photon spectra of both the nights are perfectly compatible with a
power-law with the EBL correction. However, it is shown that the values of the
spectral indices are not the same for both the nights (Abdalla et al., 2021).

We fitted the spectra of both the nights using the photohadronic model. It is
observed that the spectrum of the first night can be fitted with the spectral
index $\delta=3.0$ and the normalization factor $F_{0}=1.07\times
10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ as shown in Fig. 1. The statistical
significance of this fit is $91.7\%$. Here, we have used the $\chi^{2}$ fit to
calculate the statistical significance. To account for the EBL effect, we have
used the EBL model of Franceschini et al. (2008). However, the EBL model of
Dominguez et al. (2011) also gives similar result. Our fit shows that the
decrease in the flux is due to the EBL correction only. For $E_{\gamma}=3.3$
TeV the flux is depleted by about 73%.

In the photohadronic model $F_{0}$ and $\delta$ serve as two free parameters.
Although a best fit is obtained for $\delta=3.0$ and $F_{0}=1.07\times
10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$, it is important to estimate the
errors in both these parameters. By keeping one parameter free while the other
one is frozen at its optimum value we estimated the $1\sigma$ error on these
parameters which give $F_{0}=(1.07\pm 0.23)\times
10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ and $\delta=3.0\pm 0.2$. Also, with
the simultaneous variation of the two free parameters, the $1\sigma$
confidence interval defined by $\chi^{2}_{min}+2.3$ (Zyla et al., 2020), is
calculated and the errors in the parameters are respectively
$F_{0}=(1.07^{+0.18}_{-0.11})\times 10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$
and $\delta=3.0^{+0.16}_{-0.10}$. The blue shaded region in Fig. 1 corresponds
to these values of $F_{0}$ and $\delta$.

We have also compared our fit with the power-law plus EBL fit. For the power-
law plus EBL fit the flux is defined by
$F_{\gamma}=F_{PW}(E_{\gamma}/E_{0})^{-\gamma^{int}_{VHE}+2}\,\mathrm{e^{-\tau_{\gamma\gamma}}}$
and the central values of $\gamma^{int}_{VHE}=2.06$, $E_{0}=0.556$ TeV and
$F_{PW}=1.12\times 10^{-11}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ are used (Abdalla
et al., 2021) and the statistical significance of this fit is $86.3\%$. Taking
into account the statistical and the systematic errors, we have shown the
light grey region for this fit (with $\gamma^{int}_{VHE}=2.42$, the upper edge
and $\gamma^{int}_{VHE}=1.7$, the lower edge of the grey region). According to
the classification of the VHE flaring events of HBLs (Sahu et al., 2019),
$\delta=3.0$ corresponds to low emission state and the intrinsic flux is a
flat curve independent of energy. This value of $\delta$ corresponds to the
seed photon spectral index $\beta=1.0$. In the context of HBL flaring it is
shown that positive value of $\beta$ corresponds to seed photons in the SSC
regime. Thus, to produce this VHE spectrum, the high energy protons in the
energy range $1.8\,TeV\leq E_{p}\leq 33.0\,{\it TeV}$ should interact with the
SSC photons in the forward shock region.

In the photohadronic model, the optical depth for the $\Delta^{+}$ production
during the afterglow is
$\tau_{p\gamma}=n^{\prime}_{\gamma}\sigma_{\Delta}R^{\prime}$, where
$n^{\prime}_{\gamma}$ is the comoving SSC photon density in the forward shock
region and $R^{\prime}\sim 2.9\times 10^{16}$ cm is the comoving distance from
the central engine (Abdalla et al., 2021). For the above process we assume a
mild efficiency by taking $\tau_{p\gamma}\,<1$ and this gives
$n^{\prime}_{\gamma}<7\times 10^{10}\,\mathrm{cm^{-3}}$. Also, the $e\gamma$
interaction is taking place in the same background and we determine
$n^{\prime}_{\gamma}<5.2\times 10^{7}\,\mathrm{cm^{-3}}$. So, by taking
$n^{\prime}_{\gamma}\sim 10^{8}\,\mathrm{cm^{-3}}$ we get
$\tau_{p\gamma}\,\sim 1.5\times 10^{-3}$. The integrated VHE photon flux in
the energy range $0.18\,TeV\leq E_{\gamma}\leq 3.5\,TeV$ is $F_{\gamma}\simeq
1.8\times 10^{-11}\mathrm{erg\,cm^{-2}\,s^{-1}}$ and corresponds to luminosity
$L_{\gamma}\simeq 2.6\times 10^{44}\,\mathrm{erg\,s^{-1}}$. The isotropic-
equivalent energy emitted in VHE during the 3.6 hours period in the first
night is determined as $E^{iso}_{\gamma}\simeq 3.4\times
10^{48}\,\mathrm{erg}$. By taking $\tau_{p\gamma}\,\sim 1.5\times 10^{-3}$,
the proton luminosity is estimated to be $L_{p}\simeq 1.3\times
10^{48}\,\mathrm{erg\,s^{-1}}$. Thus we conclude that a typical LLGRB has much
lower energy output than a long GRB (Acciari et al., 2019a, b; Sahu & López
Fortín, 2020).

On the second night the GRB jet is decelerated compared to the first night and
the VHE photon energy is also decreased. So, if the spectrum of the first
night is fitted with $\delta=3.0$ corresponding to a low emission state, the
spectrum of the second night can not be fitted with $\delta<3.0$ as this
corresponds to high or very high emission state depending on the value of
$\delta$ (Sahu et al., 2019). However, the VHE spectrum of the second night is
also fitted with the photohadronic model and a good fit is obtained for
$\delta=3.0$ and $F_{0}=1.99\times 10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$.
As there are only three points, we do not calculate the statistical
significance of it. Again, as $\delta=3.0$, this is also in the low emission
state and the background photon flux is proportional to $\epsilon_{\gamma}$.
The intrinsic spectrum is a flat curve and is independent of energy. We also
estimate the $1\sigma$ errors in the parameters $\delta$ and $F_{0}$ as before
for the VHE spectrum of the second night. As this spectrum has only three data
points the estimated $1\sigma$ errors are large and we obtain $F_{0}=(1.99\pm
0.74)\times 10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ and $\delta=3.0\pm 0.35$
respectively by varying one parameter while the other one is frozen at its
optimum value. With simultaneous variation of the free parameters $F_{0}$ and
$\delta$, the $1\,\sigma$ error in these parameters are given by
$F_{0}=(1.99^{+0.70}_{-0.60})\times 10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$
and $\delta=3.0^{+0.33}_{-0.40}$ respectively. In Fig. 2, the blue shaded
region corresponds to these errors in $F_{0}$ and $\delta$ respectively. Also,
the photohadronic fit is compared with the the power-law plus EBL fit. For
$E_{\gamma}<1\,{\it TeV}$, the photohadronic fit is different from the power-
law plus EBL fit.

In the photohadronic scenario, the VHE spectrum is produced from the
interactions of high energy protons in the energy range $1.8\,TeV\leq
E_{p}\leq 14\,TeV$ with the SSC seed photons in the forward shock region as in
the first night. Although our photohadronic fit (for $\delta=3.0$) is similar
to the power-law plus EBL fit (here, the central values of
$\gamma^{int}_{VHE}=1.86$, $E_{0}=0.741$ TeV and $F_{PW}=2.03\times
10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ are used), our spectrum falls
slightly faster than the latter fit. Also, below 1 TeV, the behavior of both
the fits are slightly different as can be seen from Fig. 2. The integrated VHE
flux in the energy range $0.18\,TeV\leq E_{\gamma}\leq 1.4\,TeV$ is
$F_{\gamma}=3.0\times 10^{-12}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ and the
luminosity is $L_{\gamma}\simeq 4.4\times 10^{43}\,\mathrm{erg\,s^{-1}}$. In
the second night the VHE emission was observed for 4.7 hours. The isotropic-
equivalent energy emitted during this period is $E^{iso}_{\gamma}\simeq
7.5\times 10^{47}\,\mathrm{erg}$. Taking $R^{\prime}\sim 10^{17}$ cm and
assuming a mild efficiency for the $p\gamma$ process, we get
$n^{\prime}_{\gamma}<2\times 10^{10}\,\mathrm{cm^{-3}}$. By taking the SSC
photon density $n^{\prime}_{\gamma}\sim 10^{7}\,\mathrm{cm^{-3}}$, we get
$\tau_{p\gamma}\,\sim 5\times 10^{-4}$ and the corresponding proton luminosity
is $L_{p}\simeq 6.6\times 10^{47}\,\mathrm{erg\,s^{-1}}$. Thus, in the second
night $L_{p}$ and $E^{iso}_{\gamma}$ are about 51% and 22% of the first night
values respectively.

From the multiwavelength modeling of the X-ray to VHE $\gamma$-ray data of GRB
190829A, the value of $\Gamma$ is found to be $\Gamma=4.7$ for the first night
and $\Gamma=2.6$ for the second night respectively (Abdalla et al., 2021).
Using these values of $\Gamma$ in the photohadronic model, we estimate the
seed photon energies for both the nights. By taking $E_{\gamma}=3.3$ TeV as
the highest photon energy in the first night, the seed photon energy is found
to be $\epsilon_{\gamma}=184$ keV. Similarly, the highest photon energy for
the second night is $E_{\gamma}=1.4$ TeV, and this gives
$\epsilon_{\gamma}=56$ keV. The calculated values of $\epsilon_{\gamma}$ for
both these nights are low and belong to the synchrotron regime (Sahu et al.,
2019). According to the photohadronic scenario, since the seed photon energy
$\epsilon_{\gamma}$ should be in the SSC regime, we expect that the
$\epsilon_{\gamma}$ value should be in the $\mathcal{O}$ (few MeV) range.
Thus, by taking $\epsilon_{\gamma}\sim 1$ MeV for the first night, we get
$\Gamma\sim 11$, which is a mild value and compatible with the seed photon
energy in the SSC regime. Similarly we can also estimate $\Gamma$ for the
second night. However, it turns out that the VHE afterglow of the long GRBs,
GRB 190114C and GRB 180720B have large $\Gamma$ (Sahu & López Fortín, 2020) as
their synchrotron spectra get extended to $\sim 100$ MeV.

The photohadronic model works well for $E_{\gamma}\gtrsim 100$ GeV and to
produce gamma-rays below this energy the required seed photon energy is
$\epsilon_{\gamma}>72\,{\it MeV}$ which is in SSC regime. The density of these
seed photons is small which makes the $\Delta$ production inefficient from
$p\gamma$ interactions. Also for $E_{\gamma}\,<\,100\,{\it GeV}$ the leptonic
processes are much more important than the photohadronic process and the
latter can be neglected to explain the spectral energy distribution in this
energy regime. Also, in the photohadronic scenario, for $E_{\gamma}\gtrsim
100$ GeV, the main contribution to the VHE spectrum comes from the $p\gamma$
process. Hence, leptonic contributions to the VHE spectrum are neglected.

##  4 Conclusions

GRB 190829A is the third GRB observed in the VHE $\gamma$-rays and the second
GRB observed by the H.E.S.S. telescope in the afterglow epoch between
$T_{0}+4.3$ to $T_{0}+56$ hours. Due to many similarities between the HBL
flaring and the GRB emission, we employed the successful photohadronic model
to study the VHE afterglow of the GRB 190829A. The observed multi-TeV spectra
of two consecutive nights from GRB 190829A can be explained well by the
photohadronic model with the inclusion of the EBL correction. However, the
$\gamma\gamma$ absorption within the jet is negligible in our model. We have
shown that the interactions of high energy protons with the SSC seed photons
in the forward shock region seems to be responsible for the production of VHE
$\gamma$-rays observed in both the nights. These spectra are similar to the
spectrum observed from HBLs during low state emission corresponding to a
spectral index $\delta=3.0$. Also this value of $\delta$ implies that the SSC
photon flux depends linearly on $\epsilon_{\gamma}$. The intrinsic spectra are
flat and independent of $E_{\gamma}$ for both nights. We have also shown that
the bulk Lorentz factor of the GRB should be $\Gamma\sim 11$ to explain the
VHE spectra. Finally, it is important to mention that the photohadronic model
in its simplest form as used previously, could explain very well the sub-TeV
afterglow spectra of GRB 190114C and GRB 180720B. This time again, using the
same model, we successfully explain the multi-TeV afterglow spectra of GRB
190829A. Thus, our study shows that the three GRBs observed so far in the VHE
$\gamma$-rays in the afterglow phases by IACTs can be explained very well in
the context of the photohadronic model.

As is mentioned in the introduction, the VHE spectrum of GRB 180720B can be
explained by the interactions of the Fermi accelerated protons with the
synchrotron background, and the VHE spectra of GRB 190114C and GRB 190829A, by
the interaction of protons with the SSC background. For the synchrotron
background we should have $\delta<2.0$ as the synchrotron photon flux is
proportional to $\epsilon^{-\beta}$. On the other hand, for the SSC
background, the photon flux is proportional to $\epsilon^{\beta}$, with
$0<\beta\leq 1.0$ which corresponds to $2.5\leq\delta\leq 3.0$. In future,
through existing and forthcoming Cherenkov Telescopes, it might be possible to
observe the VHE gamma-ray spectra from nearby GRBs at redshift $\lesssim 0.5$
in their afterglow phases and some VHE gamma-ray spectra could only be
explained by employing the two different zones, one with $\delta<2.0$ and the
another with $2.5\leq\delta\leq 3.0$. Such a VHE spectrum can be interpreted
as a result of the interactions of the high energy protons with both the
synchrotron background and the SSC background in the forward shock region.

We are thankful to Alberto Rosales de León, Gabriel Sánchez Colón and Benjamín
Medina Carrillo for many useful discussions. The work of S.S. is partially
supported by DGAPA-UNAM (Mexico) Projects No. IN103019 and No. IN103522.
Partial support from CSU-Long Beach is gratefully acknowledged.

## References

## References

  * Abdalla et al. (2019) Abdalla, H., A. R. A. F., et al. 2019, Nature, 575, 464
  * Abdalla et al. (2021) Abdalla, H., et al. 2021, Science, 372, 1081, doi: 10.1126/science.abe8560
  * Acciari et al. (2019a) Acciari, V., et al. 2019a, Nature, 575, 455
  * Acciari et al. (2019b) —. 2019b, Nature, 575, 459
  * Ackermann et al. (2012) Ackermann, M., Ajello, M., Allafort, A., et al. 2012, Science, 338, 1190, doi: 10.1126/science.1227160
  * Ajello et al. (2018) Ajello, M., Baldini, L., Barbiellini, G., et al. 2018, ApJ, 863, 138, doi: 10.3847/1538-4357/aad000
  * Asano & Meszaros (2012) Asano, K., & Meszaros, P. 2012, Astrophys. J., 757, 115
  * Chand et al. (2020) Chand, V., et al. 2020, Astrophys. J., 898, 42, doi: 10.3847/1538-4357/ab9606
  * de Naurois (2019) de Naurois, M. 2019, The Astronomer’s Telegram, 13052, 1
  * Derishev & Piran (2019) Derishev, E., & Piran, T. 2019, Astrophys. J., 880, L27
  * Dermer & Schlickeiser (1993) Dermer, C. D., & Schlickeiser, R. 1993, Astrophys. J., 416, 458
  * Dominguez et al. (2011) Dominguez, A., et al. 2011, Mon. Not. Roy. Astron. Soc., 410, 2556
  * Fraija et al. (2019) Fraija, N., Duran, R. B., Dichiara, S., & Beniamini, P. 2019, Astrophys. J., 883, 162
  * Franceschini et al. (2008) Franceschini, A., Rodighiero, G., & Vaccari, M. 2008, Astron. Astrophys., 487, 837
  * Ghisellini et al. (2010) Ghisellini, G., Ghirlanda, G., Nava, L., & Celotti, A. 2010, MNRAS, 403, 926, doi: 10.1111/j.1365-2966.2009.16171.x
  * Hu et al. (2021) Hu, Y. D., et al. 2021, Astron. Astrophys., 646, A50, doi: 10.1051/0004-6361/202039349
  * Kumar & Barniol Duran (2009) Kumar, P., & Barniol Duran, R. 2009, MNRAS, 400, L75, doi: 10.1111/j.1745-3933.2009.00766.x
  * Kumar & Zhang (2015) Kumar, P., & Zhang, B. 2015, Phys. Rep., 561, 1
  * Lesage et al. (2019) Lesage, S., Poolakkil, S.and Fletcher, C., & Meegan, C. 2019, GRB Coordinates Network, 25575, 1
  * Mészáros (2006) Mészáros, P. 2006, Reports on Progress in Physics, 69, 2259, doi: 10.1088/0034-4885/69/8/R01
  * Nemmen et al. (2012) Nemmen, R. S., Georganopoulos, M., Guiriec, S., et al. 2012, Science, 338, 1445
  * Piran (2004) Piran, T. 2004, Rev. Mod. Phys., 76, 1143
  * Razzaque et al. (2010) Razzaque, S., Dermer, C. D., & Finke, J. D. 2010, Open Astron. J., 3, 150
  * Sahu (2019) Sahu, S. 2019, Rev. Mex. Fis., 65, 307
  * Sahu et al. (2020) Sahu, S., Fortin, C. E. L., Martinez, M. E. I., Nagataki, S., & de Cordoba, P. F. 2020, Mon. Not. Roy. Astron. Soc., 492, 2261
  * Sahu & López Fortín (2020) Sahu, S., & López Fortín, C. E. 2020, Astrophys. J. Lett., 895, L41, doi: 10.3847/2041-8213/ab93da
  * Sahu et al. (2019) Sahu, S., López Fortín, C. E., & Nagataki, S. 2019, Astrophys. J., 884, L17
  * Sari & Esin (2001) Sari, R., & Esin, A. A. 2001, Astrophys. J., 548, 787
  * Stecker et al. (1992) Stecker, F. W., de Jager, O. C., & Salamon, M. H. 1992, ApJ, 390, L49, doi: 10.1086/186369
  * Totani (1998) Totani, T. 1998, ApJ, 502, L13, doi: 10.1086/311489
  * Valeev et al. (2019) Valeev, A. F., Castro-Tirado, A. J., Hu, Y. D., & Garcia, E. F. 2019, GRB Coordinates Network, 25565, 1
  * Vurm & Beloborodov (2017) Vurm, I., & Beloborodov, A. M. 2017, Astrophys. J., 846, 152
  * Wang & Wei (2011) Wang, J., & Wei, J. 2011, Astrophys. J., 726, L4
  * Wang et al. (2010) Wang, X.-Y., He, H.-N., Li, Z., Wu, X.-F., & Dai, Z.-G. 2010, ApJ, 712, 1232, doi: 10.1088/0004-637X/712/2/1232
  * Wang et al. (2019) Wang, X.-Y., Liu, R.-Y., Zhang, H.-M., Xi, S.-Q., & Zhang, B. 2019, Astrophys. J., 884, 117
  * Waxman (1997) Waxman, E. 1997, ApJ, 485, L5, doi: 10.1086/310809
  * Wu et al. (2016) Wu, Q., Zhang, B., Lei, W.-H., et al. 2016, Mon. Not. Roy. Astron. Soc., 455, L1
  * Wu et al. (2011) Wu, Q., Zou, Y.-C., Cao, X., Wang, D.-X., & Chen, L. 2011, Astrophys. J., 740, L21
  * Zhang & Mészáros (2001) Zhang, B., & Mészáros, P. 2001, ApJ, 559, 110, doi: 10.1086/322400
  * Zhang et al. (2020) Zhang, B. T., Murase, K., Veres, P., & Mészáros, P. 2020, arXiv e-prints, arXiv:2012.07796.  https://arxiv.org/abs/2012.07796
  * Zyla et al. (2020) Zyla, P., et al. 2020, PTEP, 2020, 083C01, doi: 10.1093/ptep/ptaa104


11institutetext: Politecnico di Milano
Dipartimento di Elettronica e Informazione, Italy
11email<EMAIL_ADDRESS>
# A Fault Resilient Approach to Non-collective Communication Creation in MPI

Roberto Rocco  Gianluca Palermo

###### Abstract

The increasing size of HPC architectures makes the faults’ presence an
eventuality more and more frequent. This is especially relevant since MPI, the
de-facto standard for inter-process communication lacks proper fault
management functionalities. The past efforts produced extensions to the MPI
standard that enabled fault management, the most important one being ULFM. In
this paper, we introduce the support for non-collective communication creation
(MPI_Comm_create_group) in ULFM to improve the fault management capabilities.
We integrate our solution into the Legio library and measure the overhead
introduced in the application. The proposed solution removes the possibility
of turning the execution into a deadlock after a fault and can be used as an
inspiring effort to improve the ULFM repair capabilities.

###### Keywords:

Fault Management MPI ULFM

##  1 Introduction

Computational science applications require more and more resources for their
computation, leading to the growth of current HPC systems in terms of
performance, energy efficiency and complexity. HPC systems will reach the
exascale boundary in the next years (1018 FLOPS) [1], bringing more
performance but additional issues. With the increase in sizes comes a greater
probability of faults, and HPC systems (and their applications) must be able
to handle them.

The absence of fault management techniques can make the execution difficult on
HPC clusters since the mean time to failure is in the order of days [18].
Future systems will probably lower this value, making it almost impossible for
an application to complete without incurring an error. If there are no fault
management techniques, the execution terminates without producing results,
causing waste of energy and costs. Reports [6] show that the impact of faults
is already relevant in HPC. This concept also comes from the fact that the
Message Passing Interface (MPI) [4], the de-facto standard for inter-process
communication, lacks fault management techniques. The last version of the
standard (4.0) tried to reduce the possible sources of faults, isolating them
into a single process when possible. While removing the impact of faults in
local functions, it does not avoid their propagation with communication and
does not provide a way to repair the execution.

This issue has been hugely discussed in the past, producing MPI
implementations able to deal with the fault presence [3, 8, 7]. Most of them
received limited support and did not solve the problem entirely and
efficiently. The issue currently receiving the most attention is the User-
Level Fault Mitigation (ULFM) MPI extension [2]. It consists of a collection
of functions that enable the user to detect fault presence, repair the
affected structures and continue the execution seamlessly. Among the added
functionalities of ULFM, there are the shrink and agree functions, that enable
communicator reparation and resilient agreement respectively. ULFM is
currently integrated directly into the latest versions of OpenMPI, one of the
most used MPI implementations.

Many efforts started from the functionalities introduced by ULFM to produce
frameworks able to provide checkpoint and restart (C/R) with automatic
rollback [9, 13, 17]. Others leveraged those capabilities to isolate the
failed process from the execution, limiting the effect of the fault [14, 15].
Nonetheless, the possibilities introduced by ULFM are limited by a few
constraints: the reparation process is always collective and currently
involves only communicators. We tried to remove these constraints in our
previous work [15], developing support for files, windows and local repair.
Still, many other functionalities require additional attention.

In this effort, we analyze the non-collective call MPI_Comm_create_group,
which represents a challenge due to its non-collectiveness. Previous works
have already shown the relevance of this call [5], and it has been in the MPI
standard since version 3.0. Most of the ULFM functionalities are hard to use
due to the non-collectiveness of the operation which prevents the repair.
Moreover, the failure of one of the participating processes may lead to
deadlock, which is always an undesirable behaviour.

The contributions of this paper are the following:

  * •

We analyze the effects of faults on the MPI_Comm_create_group function;

  * •

We design and implement a solution that removes the main criticalities of the
function;

  * •

We integrate the solution into the Legio framework and evaluate the introduced
overhead.

The paper is structured as follows: Section 2 shows the efforts that tried to
deal with faults using ULFM and emphasizes the works done to improve it.
Section 3 shows the behaviour of non-collective operations and our proposed
solution to make them fault tolerant. Section 4 covers the experimental
campaign done to evaluate the overhead and the scalability of the proposed
solution. Lastly, Section 5 concludes the paper.

##  2 Background and previous work

The failure of an MPI process can have a remarkable impact on an application
execution since the MPI standard does not specify the behaviour of the
survivor processes. The last version of the MPI standard (4.0) introduced new
functionalities to simplify the handling of faults, limiting their impact
while possible. While providing ways to represent and react to faults, the
standard still does not contain a defined method to recover the execution from
them and only allows the application to terminate gracefully.

One of the most relevant efforts that enable the continuation of the execution
is ULFM [2]. It is an MPI standard extension proposal that focuses on fault
detection, propagation and reparation. While still under development, the ULFM
extension got included in the latest versions of OpenMPI, one of the most used
MPI implementations. The idea behind ULFM is to provide the user with the
possibility to manage faults by themselves. The latter is achieved with
changes in the application code that introduce the ULFM functions. This
approach allows maximum flexibility in fault management at the cost of
additional integration complexity: the programmer must know how and when to
handle faults, which is non-trivial.

ULFM by itself does not provide recovery mechanisms, so it is necessary to
combine it with other solutions for consistent post-failure execution. The
main direction adopted in the literature is in combining ULFM and a C/R
utility into a single framework, that introduces fault tolerance (the ability
to nullify the effect of a fault) in generic MPI applications [9, 16, 17, 10,
13]. The solution proposed in these efforts is similar: the execution restarts
from the last consistent state, removing the impact of the fault. The
integration with the application is more intuitive than using ULFM directly
since it hides the complexity within the combined framework. Some efforts [13,
12] do not require changes in the application since they leverage a heuristic
code analysis to choose the best integration between their framework and the
application.

Our previous work proposed Legio [15], a library that introduces fault
resiliency (the ability to overcome a fault) in embarrassingly parallel
applications. Applications using Legio continue after the fault detection, but
the failed process will not resume: the execution proceeds only with the
survivor processes, causing a loss of correctness but a faster recovery. These
characteristics make Legio ideal for approximate computing applications, where
the algorithms already trade correctness for speed. Moreover, Legio extends
the capabilities of ULFM by supporting one-sided communication and filesystem
functions. Applications leveraging those functionalities can fully benefit
from the additional fault management. The integration of Legio with the
application leverages the PMPI layer defined in the MPI standard: this allows
for seamless integration with no code changes needed.

Apart from Legio, few efforts aim at enhancing the fault management
possibilities of ULFM. Most of them focus on improving the performance of ULFM
[11] rather than introducing new functionalities. The focus on fault
management for non-collective communicator creation is something missing in
the literature, despite the promising result of their use [5]. In this work,
we introduce the support for non-collective communicator creation into the
Legio library.

##  3 Non-collective operations

A non-collective operation involves many processes inside a communicator, not
necessarily all. In the MPI standard, there are a few examples of non-
collective calls, one of those is the MPI_Comm_create_group call. First
proposed in [5], it got introduced with version 3.0 of the standard. The
function creates a communicator containing only the processes present in the
group structure passed as a parameter. This function must be called only by
the processes part of the group, not by all the ones participating in the
communicator (differently from the MPI_Comm_create function). Its non-
collectiveness makes the ULFM solution not sufficient: the agreement function
and the shrink one presented in the ULFM standard are collective and may
require support from processes not involved in the communicator creation.
Forcing those processes to collaborate loses the benefits of non-collective
communicator creation (less synchronization), so it is not feasible. In
general, to implement the non-collective call we cannot use any collective
operation, including most of the ULFM ones, windows and file operations.

We conducted some experiments to understand the behaviour of the non-
collective call in presence of faults. To better represent the occurrence of a
fault in a communicator, we describe it as either faulty or failed. In
particular, faulty communicators contain some failed processes but their
failure has not been acknowledged by any process. When a process discovers the
failure, the communicator becomes failed and the propagation of the failure
begins. Our tests proved that:

  * •

The non-collective call works on faulty communicators as long as no process
part of the group failed;

  * •

The non-collective call deadlocks on faulty communicators if one process part
of the group failed;

  * •

The function fails on failed communicators, returning the ULFM-defined error
code MPIX_ERR_PROC_FAILED.

The main criticality of the non-collective call behaviour resides in the
second case: while it is possible to handle the error and repair the
communicators in later calls, the deadlock prevents us from any interaction
and irremediably compromises the execution of the entire application. This
effort aims to remove the deadlock eventuality from the application behaviour
by changing the parameters passed to the function.

The function MPI_Comm_create_group takes three parameters, which are the
communicator to generate the new one from, the group of processes that
participate in the call, and a tag used to classify the calls. We may have to
change the communicator parameter: previous faults may have already caused the
communicator to shrink so we exchange it with the shrunk one. This step
introduces the possibility that a process part of the group is not part of the
communicator since a previous shrink operation removed it. To solve this issue
we need to remove those processes from the group. Figure 1 contains an example
of this first filtering process. This first exchange of parameters allows us
to limit the impact of previously fixed faults, but it does not help in the
case of unfixed ones: in the example shown in Figure 1, process 5 is still
involved in the non-collective call despite being failed, eventually leading
to the deadlock of the execution.

Figure 1: The reduction of the communicator and group parameters achieved in
the first filtering process. All the processes are represented as small
circles with their rank written within. The ones filled in red failed. Process
5 failed but its failure has not been discovered yet. The black rectangle
represents the communicator, the blue one the one used after the filtering
phase. The red rectangle represents the group, the green one the one used
after the first filtering phase.

To identify which processes failed within the filtered group, we need all the
processes to discover whose of them are still alive. If the call was a
collective operation, we could have leveraged the MPI_Allgather function: each
process could share its rank with the others and obtain data about all the
processes sharing. However, the non-collectiveness of the operation forces us
to define an algorithm using point-to-point communication only, so we defined
a tree-based algorithm. We will first describe the algorithm in a fault-free
scenario and then discuss the modifications needed for its fault management.

In a fault-free scenario, the algorithm behaves similarly to the tree
reduction algorithm followed by a tree broadcast. The two phases are not
distinct in our implementation: each process starts the broadcast after it has
completed the reduction, but it does not imply that all the others are in the
same situation. Figure 2 shows how the algorithm works considering the time
dimension. The algorithm works in a fault-free scenario but it is not
sufficient in a faulty environment: a failed process may block the propagation
towards the root in the gathering phase and the diffusion of the collected
data in the broadcast phase. Figure 3 emphasises this phenomenon, showing how
the result obtained is different among the nodes, creating a partition of the
network.

Figure 2: A representation of the algorithm used to remove the presence of
failed processes inside the group. Each column represents a process, while
green rectangles contain the data held by it. Black boxes represent MPI
operations, with the first character showing the type (Send/Recv) and the
second the rank of the other process. Figure 3: The figure shows the algorithm
depicted in Figure 2 in presence of faults. Failed processes have their column
filled in red, and also MPI calls failing.

To fix the network splitting problem, we integrated into the above algorithm a
mechanism to bypass the failed nodes. In particular, the proposed approach
works as follows: if a node fails, the next one in its subtree of competency
(the subtree having that process as root) will take its place. If that process
fails too, then the next one will take their place, and so on. This solution
ensures that if there is at least a process in a subtree whose root failed, it
will reach and be reachable by all the others. This approach requires that the
processes can learn the liveness of the others with point-to-point
communication. Following the analyses shown in [15], we know that the MPI_Recv
fails if the sender stops working. We use the receive function for
communication and fault checking, and we integrate the leader escalation
system into our algorithm, obtaining a result shown in Figure 4. The figure
shows that all the nodes get the same results, and no partitions are present.

Figure 4: The figure shows the algorithm adaptation in presence of faults.
Additional or different calls due to the adaptation are filled in purple.

The above solution with the fix of the network splitting allows us to solve
the deadlock problem since all the processes can operate on the survivor
subset of the group parameter. The only drawback is that the possible faults
discovered in the fix are propagated to all the nodes involved, turning the
faulty communicator into a failed one. This occurrence causes the function to
complete with an error despite the removal of the failed processes from the
parameters: nonetheless, we think that the deadlock removal is a crucial step
for the function usage in a faulty scenario. Moreover, the
MPI_Comm_group_create function can be used as a non-collective error checker
since it will return an error in case of fault presence, and it will work fine
otherwise. This is an improvement from the possibilities of ULFM, allowing for
error checking in a group of processes in logarithmic time complexity.

##  4 Experimental campaign

We implemented the proposed solution in the Legio library, and we conducted
some experiments to measure its scalability and impact on performance. We ran
those experiments on the Marconi100 cluster at CINECA over nodes with 2 x IBM
POWER9 AC922 16 cores 3.1 GHz processors and 256 GB of RAM. Every experiment
featured 32 processes per node, a single process per physical core.

We made a set of initial runs in a fault-free scenario to derive a basing
value (the time needed for the operation without our solution). We performed
other tests injecting single process failures, showing how the execution would
deadlock and how our solution manages to continue (despite raising an error).

In the experimental campaign, we measure the execution time of the non-
collective call integrated with the proposed solution. We run this experiment
with varying network sizes (composed of 32, 64, 128, 256 or 512 MPI processes)
and group sizes (from 1/32 to the entire network size). We execute the non-
collective call 1000 times, measuring the total time needed to complete this
task. We perform this measurement both with and without our solution, to
better evaluate the impact it has on the execution time. The campaign aims to
measure the overhead introduced and prove its logarithmic growth with regard
to the dimension of the group.

$2^{1}$$2^{2}$$2^{3}$$2^{4}$$2^{5}$$2^{6}$$2^{7}$$2^{8}$$2^{9}$$10^{-4}$$10^{-3}$$10^{-2}$Group
sizeOverhead [s]# of processes3264128256512 Figure 5: The evolution of the
overhead per MPI_Comm_create_group call of the proposed solution over networks
of different sizes.

Figure 5 shows the evolution of the overhead over different group sizes in
various networks. The image shows the logarithmic growth of the algorithm,
which seems more influenced by the size of the network rather than the group
size. While the overhead is not negligible, its scalability is still
remarkable. Moreover, the non-collectiveness of the operation reduces the
overhead impact: using it removes unneeded synchronization, which can enable
faster execution to compensate for the additional communication.

While the results are promising, we think that it is possible to achieve lower
overheads by supporting non-collective communication creation directly inside
the ULFM implementation, removing the deadlock occurrence and simplifying the
communication. Moreover, it may be possible to make the call work on failed
communicators as long as the processes inside the group are alive.

##  5 Conclusions

In this paper, we discussed the support for non-collective communicator
creation calls in ULFM. We highlighted the possibility of deadlocks in case of
faults in the involved processes, and we designed an algorithm to avoid that
occurrence. We integrated our solution into the Legio library to provide it
transparently to the MPI applications. Our proposed solution proved scalable
and the overhead introduced acceptable, but we think it can improve by
integrating support for the non-collective communication calls directly inside
the ULFM implementation.

The support for the non-collective communicator creation call directly in the
ULFM implementation can be a relevant feature. It would enable both the fault-
proof usage of the call and could also be used for non-collective communicator
substitution, removing the constraint of collectiveness in the repair
procedure. We think that this last achievement could greatly improve the
capabilities of ULFM, introducing non-collective repair of communicators, a
feature already requested in literature [15, 10].

Future work includes the support of additional MPI functionalities still not
handled by ULFM. Moreover, we are exploring the possibility to combine Legio
with C/R by applying it only to critical nodes to reduce the impact on the
filesystem.

## References

  * [1] Amarasinghe, S., Campbell, D., Carlson, W., Chien, A., Dally, W., Elnohazy, E., Hall, M., Harrison, R., Harrod, W., Hill, K., et al.: Exascale software study: Software challenges in extreme scale systems. DARPA IPTO, Air Force Research Labs, Tech. Rep pp. 1–153 (2009)
  * [2] Bland, W., Bouteiller, A., Herault, T., Bosilca, G., Dongarra, J.: Post-failure recovery of mpi communication capability: Design and rationale. The International Journal of High Performance Computing Applications 27(3), 244–254 (2013)
  * [3] Bouteiller, A., Herault, T., Krawezik, G., Lemarinier, P., Cappello, F.: Mpich-v project: A multiprotocol automatic fault-tolerant mpi. The International Journal of High Performance Computing Applications 20(3), 319–333 (2006)
  * [4] Clarke, L., Glendinning, I., Hempel, R.: The mpi message passing interface standard. In: Programming environments for massively parallel distributed systems, pp. 213–218. Springer (1994)
  * [5] Dinan, J., Krishnamoorthy, S., Balaji, P., Hammond, J.R., Krishnan, M., Tipparaju, V., Vishnu, A.: Noncollective communicator creation in mpi. In: European MPI Users’ Group Meeting. pp. 282–291. Springer (2011)
  * [6] Egwutuoha, I.P., Levy, D., Selic, B., Chen, S.: A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems. The Journal of Supercomputing 65(3), 1302–1326 (2013)
  * [7] Fagg, G.E., Dongarra, J.J.: Ft-mpi: Fault tolerant mpi, supporting dynamic applications in a dynamic world. In: European Parallel Virtual Machine/Message Passing Interface Users’ Group Meeting. pp. 346–353. Springer (2000)
  * [8] Ferreira, K., Riesen, R., Oldfield, R., Stearley, J., Laros, J., Pedretti, K., Brightwell, R.: rmpi: increasing fault resiliency in a message-passing environment. Sandia National Laboratories, Albuquerque, NM, Tech. Rep. SAND2011-2488 (2011)
  * [9] Gamell, M., Katz, D.S., Kolla, H., Chen, J., Klasky, S., Parashar, M.: Exploring automatic, online failure recovery for scientific applications at extreme scales. In: SC’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 895–906. IEEE (2014)
  * [10] Gamell, M., Teranishi, K., Heroux, M.A., Mayo, J., Kolla, H., Chen, J., Parashar, M.: Local recovery and failure masking for stencil-based applications at extreme scales. In: SC’15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1–12. IEEE (2015)
  * [11] Laguna, I., Richards, D.F., Gamblin, T., Schulz, M., de Supinski, B.R., Mohror, K., Pritchard, H.: Evaluating and extending user-level fault tolerance in mpi applications. The International Journal of High Performance Computing Applications 30(3), 305–319 (2016). https://doi.org/10.1177/1094342015623623
  * [12] Losada, N., Bosilca, G., Bouteiller, A., González, P., Martín, M.J.: Local rollback for resilient mpi applications with application-level checkpointing and message logging. Future Generation Computer Systems 91, 450–464 (2019)
  * [13] Losada, N., Cores, I., Martín, M.J., González, P.: Resilient mpi applications using an application-level checkpointing framework and ulfm. The Journal of Supercomputing 73(1), 100–113 (2017)
  * [14] Pauli, S., Arbenz, P., Schwab, C.: Intrinsic fault tolerance of multilevel monte carlo methods. Journal of Parallel and Distributed Computing 84, 24–36 (2015)
  * [15] Rocco, R., Gadioli, D., Palermo, G.: Legio: fault resiliency for embarrassingly parallel mpi applications. The Journal of Supercomputing pp. 1–21 (2021)
  * [16] Shahzad, F., Thies, J., Kreutzer, M., Zeiser, T., Hager, G., Wellein, G.: Craft: A library for easier application-level checkpoint/restart and automatic fault tolerance. IEEE Transactions on Parallel and Distributed Systems 30(3), 501–514 (2018)
  * [17] Teranishi, K., Heroux, M.A.: Toward local failure local recovery resilience model using mpi-ulfm. In: Proceedings of the 21st european mpi users’ group meeting. pp. 51–56 (2014)
  * [18] Zheng, G., Ni, X., Kalé, L.V.: A scalable double in-memory checkpoint and restart scheme towards exascale. In: IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN 2012). pp. 1–6. IEEE (2012)