11institutetext:  Universitäts-Sternwarte, Fakultät für Physik, Ludwig-
Maximilians-Universität München, Scheinerstr. 1, 81679 München, Germany
11email<EMAIL_ADDRESS>
# A Stream Come True – Connecting tidal tails, shells, streams,
and planes with galaxy kinematics and formation history

Lucas M. Valenzuela A Stream Come True – Connecting tidal tails, shells,
streams, and planes with galaxy kinematics and formation historyA Stream Come
True – Connecting tidal tails, shells, streams, and planes with galaxy
kinematics and formation history Rhea-Silvia Remus A Stream Come True –
Connecting tidal tails, shells, streams, and planes with galaxy kinematics and
formation historyA Stream Come True – Connecting tidal tails, shells, streams,
and planes with galaxy kinematics and formation history

(Received XXX / Accepted YYY)

###### Abstract

Context. The rapidly improving quality and resolution of both low surface
brightness observations and cosmological simulations of galaxies enables an
increasingly thorough investigation of the imprints of the formation history
in the outer, unrelaxed regions of galaxies, and a direct comparison to
another tracer of galaxy formation, the internal kinematics.

Aims. Using the state-of-the-art hydrodynamical cosmological simulation
_Magneticum Pathfinder_ , we identify tidal tails, shells, streams, and
satellite planes, and connect their existence to the amount of rotational
support and the formation histories of the host galaxies.

Methods. Tidal features are visually classified from a three-dimensional
rendering of the simulated galaxies by several scientists independently. Only
features that were identified by at least half of the participating
individuals are considered as existing features. The results are compared to
observations of the MATLAS survey.

Results. Shells are preferentially found around kinematically slowly rotating
galaxies in both simulations and observations, while streams can be found
around all kind of galaxies with a slightly higher probability to be present
around less rotationally supported galaxies. Tails and satellite planes,
however, appear independently of the internal kinematics of the central
galaxy, indicating that they are formed through processes that have not (yet)
affected the internal kinematics.

Conclusions. As shells are formed through radial merger events while streams
are remnants of circular merger infall, this suggests that the orbital angular
momentum of the merger event could play a more crucial role in transforming
the host galaxy than previously anticipated. The existence of a shell around a
given slow rotator can further be used to distinguish the radial merger
formation scenario from other formation pathways of slow rotators.

###### Key Words.:

galaxies: evolution – galaxies: interactions – galaxies: kinematics and
dynamics – galaxies: statistics – galaxies: structure

##  1 Introduction

The outer stellar regions of galaxies encode clues about their formation
histories because of the larger relaxation times. Due to the low surface
brightness (LSB) of these outskirts, deep imaging is necessary to detect faint
structures created through tidal interactions and mergers with other galaxies,
therefore storing information about the merger history (e.g., Johnston et al.,
2008). These tidal features, such as shells, streams, or tidal tails, have
recently been examined more closely through deep observations (e.g., Bílek et
al., 2020) and high-resolution simulations (e.g., Karademir et al., 2019) with
the aim of better understanding their connection with the merger histories of
galaxies.

To study tidal features in a more statistically significant way, LSB features
have been systematically classified in galaxy surveys, such as MATLAS (Duc et
al., 2015; Bílek et al., 2020; Sola et al., 2022) or the Stellar Stream Legacy
Survey (Martinez-Delgado et al., 2021). The features are oftentimes identified
through visual inspection, though fully automated approaches have been
developed as well. However, such automated methods can have issues with their
sensitivity to LSB features (Sola et al., 2022).

Methods of identifying tidal features have also been applied to simulations.
The possibility of tracing back individual galaxies and their tidal features
through time enables a direct connection between the LSB structures with the
formation histories of galaxies: shells typically form through mergers on
radial orbits and streams through mergers on circular orbits (e.g., Amorisco,
2015; Hendel & Johnston, 2015; Pop et al., 2018; Karademir et al., 2019).
Investigating the survival timescales in addition to the origin of different
tidal features, Mancillas et al. (2019) found that shells and streams survive
the longest and tails the shortest.

Satellite galaxies also reside in the outer galaxy regions and are expected to
merge with the central galaxy in the future, potentially forming tidal
features in the process. Already in 1976, Lynden-Bell found that the
Magellanic Clouds and a number of globular clusters form a thin plane around
the Milky Way, with more satellites identified to lie within it since then
(e.g., Kroupa et al., 2005). Other well-known systems with satellite planes
are the Andromeda Galaxy (e.g., Ibata et al., 2013) and Centaurus A (e.g.,
Müller et al., 2018). More recently, Heesters et al. (2021) found that roughly
$25\text{\,}\mathrm{\char 37\relax}$ of MATLAS galaxies feature a flattened
satellite plane in their outskirts.

Satellite planes have been studied in cosmological simulations as well: some
studies reported very few systems having such planes (e.g. Pawlowski et al.,
2014), whereas others found better agreement with observations (e.g., Wang et
al., 2013; Förster et al., 2022). Simulations further suggest that the
arrangement of satellites is connected to the surrounding cosmic web (e.g.,
Welker et al., 2018), and some studies have concluded that thin planes of
satellite galaxies are short-lived (Fernando et al., 2017). However, it is
still under debate how these satellite planes are connected to the central
galaxy.

A direct imprint of a galaxy’s formation history is also found in its
kinematics. While late-type galaxies (LTGs) usually exhibit regular rotation
patterns, early-type galaxies (ETGs) show a large variety of rotational
properties. A kinematic classification of ETGs was introduced by Emsellem et
al. (2007, 2011) for the SAURON and ATLAS3D surveys, where ETGs are split into
fast and slow rotators, depending on how rotationally supported they are.
These rotation patterns are closely related to the formation of the galaxy,
with slow rotators usually indicating a more violent accretion history (e.g.,
Schulze et al., 2018; Lagos et al., 2022).

As both the internal kinematics and the faint structures found in the
outskirts of galaxies are remnants of their formation history, we investigate
the correlation between these two tracers in simulations and observations in
this work. We first introduce the simulations and observations in Section 2,
followed by the details of our methods and the definition of the relevant
properties in Section 3. Finally, the results are presented and discussed in
Section 4, and are concluded in Section 5.

##  2 Simulations and Observations

###  2.1 The Magneticum Pathfinder Simulations

The galaxies classified in this work are extracted from the _Magneticum
Pathfinder_ 111www.magneticum.org simulation suite (Dolag et al., in prep.),
which consists of hydrodynamical cosmological simulations performed with
Gadget-3, an extended version of Gadget-2 (Springel, 2005). For details of the
implementations, see Teklu et al. (2015). In particular, we use Box4 (uhr)
(side length: $68\text{\,}\mathrm{Mpc}$, with stellar particle mass $\langle
m_{*}\rangle=$1.3\text{\times}{10}^{6}\text{\,}\mathrm{M_{\odot}}\text{\,}{\mathrm{\mathnormal{h}}}^{-1}$$
and softening length $\epsilon_{*}=$1\text{\,}\mathrm{kpc}$$). Galaxies are
identified using a modified version of SubFind adapted for baryonic matter
(Springel et al., 2001; Dolag et al., 2009).

The galaxies from Magneticum Box4 (uhr) have been shown to agree well with
observations, including their angular momenta (Teklu et al., 2015), kinematics
(Schulze et al., 2018; van de Sande et al., 2019; Schulze et al., 2020),
dynamics (Remus et al., 2017; Teklu et al., 2017; Harris et al., 2020), and
their in-situ component fractions (Remus & Forbes, 2022).

In this work, we include all main galaxies with virial masses
$M_{\mathrm{vir}}\geq$7.1\text{\times}{10}^{11}\text{\,}\mathrm{M_{\odot}}$$
and stellar masses
$M_{*}\geq$2.4\text{\times}{10}^{10}\text{\,}\mathrm{M_{\odot}}$$ to ensure
sufficient resolution for identifying stellar tidal features. Additionally, we
limit the sample to galaxies with stellar half-mass radii larger than
$2\text{\,}\mathrm{kpc}$, i.e., two times the stellar softening length
(following Schulze et al., 2018). The resulting sample consists of 520
galaxies. Their ellipticities and $\lambda_{R_{e}}$-values (see Section 3.3)
at one half-mass radius are taken from Schulze et al. (2018) and the satellite
plane properties from Förster et al. (2022).

###  2.2 The MATLAS Sample

The observational comparison sample is taken from the MATLAS survey (Duc et
al., 2015; Habas et al., 2020), which includes 177 massive ETGs with deep
optical images. The MATLAS sample is part of the ATLAS3D sample (Cappellari et
al., 2011), which comprises 260 nearby galaxies
(${<}$42\text{\,}\mathrm{Mpc}$$) as a complete volume-limited sample. The
MATLAS sample contains all ATLAS3D galaxies that neither belong to the Virgo
cluster nor are located near bright stars.

Specifically, we use the results obtained by Bílek et al. (2020) for the
existence of tidal features around the 177 MATLAS galaxies, where tidal
shells, streams, and tails were visually identified, and the identifications
of satellite planes from Heesters et al. (2021). The ellipticities and
$\lambda_{R_{e}}$-values at one effective radius are taken from Emsellem et
al. (2011).

##  3 Methods & Definitions

For the comparison between the simulated and observed galaxies, in the
following we introduce the different features and properties used in this
work. These include the tidal features classified, the determination method of
the satellite planes, and the distinction between fast and slow rotators.

###  3.1 Tidal Features: Shells, Streams, and Tidal Arms

The three types of tidal features considered are shells, streams, and tails.
These indicate past or ongoing gravitational interactions with other galaxies.
Shells appear as circular arcs centered around a galaxy (top panel of Figure
1). They have sharp outer borders and are believed to form through the radial
infall and disruption of massive satellites (e.g., Quinn, 1984; Pop et al.,
2018; Karademir et al., 2019; Bílek et al., 2022). Streams are thin slivers of
stars around the host galaxy (middle panel of Figure 1) that are formed
through the tidal disruption of an infalling galaxy on a circular orbit (e.g.,
Karademir et al., 2019). Tails are stellar arms that come out from the host
galaxy (bottom panel of Figure 1). These are the result of tidal interaction
with another galaxy, where the interaction is either ongoing or occurred
recently.

Figure 1: Mock images of simulated galaxies from Magneticum Box4 (uhr) with
prominent tidal features: shells (top), a stream (middle), and tails (bottom).
The mock images were created with the software from Martin et al., 2022 for
the r-band.

We classified the galaxies in Magneticum with respect to the existence of
these tidal features by visual inspection of the three-dimensional stellar
component, similar to the classification of Bílek et al. (2020) for the MATLAS
sample. Seven scientists participated in the classification, where they were
presented with a three-dimensional rendering of a given galaxy that was
rotated to be viewed from all sides. The local environment around the
individual galaxies was visible as well. The classification was performed
independently by the participants, with each person stating whether they
identified any tidal shells, streams, and tails, or not.

In the following, we consider a galaxy to have a given tidal feature if at
least half of the participants were in favor of it. For the MATLAS
classification by Bílek et al. (2020), this is considered to be the case if
the classification score is at least 1 (the maximum score for them is 2, where
1 means the respective feature is likely to be present).

###  3.2 Satellite Planes

A galaxy is considered to have a satellite plane when at least
$80\text{\,}\mathrm{\char 37\relax}$ of its satellite galaxies build up a
plane of thickness
${\leq}$0.2\text{\,}\mathrm{\mathnormal{R}_{\mathrm{vir}}}$$ and when the in-
plane angular momentum makes up at least $90\text{\,}\mathrm{\char 37\relax}$
of their total angular momentum. These properties are computed through the
_momentum in thinnest plane_ (MTP) method (Förster et al., 2022).

###  3.3 Fast and Slow Rotators

To distinguish fast from slowly rotating ETGs, we follow the approach taken by
Emsellem et al. (2011). They determined an empirical cut in the
$\lambda_{R_{e}}$-$\epsilon_{e}$-plane, where $\lambda_{R_{e}}$ is a
2D-projected proxy for the angular momentum within the effective radius,
$R_{e}$, and $\epsilon_{e}$ is the ellipticity at the effective radius. The
$\lambda_{R_{e}}$-value was introduced by Emsellem et al. (2007) as part of
the SAURON project. Galaxies that fulfill
$\lambda_{R_{e}}>0.31\sqrt{\epsilon_{e}}$ are classified as fast rotators, all
others as slow rotators.

We obtained these parameters for the simulated galaxies from Schulze et al.
(2018) for the edge-on projections of the galaxies, since that maximizes the
$\lambda_{R_{e}}$-values. The random projected values are also comparable to
observations (Schulze et al., 2018). The main difference between calculating
the parameters for simulations and observations is that for the former we use
the stellar mass instead of the luminosity, assuming a constant mass-to-light
ratio (see Schulze et al., 2018, for more details).

##  4 Results

###  4.1 Mass-Size Relation & Feature Statistics

Highlighting the galaxies with a given tidal feature in the mass-size relation
shows that all three types of tidal features occur at all sizes and masses,
with a clear preference for larger and more massive systems (first three
panels of Figure 2). This preference is most likely due to the limited
resolution of the simulation, with tidal features being better resolved for
more massive systems. There is no visible correlation with the deviation from
the average mass-size relation, however.

Figure 2: Mass-size relation of the Magneticum galaxy sample with galaxies
classified as having tidal features or satellite planes in color.

Considering only galaxies with stellar masses
$M_{*}\geq${10}^{11}\text{\,}\mathrm{M_{\odot}}$$, about
$10\text{\,}\mathrm{\char 37\relax}$ to $20\text{\,}\mathrm{\char 37\relax}$
have a given tidal feature (see Table 1). The occurrence of shells in
$11\text{\,}\mathrm{\char 37\relax}$ of the Magneticum galaxies agrees well
with $12\text{\,}\mathrm{\char 37\relax}$ from Atkinson et al. (2013), who
used a different stellar mass range and somewhat different categories (see the
discussion by Bílek et al., 2020), $17\text{\,}\mathrm{\char 37\relax}$ from
Bílek et al. (2020) for MATLAS (for the same stellar mass cut of
${10}^{11}\text{\,}\mathrm{M_{\odot}}$), and $18\text{\,}\mathrm{\char
37\relax}$ from Pop et al. (2018) for the Illustris simulation. The fraction
of galaxies having streams in Magneticum ($29\text{\,}\mathrm{\char
37\relax}$) is very close to the $25\text{\,}\mathrm{\char 37\relax}$ found
for MATLAS. Finally, the fraction of galaxies having tails in Magneticum
($12\text{\,}\mathrm{\char 37\relax}$) is larger than that in MATLAS
($6\text{\,}\mathrm{\char 37\relax}$). Overall, the abundance of stellar tidal
features found in the Magneticum Box4 simulation for the galaxies with larger
stellar masses is consistent with previous findings.

Table 1: Number and fraction of galaxies with stellar masses $M_{*}\geq${10}^{11}\text{\,}\mathrm{M_{\odot}}$$ having the respective tidal feature. The total number of galaxies analyzed in this mass range is 131. Feature | Number | Fraction
---|---|---
Shells | 14 | $11\text{\,}\mathrm{\char 37\relax}$
Streams | 38 | $29\text{\,}\mathrm{\char 37\relax}$
Tails | 16 | $12\text{\,}\mathrm{\char 37\relax}$

We subdivided the galaxies from Magneticum into the morphological types using
the classification from Teklu et al. (2017) based on the $b$-value (Teklu et
al., 2015). The numbers of galaxies in each morphological class containing
tidal features are listed in Table 2. While shells preferentially occur in
ETGs, streams are found equally likely in all three types, and tails are more
likely in intermediate galaxies and LTGs. A similar table for the kinematic
classes is presented in Appendix B.

Table 2: Numbers and fractions of ETGs, intermediates, or LTGs having shells, streams, tails, or satellite planes for the full classified sample of 520 galaxies. Type | All | Shells | Streams | Tails | Planes
---|---|---|---|---|---
ETGs | 295 | 17 / $6\text{\,}\mathrm{\char 37\relax}$ | 37 / $13\text{\,}\mathrm{\char 37\relax}$ | 9 / $3\text{\,}\mathrm{\char 37\relax}$ | 49 / $17\text{\,}\mathrm{\char 37\relax}$
Interm. | 172 | 4 / $2\text{\,}\mathrm{\char 37\relax}$ | 20 / $12\text{\,}\mathrm{\char 37\relax}$ | 12 / $7\text{\,}\mathrm{\char 37\relax}$ | 37 / $22\text{\,}\mathrm{\char 37\relax}$
LTGs | 53 | 0 / $0\text{\,}\mathrm{\char 37\relax}$ | 5 / $9\text{\,}\mathrm{\char 37\relax}$ | 5 / $9\text{\,}\mathrm{\char 37\relax}$ | 14 / $26\text{\,}\mathrm{\char 37\relax}$

For the satellite planes, we find that only galaxies with stellar masses
${\lesssim}${10}^{11}\text{\,}\mathrm{M_{\odot}}$$ have thin planes of
satellites around them. This is consistent with the results from Förster et
al. (2022), who found that the more massive systems have thicker planes.

###  4.2 Tidal Features and Kinematics

We investigated the correlation between two known tracers of the accretion
history of a galaxy, the appearance of tidal features and the internal galaxy
kinematics. To measure the latter, we used the position of galaxies in the
$\lambda_{R_{e}}$-$\epsilon_{e}$-plane for both the observed and simulated
galaxies (top and bottom of Figure 3, respectively). At first glance there are
no clear differences between the distributions of galaxies with tidal features
and the total sample for the observations and simulations. Only for the
simulated galaxies the trend of shells being around slow rotators is visible.
The galaxies with satellite planes are uniformly distributed and do not show
any preference for slow or fast rotation, or for particularly round or
elongated shapes (right panel of Figure 3).

Figure 3: Galaxies classified as having tidal features or a satellite plane in
the $\lambda_{R_{e}}$-$\epsilon_{e}$-plane in the MATLAS survey (_top row_)
and in Magneticum Box4 (_bottom row_). Colored data points represent galaxies
with the respective feature, large gray dots were classified as not having the
respective feature, and small gray dots were not classified (only the case for
the observed satellite planes). The values of $\lambda_{R_{e}}$ and
$\epsilon_{e}$ are determined at one effective radius for the observations and
at one half-mass radius for the simulations. For the simulations, they are
determined in the edge-on projection. The solid black lines indicate the
border between fast and slow rotators (Emsellem et al., 2011), with fast
rotators lying above the border. The measurements of shells, streams, and
tails of MATLAS galaxies are from Bílek et al. (2020) and the measurements of
the planes are from Heesters et al. (2021).

The differences between the edge-on and random projection of galaxies in the
$\lambda_{R_{e}}$-$\epsilon_{e}$-plane are discussed by Schulze et al. (2018)
in detail, especially with respect to the ellipticity, $\epsilon_{e}$, clearly
showing the problems when comparing ellipticities between simulations and
observations. In contrast, $\lambda_{R_{e}}$ is more directly comparable.
Thus, we determined the cumulative histograms of $\lambda_{R_{e}}$ for the
total simulated and observed galaxy samples, and for the galaxies with the
respective tidal features or satellite planes (Figure 4). Since the
$\lambda_{R_{e}}$-distributions of the full samples are not identical (left-
most panel), a direct comparison between the $\lambda_{R_{e}}$-distributions
of galaxies having a certain tidal feature requires a correction to the values
in the cumulative histograms. The method for this correction is described in
Appendix A.

By adapting the simulated cumulative histograms to the observed parent
distribution, the colored lines in Figure 4 are directly comparable with each
other. Very similar distributions of $\lambda_{R_{e}}$ are found for galaxies
with a given tidal feature in the simulations and observations, clearly
confirming that shells are mostly found around slow rotators. Streams have a
slight preference for appearing around slow and intermediate rotators, which
is more pronounced in the observations. Tidal tails are found around both slow
and fast rotators with similar probabilities. However, there are differences
for the presence of satellite planes: the simulations predict planes more
commonly around fast rotators, while observations find a larger probability of
planes around slow rotators, with Heesters et al. (2021) finding
$40\text{\,}\mathrm{\char 37\relax}$ around slow and $16\text{\,}\mathrm{\char
37\relax}$ around fast rotators. This discrepancy may be related to the
different methods of determining satellite planes. While the full 3D
information from the simulation is incorporated to detect planes, the
observations can only use information from the projected view. Therefore, the
results on correlations between satellite planes and galaxy kinematics remain
inconclusive.

Figure 4: Cumulative histograms of $\lambda_{R_{e}}$ for the galaxies in the
MATLAS survey (dashed lines) and in Magneticum Box4 (uhr) (solid lines). The
first plot shows how the distributions between observations and simulations
differ from each other. The cumulative histograms of the simulated galaxies in
the other four panels are adapted to match the parent distribution of MATLAS
to be directly comparable to the distributions of the MATLAS survey. Colored
lines lying above the gray dashed line (the overall distribution of the MATLAS
survey) indicate an increased presence of the respective tidal feature or
satellite plane for slow rotators compared to fast rotators. The measurements
of shells, streams, and tails of MATLAS galaxies are from Bílek et al. (2020)
and the measurements of the planes are from Heesters et al. (2021).

The result that shells are more common around slow rotators suggests that
there is a relation between the processes leading to low rotational support
and the appearance of shells. Shells have been connected to massive mergers
with radial infall (e.g., Amorisco, 2015; Pop et al., 2018; Karademir et al.,
2019), and slow rotators are found to often have formed through massive rapid
merger events (e.g., Schulze et al., 2018; Lagos et al., 2022). Therefore, the
existence of a shell is highly indicative of the observed slow rotator to have
formed in a recent massive radial merger. This provides a way to distinguish
this formation process from others like multiple small mergers (Naab et al.,
2014) or internal feedback quenching (Lagos et al., 2022).

For confirmation, we traced back the most massive galaxy with shells in time,
shown in Figure 5, with the orbit of the infalling galaxy that triggered the
formation of the shells shown in red. For such massive galaxies, the best time
and merger resolution can be expected given the storage limitations of
cosmological simulations. In fact, the shells found around this galaxy were
formed through a satellite with a mass ratio of 1:2.5 falling in radially at
$z=0.20$ (almost $2.5\text{\,}\mathrm{Gyr}$ ago), then falling back in again
on the other side at $z=0.13$ ($1.7\text{\,}\mathrm{Gyr}$ ago). The back-and-
forth movement led to the double shell structure found on two sides of the
galaxy. This strongly supports the idea that shells form through mergers on
radial orbits, albeit shown only for one example galaxy.

Figure 5: Evolution of the first example galaxy from Figure 1. The shells
develop after a massive radial merger event. The red lines trace the orbit of
the infalling galaxy until the respective point in time.

##  5 Conclusion

We classified galaxies from the Magneticum Box4 (uhr) simulation according to
the existence of tidal features in their outskirts and compared these to the
results from Bílek et al. (2020) for the MATLAS survey. The fraction of
Magneticum galaxies having shells, streams, and tails are comparable to those
found in observations and other simulations. We found a remarkable agreement
with observations for the distribution of the kinematic parameter
$\lambda_{R_{e}}$ for galaxies that have a given type of tidal feature. For
both simulations and observations, slow rotators are the most likely to have
shells, while streams preferentially occur around galaxies with low to medium
rotational support. Tracing back a massive galaxy with multiple shells showed
that these formed through an infalling satellite on a radial orbit. This
supports the idea that shells are formed through massive radial merger events
(e.g., Amorisco, 2015; Karademir et al., 2019) that also transform the central
galaxies into slow rotators, while more circular orbits of mergers do not
necessarily lead to a full suppression of the rotational support. Moreover,
the existence of shells around a slow rotator is highly indicative of a
massive rapid radial merger event, providing the means to distinguish this
formation scenario from others. Tails and satellite planes, on the other hand,
occur independently of the morphology and internal kinematics of the central
galaxy, which shows that these features are formed through processes that have
not (yet) affected the internal kinematics.

###### Acknowledgements.

We thank Pierre-Alain Duc, Elisabeth Sola, Klaus Dolag, and Lucas Kimmig for
helpful discussions, and Nick Heesters for providing us with their data on the
presence of satellite planes around MATLAS galaxies. We also thank Silvio
Fortuné, Bendix Hagedorn, Elena Hernández Martínez, Tadziu Hoffmann, Lucas
Kimmig for participating in the classification process of the galaxies in the
simulation. LMV acknowledges support by the COMPLEX project from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation program grant agreement ERC-2019-AdG 882679. The calculations for
the hydrodynamical simulations were carried out at the Leibniz Supercomputer
Center (LRZ) under the project pr83li (Magneticum). This research was
supported by the Excellence Cluster ORIGINS, funded by the Deutsche
Forschungsgemeinschaft under Germany’s Excellence Strategy –
EXC-2094-390783311. The following software was used for this work: Julia
(Bezanson et al., 2017), Matplotlib (Hunter, 2007).

## References

  * Amorisco (2015) Amorisco, N. C. 2015, MNRAS, 450, 575
  * Atkinson et al. (2013) Atkinson, A. M., Abraham, R. G., & Ferguson, A. M. N. 2013, ApJ, 765, 28
  * Bezanson et al. (2017) Bezanson, J., Edelman, A., Karpinski, S., & Shah, V. B. 2017, SIAM Review, 59, 65
  * Bílek et al. (2020) Bílek, M., Duc, P.-A., Cuillandre, J.-C., et al. 2020, MNRAS, 498, 2138
  * Bílek et al. (2022) Bílek, M., Fensch, J., Ebrová, I., et al. 2022, A&A, 660, A28
  * Cappellari et al. (2011) Cappellari, M., Emsellem, E., Krajnović, D., et al. 2011, MNRAS, 413, 813
  * Dolag et al. (2009) Dolag, K., Borgani, S., Murante, G., & Springel, V. 2009, MNRAS, 399, 497
  * Duc et al. (2015) Duc, P.-A., Cuillandre, J.-C., Karabal, E., et al. 2015, MNRAS, 446, 120
  * Emsellem et al. (2011) Emsellem, E., Cappellari, M., Krajnović, D., et al. 2011, MNRAS, 414, 888
  * Emsellem et al. (2007) Emsellem, E., Cappellari, M., Krajnović, D., et al. 2007, MNRAS, 379, 401
  * Fernando et al. (2017) Fernando, N., Arias, V., Guglielmo, M., et al. 2017, MNRAS, 465, 641
  * Förster et al. (2022) Förster, P. U., Remus, R.-S., Dolag, K., et al. 2022, arXiv e-prints, arXiv2208.05496
  * Habas et al. (2020) Habas, R., Marleau, F. R., Duc, P.-A., et al. 2020, MNRAS, 491, 1901
  * Harris et al. (2020) Harris, W. E., Remus, R.-S., Harris, G. L. H., & Babyk, I. V. 2020, ApJ, 905, 28
  * Heesters et al. (2021) Heesters, N., Habas, R., Marleau, F. R., et al. 2021, A&A, 654, A161
  * Hendel & Johnston (2015) Hendel, D. & Johnston, K. V. 2015, MNRAS, 454, 2472
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90
  * Ibata et al. (2013) Ibata, R. A., Lewis, G. F., Conn, A. R., et al. 2013, Nature, 493, 62
  * Johnston et al. (2008) Johnston, K. V., Bullock, J. S., Sharma, S., et al. 2008, ApJ, 689, 936
  * Karademir et al. (2019) Karademir, G. S., Remus, R.-S., Burkert, A., et al. 2019, MNRAS, 487, 318
  * Kroupa et al. (2005) Kroupa, P., Theis, C., & Boily, C. M. 2005, A&A, 431, 517
  * Lagos et al. (2022) Lagos, C. d. P., Emsellem, E., van de Sande, J., et al. 2022, MNRAS, 509, 4372
  * Lynden-Bell (1976) Lynden-Bell, D. 1976, MNRAS, 174, 695
  * Mancillas et al. (2019) Mancillas, B., Duc, P.-A., Combes, F., et al. 2019, A&A, 632, A122
  * Martin et al. (2022) Martin, G., Bazkiaei, A. E., Spavone, M., et al. 2022, MNRAS, 513, 1459
  * Martinez-Delgado et al. (2021) Martinez-Delgado, D., Cooper, A. P., Roman, J., et al. 2021, arXiv e-prints, arXiv2104.06071
  * Müller et al. (2018) Müller, O., Pawlowski, M. S., Jerjen, H., & Lelli, F. 2018, Science, 359, 534
  * Naab et al. (2014) Naab, T., Oser, L., Emsellem, E., et al. 2014, MNRAS, 444, 3357
  * Pawlowski et al. (2014) Pawlowski, M. S., Famaey, B., Jerjen, H., et al. 2014, MNRAS, 442, 2362
  * Pop et al. (2018) Pop, A.-R., Pillepich, A., Amorisco, N. C., & Hernquist, L. 2018, MNRAS, 480, 1715
  * Quinn (1984) Quinn, P. J. 1984, ApJ, 279, 596
  * Remus et al. (2017) Remus, R.-S., Dolag, K., Naab, T., et al. 2017, MNRAS, 464, 3742
  * Remus & Forbes (2022) Remus, R.-S. & Forbes, D. A. 2022, ApJ, 935, 37
  * Schulze et al. (2020) Schulze, F., Remus, R.-S., Dolag, K., et al. 2020, MNRAS, 493, 3778
  * Schulze et al. (2018) Schulze, F., Remus, R.-S., Dolag, K., et al. 2018, MNRAS, 480, 4636
  * Sola et al. (2022) Sola, E., Duc, P.-A., Richards, F., et al. 2022, A&A, 662, A124
  * Springel (2005) Springel, V. 2005, MNRAS, 364, 1105
  * Springel et al. (2001) Springel, V., White, S. D. M., Tormen, G., & Kauffmann, G. 2001, MNRAS, 328, 726
  * Teklu et al. (2015) Teklu, A. F., Remus, R.-S., Dolag, K., et al. 2015, ApJ, 812, 29
  * Teklu et al. (2017) Teklu, A. F., Remus, R.-S., Dolag, K., & Burkert, A. 2017, MNRAS, 472, 4769
  * van de Sande et al. (2019) van de Sande, J., Lagos, C. D. P., Welker, C., et al. 2019, MNRAS, 484, 869
  * Wang et al. (2013) Wang, J., Frenk, C. S., & Cooper, A. P. 2013, MNRAS, 429, 1502
  * Welker et al. (2018) Welker, C., Dubois, Y., Pichon, C., Devriendt, J., & Chisari, N. E. 2018, A&A, 613, A4

##  Appendix A Rescaling the Cumulative Distribution

The cumulative histogram of the simulated galaxies can be adapted to be
directly comparable with that of the observations in the following way: For a
given value of $\tilde{\lambda}_{R_{e}}$, the galaxies with values of
$\lambda_{R_{e}}\leq\tilde{\lambda}_{R_{e}}$ can be determined for all the
observed galaxies ($N_{\mathrm{obs}}({\leq}\tilde{\lambda}_{R_{e}})$), all the
simulated galaxies ($N_{\mathrm{sim}}({\leq}\tilde{\lambda}_{R_{e}})$), and
for the simulated galaxies with a given feature
($N_{\mathrm{sim,feat}}({\leq}\tilde{\lambda}_{R_{e}})$). From these values,
the cumulative number of simulated galaxies with a given feature adapted to
the observational distribution of $\lambda_{R_{e}}$ is

$N_{\mathrm{sim,feat,adap}}({\leq}\tilde{\lambda}_{R_{e}})=N_{\mathrm{obs}}({\leq}\tilde{\lambda}_{R_{e}})\cdot\frac{N_{\mathrm{sim,feat}}({\leq}\tilde{\lambda}_{R_{e}})}{N_{\mathrm{sim}}({\leq}\tilde{\lambda}_{R_{e}})}.$
(1)

Similarly, the adapted number of simulated galaxies with a given feature with
values of $\lambda_{R_{e}}>\tilde{\lambda}_{R_{e}}$ can be determined:

$N_{\mathrm{sim,feat,adap}}({>}\tilde{\lambda}_{R_{e}})=N_{\mathrm{obs}}({>}\tilde{\lambda}_{R_{e}})\cdot\frac{N_{\mathrm{sim,feat}}({>}\tilde{\lambda}_{R_{e}})}{N_{\mathrm{sim}}({>}\tilde{\lambda}_{R_{e}})}.$
(2)

From these, the adapted normalized cumulative distribution (solid colored
lines in Figure 4) is given by

$\frac{N_{\mathrm{cum}}}{N_{\mathrm{tot}}}=\frac{N_{\mathrm{sim,feat,adap}}({\leq}\tilde{\lambda}_{R_{e}})}{N_{\mathrm{sim,feat,adap}}({\leq}\tilde{\lambda}_{R_{e}})+N_{\mathrm{sim,feat,adap}}({>}\tilde{\lambda}_{R_{e}})}.$
(3)

##  Appendix B Kinematic Classes

Table 3 shows the numbers and fractions of galaxies with different kinematic
classes (classified for Magneticum Box4 by Schulze et al., 2018) that have the
respective tidal features or a satellite plane. For more information on the
kinematic classes, see Schulze et al. (2018). In the table, we combined the
distinct core (DC) and kinematically distinct core (KDC) classes into a single
class of KDCs. The results for the KDCs and prolate rotators should be taken
with caution because of the low number of galaxies with these kinematic
features. Still, the clear trend emerges that regular rotators are much less
likely to have any kind of tidal features, while especially non-rotators and
prolate rotators have significantly higher fractions of shells. Interestingly,
the KDCs are the least likely to have satellite planes, although this is based
on only one galaxy out of 26. The fact that almost every fourth KDC has
streams in its outskirts is consistent with many KDCs having decreasing
$\lambda_{R}$-profiles, which generally have merger histories dominated by
small mergers (Schulze et al., 2020). These could lead to tidal streams.

Table 3: Numbers and fractions of galaxies with the given kinematic class having shells, streams, tails, or satellite planes for the subset of our full sample that was kinematically classified by Schulze et al. (2018). This subsample comprises 436 galaxies. The kinematic classes include regular rotators (RR), non-rotators (NR), galaxies with kinematically distinct cores (KDCs), and prolate rotators (PR). Type | All | Shells | Streams | Tails | Planes
---|---|---|---|---|---
RR | 286 | 2 / $1\text{\,}\mathrm{\char 37\relax}$ | 26 / $9\text{\,}\mathrm{\char 37\relax}$ | 8 / $3\text{\,}\mathrm{\char 37\relax}$ | 63 / $22\text{\,}\mathrm{\char 37\relax}$
NR | 108 | 10 / $9\text{\,}\mathrm{\char 37\relax}$ | 16 / $15\text{\,}\mathrm{\char 37\relax}$ | 2 / $2\text{\,}\mathrm{\char 37\relax}$ | 19 / $18\text{\,}\mathrm{\char 37\relax}$
KDC | 26 | 1 / $4\text{\,}\mathrm{\char 37\relax}$ | 6 / $23\text{\,}\mathrm{\char 37\relax}$ | 2 / $8\text{\,}\mathrm{\char 37\relax}$ | 1 / $4\text{\,}\mathrm{\char 37\relax}$
PR | 16 | 3 / $19\text{\,}\mathrm{\char 37\relax}$ | 3 / $19\text{\,}\mathrm{\char 37\relax}$ | 2 / $13\text{\,}\mathrm{\char 37\relax}$ | 2 / $13\text{\,}\mathrm{\char 37\relax}$


# Analyzing Multispectral Satellite Imagery of South American Wildfires Using
Deep Learning

Christopher Sun  Monta Vista High School
Cupertino, CA, United States

###### Abstract

Since frequent severe droughts are lengthening the dry season in the Amazon
Rainforest, it is important to detect wildfires promptly and forecast possible
spread for effective suppression response. Current wildfire detection models
are not versatile enough for the low-technology conditions of South American
hot spots. This deep learning study first trains a Fully Convolutional Neural
Network on Landsat 8 images of Ecuador and the Galapagos, using Green and
Short-wave Infrared bands to predict pixel-level binary fire masks. This model
achieves a 0.962 validation F2 score and a 0.932 F2 score on test data from
Guyana and Suriname. Afterward, image segmentation is conducted on the Cirrus
band using K-Means Clustering to simplify continuous pixel values into three
discrete classes representing differing degrees of cirrus cloud contamination.
Three additional Convolutional Neural Networks are trained to conduct a
sensitivity analysis measuring the effect of simplified features on model
accuracy and train time. The Experimental model trained on the segmented
cirrus images provides a statistically significant decrease in train time
compared to the Control model trained on raw cirrus images, without
compromising binary accuracy. This proof of concept reveals that feature
engineering can improve the performance of wildfire detection models by
lowering computational expense.

###### Index Terms:

computer vision, wildfire detection, convolutional neural networks, geospatial
analysis, sensitivity analysis

##  I Introduction

In places like the Amazon Rainforest, where severe droughts are occurring more
frequently and lengthening the dry season, interconnected ecosystems are
threatened by more widespread occurrences of wildfires (Aragão et al. 2018)
[1]. It is not only important to respond to wildfires promptly using well-
informed allocation of rescue resources to minimize waste, but also to
forecast active wildfires before they become inextinguishable. Researchers
have made progress towards this task by applying remote sensing techniques and
machine learning algorithms on to detect wildfires. Matson and Holben (1987)
used Dozier’s method to extract pixel-level information from NOAA data of
regions in Brazil [8]. Schroeder et al. (2016) used the Near-Infrared and
Short-Wave Infrared bands from the Landsat 8 satellite to classify fire-
affected pixels [13]. Khryashchev and Larionov (2020) used RGB images and data
augmentation for wildfire detection [5]. Most recently, Pereira et al. (2021)
introduced three deep convolutional neural network architectures for active
wildfire detection, using these models to predict the overhead pixel-level
labels of a 256 x 256 pixel patch of land [11]. However, these automated
research designs have not conducted feature engineering apart from selecting
spectral bands, which, coupled with complex model architectures with tens of
millions of parameters, results in a high computational cost.

This paper introduces a fully convolutional neural network (FCN) with skip
connections that uses 3-channel inputs to predict corresponding fire masks.
The paper then describes a clustering approach to image segmentation of cirrus
images. The results of three additional convolutional models reveal that fires
can be detected accurately using simplified features, which is useful when
computational resources are scarce.

Figure 1: Experimental Design

##  II Materials and Data

The data used for this research was a subset of a large publicly available
database of Landsat 8 satellite images [10] processed by Pereira et al. (2021)
[11]. Aerial images of Ecuador and the Galapagos were selected to train the
model, and images of Guyana and Suriname were used as test data for a final
model evaluation. The data consisted of 14,274 images containing at least one
pixel with an active fire, and 10,685 images containing no fires. The former
set of images will be referred to as the “fire” images and the latter as the
“non-fire” images. Each image had dimensions 128$\times$128$\times$10 and was
paired with its corresponding binary mask that indicated the presence of
absence of fire in the region denoted by each pixel. These
128$\times$128$\times$1 masks were either manually annotated or generated
according to the conditions described in Schroeder et al. (2016) [13].

Figure 2: The data set included images with 10 spectral bands. Each band is
shown, with RGB visualized as one band.

The models in this paper were trained using the NVIDIA Tesla P100 GPU.

##  III Methods

###  III-A Fully Convolutional Neural Network

####  III-A1 Neural Network Design

The ultimate modeling task was to extract features from each image to predict
the corresponding mask. Accomplishing this would not require all ten bands of
data, as demonstrated by Pereira et al. (2021), who used only three bands and
generated good results [11]. Consequently, only the Green and Short-wave
Infrared bands were selected as inputs. The deep learning model assembled for
this task was based on the U-Net111The Mask R-CNN was not used because most
masks contained very few fire pixels, so the primary task was not instance
segmentation. [12]. Instead of the four downsampling and upsampling
convolutional blocks in the U-Net, a shallower model consisting of only three
downsampling and upsampling convolutional blocks was built, with a smaller
number of filters per block as well. As a result, the model contained only
2,138,785 trainable parameters, which is considerably less than the
34,525,121-parameter model designed by Pereira et al. (2021) [11]. During the
downsampling phase, 2D convolution layers enlarge the number of channels,
while maximum pooling reduces dimensionality along the first and second axes.
During the upsampling phase, deconvolution layers reduce the number of
channels and increase dimensionality along the first and second axes. The
image is encoded, passed through a bottleneck phase, and decoded by the model.
Skip connections between the convolutional blocks provide feature information
to later layers that might be lost from downsampling (Drozdzal et al. 2016)
[4]. A final 1$\times$1 convolution layer generates the
128$\times$128$\times$1 output mask.

Figure 3: The architecture of the Fully Convolutional Model used for mask
prediction.

####  III-A2 Loss and Metric Functions

Due to the class imbalance in the image masks, the model required a weighted
loss function. The standard binary cross-entropy loss function used for binary
classification was adjusted by a factor of $\mathbf{p_{c}}$, called the class
weight. This class weight was computed using:

$\mathbf{p_{c}}=\frac{\textnormal{Total\;number\;of\;pixels\;that\;equal\;0}}{\textnormal{Total\;number\;of\;pixels\;that\;equal\;1}}$

Accordingly, the loss function was defined as:

$L(\theta)=\frac{1}{m}\sum^{m}_{i=1}\sum^{k}_{n=1}\left[\mathbf{p_{c}}\cdot
y^{n}\log{(\hat{y}^{n})}+(1-y^{n})\log{(1-\hat{y}^{n})}\right]^{i},$

where $k$ is the number of pixels in each flattened mask, $y^{n}$ is the
ground truth classification of the $n^{th}$ pixel, $\hat{y}^{n}$ is the
prediction value of the $n^{th}$ pixel, and a summation is applied over every
pixel of the $i^{th}$ image for a total of $m$ images in each mini-batch.

The F-beta score with a beta value of 2 was used as a performance metric for
the FCN222As the $F_{1}$ score is more widely used, the $F_{1}$ score will
also be provided in Section IV-A2 to allow for comparison with other published
models.. The $F_{2}$ is a weighted harmonic mean of the precision and recall,
with more weight given to the recall. Maximizing recall was prioritized over
maximizing precision, since false negatives are more detrimental than false
positives during wildfire detection (Arana-Pulido et al. 2018) [2]. Let
precision $P=\frac{tp}{tp+fp}$ and let recall $R=\frac{tp}{tp+fn}$.

$F_{2}=\frac{5PR}{4P+R}.=\frac{tp}{tp+0.2fp+0.8fn}.$

####  III-A3 Training

The FCN was optimized using Adam optimization (Kingma and Ba 2014) [6] with an
exponentially decaying learning rate. The learning rate $\alpha$ as a function
of the epoch of training, $t$, was

$\alpha(t)=\alpha_{0}\,\textbf{e}^{-0.1t},$

where $\alpha_{0}$ was 1E-3 and 0.1 was a hyperparameter. The ReLU activation
function was used throughout the model, with the exception of the last layer
which used the sigmoid function. Batch Normalization was employed on the
channels axis as a regularization technique [3]. The model was trained for 100
epochs with a mini-batch size of 32 images. The 14,274 fire images were used
to train the neural network on an 85-15% train-validation split.

###  III-B Cirrus Band Analysis

Environmental research has shown that subtleties in atmospheric responses to
wildfires provide information about the sources of the fires themselves.
Veselovskii et al. (2021) highlighted the association between wildfire smoke
and cirrus clouds, using LiDAR observations to find that that smoke properties
such as surface area, volume, and concentration could be extracted
simultaneously with cirrus properties [16].

This idea motivated another avenue of experimentation: extracting features
from the cirrus bands as part of sensitivity analysis determining the effect
of feature engineering on computational expense and accuracy.

####  III-B1 Cirrus Data Preparation

Since not all images in the data set contained the cirrus cloud contamination
desirable for the analysis, images that contained cirrus clouds first needed
to be selected. The criteria for determining whether an image contained cirrus
clouds were experimentally decided. Visual observation of the cirrus bands
confirmed that images with cirrus clouds had a much larger pixel value range
than images without cirrus clouds. Accordingly, the minimum pixel value
threshold for an image to contain cirrus clouds was set as 500.333In most
cirrus images, raw pixel values hovered around 5,000. 5,420 of the 24,959
original images satisfied this condition, with 2,152 fire images and 3,268
non-fire images.

####  III-B2 K-Means Clustering for Cirrus Image Segmentation

Image segmentation was conducted to simplify the features of the cirrus
clouds. For each image, a $128^{2}\times$3 array consisting of each pixel
value along with the pixel’s coordinates on the image, was fed into the
K-Means Clustering algorithm with a K-value of 3.

####  III-B3 Clustering Hypothesis

It was hypothesized that if a K-value of 3 was used, the clustering algorithm
would identify spatial regions corresponding to degrees of cloud
contamination: “dense” cirrus, “scattered” cirrus, and “no” cirrus.

###  III-C Cirrus-to-Wildfire CNN

####  III-C1 Sensitivity Analysis

To test the hypothesis that feature simplification could reduce the train time
yet preserve the accuracy of the wildfire detection algorithm, three
convolutional neural networks were constructed using the same architecture but
trained on different data features.

  * •

Benchmark Model (Only SWIR Data) The benchmark model was trained only on the
two Short-wave Infrared bands.

  * •

Control Model (SWIR and Raw Cirrus Data) The control model was trained on the
two Short-wave Infrared bands stacked depthwise with the raw Cirrus band.

  * •

Experimental Model (SWIR and Segmented Cirrus Data) The experimental model was
trained on the two Short-wave Infrared bands stacked depthwise with the
segmented Cirrus band generated using clustering.

####  III-C2 Model Architecture

All three of the above models used the same architecture, hyperparameters, and
random seed. The models contained 1,824,937 trainable parameters. Because most
parameters were concentrated in the final two convolution layers and the first
fully connected layer, L2 Regularization with a lamdba value of 3E-3 was used
in the final two convolution layers, and dropout regularization with a dropout
probability of 0.3 was used before flattening the outputs of the last
convolution. The fully connected layers were also regularized with dropout,
with dropout probabilities decreasing by layer from 0.3 to 0.1. The 5,420
aforementioned images were used to train the neural network on an 80-20%
train-validation split.

Figure 4: The architecture of the convolutional models used for the
sensitivity analysis.

##  IV Results

###  IV-A Fully Convolutional Neural Network

####  IV-A1 Learning Curves

Figure 5: The performance of the FCN over 100 epochs.

The FCN converged quickly on the train set while the validation loss
experienced some noise until around epoch 50. After epoch 50, the validation
loss seemed to subtly increase, implying slight overfitting, but this was
interestingly not reflected in the validation $F_{1}$ and $F_{2}$ Scores.

####  IV-A2 Metrics and Confusion Matrix

TABLE I: Confusion Matrix | Actual: Fire | Actual: No Fire |
---|---|---|---
Predicted: Fire | TP = 18,691 | FP = 2,607 | 21,298
Predicted: No Fire | FN = 277 | TN = 35,072,953 | 35,073,230
| 18,968 | 35,075,560 | 35,094,528

The FCN achieved a 0.962 $F_{2}$ score and a 0.928 $F_{1}$ score on the
validation data set, corresponding to a Precision of 0.878 and a Recall of
0.989. This means that the model very rarely failed to identity an existing
fire, but was occasionally too sensitive, flagging non-fire pixels as fire
pixels.

####  IV-A3 Performance Visualizations

The following figure shows the predicted masks when there are a small number
and large number of fire pixels in the mask, respectively. The third row of
(b) shows that the model failed to identify nonfire pixels when they were
surrounded by fire pixels. This over-labeling of fires was, however, not
observed when the mask had only a fire pixels.

(a) Small number of fire pixels. Fire pixels are boxed.

(b) Large number of fire pixels.

Figure 6: (a), (b) The predicted masks resemble the true masks to a large
degree. For images with more fire pixels, the model tends to predict more
false positives.

####  IV-A4 Model Testing Scenarios

##### Geographical Test

The FCN was tested on 2,000 images from Guyana and Suriname and achieved a
0.934 $F_{2}$ score and a 0.875 $F_{1}$ score. These results confirm that the
FCN is versatile, generalizing well to geographical regions that differ in
topography and climate.

##### Type II Error Test

The FCN was also tested on 2,000 non-fire images. Only 13 pixels were
predicted to contain wildfires, implying an extremely low rate of false
positives, at 3.967E-5%.

###  IV-B Cirrus Band Segmentation

Figure 7: Cirrus band segmented by cirrus contamination level.

Visual observation supported the hypothesis that the K-Means clustering
algorithm would find centroid values that account for the differing degrees of
cirrus cloud contamination in the cirrus band.

###  IV-C Cirrus-to-Wildfire CNN

Figure 8: The learning curves of the three models used for the sensitivity
analysis over 200 epochs.

####  IV-C1 Learning Curves

##### Loss

Both the Benchmark model and the Control model experienced a high degree of
noise during training. The Experimental model experienced much smoother
training, but took longer to converge.

##### Accuracy

The Benchmark model and Control model reached 90% validation accuracy around
25 epochs earlier than the Experimental model. However, the Experimental model
maintained the plateau for around 165 more epochs while the validation
accuracies of Benchmark and Control model fluctuated unpredictably.

##### Overfitting

All three models experienced minimal overfitting, but the difference between
validation and train loss for the Experimental model trends upward at a faster
rate compared to the Benchmark and Control models. This demonstrates a
possible trade-off of the Experimental model.

##### Comparison

The Experimental model took longer to converge, but ultimately surpassed the
accuracy of the Benchmark model while falling short to the accuracy of the
Control model. The model metrics and train times are listed in the table
below, where B corresponds to Benchmark, C corresponds to Control, and E
corresponds to Experimental.

TABLE II: Performance Metrics for Sensitivity Analysis Hypotheses | Metrics | n | P-value
---|---|---|---
Binary Accuracy
$H_{0}:p_{E}-p_{B}=0$ $H_{A}:p_{E}-p_{B}>0$ | $p_{E}$ | $p_{B}$ | 1,084 | 0.00967
0.95572 | 0.93266
$H_{0}:p_{E}-p_{C}=0$ $H_{A}:p_{E}-p_{C}<0$ | $p_{E}$ | $p_{C}$ | 1,084 | 0.05750
0.95572 | 0.96863
Train Time (sec/epoch)
$H_{0}:\mu_{E}-\mu_{B}=0$ $H_{A}:\mu_{E}-\mu_{B}>0$ | $\mu_{E}$ | $\mu_{B}$ | 200 | 0.0
2.76575 | 2.57293
$\sigma_{E}$ | $\sigma_{B}$
0.05577 | 0.06129
$H_{0}:\mu_{E}-\mu_{C}=0$ $H_{A}:\mu_{E}-\mu_{C}<0$ | $\mu_{E}$ | $\mu_{C}$ | 200 | 4.6107E-8
2.76575 | 2.77919
$\sigma_{E}$ | $\sigma_{C}$
0.05577 | 0.06128

####  IV-C2 Sensitivity Analysis

The superior binary accuracy of the Experimental model compared to the
Benchmark was statistically significant at the $\alpha=0.05$ level. However,
the superior binary accuracy of the Control model compared to the Experimental
model was not statistically significant. This shows that the binary accuracy
of the Experimental model was not compromised by simplified cirrus features.

The faster train time of the Benchmark model compared to the Experimental was
statistically significant, which was expected because the Benchmark model did
not contain any cirrus inputs. The faster train time of the Experimental model
compared to the Control model was also statistically significant. This shows
that the Experimental model with simplified cirrus features provided a faster
train time than the Control model, thereby decreasing computational expense.

##  V Discussion

###  V-A Deep Learning Tasks

####  V-A1 FCN

The results of the FCN show that despite the class imbalance inherent in the
data set, the model is still a highly effective discriminator between fire and
non-fire pixels. In fact, of all fire pixels in the validation data set, the
model had a false negative rate of 1.46%. This is a crucial trait for software
used for real-time wildfire detection, since identifying regions with ground
truth fires (i.e. maximizing recall) takes priority over minimizing false
alarms. A potential use case for the FCN is Land Cover Change Detection.
Aerial vehicles can periodically conduct forward propagation, accessing the
predicted probability that each pixel of land contains a wildfire at that
instant. Aerial vehicles can take snapshots of particular wildfire hot spots
hourly, daily, etc. so that fluctuations in the pixel-wise probabilities could
be analyzed using a time-series model to forecast possible spread.

Pereira et al. (2021) referred to their 34,525,121-parameter model as the
“U-Net” and their 2,161,649-parameter model as the “U-Net-Light” [11]. The
table below compares the performance of these models to the performance of the
FCN.

TABLE III: Performance Comparison Model | No. of Param. | P | R | $F_{1}$ | $F_{2}$
---|---|---|---|---|---
“U-Net” | 34,525,121 | 0.898 | 0.888 | 0.893 | 0.890
“U-Net-Light” | 2,161,649 | 0.908 | 0.861 | 0.884 | 0.870
FCN | 2,138,785 | 0.878 | 0.985 | 0.928 | 0.962

The architecture of the FCN required less regularization than the “U-Net” and
“U-Net-Light,” which used dropout layers between convolutions. This
corroborates the findings of Sun et al. (2021), who demonstrated that high
performance can be achieved using a simpler architecture with less or no
regularization, compared to a complex architecture with heavy regularization
[14].

####  V-A2 Cirrus-to-Wildfire CNN

The Experimental model had a lower computational expense than the Control
model, yet did not significantly decrease the binary accuracy despite using
simplified cirrus features. The success of the Experimental model shows that
models with simpler features have the potential to detect active wildfires to
similar degrees of accuracy than models with complex features, while also
providing the added value of faster train times. It has already been
established that not all Landsat 8 bands are needed to detect wildfires. The
next step is to conduct feature engineering to find optimal simplifications of
the input data. In this manner, less data will be needed to train a more
versatile model that is better suited for South American wildfire hot spots
with scarce computational resources.

###  V-B Relationship Between Cirrus Clouds and Wildfire Acreage

Recall that the pixel values of the cirrus bands were segmented into three
cirrus contamination categories: dense, scattered, and none. These segmented
images were analyzed to find an association between the number of fire pixels
and the number of pixels in each contamination category. Each scatter plot
below contains 5,420 data points, with each point representing one image in
the data set for the Cirrus-to-Wildfire CNN.

Figure 9: For each image, the number of fire pixels is plotted against the
number of pixels in each cirrus contamination class. Linear regression fits
are included to give a sense of form and direction.

From the figures above, it appears that the plot of Fire Pixels vs. Dense
Cirrus Pixels is similar to a y-axis reflection of the plot of Fire Pixels vs.
No Cirrus Pixels. Furthermore, it appears that the plot of Fire Pixels vs.
Scattered Cirrus Pixels is nearly symmetric. Notice that the curves formed by
the upper bound data points take on distinct shapes. The images with the
largest number of fire pixels (points near the top of the graphs) appear on
different regions from graph to graph, containing:

  * •

More “Dense Cirrus” pixels than the mean number of “Dense Cirrus” pixels
across all images.

  * •

Similar numbers of “Scattered Cirrus” pixels as the mean number of “Scattered
Cirrus” pixels across all images.

  * •

Less “No Cirrus” pixels than the mean number of “No Cirrus” pixels across all
images.

These findings reveal the possibility of an underlying connection between a
region’s degree of cirrus contamination and probability of containing an
active wildfire, supporting the hypothesis of Veselovskii et al. (2021)[16].

##  VI Conclusions

The computer vision models designed in this study are useful for wildfire
classification and Land Cover Change Detection. A validation $F_{2}$ score of
0.962 and a high degree of generalizability, demonstrated by the test cases,
prove the versatility of the fully convolutional model. The conclusions of the
sensitivity analysis motivate avenues of future research, such as optimizing
Landsat 8 feature engineering. Future research should also investigate the
proposed relationship between the degree of cirrus cloud contamination and
presence of a wildfire.

## Acknowledgement

I would like to thank Mr. Michael Lordan for his time and feedback regarding
this paper.

## References

  * [1] Aragão, L. E., Anderson, L. O., Fonseca, M. G., Rosan, T. M., Vedovato, L. B., Wagner, F. H., … & Saatchi, S. (2018). 21st Century drought-related fires counteract the decline of Amazon deforestation carbon emissions. Nature communications, 9(1), 1-12.
  * [2] Arana-Pulido, V., Cabrera-Almeida, F., Perez-Mato, J., Dorta-Naranjo, B. P., Hernandez-Rodriguez, S., & Jimenez-Yguacel, E. (2018). Challenges of an Autonomous Wildfire Geolocation System Based on Synthetic Vision Technology. Sensors, 18(11), 3631.
  * [3] Dauphin, Y., & Cubuk, E. D. (2020, September). Deconstructing the Regularization of BatchNorm. In International Conference on Learning Representations.
  * [4] Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., & Pal, C. (2016). The importance of skip connections applications in biomedical image segmentation. In Deep learning and data labeling for medical applications (pp. 179-187). Springer, Cham.
  * [5] Khryashchev, V., & Larionov, R. (2020, March). Wildfire Segmentation on Satellite Images using Deep Learning. In 2020 Moscow Workshop on Electronic and Networking Technologies (MWENT) (pp. 1-5). IEEE.
  * [6] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  * [7] Knight, E. J., & Knight, G. (2014). Landsat-8 operational land imager design, characterization and performance. Remote sensing, 6(11), 10286-10305.
  * [8] Matson, M., & Holben, B. (1987). Satellite detection of tropical burning in Brazil. International Journal of Remote Sensing, 8(3), 509-516.
  * [9] Patro, S., & Sahu, K. K. (2015). Normalization: A preprocessing stage. arXiv preprint arXiv:1503.06462.
  * [10] Pereira, G. H. D. A., Fusioka, A.M., Nassu, B. T., & Minneto, R. (2020). A Large-Scale Dataset for Active Fire Detection/Segmentation (Landsat-8). IEEE Dataport. dx.doi.org/10.21227/t9gn-y009
  * [11] Pereira, G. H. D. A., Fusioka, A. M., Nassu, B. T., & Minetto, R. (2021). Active Fire Detection in Landsat-8 Imagery: a Large-Scale Dataset and a Deep-Learning Study. arXiv preprint arXiv:2101.03409.
  * [12] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.
  * [13] Schroeder, W., Oliva, P., Giglio, L., Quayle, B., Lorenz, E., & Morelli, F. (2016). Active fire detection using Landsat-8/OLI data. Remote sensing of environment, 185, 210-220.
  * [14] Sun, C., Sharma, J., & Maiti, M. (2021). Investigating the Relationship Between Dropout Regularization and Model Complexity in Neural Networks. arXiv preprint arXiv:2108.06628.
  * [15] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021). Unsupervised semantic segmentation by contrasting object mask proposals. arXiv preprint arXiv:2102.06191.
  * [16] Veselovskii, I., Hu, Q., Ansmann, A., Goloub, P., Podvin, T., & Korenskiy, M. (2021). Fluorescence lidar observations of wildfire smoke inside cirrus: A contribution to smoke-cirrus–interaction research. Atmospheric Chemistry and Physics Discussions, 1-29.