11institutetext: Instituto de Astrofísica de Canarias, 38205, C/ Vía Láctea,
s/n, La Laguna, Tenerife, Spain 22institutetext: Departamento de Astrofísica,
Universidad de La Laguna, 38205, La Laguna, Tenerife, Spain 33institutetext:
Institute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029
Blindern, NO-0315 Oslo, Norway 44institutetext: Rosseland Centre for Solar
Physics, University of Oslo, P.O. Box 1029 Blindern, NO-0315 Oslo, Norway
55institutetext: Institute for Solar Physics, Dept. of Astronomy, Stockholm
University, AlbaNova University Centre, 10691, Stockholm, Sweden

# Downflowing umbral flashes as an evidence of standing waves in sunspot
umbrae

T. Felipe Downflowing umbral flashes as an evidence of standing waves in
sunspot umbraeDownflowing umbral flashes as an evidence of standing waves in
sunspot umbraeDownflowing umbral flashes as an evidence of standing waves in
sunspot umbraeDownflowing umbral flashes as an evidence of standing waves in
sunspot umbrae V. M. J. Henriques Downflowing umbral flashes as an evidence of
standing waves in sunspot umbraeDownflowing umbral flashes as an evidence of
standing waves in sunspot umbraeDownflowing umbral flashes as an evidence of
standing waves in sunspot umbraeDownflowing umbral flashes as an evidence of
standing waves in sunspot umbrae J. de la Cruz Rodríguez Downflowing umbral
flashes as an evidence of standing waves in sunspot umbraeDownflowing umbral
flashes as an evidence of standing waves in sunspot umbrae H. Socas-Navarro
Downflowing umbral flashes as an evidence of standing waves in sunspot
umbraeDownflowing umbral flashes as an evidence of standing waves in sunspot
umbraeDownflowing umbral flashes as an evidence of standing waves in sunspot
umbraeDownflowing umbral flashes as an evidence of standing waves in sunspot
umbrae

###### Abstract

Context. Umbral flashes are sudden brightenings commonly visible in the core
of some chromospheric lines. Theoretical and numerical modeling suggest that
they are produced by the propagation of shock waves. According to these models
and early observations, umbral flashes are associated with upflows. However,
recent studies have reported umbral flashes in downflowing atmospheres.

Aims. We aim to understand the origin of downflowing umbral flashes. We
explore how the existence of standing waves in the umbral chromosphere impacts
the generation of flashed profiles.

Methods. We performed numerical simulations of wave propagation in a sunspot
umbra with the code MANCHA. The Stokes profiles of the Ca ii 8542 Å line were
synthesized with the NICOLE code.

Results. For freely-propagating waves, the chromospheric temperature
enhancements of the oscillations are in phase with velocity upflows. In this
case, the intensity core of the Ca ii 8542 Å atmosphere is heated during the
upflowing stage of the oscillation. If we consider a different scenario with a
resonant cavity, the wave reflections at the sharp temperature gradient of the
transition region lead to standing oscillations. In this situation,
temperature fluctuations are shifted backward and temperature enhancements
partially coincide with the downflowing stage of the oscillation. In umbral
flash events produced by standing oscillations, the reversal of the emission
feature is produced when the oscillation is downflowing. The chromospheric
temperature keeps increasing while the atmosphere is changing from a downflow
to an upflow. During the appearance of flashed Ca ii 8542 Å cores, the
atmosphere is upflowing most of the time, and only 38% of the flashed profiles
are associated with downflows.

Conclusions. We find a scenario that remarkably explains the recent empirical
findings of downflowing umbral flashes as a natural consequence of the
presence of standing oscillations above sunspot umbrae.

###### Key Words.:

Methods: numerical – Sun: chromosphere – Sun: oscillations – sunspots –
Techniques: polarimetric

##  1 Introduction

Oscillations are a ubiquitous phenomenon in the Sun. Their signatures have
been detected almost everywhere, from the solar interior to the outer
atmosphere. One of the most spectacular manifestations of solar oscillations
are umbral flashes (UFs). They were discovered by Beckers & Tallant (1969)
(see also Wittmann, 1969), who observed them in sunspot umbrae as periodic
brightness enhancements in the core of some chromospheric spectral lines.
Since their first detection, UFs were associated with magneto-acoustic wave
propagation in umbral atmospheres (Beckers & Tallant, 1969) and soon later
Havnes (1970) suggested that UFs in Ca ii lines were formed due to the
temperature enhancement during the compressional stage of these waves.

The analysis of spectropolarimetric data has become a common approach for the
study of UFs during the last two decades, since the studies undertaken by
Socas-Navarro et al. (2000b, a). They found anomalous polarization in the
Stokes profiles and interpreted it as indirect evidence of very fine structure
in UFs, unresolved in arc-sec resolution observations. Their model had one hot
upflowing component embedded in a cool downflowing medium. Further
independent, but still indirect, evidence of such fine structure was obtained
by Centeno et al. (2005). Subsequent studies, benefiting from observations
acquired with higher spatio-temporal resolution, were able to resolve those
components in imaging observations (Socas-Navarro et al., 2009; Henriques &
Kiselman, 2013; Bharti et al., 2013; Yurchyshyn et al., 2014; Henriques et
al., 2015) and in full spectropolarimetric observations (Rouppe van der Voort
& de la Cruz Rodríguez, 2013; de la Cruz Rodríguez et al., 2013; Nelson et
al., 2017). Non-local thermodynamical equilibrium (NLTE) inversions of Ca ii
8542 Å sunspot observations using the NICOLE code (Socas-Navarro et al., 2015)
have found that UF atmospheres are associated with temperature increments of
around 1000 K and strong upflows (de la Cruz Rodríguez et al., 2013). These
observational measurements exhibit a good agreement with the theoretical
picture of UFs as a manifestation of upward propagating slow magneto-acoustic
waves. The amplitude of these waves increases with height as a result of the
density stratification, leading to the development of chromospheric shocks
(Centeno et al., 2006; Felipe et al., 2010b) and UFs (Bard & Carlsson, 2010;
Felipe et al., 2014). The properties of those UF shocks have been recently
evaluated by interpreting spectropolarimetric observations with the support of
the Rankine-Hugoniot relations (Anan et al., 2019; Houston et al., 2020).

The first NLTE inversions of UFs (Socas-Navarro et al., 2001) were
computationally very expensive (typically several hours per individual
spectrum) and therefore only a few profiles from the same location and
different times were inverted. Later improvements in code optimization,
parallelization, and computer hardware allowed de la Cruz Rodríguez et al.
(2013) to analyze a few 2D maps with spatial resolution. As this trend
continued, it is now possible to make UF studies with higher spatial and
temporal resolution and much better statistics (e.g., Henriques et al., 2017;
Joshi & de la Cruz Rodríguez, 2018; Houston et al., 2018, 2020). These recent
results have revealed the existence of UFs whose Stokes profiles are better
reproduced with atmospheres dominated by downflows (Henriques et al., 2017;
Bose et al., 2019; Houston et al., 2020).

Accommodating the downflowing UFs to the widely accepted scenario of upward
propagating shock waves is a theoretical challenge. Studies employing
sophisticated synthesis tools and numerical simulations of the umbral
atmosphere have validated the upflowing UF model (Bard & Carlsson, 2010;
Felipe et al., 2014, 2018b; Felipe & Esteban Pozuelo, 2019). In those
simulations, magneto-acoustic waves could propagate from the photosphere to
the chromosphere. According to the phase relations of these propagating waves,
velocity and temperature oscillations exhibit opposite phases (considering
that positive velocities correspond to downflows), and the temperature
increase causing the flash to match an upflow. However, recent studies have
shown evidences of a resonant cavity above sunspot umbrae (see Jess et al.,
2020a; Felipe, 2020; Jess et al., 2020b; Felipe et al., 2020). This cavity is
produced by wave reflections at the strong temperature gradient in the
transition region (e.g., Zhugzhda & Locans, 1981; Zhugzhda, 2008). Waves
trapped in a resonant cavity produce standing oscillations, whose temperature
fluctuations are delayed by a quarter period with respect to the velocity
signal, instead of the half-period delay from freely propagating waves
(Deubner, 1974; Felipe et al., 2020). In this study, we use numerical
simulations and full-Stokes NLTE radiative transfer calculations to explore
the generation of intensity emission cores in the Ca ii 8542 Å line during the
downflowing stage of standing umbral oscillations, and how they can support
the existence of the recently reported dowflowing UFs.

##  2 Numerical methods

Non-linear wave propagation between the solar interior and corona was
simulated using the code MANCHA (Khomenko & Collados, 2006; Felipe et al.,
2010a) in a two-dimensional domain. The details of the simulations can be
found in Appendix A. We have computed synthetic full-Stokes spectra in the Ca
ii 8542 Å line taking snapshots from our numerical simulation and then
computing the spectra with the NICOLE code (Socas-Navarro et al., 2015, see
details in Appendix B). The code assumes statistical equilibrium. Wedemeyer-
Böhm & Carlsson (2011) showed that non-equilibrium effects in Ca ii are small
at the formation height of the infrared triplet lines and, therefore,
statistical equilibrium should suffice to model Ca ii atoms. We calculated the
transition probabilities between levels for Ca ii in a sunspot model. They
indicate characteristic transition times much shorter than one second
(Appendix B), meaning that departures from statistical equilibrium are
insignificant for the dynamics here studied.

Figure 1: Development of an umbral flash produced by propagating waves as seen
in Ca ii 8542 Å. Panel a: Temporal evolution of the vertical velocity (solid
line, left axis), temperature (black dashed line, black right axis), and
density (red dashed line, red right axis) fluctuations at $z=1.35$ Mm. Blue
shaded region corresponds to the time when an umbral flash is identified in
the Stokes profiles and the atmosphere is upflowing at $\log\tau=-5.1$. Lower
panels: Stokes I (left column), Stokes V (middle column), and atmospheric
models (right column). Each row corresponds to a different time step, as
indicated at the bottom of the first column panels and the vertical dotted
lines in panel a. In the right column, the vertical velocity (solid line, left
axis) and temperature (dashed line, right axis) of the atmospheric model is
plotted. The dashed vertical line marks the height of $\log\tau=-5.1.$ Figure
2: Same as Fig. 1, but for standing oscillations. Red shaded region in panel a
indicates the time when an umbral flash is identified in the Stokes profiles
and the atmosphere is downflowing at $\log\tau=-5.1.$

##  3 Results

We analyze how UFs are generated in the case of propagating and standing
oscillations. In Appendix C, we discuss the differences in the phase relations
between temperature and velocity fluctuations from propagating and standing
waves as determined from the analysis of numerical simulations. In the
following, for the examination of the propagating-wave case, we focus on the
first wavefront of the simulation reaching the chromosphere, before any
previous wavefronts are reflected and a standing wave is settled. We chose to
follow this approach (instead of using an independent simulation without
transition region) in order to employ exactly the same atmospheric
stratification for both cases. This way, we can assure that the differences in
the properties of the UFs are produced by the nature of the oscillations and
not by the changes in the background atmospheres.

###  3.1 Umbral flashes from propagating waves

Figure 1 illustrates the development of UFs generated by a propagating wave at
the beginning of the simulation. This case is qualitatively comparable to the
synthetic UFs analyzed by Bard & Carlsson (2010), Felipe et al. (2014), Felipe
et al. (2018b), and Felipe & Esteban Pozuelo (2019). We refer the reader to
those papers for a detailed discussion of the formation of UFs. In the present
paper, we focus on evaluating the chromospheric plasma motions during the
appearance of the UF. Similarly to Henriques et al. (2017), we define a
profile as flashed when the intensity around the core of the Ca ii 8542 Å line
is above the value of Stokes I at $\Delta\lambda=0.942$ Å.

The time steps when flashed profiles take place are marked in blue in Figure
1a. As expected, they coincide with a temperature enhancement at $z=1.35$ Mm,
which is roughly the height where the response of the Ca ii 8542 Å line to
temperature is maximum during UFs (Fig. 4). The temperature maximum precedes
the highest value of the upflow velocity by 9 s. Prior to the appearance of
the flash (Fig. 1b) the atmosphere is downflowing agreeing with observations
(Henriques et al., 2020), but the temperature fluctuation is too low to
produce an emission core (panel d). When the UF starts to develop (panel e)
the atmosphere is upflowing at the optical depth where the response of the
line is maximum ($\log\tau\approx-5.1$, Fig. 4) and downflowing at higher
layers (panel g). A similar situation can be seen at later steps (panels h and
j). The change from upflows to downflows at a certain height is a
manifestation of the propagating character of the waves since for standing
waves we expect most of the chromosphere to be oscillating in phase.

###  3.2 Umbral flashes from standing waves

The atmospheres generating UFs in the case of standing oscillations exhibit
several significant differences from those from propagating waves. First, as
discussed in Appendix C, the phase shift between velocity and temperature
fluctuations is different. In the UF illustrated in Fig. 2, the peak of the
temperature enhancement at $z=1.35$ Mm takes place 26 s before the maximum
value of the upflow velocity (panel a). Second, higher chromospheric layers
are oscillating in phase above the resonant node as a result of the standing
oscillations (right column from the bottom subset of Fig. 2).

Due to the earlier appearance of a positive temperature fluctuation (in
comparison with the case of propagating waves), they partially coincide with
the downflow of the velocity oscillation. This way, chromospheric temperatures
high enough to generate UFs are reached while a strong downflow is still
taking place (panels e, f, and g from Fig. 2). The strongest UF (at the peak
of the temperature fluctuation) is found when the chromosphere is already
upflowing (panels h, i, and j from Fig. 2). In this example, the total
duration of the UF is 45 s. During most of this time, the atmosphere at the
height where the response of the Ca ii 8542 line is maximum is upflowing.
However, in the first 12.5 s of the flash, the atmosphere exhibits a downflow
at $\log\tau=-5.1$ (red shaded region in Fig. 2). The phase difference between
velocity and temperature is not strictly $\pi/2$ (see Figs. 2a and 5c), but
the temperature peak is slightly shifted towards the time when the atmosphere
is upflowing, contributing to the longer duration of the upflowing stage of
the UF. At the beginning of the UF ($t_{2}$ in Fig. 2), the chromospheric
density is reduced (red dashed line in Fig. 2a). The UF develops during the
compressional phase of the oscillations and, thus, upflowing UFs are
associated with a higher density.

Figure 3: Properties of UFs in a simulation with a chromospheric resonant
cavity. Panel a: Wavelength shift of the center of the emission core
($\Delta$v) as a function of the vertical velocity at $\log\tau=-5.1$ (vz).
Panel b: Maximum intensity of the emission feature as a function of the
temperature fluctuation at $\log\tau=-5.1$. In panels a and b, blue dots
represent upflowing UFs and red dots correspond to downflowing UFs. Panel c:
Histogram illustrating the duration of upflowing (blue bars) and downflowing
(bars filled with red lines) UF events. Panel d: Relation between the duration
of the upflowing UFs and the preceding downflowing UFs. The color bar
indicates the number of occurrences of UFs with the corresponding durations.

###  3.3 Properties of standing waves umbral flashes

We have identified a total of 1777 flashed profiles in the numerical
simulation once the standing oscillations are established (see Appendix A).
Approximately 38% of those profiles are associated with downflows at
$\log\tau=-5.1$, whereas the rest of the cases are produced during upflows.

Figure 3 illustrates some of the properties of those UFs. Panel a shows the
relation between the wavelength shift of the emission feature and the actual
chromospheric velocity. For most of the UFs, the emission core is blueshifted,
even when the atmosphere generating the flash is downflowing (see Fig. 2e).
Henriques et al. (2017) showed how a downflowing UF can lead to an apparently
blueshifted Ca ii 8542 Å emission core. Our analysis is based on the
evaluation of the velocity at a fixed $\log\tau=-5.1$. We cannot discard
changes in the velocity response function during UFs.

Figure 3b shows the peak value of the emission core as a function of the
temperature fluctuation. The latter is defined as the difference between the
actual temperature and that from the background model (at rest) at
$\log\tau=-5.1$ (6100 K). Chromospheric temperature enhancements of at least
1000 K are required to produce a flashed profile. The intensity of the core
shows small variations for temperature perturbations up to 1600 K. Higher
temperature enhancements generate the strongest UFs found in the simulation,
exhibiting a significant increase in the intensity of the core.

As previously pointed out, even in the case of standing oscillations most of
the flashes are associated with upflows. Panels c and d from Fig. 3 examine
the duration of the upflowing and downflowing stages of UF events. We define
the duration as the temporal span while the intensity core at a certain
spatial location is continuously in emission (above the same threshold
previously defined). Since upflowing UFs are generally preceded by downflowing
UFs (Sect. 3.2), we have measured independently the duration of each of these
phases. Figure 3d shows that in the vast majority of the cases, the upflowing
stage lasts longer than the downflowing phase of the same event (white dashed
line marks the locations in the diagram where upflowing and downflowing UFs
have the same duration). They are similar to the case illustrated in Fig. 2.
For both downflowing and upflowing phases the duration is generally in the
range between 10 to 20 s, but the upflowing part of the UFs can extend longer
than 25 s, whereas almost no downflows last that long. This explains the
prevalence of detected UFs associated with upflows.

##  4 Discussion and conclusions

Umbral flashes are a well-known phenomenon and their relation to the
oscillatory processes in sunspot atmospheres is an established fact. Early
modeling of UFs has characterized them as the result of upward-propagating
waves (Havnes, 1970; Bard & Carlsson, 2010; Felipe et al., 2014). In this
scenario, most of the line formation of UFs takes place at layers where the
chromospheric flows generated by the oscillation are upflowing. First
spectropolarimetric analyses of UFs apparently confirmed this model (Socas-
Navarro et al., 2000b; de la Cruz Rodríguez et al., 2013). However, recent
studies have showed indications of a chromospheric resonant cavity above
sunspot umbrae (Jess et al., 2020a), whose presence has been proved by Felipe
et al. (2020) based on the analysis of the phase relations of chromospheric
oscillations. Umbral chromospheric waves do not propagate but form standing
waves instead.

In this study, we have modeled the development of UFs produced by standing
oscillations for the first time. The phase shift between temperature and
velocity fluctuations from standing waves differs from that of purely
propagating waves. In the latter, temperature enhancements are associated with
upflows, leading to upflowing UFs, whereas in standing oscillations the
temperature increase takes place a quarter period earlier. This way, standing
waves generate UFs partially during the downflowing stage of the oscillation.
Interestingly, recent studies have started to report downflowing UFs
(Henriques et al., 2017; Houston et al., 2020; Bose et al., 2019). Henriques
et al. (2017) suggested that dowflowing UFs may be related to the presence of
coronal loops rooted in the sunspot umbra, since previous works have detected
supersonic downflows in the transition region of such structures (e.g., Kleint
et al., 2014; Kwak et al., 2016; Chitta et al., 2016). Our model offers an
alternative explanation, interpreting downflowing UFs as a natural feature of
the global process of sunspot oscillations. This is also the recent
interpretation of Henriques et al. (2020) but here we show that such downflows
are evidence of the standing-wave nature of these oscillations. Henriques et
al. (2017) reported strong downflowing atmospheres for UFs, with velocities up
to 10 km s-1, whereas our model produces downflowing UFs with velocities
around 1 km s-1.

The development of UFs generated by standing waves is as follows. The
chromospheric temperature starts to increase during the stage when the
chromosphere is downflowing. All the chromospheric layers above the velocity
resonant node are in phase and, thus, downflowing. This temperature increase
is high enough to produce an emission core in the Ca ii 8542 Å line. During
the peak of the temperature fluctuation, the chromospheric plasma changes from
downflowing to upflowing. The profiles remain flashed for the upflowing
atmospheres until the chromospheric temperature reaches again a value closer
to that of a quiescent umbra. Unfortunately, the temporal cadence of most Ca
ii 8542 Å observations up to date, in the range between 25-28 s (e.g.,
Henriques et al., 2017; Joshi & de la Cruz Rodríguez, 2018) and 48 s (Houston
et al., 2020), is not sufficient to capture the evolution of UF events. The
analysis is also challenged by the sudden variations of the atmosphere during
the time required by Fabry-Perot spectrographs, such as the Interferometric
BIdimensional Spectrometer (IBIS, Cavallini, 2006) and the CRisp Imaging
SpectroPolarimeter (CRISP, Scharmer et al., 2008), to scan the line (Felipe et
al., 2018a). However, observations of small-scale umbral-brightenings using a
cadence of 14 s exhibit some indication of a temporal evolution consistent
with that predicted by our modeling (Henriques et al., 2020). Upcoming
observations with the National Science Foundation’s Daniel K. Inouye Solar
Telescope (DKIST, Rimmele et al., 2020) will provide the ideal data for
studying the temporal variations of UFs.

Our simulations show that the upflowing stage of UFs is generally longer than
the downflowing part (Fig. 3d). This is in agreement with the prevalence of
upflowing solutions in NLTE inversions of observed UFs (Socas-Navarro et al.,
2000b; de la Cruz Rodríguez et al., 2013), even in studies where downflowing
UFs are also reported (Houston et al., 2020).

This letter provides a missing piece of the puzzle in what has been a
convergence of observations and simulations by tying together the evidence for
downflowing UFs and the existence of cavities in the chromosphere of the umbra
as the latter will naturally generate the former. Finally, the case for a
corrugated surface where downflows are significant (Henriques et al., 2020),
together with the importance of cavities for such downflows, implies that
future observations constraining the transition region of sunspots should find
the transition region itself to be highly corrugated.

###### Acknowledgements.

Financial support from the State Research Agency (AEI) of the Spanish Ministry
of Science, Innovation and Universities (MCIU) and the European Regional
Development Fund (FEDER) under grant with reference PGC2018-097611-A-I00 is
gratefully acknowledged. VMJH is funded by the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(SolarALMA, grant agreement No. 682462) and by the Research Council of Norway
through its Centres of Excellence scheme (project 262622). JdlCR is supported
by grants from the Swedish Research Council (2015-03994) and the Swedish
National Space Agency (128/15). This project has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation program (SUNMAG, grant agreement 759548). The authors
wish to acknowledge the contribution of Teide High-Performance Computing
facilities to the results of this research. TeideHPC facilities are provided
by the Instituto Tecnológico y de Energías Renovables (ITER, SA). URL:
http://teidehpc.iter.es.

## References

  * Al et al. (1998) Al, N., Bendlin, C., & Kneer, F. 1998, A&A, 336, 743
  * Anan et al. (2019) Anan, T., Schad, T. A., Jaeggli, S. A., & Tarr, L. A. 2019, ApJ, 882, 161
  * Avrett (1981) Avrett, E. H. 1981, in The Physics of Sunspots, ed. L. E. Cram & J. H. Thomas, 235–255
  * Bard & Carlsson (2010) Bard, S. & Carlsson, M. 2010, ApJ, 722, 888
  * Beckers & Tallant (1969) Beckers, J. M. & Tallant, P. E. 1969, Sol. Phys., 7, 351
  * Berenger (1994) Berenger, J. P. 1994, Journal of Computational Physics, 114, 185
  * Bharti et al. (2013) Bharti, L., Hirzberger, J., & Solanki, S. K. 2013, A&A, 552, L1
  * Bose et al. (2019) Bose, S., Henriques, V. M. J., Rouppe van der Voort, L., & Pereira, T. M. D. 2019, A&A, 627, A46
  * Cavallini (2006) Cavallini, F. 2006, Sol. Phys., 236, 415
  * Centeno et al. (2006) Centeno, R., Collados, M., & Trujillo Bueno, J. 2006, ApJ, 640, 1153
  * Centeno et al. (2005) Centeno, R., Socas-Navarro, H., Collados, M., & Trujillo Bueno, J. 2005, ApJ, 635, 670
  * Chitta et al. (2016) Chitta, L. P., Peter, H., & Young, P. R. 2016, A&A, 587, A20
  * Christensen-Dalsgaard et al. (1996) Christensen-Dalsgaard, J., Dappen, W., Ajukov, S. V., & 30 co-authors. 1996, Science, 272, 1286
  * de la Cruz Rodríguez et al. (2013) de la Cruz Rodríguez, J., Rouppe van der Voort, L., Socas-Navarro, H., & van Noort, M. 2013, A&A, 556, A115
  * Deubner (1974) Deubner, F.-L. 1974, Sol. Phys., 39, 31
  * Felipe (2019) Felipe, T. 2019, A&A, 627, A169
  * Felipe (2020) Felipe, T. 2020, Nature Astronomy [arXiv:2007.10471]
  * Felipe & Esteban Pozuelo (2019) Felipe, T. & Esteban Pozuelo, S. 2019, A&A, 632, A75
  * Felipe et al. (2010a) Felipe, T., Khomenko, E., & Collados, M. 2010a, ApJ, 719, 357
  * Felipe et al. (2011) Felipe, T., Khomenko, E., & Collados, M. 2011, ApJ, 735, 65
  * Felipe et al. (2010b) Felipe, T., Khomenko, E., Collados, M., & Beck, C. 2010b, ApJ, 722, 131
  * Felipe et al. (2020) Felipe, T., Kuckein, C., González Manrique, S. J., Milic, I., & Sangeetha, C. R. 2020, ApJ, 900, L29
  * Felipe et al. (2018a) Felipe, T., Kuckein, C., & Thaler, I. 2018a, A&A, 617, A39
  * Felipe et al. (2014) Felipe, T., Socas-Navarro, H., & Khomenko, E. 2014, ApJ, 795, 9
  * Felipe et al. (2018b) Felipe, T., Socas-Navarro, H., & Przybylski, D. 2018b, A&A, 614, A73
  * González-Morales et al. (2018) González-Morales, P. A., Khomenko, E., Downes, T. P., & de Vicente, A. 2018, A&A, 615, A67
  * Havnes (1970) Havnes, O. 1970, Sol. Phys., 13, 323
  * Henriques & Kiselman (2013) Henriques, V. M. J. & Kiselman, D. 2013, A&A, 557, A5
  * Henriques et al. (2017) Henriques, V. M. J., Mathioudakis, M., Socas-Navarro, H., & de la Cruz Rodríguez, J. 2017, ApJ, 845, 102
  * Henriques et al. (2020) Henriques, V. M. J., Nelson, C. J., Rouppe van der Voort, L. H. M., & Mathioudakis, M. 2020, A&A, 642, A215
  * Henriques et al. (2015) Henriques, V. M. J., Scullion, E., Mathioudakis, M., et al. 2015, A&A, 574, A131
  * Houston et al. (2018) Houston, S. J., Jess, D. B., Asensio Ramos, A., et al. 2018, ApJ, 860, 28
  * Houston et al. (2020) Houston, S. J., Jess, D. B., Keppens, R., et al. 2020, ApJ, 892, 49
  * Jess et al. (2020a) Jess, D. B., Snow, B., Houston, S. J., et al. 2020a, Nature Astronomy, 4, 220
  * Jess et al. (2020b) Jess, D. B., Snow, B., Fleck, B., Stangalini, M., & Jafarzadeh, S. 2020b, Nature Astronomy
  * Joshi & de la Cruz Rodríguez (2018) Joshi, J. & de la Cruz Rodríguez, J. 2018, A&A, 619, A63
  * Khomenko & Collados (2006) Khomenko, E. & Collados, M. 2006, ApJ, 653, 739
  * Kleint et al. (2014) Kleint, L., Antolin, P., Tian, H., et al. 2014, ApJ, 789, L42
  * Kwak et al. (2016) Kwak, H., Chae, J., Song, D., et al. 2016, ApJ, 821, L30
  * Maltby et al. (1986) Maltby, P., Avrett, E. H., Carlsson, M., et al. 1986, ApJ, 306, 284
  * Nelson et al. (2017) Nelson, C. J., Henriques, V. M. J., Mathioudakis, M., & Keenan, F. P. 2017, A&A, 605, A14
  * Rimmele et al. (2020) Rimmele, T. R., Warner, M., Keil, S. L., et al. 2020, Sol. Phys., 295, 172
  * Rouppe van der Voort & de la Cruz Rodríguez (2013) Rouppe van der Voort, L. & de la Cruz Rodríguez, J. 2013, ApJ, 776, 56
  * Scharmer et al. (2008) Scharmer, G. B., Narayan, G., Hillberg, T., et al. 2008, ApJ, 689, L69
  * Socas-Navarro et al. (2015) Socas-Navarro, H., de la Cruz Rodríguez, J., Asensio Ramos, A., Trujillo Bueno, J., & Ruiz Cobo, B. 2015, A&A, 577, A7
  * Socas-Navarro et al. (2009) Socas-Navarro, H., McIntosh, S. W., Centeno, R., de Wijn, A. G., & Lites, B. W. 2009, ApJ, 696, 1683
  * Socas-Navarro et al. (2000a) Socas-Navarro, H., Trujillo Bueno, J., & Ruiz Cobo, B. 2000a, ApJ, 544, 1141
  * Socas-Navarro et al. (2000b) Socas-Navarro, H., Trujillo Bueno, J., & Ruiz Cobo, B. 2000b, Science, 288, 1396
  * Socas-Navarro et al. (2001) Socas-Navarro, H., Trujillo Bueno, J., & Ruiz Cobo, B. 2001, ApJ, 550, 1102
  * Wedemeyer-Böhm & Carlsson (2011) Wedemeyer-Böhm, S. & Carlsson, M. 2011, A&A, 528, A1
  * Wittmann (1969) Wittmann, A. 1969, Sol. Phys., 7, 366
  * Yurchyshyn et al. (2014) Yurchyshyn, V., Abramenko, V., Kosovichev, A., & Goode, P. 2014, ApJ, 787, 58
  * Zhugzhda (2008) Zhugzhda, Y. D. 2008, Sol. Phys., 251, 501
  * Zhugzhda & Locans (1981) Zhugzhda, Y. D. & Locans, V. 1981, Soviet Astronomy Letters, 7, 25

##  Appendix A Numerical setup

The ideal magnetohydrodynamic (MHD) equations were solved using the code
MANCHA (Khomenko & Collados 2006; Felipe et al. 2010a). The numerical
simulations were computed using the 2.5D approximation. We employed a two-
dimensional domain, but vectors keep three-dimensional coordinates. code
solves the non-linear equations since UFs are characterized by acoustic shock
waves. The shock capturing capabilities of the code have been validated by
comparing its performance with analytic solutions of shock tube tests in both
the MHD approximation (Felipe et al. 2010a) and including the effects of
partial ionization (González-Morales et al. 2018). In this paper, we have
restricted the analysis to ideal MHD. The vertical direction of the
computational domain spans from $z=-1140$ km to $z=3500$ km, with $z=0$ set at
the height where the optical depth at 5000 Å is unity ($\log\tau=0$) and using
a constant spatial step of 10 km. At top and bottom boundaries we imposed a
perfectly matched layer (Berenger 1994), which damps waves with minimum
reflection. The horizontal domain includes 96 points with a spatial step of 50
km. Periodic boundary conditions were imposed in the horizontal direction.

The code computes the temporal evolution of waves driven below the solar
surface as they propagate in a background umbral model. The background
atmosphere is constant in the horizontal direction. The vertical
stratification of a modified (Avrett 1981) model was established at all
horizontal locations. The modifications of this umbral atmosphere include the
extension of the model to deeper layers by smoothly merging it with model S
(Christensen-Dalsgaard et al. 1996) and the shift of the chromospheric
temperature increase to higher layers. The later was introduced to employ a
model that generates Ca ii 8542 Å profiles comparable to those observed in
actual observations, similar to Felipe et al. (2018b). The model includes a
sharp temperature gradient from chromospheric temperature values to coronal
temperatures starting at $z\approx 1.8$ Mm. The atmosphere is permeated by a
vertical magnetic field with constant strength of 2000 G. Waves are driven
with a vertical force imposed at $z=-180$ km. The temporal and spatial
dependence of the driver were extracted from temporal series measured in the
photospheric Si i 10827 Å line (Felipe et al. 2018a) following the methods
described in Felipe et al. (2011).

A total of 55 min of solar time were computed. The simulated atmospheres were
saved with a temporal cadence of 5 s. For the analysis of the properties of
UFs produced by standing waves (Sect. 3.3) we have avoided the first 15 min of
the simulation since during this time the first wavefronts are arriving in the
chromosphere and the standing oscillations are not established. Thus, the
analysis is restricted to the last 40 min of simulations.

Table 1:  Transition probabilities per second at $\log\tau=-5$ in Maltby M
sunspot model (Maltby et al. 1986)

Transition | P3k (s-1) | Pk3 (s-1)
---|---|---
1 | 2669 | 19407
2 | 73605 | 49744
4 | 23606 | 596
5 | 1807920 | 26141
6 | 4.10796$\times 10^{-2}$ | 1.47390$\times 10^{-1}$

  * •

Middle column indicates the transition probabilities from level 3 (Ca ii 8542
Å lower level) to all other levels $k$ (given by the first column). Right
column shows the transition probabilities from level $k$ to level 3. Level
$k=6$ is the Ca ii continuum.

Figure 4: Response functions of the intensity profile of the Ca ii 8542 Å line
during the development of an UF. Panel a: Response function of Stokes I to the
temperature at the core of the line (dashed line) and to the velocity at the
wavelength where the response is maximum (8542.2 Å, solid line) at $t=1405$ s
(see profile in Fig. 2k). Lower axis corresponds to the optical depth and
upper axis to the geometrical height. Panel b: Optical depth of the maximum of
the response function to temperature. Panel c: Geometrical height of the
maximum of the response function to temperature. Color-shaded regions in
panels b and c have the same meaning as in Fig. 2. Figure 5: Numerical
simulations of wave propagation in a sunspot umbra. Panel a: Vertical velocity
at $z=1.35$ Mm for all the horizontal positions. Panel b: Vertical velocity
(red line, left axis) and temperature fluctuations (blue line, right axis) at
$X=-1.1$ Mm (location indicated by the green vertical line in panel a). Panel
c: Average phase difference between velocity and temperature oscillations for
waves with a frequency between 5 and 6 mHz as a function of atmospheric
height. The red line represents the phase shift for standing waves from a
simulation with a resonant cavity. The blue line corresponds to propagating
waves from a simulation without a transition region. Dotted (dashed) lines
indicate a phase shift of $\pm\pi/2$ (0).

##  Appendix B Spectral synthesis

The Stokes profiles of the Ca ii 8542 Å line were synthesized for all the
spatial locations and time steps of the simulation using the NLTE code NICOLE
(Socas-Navarro et al. 2015). NICOLE was fed with the velocity, temperature,
gas pressure, density, and magnetic field vector stratification of each
atmosphere (for a specific horizontal position and time) in a geometrical
scale, as given by the output of the numerical simulation. An artificial
macroturbulence of 1.8 km s-1 was added to balance the absence of small scale
motions in the numerical calculations and generate broader line profiles,
similar to those measured in actual observations. The Stokes profiles were
first synthesized with a spectral resolution of 5 mÅ. They were then degraded
to a resolution of 55 mÅ by convolving them with a 100 mÅ full width half
maximum Gaussian filter centered at the wavelengths chosen for the spectral
sampling.

Table 1 shows the transition probabilities from the lower level of the Ca ii
8542 Å line to all other levels and vice versa. The large values point to very
short characteristic transition times (below one second), confirming that the
assumption of statistical equilibrium leads to very small decay times, shorter
than the temporal scales of interest for our study. The only exception are the
rates to and from the Ca ii continuum ($k=6$). The characteristic times for
this transition is of the order of minutes, which means that ionization to Ca
iii might be lagging behind the dynamics. However, this is not relevant for
our purposes since Ca ii is the dominant species.

Figure 4 illustrates the response function of the Ca ii 8542 Å intensity to
the temperature during the development of an UF. The main contribution to
Stokes I comes from an optical depth around $\log\tau=-5.1$. We have chosen
this optical depth as the reference for the analyses presented in this study
since it is maintained as the main contribution to the formation of the line
during the whole temporal span of UFs (color-shaded regions in Fig. 4b).
During this time, $\log\tau=-5.1$ samples a geometrical height around $z=1350$
km (Fig. 4c). Remarkable changes in the chromospheric region with higher
sensitivity to the temperature are found between flashed and non-flashed
profiles (ranging between $\log\tau=-4.7$ and $\log\tau=-5.6$).

##  Appendix C Standing and propagating oscillations

Figure 5 illustrates the chromospheric oscillations from the numerical
simulation. Panel a shows the velocity wavefield for all the umbral spatial
positions included in the simulation. We use the usual spectroscopic velocity
convention in astrophysics where negative velocities correspond to upflows
(blueshifts, indicated in blue in Fig. 5a) and positive velocities represent
downflows (redshifts, red in Fig. 5a). Figure 5b exhibits the temporal
evolution of velocity (red line) and temperature (blue line) fluctuations at a
randomly selected umbral location. Temperature wavefronts are delayed with
respect to velocity wavefronts. For a better examination, Fig. 5c shows the
phase difference between temperature and velocity oscillations for waves with
frequencies in the range 5-6 mHz (red line). Only the atmospheric heights of
interest for the formation of the Ca ii infrared lines are plotted. The phase
shift is around $\pi/2$, indicating that temperature fluctuations lag velocity
oscillations by a quarter period, in agreement with theoretical estimates of
standing waves (Deubner 1974; Al et al. 1998).

The blue line in Fig. 5c corresponds to the phase-shift derived from a
numerical simulation where the transition region is removed. Instead, a
constant temperature atmosphere is set above $z=1.5$ Mm (Felipe 2019; Felipe
et al. 2020). In this situation, waves can freely propagate in the upper
atmospheric layers and the phase relation between velocity and temperature
fluctuations tends to $\pi$, the expected value for propagating waves. The
phase-shift from purely propagating waves (strictly $\pi$) is not obtained due
to two reasons. First, there are temperature gradients in the model (the
temperature increase from the photospheric to the chromospheric temperature),
which produce some partial wave reflections at $z\approx 1.6$ Mm. Second, as
waves propagate upwards they develop into shocks and temperature enhancements
take place earlier, shifting towards the compression phase of the wave.
Summarizing, in the propagating-waves scenario (blue line in Fig. 5c)
temperature fluctuations are approximately in anti-phase with velocity
oscillations, with temperature enhancements corresponding to upflows. If a
chromospheric resonant cavity is present (red line), temperature oscillations
are shifted backward and their increase partially coincides with the
downflowing stage of the oscillations. For waves with $\approx$5.5 mHz
frequency, from $z=1.1$ Mm to $z=1.8$ Mm the shift of temperature fluctuations
is in the range between 25 and 80 s.


# Can We Automate the Analysis of Online Child Sexual Exploitation Discourse?

Darren Cook
University of Liverpool
Liverpool, UK
<EMAIL_ADDRESS>
Miri Zilka
University of Cambridge
Cambridge, UK
<EMAIL_ADDRESS>
Heidi DeSandre
University of Liverpool
Liverpool, UK
Susan Giles
University of Liverpool
Liverpool, UK
Adrian Weller
University of Cambridge
The Alan Turing Institute
UK
Simon Maskell
University of Liverpool
Liverpool, UK

###### Abstract

Social media’s growing popularity raises concerns around children’s online
safety. Interactions between minors and adults with predatory intentions is a
particularly grave concern. Research into online sexual grooming has often
relied on domain-experts to manually annotate conversations, limiting both
scale and scope. In this work, we test how well automated methods can detect
conversational behaviors and replace an expert human annotator. Informed by
psychological theories of online grooming, we label $6772$ chat messages sent
by child-sex offenders with one of eleven predatory behaviors. We train bag-
of-words and natural language inference models to classify each behavior, and
show that the best performing models classify behaviors in a manner that is
consistent, but not on-par, with human annotation.

##  1 Introduction

Social media’s growing popularity amongst children and young adults raises
serious concerns for their safety. The threat of online sexual grooming is an
increasing problem in the digital age (Greene-Colozzi et al., 2020). In 2021
alone, UK police forces recorded over 5000 offences relating to sexual
communications with a child, with social media apps such as Instagram and
Snapchat popular amongst online predators (NSPCC, 2021). Minors who fall
victim to offenders suffer considerably, with many abusers seeking to
establish physical contact offline (Shelton et al., 2016). To assist law
enforcement, academics have been trying to identify in advance those predators
that steer the relationship towards physical encounters (Briggs et al., 2011;
Winters et al., 2017; O’Connell, 2003; Williams et al., 2013). This important
work predominantly relies on time-consuming human-effort to manually annotate
conversations involving child sex offenders. In this work, we explore the
extent to which automated annotation can mimic an expert human annotator.

Automatically detecting predatory behavior is a challenging task. Offenders
use a variety of subtle behaviors to manipulate the flow of the conversation.
Predators may use flattery to build trust (Barber and Bettez, 2021), make
threats or bribe a child as a coercion tactic (Joleby et al., 2021). Whilst
human experts can identify these contextual psychological behaviors in text,
their implicitness can be problematic for machines (Buckingham and Alali,
2020). In computational social sciences, dictionary-based approaches (Tausczik
and Pennebaker, 2010) are often used to identify psychological characteristics
such as neuroticism (Bogdanova et al., 2014). However, these methods heavily
rely on the included vocabulary, causing a large numbers of false positives
for some behaviors (Kaur et al., 2021) while overlooking others (Broome et
al., 2020). Prior work utilising machine learning has focused on identifying
predators from a mixed corpus of illicit and everyday conversations (Inches
and Crestani, 2012; Pendar, 2007; Miah et al., 2011; Ebrahimi et al., 2016;
Gupta et al., 2012). While valuable in its own right, this line of research
does not offer significant value to grooming experts as it identifies grooming
behavior after-the-fact (Razi et al., 2021) and does not link or contribute to
psychological insight.

This work is novel in its use of supervised and deep learning methods to
analyse an expert-annotated corpus of real conversations between sexual
predators and decoys pretending to be early teens. We report results and
highlight the performance possible without incurring the cost of expert-
labelling. The paper is organised as follows: the methods and experimental
setup are described in Section 2; Section 3 then summarises results (with
further details provided in the appendices) before Section 4 concludes the
paper.

##  2 Method and Experimental Setup

Region | Num. Msgs | % of Corpus
---|---|---
Train | 4712 | 70%
Test | 1355 | 20%
Validation | 704 | 10%
Table 1: The number of messages included for training, testing, and validation regions. Category | Coverage (%) | Model | Precision | Recall | F1
---|---|---|---|---|---
Communication/Coordination | 73.1 | NLI (1) |  $84.0$ ($\pm 0.6$) |  $87.1$ ($\pm 0.5$) |  $85.5$ ($\pm 0.2$)
Rapport Building | 15.2 | NLI (5) |  $83.0$ ($\pm 9.3$) |  $80.1$ ($\pm 15.7$) |  $81.3$ ($\pm 12.6$)
Control | 20.8 | NLI (5) |  $61.5$ ($\pm 1.7$) |  $59.5$ ($\pm 1.4$) |  $60.4$ ($\pm 0.2$)
Challenge | 4.5 | NLI (5) |  $36.1$ ($\pm 1.7$) |  $24.2$ ($\pm 4.1$) |  $28.9$ ($\pm 3.5$)
Negotiation | 20.9 | NLI (5) |  $62.9$ ($\pm 2.9$) |  $66.7$ ($\pm 1.7$) |  $64.7$ ($\pm 2.3$)
Use of Emotions | 16.4 | NLI (5) |  $58.2$ ($\pm 1.4$) |  $53.1$ ($\pm 1.65$) |  $55.6$ ($\pm 1.6$)
Testing Boundaries | 31.2 | NLI (5) |  $69.6$ ($\pm 0.8$) |  $76.1$ ($\pm 2.7$) |  $72.7$ ($\pm 0.9$)
Use of Sexual Topics | 18.3 | NLI (5) |  $64.5$ ($\pm 1.6$) |  $69.3$ ($\pm 3.6$) |  $66.8$ ($\pm 2.1$)
Mitigation | 3.0 | NLI (3) |  $62.2$ ($\pm 10.7$) |  $38.4$ ($\pm 7.0$) |  $47.5$ ($\pm 8.5$)
Encouragement | 8.0 | NLI (1) |  $43.2$ ($\pm 4.8$) |  $22.0$ ($\pm 1.6$) |  $29.1$ ($\pm 2.0$)
Risk Management | 4.6 | NLI (1) |  $63.1$ ($\pm 4.0$) |  $49.5$ ($\pm 4.1$) |  $55.4$ ($\pm 3.5$)
Table 2: Eleven offender behavior categories derived and labelled by domain-
experts, alongside their prevalence within the corpus. We report performance
for the best performing model (based on F1) for each category. NLI (5) denotes
the 5-message-input Natural Language Inference model, and similarly for NLI
(3) and NLI (1).

#### Dataset and Labelling.

The Perverted-Justice (n.d) website is an online repository of real, chat-
based conversations between adults who were later convicted of grooming
offences and decoys posing as children. Twenty-four chats, comprising of
12,942 messages, were labelled by a domain-expert with a background in
forensic psychology. Chats were annotated in concordance with a theory of
child grooming known as “self-regulation” (Elliott, 2017) – the notion that
online predation contains a potentiality phase, where the predator attempts to
form a positive relationship with the victim; and a disclosure phase, where
the predator becomes more explicitly goal-oriented. The following behaviors
were identified and applied to the offender messages: (1)
communication/coordination, (2) rapport building, (3) control, (4) challenges,
(5) negotiation, (6) use of emotions, (7) testing boundaries, (8) use of
sexual topics, (9) mitigation, (10) encouragement, (11) risk management. A
qualitative description of these behaviors is given in Section A of the
Appendix. Overall 6,772 messages sent by the offender were labelled, with each
message assigned a ‘Yes’ or ‘No’ for each of the above categories, based on
the annotator’s judgment. Due to the subjective nature of this assessment,
labelling performed by different expert annotators may not be in complete
agreement.

We construct binary classification tasks and predict whether each message is
an example of each behavior category. Offender messages were split into
training, testing, and validation regions (see Table 1 for information on data
splits), and were stratified to ensure equal distribution of behaviors per
region. To increase model confidence, we cross validated each experiment three
times through random re-sampling. In our results, we report the average ($\pm
SD$) scores for precision, recall and F1 as evaluation metrics.

#### Supervised Machine Learning.

Offender messages were tokenized, part-of-speech (POS) tagged, and lemmatized
with spaCy. We removed stop-words (i.e., the, a, am, at, be, is), and used the
frequency counts of the remaining unigrams as input features. Consistent with
previous work (Miah et al., 2011; Bogdanova et al., 2014), we compared the
performance of four classifiers: Random Forest, Logistic Regression, Support
Vector Machines, and Naive Bayes. For each algorithm, we optimized
hyperparameters111Hyperparameters and the range of values explored are
included in Section B of the Appendix. using the GridSearch class in Scikit-
learn (Pedregosa et al., 2011) with 3-fold cross validation.

#### Deep Learning.

To make predictions using transformer-based deep learning, we formulate the
problem as a natural language inference (NLI) task. In NLI, the objective is
to determine whether two sentences logically complement or contradict one
another (Bowman et al., 2015). For our purposes, we compare each message with
a standardized sentence crafted from each behavior label (i.e., “This message
is an example of building rapport”). Each message/label sentence pair is then
used as an input to the deep learning model, where a softmax activation
function calculates the probability the two sentences are logically related.
For training222We used a Tesla P100 GPU for training., we use RoBERTa-large
(Liu et al., 2019) with an implementation built in Pytorch. This model has
been fine-tuned for NLI tasks using the Multi-Genre Natural Language Inference
corpus (Williams et al., 2018). We ran both zero-shot and few-shot training
setups. For the zero-shot model, predictions were made on the test set without
any domain-specific training. In few-shot settings, we experiment with
different amounts of positive training examples up to the full training set.
As per (Wang et al., 2021), models were trained for 10 epochs with batch size
32 and a learning rate of $10^{-5}$.

#### Multi-Message Input.

To explore whether the surrounding messages increase the contextual
understanding of the transformer, we repeat full-shot experiments but expand
the message window to include multiple prior messages sent by both speakers.
To do this, we concatenate multiple messages into a single input which is then
passed to the transformer. We compare the performance of NLI models with an
input of 1, 3, and 5 messages.

#### Human-in-the-Loop Validation.

To establish how comparable the model’s output is to labelling performed by a
human annotator, we scored the predictions made by the best performing model
for $\sim 10\%$ of the corpus ($645$ predator messages). Each message was
scored on a 1-3 scale of agreement (1=disagree, 2=uncertain, 3=agree).
Sections of chats were chosen at random. To mitigate potential bias,
annotations generated during the initial labelling were shielded from view.

We use Cohen’s $\kappa$ (Cohen, 1960) to measure pairwise agreement between
the two human raters and the predictions generated by the best performing
model for each behavior. We report the mean of these scores as an index of
overall agreement.

##  3 Results

Transfomer-based deep learning models achieved best performance across all
categories, with the 5 message input NLI model performing best for most
categories. Evaluation metrics for the highest performing model for each
behavior are shown in Table 2. The performance of all models on all categories
is reported in Section E of the Appendix. Eight of the eleven behaviors obtain
a maximum F1 score above 50%, with the most prevalent category,
communication/coordination, performing best with F1=$86\%$. Automatically
labelling encouragement and challenge was least successful, with a maximum F1
score of $29\%$ for both. For all eleven categories, a transformer-based
model, trained on the full training set, was more successful than the
traditional supervised machine learning algorithms. Averaging across
behaviors, transformers improved performance of single message classification
from $33\%$ to $53\%$ compared to the best performing supervised model. In 8
out of 11 cases, the supervised models, trained on the full training set,
outperformed the zero-shot transformer. However, in 10 out of 11 cases, only
50 labelled instances were required by the transformer to exceed the best
performing bag-of-words models. As many of the behaviors rely on contextual
understanding, we increased the input size from a single message to 3 and 5
messages. The first contains two messages sent by the offender and one by the
decoy, and the latter contains three by the offender and two by the decoy.
Increasing the window size indeed improved performance for all behaviors333The
majority of messages ($73\%$) were labelled as communication/coordination. As
a result, this behavior was not tested in a multi-message setting as grouping
messages meant that all inputs would be positively labelled. except for
encouragement and risk management. On average, while 3-message classification
did not improve single-message performance, 5-message classification improved
on single-message F1 from $50\%$ to $56\%$.

Figure 1: Model performance (F1) on the eleven behavioral categories for
single message classification, as the number of positive training examples
increases (up to full training set). Error bars are standard deviations.

Figure 1 shows the change in F1 for single message classification as the
number of training samples increases. Whilst predictions for all behaviors
improved with additional training, large variation in performance between the
categories remains even for a fixed amount of positive training examples. To
better understand this variation, we manually examined the predictions made by
the best performing single message model for $\sim 10\%$ of the corpus. We
observed that the transformer has learned some rules that caused a number of
false positives. For example, the testing boundaries model correctly predicted
a number of questions aimed at assessing the victim’s willingness to engage,
e.g., "what are you looking for?", but also included several logistical
questions, e.g., "how about i come by at 8pm?". The rapport building model
correctly recognised complements and sweet talk as positive examples, but
missed more everyday examples of rapport building, e.g., "how was your spring
break?". Some aspects of control required coverage over longer ranges than we
have focused on. For example, persistently asking the same question is a sub-
category of control that was often misclassified. In this case, increasing the
the message window to 3 messages was not enough to improve performance,
however, the larger 5 message window did notably improve performance, with the
F1 increasing from $54\%$ to $60\%$ due to better recall. In trying to predict
encouragement, the worst performing category, the transformer appeared to
overfit on short verbal nods (i.e., "kool" and "sure", which appeared in the
corpus either as evidence of encouragement or simple linguistic fillers) which
increased the false positive rate. Over-reliance on this simple rule would
explain why the model did not utilise the additional information provided in
the multi-message setting. Risk management achieved reasonable performance
(F1=$55\%$) despite being one of the rarest behaviors in the corpus. This was
largely a consequence of the transformer recognising instances where the
offender attempts to establish the presence of a parent, e.g., “when are they
getting home?”.

In addition to drawing qualitative observations on the ability of the
transformer-based model to recognise different offender behaviors, we quantify
how comparable the best performing single-message model is to a human
annotator. We compare the labelled classes between the original labels (Rater
1), with the predictions made by the best performing models (Model), and the
labels verified during post-validation (Rater 2). We report a mean $\kappa$
score of 0.68, indicating ‘substantial agreement’ between raters (McHugh,
2012). Pairwise agreement is reported in Table 3. Agreement was highest
between the two human raters ($\kappa=0.8$). The average agreement between
each rater and the predictions generated by the model was $\kappa=0.62$ (see
Section D in the Appendix for pairwise agreement per behavior).

| Rater 1 | Rater 2 | Model
---|---|---|---
Rater 1 | - | 0.8 (0.78) | 0.59
Rater 2 |  | - | 0.65 (0.66)
Table 3: Pairwise agreement (Cohen’s $\kappa$) between raters over all
behaviors. A score of 1.0 indicates perfect agreement. Value in parentheses
indicate Rater 2 agreement when ‘uncertain’ validation ratings are included.

##  4 Conclusion

Manually labeling the 24 chat logs used in this work took over 600 hours.
Given that the full Perverted-Justice corpus contains 850 chat logs, it would
be infeasible to label the entire corpus without the help of automated
methods. We find that a transformer-based deep learning approach yields
promising results when applied to the detection of online predatory behavior.
However, even with training, the agreement between the model and a human
annotator is not comparable to the agreement between two human annotators. The
model’s success in predicting behaviors varies significantly between the
categories. Performance is better for more common behaviors, however, we show
that the variation is not only due to the number of positive examples in the
training data. Extending the prediction task to include multiple messages,
increasing the contextual window, boosted detection for certain behaviors. The
F1 score for rapport building, for example, increased from $61\%$ for single
messages to $81\%$ when classification was based on five-messages. Performing
post-validation on the automatic classifications allowed us to gain
qualitative insight about the model’s performance, which may be used to design
better prompts and improve performance further. Overall, our results are an
encouraging step towards building an automated, psychology-informed model to
detect online sexual exploitation.

## References

  * Greene-Colozzi et al. (2020) Emily A Greene-Colozzi, Georgia M Winters, Brandy Blasko, and Elizabeth L Jeglic.  Experiences and perceptions of online sexual solicitation and grooming of minors: a retrospective report.  _Journal of child sexual abuse_ , 29(7):836–854, 2020.  doi:10.1080/10538712.2020.1801938.
  * NSPCC (2021) NSPCC.  Record high number of recorded grooming crimes lead to calls for stronger online safety legislation, 2021.  URL https://www.nspcc.org.uk/about-us/news-opinion/2021/online-grooming-record-high.
  * Shelton et al. (2016) Joy Shelton, Jennifer Eakin, Tia Hoffer, Yvonne Muirhead, and Jessica Owens.  Online child sexual exploitation: An investigative analysis of offender characteristics and offending behavior.  _Aggression and violent behavior_ , 30:15–23, 2016.  doi:10.1016/j.avb.2016.07.002.
  * Briggs et al. (2011) Peter Briggs, Walter T Simon, and Stacy Simonsen.  An exploratory study of internet-initiated sexual offenses and the chat room sex offender: Has the internet enabled a new typology of sex offender?  _Sexual Abuse_ , 23(1):72–91, 2011.  doi:10.1177/1079063210384275.
  * Winters et al. (2017) Georgia M Winters, Leah E Kaylor, and Elizabeth L Jeglic.  Sexual offenders contacting children online: an examination of transcripts of sexual grooming.  _Journal of sexual aggression_ , 23(1):62–76, 2017.  doi:10.1080/13552600.2016.1271146.
  * O’Connell (2003) Rachel O’Connell.  A typology of child cybersexploitation and online grooming practices, 2003\.  URL http://image.guardian.co.uk/sys-files/Society/documents/2003/07/17/Groomingreport.pdf.
  * Williams et al. (2013) Rebecca Williams, Ian A Elliott, and Anthony R Beech.  Identifying sexual grooming themes used by internet sex offenders.  _Deviant Behavior_ , 34(2):135–152, 2013.  doi:10.1080/01639625.2012.707550.
  * Barber and Bettez (2021) Connie S Barber and Silvia Cristina Bettez.  Exposing patterns of adult solicitor behaviour: towards a theory of control within the cybersexual abuse of youth.  _European Journal of Information Systems_ , 30(6):591–622, 2021.  doi:10.1080/0960085x.2020.1816146.
  * Joleby et al. (2021) Malin Joleby, Carolina Lunde, Sara Landström, and Linda S Jonsson.  Offender strategies for engaging children in online sexual activity.  _Child Abuse & Neglect_, 120:105214, 2021.  doi:10.1016/j.chiabu.2021.105214.
  * Buckingham and Alali (2020) Louisa Buckingham and Nusiebah Alali.  Extreme parallels: a corpus-driven analysis of isis and far-right discourse.  _Kōtuitui: New Zealand Journal of Social Sciences Online_ , 15(2):310–331, 2020.  doi:10.1080/1177083x.2019.1698623.
  * Tausczik and Pennebaker (2010) Yla R Tausczik and James W Pennebaker.  The psychological meaning of words: Liwc and computerized text analysis methods.  _Journal of language and social psychology_ , 29(1):24–54, 2010.  doi:10.1177/0261927x09351676.
  * Bogdanova et al. (2014) Dasha Bogdanova, Paolo Rosso, and Thamar Solorio.  Exploring high-level features for detecting cyberpedophilia.  _Computer speech & language_, 28(1):108–120, 2014.  doi:10.1016/j.csl.2013.04.007.
  * Kaur et al. (2021) Simrat Kaur, Sarbjeet Singh, and Sakshi Kaushal.  Abusive content detection in online user-generated data: A survey.  _Procedia Computer Science_ , 189:274–281, 2021.  doi:10.1016/j.procs.2021.05.098.
  * Broome et al. (2020) Laura Jayne Broome, Cristina Izura, and Jason Davies.  A psycho-linguistic profile of online grooming conversations: A comparative study of prison and police staff considerations.  _Child Abuse & Neglect_, 109:104647, 2020.  doi:10.1016/j.chiabu.2020.104647.
  * Inches and Crestani (2012) Giacomo Inches and Fabio Crestani.  Overview of the international sexual predator identification competition at pan-2012.  In _CLEF (Online working notes/labs/workshop)_ , volume 30, 2012.
  * Pendar (2007) Nick Pendar.  Toward spotting the pedophile telling victim from predator in text chats.  In _International Conference on Semantic Computing (ICSC 2007)_ , pages 235–241. IEEE, 2007.  doi:10.1109/icsc.2007.32.
  * Miah et al. (2011) Md Waliur Rahman Miah, John Yearwood, and Sid Kulkarni.  Detection of child exploiting chats from a mixed chat dataset as a text classification task.  In _Proceedings of the Australasian Language Technology Association Workshop 2011_ , pages 157–165, 2011.
  * Ebrahimi et al. (2016) Mohammadreza Ebrahimi, Ching Y. Suen, and Olga Ormandjieva.  Detecting predatory conversations in social media by deep convolutional neural networks.  _Digital Investigation_ , 18:33–49, 2016.  ISSN 1742-2876.  doi:https://doi.org/10.1016/j.diin.2016.07.001.  URL https://www.sciencedirect.com/science/article/pii/S1742287616300731.
  * Gupta et al. (2012) Aditi Gupta, Ponnurangam Kumaraguru, and Ashish Sureka.  Characterizing pedophile conversations on the internet using online grooming.  _arXiv preprint arXiv:1208.4324_ , 2012.  doi:https://doi.org/10.48550/arXiv.1208.4324.
  * Razi et al. (2021) Afsaneh Razi, Seunghyun Kim, Ashwaq Alsoubai, Gianluca Stringhini, Thamar Solorio, Munmun De Choudhury, and Pamela J Wisniewski.  A human-centered systematic literature review of the computational approaches for online sexual risk detection.  _Proceedings of the ACM on Human-Computer Interaction_ , 5(CSCW2):1–38, 2021.  doi:10.1145/3479609.
  * Perverted-Justice (n.d) Perverted-Justice.  http://perverted-justice.com/, n.d.
  * Elliott (2017) Ian A Elliott.  A self-regulation model of sexual grooming.  _Trauma, Violence, & Abuse_, 18(1):83–97, 2017\.  doi:10.1177/1524838015591573.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  _Journal of machine learning research_ , 12(Oct):2825–2830, 2011.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_ , 2015.  doi:10.18653/v1/d15-1075.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.  doi:10.48550/arXiv.1907.11692.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics, 2018.  doi:10.18653/v1/n18-1101.  URL http://aclweb.org/anthology/N18-1101.
  * Wang et al. (2021) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma.  Entailment as few-shot learner.  _arXiv preprint arXiv:2104.14690_ , 2021.  doi:10.48550/arXiv.2104.14690.
  * Cohen (1960) Jacob Cohen.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46, 1960.  doi:10.1177/001316446002000104.
  * McHugh (2012) Mary L McHugh.  Interrater reliability: the kappa statistic.  _Biochemia medica_ , 22(3):276–282, 2012.  doi:10.11613/bm.2012.031.

##  Appendix A Content Label Descriptions

### Communication/Coordination

#### Description:

“Communication Coordination” is used to start and maintain communication as
offenders: (i) exchange and clarify information with their intended victim,
(ii) present reason/excuses, (iii) assess the level of engagement of the
victim, (iv) find new ways to communicate (i.e., media exchange), (v)
strategically use humour or linguistic fillers (i.e., “lol”, “hehe”), and (vi)
redirect the flow of conversation. One of the offenders’ main purposes of this
category is to maximize gain and potentially minimize time spent on non-
compliant victims.

### Rapport Building

#### Description:

Offenders use positive behavior to mimic romantic relationships, making it
easier to introduce sexual topics [Elliott, 2017]. Offenders use ‘Rapport’ to
infiltrate victims’ offline/online social and emotional life to create an
illusion of exclusivity, reinforcing the offender as a trusted other. This is
achieved through compliments/sweet talk, showing interest, and shared
experiences. This special connection or bond is usually created in a short
amount of time through excessive saturation and exposure to constant positive
statements [Elliott, 2017].

### Control/Regulation

#### Description:

Control/Regulation’ occurs when offenders use power to direct the flow of
communication by influencing or directing the victim’s behavior. Controlling
the conversation can occur through subtle (e.g., illusion of control,
rhetorical questions, checking for willingness to engage, or permissive
behavior) or direct strategies (e.g., making demands, persistence, use of
coercion). Offenders may attempt to take control of the conversation through
patronizing language, persistence, frequently checking for engagement, making
demands, or by asking questions that give the illusion of consent - giving the
impression that victims have control over what happens during an exchange.

### Challenges

#### Description:

An offender may challenge a victim when opposing motivations appear. As a
result, confrontation ensues directly (e.g., offence, control, aggression) or
indirectly (e.g., joke, mockery, irony). Offenders often challenge the victim
as a way of authenticating identity, or to exert more control.

### Negotiation

#### Description:

“Negotiation” can occur at any time during the exchange and is the process
where offenders attempt to make decisions, compromise, incentivize continued
interaction, or reach goal achievement (e.g., confirming a plan to meet).
“Negotiations” can be brief or extensive depending on what goals the offender
is trying to achieve. Incentives are particularly important when negotiating
goals, and can be either financial or emotional.

### Use of Emotion

#### Description:

Offenders use emotive language to manipulate the victim’s emotions in order to
influence their behavior. “Use of emotions” can be positive or negative, and
include sub-behaviors such as manipulation, expressing empathy, guilt
tripping, vilifying third parties, offering reassurance, or by playing the
victim. Offenders may employ positive strategies to isolate victims, and use
negative emotions increase compliance.

### Testing Boundaries

#### Description:

“Testing Boundaries” determines whether the conversation continues or ends.
Offenders seek to test boundaries directly or indirectly to determine whether
it is possible to desensitize victims through exposure to sexual topics
[Elliott, 2017].

### Use of Sexual Topics

#### Description:

Offenders intentionally use sex to desensitize victims. This is done by
directing conversation toward the victim’s prior sexual experiences,
discussing fantasies, use of explicit language, determining sexual
preferences, suggesting media production, alluding to traveling for sex, and
acting as a sexual mentor.

### Mitigation

#### Description:

“Mitigation” is a strategy that aims to soften or downgrade the intensity or
seriousness of what is being expressed to convince the victim to participate.
Offenders may use this technique in an attempt to normalize the sexual
exchange by lessening the idea of harm or criminality. Specific sub-behaviors
include indirectly stating a sexual preference for children, implicating
oneself in a previous crime, normalizing sexual conversations, or discussing
differences in age. Normalization occurs by talking about sex often without
reservation and is the process of desensitizing the victim to sexual topics or
acts.

### Encouraging

#### Description:

Offenders use encouragement to comply with the victims requests, or to show
support by acting as a mentor/trusted other.

### Risk Management

#### Description:

“Risk Management” occurs when offenders assess risk and take steps to prevent
discovery. This may be through incentivizing secrecy using emotional
manipulation, asking the decoy to delete messages/images, enquiring after
third parties (e.g., the location of parents), acknowledge previous
wrongdoing, and discussing the consequences of getting caught.

##  Appendix B Hyperparameter Tuning

Algorithm | Parameter | Value Range
---|---|---
Random Forest | n_estimators |  $50$, $100$*, $150$
max_depth |  $10$, $50$, $100$
Logistic Regression | solver | liblinear, saga
penalty |  l1, l2*
Support Vector Machine | C |  $0.1$, $0.5$, $1$*
kernel | linear, poly, rbf*
Naive Bayes | alpha |  $0.0$, $1.0$*
fit_prior | True*, False
Table 4: Range of settings evaluated for each supervised algorithm. As
GridSearch is an exhaustive optimization technique, effort was taken to
minimize the number of models tested. Algorithms were implemented via Scikit-
learn. * indicates the default setting.

##  Appendix C Precision, Recall, and Accuracy Plots for Full-Shot Single-
Classification Models

Figure 2: Precision, recall, and accuracy of single message classification as
the number of positive training examples increases (up to full training set).
Error bars are standard deviations.

##  Appendix D Inter-Rater Agreement Scores for Each Behavior

Behavior | Rater 1/Rater 2 | Rater 1/Model | Rater 2/Model | Overall
---|---|---|---|---
Communication/Coordination |  $0.66$ ($0.65$) | $0.53$ |  $\mathbf{0.71}$ ($0.72$) |  $0.63$ ($0.62$)
Rapport Building |  $\mathbf{0.73}$ ($0.7$) | $0.63$ |  $0.62$ ($0.65$) |  $0.66$ ($0.65$)
Control |  $\mathbf{0.62}$ ($0.6$) | $0.52$ |  $0.55$ ($0.59$) |  $0.56$ ($0.57$)
Challenge |  $\mathbf{0.89}$ ($0.89$) | $0.46$ |  $0.51$ ($0.51$) |  $0.62$ ($0.62$)
Negotiation |  $\mathbf{0.95}$ ($0.93$) | $0.67$ |  $0.65$ ($0.67$) |  $0.76$ ($0.76$)
Use of Emotions |  $\mathbf{0.78}$ ($0.74$) | $0.6$ |  $0.77$ ($0.79$) |  $0.72$ ($0.7$)
Testing Boundaries |  $\mathbf{0.88}$ ($0.84$) | $0.5$ |  $0.52$ ($0.53$) |  $0.63$ ($0.62$)
Use of Sexual Topics |  $\mathbf{0.74}$ ($0.73$) | $0.6$ |  $0.64$ ($0.65$) |  $0.66$ ($0.66$)
Mitigation |  $\mathbf{0.91}$ ($0.91$) | $0.52$ |  $0.52$ ($0.52$) |  $0.65$ ($0.65$)
Encouragement |  $0.66$ ($0.66$) | $0.61$ |  $\mathbf{0.84}$ ($0.84$) |  $0.7$ ($0.71$)
Risk Management |  $\mathbf{0.88}$ ($0.85$) | $0.67$ |  $0.67$ ($0.68$) |  $0.74$ ($0.72$)
Total |  $\mathbf{0.8}$ ($0.78$) | $0.59$ |  $0.65$ ($0.66$) |  $0.68$ ($0.67$)
Table 5: Pairwise Cohen’s $\kappa$ per behavior label. Overall score is the
mean of the pairwise scores. $1.0$ indicates perfect agreement; 0.0 indicates
no agreement. Values greater than $\kappa=0.6$ are generally indicative of
“moderate to strong” agreement, as per guidelines in [McHugh, 2012]. $\kappa$
in bold indicates largest pairwise agreement per behavior label. Rater 1 is
the actual labels used in training (i.e., ground-truth), Model refers to the
predictions generated by the best performing single-classification model for
each behavior, and Rater 2 refers to manual ratings generated during post-
validation. Values in parentheses indicate $\kappa$ when “unsure” ratings
given by Rater 2 are included as positive agreement.

##  Appendix E Full Model Results

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $73.8$ ($\pm 0.26$) |  $74.28$ ($\pm 0.11$) |  $98.18$ ($\pm 0.8$) |  $84.57$ ($\pm 0.23$)
Logistic Regression |  $73.73$ ($\pm 0.2$) |  $75.5$ ($\pm 0.11$) |  $94.85$ ($\pm 0.61$) |  $84.08$ ($\pm 0.18$)
Support Vector Machine |  $74.59$ ($\pm 0.62$) |  $75.41$ ($\pm 0.26$) |  $96.84$ ($\pm 0.61$) |  $84.79$ ($\pm 0.39$)
Naive Bayes |  $74.47$ ($\pm 1.35$) |  $76.5$ ($\pm 0.58$) |  $93.95$ ($\pm 1.31$) |  $84.33$ ($\pm 0.88$)
Transformer |  |  |  |
0-shot |  $61.7$ ($\pm 0.87$) |  $72.71$ ($\pm 0.46$) |  $76.25$ ($\pm 0.84$) |  $74.44$ ($\pm 0.64$)
25-shot |  $73.43$ ($\pm 1.01$) |  $81.91$ ($\pm 0.68$) |  $81.74$ ($\pm 0.24$) |  $81.81$ ($\pm 0.98$)
50-shot |  $75.45$ ($\pm 0.36$) |  $81.11$ ($\pm 0.28$) |  $86.61$ ($\pm 1.13$) |  $83.77$ ($\pm 0.38$)
100-shot |  $73.09$ ($\pm 0.69$) |  $82.59$ ($\pm 0.61$) |  $80.09$ ($\pm 0.85$) |  $81.32$($\pm 0.51$)
200-shot |  $74.15$ ($\pm.1.08$) |  $81.54$ ($\pm 1.2$) |  $83.65$ ($\pm 3.8$) |  $82.53$ ($\pm 1.28$)
300-shot |  $75.13$ ($\pm 0.84$) |  $82.25$ ($\pm 1.58$) |  $84.26$ ($\pm 3.54$) |  $83.19$ ($\pm 1.02$)
500-shot |  $75.38$ ($\pm 0.37$) |  $81.31$ ($\pm 0.58$) |  $86.14$ ($\pm 1.41$) |  $83.65$ ($\pm 0.41$)
1000-shot |  $75.15$ ($\pm 0.57$) |  $81.84$ ($\pm 0.44$) |  $84.86$ ($\pm 1.06$) |  $83.32$ ($\pm 0.47$)
Full-shot (3466) |  $78.4$ ($\pm 0.35$) |  $83.98$ ($\pm 0.55$) |  $87.08$ ($\pm 0.53$) |  $85.5$ ($\pm 0.21$)
Table 6: Performance of classification models for Communication/Coordination.
Full-shot value refers to the number of positive examples present in the full
training set. Best performing model is in bold.

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $87.09$ ($\pm 0.52$) |  $71.26$ ($\pm 6.61$) |  $25.57$ ($\pm 2.24$) |  $37.55$ ($\pm 2.51$)
Logistic Regression |  $87.16$ ($\pm 0.07$) |  $72.28$ ($\pm 3.17$) |  $25.41$ ($\pm 1.96$) |  $37.52$ ($\pm 1.67$)
Support Vector Machine |  $86.32$ ($\pm 0.31$) |  $69.8$ ($\pm 1.37$) |  $17.8$ ($\pm 4.07$) |  $28.18$ ($\pm 5.25$)
Naive Bayes |  $86.54$ ($\pm 0.19$) |  $72.17$ ($\pm 3.38$) |  $18.77$ ($\pm 0.28$) |  $29.78$ ($\pm 0.4$)
Transformer |  |  |  |
0-shot |  $42.07$ ($\pm 1.67$) |  $17.63$ ($\pm 0.73$) |  $76.54$ ($\pm 3.16$) |  $28.66$ ($\pm 1.15$)
25-shot |  $85.31$ ($\pm 6.27$) |  $53.34$ ($\pm 23.53$) |  $44.5$ ($\pm 17.21$) |  $48.45$ ($\pm 19.82$)
50-shot |  $84.63$ ($\pm 4.45$) |  $49.33$ ($\pm 14.86$) |  $49.03$ ($\pm 15.13$) |  $49.18$($\pm 15.0$)
100-shot |  $87.11$ ($\pm 2.32$) |  $58.15$ ($\pm 8.51$) |  $52.1$ ($\pm 11.53$) |  $54.86$($\pm 10.08$)
200-shot |  $87.33$ ($\pm 1.36$) |  $58.79$ ($\pm 4.96$) |  $56.63$ ($\pm 2.19$) |  $57.67$($\pm 3.53$)
300-shot |  $87.01$ ($\pm 0.71$) |  $57.36$ ($\pm 2.36$) |  $56.8$ ($\pm 2.7$) |  $57.07$($\pm 2.41$)
500-shot |  $87.82$ ($\pm 0.41$) |  $59.87$ ($\pm 1.59$) |  $60.52$ ($\pm 0.28$) |  $60.18$($\pm 0.8$)
1000-shot* |  |  |  |
Full-shot (721) |  $89.0$ ($\pm 0.45$) |  $66.08$ ($\pm 2.58$) |  $57.12$ ($\pm 1.22$) |  $61.24$ ($\pm 0.59$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $91.48$ ($\pm 6.31$) |  $80.47$ ($\pm 15.23$) |  $77.97$ ($\pm 17.94$) |  $79.16$ ($\pm 16.58$)
Full-shot 5 Msg |  $89.79$ ($\pm 6.34$) |  $82.95$ ($\pm 9.27$) |  $80.05$ ($\pm 15.71$) |  $81.33$ ($\pm 12.58$)

Table 7: Performance of classification models for Rapport Building. Full-shot
value refers to the number of positive examples present in the full training
set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $79.75$ ($\pm 0.54$) |  $53.84$ ($\pm 3.61$) |  $19.03$ ($\pm 1.34$) |  $28.12$ ($\pm 1.95$)
Logistic Regression |  $79.85$ ($\pm 1.0$) |  $56.08$ ($\pm 8.61$) |  $15.25$ ($\pm 2.16$) |  $23.96$ ($\pm 3.38$)
Support Vector Machine |  $80.17$ ($\pm 0.74$) |  $59.35$ ($\pm 6.21$) |  $14.89$ ($\pm 3.15$) |  $23.74$ ($\pm 4.37$)
Naive Bayes |  $79.41$ ($\pm 0.48$) |  $52.87$ ($\pm 5.14$) |  $11.35$ ($\pm 1.42$) |  $18.64$ ($\pm 2.08$)
Transformer |  |  |  |
0-shot |  $43.57$ ($\pm 0.86$) |  $17.24$ ($\pm 0.51$) |  $45.04$ ($\pm 1.23$) |  $24.94$ ($\pm 0.71$)
25-shot |  $80.64$ ($\pm 3.13$) |  $54.44$ ($\pm 12.75$) |  $33.81$ ($\pm 12.56$) |  $41.53$ ($\pm 13.31$)
50-shot |  $80.59$ ($\pm 2.11$) |  $54.32$ ($\pm 7.12$) |  $39.48$ ($\pm 10.13$) |  $45.5$ ($\pm 8.72$)
100-shot |  $80.98$ ($\pm 2.01$) |  $56.33$ ($\pm 6.7$) |  $42.55$ ($\pm 2.16$) |  $48.3$ ($\pm 2.32$)
200-shot |  $81.55$ ($\pm 0.7$) |  $57.82$ ($\pm 2.7$) |  $42.35$ ($\pm 2.49$) |  $48.89$ ($\pm 1.75$)
300-shot |  $82.36$ ($\pm 0.78$) |  $60.13$ ($\pm 2.38$) |  $45.15$ ($\pm 2.56$) |  $51.57$ ($\pm 2.52$)
500-shot |  $81.99$ ($\pm 1.34$) |  $59.11$ ($\pm 4.38$) |  $43.73$ ($\pm 3.22$) |  $50.27$ ($\pm 3.68$)
1000-shot* |  |  |  |
Full-shot (986) |  $82.36$ ($\pm 0.37$) |  $59.2$ ($\pm 1.0$) |  $49.05$ ($\pm 1.43$) |  $53.65$ ($\pm 1.2$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $76.53$ ($\pm 0.33$) |  $59.12$ ($\pm 1.1$) |  $41.52$ ($\pm 0.67$) |  $48.78$ ($\pm 0.48$)
Full-shot 5 Msg |  $71.32$ ($\pm 0.85$) |  $61.47$ ($\pm 1.74$) |  $59.47$ ($\pm 1.39$) |  $60.43$ ($\pm 0.15$)

Table 8: Performance of classification models for Control. Full-shot value
refers to the number of positive examples present in the full training set.
Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $95.43$ ($\pm 0.01$) |  $20.0$ ($\pm 20.0$) |  $1.67$ ($\pm 1.67$) |  $3.08$ ($\pm 3.08$)
Logistic Regression |  $95.42$ ($\pm 0.01$) |  $13.33$ ($\pm 23.1$) |  $1.11$ ($\pm 1.93$) |  $2.05$ ($\pm 3.55$)
Support Vector Machine |  $95.55$ ($\pm 0.01$) |  $0.0$ ($\pm 0.0$) |  $0.0$ ($\pm 0.0$) |  $0.0$ ($\pm 0.0$)
Naive Bayes |  $95.55$ ($\pm 0.01$) |  $50.5$ ($\pm 50.5$) |  $1.11$ ($\pm 0.96$) |  $2.17$ ($\pm 1.88$)
Transformer |  |  |  |
0-shot |  $14.29$ ($\pm 0.54$) |  $4.52$ ($\pm 0.09$) |  $91.11$ ($\pm 2.55$) |  $8.6$ ($\pm 0.18$)
25-shot |  $94.88$ ($\pm 0.57$) |  $35.56$ ($\pm 10.81$) |  $17.78$ ($\pm 3.85$) |  $23.66$ ($\pm 5.78$)
50-shot |  $95.3$ ($\pm 0.35$) |  $40.25$ ($\pm 12.39$) |  $12.78$ ($\pm 4.19$) |  $19.39$ ($\pm 6.25$)
100-shot |  $94.54$ ($\pm 1.15$) |  $27.73$ ($\pm 19.01$) |  $10.0$ ($\pm 2.36$) |  $14.35$ ($\pm 5.41$)
200-shot |  $95.2$ ($\pm 0.52$) |  $41.07$ ($\pm 12.63$) |  $17.5$ ($\pm 3.54$) |  $24.51$ ($\pm 5.74$)
300-shot* |  |  |  |
500-shot* |  |  |  |
1000-shot* |  |  |  |
Full-shot (211) |  $95.25$ ($\pm 0.23$) |  $41.04$ ($\pm 0.62$) |  $16.67$ ($\pm 2.89$) |  $23.7$ ($\pm 3.93$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $91.96$ ($\pm 0.24$) |  $15.53$ ($\pm 1.79$) |  $5.36$ ($\pm 1.79$) |  $17.97$ ($\pm 2.67$)
Full-shot 5 Msg |  $88.31$ ($\pm 1.12$) |  $36.14$ ($\pm 1.68$) |  $24.18$ ($\pm 4.08$) |  $28.9$ ($\pm 3.51$)

Table 9: Performance of classification models for Challenge. Full-shot value
refers to the number of positive examples present in the full training set.
Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $80.57$ ($\pm 1.08$) |  $59.57$ ($\pm 6.74$) |  $21.2$ ($\pm 4.95$) |  $31.14$ ($\pm 6.18$)
Logistic Regression |  $81.45$ ($\pm 1.08$) |  $67.55$ ($\pm 9.62$) |  $22.5$ ($\pm 1.14$) |  $33.65$ ($\pm 1.76$)
Support Vector Machine |  $80.57$ ($\pm 0.66$) |  $65.05$ ($\pm 7.93$) |  $13.43$ ($\pm 3.08$) |  $22.3$ ($\pm 4.46$)
Naive Bayes |  $81.35$ ($\pm 0.7$) |  $66.58$ ($\pm 7.39$) |  $22.5$ ($\pm 3.19$) |  $33.42$ ($\pm 3.04$)
Transformer |  |  |  |
0-shot |  $43.1$ ($\pm 1.47$) |  $21.92$ ($\pm 0.62$) |  $67.26$ ($\pm 2.07$) |  $33.06$ ($\pm 0.86$)
25-shot |  $77.91$ ($\pm 1.9$) |  $47.73$ ($\pm 5.8$) |  $38.28$ ($\pm 10.45$) |  $41.49$ ($\pm 4.78$)
50-shot |  $79.93$ ($\pm 2.11$) |  $51.87$ ($\pm 5.23$) |  $50.65$ ($\pm 7.21$) |  $51.22$ ($\pm 6.14$)
100-shot |  $80.47$ ($\pm 1.65$) |  $53.54$ ($\pm 4.26$) |  $49.0$ ($\pm 4.45$) |  $51.15$ ($\pm 4.22$)
200-shot |  $81.33$ ($\pm 0.27$) |  $56.21$ ($\pm 0.36$) |  $47.94$ ($\pm 4.54$) |  $51.67$ ($\pm 2.76$)
300-shot |  $80.69$ ($\pm 1.01$) |  $54.27$ ($\pm 2.81$) |  $48.29$ ($\pm 1.78$) |  $51.1$ ($\pm 2.16$)
500-shot |  $81.3$ ($\pm 0.52$) |  $55.83$ ($\pm 1.58$) |  $50.53$ ($\pm 0.71$) |  $53.03$ ($\pm 0.55$)
1000-shot* |  |  |  |
Full-shot (991) |  $83.22$ ($\pm 0.15$) |  $61.8$ ($\pm 0.92$) |  $51.59$ ($\pm 1.27$) |  $56.22$ ($\pm 0.39$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $80.43$ ($\pm 1.13$) |  $63.31$ ($\pm 2.41$) |  $61.99$ ($\pm 1.41$) |  $62.64$ ($\pm 1.84$)
Full-shot 5 Msg |  $74.74$ ($\pm 1.96$) |  $62.85$ ($\pm 2.93$) |  $66.67$ ($\pm 1.71$) |  $64.7$ ($\pm 2.31$)

Table 10: Performance of classification models for Negotiation. Full-shot
value refers to the number of positive examples present in the full training
set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $84.82$ ($\pm 0.69$) |  $66.26$ ($\pm 8.67$) |  $14.87$ ($\pm 2.95$) |  $24.25$ ($\pm 4.4$)
Logistic Regression |  $84.38$ ($\pm 0.53$) |  $57.47$ ($\pm 0.52$) |  $17.71$ ($\pm 2.27$) |  $27.07$ ($\pm 3.17$)
Support Vector Machine |  $84.4$ ($\pm 0.55$) |  $68.5$ ($\pm 11.75$) |  $8.56$ ($\pm 2.51$) |  $15.2$ ($\pm 4.27$)
Naive Bayes |  $84.77$ ($\pm 0.31$) |  $65.59$ ($\pm 4.21$) |  $14.87$ ($\pm 0.9$) |  $24.24$ ($\pm 1.47$)
Transformer |  |  |  |
0-shot |  $26.83$ ($\pm 1.4$) |  $16.77$ ($\pm 0.55$) |  $87.34$ ($\pm 2.39$) |  $28.13$ ($\pm 0.89$)
25-shot |  $81.28$ ($\pm 0.81$) |  $41.86$ ($\pm 1.31$) |  $35.89$ ($\pm 5.63$) |  $38.42$ ($\pm 2.82$)
50-shot |  $85.17$ ($\pm 1.76$) |  $56.69$ ($\pm 7.54$) |  $45.2$ ($\pm 3.62$) |  $50.02$ ($\pm 2.98$)
100-shot |  $85.02$ ($\pm 1.11$) |  $54.73$ ($\pm 4.03$) |  $50.0$ ($\pm 6.76$) |  $52.12$ ($\pm 4.62$)
200-shot |  $84.65$ ($\pm 0.38$) |  $53.54$ ($\pm 1.24$) |  $47.45$ ($\pm 2.56$) |  $50.3$ ($\pm 1.97$)
300-shot |  $85.22$ ($\pm 0.37$) |  $55.41$ ($\pm 0.94$) |  $49.7$ ($\pm 3.83$) |  $52.37$ ($\pm 2.5$)
500-shot |  $85.09$ ($\pm 0.2$) |  $55.02$ ($\pm 0.47$) |  $49.25$ ($\pm 2.31$) |  $51.96$ ($\pm 1.49$)
1000-shot* |  |  |  |
Full-shot (777) |  $85.27$ ($\pm 0.26$) |  $55.02$ ($\pm 0.91$) |  $55.26$ ($\pm 0.52$) |  $55.13$ ($\pm 0.28$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $78.89$ ($\pm 1.03$) |  $52.16$ ($\pm 2.72$) |  $44.62$ ($\pm 2.61$) |  $48.09$ ($\pm 2.67$)
Full-shot 5 Msg |  $73.64$ ($\pm 0.85$) |  $58.21$ ($\pm 1.44$) |  $53.13$ ($\pm 1.65$) |  $55.55$ ($\pm 1.56$)

Table 11: Performance of classification models for Use of Emotions. Full-shot
value refers to the number of positive examples present in the full training
set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $72.87$ ($\pm 1.39$) |  $63.03$ ($\pm 4.09$) |  $31.52$ ($\pm 2.82$) |  $42.02$ ($\pm 3.41$)
Logistic Regression |  $72.96$ ($\pm 1.32$) |  $62.47$ ($\pm 4.2$) |  $33.73$ ($\pm 1.66$) |  $43.79$ ($\pm 2.31$)
Support Vector Machine |  $73.31$ ($\pm 0.75$) |  $63.39$ ($\pm 2.71$) |  $34.44$ ($\pm 0.55$) |  $44.62$ ($\pm 1.0$)
Naive Bayes |  $73.65$ ($\pm 0.7$) |  $63.34$ ($\pm 2.22$) |  $37.12$ ($\pm 0.47$) |  $46.8$ ($\pm 0.95$)
Transformer |  |  |  |
0-shot |  $32.77$ ($\pm 0.85$) |  $30.22$ ($\pm 0.39$) |  $88.1$ ($\pm 0.83$) |  $45.0$ ($\pm 0.54$)
25-shot |  $71.93$ ($\pm 4.84$) |  $55.22$ ($\pm 8.07$) |  $53.03$ ($\pm 8.18$) |  $54.1$ ($\pm 8.1$)
50-shot |  $72.01$ ($\pm 2.87$) |  $55.05$ ($\pm 4.66$) |  $56.82$ ($\pm 4.29$) |  $55.91$ ($\pm 4.39$)
100-shot |  $72.92$ ($\pm 2.61$) |  $57.88$ ($\pm 5.58$) |  $51.3$ ($\pm 3.34$) |  $54.22$ ($\pm 2.34$)
200-shot |  $73.78$ ($\pm 0.47$) |  $58.9$ ($\pm 0.66$) |  $52.88$ ($\pm 1.9$) |  $55.72$ ($\pm 1.29$)
300-shot |  $74.44$ ($\pm 2.31$) |  $59.78$ ($\pm 4.19$) |  $56.03$ ($\pm 3.58$) |  $57.79$ ($\pm 3.28$)
500-shot |  $75.15$ ($\pm 0.76$) |  $61.1$ ($\pm 1.56$) |  $56.27$ ($\pm 0.24$) |  $58.58$ ($\pm 0.81$)
1000-shot |  $75.03$ ($\pm 0.33$) |  $61.01$ ($\pm 0.61$) |  $55.48$ ($\pm 1.9$) |  $58.11$ ($\pm 0.74$)
Full-shot (1479) |  $77.74$ ($\pm 0.79$) |  $65.17$ ($\pm 2.1$) |  $61.86$ ($\pm 1.43$) |  $63.44$ ($\pm 0.34$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $68.76$ ($\pm 0.52$) |  $61.53$ ($\pm 1.29$) |  $59.94$ ($\pm 2.02$) |  $60.69$ ($\pm 0.41$)
Full-shot 5 Msg |  $69.77$ ($\pm 0.51$) |  $69.63$ ($\pm 0.77$) |  $76.07$ ($\pm 2.65$) |  $72.68$ ($\pm 0.92$)
Table 12: Performance of classification models for Testing Boundaries. Full-
shot value refers to the number of positive examples present in the full
training set. Best performing model is in bold.

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $84.63$ ($\pm 0.81$) |  $71.73$ ($\pm 4.3$) |  $26.34$ ($\pm 4.31$) |  $38.43$ ($\pm 5.12$)
Logistic Regression |  $85.22$ ($\pm 0.73$) |  $74.59$ ($\pm 2.49$) |  $29.17$ ($\pm 5.26$) |  $41.76$ ($\pm 5.52$)
Support Vector Machine |  $84.38$ ($\pm 0.38$) |  $81.49$ ($\pm 1.64$) |  $18.95$ ($\pm 2.42$) |  $30.7$ ($\pm 3.23$)
Naive Bayes |  $85.12$ ($\pm 0.63$) |  $75.04$ ($\pm 4.48$) |  $28.09$ ($\pm 3.03$) |  $40.81$ ($\pm 3.45$)
Transformer |  |  |  |
0-shot |  $38.23$ ($\pm 0.52$) |  $19.58$ ($\pm 0.73$) |  $76.48$ ($\pm 3.88$) |  $31.18$ ($\pm 1.24$)
25-shot |  $75.57$ ($\pm 2.1$) |  $35.92$ ($\pm 3.11$) |  $41.53$ ($\pm 2.02$) |  $38.4$ ($\pm 0.9$)
50-shot |  $84.85$ ($\pm 2.09$) |  $60.61$ ($\pm 7.39$) |  $50.94$ ($\pm 3.75$) |  $55.25$ ($\pm 4.59$)
100-shot |  $85.76$ ($\pm 1.02$) |  $63.77$ ($\pm 3.86$) |  $51.62$ ($\pm 1.61$) |  $57.04$ ($\pm 2.43$)
200-shot |  $85.66$ ($\pm 0.65$) |  $63.72$ ($\pm 3.64$) |  $51.08$ ($\pm 2.83$) |  $56.58$ ($\pm 0.33$)
300-shot |  $86.0$ ($\pm 0.9$) |  $64.97$ ($\pm 3.56$) |  $51.21$ ($\pm 2.82$) |  $57.25$ ($\pm 2.66$)
500-shot |  $86.4$ ($\pm 0.42$) |  $65.17$ ($\pm 1.03$) |  $55.11$ ($\pm 2.46$) |  $59.71$ ($\pm 1.78$)
1000-shot* |  |  |  |
Full-shot (867) |  $85.9$ ($\pm 0.15$) |  $61.86$ ($\pm 0.33$) |  $55.48$ ($\pm 1.3$) |  $60.88$ ($\pm 0.73$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $84.11$ ($\pm 0.51$) |  $65.83$ ($\pm 1.0$) |  $62.59$ ($\pm 1.79$) |  $64.16$ ($\pm 1.37$)
Full-shot 5 Msg |  $79.46$ ($\pm 1.08$) |  $64.53$ ($\pm 1.63$) |  $69.26$ ($\pm 3.58$) |  $66.78$ ($\pm 2.11$)

Table 13: Performance of classification models for Use of Sexual Topics. Full-
shot value refers to the number of positive examples present in the full
training set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $96.97$ ($\pm 0.2$) |  $44.44$ ($\pm 38.49$) |  $4.88$ ($\pm 4.88$) |  $8.7$ ($\pm 8.5$)
Logistic Regression |  $96.93$ ($\pm 0.19$) |  $42.05$ ($\pm 20.31$) |  $10.57$ ($\pm 7.04$) |  $16.7$ ($\pm 10.72$)
Support Vector Machine |  $96.97$ ($\pm 0.01$) |  $0.0$ ($\pm 0.0$) |  $0.0$ ($\pm 0.0$) |  $0.0$ ($\pm 0.0$)
Naive Bayes |  $97.02$ ($\pm 0.04$) |  $58.89$ ($\pm 8.39$) |  $4.87$ ($\pm 2.44$) |  $8.93$ ($\pm 4.2$)
Transformer |  |  |  |
0-shot |  $44.11$ ($\pm 0.89$) |  $3.22$ ($\pm 0.23$) |  $60.16$ ($\pm 0.43$) |  $6.12$ ($\pm 0.43$)
25-shot |  $96.73$ ($\pm 0.33$) |  $40.3$ ($\pm 12.74$) |  $13.01$ ($\pm 2.82$) |  $19.38$ ($\pm 3.86$)
50-shot |  $97.34$ ($\pm 0.39$) |  $66.52$ ($\pm 17.64$) |  $23.58$ ($\pm 8.57$) |  $34.7$ ($\pm 11.48$)
100-shot |  $97.32$ ($\pm 0.01$) |  $64.08$ ($\pm 2.95$) |  $26.83$ ($\pm 7.32$) |  $37.28$ ($\pm 6.8$)
200-shot* |  |  |  |
300-shot* |  |  |  |
500-shot* |  |  |  |
1000-shot* |  |  |  |
Full-shot (144) |  $96.78$ ($\pm 0.15$) |  $44.9$ ($\pm 4.34$) |  $30.08$ ($\pm 5.08$) |  $36.0$ ($\pm 5.06$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $96.75$ ($\pm 0.51$) |  $62.22$ ($\pm 10.72$) |  $38.39$ ($\pm 7.0$) |  $47.47$ ($\pm 8.45$)
Full-shot 5 Msg |  $94.83$ ($\pm 0.11$) |  $50.88$ ($\pm 1.52$) |  $38.27$ ($\pm 2.14$) |  $43.64$ ($\pm 1.18$)

Table 14: Performance of classification models for Mitigation/Minimization.
Full-shot value refers to the number of positive examples present in the full
training set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $91.51$ ($\pm 0.27$) |  $36.6$ ($\pm 6.36$) |  $7.03$ ($\pm 1.91$) |  $11.67$ ($\pm 2.59$)
Logistic Regression |  $91.61$ ($\pm 0.3$) |  $30.51$ ($\pm 15.39$) |  $3.36$ ($\pm 1.91$) |  $6.05$ ($\pm 3.39$)
Support Vector Machine |  $91.88$ ($\pm 0.01$) |  $42.11$ ($\pm 11.07$) |  $3.67$ ($\pm 2.43$) |  $6.67$ ($\pm 4.32$)
Naive Bayes |  $91.88$ ($\pm 0.07$) |  $27.78$ ($\pm 25.46$) |  $0.61$ ($\pm 0.53$) |  $1.2$ ($\pm 1.0$)
Transformer |  |  |  |
0-shot |  $61.62$ ($\pm 0.9$) |  $11.49$ ($\pm 1.26$) |  $56.27$ ($\pm 6.44$) |  $19.08$ ($\pm 2.1$)
25-shot |  $88.98$ ($\pm 0.37$) |  $18.36$ ($\pm 3.51$) |  $10.7$ ($\pm 1.91$) |  $13.52$ ($\pm 2.48$)
50-shot |  $90.65$ ($\pm 0.87$) |  $37.07$ ($\pm 7.77$) |  $21.1$ ($\pm 0.92$) |  $26.72$ ($\pm 2.11$)
100-shot |  $89.91$ ($\pm 0.6$) |  $30.33$ ($\pm 4.72$) |  $19.88$ ($\pm 1.84$) |  $23.55$ ($\pm 2.79$)
200-shot |  $91.61$ ($\pm 0.04$) |  $45.13$ ($\pm 0.7$) |  $19.88$ ($\pm 1.4$) |  $27.58$ ($\pm 1.44$)
300-shot |  $91.37$ ($\pm 0.46$) |  $43.16$ ($\pm 4.83$) |  $22.02$ ($\pm 1.59$) |  $29.1$ ($\pm 2.04$)
500-shot* |  |  |  |
1000-shot* |  |  |  |
Full-shot (380) |  $91.14$ ($\pm 0.22$) |  $39.88$ ($\pm 2.78$) |  $19.88$ ($\pm 1.4$) |  $26.53$ ($\pm 1.89$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $86.43$ ($\pm 0.2$) |  $33.99$ ($\pm 1.09$) |  $18.0$ ($\pm 0.01$) |  $23.53$ ($\pm 0.27$)
Full-shot 5 Msg |  $78.75$ ($\pm 1.4$) |  $33.08$ ($\pm 4.8$) |  $22.47$ ($\pm 3.89$) |  $26.69$ ($\pm 4.02$)

Table 15: Performance of classification models for Encouragement. Full-shot
value refers to the number of positive examples present in the full training
set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot

Model | Accuracy | Precision | Recall | F1
---|---|---|---|---
Supervised - Bag of Words |  |  |  |
Random Forest |  $95.75$ ($\pm 0.35$) |  $63.83$ ($\pm 17.9$) |  $22.04$ ($\pm 6.52$) |  $31.79$ ($\pm 6.45$)
Logistic Regression |  $95.7$ ($\pm 0.3$) |  $62.86$ ($\pm 14.5$) |  $18.28$ ($\pm 4.06$) |  $27.81$ ($\pm 3.99$)
Support Vector Machine |  $95.47$ ($\pm 0.01$) |  $54.37$ ($\pm 25.54$) |  $4.3$ ($\pm 3.36$) |  $7.89$ ($\pm 5.93$)
Naive Bayes |  $95.57$ ($\pm 0.13$) |  $67.14$ ($\pm 15.45$) |  $6.45$ ($\pm 1.61$) |  $11.75$ ($\pm 2.84$)
Transformer |  |  |  |
0-shot |  $48.61$ ($\pm 0.52$) |  $4.75$ ($\pm 0.35$) |  $53.76$ ($\pm 4.06$) |  $8.74$ ($\pm 0.65$)
25-shot |  $95.25$ ($\pm 0.09$) |  $44.74$ ($\pm 2.35$) |  $15.59$ ($\pm 1.86$) |  $23.07$ ($\pm 10.94$)
50-shot |  $95.84$ ($\pm 0.57$) |  $56.11$ ($\pm 9.48$) |  $38.17$ ($\pm 10.74$) |  $45.27$ ($\pm 10.65$)
100-shot |  $96.11$ ($\pm 0.37$) |  $63.04$ ($\pm 8.13$) |  $37.1$ ($\pm 6.45$) |  $46.49$ ($\pm 6.06$)
200-shot |  $96.36$ ($\pm 0.26$) |  $63.08$ ($\pm 3.97$) |  $49.46$ ($\pm 4.06$) |  $55.39$ ($\pm 3.48$)
300-shot* |  |  |  |
500-shot* |  |  |  |
1000-shot* |  |  |  |
Full-shot (218) |  $96.29$ ($\pm 0.09$) |  $65.85$ ($\pm 2.39$) |  $39.25$ ($\pm 0.93$) |  $49.16$ ($\pm 0.58$)
Transformer - Multi-Message Input |  |  |  |
Full-shot 3 Msg |  $94.47$ ($\pm 0.44$) |  $61.83$ ($\pm 6.42$) |  $29.63$ ($\pm 6.42$) |  $39.98$ ($\pm 7.28$)
Full-shot 5 Msg |  $92.64$ ($\pm 0.39$) |  $65.48$ ($\pm 3.21$) |  $47.62$ ($\pm 1.18$) |  $55.13$ ($\pm 1.86$)

Table 16: Performance of classification models for Risk Management. Full-shot
value refers to the number of positive examples present in the full training
set. Best performing model is in bold.

* The number exceeds the total number of positive examples in the dataset and therefore results are analogous to full-shot