# Interpretable Explainability in Facial Emotion Recognition and Gamification
for Data Collection

Krist Shingjergji  Educational Sciences
Open University of the Netherlands
Heerlen, The Netherlands
<EMAIL_ADDRESS>Deniz Iren  Center for Actionable Research
Open University of the Netherlands
Heerlen, The Netherlands
<EMAIL_ADDRESS>Felix Böttger  Center for Actionable Research
Open University of the Netherlands
Heerlen, The Netherlands
Corrie Urlings  Educational Sciences
Open University of the Netherlands
Heerlen, The Netherlands
Roland Klemke  Educational Sciences
Open University of the Netherlands
Heerlen, The Netherlands

###### Abstract

Training facial emotion recognition models requires large sets of data and
costly annotation processes. To alleviate this problem, we developed a
gamified method of acquiring annotated facial emotion data without an explicit
labeling effort by humans. The game, which we named Facegame, challenges the
players to imitate a displayed image of a face that portrays a particular
basic emotion. Every round played by the player creates new data that consists
of a set of facial features and landmarks, already annotated with the emotion
label of the target facial expression. Such an approach effectively creates a
robust, sustainable, and continuous machine learning training process. We
evaluated Facegame with an experiment that revealed several contributions to
the field of affective computing. First, the gamified data collection approach
allowed us to access a rich variation of facial expressions of each basic
emotion due to the natural variations in the players’ facial expressions and
their expressive abilities. We report improved accuracy when the collected
data were used to enrich well-known in-the-wild facial emotion datasets and
consecutively used for training facial emotion recognition models. Second, the
natural language prescription method used by the Facegame constitutes a novel
approach for interpretable explainability that can be applied to any facial
emotion recognition model. Finally, we observed significant improvements in
the facial emotion perception and expression skills of the players through
repeated game play.

###### Index Terms:

Affective computing, facial emotion recognition, gamification, explainable AI,
interpretable machine learning

##  I Introduction

Facial expressions are imperative to non-verbal human communication as they
provide a means of conveying information regarding the emotional state [1] as
well as the behavioral intentions [2] of the individual. Emotions are
fundamental components of social interaction [3], and the ability to express
and perceive emotions is an invaluable asset for building social connections.
The holy grail of affective computing is to empower computer systems with the
ability to perceive and express emotions, and be able to form social ties with
human users [4]. Until very recently, this ability has been considered unique
to humans. However, especially with the recent advances in Artificial
Intelligence (AI), many studies have been conducted that focus on the
automated recognition of emotions [5].

The common approach of training machine learning models for facial emotion
recognition (FER) is supervised learning, which requires large sets of data
[6]. Specifically, deep FER models are challenged by a lack of sufficient data
for training [7]. Collecting and curating such large datasets is a costly and
time-consuming endeavour since labeling by human annotators is necessary [8].
This poses an obstacle to achieving significant performance improvements in
emotion recognition research.

Another major challenge lies in the explainability and interpretability of
emotion recognition models. Studies mostly evaluate emotion recognition models
using accuracy and confusion matrices; however, these metrics often fall short
in reporting the utility of the models for humans. Interpretable models should
provide explanations that are simple enough to be understood by their users,
and are given in a language that is meaningful to them [9]. The explainability
of emotion recognition models has been very rarely addressed in literature.
The approaches to achieve explainability are limited to model-agnostic methods
that explain the output of the model based on the inputs, and model-
transparent methods (e.g., [10], [11]) that highlight the activation in
different layers of artificial neural networks [12]. However, neither approach
necessarily provides human-friendly explanations that are interpretable by
their users.

The challenges regarding collecting and curating excessive amounts of labeled
data for training FER models, and yielding interpretable explanations from
such models call for heterodox methods. In this study, we propose a
gamification approach towards the collection of annotated facial emotion data.
The proposed game, which is named; Facegame, embodies a method for providing
natural language prescription as feedback to the players, effectively serving
as a means of achieving interpretable explainability. In summary, our
contributions to the field of affective computing are as follows:

  * •

We present a gamification approach for rapidly collecting annotated facial
emotion data that is rich in variety of facial expressions, in a low-cost,
low-effort manner.

  * •

We propose a novel approach for interpretable explanability by translating the
intermediary facial features into natural language prescriptions, and
providing them as an explanation for the emotion classifications provided by
any FER model.

  * •

The presented gamification approach leads to significant improvements in the
facial emotion perception and expression skills of the players.

The remaining of this paper is structured as follows. Section II provides a
literature review on emotion recognition, explainable AI, and gamified data
collection. Section III describes the core contributions. Section IV presents
the details of our experimental study. Section V discloses the results of our
experiments. Finally, Section VI provides a discussion on the theoretical and
practical implications of our contributions, and concludes the paper.

##  II Related Work

###  II-A Face expressions, action units, and their automated recognition

Facial expressions are a means for humans to express their emotions; thus,
their automated detection is an important goal of affective computing. Facial
expressions are movements and positions of the facial muscles that can be
identified by Action Units (AUs); hierarchical components of movements of
individual or groups of facial muscles that describe the changes in facial
expressions [13]. There are studies focusing on the correlation between AUs
and the basic emotions, namely, happiness, sadness, fear, disgust, anger, and
surprise [14]. For example, Reisenzein et al. [15] reported coherence between
amusement and smiling, and Wegrzyn et al. [16] presented a detailed mapping
between the basic emotions and different parts of the face, e.g., lid raiser
is essential for fear detection and lid tightener for anger. Apart from the
basic emotions, there are studies aiming at detecting more complex emotional
states, such as confusion, by utilizing AUs [17].

The strong relationship between emotional facial expressions and AUs has
motivated researchers to develop AU-detection algorithms as well as curating
AU-labeled face expression datasets such as CK+ [18] and DISFA [19]. For
instance, Baltrušaitis et al. [20] presented an AU occurrence and intensity
algorithm based on Histogram of Oriented Gradients (HOG); a method to describe
an image by the distribution of intensity gradients or edge direction [21]),
and geometrical features (e.g., shape and landmarks; detection and
localization of certain characteristic points on the face [22]). This work
also highlights the positive impact of using various datasets to the
generalizability of the model performance. Shao et al. [23] presented a
framework for detecting 10 AUs using the attention mechanism, i.e., finding
the region-of-interest for each AU. Jacob and Stenger [24] outperformed their
previous model by employing a correlation network based on a transformer-
encoder architecture, to capture the relationship between different AUs for a
wide range of expressions of emotions. Other prominent examples of
architectures for AU detection are the JAA-Net [25], which uses high-level
features of face alignment for AU detection, and DRML [26] which uses feed-
forward functions to induce regions on the face that are important.

###  II-B Explainable and Interpretable AI

As AI finds widespread application across many domains, the need for
explainable AI (XAI) is rapidly growing as well. However, most explainability
approaches do not target end-users, and their outcomes are not directly
interpretable by humans. One way to address this issue is to improve the
transparency of AI models. Model transparency focuses on explaining “how the
system made a decision” [27]. There are models that are transparent by design,
e.g., decision trees, and others that are “black box” and require additional
tools for explainability [28]. In recent years, explanation tools have been
designed to provide users insights on the decision-making process of a system.
The study of Jeyakumar [12] showed that the users prefer explanations by
example in most of the cases. Rosenfeld [29] presents a set of metrics that
are suitable for evaluating the effectiveness of explainable AI, namely, i)
the difference between the explanations’ logic and the agent’s actual
performance, ii) the number of rules in the agent’s explanation, iii) the
number of features used to construct the explanation, and iv) the stability of
the agent’s explanation.

###  II-C Crowdsourcing and Gamification for Data Collection

Crowdsourcing is defined as the act of outsourcing a task that is commonly
performed by designated agents to a large number of individuals [30].
Crowdsourcing has been used both by industry and scientific communities for a
variety of purposes, one of which is labeled data collection. In most cases,
crowd workers complete a task with the motivation of monetary gain. Even
though this approach has been proven cost-effective, it has also been
criticized because it potentially leads to questionable data quality unless
the necessary quality assurance mechanisms are put in place [31]. A subtype of
crowdsourcing, games-with-a-purpose [32], provides a different kind of
incentive [33] for the workers to complete the tasks to the best of their
abilities, and it generally incurs no additional costs to the employer. The
design of crowdsourcing tasks in the form of a game is considered a part of a
much larger concept; gamification. Gamification can be defined as a technique
of using game elements in non-game systems to improve user experience and
engagement [34], increasing the motivation of the respondents by satisfying
psychological and social needs [35]. Gamification of data collection finds
application in different domains [36], such as education [37][38] and health
[39].

##  III Gamified Data Collection and Interpretable FER

In this section, we present our solution that addresses the challenges of
labeled data collection for FER model training and devising human-friendly
explanations for emotion recognition systems. Specifically, we elaborate on
the gamified data collection approach of Facegame and the underlying
interpretable FER method.

###  III-A Facegame

Emotional facial expressions emerge rapidly and mostly involuntarily on human
faces. Nevertheless, humans are exceptionally good at recognizing even the
subtlest cues that appear on the faces of others. Even though humans
inherently possess these abilities, it is surprisingly challenging to exercise
them deliberately. The motivation of Facegame is to provide the players with a
challenging way to exercise the skills of facial expression perception and
mimicking.

In Facegame, the goal of the players is to mimic the facial expression shown
on a target image. The interface of Facegame (Fig. 1) displays two images
together; (a) a target image from the database of the game, which contains a
face that exhibits a certain basic emotion and (b) the player’s camera feed.
Thus, the interface allows the player to compare both faces and try to imitate
the target face.

All target images in the database are labeled based on six basic emotions [13]
by human experts, as well as the 20 AUs (Table I ) automatically using a pre-
trained classifier provided by Py-Feat [40]. The classifier’s inputs are the
following two vectors: the facial landmarks, a $(68\times 2)$ vector of the
landmark locations that is computed with the dlib package[41], and the HOGs; a
vector of $(5408\times 1)$ features that describe an image as a distribution
of orientations [21]. The HOGs are calculated for the faces that are aligned
using the position of the two eyes, and masked using the positions of the
landmarks. The model’s output is a list of AUs detected on the face image. The
pipeline of the AU detection is shown in Fig. 2.

The player is given five seconds to mimic the target expression. Afterward,
the player’s image is processed and automatically labeled with AUs. The set of
AUs on the target image are already known beforehand. Consecutively, the
Jaccard Index of the two AU sets yields the score as seen in the Equation 1; P
and T depicting the AU sets of the player and target respectively. Players can
retry imitating the same facial image to increase their scores or move on to
another image.

Figure 1: The interface of the Facegame.

Figure 2: The pipeline of the AU detection.

$Score(P,T)=\frac{|P\cap T|}{|P\cup T|}$ (1)

###  III-B Quality of data collected from the game

Every time a player plays the game, new data are generated. The score as a
game element provides feedback to the player, and the players are motivated to
do better and improve their scores, thus, generating better representations of
the target facial emotion. This acts as an inherent quality assurance
mechanism. Nevertheless, some players may show poor performance due to various
reasons. For instance, they might be just exploring and testing the game, or
their lighting conditions might be sub-optimal, or they simply do not feel
motivated to do good in the game. In any case, such turns would yield low
scores, and the data originating from players that consistently score low can
easily be eliminated, thus, diminishing noise in data.

The turns that yield a high score are considered good representations of the
emotional face expression on the target image, which is already labeled with
one of six basic emotions. Thus, the players’ image can automatically be
annotated with the same label as the target image. The minor differences
between the player and target AU sets provide a desirable variance in the
distribution of AUs corresponding to a certain emotional face expression. This
way, the variance in the AU distribution is created naturally by human players
instead of automatically generated by means of simulation, potentially
improving the in-the-wild performance of FER when used in training.

AU1 |  AU2 |  AU4 |  AU5
---|---|---|---
|  |  |
Inner Brow Raiser |  Outer Brow Raiser |  Brow Lowerer |  Upper Lid Raiser
AU6 |  AU7 |  AU9 |  AU10
|  |  |
Cheek Raiser |  Lid Tightener |  Nose Wrinkler |  Upper Lip Raiser
AU11 |  AU12 |  AU14 |  AU15
|  |  |
Nasolabial Deepener |  Lip Corner Puller |  Dimpler |  Lip Corner Depressor
AU17 |  AU20 |  AU23 |  AU24
|  |  |
Chin Raiser |  Lip Stretcher |  Lip Tightener |  Lip Pressor
AU25 |  AU26 |  AU28 |  AU43
|  |  |
Lips Part |  Jaw Drop |  Lip Suck |  Eyes Closed
TABLE I: The 20 AUs predicted by the AU detection model.

###  III-C Interpretable FER Explanation

Even though there is no precise formula for how combinations of AUs translate
into emotional expressions, some strong correlations exist. For instance, a
happy face generally exhibits a smile, which is characterized by the existence
of the “lip corner puller.” We propose using AUs as a means for explaining the
the outcome of FER models. Specifically, we utilize AU detection parallel to
FER, and translate the identified AUs into natural language descriptions,
which constitute human-friendly, interpretable explanations of FER (Figure 3).
The natural language descriptions are generated by a rule-based dictionary
approach.

We created the dictionary of AU descriptions based on the definitions on
Facial Action Coding System [13]. The dictionary contains AU combinations
categorized based on the facial muscle types and areas of appearance: cheeks,
eyebrows, eyelids, lips, chin and nose, mouth, horizontal, oblique, and
orbital. Every combination of the AUs that fall into these categories are
represented in the dictionary. One example entry from the dictionary is as
follows:

> Eyebrows, AU4: “brow lowerer”
>
> description: “eyebrows are lowered.”
>
> prescription (+): “lower your eyebrows.”
>
> prescription (-): “do not lower your eyebrows.”

The prescriptions of what players must do in order to improve their scores are
given based on the outcome of a comparison between the target AU set and the
player AU set. The intersection of both sets (P for the player AUs and T;
target AUs) are the correctly mimicked AUs, while the difference between them
show two kinds of mistakes; The set P-T includes the AUs that should be
removed from the player’s expression, and the set T-P covers the AUs that are
missing on the player’s expression to mimic the target successfully. The AUs
in both sets of mistakes are expressed as natural language prescriptions in
different polarities; for example; “raise your eyebrows” and “do not raise
your eyebrows”.

Figure 3: A graphical representation of the overall explainability model.

##  IV Methodology

In this section, we present the overall methodology of this study, and
describe the experimental setup for the data collection and analysis. In order
to evaluate the efficacy and utility of the proposed gamified data collection
method, i.e., Facegame, we pose the first research question as follows.

> RQ1: Do the data generated by the players represent emotional facial
> expressions accurately?

We also hypothesize that through playing the game repeatedly, players exercise
deliberate practice of their face expression and perception skills, and thus,
be able to improve them. Hence, our second research question is as follows.

> RQ2: Do the players improve their facial expression and perception skills
> through repeated play?

Finally, to evaluate how the proposed explanation method is perceived, and
whether this yield an improved level of understanding regarding the outcome of
the FER model. Hence, we formulate the third research question as follows.

> RQ3: Do the natural language explanations help players understand the
> outcome of the FER model?

###  IV-A Experiment setup and procedure

In the experiment design, we adjusted the Facegame slightly. Participants were
asked to play six rounds of the game, each round corresponding to one of the
six target images, and each image representing one basic emotion. Participants
were shown each target image five times in a row to be able observe the score
change in each try. They were given five seconds, indicated by a countdown on
the screen, to mimic the face. Following, the score of their attempt was
displayed (Eq. 1). To examine the potential effect of the natural language
prescriptions, we divided the participants into two groups. One group received
only the score (control group), and the other received the natural language
prescriptions as well as the score (treatment group)

###  IV-B Analysis of the collected emotional facial image data

To answer the first research question, we analyzed the collected emotional
facial image data, and evaluate to what extend the collected data can be used
to model facial expressions of emotions. To this end, we used two commonly
used in-the-wild data sets; Facial Expression Recognition 2013 (FER2013) [42],
and The Real-world Affective Face Database (RAF-DB) [43] [44], and we trained
a simple neural network classifier.

FER2013 is a large-scale dataset that includes images collected automatically
by the Google image search API labelled with the six basic emotions and
neutral. RAF-DB is a real-world dataset that includes images of faces
collected from the Internet and manually labelled with crowdsourcing. The
neural network architecture we trained is similar to the baseline model used
in the study of Bishay et al. [45], a shallow convolutional neural network
(CNN) including four convolutional layers with 32, 32, 64, and 64 filters
respectively and the ReLU activation function. The first three layers were
followed by a max-pooling layer with a $2\times 2$ filter, and the last one by
a flatten layer. The final two layers are fully-connected. The first fully-
connected layer has 96 neurons, while the second has six sigmoid units
representing the predictions of the six emotions.

Considering that FER2013 and RAF-DB are large-scale, unbalanced datasets,
samples were taken in order to respect the size and class distribution of the
Facegame data. The samples were created by random sampling without replacement
200 and 50 instances from each of the six emotion from the training and
testing set, respectively. With this technique we obtained a balanced sample
of each set containing 1,200 instances for training and 300 instances for
testing. To achieve generalizabile observations, we formed five different
samples from each of both FER2013 and RAF-DB datasets. Finally, we compared
the performance of the models trained on each sample set twice; first, without
the inclusion of the Facegame Data, second, with the inclusion of Facegame
Data.

###  IV-C Participant Survey

To receive quantitative and qualitative feedback regarding the design of
Facegame, and to partially answer the third research question, we prepared a
questionnaire. The participants were asked to fill in the questionnaire after
completing the game experiment. The questions covered demographics questions,
i.e., age and gender, technical information, i.e., type of device and browser
used for the game, and their quantitative and qualitative feedback on the
game. The quantitative feedback was given with a 5-Likert scale score (Very
Satisfied, Somewhat Satisfied, Neutral, Somewhat Unsatisfied, Very
Unsatisfied) on different aspects of the game; the ease of use, time to load
and browser compatibility, and design. The survey for the participants in the
treatment group included additional information regarding the natural language
prescriptions. Specifically, they were asked to give a score on the usefulness
and the design of the prescriptions. The qualitative feedback was sought with
two open-ended questions about comments and suggestions regarding the
functionality and the design of the game. Similar to the quantitative
feedback, the participants of the treatment group were asked two additional
open-ended questions regarding the clarity and other comments on the
prescriptions that were displayed. The analysis of the survey data consisted
of descriptive statistics on the age of the participants, quantitative
analysis of the satisfaction scores regarding the design aspects of the game,
and qualitative analysis on the open-ended questions.

###  IV-D Analysis of Skill Improvement

To answer the second and the third research questions, we analyzed the
consecutive scores of the players in each round. Specifically, we compared the
first score of each round against the mean of the remaining four scores of the
same round. We tested the significance of the difference between the mean of
the scores for both groups, and also between the groups using a t-test.

##  V Results

###  V-A Facegame Scores and Skill Improvement

A total number of 36 individuals participated in the experiment of which 18
received the natural language prescriptions while the remaining 18 received
only a score. In total, 216 games were played, each game yielding five
consecutive scores: $S_{1},S_{2},S_{3},S_{4},$ and $S_{5}$. We examined the
score change by comparing the distributions of S1 and the mean of the rest;
$M_{rest}=M(S_{2},S_{3},S_{4},S_{5})$.

Our results show that, for all games $(N=216)$ the score increased
significantly between $S_{1}$ $(M=0.4$, $SD=0.23)$ and $M_{rest}$ $(M=0.45$,
$SD=0.21)$ with $t_{(215)}=2.61$, $p<0.01$ which is a clear indication of the
learning effect of Facegame.

For the control group $(N=108)$, the same comparison also yielded a
significant increase in the distributions $S_{1}(M=0.41,SD=0.23)$ and
$M_{rest}(M=0.46,SD=0.21)$, $t_{(107)}=2.26$, $p=0.02.$. For the treatment
group $(N=108)$ that received natural language prescriptions, the comparison
showed an insignificant increase in the score distributions
$S_{1}(M=0.40,SD=0.24)$ and $M_{rest}(M=0.44,SD=0.22)$, $t_{(107)}=1.44$,
$p=0.15.$

Additionally, we investigated the number of times a game ends with an
increased score for both groups. Our observations showed that $62.9\%$ of the
games resulted with an increased score when the participants received natural
language descriptions, while the score increased $57.4\%$ when the
participants received only a score.

###  V-B Survey Results and Player Feedback

In the online survey, 36 participants (22 male; 14 female) provided feedback.
The age of the participants ranged between 25 and 55 years ($M=33.77$,
$SD=7.66)$. Fig. 4 presents the satisfaction scores provided by all
participants regarding various aspects of the design of the game. Fig. 5 shows
the feedback of the participants who received natural language prescriptions
($N=18$) in the experiment regarding their content, design, and usefulness.
The results revealed that most of the participants were satisfied or neutral
to the content and the design of the prescriptions while attention should be
given to the usefulness prescriptions.

We manually semantically grouped the answers from the two open-ended
questions. The open-ended question that inquires participants’ suggestions on
the prescriptions shows that participants find the use of visualization,
prioritization, and personalization useful. Specifically, four participants
mentioned that on-screen visualization of the part of the face that they
needed to change would help them follow the natural language prescriptions
better. Three of the participants found the text too long to read in a short
time span and suggested the display of a few of the most important instead.
Two participants indicated that more personalized prescriptions would be
helpful. Lastly, the 12 participants that commented on the natural language of
the prescriptions stated that they found them understandable.

Figure 4: Likert scale Figure 5: Likert scale

###  V-C Facial Emotion Recognition with Facegame Data

The Facegame Data that were collected in the experiment consist of 636 images
depicting the six basic emotions (86 angry, 89 disgusted, 127 fearful, 132
happy, 99 sad, and 103 surprised). The accuracy for each of the five samples
for FER2013 and RAF-DB is shown in Fig. 6 and Fig. 7, respectively. The
average accuracy of the model trained on instances from FER2013 is 32.80%,
while for the model trained on the instances from the combination of FER2013
and the Facegame Data is 33,20%. Similarly, the average accuracy of the model
trained on instances from RAF-DB is 51,27%, while for the model trained on the
instances from the combination of FER2013 and the Facegame Data is 51,87%. The
observed low accuracies of the models were expected considering the nature of
the sets and the simplicity of our neural network architecture. The images in
FER2013 and RAF-DB represent in-the-wild conditions with subtle facial
expressions. Therefore, the emotion recognition task is more challenging, and
it requires complex classifiers to achieve better accuracy. Moreover, our goal
was to examine the quality of the data collected via Facegame in comparison
with the existing in-the-wild sets, and for this purpose, a simple neural
network sufficed. The results show the we were able to collect labelled data
that can potentially improve FER in-the-wild.

Figure 6: Performance of the models trained on FER2013 dataset and FER2013
dataset combined with the Facegame Dataset (FG)

Figure 7: Performance of the models trained on RAF-DB and RAF-DB combined with
the Facegame Dataset (FG)

###  V-D The Mapping of AUs to Emotion Classes and the Variability

The results of the correlation between the six basic emotions and the AUs are
shown in Fig. 8. For this analysis, the data from the trials that scored below
$\frac{1}{3}$ were excluded. The results suggest that we were able to capture
some strong correlations between certain emotional facial expressions and
their signature AUs. For instance, lips part (AU25) and jaw drop (AU26) highly
correlate with both surprise and fear, while lip corner puller (AU12) highly
correlates with happiness. The high occurrence number of nasolabial deepener
(AU11) is an indication of the underlying Py-Feat model creating a high false-
positive rate for AU11, which needs further examination. The results show that
we were able to define the emotion classes as a distribution of multiple AUs.
Such naturally occurring variety in facial expression data can potentially be
used to improve FER in-the-wild.

Figure 8: Heatmap of the occurrences of AUs detected on the participants and
the emotions of the targeted image (threshold = $\frac{1}{3}$).

##  VI Discussion and Conclusion

In this paper, we introduced a novel approach for explainable FER that
promises three related contributions. First, by means of gamification, we have
developed a method for collecting annotated face expression data continuously,
which allows us to describe the facial expressions of six basic emotions as a
distribution of AUs. Secondly, we proposed and evaluated an interpretable FER
explainability method that uses AUs as features to describe the outcomes of
FER models, i.e., facial emotion classes. The experimental observations
indicate that the natural language explanations of face expressions are
interpretable by humans. Our quantitative and qualitative results highlight
improvement opportunities regarding the design of Facegame and how we
communicate the face expression explanations. Finally, we observed that the
players were able to improve their face expression and perception skills by
playing the game.

Our results have potential theoretical and practical implications. Our method
of acquiring nuanced face expressions (i.e., distributions of AUs) that
correlate with facial emotion classes provides a means to improve the
performance of in-the-wild FER models. Such performance improvements may pave
the way for novel practical approaches in many domains, such as online
synchronous learning [46]. Moreover, the gamification approach offers a
sustainable, continuous self-training process of FER models. Our
explainability method that uses AUs as intermediary features to describe
facial emotions provides a novel approach towards achieving interpretable,
human-friendly explanations of FER models. In this study, we conducted our
experiment and analyses on a limited sample. In the future, based on the
observations and feedback collected during this study, we will continue this
line of research and improve the ways of providing prescriptions to the
players of Facegame by combining the natural language explanations with
graphical methods. Finally, the quality of the data was assured by using a
threshold of the score as a selection criterion. The score of the trial was
calculated by the presence/absence of the AUs regardless of the intensity
which has the risk of including data with exaggerating facial expressions and
introducing bias to the data set [47]. In future studies, we aim to include
the intensity of the AUs as well as a higher threshold in the selection
criteria to increase the accuracy and fairness of the collected data set.

The goal of this study is limited to the evaluation of gamification as a data
collection method. In the future, we aim to collect more data by crowdsourcing
a large number of players which will result in a more in-depth analysis of the
collected data set. In this setting, the player was asked to mimic the facial
expressions of a face displaying one emotion and not triggered to experience
that emotion. This approach has limitations considering that there are
differences between the posed and spontaneous facial expressions in
morphological and dynamic aspects of certain emotions [48] as well as
demographical mismatches between the target image and the player. In future
studies, we aim to increase the variance of the dataset by including in the
set of the target images faces with spontaneous facial expressions as well as
increasing the demographic diversity of the target images.

## Ethical Impact Statement

In this study, the personal data collection was limited to facial images of
the players. The collected data were securely stored in the server of the
research institute which can only be accessed by the researchers of this
study, The participants were informed regarding the experiment and data
collection, and they provided consent prior to taking part in the experiment.

## References

  * [1] K. R. Scherer, and P. H. Tannenbaum, “Emotional experiences in everyday life: A survey approach,” Motivation and Emotion, vol. 10, pp. 295–314, 1986
  * [2] A.J. Fridlund, “Human facial expression: An evolutionary view,” Academic press, March 2014.
  * [3] D. Matsumoto, K. Dacher, M. N. Shiota, M. O’Sullivan, and M. Frank, “Facial expressions of emotion,”The Guilford Press, 2008.
  * [4] R. W. Picard, “Affective computing,” MIT press, July 2000.
  * [5] M. Maithri, U. Raghavendra, A. Gudigar, J. Samanth, D. B. Prabal, M. Murugappan, et al. “Automated Emotion Recognition: Current Trends and Future Perspectives,” Computer Methods and Programs in Biomedicine, pp. 106646, 2022.
  * [6] B. C. Ko, “A brief review of facial emotion recognition based on visual information,” sensors, vol. 18, pp.401, February 2018.
  * [7] S. Li, and D. Weihong, “Deep facial expression recognition: A survey,” IEEE transactions on affective computing, 2020.
  * [8] S. Li, and D. Weihong, “Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition,” IEEE Transactions on Image Processing, vol. 28, pp.356–370, 2018.
  * [9] L.H. Gilpin, D. Bau, B.Z. Yuan BZ, A. Bajwa, M. Specter, and L. Kagal. “Explaining explanations: An overview of interpretability of machine learning,” 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80-89, October 2018.
  * [10] P. Kumar P, Kaushik V, and B. Raman. “Towards the Explainability of Multimodal Speech Emotion Recognition,” Interspeech, pp 1748–1752, 2021
  * [11] E. Ghaleb, A. Mertens, S. Asteriadis, and G. Weiss, “Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks,” 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–8, 2021.
  * [12] J.V. Jeyakumar, J. Noor, Y.H Cheng, L. Garcia, and M. Srivastava, “How can i explain this to you? an empirical study of deep neural network explanation methods,” Advances in Neural Information Processing Systems, vol. 33, pp. 4211–4222, 2020.
  * [13] P. Ekman, and W. Friesen, “Facial action coding system,” Environmental Psychology & Nonverbal Behavior, 1978.
  * [14] P. Ekman, “Facial expression and emotion,” American psychologist vol. 48, pp. 384, April 1993.
  * [15] R. Reisenzein, M. Studtmann, and G. Horstmann. “Coherence between emotion and facial expression: Evidence from laboratory experiments,” Emotion Review, vol. 5, pp. 16–23, January 2013.
  * [16] M. Wegrzyn, M. Vogt, B. Kireclioglu, J. Schneider, and J. Kissler, “Mapping the emotional face. How individual face parts contribute to successful emotion recognition,” PloS one, vol. 12, pp. 11–12, May 2017.
  * [17] N. Borges, L. Lindblom, B. Clarke, A. Gander, and R. Lowe, “Classifying confusion: autodetection of communicative misunderstandings using facial action units,” 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 401–406, September 2019.
  * [18] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression,” 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp. 94–101, June 2010.
  * [19] S.M. Mavadati, M.H Mahoor, K. Bartlett, P. Trinh, and J.F. Cohn, “Disfa: A spontaneous facial action intensity database,” IEEE Transactions on Affective Computing, pp. 151–160 2013 Mar 7;4(2):151–60.
  * [20] T. Baltrušaitis, M. Mahmoud, and P. Robinson, “Cross-dataset learning and person-specific normalisation for automatic action unit detection,” 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 6, pp. 1–6, May 2015.
  * [21] N. Dalal, and B. Triggs, “Histograms of oriented gradients for human detection,” In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1, pp. 886–893, June 2005.
  * [22] O. Çeliktutan, S. Ulukaya, and B. Sankur, “A comparative study of face landmarking techniques,” EURASIP Journal on Image and Video Processing, pp. 1–27, December 2013.
  * [23] Z. Shao, Z. Liu, J. Cai, Y. Wu, and L, Ma, “Facial action unit detection using attention and relation learning,” IEEE transactions on affective computing, October 2019.
  * [24] G.M. Jacob, and B. Stenger, “Facial action unit detection with transformers,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7680–7689, 2021.
  * [25] Z. Shao, Z. Liu, J. Cai, Y. Wu, and L, Ma, “JAA-Net: joint facial action unit detection and face alignment via adaptive attention,” International Journal of Computer Vision, vol. 129, pp. 321–340, February 2021.
  * [26] K. Zhao, W.S Chu, and Zhang, “Deep region and multi-label learning for facial action unit detection,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3391–3399, 2016.
  * [27] A. Rosenfeld, and A. Richardson, “Explainability in human–agent systems,” Autonomous Agents and Multi-Agent Systems, vol. 33, pp. 673–705 November 2019.
  * [28] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, “A survey of methods for explaining black box models,” ACM computing surveys (CSUR). 2018 Aug 22;51(5):1-42.
  * [29] A. Rosenfeld, “Better metrics for evaluating explainable artificial intelligence,” Proceedings of the 20th international conference on autonomous agents and multiagent systems, pp. 45–55, May 2021.
  * [30] J. Howe J, “The rise of crowdsourcing,” Wired magazine, vol. 6, pp. 1–4, June 2006.
  * [31] D. Iren, S. Bilgen, “Cost of quality in crowdsourcing,” Human Computation, vol. 1, December 2014.
  * [32] A.J Quinn, and B.B Bederson, “Human computation: a survey and taxonomy of a growing field,” Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1403–1412, May 2011.
  * [33] M. Hosseini, A. Shahri, K. Phalp, J. Taylor, and R. Ali, “Crowdsourcing: A taxonomy and systematic mapping study,” Computer Science Review, vol. 17, pp. 43–69, August 2015.
  * [34] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design elements to gamefulness: defining” gamification,” Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments, pp. 9–15, September 2011
  * [35] A.F. Aparicio, F.L. Vela, J.L. Sánchez, and J.L Montes, “Analysis and application of gamification,” Proceedings of the 13th International Conference on Interacción Persona-Ordenador, pp. 1–2, October 2012.
  * [36] V. Gurav, M. Parkar, and P. Kharwar, “Accessible and Ethical Data Annotation with the Application of Gamification,” International Conference on Recent Developments in Science, Engineering and Technology, pp. 68–78, November 2019.
  * [37] J.J López-Jiménez, J.L. Fernández-Alemán, L.L González, O.G Sequeros, B.M Valle, J.A García-Berná, et al. “Taking the pulse of a classroom with a gamified audience response system,” Computer Methods and Programs in Biomedicine, vol. 213, pp. 106459, January 2022.
  * [38] A.I. Wang, and R. Tahir, “The effect of using Kahoot! for learning–A literature review” Computers & Education, vol. 149, pp. 103818, May 2020.
  * [39] P. Tobien, L. Lischke, M. Hirsch, R. Krüger, P. Lukowicz, A. Schmidt, “Engaging people to participate in data collection,” Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp. 209-212, September 2016.
  * [40] J.H. Cheong, T. Xie, S. Byrne, and L.J. Chang, “Py-feat: Python facial expression analysis toolbox,” arXiv preprint arXiv:2104.03509, April 2021.
  * [41] D. E. King, “Dlib-ml: A machine learning toolkit,” The Journal of Machine Learning Research, vol. 10, pp. 1755–1758, 2009.
  * [42] I .J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner et al, “Challenges in representation learning: A report on three machine learning contests,” International conference on neural information processing, Springer, Berlin, pp. 117–124, November 2013.
  * [43] S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2852–2861, 2017.
  * [44] S. Li, and W. Deng, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition,” IEEE Transactions on Image Processing, vol. 28, pp. 356–370, 2019.
  * [45] M. Bishay, A. Ghoneim, M. Ashraf, M. Mavadati, “Which CNNs and Training Settings to Choose for Action Unit Detection? A Study Based on a Large-Scale Dataset,” 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–5, 2021.
  * [46] K. Shingjergji, D. Iren, C. Urlings, and R. Klemke, “Sense the classroom: AI-supported synchronous online education for a resilient new normal,” EC-TEL (Doctoral Consortium), pp. 64–70, January 2021.
  * [47] M. Mäkäräinen, J. Kätsyri, and T. Takala, “Exaggerating facial expressions: A way to intensify emotion or a way to the uncanny valley?,” Cognitive Computation, vol. 6. pp. 708–721, Decembe 2014.
  * [48] S. Namba, S. Makihara, R.S. Kabir, M. Miyatani, and T. Nakao, “Spontaneous Facial Expressions Are Different from Posed Facial Expressions: Morphological Properties and Dynamic Sequences,” Current Psychology, vol 36, pp. 593–605, 2017


2021

[1]Veronika Solopova

[1]Dahlem Center for Machine Learning and Robotics, Freie Universität Berlin,
Germany

2]German Aerospace Center, Jena, Germany

# Automated multilingual detection of Pro-Kremlin propaganda in newspapers and
Telegram posts

<EMAIL_ADDRESS>Oana-Iuliana Popescu
<EMAIL_ADDRESS>Christoph Benzmüller  c.benzmueller@fu-
berlin.de Tim Landgraf<EMAIL_ADDRESS>*  [

###### Abstract

The full-scale conflict between the Russian Federation and Ukraine generated
an unprecedented amount of news articles and social media data reflecting
opposing ideologies and narratives. These polarized campaigns have led to
mutual accusations of misinformation and fake news, shaping an atmosphere of
confusion and mistrust for readers worldwide. This study analyses how the
media affected and mirrored public opinion during the first month of the war
using news articles and Telegram news channels in Ukrainian, Russian, Romanian
and English. We propose and compare two methods of multilingual automated pro-
Kremlin propaganda identification, based on Transformers and linguistic
features. We analyse the advantages and disadvantages of both methods, their
adaptability to new genres and languages, and ethical considerations of their
usage for content moderation. With this work, we aim to lay the foundation for
further development of moderation tools tailored to the current conflict.

###### keywords:

Propaganda, Fake news, NLP, Kremlin, Ukraine, Automated Content Moderation

##  1 Introduction

Propaganda influences an audience to support a political agenda. Smith (2022);
202 (2022). Propaganda has been shown to play a vital role in the Russian
invasion of Ukraine, shaping the war approval rate Khvostunova (2022) by e.g.
fabricating explanations for war crimes Roth (2022). As a result, fake news
also spreads through Ukrainian, Central European and Western media Heritage
(2014), seeding mistrust and confusion Paul (2022).

With every day of the war having a large amount of potentially false
information produced, human quality control thereof is limited. Especially
during a war, the journalistic virtue of fact-checking may be substantially
obstructed. This poses the question of whether statistical analysis can
provide us with a reliable prediction of the intent behind a piece of news.
Given a sufficiently high separability, automatic moderation tools could
process the news and warn readers about the potential disinformation instead
of fully placing the responsibility on human moderators Steiger et al. (2021).
The objective of this study is to detect war propaganda produced in the
context of the 2022 Russian invasion of Ukraine in a transparent, explainable
way, as such a tool can be used for content moderation in social media.
While Russian propaganda creates a ‘cacophony’ of fabricated opinions and
sources in its media ecosystem of Excellence (2016), it also has a number of
uniform strategies and approaches which are recurrently mentioned in research
throughout the whole of Russian-Ukrainian war by international bodies and
independent researchersCarroll (2017); Meister (2016); Fortuin (2022); gec
(2020). Hence, we hypothesize and aim to prove that propaganda can be
successfully detected using certain stylistic and syntactical features behind
these strategies, independent of keywords. Naturally, keywords change
depending on the course of events, while the tactics of the propaganda stay
similar. Traditional algorithms empowered by such features are inherently
interpretable and may perform on par with intransparent neural networks. Here,
we propose a linguistics-based approach for detecting war propaganda in the
context of the ongoing war between Ukraine and Russia since the start of the
full-scale invasion in 2022, using models trained on news from media outlets
from Ukraine, Romania, Great Britain and the USA. We extracted news from fact-
checked outlets identified as credible sources and from outlets recognized as
spreading fake news. We train and evaluate classifiers using a set of
linguistic features and keywords, and a multilingual Transformer to classify
articles as containing pro-Kremlin and pro-Western narrative indicators.
With this work, we provide an open-source tool for the identification of such
fake news and propaganda in Russian and Ukrainian, which, to the best of our
knowledge, is the first of its kind. We demonstrate that discriminating
propaganda from neutral news is not entirely possible in the current
situation, as news from both sides may contain war propaganda, and Western
media is heavily dependent on Ukrainian official reports.

In Section 2 we present previous work related to our research. In Section 3 we
first describe the sources of our data and its collection process (3.1-3.2),
then, we expand upon the linguistic features (3.3) and the keywords that we
extract (3.4). In Section 4 we introduce our training setup for each
experiment, describing the data and model configurations, while in Section 5
we present the results for each setting. In Section 6 we provide additional
analyses, looking into feature importance coefficients of some models (6.1)
and distributional exploratory analysis of the classes (6.2), exploring
chronological, language and narrative-specific differences. In Section 7 we
consider the main findings and limitations of our work. We lay the ground for
future work opportunities and delve into ethical dangers in terms of its
potential usage for automated content moderation. In Section 8 we summarize
the main contributions of our study.

##  2 Related Work

Our work is motivated by the fact that despite the assumed involvement of
Russia in Brexit Narayanan (2017) and the 2016 US presidential elections
Cosentino (2020a), there is still only a small number of peer-reviewed
research publications investigating Russian state-sponsored fake news Elswah
and Howard (2020); Beskow and Carley (2020). Furthermore, existing
publications are not always neutral, with some using accusations and emotional
lexicon Rosulek (2019), while others accuse Western media of propaganda used
to discredit Russia Chudinov et al. (2019).

Wilbur (2021) examines claims of Russian propaganda targeting the US
population by analysing weblogs and media sources in terms of their attitude
towards Russia and finding a positive correlation between the Russian media
outlet Sputnik and several US media sources.

Timothy Snyder in his books Snyder (2018); Sly (2017) analyses the Kremlin’s
informational campaign against the sovereignty of the United States and the
integrity of the European Union.

Several studies investigated Russian bots in social media. Alsmadi and O’Brien
(2020) used a decision tree classifier on features extracted from the tweets,
concluding that bot accounts tend to sound more formal or structured, whereas
real user accounts tend to be more informal, containing slang, slurs, and
cursing; Beskow and Carley (2020) analyzed the geography and history of the
accounts, as well as their market share using Botometer, pointing to a
‘sophisticated and well-resourced campaign by Russia’s Internet Research
Agency’. Narayanan (2017) performed a basic descriptive analysis to discern
how bots were being used to amplify political communication during the Brexit
campaign. There is a considerable amount of research focusing on fake news
detection. Asr and Taboada (2019) show-cased that significant performance can
be achieved even with n-grams; Antoun et al. (2020); Mahyoob et al. (2020);
Rashkin et al. (2017) implemented different linguistic feature-based
solutions, while Li et al. (2021) demonstrated the application of
Transformers. While fake news makes up a big part of the propaganda toolkit,
propaganda also relies on specific wording, appealing to emotions or
stereotypes, flag-waving and distraction techniques such as red herrings and
whataboutisms Yu et al. (2021). Research on propaganda detection is less
frequent. Although existing works proposed using both feature-based and
contextual embedding approaches Tundis et al. (2020); Yu et al. (2021);
Oliinyk et al. (2020); Dadu et al. (2020), these studies focused mostly on the
English language. To the best of our knowledge, there are no benchmark corpora
and no open-source multilingual tools available.

##  3 Methods

We implement a binary classification using the following models for input
vectors consisting of 41 handcrafted linguistic features and 116 keywords
(normalized by the length of the text in tokens): decision tree, linear
regression, support vector machine (SVM) and neural networks, using stratified
5-fold cross-validation (10% for test and 90% for training). For comparison
with learned features, we extract embeddings using a multilingual BERT model
Devlin et al. (2018) and train a linear model using them.

We performed 3 sets of experiments contrasting the handcrafted and learned
features:
Experiment 1. Training models on Russian, Ukrainian, Romanian and English
newspaper articles, and evaluating them on the test sets of these languages
(1.1) and on French newspaper articles (1.2). We add the French newspapers to
benchmark the multilingualism of our models. We choose French because it is in
the same language family as Romanian.
Experiment 2. Training models on Russian, Ukrainian, Romanian, English and
French newspaper articles, and validating them on the test set (2.1).
Additionally, we use this model to test the Russian and Ukrainian Telegram
data (2.2.). Here the goal is to investigate whether this model will perform
well out-of-the-box for the Telegram articles, which are 10 to 20 times
shorter. See an example of the genre-related difference in distributions in
Figure 1.
Experiment 3. Training models on the combined newspaper and Telegram data and
applying them to the test set. Here we verify whether adding the Telegram data
to the training set can improve generalization power, although data
distributions differ.

Table 1: Corpus statistics, including the sources per language and stance.

Language | Source | Amount of texts
---|---|---
Pro-Western newspapers
Ukrainian | ‘Europeiska Pravda’,‘Ukrainska Pravda’, ‘Espresso’, ‘5.ua’, ‘Hhromadske’, ‘Liga.net’ | 3298
Romanian | ‘digi24’, ‘mediafax’, ‘g4media’ | 4049
English | ‘The Guardian’, ‘BBC’, ‘The New York Times’, ‘Reuters’) | 1060
French | ‘Tv5monde’, ‘Le Monde’ and ‘Le Figaro’ | 458
Russian | ‘Raintv’ | 7
Pro-Kemlin newspapers
Ukrainian | ‘Newsua’, ‘Strana.ua’, ‘Vesti.ua’, ‘Ukranews’, ‘Zik’ | 3579 (474 in Ukrainian and 3105 in Russian)
Romanian | ‘Antena3’, ‘Stiripesurse’, ‘Romaniatv.net’, ‘Cyd.ro’, ‘Activenews’ and ‘Dcnews’. | 3007
French | ‘RT’ French edition | 123
Russian | ‘Ria news’, ‘Russia Today’, ‘Interfax’, ‘Lenta.ru’ and ‘Ukraine.ru’. | 2648
Telegram posts
Ukrainian | ‘Goncharenko’, ‘InformNapalm’,‘Brati po zbroi’, ‘Spravdi’,‘Operativni ZSU’ | 7263 ( 1568 in Ukrainian and 1568 in Russian)
Russian | ‘Rybar’,‘Siloviki’,‘Vysokigovorit’ | 61595

##  4 Data

###  4.1 Newspapers

We automatically scraped articles from online news outlets using the
newspaper111https://newspaper.readthedocs.io/en/latest/ framework. Our data
collection spans the period from the 23rd of February, on the eve of the
Russian full-scale attack on Ukraine, until the fourth of April, and we sample
at eight time points during that period.

Our choice of media outlets and languages is based on the geopolitical zones
which might have been affected by propaganda. We collected news from Ukrainian
and Russian media outlets, choosing sources that support pro-Kremlin
narratives in Ukraine that have been confirmed by journalistic investigations
to directly copy pieces of news from Russian news outlets UkraineWorld.org
(2022). We included American and British English-speaking outlets as a
positive control of widely recognised high-quality news, as well as French
news sources. We also added Romanian news as representative of the Central
European block, which is one of the secondary targets of propaganda campaigns
Rosulek (2019), and used websites that have been categorized by
Rubrika.ro222https://rubrika.ro/extensie-browser as containing fake news.
Except for English, all languages have two subsets, one supporting the Russian
side of the conflict, and one supporting the Ukrainian one. In total, we
collected 18,229 texts: 8872 texts featuring pro-Western narratives and 9357
reflecting the official position of the Russian government. The sources are
listed in Table 1.

Note that the ground-truth labels were assigned only according to the news
source without manual labelling.

###  4.2 Telegram posts

Since the start of the war, many Telegram channels became widely used in
Ukraine and Russia for quicker updates on the war and for posting uncensored
reports Bergengruen (2022); O’Brien (2022); Valeriya Safronova (2022).
However, it is a source without moderation and fact-checking, hence fake news
and emotional lexicon, including profanity and propaganda, are not unusual
Sweney (2022). Therefore, we included both Russian and Ukrainian Telegram news
in our data collection.

###  4.3 Linguistic Feature Selection

We start processing the collected texts by extracting per-article linguistic
features. The first set of features have been used previously in Mahyoob et
al. (2020) to detect fake news: a number of negations, adverbs, average
sentence length, proper nouns, passive voice, quotes, conjunctions (we also
count separately the frequency of the conjunction ‘but’ to capture
contrasting), comparative and superlative adjectives, state verbs, personal
pronouns, modal verbs, interrogatives. Since fake news and propaganda can be
associated with ‘overly emotional’ language Asr and Taboada (2019), we
generate word counts for each basic emotion category: anger, fear,
anticipation, trust, surprise, sadness, joy, disgust, and identify two
sentiment classes, negative and positive, using the NRC Word-Emotion
Association Lexicon Mohammad and Turney (2013). Following Rashkin et. al.
Rashkin et al. (2017), we also extract the number of adjectives, the overall
number of verbs and action verbs, as well as abstract nouns (e.g. ‘truth’,
‘freedom’), money symbols, assertive words, and second person pronouns
(‘you’), and the first person singular (‘I’). Inspired by the journalistic
rules of conduct for neutrality333 https://www.spj.org/ethicscode.asp, we
count the number of occurrences of words from several dictionaries: survey,
reporting words, discourse markers, reflecting the surface coherence of the
article, words denoting claims(e.g. ‘reckon’, ‘assume’), high modality words
(e.g. ‘obviously’, ‘certainly’). As syntactic features, we measure the number
of subordinate clauses of concession (using subordinate conjunctions similar
to ‘although’, and ‘even if’), reason (e.g. ‘because’, ‘as’), purpose (e.g.
‘in order to’, ‘to’), condition (e.g. ‘provided’, ‘if’), time (e.g. ‘when’,
‘as soon as’) and relative clauses, which reflect different ways of
justification and argumentation.

All features are automatically extracted in a pipeline and translated for each
language using simplemma444https://adrien.barbaresi.eu/blog/simple-
multilingual-lemmatizer-python.html and
pymorphy2555https://pymorphy2.readthedocs.io/en/stable/ for part-of-speech
extraction in Ukrainian and spacy666https://spacy.io for Russian, English,
Romanian and French. The code will be released publicly upon acceptance.

Figure 1: Examples of genre-related differences between Newspapers and Telegram subsets. The boxplot represents 25% around the median, the whiskers show the first and last quartiles. Ukrainian news has much more adjectives than Telegram posts, while it is vice-versa for Russian news. Sentences are much longer in Telegram for both languages.  Table 2: Comparative results achieved on best folds. Algorithm | Cohen’s $\kappa$ | F1 | False positives | False negatives
---|---|---|---|---
Experiment 1.1 Test on subset (1768 texts)
Decision tree | 0.49 | 0.73 | 16 | 450
Linear logistic model | 0.58 | 0.79 | 113 | 265
SVM | 0.75 | 0.87 | 156 | 151
MLP | 0.64 | 0.80 | 103 | 229
BERT | 0.84 | 0.92 | 97 | 42
Experiment 1.2 Test on French (20 texts)
SVM | 0.01 | 0.5 | 6 | 16
BERT | 0.05 | 0.52 | 19 | 0
Experiment 2.1 Test on subset (1827 texts)
SVM | 0.75 | 0.884 | 120 | 151
BERT | 0.86 | 0.93 | 111 | 12
Experiment 2.2 Test on Telegram (14525 texts)
SVM | 0.25 | 0.637 | 2013 | 3402
BERT | 0.17 | 0.58 | 5770 | 212
Experiment 3. Test on subset (8709 texts)
SVM | 0.66 | 0.88 | 707 | 267
BERT | 0.81 | 0.92 | 136 | 162

###  4.4 Keywords

As a list of keywords, we use the
glossary777https://www.rnbo.gov.ua/files/2021 prepared by the National
Security and Defense Council of Ukraine. It contains a list of names, terms
and phrases recommended for use by public authorities and diplomatic missions
of Ukraine and as well as versions of these terms used in Russian political
discourse. We translate this glossary to the target languages and add a short
list from the military lexicon being avoided by Russian officials (e.g. ’war’,
’victims’, ’children’, ’casualties’) Gessen (2022).

##  5 Results

We evaluate the performance of our models using Cohen’s $\kappa$ Cohen (1960)
and F-measure Powers (2008). While F1-score is easy to interpret and most
frequently used, subtracting the Expected Accuracy, Cohen’s Kappa removes the
intrinsic dissimilarities of different data sets, which makes two
classification problems comparable, as K can compare the performances of two
models on two different cases Grandini et al. (2020). We also evaluate the
number of false positives and negatives, which help build a complete picture
of the model’s performance. The results for all settings, averaged over five
models, can be found in Table 2. Details about the models and hyperparameters
can be found in Appendix.

Experiment 1. When training on Russian, Ukrainian, Romanian and English
newspaper articles, the best result on the handcrafted linguistic features (no
keywords) was achieved with an SVM: 0.87 F1-measure and 0.75 $\kappa$. The
model is almost equally prone to false positives (108) and false negatives
(120) across 1768 test samples (FP-rate: 0.06, FN-rate: 0.06). Linear models
and a 2-layer neural network performed considerably worse (F1: 0.8). As the
SVM performed best, we continued our experiments with this model, added our
extracted keywords to the dataset, but found no improvement.

The linear model using BERT embeddings outperformed the handcrafted feature
models (F1: 0.92, and $\kappa$: 0.84). While it produces a similar quantity of
false positives as the SVM, the false negative rate decreases considerably.
When testing on 40 French texts (20 pro-Kremlin, 20 pro-Ukrainian), the
performance drops considerably for the feature-based approach (F1: 0.5,
$\kappa$: 0.01) with 14 false negatives and 6 false positives, and for BERT
embeddings (F1: 0.52, $\kappa$: 0.05) with 19 false positives and only one
true negative.

Experiment 2. The addition of French newspaper articles to the training set
increased the F1-score by 0.08 for both SVM and embeddings-based models.
However, the models do not perform well when tested on Telegram data. Without
keywords, the SVM model scored 0.61 F1-measure, with a very low $\kappa$ of
0.24, 2078 false positives and 3422 false negatives out of 14525 test samples.
Adding keywords increases performance (F1: 0.62, $\kappa$: 0.25), lowering the
false positive and false negative (FP-rate: 0.13, FN-rate: 0.23). The
embeddings-based model scores even lower (F1: 0.58, $\kappa$: 0.17, FP-rate:
0.39, FN-rate: 0.014)

Experiment 3. Finally, we train on the full dataset with both newspaper
articles and Telegram posts. The handcrafted feature-based model increases the
F1-score to 0.88, but decreases $\kappa$ to 0.66, with 707 false positives and
267 false negatives out of 8709 test samples. The embeddings-based model
reaches 0.90 F1-measure and 0.81 $\kappa$, with 136 false positives and 162
false negatives.
Both models make disproportionately more errors when tasked with the
classification of Romanian texts.

Figure 2: Permutation importance (drop of F1 score in %) for an SVM with
linear kernel. Keyword importance is on the left side and the importance of
linguistic features is on the right. Negative bars indicate features that are
important for classifying a data point as pro-Western, while positive bars
represent features indicative of pro-Kremlin propaganda.

##  6 Additional Analysis

###  6.1 Feature Importance

We further analyse our best-performing SVM model to obtain feature importance
for both linguistic features and keywords using the feature permutation method
Breiman (2004). We find that various subordinate clauses prove to be important
for the model, with the presence of the clause of the reason being the most
indicative of pro-Western narratives, as well as passive voice. To a lesser
degree, the following features were also deemed as important: superlatives,
money symbols and words, clauses of condition, state verbs, comparatives and
words indicating claims. For those features, it can be stated that they are
unlikely to be found in pro-Kremlin news. At the same time, discourse markers
(e.g. ‘however’, ‘finally’) as well as clauses of concession, clauses of
purpose, conjunctions, negations and clauses of time separate pro-Kremlin news
the best.

We find many keywords coming from the list provided by the Ukrainian Security
Council glossary to be important. However, some of them need cultural and
historical context to be understood. We find that the formulation ‘in Ukraine’
is the most reliable marker of pro-Western news, while in Russian news the
formulation is ‘on Ukraine’, which indicates its use as ‘on a territory’ and
not ‘in the country’. Interestingly, the use of ‘in Donbas’ is the second
highest indicator for Russian news. While it is a conventional name for the
territory shared by two Ukrainian regions, it would preferably be used with
the preposition ‘on’, e.g. ‘on the Western Balkans’. The usage of ’in’ gives
linguistic legitimacy to the idea of the independence of the quasi-republics.
The variant of the country’s name ‘Belarus’ is highly indicative of the
Western side, while ‘Belorussia’, the version found in Russian news, presents
the neighbouring country’s name rather as ‘white’ Russia, and not as ‘Rus’,
the historical area in Eastern Europe. The formulation ‘special
operation’888https://www.un.org/press/en/2022/sc14803.doc.htm is a euphemism
for the word ‘war’ used by the Russian government and the pro-governmental
news agencies. It is a strong indicator towards a pro-Kremlin narrative. On
the Western side, we observe that the word ‘invasion’ has a higher frequency.
Other words with high importance values for pro-Kremlin narratives are
demilitarisation (of Ukraine), ‘self-defence’, ‘militia’, ‘Bandera’
999Politician and theorist of the militant wing of the Organization of
Ukrainian Nationalists in 20th-century Marples (2006)., ‘Baltic countries’,
also commonly called ‘Pribaltika’ in Russian, again presented more as a
territory, and finally ‘foreign agent’.

Many of the words we assumed will not be used in pro-governmental Russian
articles were found to be important markers. Hence, words commonly used by the
pro-Ukrainian news describing the disastrous consequences of war for both
sides, e.g. ‘children’, ‘looting’, ‘war crimes’, ‘deceased’, ‘victims’,
‘rape’, ‘sanctions’, ’embargo’, are attributed high importance in Western
media. Some other curious keywords often occurring in Western media are
‘Russian world’ and ‘russcists’, which are mainly used by Ukrainian media as
means of referring to the ideology of the Russian military expansionism
Gaufman (2016).

###  6.2 Distributional exploratory analysis

Chronological analysis. We also carried out an exploratory study of the
feature and keyword distributions over 5 data collections: the 23rd of
February, the 1st of March, the eighth of March, the 18th of March and the
fourth of April. We looked at the contrast between different languages and
between pro-Kremlin and pro-Western media within one language, with the aim to
explain frequent model errors and observe how media reflects the events of the
war.

The most noticeable observation for Ukrainian pro-Western media is an increase
in many features on the 18th of March, following the bombing of the Mariupol
theatre on the 16th Daniel Boffey and Borger (2022): abstract nouns, claims,
and negative emotional lexicon (namely words of surprise, disgust, sadness,
fear and anger). Some indicators, like reporting words, negations, proper
nouns, and modal verbs drop in frequency in March and seem to come back to the
pre-war level in April. The use of the word war is constantly increasing
throughout our data collection.

In contrast, in Russian pro-governmental media, the collection date with the
most deviation from the overall average is the 1st of March, when we can
observe a drastic increase in the number of adjectives, average sentence
length, assertive words, clauses of purpose, but also negative emotional
lexicon (including words of trust and anger), and positive emotional lexicon.
1st of March corresponds to the end of the first week of the war when it
became clear that the war might extend for a longer period Harding (2022).

British and American media remained quite stable throughout this time period,
although we can observe an increase in superlatives on the fourth of April,
which follows closely the discovery of the war crimes in Bucha, where 412
civilians were killed Horton et al. (2022). Pro-Western Romanian data also did
not change considerably, with an exception of a slight increase with each
collection in clauses of reason, words of surprise and the keyword ‘war’. At
the same time, in the Romanian media flagged as fake news, there is a drop in
words of anger and an increase in words of disgust, surprise, happiness and
expectation, as well as abstract nouns, modal verbs, clauses of purpose when
compared to the pre-war collection.

Language and narrative specific feature differences. When comparing media in
different languages we observe interesting trends, which, however, did not
account much for the decision of the classifiers. For instance, English and
Ukrainian pro-Western media have the highest personal pronouns frequency,
while newspapers from Russian media have the highest amount of quotations and
are the only using keywords such as: ‘coup d’etat’, ‘DNR’, ‘LNR’, ‘Kiev
regime’, ‘russophobia’. Romanian articles from trusted sources have the
longest sentences, and all articles from Romanian media have the lowest use of
the conjunction ‘but’. Furthermore, all articles have the highest occurrence
of comparatives, superlatives and state verbs, which we believe is language
specific. This might be the reason for the low performance when applied to
Romanian articles since these three features have high importance for the SVM
model.

Articles from Ukrainian media generally have a high frequency of adjectives.
At the same time, Ukrainian pro-Western news has the highest amount of
emotional words (sadness, expectation, disgust, surprise, fear, anger), while
pro-Russian Ukrainian articles do not show such a tendency, and thus might not
reflect the same emotional atmosphere. It also has the same distribution of
clauses of time as Russian pro-governmental news and the equally lowest usage
of passive verbs.

Thus, we do not see a clear tendency for Russian propaganda to be homogeneous
among the countries we selected. The only example would be the use of the
keyword ‘Soros’, which is used uniquely in pro-Kremlin media in Ukraine and
Russia, as well as in fake-news-flagged Romanian media. This can be explained
by invocations of antisemitic conspiracy theories explored in Timothy Snyder’s
‘Road to Unfreedom’Snyder (2018) as manipulation strategies. Otherwise, media
in Romania seems to be much more adapted to the local journalistic specifics,
while in Ukraine the pro-Kremlin articles have much more in common with their
origin.

##  7 Discussion

Our study is the first attempt to quantitatively and qualitatively analyse and
automatically detect Russian propaganda, which does not necessarily use an
emotional lexicon. Surprisingly, the propaganda style seemingly mimics
international codes of conduct for journalism and adapts to each target
language and it is country-specific. Many features of the Western news class
can be found in the aforementioned related works for fake news detection,
while pro-Kremlin features taken out of context could be interpreted as
neutral journalistic style. This indicates that morpho-syntax modelling and
analysis without semantics may be misleading. Both sides use propaganda. We
found that their features differ, which may be explained by different
ideologies and cultural specifics, but it may indicate different goals.
Russian-backed media justifies the war and downplays adverse effects, while
Ukrainian war propaganda focuses on various emotions, from ridiculing Russian
forces, instigating hate against the Russian people as a whole or proposing an
over-optimistic view of the military situation. In future work, we propose
including an additional class representing neutral Ukrainian, Russian, and
international news. However, labelling such datasets would require much more
time for manual annotation.
Since the appearance of state-of-the-art Transformer-based models, the trade-
off between transparency and accuracy is a topical issue in the NLP community.
We show that transparent statistical models based on linguistic expert
knowledge can still be competitive. Our best embeddings-based model has only
around 0.04 F1-score advantage, but it is less explainable. We cannot control
if the BERT model learnt the style of the media outlets instead of propaganda
itself, while we can be sure that SVM indeed captures Kremlin’s manipulative
lexicon. While there are methods to interpret decisions De Cao et al. (2020)
of such models, we leave this for future work. As BERT models can capture
syntax Jawahar et al. (2019), we believe that such embeddings might still be
less flexible towards changes in themes and topics, and need retraining if
major vocabulary changes occur. The BERT-based model has a clear tendency
towards false positives and performs slightly worse on the data from different
distributions. In the context of automated content moderation, false positives
would mean flagging/filtering a post or banning a person, limiting the freedom
of speech. False negatives might lead to posts with propaganda reaching more
targets.

Keywords were only beneficial for SVM when applied to new data, where the
algorithm had to base its decisions on semantics more than morpho-syntax. The
overall journalistic style captured by handcrafted features is more reliable,
as the model performance does not drastically change for any of the languages
in focus, even in the face of such events as war. Scalability is, however, a
major drawback of feature-based models, as new predictions require first-
feature extraction, while models using BERT embeddings can be used out of the
box. However, BERT models have important token length limitations, whereas
with SVM we pass a stable vector of feature counts normalised by the text
length. While it might seem natural to choose these high-performing models in
industrial settings, we believe that for the sake of transparency, models
using handcrafted features that are competitive can still be used.
Both approaches can turn out to be inefficient after a certain period of time,
especially in light of the new tendencies towards automatically generated
news.

We see our work as a first step towards a browser extension flagging harmful
content to raise individual awareness and assist us in filtering the content
we consume. However, the classifier can be used to block pro-Western news as
well, ensuring the impenetrability of echo chambers, amplifying the effects of
propaganda instead of helping to fight it.

Figure 3: Advantages and disadvantages of the presented methods.

##  8 Conclusion

We presented two alternative methods to automatically identify pro-Kremlin
propaganda in newspaper articles and Telegram posts. Our analysis indicates
that there are strong similarities in terms of rhetoric strategies in the pro-
Kremlin media in both Ukraine and Russia. While being relatively neutral
according to surface structure, pro-Kremlin sources use artificially modified
vocabulary to reshape important geopolitical notions. They also have, to a
lesser degree, similarities with the Romanian news flagged as fake news,
suggesting that propaganda may be adapted to each country and language in
particular. Both Ukrainian and Russian sources lean towards strongly
opinionated news, pointing towards the use of war propaganda in order to
achieve strategic goals.

Russian, Romanian and Ukrainian languages are under-researched in terms of NLP
tools in comparison to English. We hope that our study contributes to social
media and individual efforts to moderate harmful content. We share our data
and code as open-source tools for the detection of automated fake news or
propaganda in order to help local human moderators and common users in those
countries.

##  9 Acknowledgements

The research was completed while the second author was still affiliated with
Freie Universität Berlin.

## Declarations

  * •

Funding
Not applicable

  * •

Competing interests
The corresponding author VS is of Ukrainian nationality but is not affiliated
with any of the Ukrainian governmental or private institutions. Other authors
have no competing interests to declare.

  * •

Ethics approval
Not applicable

  * •

Consent to participate
Not applicable

  * •

Consent for publication
Not applicable

  * •

Availability of data and materials
All data is available in a fully anonymous until publication GitHub
depository101010https://github.com/anonrep/pro-kremlin_propaganda.

  * •

Code availability
All code is available in the same GitHub repository.

  * •

Authors’ contributions
Conceptualization: [Veronika Solopova]; Methodology: [Veronika Solopova, Oana-
Iuliana Popescu]; Formal analysis and investigation: [Veronika Solopova, Oana-
Iuliana Popescu]; Writing - original draft preparation: [Veronika Solopova];
Writing - review and editing: [Veronika Solopova, Tim Landgraf, Oana-Iuliana
Popescu]; Supervision: [Tim Landgraf, Christoph Benzmüller].

##  Appendix A Hyper-parameters

We performed a Grid search and found out that the best results are achieved
with Radial basis function kernel, gamma=100, and C=46 parameters.

For the neural network used with linguistic features, our best setup was
achieved with 2 hidden layers, a Limited-memory BFGS solver, tanh activation
function, and alpha=1e-5.

For the linear model used with BERT, we used a learning rate of 1e-4, 4 epochs
and batch size 16.

## References

  * Smith (2022) Smith, B. L. propaganda. _Encyclopedia Britannica_ 2022, _24_.
  * 202 (2022) propaganda. _n. OED Online. Oxford University Press_ 2022, _11_.
  * Khvostunova (2022) Khvostunova, O. Do Russians Really “Long for War” in Ukraine? _Foreign Policy Research Institute_ 2022,
  * Roth (2022) Roth, A. Kremlin reverts to type in denial of alleged war crimes in Ukraine’s Bucha. _The Guardian_ 2022,
  * Heritage (2014) Heritage, T. Russia launches new media to lead "propaganda war" with West. _Reuters_ 2014,
  * Paul (2022) Paul, K. Flood of Russian misinformation puts tech companies in the hot seat. _The Guardian_ 2022,
  * Steiger et al. (2021) Steiger, M.; Bharucha, T. J.; Venkatagiri, S.; Riedl, M. J.; Lease, M. The Psychological Well-Being of Content Moderators. 2021.
  * of Excellence (2016) of Excellence, N. S. C. C. _The manipulative techniques of Russian’s Information Campaign, Euro-Atlantic values and Russia’s Strategic Communication in Euro-Atlantic Space Executive Summary_ ; 2016.
  * Carroll (2017) Carroll, J. J. Image and imitation. The visual rhetoric of Pro-Russian Propaganda. _Ideology and Politics_ 2017,
  * Meister (2016) Meister, S. Isolation and propaganda, the roots and instrumnts of Russia’s disinformation campaign. _Transatlantic Academy 2015-16 Paper Series_ 2016,
  * Fortuin (2022) Fortuin, E. “Ukraine commits genocide on Russians”: the term “genocide” in Russian propaganda. _Russ Linguist_ 2022, 313–347.
  * gec (2020) _Pillars of Russia’s Disinformation and Propaganda Ecosystem_ ; 2020\.
  * Narayanan (2017) Narayanan, V. P. Russian Involvement and Junk News during Brexit COMPROP DATA MEMO 2017.10 / December 19th 2017. 2017.
  * Cosentino (2020a) Cosentino, G. Polarize and conquer: Russian influence operations in the united states. In Social Media and the Post-Truth World Order. _Springer_ 2020a, 33–57.
  * Elswah and Howard (2020) Elswah, M.; Howard, P. N. “Anything that Causes Chaos”: The Organizational Behavior of Russia Today (RT). _Journal of Communication_ 2020, _70_ , 623–645.
  * Beskow and Carley (2020) Beskow, D.; Carley, K. _Characterization and Comparison of Russian and Chinese Disinformation Campaigns_ ; 2020; pp 63–81.
  * Rosulek (2019) Rosulek, P. The post-truth age, the fake news industry, the Russian Federation and the Central European area. _Trendy v podnikání_ 2019, _9_ , 46–53.
  * Chudinov et al. (2019) Chudinov, A.; Koshkarova, N.; Ruzhentseva, N. Linguistic Interpretation of Russian Political Agenda Through Fake, Deepfake, Post-Truth. _Journal of Siberian Federal University. Humanities & Social Sciences_ 2019, 1840–1853.
  * Wilbur (2021) Wilbur, D. Propaganda or Not: Examining the Claims of Extensive Russian Information Operations within the United States. 2021, _20_ , 146–156.
  * Snyder (2018) Snyder, T. _The road to unfreedom : Russia, Europe, America / Timothy Snyder_ , first edition. ed.; Tim Duggan Books New York, 2018; p 359 pages ;.
  * Sly (2017) Sly, J. Timothy Snyder. On Tyranny: Twenty Lessons from the Twentieth Century . New York: Tim Duggan Books, 2017. 126p. Paper, $7.99 (ISBN 978-0-8041-9011-4). _College and Research Libraries_ 2017, _78_ , 868.
  * Alsmadi and O’Brien (2020) Alsmadi, I.; O’Brien, M. How Many Bots in Russian Troll Tweets? _Information Processing & Management_ 2020, _57_ , 102303\.
  * Asr and Taboada (2019) Asr, F. T.; Taboada, M. Big Data and quality data for fake news and misinformation detection. _Big Data & Society_ 2019, _6_ , 2053951719843310\.
  * Antoun et al. (2020) Antoun, W.; Baly, F.; Achour, R.; Hussein, A.; Hajj, H. State of the Art Models for Fake News Detection Tasks. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT). 2020; pp 519–524.
  * Mahyoob et al. (2020) Mahyoob, M.; Algaraady, J.; Alrahaili, M. Linguistic-Based Detection of Fake News in Social Media. _International Journal of English Linguistics_ 2020, _11_ , 99.
  * Rashkin et al. (2017) Rashkin, H.; Choi, E.; Jang, J. Y.; Volkova, S.; Choi, Y. Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, 2017; pp 2931–2937.
  * Li et al. (2021) Li, X.; Xia, Y.; Long, X.; Li, Z.; Li, S. Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English. 2021,
  * Yu et al. (2021) Yu, S.; Martino, G. D. S.; Mohtarami, M.; Glass, J. R.; Nakov, P. Interpretable Propaganda Detection in News Articles. _CoRR_ 2021, _abs/2108.12802_.
  * Tundis et al. (2020) Tundis, A.; Mukherjee, G.; Mühlhäuser, M. Mixed-Code Text Analysis for the Detection of Online Hidden Propaganda. Proceedings of the 15th International Conference on Availability, Reliability and Security. New York, NY, USA, 2020.
  * Oliinyk et al. (2020) Oliinyk, V.-A.; Vysotska, V.; Burov, Y.; Mykich, K.; Fernandes, V. B. Propaganda Detection in Text Data Based on NLP and Machine Learning. MoMLeT+DS. 2020.
  * Dadu et al. (2020) Dadu, T.; Pant, K.; Mamidi, R. Towards Detection of Subjective Bias using Contextualized Word Embeddings. _CoRR_ 2020, _abs/2002.06644_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. _CoRR_ 2018, _abs/1810.04805_.
  * UkraineWorld.org (2022) UkraineWorld.org, «Страна»: популярне і проросійське медіа в Україні. _Radio Svoboda_ 2022,
  * Bergengruen (2022) Bergengruen, V. How Telegram Became the Digital Battlefield in the Russia-Ukraine War. _The Time_ 2022,
  * O’Brien (2022) O’Brien, P. How Telegram Became the Digital Battlefield in the Russia-Ukraine War. _France 24_ 2022,
  * Valeriya Safronova (2022) Valeriya Safronova, A. S., Neil MacFarquhar Where Russians Turn for Uncensored News on Ukraine. _The New Your Times_ 2022,
  * Sweney (2022) Sweney, M. Telegram: the app at the heart of Ukraine’s propaganda battle. _The Guardian_ 2022,
  * Mohammad and Turney (2013) Mohammad, S. M.; Turney, P. D. Crowdsourcing a Word-Emotion Association Lexicon. _Computational Intelligence_ 2013, _29_ , 436–465.
  * Gessen (2022) Gessen, M. The War That Russians Do Not See. _The New Yorker_ 2022,
  * Cohen (1960) Cohen, J. A Coefficient of Agreement for Nominal Scales. _Educational and Psychological Measurement_ 1960, 37–46.
  * Powers (2008) Powers, D. Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation. _Mach. Learn. Technol._ 2008, _2_.
  * Grandini et al. (2020) Grandini, M.; Bagli, E.; Visani, G. Metrics for Multi-Class Classification: an Overview. _ArXiv_ 2020, _abs/2008.05756_.
  * Breiman (2004) Breiman, L. Random Forests. _Machine Learning_ 2004, _45_ , 5–32.
  * Marples (2006) Marples, D. R. Stepan Bandera: The resurrection of a Ukrainian national hero, Europe-Asia Studies, 58:4, 555-566. _DOI: 10_ 2006, 1080/09668130600652118.
  * Gaufman (2016) Gaufman, E. _Security Threats and Public Perception: Digital Russia and the Ukraine Crisis_ ; New Security Challenges, 2016; p 107.
  * Daniel Boffey and Borger (2022) Daniel Boffey, D. S.; Borger, J. Mariupol theatre bombing killed 300, Ukrainian officials say. _The Guardian_ 2022,
  * Harding (2022) Harding, L. How Ukrainian defiance has derailed Putin’s plans. _The Guardian_ 2022,
  * Horton et al. (2022) Horton, J.; Sardarizadeh, S.; Schraer, R.; Robinson, O.; Coleman, A.; Palumbo, D.; Cheetham, J. Bucha killings: Satellite image of bodies site contradicts Russian claims. _Reality Check and BBC Monitoring, BBC News_ 2022,
  * De Cao et al. (2020) De Cao, N.; Schlichtkrull, M.; Aziz, W.; Titov, I. How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking. 2020; pp 3243–3255.
  * Jawahar et al. (2019) Jawahar, G.; Sagot, B.; Seddah, D. What Does BERT Learn about the Structure of Language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy, 2019; pp 3651–3657.