# An Emotion-Aware Multi-Task Approach to Fake News and Rumour Detection using
Transfer Learning

Arjun Choudhry111https://github.com/nikicc/twitter-emotion-recognition
Biometric Research Laboratory
Delhi Technological University
New Delhi, India
<EMAIL_ADDRESS>Inder Khatri111https://github.com/nikicc/twitter-
emotion-recognition
Biometric Research Laboratory
Delhi Technological University
New Delhi, India
<EMAIL_ADDRESS>&Minni Jain
Department of Computer Science
Delhi Technological University
New Delhi, India
<EMAIL_ADDRESS>&Dinesh Kumar Vishwakarma
Biometric Research Laboratory
Delhi Technological University
New Delhi, India
<EMAIL_ADDRESS>
###### Abstract

Social networking sites, blogs, and online articles are instant sources of
news for internet users globally. However, in the absence of strict
regulations mandating the genuineness of every text on social media, it is
probable that some of these texts are fake news or rumours. Their deceptive
nature and ability to propagate instantly can have an adverse effect on
society. This necessitates the need for more effective detection of fake news
and rumours on the web. In this work, we annotate four fake news detection and
rumour detection datasets with their emotion class labels using transfer
learning. We show the correlation between the legitimacy of a text with its
intrinsic emotion for fake news and rumour detection, and prove that even
within the same emotion class, fake and real news are often represented
differently, which can be used for improved feature extraction. Based on this,
we propose a multi-task framework for fake news and rumour detection,
predicting both the emotion and legitimacy of the text. We train a variety of
deep learning models in single-task and multi-task settings for a more
comprehensive comparison. We further analyze the performance of our multi-task
approach for fake news detection in cross-domain settings to verify its
efficacy for better generalization across datasets, and to verify that
emotions act as a domain-independent feature. Experimental results verify that
our multi-task models consistently outperform their single-task counterparts
in terms of accuracy, precision, recall, and F1 score, both for in-domain and
cross-domain settings. We also qualitatively analyze the difference in
performance in single-task and multi-task learning models.††*Equal
Contribution

_K_ eywords Multi-task Learning $\cdot$ Emotion Classification $\cdot$
Principal Component Analysis $\cdot$ Plutchik’s emotions $\cdot$ Ekman’s
emotions $\cdot$ Cross-domain analysis.

##  1 Introduction

In recent years, we have witnessed a substantial increase in the usage of
social networking sites, news services, and blogs. This increase in the user
base, and thus, the usage of social media has brought along an increase in the
spread of _fake news_ and _rumours_. According to the work by Meel and
Vishwakarma (2019), fake news and rumours both fall under _false information_.
Fake news refers to the spread of false information under the pretext of being
correct, while rumours are unverified pieces of information that may or may
not be false. While there are noticeable differences in the semantics of a
rumour and a fake news text, machine learning and deep learning approaches
used for the detection of both remain largely similar. Further, both fake news
and rumours have often been used interchangeably in research works. For
example, Ajao et al. (2019) proposed a sentiment-aware approach to fake news
detection, but tested their approach on the PHEME (Kochkina et al., 2018b)
dataset, predominantly used for rumour detection. Thus, both tasks can be
countered with common model architectures and solutions.

Due to the deceptive nature of fake news and rumours, it is difficult for
people to understand which text is real, and which is fake (or a rumour). For
example, Starbird et al. (2014) found that people on social media were unable
to distinguish between rumours and non-rumours for the 2013 Boston Marathon
bombings. For every tweet tweeted debunking a rumour, as many as 44 tweets
were tweeted in support of it. This proves the inability of users on social
media to identify fake news and rumours. Hence, we can say that social media
has been unable to keep fake news and rumours off their platforms. This calls
into question the credibility of social media and the web as a source of
information.

Significant research has been done in recent years towards improved detection
of fake news and rumours. Researchers have focused on a variety of directions
to implement models that can reliably detect fake news and rumours. Some works
have focused on improved classifier architectures for better learning the
semantic and syntactic features that differentiate fake news texts from real
news texts. For example, Han et al. (2020) used Graph Neural Networks with
continual learning to differentiate between the propagation patterns of fake
news and real news on social media. Other works have focused on multi-modal
data to extract features from the text, as well as the accompanying
multimedia. This has been shown to further improve the performance of
classifiers for the detection of fake news and rumours. For example, Singhal
et al. (2019) proposed a multi-modal framework for the detection of fake news,
and found that the performance of classifiers toward the detection of fake
news was significantly improved when compared with models targeting a single
modality. Singhal et al. (2020) further observed improved performance by using
transfer learning for large transformer models.

Some research works have also focused on the extraction of specific features
from the texts, which are found to aid them in the detection of fake news and
rumours. For example, Kochkina et al. (2018a) utilized veracity and stance
detection features for the detection of rumours, and observed noticeably
better performance from their classifiers, as compared to the classifiers
trained without these features. Ajao et al. (2019) used the sentiment ratio
extracted from the text using Latent Dirichlet Allocation and concatenated the
extracted vector for the sentiment ratio with the feature matrix. They
observed improved performance on the PHEME 6 dataset. Some recent works have
also shown the correlation between the legitimacy of a text and its emotion
(Vosoughi et al., 2018).

In this work, we verify the correlation between the legitimacy of a text and
its intrinsic emotion and show that for the same emotion class, real news and
fake news are often represented slightly differently. This work is the full
paper for our abstract (Choudhry et al., 2022) and contains suitable
extensions for a more thorough evaluation. Based on the correlation between
fake news and emotions, we further propose a multi-class emotion-aware multi-
task approach for the detection of fake news and rumours, where classifiers
implicitly learn features associated with fake news or rumour detection, as
well as features associated with emotions portrayed by the text, with the help
of transfer learning. We annotate datasets with their emotion class labels
using transfer learning. We treat fake news detection and rumour detection as
similar tasks and verify our approach’s efficacy on the PHEME 9 dataset for
rumour detection, FakeNews AMT dataset for multi-domain fake news detection,
Celeb and Gossipcop datasets. Our experimental evaluations prove that implicit
extraction of emotion-based features in a multi-task setting helps improve the
model performance across a variety of model architectures and datasets, both
in in-domain and cross-domain settings. Our contributions can be summarized
as:

  * •

We verify the correlation between the legitimacy of a text and its portrayed
emotions by dimensionality reduction. We experimentally use this correlation
for improved fake news and rumour detection by proposing a multi-task
framework with fake news or rumour detection as the primary task, and multi-
class emotion detection as the auxiliary task.

  * •

We show that for the same emotion, rumours and non-rumours are often
represented slightly differently in the feature space upon dimensionality
reduction. These differences will vary from dataset to dataset due to
differences in the domain and other biases. We suggest that these differences
can aid the model to better incorporate this correlation for improved rumours
and fake news detection.

  * •

We further discover that emotion-guided implicit features are more domain-
independent than features extracted by models without the use of emotions, due
to improved performance observed in cross-domain fake news detection.

  * •

We evaluate both Ekman’s and Plutchik’s emotion classes for fake news and
rumour detection, showing that Ekman’s emotion classes lead to similar or
better performance than Plutchik’s emotion classes. To the best of our
knowledge, no previous work on fake news detection considers Ekman’s emotion
classes.

  * •

We evaluate the performance of the single-task and multi-task models in in-
domain and cross-domain settings for different deep learning architectures and
datasets. Based on the results obtained, we verify that emotion-guided
implicit features extracted by our multi-task models aid in cross-domain fake
news and rumour detection.

The organization of our paper is as follows: Section 2 contains the Related
Works and associated literature review, Section 3 contains the Proposed
Methodology, Section 4 contains the Results and Discussion, Section 5 consists
of the Broader Impact and Ethical Considerations regarding our work, Section 6
conveys the Current Limitation of Our Approach, and Section 7 contains the
Conclusion.

##  2 Related Work

In recent years, significant work has been done to tackle the problem of _fake
news_ and _rumour detection_. Bhutani et al. (2019) used a combination of
cosine similarity, Tf-Idf and sentiment scores as features for the detection
of fake news. Pérez-Rosas et al. (2018) presented two novel datasets,
_FakeNews AMT_ and _Celeb_ , for fake news detection across multiple domains,
and used hand-crafted linguistic features and an SVM model for fake news
detection. Saikh et al. (2019) treated fake news detection as a text
classification task and presented two deep learning models for fake news
detection on FakeNews AMT and Celeb datasets.

Bhattacharya et al. (2021) proposed a Blockchain and LSTM-based classification
technique incorporating Q-GloVe word embeddings for the detection of fake
news. They efficiently used Blockchain to protect real news from adversaries
to prevent its alteration. Verma et al. (2021) curated a new generalized
dataset by combining four previously available benchmark datasets, to reduce
bias associated with domains specific to each dataset. They generated
embeddings from the linguistic features of the combined dataset and were used
for classification using an ensemble of multiple machine learning classifiers.
Shahid et al. (2022) further presented a detailed survey of topics,
challenges, and future areas of research associated with fake news detection,
providing a detailed overview of dissemination, detection, and mitigation of
fake news. They further focused on a specific potential future research
agenda, i.e., using artificial intelligence explainable fake news credibility
system.

Some contemporary works have studied rumour detection from the direction of
network sciences, focusing on its evolution and impact on users. Cheng et al.
(2021b) modelled a misinformation network for the spread of rumours during the
COVID-19 pandemic. Each node in the network represented a different rumour,
while the edges indicated the similarity between the rumours. This work showed
that information propagation could be controlled by deleting some central
nodes in the misinformation graph, and further proposed a deep learning
framework to predict Twitter posts that could act as central nodes. Verma et
al. (2022), on the other hand, studied the impact of misinformation about the
COVID-19 pandemic on the mental health of people and found that individuals
who consumed a higher amount of misinformation suffered from high stress and
anxiety, and exhibited an increased tendency for suicidal thoughts.

Recent research works have incorporated advanced mainstream deep learning
approaches like Generative Adversarial Networks (GANs) and Transformers for
fake news detection and rumour detection to train detectors in an unsupervised
manner. Cheng et al. (2021a) proposed a GAN-based layered model for rumour
detection, which was trained in a label-free manner without using any verified
database. The Generator and the discriminator were trained adversarially in a
sequential manner, where the generator produced rumours by fabricating the
non-rumours, and the discriminator learnt to detect rumours and non-rumours
with the help of the produced rumours. Roy et al. (2022) proposed a
transformer-based architecture with a modified multi-head attention mechanism
used to learn multiple contextual correlations for enhancing feature
representation learning.

Kochkina et al. (2018a) proposed a branch LSTM-based multi-task learning model
trained using veracity and stance detection as auxiliary tasks to rumour
detection. The model trained with both veracity and stance as auxiliary tasks
outperformed single-task models, as well as multi-task models with either
veracity or stance as an auxiliary task. Building upon this work, Cheng et al.
(2020) also used multi-task learning for the rumour classification, but unlike
Kochkina et al. (2018a), they tackled all four component tasks related to
rumour detection using the same model: rumour detection, rumour tracking,
stance classification, and veracity classification. To improve the overall
performance on the component tasks, they utilized an LSTM-based Variational-
Autoencoder (VAE) to get classifier-friendly latent representations. Our
proposed approach also leverages multi-task learning for rumour and fake news
detection. While the above works utilize multi-task learning to tackle the
different component tasks relate to rumour detection simultaneously, we
exploit multi-task learning to improve implicit feature representation using
emotion classification as an auxiliary task.

Various works in fake news and rumour detection have shown the efficacy of
using sentiment analysis and text polarity for fake news and rumour detection.
Ajao et al. (2019) calculated _emoratio_ (ratio of negative polarity to
positive polarity in text) for the PHEME dataset, showing a significant
difference in values for rumours and non-rumours. Augmentation of emoratio
into the feature matrix showed noticeable improvements to their model’s
performance. Yang et al. (2018) showed that real news has higher median values
and lower standard deviation for both positive and negative sentiments than
fake news. The median values for negative sentiment for fake news were also
higher than the median values for positive sentiment. Both indicate that more
negative sentiment is associated with fake news.

Some recent works have started proving that there exists an affinity between
certain emotions and the legitimacy of news, and this affinity can be
leveraged for better detection of fake news. K et al. (2020) introduced a
technique to develop emotion-amplified (_emotionized_) text representations
for the detection of fake health news articles, based on the assumption that
fake and real health news articles portray a different affective character,
and this can be used to aid in the detection of fake news. Vosoughi et al.
(2018) analyzed all of the verified fake and real news stories propagated on
Twitter from 2006 to 2017 and labeled them as real or fake using information
extracted from six fact-checking organizations. Upon evaluating the replies on
each of these stores using the lexicon curated by the National Research
Council Canada (NRC) based on Plutchik’s basic emotions, they detected that
false stories typically garnered surprise, disgust and fear in replies, while
real stories garnered anticipation, joy, trust and sadness.

Fersini et al. (2020) proposed an approach incorporating the modeling of users
based on four characteristics, i.e. stylometry, personality, emotions, and
feed embeddings, for the detection of fake news spreaders for the Profiling
Fake News Spreaders on Twitter shared task at PAN 2020 (Pardo et al., 2020).
They observed that stylometry, emotions, and embeddings contributed positively
to the detection capability of the model. Kumari et al. (2021) proposed a
common framework for novelty detection, sentiment analysis, emotion analysis,
and fake news detection using a hybrid LSTM and BERT-based model. They map the
emotions into two classes, _emotion true_ and _emotion false_ , which are used
in their framework. Guo et al. (2019) proposed a novel Emotion-based Fake News
Detection (EFN) framework utilizing the emotions extracted from both the
content and the comments, i.e., the emotions portrayed by the publishers and
the users respectively, apart from the content, for the detection of fake
news. Zhang et al. (2021) further built upon the work by Guo et al. (2019), by
extracting the _Emotion Gap_ between the _publisher emotion_ and the _social
emotion_ , and further proposed a _Dual Emotions Features_ set for the
detection of fake news. Most of the above emotions-based works present
analytical studies about the correlation between fake news and emotions on
different datasets. Only a few of them actually use emotions for detection.
Further, most of these works which utilize emotions to aid in rumour or fake
news detection, take the emotion labels as input or exploit the emotional
inconsistencies between the main content and the comments. This makes them
highly dependent on additional inputs like emotion labels, comments, and
thread information for evaluation. Our work, in contrast, intends to align the
feature representations according to the emotion labels in a multi-task
learning setup, which does not require any additional inputs during inference
time.

Some very recent works, done parallelly to our original abstract (Choudhry et
al., 2022), incorporate multi-task learning and utilize emotion recognition as
an auxiliary task to aid in their main classification tasks. Saha et al.
(2022) proposed a combined sentiment and emotion-aided framework for tweet act
classification. They utilized the multi-modal feature extractors to obtain the
textual and emoticon features. Another work (Kamila et al., 2022) investigated
the impact of emotions on the temporal orientation of tweets, and further
proposed a multi-task learning-based approach to leverage the influence of
emotions on the temporal orientation of the text. It parallelly trained the
CNN and Bi-GRU networks to learn the shared input embeddings for the primary
task classifier, i.e., to detect the temporal orientation among three the
classes (past, present, and future), and the auxiliary task classifier, i.e.,
to detect emotions among four labels (joy, sadness, anger or fear). Although
these works utilize emotion classification as an auxiliary task in multi-task
frameworks and are similar to our proposed approach in terms of architecture,
their base problems (tweet act classification and temporal orientation of
tweets) are significantly different from our task.

We observe that, contrary to the claim by Vosoughi et al. (2018), rumours and
non-rumours cannot always be associated with specific types of emotions, as
these may vary depending on the dataset. Thus, for smaller datasets, it may
not be correct to generalize rumours being associated with specific emotion
classes respectively. We incorporate multi-class emotion classification to aid
in fake news and rumour detection on various datasets for in-domain and cross-
domain evaluation. We further use the basic emotions considered by
psychologists defined by Ekman and Plutchik respectively, which form the
complete set of basic emotions, in relation to fake news detection. We portray
the efficacy of using multi-class emotion classification as an auxiliary task,
which aids in generalizing a model for use over a wider set of datasets.

Figure 1: Flowchart representing our proposed methodology.

##  3 Proposed Methodology

This section elaborates on our proposed methodology and is structured as
follows: Subsection 3.1 elucidates the dataset preprocessing steps we used.
Subsection 3.2 elaborates our approach for choosing the basic emotion classes
for annotation and annotating the datasets. Subsection 3.3 explains the
intuition behind the use of emotions as an auxiliary feature for fake news and
rumour detection and shows why they aid in improving the performance of
classifiers. Subsection 3.4 elaborates upon the multi-task learning framework
we used for various classifiers to incorporate emotion features into the
model. Subsection 3.5 elucidates the process for cross-domain analysis, the
reason why we evaluated cross-domain results, and their implication. Figure 1
shows a general overview of our work.

###  3.1 Preprocessing the Data

In this work, we use the PHEME 9 (Kochkina et al., 2018b), FakeNews AMT
(Pérez-Rosas et al., 2018), Celeb (Pérez-Rosas et al., 2018) and Gossipcop
datasets for evaluating our approach. We preprocess all the datasets using the
same approach. We first convert all text to lowercase. We define a contracted
verb’s dictionary with general as well as specific cases that commonly occur
in normal use. Using this contracted verb’s dictionary, we de-contract verb
forms to their root forms (eg. “I’ll” to “I will”. We then remove all
punctuation marks and numbers. Finally, we tokenize the text and store it in a
list for further use.

###  3.2 Emotion Classes & Annotating the Datasets

Incorporating multi-class emotion classification as an auxiliary task in our
multi-task framework poses two challenges: choosing a set of basic emotions,
and annotating the datasets with the chosen emotion classes. There are two
widely accepted theories on basic emotion classes, proposed by Plutchik (1982)
and Ekman (1992) respectively. Plutchik (1982) designed a wheel with 8 basic
emotions (Joy, Surprise, Trust, Anger, Anticipation, Sadness, Disgust, Fear)
based on his _Ten Postulates_. Combinations of these 8 basic emotions can be
used to form 56 other emotions at one intensity level. Previous psychological
works (Vosoughi et al., 2018) on the correlation between emotions and fake
news have made use of Plutchik’s basic emotion classes using lexicon-based
approaches. During a cross-cultural study, Ekman (1992) inferred that there
are 6 basic emotions (Joy, Surprise, Anger, Sadness, Disgust, Fear). Each
emotion was considered a discrete category, rather than an individual emotion.
Bann and Bryson (2014) later found Ekman’s basic emotion set to be the most
semantically distinct.

We incorporate the use of Ekman’s basic emotions in our work as well as
Plutchik’s basic emotions and see if they lead to significantly different
results in the detection of fake news. Due to the unavailability of fake news
datasets annotated with emotion classes, we use the _Unison model_
111https://github.com/nikicc/twitter-emotion-recognition (Colneric and Demsar,
2018) to generate both Plutchik and Ekman emotion tags to annotate our
datasets. The Unison model achieved state-of-the-art results for the
classification of Twitter data into their respective emotions. Colneric and
Demsar (2018) used a dataset of 73 billion tweets to train and test their
approach. Annotation using the Unison model enables us to implicitly transfer
some emotion-based features from the Unison model to our classifiers when
trained in a multi-task setting. While the Unison model achieves an F1-score
of around 0.70 for various basic emotion classes, it is a relatively robust
model trained over 73 billion tweets from a variety of domains. It is also the
only available model which can annotate for both Ekman’s and Plutchik’s basic
emotion classes, an important factor for evaluating any differences between
the two emotion sets. Further, as it is trained on an extremely large real-
world dataset, it is less likely to show biases than smaller models trained on
smaller corpora. However, in the future, as better and more robust emotion
classification models become available, we recommend their use to minimize
misannotations of emotion tags and further optimize performance. We compare
the performance of various classifiers in multi-task settings on both emotion
sets.

###  3.3 Intuition behind use of Emotions

We incorporate emotion-guided features in our fake news detection model by
implementing a multi-task framework incorporating a multi-class emotion
classification task in addition to the original fake news or rumour detection
task. We further hypothesize that there is a prominent correlation between the
legitimacy of a text and its intrinsic Ekman’s and Plutchik’s emotion, and
within the same emotion, real and fake texts are sometimes represented
differently. We further suggest that this correlation can positively affect a
classifier’s capability in the detection of fake news or rumours.

Figure 2 represents 3-dimensional Principal Component Analysis (PCA) graphs
for the PHEME dataset, computed using embeddings generated from the Unison
model for Ekman emotions. The 3-dimensional principal components are computed
over the whole PHEME dataset but are plotted separately for rumours and non-
rumours. If there wasn’t a correlation between emotions and the legitimacy of
text, then the two graphs obtained should have been nearly identical, except
for the volume of sample points. However, we observe some stark differences
between the plots for rumours and non-rumours.

(A)

(B)

Figure 2: 3-dimensional Principal Component Analysis (PCA) plots for Rumours
and Non-rumours from the PHEME 9 dataset, colored by Ekman emotions. (A):
Upper View (B): Lower View

(A)

(B)

Figure 3: 3-dimensional Principal Component Analysis (PCA) plots for Rumours
and Non-rumours from the PHEME 9 dataset, colored by Plutchik emotions. (A):
Upper View (B): Lower View

Non-rumours show better-formed clusters and a higher percentage of Joy and
Fear than rumours, which are more scattered and show a higher percentage of
Sadness and Surprise. A similar conclusion can be drawn from Figure 3, which
represents 3-dimensional Principal Component Analysis (PCA) graphs for the
PHEME dataset, computed using embeddings generated from the Unison model for
Plutchik emotions. Non-rumours show better-formed clusters and a higher
percentage of Trust and Fear than rumours, which are again more scattered,
with a higher percentage of Sadness and Surprise.

We further observe that for the same emotion, rumours and non-rumours clusters
are plotted differently. Thus, even when two texts portray the same emotion,
they are sometimes represented differently in the feature space for rumours
and non-rumours. While the distribution of texts into emotion classes will
vary from dataset to dataset, primarily due to differences in domains and
various biases in each dataset, the fact that distributional and positional
variations in the feature space occur between rumours and non-rumours for the
same emotion should hold true for a variety of datasets, as these help form
better decision boundaries for improved rumour or fake news detection in our
multi-task models, leading to higher performance. The comparative non-emotion
models, in comparison, perform worse and aren’t able to correctly label texts
as rumours or non-rumours (or fake and real news) as they are not guided by
the variations in emotions between the classes. This indicates that a
correlation exists between the legitimacy of a text and its underlying
emotion, thereby supporting our proposition that emotion tags can aid in
rumour detection and fake news detection.

Figure 4: Generalized multi-task learning framework with fake news detection
as the primary task and multi-class emotion classification as the auxiliary
task.

###  3.4 Multi-task Learning for Fake News Detection

Multi-task learning (MTL) uses shared representations between the primary and
auxiliary tasks for better extraction of common features, which can otherwise
get ignored, and to generalise better on the primary task. We train our models
to predict both the legitimacy and emotion of a text to verify the
relationship between its legitimacy and emotion and evaluate them against
their single-task (STL) counterparts. We further leverage the domain tags for
texts in FakeNews AMT dataset and evaluate the performance of multi-task
learning models predicting the legitimacy and domain of the text, to compare
with multi-task learning models predicting the legitimacy and emotion of the
text. Figure 4 shows the multi-task learning architecture we use, with fake
news detection as the primary task, and emotion classification as the
auxiliary task.

###  3.5 Cross Domain Analysis

Previous works by Pérez-Rosas et al. (2018) and Saikh et al. (2019) show that
classifiers exhibit a significant drop in performance when training the
classifier on a dataset from one domain and testing it on a dataset from a
different domain, as compared to its performance on the test set from its own
domain. This is primarily due to the implicit extraction of domain-dependent
features by the models, which reduces its efficacy in correctly detecting fake
news from other domains. We evaluate the performance of LSTM and CNN-LSTM
models in both single-task and multi-task settings for cross-domain analysis
by training them on FakeNews AMT dataset and testing on Celeb dataset and vice
versa. This helps us verify our hypothesis of the applicability of emotions
for better generalization across domains and datasets for the detection of
fake news.

##  4 Experimental Analysis & Results

This section describes the experimental analysis, results, and implications.
Subsection 4.1 describes the datasets that we have used. Subsection 4.2
describes our $\alpha$-$\beta$ tuning approach applied to ensure optimal
performance in multi-task settings. Subsection 4.3 elucidates the various
standard classification models we have used with our framework to show the
wide applicability of our approach. It further explains how we have balanced
the loss weights of the two branches in the multi-task learning models for
optimum performance. Subsection 4.4 contains the performance metrics we have
used to evaluate the classifiers. Subsection 4.5 contains the results and the
implications we draw from them.

###  4.1 Datasets

We have used the PHEME 9 (Kochkina et al., 2018b), FakeNews AMT (Pérez-Rosas
et al., 2018), Celeb (Pérez-Rosas et al., 2018) and
Gossipcop222https://www.gossipcop.com/ datasets for evaluating our approach in
this paper. The PHEME 9 dataset contains rumours and non-rumours posted on
Twitter for 9 different events. Table 1 illustrates the distribution of the
PHEME 9 dataset. The FakeNews AMT dataset is a multi-domain fake news dataset
covering business, education, entertainment, politics, sports, and technology,
while the Celeb dataset contains news about celebrities only. The Gossipcop
dataset contains entertainment news articles scrapped from the web, containing
over 12,000 samples. Hereon, the terms rumour and non-rumour are used
specifically in the context of the PHEME 9 dataset or its evaluation, while
the term fake news is used specifically in the context of the FakeNews AMT,
Celeb and Gossipcop datasets, or their evaluation.

Table 1: Distribution of the PHEME 9 dataset by the event. The total number of non-rumours are 67 percent more than the number of rumours. Events | Threads | Tweets | Rumours | Non-rumours
---|---|---|---|---
Charlie Hebdo | 2,079 | 38,268 | 458 | 1,621
Sydney Siege | 1,221 | 23,996 | 522 | 699
Ferguson | 1,143 | 24,175 | 284 | 859
Ottawa shooting | 890 | 12,284 | 470 | 420
Germanwings-crash | 469 | 4,489 | 238 | 231
Putin missing | 238 | 835 | 126 | 112
Prince Toronto | 233 | 902 | 229 | 4
Gurlitt | 138 | 179 | 61 | 77
Ebola Essien | 14 | 226 | 14 | 0
Total | 6,425 | 105,354 | 2,402 | 4,023

###  4.2 Parameter Tuning for Multi-task Models

We optimize the performance of the multi-task learning models by rebalancing
the weights for the loss function for the primary and auxiliary branches of
the models. We take the loss weight for the emotion classification branch as
__, and the loss weight for the fake news or rumour detection branch as __.
The relation between __and __is in Equation 1:

$\displaystyle\beta$ $\displaystyle=1-\alpha$ (1)

The primary branch, tasked with the detection of the legitimacy of the text,
gives a logits vector $Z_{t}$ of size $2\times 1$ as output, as shown in
Equation 2. The auxiliary branch, which detects the Ekman or Plutchik emotion
portrayed by the text, gives a logits vector $Z_{e}$ of size $6\times 1$ (for
Ekman emotion classes) or $8\times 1$ (for Plutchik emotion classes) as
output, as shown in Equation 3.

$\displaystyle Z_{t}$ $\displaystyle=\begin{bmatrix}Z_{t}^{0}\\\ Z_{t}^{1}\\\
\end{bmatrix}$ (2)

$\displaystyle Z_{e}$ $\displaystyle=\begin{bmatrix}Z_{e}^{0}\\\ Z_{e}^{1}\\\
\vdots\\\ Z_{e}^{n-1}\end{bmatrix}$ (3)

$Z_{e}$ and $Z_{t}$ are further activated using the Softmax activation
function to generate the probability distributions $\hat{y_{t}}$ and
$\hat{y_{e}}$ for the legitimacy classes and emotion classes respectively.
$\hat{y_{t}}$ is represented as shown in Equation 4

$\displaystyle\hat{y_{t}}$ $\displaystyle=\begin{bmatrix}\hat{y_{t}^{0}}\\\
\hat{y_{t}^{1}}\\\ \end{bmatrix}=Softmax(Z_{t})$ (4)

where $\hat{y_{t}^{j}}$ is defined as Equation 5

$\displaystyle\hat{y_{t}^{j}}=\frac{e^{z_{t}^{j}}}{\sum_{k=0}^{1}e^{z_{t}^{k}}}$
(5)

Similarly, for the emotion branch, we get $\hat{y_{e}}$ as shown in Equation 6

$\displaystyle\hat{y_{e}}$ $\displaystyle=\begin{bmatrix}\hat{y_{e}^{0}}\\\
\hat{y_{e}^{1}}\\\ \vdots\\\ \hat{y_{e}^{n-1}}\end{bmatrix}=Softmax(Z_{e})$
(6)

where $\hat{y_{e}^{j}}$ is defined as in Equation 7.

$\displaystyle\hat{y_{e}^{j}}=\frac{e^{z_{e}^{j}}}{\sum_{k=0}^{n}-1e^{z_{e}^{k}}}$
(7)

Further, we use cross entropy loss to penalize our model for both the fake
news detection branch and the emotion detection branch, as shown in Equations
8 and 9.

$\displaystyle Loss_{t}=-\sum_{i=0}^{1}\hat{y_{t}^{i}}*log(\hat{y_{t}^{i})}$
(8)

$\displaystyle Loss_{e}=-\sum_{i=0}^{n-1}\hat{y_{e}^{i}}*log(\hat{y_{e}^{i})}$
(9)

The total loss for the multi-task learning models is defined in Equation 10.

$\displaystyle Loss_{(}Total)=\alpha\times Loss_{e}+\beta\times Loss_{t}$ (10)

Therefore, using Equations 1, 8, 9 and 10, we get 11.

$\displaystyle
Loss_{Total}=\alpha\times-\sum_{i=0}^{n-1}\hat{y_{e}^{i}}*log(\hat{y_{e}^{i})}$
$\displaystyle+(1-\alpha)\times-\sum_{i=0}^{1}\hat{y_{t}^{i}}*log(\hat{y_{t}^{i})}$
(11)

###  4.3 Classification Models and Hyperparameters

We have evaluated the performance of a variety of commonly used deep learning
architectures in STL and MTL settings to show the wide applicability of our
approach. Across all datasets, we have trained LSTM, CNN-LSTM, and BERT
(Devlin et al., 2019) models in both MTL and STL settings. The CNN layer in
the CNN-LSTM model aids in mirroring the local and position invariant features
and preserving n-gram representations, while the LSTM layer helps to capture
the long-term dependencies between word sequences better. For the PHEME 9
dataset, we have also evaluated the performance of CNN, CapsuleNet (Sabour et
al., 2017) and HAN (Yang et al., 2016) models, which have previously been used
for rumour detection (Guo et al., 2018). Comparison across multiple models
enables us to better judge the efficacy of using emotion classification as an
auxiliary task in a multi-task learning setup for fake news and rumour
detection. This further ensures that our approach is applicable across a
variety of architectures, and isn’t just dependent on the model. We have made
our code available online.333https://github.com/Arjun7m/Emotion-Aware-MTL-FND-
TCSS

We used the pre-trained _bert-base-uncased_ model, available in the
Huggingface Transformers (Wolf et al., 2020) library, and further fine-tuned
it for our task. Additional dense layers with softmax activation have been
added to the ends of each branch, to predict the legitimacy of the text and
its emotion. The single-task and multi-task BERT models were trained for 5
epochs with a batch size of 32 on the PHEME 9 dataset, while for the FakeNews
AMT and Celeb datasets, the BERT models have a batch size of 64, and were
trained for up to 50 epochs with early stopping to prevent over-fitting.

The CNN layer in the CNN and CNN-LSTM models consists of 32 filters with a
filter size of 5, followed by a max pooling layer. The LSTM layer in the LSTM
and CNN-LSTM models has 100 units. The CNN, LSTM and CNN-LSTM models were
trained for 5 epochs, with a batch size of 64. The HAN and CapsuleNet models
were trained for 5 epochs, with a batch size of 32. All models were trained
using the Adam optimizer, with a learning rate of 2e-5 and cross-entropy loss
function. The models trained for evaluation on the Gossipcop dataset follow
the same parameters as those for PHEME 9.

During loss weights rebalancing, we took the initial value of $\alpha$ as
0.20, and after the training and testing of a model for a given set of values
for $\alpha$ and $\beta$, we incremented $\alpha$ by 0.05, consequently
decreasing $\beta$ by 0.05. We trained each model for thirteen values of
$\alpha$ and $\beta$, till $\alpha$ reaches 0.80, and consequently $\beta$
reaches 0.20. Figure 5 illustrates the line graphs showing the variation in
the accuracy of CNN, LSTM, and CNN-LSTM models on the PHEME 9 dataset with
variations in $\alpha$ and $\beta$ for Ekman’s and Plutchik’s emotions.

###  4.4 Performance Metrics

To compare the performance of the classifiers in single-task and multi-task
settings, we calculated the metrics accuracy, precision, recall and F1-score
for all models.

To calculate these metrics, we need to first determine the number of True
Positives, True Negatives, False Positives, and False Negatives for our models
using the confusion matrix. These metrics are defined as:

  * •

True Positive: Test samples whose labels were correctly predicted as Positive.

  * •

True Negative: Test samples whose labels were correctly predicted as Negative.

  * •

False Positive: Test samples whose labels were incorrectly predicted as
Positive, but were in fact Negative.

  * •

False Negative: Test samples whose labels were incorrectly predicted as
Negative, but were in fact Positive.

Based on the number of True Positives (TP), True Negatives (TN), False
Positives (FP), and False Negatives (FN), accuracy, precision, recall and
F1-score are defined as in Equations 12, 13, 14 and 15.

$\displaystyle Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (12)

$\displaystyle Precision=\frac{TP}{TP+FP}$ (13)

$\displaystyle Recall=\frac{TP}{TP+FN}$ (14)

$\displaystyle F1-score=\frac{2TP}{2TP+FP+FN}$ (15)

Accuracy is used to evaluate the model performance across all classes when all
classes are given equal importance. Mathematically, it represents the ratio of
the number of correct predictions by a model to the total number of samples in
the testing set.

Precision exhibits a model’s performance in classifying a sample as positive.
It is a measure of _exactness_ and tends to reduce if the number of samples in
the minority classes is increased. Mathematically, it represents the ratio of
the number of labels that are correctly predicted as positive to those that
are labeled as positive.

Recall exhibits a model’s performance in of correctly predicting true
positives (TPs). It tends to increase if the number of samples in the minority
classes is increased. Mathematically, it represents the ratio of the number of
samples that are correctly predicted as positive to those that are actually
positive.

F1-score acts as an alternative to accuracy, as it takes into account
precision and recall scores of a model equally, but doesn’t require the value
of the total number of observations. Mathematically, F1-score is represented
as the harmonic mean of precision and recall.

Table 2: Performance evaluation on FakeNews AMT and Celeb datasets for fake news detection, using accuracy, precision, recall, and F1-Score. Multi-task learning models outperform their single-task counterparts. Dataset | Model | Setting | Accuracy | Precision | Recall | F1
---|---|---|---|---|---|---
FAMT | LSTM | STL | 0.725 | 0.829 | 0.566 | 0.673
MTL (Ekman) | 0.775 | 0.753 | 0.816 | 0.784
MTL (Plutchik) | 0.758 | 0.712 | 0.866 | 0.781
MTL (Domain) | 0.775 | 0.770 | 0.783 | 0.776
CNN-LSTM | STL | 0.733 | 0.725 | 0.750 | 0.737
MTL (Ekman) | 0.758 | 0.754 | 0.766 | 0.760
MTL (Plutchik) | 0.766 | 0.735 | 0.833 | 0.781
MTL (Domain) | 0.758 | 0.803 | 0.683 | 0.738
BERT | STL | 0.816 | 0.851 | 0.766 | 0.806
MTL (Ekman) | 0.875 | 0.868 | 0.883 | 0.875
MTL (Plutchik) | 0.866 | 0.866 | 0.867 | 0.866
MTL (Domain) | 0.866 | 0.854 | 0.883 | 0.868
Celeb | LSTM | STL | 0.736 | 0.780 | 0.639 | 0.702
MTL (Ekman) | 0.752 | 0.758 | 0.721 | 0.739
MTL (Plutchik) | 0.712 | 0.735 | 0.639 | 0.684
CNN-LSTM | STL | 0.704 | 0.740 | 0.606 | 0.666
MTL (Ekman) | 0.712 | 0.692 | 0.737 | 0.714
MTL (Plutchik) | 0.696 | 0.744 | 0.573 | 0.648
BERT | STL | 0.816 | 0.816 | 0.815 | 0.815
MTL (Ekman) | 0.856 | 0.859 | 0.854 | 0.855
MTL (Plutchik) | 0.880 | 0.881 | 0.879 | 0.879
Table 3: Performance evaluation on the PHEME 9 dataset for rumour detection, using accuracy, precision, recall, and F1-score. MTL (Ekman) and MTL (Plutchik) models outperform their STL counterparts across most metrics. Dataset | Model |  Setting |  Accuracy |  Precision |  Recall |  F1
---|---|---|---|---|---|---
PHEME 9 | CNN | STL | 0.870 | 0.819 | 0.826 | 0.822
MTL (Ekman) | 0.874 | 0.843 | 0.800 | 0.822
MTL (Plutchik) | 0.875 | 0.825 | 0.834 | 0.830
LSTM | STL | 0.857 | 0.807 | 0.800 | 0.804
MTL (Ekman) | 0.880 | 0.835 | 0.837 | 0.836
MTL (Plutchik) | 0.881 | 0.847 | 0.823 | 0.835
CNN-LSTM | STL | 0.860 | 0.786 | 0.849 | 0.816
MTL (Ekman) | 0.876 | 0.822 | 0.843 | 0.832
MTL (Plutchik) | 0.872 | 0.835 | 0.809 | 0.821
CapsuleNet | STL | 0.853 | 0.818 | 0.748 | 0.782
MTL (Ekman) | 0.858 | 0.804 | 0.789 | 0.797
MTL (Plutchik) | 0.863 | 0.817 | 0.784 | 0.800
HAN | STL | 0.847 | 0.790 | 0.769 | 0.779
MTL (Ekman) | 0.867 | 0.859 | 0.742 | 0.796
MTL (Plutchik) | 0.867 | 0.828 | 0.784 | 0.805
BERT | STL | 0.859 | 0.844 | 0.854 | 0.848
MTL (Ekman) | 0.884 | 0.873 | 0.874 | 0.873
MTL (Plutchik) | 0.874 | 0.859 | 0.870 | 0.864
Table 4: Cross-domain evaluation on FakeNews AMT and Celeb datasets using accuracy, precision, recall, and F1-score. We notice a drop in performance for all models compared to the in-domain setting. MTL models outperform their STL counterparts, indicating better generalization and domain-independent feature extraction. Train Set | Test Set | Model | Setting | Accuracy | Precision | Recall | F1
---|---|---|---|---|---|---|---
FAMT | Celeb | LSTM | STL | 0.560 | 0.557 | 0.475 | 0.513
MTL (Ekman) | 0.576 | 0.554 | 0.672 | 0.607
MTL (Plutchik) | 0.632 | 0.641 | 0.557 | 0.596
CNN-LSTM | STL | 0.568 | 0.559 | 0.540 | 0.550
MTL (Ekman) | 0.632 | 0.600 | 0.737 | 0.661
MTL (Plutchik) | 0.632 | 0.608 | 0.688 | 0.646
Celeb | FAMT | LSTM | STL | 0.558 | 0.549 | 0.650 | 0.595
MTL (Ekman) | 0.591 | 0.557 | 0.883 | 0.683
MTL (Plutchik) | 0.608 | 0.594 | 0.683 | 0.635
CNN-LSTM | STL | 0.525 | 0.529 | 0.450 | 0.486
MTL (Ekman) | 0.575 | 0.563 | 0.666 | 0.610
MTL (Plutchik) | 0.583 | 0.556 | 0.816 | 0.662
Table 5: Performance evaluation on the Gossipcop dataset for fake news detection, using accuracy, precision, recall, and F1-score. MTL (Ekman) and MTL (Plutchik) models outperform their STL counterparts across most metrics. Dataset | Model |  Setting |  Accuracy |  Precision |  Recall |  F1
---|---|---|---|---|---|---
Gossipcop | LSTM | STL | 0.806 | 0.845 | 0.932 | 0.886
MTL (Ekman) | 0.835 | 0.847 | 0.973 | 0.906
MTL (Plutchik) | 0.833 | 0.844 | 0.975 | 0.905
CNN | STL | 0.822 | 0.850 | 0.948 | 0.897
MTL (Ekman) | 0.835 | 0.850 | 0.969 | 0.906
MTL (Plutchik) | 0.836 | 0.853 | 0.964 | 0.905
CNN-LSTM | STL | 0.820 | 0.842 | 0.957 | 0.896
MTL (Ekman) | 0.838 | 0.847 | 0.978 | 0.907
MTL (Plutchik) | 0.837 | 0.845 | 0.980 | 0.907
CapsuleNet | STL | 0.824 | 0.828 | 0.977 | 0.897
MTL (Ekman) | 0.834 | 0.838 | 0.978 | 0.902
MTL (Plutchik) | 0.833 | 0.834 |  0.983 | 0.903
HAN | STL | 0.864 | 0.903 | 0.927 | 0.915
MTL (Ekman) | 0.869 | 0.882 | 0.962 | 0.921
MTL (Plutchik) | 0.870 | 0.888 | 0.955 | 0.920
BERT | STL | 0.851 | 0.803 | 0.716 | 0.745
MTL (Ekman) | 0.866 | 0.823 | 0.750 | 0.777
MTL (Plutchik) | 0.856 | 0.790 | 0.768 | 0.778
RoBERTa | STL | 0.841 | 0.764 | 0.760 | 0.762
MTL (Ekman) | 0.867 | 0.833 | 0.743 | 0.774
MTL (Plutchik) | 0.872 | 0.836 | 0.751 | 0.782

###  4.5 Results and Discussion

We evaluated the performance of our proposed framework on three datasets and
several deep learning models, using the same train-test split for each dataset
across all classifiers. Table 3 illustrates the results of our experiments on
the PHEME 9 dataset, Table 2 compares the results of our MTL models with their
STL counterparts on the FakeNews AMT and Celeb datasets, Table 5 shows the
results of our experiments on the Gossipcop dataset, and Table 4 compares the
performance of our multi-task learning models with their single-task
counterparts in cross-domain settings. Some interesting findings observed are:

MTL models outperform their STL counterparts On comparing the performance of
various classifiers in both single-task and multi-task settings, we observed
an improvement in the performance of the classifiers in multi-task settings
over the single-task setting across most metrics and on all datasets,
verifying the correlation between the legitimacy of a text and its intrinsic
emotion. The LSTM and CNN-LSTM models on the Celeb dataset unexpectedly showed
slightly worse performance in the multi-task settings with Plutchik emotions
compared to the single-task setting. However, the same models in the multi-
task settings with Ekman’s emotions outperform their single-task counterparts
handily in terms of accuracy, recall, and F1 score. Further, the BERT model
trained in the multi-task setting with Plutchik’s emotions handily outperforms
its multi-task counterpart with Ekman’s emotions and its single-task
counterpart across all metrics.

MTL models with Ekman’s and Plutchik’s emotions perform comparably We observed
largely comparable performance between multi-task models trained on Ekman’s
and Plutchik’s emotions across all datasets, except for LSTM and CNN-LSTM
models on the Celeb dataset, which unexpectedly performed worse with
Plutchik’s emotions. Multi-task models with Ekman’s emotions generally
performed slightly better on the FakeNews AMT, Celeb, and Gossipcop datasets
while multi-task models with Plutchik’s emotions performed slightly better on
the PHEME 9 dataset. However, multi-task models trained with either emotion
set generally outperformed the single-task models across most metrics on all
datasets. Thus, we verify the correlation between the legitimacy of a text and
its portrayed Ekman emotion and concur that Ekman’s emotions can also act as
an alternative to Plutchik’s emotions as a feature for fake news detection,
particularly for fake news detection datasets.

MTL models with domain classification as auxiliary task outperform their STL
counterparts Classifiers trained to identify both domain and legitimacy of
news (_MTL (Domain)_ in Table 2) outperformed their single-task counterparts
on the FakeNews AMT dataset, achieving performance comparable to those of
emotion-based MTL models. This indicates a deeper correlation between the
legitimacy of news and its domain. We intend to further build upon this
correlation in future work for building a more generalizable framework.

Performance drops in cross-domain settings for all models, but MTL models
outperform their STL counterparts We observed an all-around drop in the
performance of classifiers in cross-domain settings, both when trained on the
FakeNews AMT dataset and tested on the Celeb dataset, and vice versa. A
similar drop in performance was also observed by Pérez-Rosas et al. (2018) and
Saikh et al. (2019). However, our multi-task models still outperformed their
single-task counterparts, implying improved extraction of shared features and
better generalization. The results further indicate that intrinsic emotions
aid in domain-independent implicit feature extraction for the detection of
fake news.

(A)

(B)

Figure 5: Line graph plot for variation in accuracy of CNN, LSTM and CNN-LSTM
models in multi-task setting on the PHEME 9 dataset with variation in __(and
consequently __) for:(A) Ekman’s emotions (B) Plutchik’s emotions.

Rebalancing weights of the loss function for fake news or rumour detection and
emotion classification branches improves the performance of multi-task models
We observed an improvement in the performance of all multi-task models on
rebalancing the weights of the loss function for each branch’s output. We
observe from Figure 5 that the performance of the CNN, LSTM, and CNN-LSTM
models on the PHEME 9 dataset reaches the maximum for __between 0.35 and 0.55.
Upon increasing the value of __beyond that, we see a sharp decrease in the
performance of the models. This can be attributed to the fact that the loss
function penalizes the emotion classification branch significantly more than
it penalizes the rumour detection branch (in the case of the PHEME 9 dataset).
For __= 0.80, the value of __is 0.20. This leads to a 4:1 ratio in the weights
of the loss function, in favor of the emotion classification branch, which
leads to the model focusing significantly more on the performance of the
emotion classification branch.

Table 6: Qualitative analysis of the results obtained from single-task and
multi-task learning models on the PHEME 9 dataset, where multi-task models
correctly predicted the label. Label 0 corresponds to Non-rumours, while label
1 corresponds to Rumours.

Text | Label |  Ekman Emotion |  Plutchik Emotion | STL |  MTL (Ekman) |  MTL (Plutchik)
---|---|---|---|---|---|---
Family claims #CorneliusGurlitt was mentally unfit. Read the news @Artlyst http://t.co/1tmcwtUYC8 http://t.co/tfH3caD1EP | 1 | Sadness | Surprise | 0 | 1 | 1
Such scary news from Sydney. Thoughts & prayers go out to the hostages &; their family’s. | 0 | Fear | Fear | 1 | 0 | 0
Video footage shows police storming Sydney cafe http://t.co/3U4aWpVmdM #SydneySiege | 0 | Surprise | Surprise | 1 | 0 | 0
This is trending on French twitter #MouradHamydInnocent started by classmates who say Mourad was in class when #CharlieHebdo attack happened | 1 | Fear | Fear | 0 | 1 | 1

Table 7: Qualitative analysis of the results obtained from single-task and
multi-task learning models on the PHEME 9 dataset, where multi-task models
with Ekman’s emotions correctly predicted the label, while multi-task models
with Plutchik’s emotions incorrectly predicted the label. Label 0 corresponds
to Non-rumours, while label 1 corresponds to Rumours.

Text | Label |  Ekman Emotion |  Plutchik Emotion | STL |  MTL (Ekman) |  MTL (Plutchik)
---|---|---|---|---|---|---
Putin juggling enough instability. He would make a live appearance by now to squash death/coup jitters. | 0 | Fear | Trust | 1 | 0 | 1
Australian Mufti condemns hostage taking in #Sydney - Statement #SydneyHostageCrisis #SydneySiege #SydneyCafe http://t.co/n38XZVq5iu | 1 | Surprise | Trust | 0 | 1 | 0
Whenever you want to justify shooting an unarmed person you always say they reached for your gun. George Zimmerman taught us that #Ferguson | 0 | Fear | Fear | 1 | 0 | 1

Table 8: Qualitative analysis of the results obtained from single-task and
multi-task learning models on the PHEME 9 dataset, where multi-task models
with Plutchik’s emotions correctly predicted the label, while multi-task
models with Ekman’s emotions incorrectly predicted the label. Label 0
corresponds to Non-rumours, while label 1 corresponds to Rumours.

Text | Label |  Ekman Emotion |  Plutchik Emotion | STL |  MTL (Ekman) |  MTL (Plutchik)
---|---|---|---|---|---|---
Latest footage from #sydneysiege shows security forces throwing objects into the cafe http://t.co/HdTPHSMjUQ https://t.co/2pu3pWqHhz | 0 | Surprise | Trust | 1 | 1 | 0
It just feels very odd.What’ss going on in #Kremlin? Military coup and putin removal?I’m afraid the next Russia’s leader won’t be any better | 1 | Fear | Trust | 0 | 0 | 1
PM Tony Abbott will hold a press conference on the #SydneySiege in 20 minutes. | 0 | Surprise | Anticipation | 0 | 1 | 0

Table 9: Qualitative analysis of the results obtained from single-task and
multi-task learning models on the PHEME 9 dataset, where multi-task models
with both Ekman’s and Plutchik’s emotions incorrectly predicted the label.
Label 0 corresponds to Non-rumours, while label 1 corresponds to Rumours.

Text | Label |  Ekman Emotion |  Plutchik Emotion | STL |  MTL (Ekman) |  MTL (Plutchik)
---|---|---|---|---|---|---
#sydneysiege is a misnomer. It’s a hostage situation. Inside a cafe. You want 2 know what being under siege looks like? look @ #Gaza. | 1 | Fear | Trust | 1 | 0 | 0
Gun fire exchange in Parliament Hill building #Ottawa #parliament #shooting #Canada #BeSafeOttawa -https://t.co/9RDjNvVkJW via @martinvass2 | 1 | Joy | Trust | 1 | 0 | 0

Upon qualitatively analyzing the results of the single-task and multi-task
learning models, we found many instances where by leveraging the emotion-
specific features, the multi-task learning models were able to accurately
detect the legitimacy of the text. Table 6 summarises some of the tweets found
in the PHEME 9 dataset that was incorrectly labelled by the majority of the
single-task learning models, but using the emotion-based features, the multi-
task learning models correctly predicted the labels. We observed that the
multi-task learning models were able to correctly predict the legitimacy label
for a sample even if the sample portrayed an emotion typically associated with
the other class. This can be attributed to the fact that even within the same
emotions, fake news and real news differ in their semantic and syntactic
features, as shown in the PCA graphs in Figures 2 and 3.

We also found a few instances where multi-task learning models leveraging
either emotion sets incorrectly predicted the labels, sometimes even when the
single-task learning models correctly predicted the labels. These are
summarised in Tables 7, 8 and 9. We can see from these results that multi-task
learning models with Ekman’s emotions can sometimes incorrectly predict
rumours as non-rumours when the intrinsic emotions detected are _Fear_ or
_Joy_. Similarly, multi-task learning models with Plutchik’s emotions can
sometimes incorrectly predict rumours as non-rumours when the intrinsic
emotions detected are _Fear_ or _Trust_. However, the instances where single-
task learning models were correct and multi-task learning models were
incorrect were significantly lesser than the other way around.

##  5 Broader Impact and Ethical Considerations

The spread of misinformation has become a significant problem for society with
adverse consequences. Our research further improves upon previous machine
learning-based fake news and rumour detection approaches by incorporating the
correlated features between the legitimacy of a text and its intrinsic
emotion, to more accurately detect fake news or rumours. Instead of taking the
sentiment or emotion features as input to the models like in previous works,
we apply a multi-task learning approach to detect fake news and rumours. This
mitigates the need to generate emotion-based features, either computationally
or manually, to be used as input to the fake news detection or rumour
detection models during testing or deployment. Instead, the models implicitly
detect the emotion-guided features and use them for the detection of fake news
or rumours. Evaluation across various datasets for rumour detection as well as
fake news detection, as well as on multiple deep learning architectures proves
the wide applicability of our approach, both for in-domain and cross-domain
settings. Our approach can further be implemented with any state-of-the-art
deep learning model used for fake news detection or rumour detection, to
further improve the performance of the model. Hence, systems incorporating our
approach can aid in curbing the spread of false information on social media
with increased accuracy.

From an ethical point of view, our models may be affected by biases inherited
from the training dataset. This may be biased toward gender, race, or a
different set of people. We hope that further works lead to the creation of
more robust large-scale fake news detection and rumour detection datasets
annotated with emotion classes, which can help minimize biases in the data,
and hence the models. However, we do not foresee our proposed approach causing
any negative impact on society, directly or indirectly.

##  6 The Current Limitations of Our Approach

One limitation of our proposed approach is dependence on the Unison model for
annotating the datasets with Ekman’s and Plutchik’s emotion tags. Despite
giving state-of-the-art results for emotion classification for both Ekman’s
and Plutchik’s emotions, the Unison model achieves a micro F1-score of around
0.70 for both Ekman’s and Plutchik’s emotion classes. This can sometimes lead
to incorrect annotations for some texts in the datasets. These incorrect
annotations can lead to contradictory information being learned by the models,
leading to lower performance. While we achieved positive performance gains in
our experimental evaluations, future advancements in emotion classification
models as well as manually annotated fake news detection and rumour detection
datasets with emotion tags can help in further improving the performance of
classifiers trained to detect fake news or rumours in our proposed multi-task
setting.

##  7 Conclusion

In this work, we verified the correlation between the legitimacy of a text and
its intrinsic emotion. We annotated the datasets with their emotion classes
using the Unison model, to help our models learn the associated emotion
features by transfer learning. We used Plutchik’s basic emotions, based on
previous psychological studies, as well as proposed the use of Ekman’s emotion
classes for improved feature extraction in our classifiers. For implicit
utilization of features linked to the emotions portrayed by a text, we
proposed a multi-task learning approach using multi-class emotion detection as
an auxiliary task for the detection of fake news and rumours on FakeNews AMT,
Celeb, Gossipcop, and PHEME 9 datasets. Our experimental evaluation revealed
that most multi-task models consistently outperformed their single-task
counterparts in terms of accuracy, precision, recall, and F1-score for both
Ekman’s and Plutchik’s emotions for in-domain and cross-domain settings. Our
cross-domain evaluations showed that emotion-guided features are more domain
generalizable, as we observed similar gains in cross-domain settings in MTL
settings over STL settings as in in-domain settings. Upon qualitative
analysis, we observed that MTL models showed improved performance not only for
fake news samples portraying emotions normally associated with fake news and
for real news portraying emotions normally associated with real news, but also
for fake samples portraying emotions associated with real news and real
samples portraying emotions associated with fake news. We further showed the
efficacy of the use of the domain of news as another plausible feature for
fake news detection on the FakeNews AMT dataset.

## References

  * Ajao et al. (2019) Ajao, O., Bhowmik, D., Shahrzad., Z., 2019.  Sentiment Aware Fake News Detection on Online Social Networks, in: IEEE 44th International Conference on Acoustics, Speech and Signal Processing, (ICASSP), IEEE, Brighton, United Kingdom. pp. 2507–2511.
  * Bann and Bryson (2014) Bann, E., Bryson, J., 2014.  The conceptualisation of emotion qualia: semantic clustering of emotional tweets, in: Mayor, J., Gomez, P. (Eds.), Computational Models of Cognitive Processes, World Scientific. pp. 249–263.  NCPW 13th Neural Computation and Psychology Workshop.
  * Bhattacharya et al. (2021) Bhattacharya, P., Patel, S.B., Gupta, R., Tanwar, S., Rodrigues, J.J.P.C., 2021.  Satya: Trusted bi-lstm-based fake news classification scheme for smart community.  IEEE Transactions on Computational Social Systems , 1–10doi:10.1109/TCSS.2021.3131945.
  * Bhutani et al. (2019) Bhutani, B., Rastogi, N., Sehgal, P., Purwar, A., 2019\.  Fake news detection using sentiment analysis, in: 2019 Twelfth International Conference on Contemporary Computing (IC3), pp. 1–5.  doi:10.1109/IC3.2019.8844880.
  * Cheng et al. (2021a) Cheng, M., Li, Y., Nazarian, S., Bogdan, P., 2021a.  From rumor to genetic mutation detection with explanations: a gan approach.  Scientific Reports 11, 5861\.  doi:10.1038/s41598-021-84993-1.
  * Cheng et al. (2020) Cheng, M., Nazarian, S., Bogdan, P., 2020.  Vroc: Variational autoencoder-aided multi-task rumor classifier based on text.  Proceedings of The Web Conference 2020 .
  * Cheng et al. (2021b) Cheng, M., Yin, C., Nazarian, S., Bogdan, P., 2021b.  Deciphering the laws of social network-transcendent covid-19 misinformation dynamics and implications for combating misinformation phenomena.  Scientific Reports 11.  doi:10.1038/s41598-021-89202-7.
  * Choudhry et al. (2022) Choudhry, A., Khatri, I., Jain, M., 2022.  An emotion-based multi-task approach to fake news detection (student abstract).  Proceedings of the AAAI Conference on Artificial Intelligence 36, 12929–12930.  URL: https://ojs.aaai.org/index.php/AAAI/article/view/21601, doi:10.1609/aaai.v36i11.21601.
  * Colneric and Demsar (2018) Colneric, N., Demsar, J., 2018\.  Emotion Recognition on Twitter: Comparative Study and Training a Unison Model.  IEEE Transactions on Affective Computing 11, 433–446.  doi:10.1109/TAFFC.2018.2807817.
  * Devlin et al. (2019) Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Stroudsburg, PA, USA. pp. 4171–4186.  doi:10.18653/v1/N19-1423, arXiv:1810.04805v1.
  * Ekman (1992) Ekman, P., 1992.  An argument for basic emotions.  Cognition & Emotion 6, 169--200.
  * Fersini et al. (2020) Fersini, E., Armanini, J., D’Intorni, M., 2020.  Profiling fake news spreaders: Stylometry, personality, emotions and embeddings, in: CLEF.
  * Guo et al. (2019) Guo, C., Cao, J., Zhang, X., Shu, K., Yu, M., 2019\.  Exploiting emotions for fake news detection on social media.  ArXiv abs/1903.01728.
  * Guo et al. (2018) Guo, H., Cao, J., Zhang, Y., Guo, J., Li, J., 2018\.  Rumor detection with hierarchical social attention network, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA. p. 943–951.  doi:10.1145/3269206.3271709.
  * Han et al. (2020) Han, Y., Karunasekera, S., Leckie, C., 2020.  Graph neural networks with continual learning for fake news detection from social media.  CoRR abs/2007.03316.  arXiv:2007.03316.
  * K et al. (2020) K, A., P, D., L, L.V., 2020\.  Emotion cognizance improves health fake news identification, in: Proceedings of the 24th Symposium on International Database Engineering & Applications, Association for Computing Machinery.  doi:10.1145/3410566.3410595.
  * Kamila et al. (2022) Kamila, S., Hasanuzzaman, M., Ekbal, A., Bhattacharyya, P., 2022\.  Investigating the impact of emotion on temporal orientation in a deep multitask setting.  Scientific Reports 12.  doi:10.1038/s41598-021-04331-3.
  * Kochkina et al. (2018a) Kochkina, E., Liakata, M., Zubiaga, A., 2018a.  All-in-one: Multi-task learning for rumour verification, in: COLING.
  * Kochkina et al. (2018b) Kochkina, E., Liakata, M., Zubiaga, A., 2018b.  Pheme dataset for rumour detection and veracity classification.  doi:10.6084/m9.figshare.6392078.v1.
  * Kumari et al. (2021) Kumari, R., Ashok, N., Ghosal, T., Ekbal, A., 2021\.  A multitask learning approach for fake news detection: Novelty, emotion, and sentiment lend a helping hand, in: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1--8.  doi:10.1109/IJCNN52387.2021.9534218.
  * Meel and Vishwakarma (2019) Meel, P., Vishwakarma, D., 2019\.  Fake news, rumor, information pollution in social media and web: A contemporary survey of state-of-the-arts, challenges and opportunities.  Expert Systems with Applications 153, 112986.  doi:10.1016/j.eswa.2019.112986.
  * Pardo et al. (2020) Pardo, F.M.R., Giachanou, A., Ghanem, B., Rosso, P., 2020\.  Overview of the 8th author profiling task at pan 2020: Profiling fake news spreaders on twitter, in: Cappellato, L., Eickhoff, C., 0001, N.F., Névéol, A. (Eds.), Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, CEUR-WS.org.
  * Pérez-Rosas et al. (2018) Pérez-Rosas, V., Kleinberg, B., Lefevre, A., Mihalcea, R., 2018\.  Automatic detection of fake news, in: Proceedings of the 27th International Conference on Computational Linguistics, ACL, Santa Fe, New Mexico, USA. pp. 3391--3401.
  * Plutchik (1982) Plutchik, R., 1982.  A psychoevolutionary theory of emotions.  Social Science Information 21, 529--553.
  * Roy et al. (2022) Roy, S., Bhanu, M., Saxena, S., Dandapat, S., Chandra, J., 2022.  gdart: Improving rumor verification in social media with discrete attention representations.  Information Processing & Management 59, 102927.  doi:https://doi.org/10.1016/j.ipm.2022.102927.
  * Sabour et al. (2017) Sabour, S., Frosst, N., Hinton, G.E., 2017.  Dynamic routing between capsules, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. p. 3859–3869.
  * Saha et al. (2022) Saha, T., Upadhyaya, A., Saha, S., Bhattacharyya, P., 2022\.  A multitask multimodal ensemble model for sentiment- and emotion-aided tweet act classification.  IEEE Transactions on Computational Social Systems 9, 508--517.  doi:10.1109/TCSS.2021.3088714.
  * Saikh et al. (2019) Saikh, T., De, A., Ekbal, A., Bhattacharyya, P., 2019.  A deep learning approach for automatic detection of fake news, in: Proceedings of the 16th International Conference on Natural Language Processing, NLP Association of India, International Institute of Information Technology, Hyderabad, India. pp. 230--238.
  * Shahid et al. (2022) Shahid, W., Jamshidi, B., Hakak, S., Isah, H., Khan, W.Z., Khan, M.K., Choo, K.K.R., 2022.  Detecting and mitigating the dissemination of fake news: Challenges and future research opportunities.  IEEE Transactions on Computational Social Systems , 1--14doi:10.1109/TCSS.2022.3177359.
  * Singhal et al. (2020) Singhal, S., Kabra, A., Sharma, M., Shah, R.R., Chakraborty, T., Kumaraguru, P., 2020\.  Spotfake+: A multimodal framework for fake news detection via transfer learning (student abstract).  Proceedings of the AAAI Conference on Artificial Intelligence 34, 13915--13916.  doi:10.1609/aaai.v34i10.7230.
  * Singhal et al. (2019) Singhal, S., Shah, R., Chakraborty, T., Kumaraguru, P., Satoh, S., 2019.  Spotfake: A multimodal framework for fake news detection, in: IEEE BigMM.
  * Starbird et al. (2014) Starbird, K., Maddock, J., Orand, M., Achterman, P., Mason, R., 2014.  Rumors, false flags, and digital vigilantes: Misinformation on twitter after the 2013 boston marathon bombing.  IConference doi:10.9776/14308.
  * Verma et al. (2022) Verma, G., Bhardwaj, A., Aledavood, T., Choudhury, M., Kumar, S., 2022.  Examining the impact of sharing covid-19 misinformation online on mental health.  Scientific Reports 12, 8045\.  doi:10.1038/s41598-022-11488-y.
  * Verma et al. (2021) Verma, P.K., Agrawal, P., Amorim, I., Prodan, R., 2021\.  Welfake: Word embedding over linguistic features for fake news detection.  IEEE Transactions on Computational Social Systems 8, 881--893.  doi:10.1109/TCSS.2021.3068519.
  * Vosoughi et al. (2018) Vosoughi, S., Roy, D., Aral, S., 2018.  The spread of true and false news online.  Science 359, 1146--1151.  doi:10.1126/science.aap9559.
  * Wolf et al. (2020) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M., 2020\.  Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online. pp. 38--45.
  * Yang et al. (2018) Yang, Y., Zheng, L., Zhang, J., Cui, Q., Li, Z., Yu, P.S., 2018.  TI-CNN: convolutional neural networks for fake news detection.  arXiv:1806.00749.
  * Yang et al. (2016) Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E., 2016.  Hierarchical attention networks for document classification, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics. pp. 1480--1489.  doi:10.18653/v1/N16-1174.
  * Zhang et al. (2021) Zhang, X., Cao, J., Li, X., Sheng, Q., Zhong, L., Shu, K., 2021.  Mining dual emotion for fake news detection, in: Proceedings of the Web Conference 2021, Association for Computing Machinery, New York, NY, USA. p. 3465–3476.  doi:10.1145/3442381.3450004.