# Arthur: a new ECA that uses Memory to improve Communication

Paulo Knob, Willian S. Dias, Natanael Kuniechick, Joao Moraes, Soraia Raupp
Musse  Graduate Programa in Computer Science, School of Technology, Pontifical
Catholic University of Rio Grande do Sul
Porto Alegre, Brazil
Email<EMAIL_ADDRESS>
###### Abstract

This article proposes an embodied conversational agent named Arthur. In
addition to being able to talk to a person (using text and voice), he is also
able to recognize the person he is talking to and detect his/her expressed
emotion through facial expressions. Arthur uses these skills to improve
communication with the user, also using his artificial memory, which stores
and retrieves data about events and facts, based on a human memory model. We
conducted some experiments to collect quantitative and qualitative
information, which show that our model provides a consistent impact on users.

##  I Introduction

Human beings are the only known species that use spoken language to
communicate, having a skill developed in communication that uses factors other
than speech, such as, for example, body expressions and gazing [1]. In this
context, Embodied Conversational Agents (ECAs) are virtual agents which are
able to interact and talk with humans in a natural way. In the last years,
many research have been conducted to improve the quality of the communication
abilities of such ECAs, both verbal and non-verbal [2, 3, 4]. A fair amount of
effort has been focused on ECAs that help people to have a healthier life [5,
6, 7], in clinical interviews [8, 9], training specific skills [10, 11], among
others.

The present work aims to propose an Embodied Conversational Agent (ECA) with
general purpose, called Arthur, endowed with a memory module and many other
abilities. Besides a conversational module, using text and voice, this ECA is
able to recognize the person he is talking to, as well to assess the user
emotional state through his/her facial expressions. Also, Arthur is able to
demonstrate different levels of emotion through his facial expressions.
Lastly, Arthur is equipped with a memory module, which tries to replicate the
behavior of human memory and, thus, allows for Arthur to learn information
with and from the user while interacting. Moreover, it allows Arthur to
remember such information later in the conversation or, even, in a different
interaction.

Section II presents several work related with the model proposed in this work.
Section III details our methodology, while Section IV shows some results
achieved when users know Arthur. Finally, Section V presents our final
considerations.

##  II Related Work

This Section presents some work related with methodologies used to build
Arthur. Section II-A presents two interesting researches developed in the
context of chatbots, while Section II-B presents papers that discuss the
behaviour of human memory and its use for virtual agents. Finally, Section
II-C discusses the importance of appearance and expressiveness for virtual
agents, also presenting work related with Embodied Conversational Agents
(ECAs).

###  II-A Chatbots

Zhou et al. [12] focus their work on the proposal of the Emotional Chat
Machine (ECM), a chatbot that can generate relevant responses to content that
are also emotionally consistent. As discussed by the authors, there are three
main challenges: obtaining high-quality data marked by emotions in a large-
scale corpus, considering emotions in a natural and coherent way and
incorporating that information of emotions into a neural model. The results
achieved by their work show that ECM can generate appropriate responses, if
the emotion category of the response and the emotion of the post both belong
to one of the frequent Emotion Interaction Patterns (EIP). EIP is defined as
the pair: categories of emotion of the publication and its response.

Zhang et al. [13] propose to solve the problem of consistency on chatbot
responses, concerning both context and personas (casual speaker). In their
work, they present a self-supervised approach that uses the natural structure
of conversational data to learn and leverage both topic and persona features.
For the tests, two data sets are used: Twitter FireHose, collected from 2012
until 2016; and Maluuba dataset. The results achieved indicate that the
proposed model is able to capture meaningful topics and personas’ features.
Also, the incorporation of the learned features helps to significantly improve
the quality of generated responses on both data sets, even when comparing with
models which explicit persona information.

In our method, we mix a pre-defined dialog with an existent technology to
provide chat and voice, in order to build the communication module of Arthur.
More details are given in Section III-B.

###  II-B Human and Agent Memory

One of the most accepted models concerning human memory cited in literature is
known as Autobiographical Memory. As defined by Bluck et al. [14],
autobiographical memory is ”a system that encodes, stores and guides retrieval
of all episodic information related to our person experiences”. Also,
according to Conway et al. [15], autobiographical memory can be grouped in
three levels: lifetime periods, general events and event-specific knowledge.
So, such memories can be directly accessed if the cues are specific and
relevant to the person. Otherwise, if the cues are too general, a generative
retrieval process must be used to produce more specific cues for the retrieval
of relevant memories.

Following this definition of autobiographical memory, Wang et al. [16] build a
model to mimic such behavior. Their model, known as Autobiographical Memory-
Adaptive Resonance Theory (AM-ART), is a three-layer neural network that
encode lifetime periods, general events and event-specific knowledge,
respectively, being, therefore, consistent with the model presented by Conway
et al. [15]. Also, it encodes the 5W1H schema, which represents when an event
occurred, where it happened, who was involved, what happened, which pictorial
memory was associated with the event and how was the person feeling during the
event. The results achieved by their work show that AM-ART was able to perform
better that the keyword-based query method, since the last can not deal with
noisy cues in many existing photo or memory repositories.

The main difference between the work presented in this Section and ours is
that we do not train a neural network. We modeled our memory in a procedural
way, similarly to [17]. Also, our memory retrieval can be guided by the
emotional state and, even, change the mood of the agent. More details are
presented in Section III-E.

###  II-C Appearance and Expressiveness

When developing an interactive agent, there is a concern about how it presents
itself. It is important that the interaction occurs so that the subject in
contact with this agent does not feel uncomfortable and/or embarrassed. As
shown in Dill et al. [18], when trying to present a character who pretends to
be human, there is a certain strange feeling (Uncanny Valley) when it seems
human, but not as close as expected. It should be noted that, considering the
face of a character, the elements that can cause more strangeness are the eyes
and the mouth.

Real human eyes tend to not stay static for too much time, having some
involuntary or voluntary movement at some point. Such phenomena is known as
saccade behavior, being defined by Leigh and Zee [19] as rapid movements of
both eyes from one gaze position to another. Lee et al. [20] propose an
algorithm to simulate such behavior. The authors obtained a statistical model
through the analysis of eye-tracking images from a subject during a
conversation with a software. In combination with literature data, they
achieved a model capable of generate movements considering various aspects,
like magnitude and duration, of a human saccade. As result, the model obtained
a natural and friendly perception from subject into author’s survey.

In this work, we chose to model our virtual agent in a cartoon manner to avoid
the strange feeling discussed by Dill et al. [18]. Also, we included saccade
eyes movement, following the algorithm proposed by Lee et al. [20]. More
details are presented in Section III-F.

##  III Proposed Model

This work aims to build an interactive empathetic ECA which can store and
recover memories about previous interactions: Arthur. One of his features is
being able to recognize the person he is talking to. Also, Arthur has a memory
to store knowledge about people he interacts, such as their face, emotion and
interactions. Finally, in order to keep the interactions as natural as
possible, the agent is able to properly react to interactions with people,
delivering meaningful responses and facial cues.

###  III-A Overview

The overview of Arthur is illustrated in Figure 1. This work was mostly
developed using Unity3D [21]. Some modules were developed using Python, as,
for example, the Face Recognition module.

Figure 1: Overview of the proposed model. Behavior Control is responsible to
define the appropriate behavior of the virtual agent, while the Memory Control
is responsible of store and retrieve memories. The dolls inside the boxes
represent the modules where an user input exist.

Behavior Control is responsible for defining Arthur’s appropriate behavior.
This module uses all the information available to Arthur, that is, the person
recognized by Arthur and with whom he is interacting, the user’s facial
expressions, facts and information stored in Arthur’s memory, the user’s
emotion detected and the dialogue through text/voice.

The Chat and Voice Detection modules are responsible by the interaction
between the user and the agent, in the scope of verbal and textual behavior,
where the Chat Module allows for the user to type some sentence and the Voice
Detector Module is able to transform the voice of the user into words, which
are used as input to the agent. The Face Recognition Module allows Arthur to
recognize the person he is talking to, while the Emotion Detection Module
allows him to identify the person’s perceived emotion. The Facial Expressions
Module is responsible to animate the facial expressions of Arthur, such as
emotions and eyes movement.

Finally, Memory Control is responsible for managing the memory of the virtual
agent and is linked to all memory resources (i.e., memory learning, memory
retrieval, memory consolidation, general events and ESK). These aspects will
be explained in Section III-E. We use a common method to define artificial
memories that is to use Short and Long term memories, where information is
transferred from one to the other (STM and LTM, respectively) [22]. So, during
the interaction between Arthur and the user, the information is stored in STM.
Then, a phase called Memory Consolidation Module deals with the consolidation
of the Long-Term Memory (LTM) of the agent. Therefore, saved memories can be
forgotten by Arthur if their importance is too low or if they are not used for
too long. Such process is detailed in Section III-E4.

Next sections give more details about each of the modules.

###  III-B Chat and Voice Detector

The Chat module allows for the user to communicate with the virtual agent, by
speaking or writing words and sentences. The user can use its own voice to
interact with the virtual agent or write sentences. The Voice Detection module
is able to transform the voice input into words, which can be understood by
the agent. To do so, the DictationRecognizer
111https://docs.unity3d.com/ScriptReference/Windows.Speech.
DictationRecognizer.html class was used. It is available for C# and, thus, for
Unity3D, allowing to access the system scripts which deal with voice
translation to text. Such tool was used for the ease of use, once it is
already integrated with Windows itself.

####  III-B1 Arthur Conversation

At the beginning of the conversation, we modeled some initial questions that
our agent can ask the person it is talking to. These questions serve both as
an ice-breaker and to know some basic information about that person, that are
going to be stored in the memory. To the extent of this work, the modeled
questions are:

  * •

How old are you?

  * •

Do you work?

  * •

Do you study?

  * •

Do you have children?

Each question can, also, have another question(s) linked with them. For
example, if the person answers that it has children, the virtual agent can ask
how many children, or the names of them. All this information is stored in the
memory of the agent, as further explained in Section III-E. If, at any time,
the user asks something about a certain subject that Arthur already knows, he
can answer. For example, suppose that John tells Arthur that he is 42 years
old. Then, in another conversation, someone asks the virtual agent ”how old is
John?”. In this case, Arthur is able to answer that John is 42 years old.
Finally, if Arthur is unsure about how to handle a question, the user’s
written/spoken sentence is sent to a chatbot API 222https://rapidapi.com that
processes the question and returns an answer, which is presented to the user.
Since we are using an API for chatbot responses, it is quite easy to use
another chatbot model, if necessary.

###  III-C Face Recognition

The face recognition module allows Arthur to recognize the person he is
talking to. To do this, we base our work on the Python library for face
recognition 333Available at https://github.com/ageitgey/face_recognition. In
summary, this library is a trained neural network that allows to recognize
people’s faces by comparing a particular image with other images saved in a
data directory. If the person photographed has a low chance of being in the
database (returned by the method), it is a new person, so it is stored in the
dataset for further consultation. For more information, see the link in the
footnote.

###  III-D Emotion Detection

The Emotion Detection module provides to Arthur the skill to detect the
emotion of the user. We use the Affectiva 444https://developer.affectiva.com/
plugin, available for Unity. Affectiva is a well accepted framework which uses
a trained neural network to identify many points on the person’s face.
Depending on the arrangement of such points, the network is able to detect the
emotion that the person is expressing. For more information, see the link in
the footnote.

###  III-E Agent Memory

Arthur’s memory is used to record data collected during interactions with
people. For this, we adopted a model known as Autobiographical Memory. In
short, as defined by Bluck et al. [14], autobiographical memory is ”a system
that encodes, stores and guides retrieval of all episodic information related
to our person experiences”. Therefore, it is capable of storing different
types of information (for example, text, episodes, images, voice) and
retrieving it, if a certain suggestion (or suggestions) is informed. For
example, when someone meets a new person, they usually greet each other and
say their names. If the memory serves that person well, he/she will be able to
remember the other person’s name when they meet again.

According to Conway et al. [15], autobiographical memory can be grouped in
three levels: Lifetime Periods, General Events and Event-Specific Knowledge.
Lifetime Periods serve as an index to cue General Events and can be linked to
different periods across the lifespan of a person. For example, someone’s
first job can be seen as a lifetime period. General Events are events that
occur within a certain Lifetime Period. Using the first example of work given
earlier, some general events within this can be the first day of work, meeting
a new colleague or a happy hour with people in the office. Finally, the Event
Specific Knowledge (ESK) includes a set of resources that form the memory of a
given event. This information can be stored in different types, such as
images, grammatical or audio.

As it is possible to see in Figure 1, we model our memory using General Events
and ESK. As the intent of our virtual agent is to interact with people, we
chose not to model Lifetime Periods. General Events represent the events which
occur during the interaction between Arthur and the user. For example, when
Arthur meets someone new, a new General Event is generated (i.e. Meet new
person). More details about General Events are given in Section III-E1.
Moreover, following the autobiographical memory model, General Events are
comprised of a set of resources known as ESK. In this work, we store such
resources in two levels: Short Term Memory (STM) and Long Term Memory (LTM).
More details are given in Section III-E2.

####  III-E1 General Events

In our model, General Events are comprised of:

  * •

Timestamp: the moment this event was created or updated.

  * •

ID: an unique ID which refers to this event.

  * •

Type: refers to the type of the event and there are currently three possible
types: i) Meet new person: when the agent meets someone new; ii) Learn thing:
when the agent learn something new; and iii) Interaction: when it does not fit
in any of the other types).

  * •

Emotion: the emotion associated with the event, recognized from the Emotion
Detection module. For example, if a person is talking about the death of its
beloved pet, it is expected that such person assumes a sad face. So, this
general event should be associated with the Sadness emotion.

  * •

Polarity [-1,1]: the polarity of the sentence, it means, if it is a positive
or negative sentence. For example, the sentence ”I am feeling good” would be a
positive sentence, while the sentence ”I am not feeling good” would be a
negative one. When we divide the sentence into tokens, we also use the
sentiment library555https://www.nltk.org/howto/sentiment.html, provided by
NLTK, to calculate the polarity of the given sentence. To do so, we chose to
work with Vader method. It returns a float value lying between -1 and 1, where
negative values reflect a negative polarity and positive values reflect a
positive polarity.

  * •

Resources: a list with the resources (i.e. ESK) associated with this General
Event. Using the previous example, some resources associated with this event
could be a picture of the deceased pet and some grammatical information like
the pet name, death and when it passed away.

####  III-E2 ESK: STM and LTM

As commented before, we store the resources (i.e. ESK) in two levels: STM and
LTM. According to Loftus et al. [22], STM is used to store important
information for a short period of time, while LTM is an information storage
with virtually unlimited capacity that each human being has. In our work, we
model STM and LTM separately. The STM is comprised of a list of resources.
This list of resources can have, at most, seven items, as defined by Miller’s
Law [23]. If a new resource should enter the STM, the less important
information is forgotten according its weight (i.e. the resource with the
lower weight is removed). Each resource has the following information:

  * •

Timestamp: the moment this resource was created or updated.

  * •

ID: an unique ID which refers to this resource.

  * •

Type: refers to the type of the resource (e.g. grammatical, image, audio).

  * •

Information: refers to the information itself that the resource should
reflect. For example, if the resource is an image, the information is going to
contain the path to such image.

  * •

Activation [0,1]: As described in Loftus et al. [22], an information present
in the short-term memory is rapidly forgotten (around 15 seconds), unless it
is rehearsed, it means, repeated over and over. The activation represents this
rehearsal process. When a new resource enters the STM of Arthur, activation is
set at its maximum value (i.e. 1). As time passes by, a logarithmic-based
decay function is applied for each resource, diminishing their activation
value. It is defined as follows: $A_{ID_{STM}}^{*}=Log(A_{ID_{STM}}+1)$, where
$A_{ID_{STM}}^{*}$ is the new activation value of STM for resource $ID$,
$A_{ID_{STM}}$ is the current activation value and $Log()$ is a logarithmic
function. If any resource in the memory is rehearsed (e.g. remembered, seen
again), its activation value is set back to 1. This attribute exists only for
STM.

  * •

Weight [0,1]: represents the importance of the resource. For example, meeting
a new person can be considered more important than talking about the weather.
We empirically defined the initial importance of each resource (at STM) based
on the Type of the General Event it belongs to, as follows: Meet new person:
0.9; Learn thing: 0.9; Interaction: 0.1. At LTM this attribute is not
initialized but indeed affected by the values of
$W_{ID_{LTM}}^{*}=f(A_{ID_{STM}}^{*},W_{ID_{STM}}^{*})$ at STM, in the
consolidation of the memory, as explained in Section III-E4.

In order to store grammatical resources, we divide a given sentence in
significant tokens and keep each of them separately. To do so, we use the
Natural Language ToolKit (NLTK)666https://www.nltk.org/. since it is a well
accepted tool for natural language processing. It is a platform for building
Python programs to work with human language data. Such tool is able to
”tokenize” a text, it means, split it in sentence or word tokens. It is also
able to remove ”stop words” from the text, it means, words which have low or
no meaningfulness for the context of the sentence. We built a script which is
able to remove the ”stop words” from the text and ”tokenize” it word by word.
For example, assuming the user told to the virtual agent ”I am going on
vacation with my dad to Glasgow”, the tokens returned by the NLTK script would
be ”vacation”, ”dad” and ”Glasgow”.

The same type of resources can be stored in STM and LTM, but the LTM is
theoretically unlimited, it means, can maintain an unlimited number of
resources. To keep track of the content of the Long-term memory, we save all
its information in a database, so we can retrieve it at any moment.

####  III-E3 Memory Retrieval

Assuming the information is stored, how can one retrieve such information if
required? The Memory Retrieval module deals with the retrieval of the
information from the storage. According to Conway et al. [15], there are two
types of memory construction: Generative Retrieval and Direct Retrieval.
Generative retrieval is a method guided by cues, it means, the retrieval
depends on some ”hint” in order to find some information. For example, the
word ”dog” can make some people remember of its beloved pet. Thus, the word
”dog” acted like a cue to the generative retrieval method, constructing a
memory of a special dog. In its turn, Direct Retrieval method refers to
spontaneously recalled memories, it means, memories that are recalled
automatically, with no apparent cue. According to Berntsen [24], such process
occurs between two to three times each day. In our work, the Memory Retrieval
is developed following the Generative Retrieval method. When the user
interacts with the virtual agent, the information provided can be used as
cue(s) to the Retrieval method. For example, if the user tells ”I went fishing
with my dad”, the words ”fish” and ”dad” are used as cues for the retrieval
method, so we proceed with a search of those words in the database.

####  III-E4 Memory Consolidation: from STM to LTM

This module is responsible for consolidating the Long-term Memory of the
virtual agent based on data available at STM. According Klinzing et al. [25],
the formation of such LTM is a major function of sleep. As their work affirms,
the authors ”consider the formation of long-term memory during sleep as an
active systems consolidation process that is embedded in a process of global
synaptic downscaling”. In short, the consolidation process prioritizes
important memories over mundane ones. For example, emotional memories are more
important and have more impact than neutral memories. Therefore, less
important information can be placed in a second plan or, even, forgotten.

In our work, we developed a module to simulate such process, which uses two
information from the STM’s resources: Activation and Weight. If the Activation
value $A_{ID}$ of a given resource $ID$ is below an empirically defined
threshold (i.e. $A_{ID}<0.2$), the Weight attribute of this resource is
reduced at LTM, as follows: $W_{ID_{LTM}}^{*}=Log(W_{ID_{STM}}+1)$, where
$W_{ID_{LTM}}^{*}$ is the new Weight this resource will have at LTM,
$W_{ID_{STM}}$ is the current weight at STM, and $Log()$ is a logarithmic
function. Such process is repeated for all stored resources at STM. Then,
resources which have low importance are wiped out from the LTM. We empirically
define that a resource has low importance if its weight drops below 0.2. In
its turn, General Events are not forgotten by the virtual agent, unless all
the resources belonging to a given event are also forgotten. We do so to make
possible for the agent to forget just parts of the information, just like it
happens with a real person. In addition, please notice that during the memory
consolidation, all data at STM is erased.

###  III-F Facial Expressions

In order to model our agent facial characteristics, we proceeded the following
way: the various parts of agent’s face (such as eyes, mouth, etc) where
created using Unity3D sprites. Then, after placing each of them into their
position, a set of constraints were defined in order to avoid further
distortions, e.g., when agent’s head moves, the other parts follow that
movement in a proper way. Finally, the different expression where modeled
using a plug-in called Anima2D
777https://assetstore.unity.com/packages/essentials/unity-anima2d-79840.

The processes of animation consists in the usage of skeletal animation. For a
given mesh in the face that should suffer some kind of deformation, such as
the eyebrows and mouth, a set of bones is defined and linked to this mesh.
After that, it is possible for us to animate the agent through the time just
by applying transformations into the bones, which is done through
Anima2D888See footnote 7. Therefore, after animating a certain facial
expression (which can by either animated or a static one), we can save it for
further usage. The expressions of our agent are: anger, disgust, doubt, fear,
joy, sadness, surprise and worry. Also, there is a neutral expression and one
for sleeping (in order to indicate that the memory consolidation process was
triggered).

Finally, in order to give some expression to the eyes of the virtual agent, it
was implemented saccade movements on them, according to Lee et al [20]. As
commented in Section II-C, human eyes are not static for much time, tending to
have involuntary movements, even when the person is focusing its vision on
something. Such phenomena is known as saccade, from the french word saccade,
which means jerk. As defined by Leigh and Zee [19], saccades are rapid
movements of both eyes from one gaze position to another.

##  IV Results

With all features presented in Section III working altogether, we developed an
Embodied Conversational Agent (ECA), named Arthur, able to recognize the
person he is talking to, as well the emotions hinted by this person’s facial
expressions.Arthur is also capable of demonstrate some level of emotions,
depending the situation. Also, it can learn things from the user and remember
such information when needed.

Figure 2 shows Arthur and its interface. We chose to model Arthur in a cartoon
manner to avoid the Uncanny Valley, as commented in Section II-C. At the
bottom, the image captured by the camera is shown, with the face of the person
talking with Arthur. The green square is where the name of the recognized
person is presented. The blue square is where the person can chat with Arthur.
The larger text area is where Arthur speaks, while the smaller input text
below is where the user can write his/her sentences. The red square is where
the person’s emotion is shown. Finally, in the yellow square, two buttons:
Voice and Sleep. The Voice button toggles the voice management, allowing for
the user to talk with Arthur using its own voice, while the Sleep button puts
Arthur to sleep, triggering the memory consolidation process, as explained in
Section III-E4.

Figure 2: Arthur and its interface. The green square is where the name of the
recognized person is presented. The blue square is where the person can chat
with Arthur. The red square is where the person’s emotion is shown. Finally,
in the yellow square we have the Voice and Sleep buttons, which allow for the
user to talk naturally with Arthur and puts it to sleep, respectively.

In order to test our method, we developed some experiments exploring the
various features of Arthur. Section IV-A shows some emergent behaviors
regarding Arthur’s memory while Section IV-B presents research carried out
with subjects on the functioning and behavior of Arthur.

###  IV-A Agent Memory

In order to evaluate the Arthur’s memory, we test some possible scenarios:

  * •

Introduction Scenario: Arthur meets a new person and starts to asking
questions about him/her. After that, we check if Arthur can remember all
information.

  * •

Learning Scenario: we feed Arthur with information about some objects. Later,
we check if it recognizes such objects.

####  IV-A1 Introduction Scenario

In this scenario, Arthur begins with no knowledge about the person it is
seeing on the webcam, thus, its first action should be to ask who it is.
Figure 3 shows how Arthur deals with this situation. In Figure 3 (a), Arthur
is meeting a new person and, therefore, asks for this person’s name with the
statement ”Hello stranger! May I know your name?”. After the proper
introduction (i.e. the person in question answers that its name is ”Knob”),
Arthur stores the name of this person on its memory. When they meet again
(Figure 3 (b)), Arthur is able to remember both the face and the name of this
person. Then, it proceeds to a cordial greeting (”Greetings Knob!”). Plus,
Arthur tries to maintain the conversation asking questions about the person.
In Figure 3 (b), it is possible to see Arthur asking the age of the person
(”How old are you?”). When the answer is given, such information is also
stored into the memory, increasing the information Arthur knows about this
user.

(a) Arthur does not know this user.

(b) After introduction, Arthur is able to remember this person.

Figure 3: Arthur is meeting a new person. In (a), he asks for this user’s
name. After a proper introduction, Arthur stores that person’s name in his
memory and then he can remember that person’s face and name, greeting him/her
when they see each other again (b).

####  IV-A2 Learning Scenario

As commented in Section III-E, Arthur is able to learn information from the
person he is talking to. Figure 4 shows Arthur learning 2 types of
information. As commented in Section IV-A1, Arthur tries to starts or maintain
a conversation with the person he is interacting, e.g., asking questions about
him/her. Figure 4 (a) shows Arthur asking if the person studies. All possible
questions are presented in Section III-B. In Figure 4 (b), Arthur answers the
question about the age of Knob, saying he is 31 years old. It is able to do so
because the person (i.e. Knob) already informed Arthur about his age in a past
interaction.

Besides learning information about the person Arthur is talking to, he can
also learn about new things. For example, in Figure 4 (c), it is asked if
Arthur knows what a cellphone is. ”No, it does not!”, is the Arthur answer.
Then, he asks the person if he/she desires to show him a picture of a
cellphone. If the person agrees, a new term is stored in Arthur’s memory, next
to the image provided. Then, when Arthur is asked again about a cellphone
(Figure 4 (d)), he is able to remember what it is, showing the image which was
provided before.

(a) Arthur asking a question to the person.

(b) After being asked about the age of Knob, Arthur gives the answer.

(c) After being asked about if it knows a cellphone, Arthur says it does not.

(d) After it learns what a cellphone is, Arthur is able to identify it.

Figure 4: Arthur learning new things. In (a) and (b), it is shown the process
where Arthur learns new information about a given person. In (c) and (d),
Arthur learns what a cellphone is.

###  IV-B Study with Subjects

In order to test our hypothesis with a qualitative evaluation, we conducted an
experiment with subjects. Each person was presented to three video sequences.
In the first video sequence, Arthur meets a new person and asks questions
about he/she, as explained in Section III-B, but the Memory Module is
deactivated, thus, it does not retain information. In the second video
sequence, the same interaction is conducted, but the Memory Module is
activated, thus, Arthur can retain information and remember it further.
Finally, in the third video sequence is specifically presented the process
where Arthur learns a new information and remembers it further. Arthur is
asked if he knows an object (i.e. cellphone), which he does not. Then, we show
to him what a cellphone is and ask him again if he knows a cellphone. Since
the Memory Module is activated, Arthur remembers what a cellphone is and
answer the person. After each video sequence, each subject was asked to answer
some questions:

  * •

Q1: How do you evaluate your comfort level regarding agent’s appearance?

  * •

Q2: How do you evaluate the agent’s facial expressions realism level?

  * •

Q3: How do you evaluate agent’s information comprehension abilities?

  * •

Q4: After watching the interaction, how do you evaluate agent’s memory?

Subjects answered each question following a Likert scale, where:

$1$: Very low, $2$: Low, $3$: Moderate, $4$: Good and $5$: Very good.

Regarding our expectations, we formulate the following hypothesis:

  * •

H1: Since the facial expressions are the same for all three videos, we expect
little or no difference of how people will evaluate both their comfort level
and the realism level of the agent’s facial expressions (Q1 and Q2),
throughout the video sequences.

  * •

H2: We believe that the memory of Arthur is going to help him to understand
the context of the interaction better. Therefore, in the videos where the
Memory Module is activated, we believe the participants are going to rate
Arthur’s comprehension abilities better (Q3).

  * •

H3: We expect that people rate Arthur’s memory higher in Videos 2 and 3, when
compared with Video 1 (Q4).

Concerning the subjects, we had 51 answers, where 39 were male and 12 were
female. Also, we had participants ranging from 17 to 61 years old. The average
age was 31.49 with a standard deviation of 12.49. All participants agreed with
the terms of the research and conducted the experiment until the end.

TABLE I: Quantity of answers acquired by the experiment, separated by each
video sequence. The head numbers (i.e. 1, 2, 3, 4 and 5) represent each value
of the Likert scale.

Video 1 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Q1 | 4 | 7 | 19 | 19 | 2
Q2 | 8 | 12 | 19 | 10 | 2
Q3 | 3 | 14 | 15 | 15 | 4
Q4 | 7 | 8 | 17 | 15 | 4

Video 2 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Q1 | 6 | 9 | 15 | 19 | 2
Q2 | 5 | 12 | 20 | 12 | 2
Q3 | 4 | 4 | 14 | 23 | 6
Q4 | 5 | 4 | 11 | 22 | 9

Video 3 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Q1 | 4 | 7 | 16 | 19 | 5
Q2 | 5 | 12 | 19 | 10 | 5
Q3 | 2 | 6 | 12 | 19 | 12
Q4 | 1 | 5 | 13 | 19 | 13

Table I shows the quantity of answers obtained per question, for each video
sequence. In order to investigate the hypothesis variation of response of the
users, we rely on ANOVA test. The results reveal an uniformity in the
participants answers concerning Q1 (F(3.05)=0.63,p=.53)) and Q2
(F(3.05)=0.64,p=.52)), thus, there is little variation on the answers about
the appearance and realism level of the agent (i.e. Q1 and Q2), which confirms
H1. For Q1, most answers were between Moderate (3) and Good (4). For Q2, the
most answers were Moderate (3). With this, we can also conclude that the
appearance of Arthur has some space to improve, specially concerning its
facial expressions.

An unexpected behavior was observed on the answers about the comprehension
abilities and memory of the agent (i.e. Q3 and Q4). Both questions had a
significant amount of answers between Moderate (3) and Good (4) for Video 1.
For example, if we take Q4, it was expected to have most answers lying between
Very Low (1) and Low (2), since the agent has the Memory Module deactivated.
However, we had 17 answers for Moderate (3) and 15 for Good (4), while having
a total of 15 for the ones we were expecting. One possible explanation could
be the order of the sentences that Arthur speaks. Since they are presented to
the user in an orderly way, it may cause the impression that Arthur is indeed
understanding and processing the information given by the user. Further
investigation would be necessary to confirm or disprove it.

Even with such unexpected behavior, it seems that H2 and H3 were also
confirmed. In Video 1, we had a total of 19 answers for Good (4) and Very Good
(5), for both Q3 and Q4. In Video 2, with the Memory Module activated, we had
a total of 29 answers, in the same range, for Q3 and 31 answers for Q4. In
Video 3, we had 31 for Q3 and 32 for Q4. Also, if we look at the amount of
answers Very Low (1) and Low (2), we can clearly perceive that it diminishes
in Videos 2 and 3, when compared with Video 1. Finally, the results of the
ANOVA test show a change of perception when comparing Video 1 and Video 3, for
both Q3 (F(3.93)=7.57,p=.007)) and Q4 (F(3.93)=11.31,p=.001)).

##  V Final Considerations

This work presented a model of an Embodied Conversational Agent (ECA) endowed
with a memory module and many other abilities, like face recognition, emotion
detection and expressiveness. We conducted some experiments in order to test
our model and collect both quantitative and qualitative information. The
results achieved seem to confirm that Arthur presented the expected behavior.
Concerning the appearance of the virtual agent, despite having a good result
in the user study, we conclude that it has room to improve. A few changes
could be made regarding the geometric model and the animations, pursuing a
better comfort level and higher realism. Also, concerning the memory model, a
satisfactory result was achieved. In both quantitative and qualitative tests,
the results achieved showed that the memory worked as intended and its
influence on the interaction could be perceived.

As for future work, there are a few paths that can be followed. As have
already been commented, we plan to improve the appearance of the agent. Also,
according the literature, as presented in Sections II-B and III-E, the human
memory is able to store information in different types, like text, images and
voice. This work presents a memory model which deals with text and images,
thus, the next logical step would be to make it able to deal with audio also.
Moreover, besides being able to recognize the person it is talking with, the
virtual agent could be able to recognize objects present in the image, both to
learn what such object is and identify it later.

## Acknowledgments

The authors would like to thank to Brazilian agencies CNPq e CAPES for
partially funding this project.

## References

  * [1] J. Cassell, J. Sullivan, E. Churchill, and S. Prevost, _Embodied conversational agents_. MIT press, 2000\.
  * [2] Ö. N. Yalçın, “Empathy framework for embodied conversational agents,” _Cognitive Systems Research_ , vol. 59, pp. 123–132, 2020.
  * [3] B. Biancardi, C. Wang, M. Mancini, A. Cafaro, G. Chanel, and C. Pelachaud, “A computational model for managing impressions of an embodied conversational agent in real-time,” in _2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2019, pp. 1–7.
  * [4] P. Sajjadi, L. Hoffmann, P. Cimiano, and S. Kopp, “A personality-based emotional model for embodied conversational agents: Effects on perceived social presence and game experience of users,” _Entertainment Computing_ , vol. 32, p. 100313, 2019.
  * [5] L. Kramer, S. ter Stal, B. Mulder, E. de Vet, and L. van Velsen, “Developing embodied conversational agents for healthy lifestyles: a scoping review,” in _ARPH_ , 2019.
  * [6] M. Spitale, F. Catania, P. Crovari, and F. Garzotto, “Multicriteria decision analysis and conversational agents for children with autism,” in _Proceedings of the 53rd Hawaii International Conference on System Sciences_ , 2020.
  * [7] K. S. J. Das, T. Beinema, H. Op Den Akker, and H. Hermens, “Generation of multi-party dialogues among embodied conversational agents to promote active living and healthy diet for subjects suffering from type 2 diabetes,” in _5th International Conference on Information and Communication Technologies for Ageing Well and e-Health, ICT4AWE 2019_. SCITEPRESS, 2019, pp. 297–304.
  * [8] P. Philip, L. Dupuy, M. Auriacombe, F. Serre, E. de Sevin, A. Sauteraud, and J.-A. Micoulaud-Franchi, “Trust and acceptance of a virtual psychiatric interview between embodied conversational agents and outpatients,” _npj Digital Medicine_ , vol. 3, no. 1, pp. 1–7, 2020.
  * [9] J. Martínez-Miranda, A. Martínez, R. Ramos, H. Aguilar, L. Jiménez, H. Arias, G. Rosales, and E. Valencia, “Assessment of users’ acceptability of a mobile-based embodied conversational agent for the prevention and detection of suicidal behaviour,” _Journal of medical systems_ , vol. 43, no. 8, p. 246, 2019.
  * [10] G. Chetty and M. White, “Embodied conversational agents and interactive virtual humans for training simulators,” in _Proc. The 15th International Conference on Auditory-Visual Speech Processing_ , 2019, pp. 73–77.
  * [11] E. Ayedoun, Y. Hayashi, and K. Seta, “Adding communicative and affective strategies to an embodied conversational agent to enhance second language learners’ willingness to communicate,” _International Journal of Artificial Intelligence in Education_ , vol. 29, no. 1, pp. 29–57, 2019.
  * [12] H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu, “Emotional chatting machine: Emotional conversation generation with internal and external memory,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [13] Y. Zhang, X. Gao, S. Lee, C. Brockett, M. Galley, J. Gao, and B. Dolan, “Consistent dialogue generation with self-supervised feature learning,” _arXiv preprint arXiv:1903.05759_ , 2019.
  * [14] S. Bluck and L. J. Levine, “Reminiscence as autobiographical memory: A catalyst for reminiscence theory development,” _Ageing & Society_, vol. 18, no. 2, pp. 185–208, 1998.
  * [15] M. A. Conway and C. W. Pleydell-Pearce, “The construction of autobiographical memories in the self-memory system.” _Psychological review_ , vol. 107, no. 2, p. 261, 2000.
  * [16] D. Wang, A.-H. Tan, and C. Miao, “Modeling autobiographical memory in human-like autonomous agents,” in _Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems_. International Foundation for Autonomous Agents and Multiagent Systems, 2016, pp. 845–853.
  * [17] A. Kope, C. Rose, and M. Katchabaw, “Modeling autobiographical memory for believable agents,” in _Ninth Artificial Intelligence and Interactive Digital Entertainment Conference_ , 2013.
  * [18] V. Dill, L. M. Flach, R. Hocevar, C. Lykawka, S. R. Musse, and M. S. Pinho, “Evaluation of the uncanny valley in cg characters,” in _International Conference on Intelligent Virtual Agents_. Springer, 2012, pp. 511–513.
  * [19] R. J. Leigh and D. S. Zee, _The neurology of eye movements_. OUP USA, 2015.
  * [20] S. P. Lee, J. B. Badler, and N. I. Badler, “Eyes alive,” in _Proceedings of the 29th annual conference on Computer graphics and interactive techniques_ , 2002, pp. 637–644.
  * [21] U. Technologies. (2020) Unity - game engine. [Online]. Available: https://unity3d.com/
  * [22] G. R. Loftus and E. F. Loftus, _Human memory: The processing of information_. Psychology Press, 2019.
  * [23] G. A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” _Psychological review_ , vol. 63, no. 2, p. 81, 1956.
  * [24] D. Berntsen, “Involuntary autobiographical memories,” _Applied Cognitive Psychology_ , vol. 10, no. 5, pp. 435–454, 1996.
  * [25] J. G. Klinzing, N. Niethard, and J. Born, “Mechanisms of systems memory consolidation during sleep,” _Nature neuroscience_ , vol. 22, no. 10, pp. 1598–1610, 2019.


††footnotetext: Copyright © 2021 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY
4.0).11institutetext: TIB – Leibniz Information Centre for Science and
Technology, Hannover, Germany
22institutetext: L3S Research Center, Leibniz University Hannover, Germany
22email<EMAIL_ADDRESS>
# On the Role of Images for Analyzing Claims in Social Media

Gullal S. Cheema 11 0000-0003-4354-9629 Sherzod Hakimov 11 0000-0002-7421-6213
Eric Müller-Budack 11 0000-0002-6802-1241 Ralph Ewerth 1122
0000-0003-0918-6297

###### Abstract

Fake news is a severe problem in social media. In this paper, we present an
empirical study on visual, textual, and multimodal models for the tasks of
claim, claim check-worthiness, and conspiracy detection, all of which are
related to fake news detection. Recent work suggests that images are more
influential than text and often appear alongside fake text. To this end,
several multimodal models have been proposed in recent years that use images
along with text to detect fake news on social media sites like _Twitter_.
However, the role of images is not well understood for claim detection,
specifically using transformer-based textual and multimodal models. We
investigate state-of-the-art models for images, text (Transformer-based), and
multimodal information for four different datasets across two languages to
understand the role of images in the task of claim and conspiracy detection.

###### Keywords:

Fake News Detection Claim Detection Conspiracy Detection Multimodal Analysis
Multilingual NLP Computer Vision Transformers COVID-19 5G Twitter

##  1 Introduction

Social media platforms have become an integral part of our everyday lives,
where we use them to connect with people and consume news, entertainment, and
buy or sell products. In the last decade, social media has seen exponential
growth, with more than a couple of billion users and the increasing presence
of prominent people like politicians and celebrities (also called
Influencers), organizations, and political parties. On the one hand, this
allows influential people or organizations to reach millions of users
directly, but it also allows for fake and unverified information to rise and
spread faster [42] due to the nature of social media. To deal with
misinformation and false claims on online platforms, several independent fact-
checking projects like Snopes, Alt News, Our.News have been launched that
manually fact-check news and publish their outcomes for public use. Although
more such initiatives are coming up worldwide, they cannot keep up with the
rate of news or information production on online platforms. Therefore, fake
news detection has gathered much interest in computer science for developing
automated methods to speed and scale up to handle the continuous fast
streaming social media data.

As social media is inherently multimodal in nature, fact-checking initiatives
and computation methods consider not only text but also image content [14, 21,
39, 43] as it can be easily fabricated and manipulated due to the availability
of free image and video editing tools. In this paper, we investigate the role
of images in the context of claim and conspiracy detection. Claim detection is
one of the first vital steps to identify fake news where the purpose is to
flag a statement if it contains check-worthy facts and information, while the
claim may be true or false. Whereas in conspiracy detection, a statement that
includes a conspiracy theory is fake news and consists of manipulated facts.
Although fake news on social media has been explored recently from a
multimodal perspective, images have hardly been considered for claim detection
except in recent work by Zlatkova et al. [48]. Here, meta-information of
images is treated as features, and reverse image search is performed to
compare the claim text. However, the image’s semantic information is not
considered, and the authors highlight that images are more influential than
text and appear alongside fake text or unverified news.

Since we are interested in the impact of using images in a multimodal
framework, to keep our models simple, we focus on extracting only semantic or
contextual features from text and do not consider its structure or syntactic
information. To this end, we mainly consider deep transformer Bidirectional
Encoder Representations from Transformers (BERT) to extract contextual
embeddings and use them along with image embeddings. Taking inspiration from
recent work by Cao et al. [4], we extract image sentiment features that are
widely applied for image credibility or fake news detection in addition to
object and scene information for the semantic overlap with textual
information.

Figure 1: Examples from CLEF-English [36] (a) check-worthy claims dataset and
MediaEval [33] (b) conspiracy detection dataset

To carry out this study111Code: https://github.com/cleopatra-
itn/image_text_claim_detection, we experiment with four _Twitter_
datasets222Dataset: https://zenodo.org/record/4592249 on binary classification
tasks, two of which are from the recent _CLEF-CheckThat! 2020_ [2], one in
English [36] and the other one in Arabic [17]. The third one is an English
dataset from _MediaEval 2020_ [33] on conspiracy detection, and the last one
is a recent claim detection dataset (English) from Gupta et al. [16] on
_COVID-19_ tweets. Four examples for claim and conspiracy detection are shown
in Figure 1. To train our unimodal and multimodal models, we use Support
Vector Machines (SVM) [40] and Principal Component Analysis (PCA) [45] for
dimensionality reduction due to the small datasets and large size of combined
features. We also fine-tune BERT models on the text input to see the extent of
the unimodal model’s performance on limited-sized datasets and use different
pre-trained _BERT_ models to see the effect of domain gap. Furthermore, we
investigate the recently proposed transformer-based _ViLBERT_ [25] (Vision-
and-Language BERT) model that learns semantic features via co-attention on
image and textual inputs. Just like _BERT_ models, we perform fixed embedding
and fine-tuning experiments using _ViLBERT_ to see if a large transformer-
based multimodal model can learn meaningful representation and perform better
on small-sized datasets.

The remainder of the paper is organized as follows. Section 2 briefly
discusses related work on fake news detection and the sub-problems of claim
and conspiracy detection. Section 3 presents details of image, text, and
multimodal features as well as the fine-tuned and applied models. Section 4
describes the experimental setup, results and summarizes our findings. Section
5 concludes the paper with future research directions.

##  2 Related Work

There is a wide body of work on fake news detection that goes well beyond this
paper’s scope. Therefore, we restrict this section to multimodal fake news,
claim detection, and conspiracy detection.

###  2.1 Unimodal Approaches

The earliest claim detection works go back a decade. Rosenthal et al. [34] in
their pioneering work extracted claims from _Wikipedia_ discussion forums.
They classified them via logistic regression using the sentiment, syntactic
and lexical features like POS (Part-of-Speech) tags and n-grams, and other
statistical features over text. Since then, researchers have proposed context
dependent [22], context independent [23], and cross-domain [10] and in-domain
approaches for claim detection. Recently, the transformer-based models [6]
have replaced structure-based claim detection approaches due to their success
in several downstream natural language processing (NLP) tasks.

For claim detection on social media in particular, recently _CLEF-CheckThat!
2020_ [2] hosted a challenge to detect check-worthy claims in _COVID-19_
related English tweets and several other topics in Arabic. The challenge
attracted several models with top submissions [7, 32, 44] all using some
version of transformer-based models like _BERT_ [11] and _RoBERTa_ [24] along
with tweet meta-data and lexical features. Outside of CLEF challenges, some
works [12, 27] have also conducted a detailed study on detecting check-worthy
tweets in U.S. politics and proposed real-time systems to monitor and filter
them. Taking inspiration from [10], Gupta et al. [16] address the limitations
of current methods in cross-domain claim detection by proposing a generalized
claim detection model called _LESA_ (Linguistic Encapsulation and Semantic
Amalgamation). Their model combines contextual transformer features with
learnable POS and dependency relation embeddings via transformers to achieve
impressive results on several datasets. For conspiracy detection, _MediaEval
2020_ [33] saw interesting methods to automatically detect 5G and Coronavirus
conspiracy in tweets. Top submissions used BERT [8, 28] pre-trained on _COVID_
Twitter data, tweet meta-data, graph network data and RoBERTa models [9] along
with Graph Convolutional Neural (GCN) networks.

###  2.2 Multimodal Approaches

For multimodal fake news in general, several benchmark datasets have been
proposed in the last few years, generating interest in developing multimodal
visual and textual models. In one of the relatively early works, Jin et al.
[20] explored rumor detection on Twitter using text, social context
(emoticons, URLs, hashtags), and the image by learning a joint representation
with attention from LSTM outputs over image features. The authors observed the
benefit of using the image and social context in addition to text by improving
the detection of fake news in Twitter and Weibo datasets. Later, Wang et al.
[43], proposed an improved model that learns a multi-task model to detect fake
news as one task and event discriminator as another task to learn event
invariant representations. Since then, improvements have been proposed via
using multimodal variational autoencoders [21], transfer learning [15, 39]
with transformer-based text and deep visual CNN models. Recently, Nakamura
[30] et al. proposed a fake news dataset r/Fakeddit mined from Reddit with
over 1 million samples, which includes text, images, meta-data, and comments
data. The data is labeled through distant supervision into 2-way, 3-way, and
6-way classification categories. In addition to our different tasks, another
difference with the approaches mentioned above is that the size of the
datasets is moderate (several thousand) to large (millions) in comparison to a
few hundred or a couple of thousand samples in our four datasets for claim and
conspiracy detection.

##  3 Methodology

In this section, we provide details of different image (Section 3.1), textual
(Section 3.2), and multimodal (Section 3.3) models and their feature encoding
process and how classification models (Section 3.4) are built. An overview of
classification models are presented in Figure 2.

Figure 2: Workflow of the proposed solutions for claim and conspiracy
detection in tweets using multimodal information from text (orange) and image
(blue). Three approaches are investigated: (a) Training of an SVM based on the
combined features extracted from pre-trained models for visual and/or textual
information extraction. (b) Fine-tuning a transformer network (_BERT_) solely
using textual information. (c) Fine-tuning ViLBERT [25] or training an SVM
based on its multimodal embeddings extracted from text and image.

###  3.1 Image Models ($I$)

The purpose of image models is to encode the presence of different objects,
scene, place or background, and affective image content. When learning a
multimodal model or a classifier, specific overlapping patterns between image
and text can act as discriminatory features for claim detection.
Object Features ($I_{o}$) In order to encode objects and the overall image
content, we extract features from a pre-trained _ResNet_ [19] model trained on
ImageNet [35] dataset. The pre-trained model has been shown to boost
performance over low-level features in several computer vision tasks. We use
widely recognized _ResNet_ -152 and its last convolution layer to extract
features instead of the object categories (final layer). The final
convolutional layer outputs 2048 feature maps each of size $7\times 7$, which
is then pooled with a global average to get a 2048-dimensional vector.
Place and Scene Features ($I_{p}$) In order to encode the scene information in
an image, we extract features from a pre-trained _ResNet_ [19] model trained
on Places365 [47] dataset. In this case, we use _ResNet_ -101 and follow the
same encoding process as described for object features.
Hybrid Object and Scene Features ($I_{h}$) We also experiment with a hybrid
model trained on both ImageNet and Places365 datasets that encodes object and
scene information in a single model. To extract these features, we again use a
_ResNet_ -101 model and follow the same encoding process.
Image Sentiment ($I_{s}$) To encode the image sentiment, we use a pre-trained
model [41] that is trained on three million images using weak supervision of
sentiment label from the tweet text. Although the image labels are noisy, the
model has shown superior performance on unseen Twitter testing datasets. We
use their best CNN model based on _VGG-19_ [38]. The image sentiment
embeddings ($I_{se}$) are extracted from the last layer in the model, which
are 4096-dimensional vectors. Additionally, we extract the image sentiment
predictions ($I_{sp}$) from the classification layer that outputs a three-
dimensional vector corresponding to the probabilities of three sentiment
classes (Negative, Neutral and Positive).

###  3.2 Textual Models ($T$)

Since context and semantics of the sentence is shown [2, 6] to be important
for claim detection, we use transformer-based _BERT_ -Base [11] ($T_{BB}$), to
extract contextual word embeddings and employ different pooling strategies to
get a single embedding for the tweet. As different layers of _BERT_ capture
different kinds of information, we experiment with four combinations, i.e., 1)
concatenate the last four hidden layers, 2) sum of the last four hidden
layers, 3) the last hidden layer, and 4) the second last hidden layer. We
finally take an average over the word embeddings to obtain a single vector.

To reduce the domain gap for our Twitter datasets in English, we experiment
with two _BERT_ models. The first variant is called _BERTweet_ [31] ($T_{BT}$)
a BERT-base model that is further pre-trained on 850 million English tweets,
and the second one called _COVID-Twitter-BERT_ [29] ($T_{CT}$), a BERT-large
model trained on 97 million English tweets on the topic of COVID-19. For
Arabic tweets, we experiment with the _AraBERT_ [1] ($T_{AB}$) that is trained
on Arabic news corpus called _OSIAN_ [46] and 1.5 Billion words Arabic corpus
[13]. We also perform two experiments, one with raw tweets and the other with
pre-processing tweets as part of the _AraBERT’s_ language-specific text
processing method.

For English text, with vanilla BERT-base model, we pre-process the text by
following the steps mentioned in Cheema et. al. [7] using the publicly
available text processing tool Ekphrasis [3]. We also show the performance of
vanilla _BERT_ -base on raw tweets ($T_{BB}^{Raw}$) to reflect its sensitivity
towards text pre-processing ($T_{BB}^{Clean}$). For both _BERTweet_ and
_COVID-Twitter-BERT_ , we follow their pre-processing steps, which normalize
text, and additionally replaces user mentions, emails, URLs with special
keywords.

###  3.3 Multimodal Models ($M$)

####  3.3.1 ViLBERT (Vision-and-Language BERT)

We use _ViLBERT_ [25], one of the recent multimodal transformer architectures
that process image and text inputs through two separate transformer-based
streams and combines them through transformer layers with the co-attention. It
eventually outputs co-attended image and text features that can be combined
(added, multiplied or concatenated) to learn a classifier for vision and
language tasks. The authors proposed to use visual grounding as a self-
supervised pre-training task on a large conceptual captions dataset [37]. They
used the model for various downstream tasks involving vision and language,
such as visual question answering, visual commonsense reasoning, and caption-
based image retrieval.

For the image branch, _ViLBERT_ uses state-of-the-art object detection model
_Mask R-CNN_ [18] and extracts top 100 region proposals (boxes) and their
corresponding features. These features are used in a sequence through a
5-layer image transformer, which outputs the image region embeddings. For the
text branch, it uses BERT-base model to get the contextual word embeddings. A
6-layer transformer block with the co-attention follows the individual streams
that outputs the co-attended image and text embeddings.

####  3.3.2 Feature Extraction

In our fixed embedding experiments with a SVM, we experiment with the output
of pooling and last layers of image and text branches. With pooling layers, we
directly concatenate ($M_{pool}^{CAT}$) the image and text outputs. With last
layer outputs we average the image region embeddings and word embeddings to
get one single embedding per modality and then concatenate them
($M_{avg}^{CAT}$). From pooling layers, each modality’s embedding size is a
1024-dimensional vector, and the last layer average of embeddings gives 1024
and 768-dimensional vectors for image and text, respectively. For fine-tuning,
we follow _ViLBERT_ ’s downstream task approach, where the pooling layer
outputs are either added ($M_{pool}^{ADD}$) or multiplied ($M_{pool}^{MUL}$)
and passed to a classifier. For Arabic text, we use Google Translate to
convert the text into English because all _ViLBERT_ models are trained on
English text.

_ViLBERT_ is fine-tuned on several downstream tasks which can be relevant for
encapsulating image-text relationship for our claim detection problem.
Therefore, we experiment with four different pre-trained models, namely,
conceptual captions , image retrieval (_Image-Ret_), grounding referring
expressions (localize an image region given a natural language reference)
(_RefCOCO_), and a multi-task model [26] that is trained on 12 different
tasks.

###  3.4 Classification of Tweets

For our fixed embedding experiments, we train SVM models with each type of
image and text embeddings for binary classification of tweets as shown in
Figure 2 (a). For fine-tuning textual models (Figure 2 (b)), given that we
have relatively small-sized datasets, we only experiment with fine-tuning the
last two and four layers of transformer models for each dataset. We
concatenate the image and text features for multimodal fixed embedding
experiments and train an SVM model over them for classification.

In the case of _ViLBERT_ (Figure 2 (c)), we again train SVM over the extracted
pooled image and text outputs for classification. For fine-tuning, we fix the
individual transformer branches and experiment with fine-tuning the last two
and four co-attention layers to activate the interaction between modalities.
It enables us to see the effect of only the attention mechanism that can show
the benefit of an image and text in claim detection. We use a simple
classifier on top of _ViLBERT_ outputs as recommended by the authors of
_ViLBERT_ , which includes a linear layer for down projecting outputs to 128
dimensions, followed by _ReLU_ (Rectified Linear Unit) non-linear activation
function, a normalization layer and finally a binary classification layer.
Dropout is used to avoid over-fitting, and the fine-tuning is performed by
minimizing the cross-entropy loss.

##  4 Experiments and Results

In this section, we describe all the datasets and their statistics, training
details and hyper-parameters, model details, experimental results, and discuss
them as obtained by different models mentioned in Section 3.

###  4.1 Datasets

We selected the following four publicly available _Twitter_ datasets with
high-quality annotations (which excludes [30], besides its focus on fake
news), three of which are on claim detection and one on conspiracy detection.
The number of tweets in the original datasets is four to fifteen times more as
they were mined for text-based fake news detection. We only selected tweets
that have an image.

CLEF-En [36] \- Released as a part of _CLEF-CheckThat! 2020_ challenge, the
purpose is to identify _COVID-19_ related tweets that are check-worthy claims
vs not check-worthy claims. Only 281 English tweets in the dataset include
images, whereas the original dataset included 964 tweets.

CLEF-Ar [17] \- Released in the same challenge, the dataset consists of 15
topics related to middle east including _COVID-19_ and the purpose is to
identify check-worthy claims. It consists of 2571 Arabic tweets and
corresponding images.

MediaEval [33] \- Released in _MediaEval 2020_ workshop [33] challenge on
identifying 5G and Coronavirus conspiracy tweets. The original dataset has
three classes, 5G and Corona conspiracy, other conspiracies, and no
conspiracy. To make the problem consistent with other datasets in this paper,
we combine conspiracy classes (Corona and others) and treat it as a binary
classification problem. It consists of $1724$ tweets and images.

LESA [16] \- This is a recently proposed dataset of _COVID-19_ related tweets
on the problem of claim detection. Here, the problem is identifying whether a
tweet is a claim or not, and not the claim check-worthiness as in _CLEF-En_.
The original dataset consists of $10\,000$ tweets in English, out of which
only $1395$ consists of images.

We applied 5-fold cross-validation to overcome the issue of low number of
samples in each dataset. We used the ratio of around 72:10:18 for training,
validation, and testing in each data split. Next, we report the experimental
results for different model configurations. The reported results are averaged
across five splits of each dataset. We report accuracy and weighted-F1 measure
to account for label imbalance in all the datasets.

###  4.2 Setup and Hyper-parameters

SVM hyper-parameters: we perform grid search over PCA energy (%) conservation,
regularization parameter C and RBF kernel’s gamma. The parameter range for PCA
varies from 100% (original features) to 95% with decrements of 1. The
parameter range for C and gamma vary between $-1$ to $1$ on a log-scale with
15 steps. For experiments only on the _CLEF-En_ dataset, we use the range
between $-2$ to $0$ for C and gamma, as the number of samples are very low and
needs aggressive regularization. We normalize the final embedding so that $l2$
norm of the vector is $1$.

Fine-tuning BERT and VilBERT: we use a batch size of 4 for _CLEF-En_ and 16
for the other datasets. We train all the models for 6 epochs with a starting
learning rate of $5e-5$ and a linear decay. A dropout with ratio $0.2$ is
applied after the first linear layer in the classifier for regularization
during fine-tuning.

Table 1: The classification results on all datasets using the textual and visual features (see Sections 3.1 and 3.2). Models marked with† are fine-tuning results and the rest are SVM-based. The best result for each group (bold), and the best result for each dataset (bold and underlined) are highlighted.  Model |  CLEF-En [36] |  CLEF-Ar [17] |  LESA [16] |  MediaEval [33]
---|---|---|---|---
| ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
$I_{o}$ | 0.6748 | 0.6180 | 0.6991 | 0.6770 | 0.8223 | 0.7775 | 0.7189 | 0.6968
$I_{p}$ | 0.6033 | 0.5966 | 0.6961 | 0.6558 | 0.8159 | 0.7687 | 0.7215 | 0.6355
$I_{h}$ | 0.6551 | 0.6108 | 0.7052 | 0.6776 | 0.8223 | 0.7744 | 0.7260 | 0.6581
$I_{se}$ | 0.6384 | 0.6223 | 0.7073 | 0.6563 | 0.8143 | 0.7516 | 0.708 | 0.6822
$T_{BB}^{Clean}$ | 0.7501 | 0.7514 | - | - | 0.8279 | 0.8015 | 0.8130 | 0.8026
$T_{BB}^{Raw}$ | 0.7459 | 0.7346 | - | - | 0.8119 | 0.7873 | 0.8298 | 0.8261
$T_{BT}$ | 0.7656 | 0.7661 | - | - | 0.8255 | 0.8023 | 0.8272 | 0.8232
$T_{CT}$ | 0.7178 | 0.7123 | - | - | 0.8175 | 0.8045 | 0.8479 | 0.8479
$T_{AB}$ | - | - | 0.8362 | 0.8307 | - | - | - | -
$T{{}_{BB}^{Clean}}^{\dagger}$ | 0.6942 | 0.6804 | - | - | 0.8319 | 0.809 | 0.8046 | 0.7952
$T_{BT}^{\dagger}$ | 0.7420 | 0.7363 | - | - | 0.84862 | 0.83032 | 0.84072 | 0.83422
$T_{CT}^{\dagger}$ | 0.7146 | 0.6784 | - | - | 0.8303 | 0.8075 | 0.8627 | 0.8604
$T_{AB}^{\dagger}$ | - | - | 0.8431 | 0.8432 | - | - | - | -

###  4.3 Results

Table 1 and Table 2 show the unimodal and multimodal models’ performance for
all the four datasets based on type of features and feature combinations
respectively.

_Unimodal Results_ \- In Table 1, it can be seen that all the visual features
perform poorly in comparison to textual features. This is expected as visual
information on its own cannot indicate whether a social media post makes a
claim unless it has text or it’s a video. Among the four types of visual
models, _Object_ ($I_{o}$) and _Hybrid_ ($I_{h}$) features are slightly
better, probably because the place or scene information (lowest F1 for all
datasets) on its own is not a useful indicator in images for claim detection.
With textual features, _BERT_ models that are further pre-trained on tweets
($T_{BT},T_{BT}^{\dagger}$) and _COVID_ -related data
($T_{CT},T_{CT}^{\dagger}$) perform better in comparison to vanilla _BERT_
($T_{BB}^{Clean},T{{}_{BB}^{Clean}}^{\dagger}$) in at-least three datasets. It
suggests that the tweets’ structure and the domain gap are better captured and
reduced respectively in Twitter corpus pre-trained models. Further,
normalizing ($T_{BB}^{Clean}$) the tweet text delivers better performance than
using the raw text ($T_{BB}^{Raw}$). In SVM training, we observed the sum of
the last four layers of BERT to compute the embeddings performs better than
the other pooling combinations. It indicates that downstream tasks can benefit
from the diverse information in different layers of BERT. Similarly, fine-
tuning the last four layers instead of two (marked with2) gives better
performance across all the datasets with _BERT-base_ ($T_{BB}^{Clean}$†),
_COVID-Twitter-BERT_ ($T_{CT}$†) and _AraBERT_ ($T_{AB}$†).

_Multimodal Results_ \- In Table 2, we can see the effect of combining visual
features with textual features by using a simple concatenation in SVM and also
with multimodal co-attention transformer _ViLBERT_. Although we do not see any
benefit of using the image sentiment embeddings ($I_{se}$) in unimodal models,
here instead, we use the image sentiment predictions ($I_{sp}$) that perform
better or equivalent in comparison to other visual features. For instance, in
case of _CLEF-Ar_ , sentiment predictions $I_{sp}$ with _AraBERT_ ($T_{AB}$†)
gives the best fixed embedding performance. Similarly, combining hybrid
features ($I_{h}$) with _BERT-base_ ($T_{BB}^{Clean}$†) and object features
with _COVID-Twitter-BERT_ ($T_{CT}$†) in case of _LESA_ and _MediaEval_
improves the metrics by 1% over textual SVM models.

With _ViLBERT_ , it is interesting to see that with fixed visual and textual
branches, it can capture some information from image and text with co-
attention to boost performance in case of _LESA_ and _MediaEval_. It is worth
mentioning that the best unimodal textual models for English and Arabic are
pre-trained models further trained on Twitter and language-specific data
corpus. In the case of _ViLBERT_ , there is a wider domain gap, and for
Arabic, the translation process loses quite a bit of information that results
in a drop in performance. Different pooling operations applied for pre-trained
_ViLBERT_ models show more difference in fixed-embedding SVM experiments where
the average pooling ($M_{avg}^{CAT}$) yields a considerable performance, which
we also observed in unimodal SVM experiments. We observed that pre-training
tasks (best two reported in Table 2) also matter, where image retrieval
(Image-Ret) and language reference grounding (RefCOCO) features perform much
better for all the datasets. It is explainable since both tasks require
capturing complex relationships and linking text to specific image regions in
the image, enabling them to perform better for our tasks.

Table 2: Multimodal classification results on all datasets based on combination of textual, visual and multimodal features (see Sections 3.1, 3.2 and 3.3). Models marked with† are fine-tuning results on VilBERT and the rest are SVM-based. _ReCOCO_ and _Image-Ret_ refers to pre-trained _ViLBERT_ models. Layers refers to the number of fine-tuned co-attention layers in VilBERT. Model |  CLEF-En [36] |  CLEF-Ar [17] |  LESA [16] |  MediaEval [33]
---|---|---|---|---
| ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
Best Unimodal | 0.7656 | 0.7661 | 0.8431 | 0.8432 | 0.8486 | 0.8303 | 0.8627 | 0.8604
$T$ -> | $T_{BB}^{Clean}$ |  $T_{AB}$ [1] | $T_{BB}^{Clean}$ |  $T_{CT}$ [29]
$I_{o}$ \+ $T$ | 0.7219 | 0.7053 | 0.8054 | 0.8053 | 0.8311 | 0.7953 | 0.8594 | 0.8566
$I_{p}$ \+ $T$ | 0.7336 | 0.7296 | 0.8184 | 0.8168 | 0.8223 | 0.7955 | 0.8472 | 0.8460
$I_{h}$ \+ $T$ | 0.7259 | 0.7003 | 0.8085 | 0.8060 | 0.8335 | 0.7907 | 0.8549 | 0.8527
$I_{sp}$ \+ $T$ | 0.7557 | 0.7575 | 0.8370 | 0.8319 | 0.8271 | 0.8009 | 0.8485 | 0.8483
Pre-trained -> | RefCOCO | Image-Ret | RefCOCO | Image-Ret
$M_{pool}^{CAT}$ | 0.6980 | 0.6941 | 0.7125 | 0.6990 | 0.8175 | 0.7842 | 0.7357 | 0.7142
$M_{avg}^{CAT}$ | 0.7062 | 0.7022 | 0.7454 | 0.7245 | 0.8175 | 0.7910 | 0.7892 | 0.7832
Layers -> | 2 | 4 | 4 | 4
$M{{}_{pool}^{ADD}}^{\dagger}$ | 0.7339 | 0.7322 | 0.7449 | 0.7214 | 0.8446 | 0.8196 | 0.7989 | 0.7820
$M{{}_{pool}^{MUL}}^{\dagger}$ | 0.7336 | 0.7341 | 0.7449 | 0.7389 | 0.8446 | 0.8191 | 0.7937 | 0.7772
$M{{}_{avg}^{CAT}}^{\dagger}$ | 0.7182 | 0.7121 | 0.7466 | 0.7415 | 0.8319 | 0.8063 | 0.8014 | 0.7900

###  4.4 Discussion of Results

We can summarize the findings of our experiments as follows: 1) Domain-
specific languages models should be preferred for downstream tasks such as
claim detection or fake news, where underlying meaning and context of certain
words (like COVID) is essential, 2) Multimodality certainly helps as seen with
multimodal transformer models, where activating interaction through co-
attention layers between fixed unimodal embeddings improves the performance in
two datasets, 3) To further understand underlying multimodal dynamics it might
be better to explicitly model multimodal relationships, for instance,
importance of image or correlation between image-text in addition to claim
detection, 4) Certain pre-training tasks in _ViLBERT_ are better suited for
downstream tasks and need further introspection on larger datasets, and
lastly, 5) Visual models need to be better adapted to social media images, for
instance, the models used here are not sufficient for diagrams or images with
large text, which constitute around 30-40% of _LESA_ and _MediaEval_ datasets.

##  5 Conclusion

In this paper, we have investigated the role of images and tweet text for two
problems related to fake news, claim, and conspiracy detection. For this
purpose, we combined several state-of-the-art CNN features for images with
BERT features for text. We observed the performance improvement over unimodal
models in two out of four _Twitter_ datasets over two languages. We also
experimented with the recently proposed multimodal co-attention transformer
_ViLBERT_ and observed a promising performance using both image and text even
with relatively small-sized datasets. In future work, we will look into other
ways to include external knowledge in domain-independent claim detection
models without relying on different domain-specific language models. Second,
we plan to investigate multimodal transformers in more detail and analyze if
the performance does scale with more data in similar tasks. Finally, to
address the limitation of visual models, we will consider models that can deal
with text and graphs in images and extract suitable features.

## Acknowledgements

This work was funded by European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement no 812997.

## References

  * [1] Antoun, W., Baly, F., Hajj, H.: AraBERT: Transformer-based model for Arabic language understanding. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. pp. 9–15. European Language Resource Association (2020)
  * [2] Barrón-Cedeno, A., Elsayed, T., Nakov, P., Da San Martino, G., Hasanain, M., Suwaileh, R., Haouari, F., Babulkov, N., Hamdan, B., Nikolov, A., et al.: Overview of checkthat! 2020: Automatic identification and verification of claims in social media. In: International Conference of the Cross-Language Evaluation Forum for European Languages. pp. 215–236. Springer (2020)
  * [3] Baziotis, C., Pelekis, N., Doulkeridis, C.: DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pp. 747–754. Association for Computational Linguistics (2017)
  * [4] Cao, J., Qi, P., Sheng, Q., Yang, T., Guo, J., Li, J.: Exploring the role of visual content in fake news detection. Disinformation, Misinformation, and Fake News in Social Media pp. 141–161 (2020)
  * [5] Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.): Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org (2020)
  * [6] Chakrabarty, T., Hidey, C., McKeown, K.: IMHO fine-tuning improves claim detection. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 558–563. Association for Computational Linguistics (2019)
  * [7] Cheema, G.S., Hakimov, S., Ewerth, R.: Check_square at checkthat! 2020 claim detection in social media via fusion of transformer and syntactic features. In: Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece. CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org (2020)
  * [8] Cheema, G.S., Hakimov, S., Ewerth, R.: Tib’s visual analytics group at mediaeval’20: Detecting fake news on corona virus and 5g conspiracy. MediaEval 2020 Workshop (2020)
  * [9] Claveau, V.: Detecting fake news in tweets from text and propagation graph: Irisa’s participation to the fakenews task at mediaeval 2020. In: MediaEval 2020 Workshop (2020)
  * [10] Daxenberger, J., Eger, S., Habernal, I., Stab, C., Gurevych, I.: What is the essence of a claim? cross-domain claim identification. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2055–2066. Association for Computational Linguistics (2017)
  * [11] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. pp. 4171–4186 (2019)
  * [12] Dogan, F., et al.: Detecting Real-time Check-worthy Factual Claims in Tweets Related to US Politics. Ph.D. thesis (2015)
  * [13] El-Khair, I.A.: 1.5 billion words arabic corpus. ArXiv abs/1611.04033 (2016)
  * [14] Giachanou, A., Rosso, P., Crestani, F.: Leveraging emotional signals for credibility detection. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France. pp. 877–880. ACM (2019)
  * [15] Giachanou, A., Zhang, G., Rosso, P.: Multimodal multi-image fake news detection. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA). pp. 647–654. IEEE (2020)
  * [16] Gupta, S., Singh, P., Sundriyal, M., Akhtar, M.S., Chakraborty, T.: Lesa: Linguistic encapsulation and semantic amalgamation based generalised claim detection from online content. arXiv preprint arXiv:2101.11891 (2021)
  * [17] Hasanain, M., Haouari, F., Suwaileh, R., Ali, Z., Hamdan, B., Elsayed, T., Barrón-Cedeño, A., Da San Martino, G., Nakov, P.: Overview of CheckThat! 2020 Arabic: Automatic identification and verification of claims in social media. In: Cappellato et al. [5]
  * [18] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy. pp. 2980–2988. IEEE Computer Society (2017)
  * [19] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA. pp. 770–778. IEEE Computer Society (2016)
  * [20] Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on Multimedia. pp. 795–816 (2017)
  * [21] Khattar, D., Goud, J.S., Gupta, M., Varma, V.: MVAE: multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference, WWW 2019, San Francisco, CA, USA. pp. 2915–2921. ACM (2019)
  * [22] Levy, R., Bilu, Y., Hershcovich, D., Aharoni, E., Slonim, N.: Context dependent claim detection. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. pp. 1489–1500. Dublin City University and Association for Computational Linguistics (2014)
  * [23] Lippi, M., Torroni, P.: Context-independent claim detection for argument mining. In: Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina. pp. 185–191. AAAI Press (2015)
  * [24] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [25] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada. pp. 13–23 (2019)
  * [26] Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA. pp. 10434–10443. IEEE (2020)
  * [27] Majithia, S., Arslan, F., Lubal, S., Jimenez, D., Arora, P., Caraballo, J., Li, C.: ClaimPortal: Integrated monitoring, searching, checking, and analytics of factual claims on Twitter. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 153–158. Association for Computational Linguistics (2019)
  * [28] Manh Duc Tuan, N., Quang Nhat Minh, P.: Fakenews detection using pre-trained language models and graph convolutional networks. In: MediaEval 2020 Workshop (2020)
  * [29] Müller, M., Salathé, M., Kummervold, P.E.: Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. arXiv preprint arXiv:2005.07503 (2020)
  * [30] Nakamura, K., Levy, S., Wang, W.Y.: Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In: Proceedings of the 12th Language Resources and Evaluation Conference. pp. 6149–6157. European Language Resources Association (2020)
  * [31] Nguyen, D.Q., Vu, T., Tuan Nguyen, A.: BERTweet: A pre-trained language model for English tweets. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 9–14. Association for Computational Linguistics (2020)
  * [32] Nikolov, A., Da San Martino, G., Koychev, I., Nakov, P.: Team_Alex at CheckThat! 2020: Identifying check-worthy tweets with transformer models. In: Cappellato et al. [5]
  * [33] Pogorelov, K., Schroeder, D.T., Burchard, L., Moe, J., Brenner, S., Filkukova, P., Langguth, J.: Fakenews: Corona virus and 5g conspiracy task at mediaeval 2020\. In: MediaEval 2020 Workshop (2020)
  * [34] Rosenthal, S., McKeown, K.: Detecting opinionated claims in online discussions. In: 2012 IEEE sixth international conference on semantic computing. pp. 30–37. IEEE (2012)
  * [35] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211–252 (2015)
  * [36] Shaar, S., Nikolov, A., Babulkov, N., Alam, F., Barrón-Cedeño, A., Elsayed, T., Hasanain, M., Suwaileh, R., Haouari, F., Da San Martino, G., Nakov, P.: Overview of CheckThat! 2020 English: Automatic identification and verification of claims in social media. In: Cappellato et al. [5]
  * [37] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565. Association for Computational Linguistics (2018)
  * [38] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, Conference Track Proceedings (2015)
  * [39] Singhal, S., Shah, R.R., Chakraborty, T., Kumaraguru, P., Satoh, S.: Spotfake: A multi-modal framework for fake news detection. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). pp. 39–47. IEEE (2019)
  * [40] Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural processing letters 9(3), 293–300 (1999)
  * [41] Vadicamo, L., Carrara, F., Cimino, A., Cresci, S., Dell’Orletta, F., Falchi, F., Tesconi, M.: Cross-media learning for image sentiment analysis in the wild. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). pp. 308–317 (2017)
  * [42] Vosoughi, S., Roy, D., Aral, S.: The spread of true and false news online. Science 359(6380), 1146–1151 (2018)
  * [43] Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., Gao, J.: EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK. pp. 849–857. ACM (2018)
  * [44] Williams, E., Rodrigues, P., Novak, V.: Accenture at checkthat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models. In: Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece. CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org (2020)
  * [45] Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemometrics and intelligent laboratory systems 2(1-3), 37–52 (1987)
  * [46] Zeroual, I., Goldhahn, D., Eckart, T., Lakhouaja, A.: OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop. pp. 175–182. Association for Computational Linguistics (2019)
  * [47] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017)
  * [48] Zlatkova, D., Nakov, P., Koychev, I.: Fact-checking meets fauxtography: Verifying claims about images. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 2099–2108. Association for Computational Linguistics (2019)