# Duluth at SemEval-2020 Task 12:
Offensive Tweet Identification in English with Logistic Regression

Ted Pedersen
Department of Computer Science
University of Minnesota
Duluth, MN 55812 USA
<EMAIL_ADDRESS>
###### Abstract

This paper describes the Duluth systems that participated in SemEval–2020 Task
12, Multilingual Offensive Language Identification in Social Media
(OffensEval–2020). We participated in the three English language tasks. Our
systems provide a simple Machine Learning baseline using logistic regression.
We trained our models on the distantly supervised training data made available
by the task organizers and used no other resources. As might be expected we
did not rank highly in the comparative evaluation: 79th of 85 in Task A, 34th
of 43 in Task B, and 24th of 39 in Task C. We carried out a qualitative
analysis of our results and found that the class labels in the gold standard
data are somewhat noisy. We hypothesize that the extremely high accuracy ($>$
90%) of the top ranked systems may reflect methods that learn the training
data very well but may not generalize to the task of identifying offensive
language in English. This analysis includes examples of tweets that despite
being mildly redacted are still offensive.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

The goal of the OffensEval–2020 task [Zampieri et al., 2020] is to identify
offensive language in tweets, and to determine if specific individuals or
groups are being targeted. We relied on traditional Machine Learning methods
implemented in Scikit [Pedregosa et al., 2011] to build logistic regression
classifiers from distantly supervised training examples of offensive tweets
[Rosenthal et al., 2020]. Our methods are well known and so will only be
described briefly. Instead, our primary focus in this paper is on a post-
evaluation qualitative analysis of both our results and the underlying task
data.

Identifying offensive, abusive, and hateful language is a challenging problem
that is drawing increasing attention both among the general public and in the
research community (e.g.,[Fortuna and Nunes, 2018, Schmidt and Wiegand,
2017]). These are difficult problems since what is offensive depends not only
on the words being used but also on the situation in which they occur. Whether
something is offensive may depend on answers to questions like : Is the source
of such language in a position of power? Is the target a member of a
marginalized group? Is there a difference in age, race, religion, or social
status of the source and the target? The number of real-world factors that may
determine if language is offensive is impossible to enumerate, and leads to a
high degree of polysemy in candidate offensive words that can only be
unraveled by considering the social situation in which they are used. For
example, words that are usually considered slurs may be used within a
marginalized group as a means of bonding or identifying. The same is true of
profanity, which when used within an in-group or among friends may not be
offensive, while the same language directed at an outsider might well be
offensive [Waseem et al., 2018, Sap et al., 2019, Wiegand et al., 2019].

##  2 Experimental Data

OffensEval–2020 is made up of three tasks that went through the final
evaluation stage in late February and early March 2020. Task A is to classify
a tweet as offensive (OFF) or not (NOT). Task B takes the tweets identified as
OFF from Task A and determines if hey are targeted insults (TIN) or not (UNT).
Task C considers the targeted insult tweets from Task B and classifies them as
being directed against an individual (IND), group (GRP) or other entity (OTH).
There is a cascading relationship between the tasks, where Task C requires the
output of Task B, and Task B requires the output of Task A. This is the same
set of labels and tasks as used in OffensEval–2019.

OffensEval–2020 provided a large corpus of training tweets known as SOLID that
was created via distant supervision [Rosenthal et al., 2020]. Task A included
9,089,139 tweets, Task B provided 188,974 and Task C 188,973. The distant
supervision used to create SOLID was seeded with the the manually labeled OLID
corpus from OffensEval–2019 [Zampieri et al., 2019a]. OLID is a much smaller
corpus of 13,240 training tweets and 860 test tweets which was also available
to OffensEval–2020 participants.

SOLID and OLID are different in that tweets in OLID are labeled with
categories whereas in SOLID tweets are scored on a continuous scale of 0 to 1
to reflect the collective judgment of the models used as a part of distant
supervision. As such the SOLID data did not provide a specific indication as
to the boundaries between categories.

##  3 Assigning Labels to the Experimental Data

We made a few significant decisions early on regarding the data. First, we
elected not to use the 2019 OLID data. We participated in OffensEval–2019 and
observed some potential inconsistencies in the OLID training data [Pedersen,
2019]. We also felt that since SOLID was seeded with OLID that there would be
no particular advantage to also using OLID. There is a tradeoff here between
using a smaller amount of manually annotated data (OLID) versus a much larger
sample of potentially noisier distantly supervised data (SOLID). Second, we
decided to directly map the 2020 SOLID training data to categories. This
required us to draw somewhat perilous and arbitrary boundaries through a real
valued space for each task.

We studied the distribution of scores in Task A and observed that the median
in the SOLID training data was .25, and that the standard deviation was .185.
This suggested that the vast majority of tweets were not considered offensive,
and so we selected the value of .8 for our cutoff. Based on our manual review
of the training data we felt that tweets in Task A with a score greater than
.8 were very likely to be offensive. We realized that we could set this
boundary lower (perhaps .75 or .70) and still include many offensive tweets,
but wanted to choose a boundary that might at least give the possibility of
high precision results.

After making this cutoff, only about 4% (356,811) of the training tweets were
labeled as offensive. We used this same threshold with Task B and C. For Task
B, any tweet with a score less than .2 was considered targeted. In Task C
separate scores were provided for each of the three possible categories, so
any tweet with a score greater than .8 was considered to be targeted against
an individual, group or other.

##  4 Methods

The Duluth system is a slightly modified version of a tweet emoticon
classification system developed for SemEval–2018 [Jin and Pedersen, 2018].
This system does some light pre-processing that largely leaves the tweets
intact (but does basic cleaning and regularization of punctuation) and then
identifies unigrams and bigrams as candidate features. We took the same
approach for each task, where we learnt a logistic regression classifier from
the training data where category boundaries were set as described in Section
2.

##  5 Experimental Results

The official rankings in OffensEval were based on macro–averaged F1, and we
also report accuracy and per-class precision, recall, and $F1$ scores.

The results of the Duluth systems were significantly below top ranked systems.
For Task A our macro F-1 score was .7713 while the top ranked system attained
.92226, for Task B we obtained .53816 versus a top rank of .74618, and for
Task C the Duluth system scored .57437 against .71450.

That said, our qualitative analysis of our results suggests that there is some
noise in the class labels of the gold data, and so we are uncertain if
significantly higher accuracy would translate into meaningful performance on
the task of identifying offensive language in English. In the confusion
matrices that follow, the distribution of gold answers (ground truth) are
shown on the columns, and the system predictions are on the rows.

###  5.1 Task A

Task A asks whether a tweet is offensive (OFF) or not (NOT). This task had the
largest amount of training data (9,0899,140 examples), of which we considered
4% offensive (OFF) and 96% not (NOT).

We found that the Duluth system was very conservative in predicting a tweet as
OFFensive, assigning that label to only 16% (603) of the 3,887 test instances.
This resulted in a classifier that achieved high precision in identifying
offensive tweets. We hypothesized that this would be the case when we set the
cutoff of .8 for offensive tweets (see Section 3).

Table 1 shows that the most significant confusion for our system was
disagreement with the gold standard over what was offensive. Our system
considered fully half (540) of the gold standard offensive tweets to be NOT
offensive (shown in bold face in Table 1).

| OFF | NOT |  | P | R | F1
---|---|---|---|---|---|---
OFF | 540 | 63 | 603 | .896 | .500 | .642
NOT | 540 | 2,744 | 3,284 | .164 | .978 | .281
| 1,080 | 2,807 | 3,887 |  |  |
Table 1: Task A Duluth : Logistic Regression, accuracy = .845, majority classifier = .722 A1601 | @USER Not ashamed to say it, but i jacked off to the thought of having
---|---
| sex with this guy. What i wouldnt do to go to bed with him
A848 | @USER @USER 100% liar Pakistan is the country which damaged the image of Muslim
| nation around the world
A388 | dear white men, pink dick is out of season
A2403 | @USER Back to Somila. You hate America. You hate her people.
| You promote terror. We don’t need you.
A1267 | It’s always that 1 b**ch that wanna be related to everybody
A1448 OFF? | @USER 6’9 with the soul of a 7’4 n**ga
A2127 OFF?? | That. Ain’t. Got. Shit. To. Do. With. Me. Idc who came before me, it’s IRRELEVANT.
| cause I’ve been b**ched about it before Do you feel it’s a must to disclose
who’ve you slept
| with to someone during the talking stage?
A2816 OFF?? | @USER He will fight them on the beaches … what is it with these pathetic little boys
| and their dreams of glory?
A3005 OFF?? | @USER @USER Such left leaning bullshit rules that don’t apply to the LEFT as they
| punish the RIGHT instead
A237 OFF?? | @USER Spoiler alert.. George was set up by the corrupt Dems and Hillary,
| just like President Trump! #FisaBringsDownTheHouse
Table 2: Task A OFF-NOT category examples - gold standard indicates offensive
while Duluth system predicts not offensive.

Given the significant disagreement between gold OFFensive and predicted NOT,
we carried out a qualitative analysis of randomly selected test instances that
fell into this category. Specifically, we randomly selected 10 instances from
the OFF-NOT category 10 different times. Table 2 shows one of these ten trials
(which was itself randomly selected). We’ve indicated (with OFF??) five
instances where we believe a case can be made that the tweet is not offensive.
Overall in our ten trials we found anywhere from two to five instances per
trial where there could be reasonable doubt as to whether the tweet was
offensive. We carried out a similar analysis with the three other cross-
classification categories for Task A and show examples of that in Tables 9,
10, and 11 in the Appendix.

We noted in-group uses of the N-word and the B-word seemed to be automatically
considered offensive (in all tasks). A1448 is an example, where the tweet may
in fact be intended as a compliment. The automatic classification of profanity
and slurs as offensive is a known and significant problem, since this can lead
to all in-group speech among members of a marginalized group as being unfairly
labeled as offensive [Waseem et al., 2018, Sap et al., 2019, Wiegand et al.,
2019].

These observations suggest that a highly accurate classifier trained on this
data may simply be learning anomalies of this sort and may not generalize well
to the problem of identifying offensive language.

###  5.2 Task B

Task B takes the tweets labeled as OFFensive in Task A and determines if they
are targeted insults (TIN) or not (UNT). We can see in Table 3 that the Duluth
system was again very conservative, this time in considering a tweet to be
targeted. In this task our precision has fallen fairly dramatically, and our
accuracy lags behind even a majority classifier. Our system disagrees with the
gold standard 70% of the time (in 593 of 850 tweets, shown in bold face in
Table 3) and is far less likely to consider a tweet a targeted insult.

| UNT | TIN |  | P | R | F1
---|---|---|---|---|---|---
UNT | 533 | 593 | 1,126 | .473 | .932 | .628
TIN | 39 | 257 | 296 | .132 | .068 | .090
| 572 | 850 | 1,422 |  |  |
Table 3: Task B Duluth : Logistic Regression, accuracy = .556, majority classifier = .598 BC162 | can’t believe it went from “idc i’ll eat ur ass” to “i’m actually not sexual anymore”
---|---
| b**ch i’m gonna kill u
BC962 |  She just wanna f*ck me cuz I be making money
BC406 |  B**ches who got they shit together ain’t bout to argue or be in drama everyday.
| That’s a BROKE B**CH HOBBY!
BC233 | @USER Facebook sucks ass, used to use it all the time under my real name up
| until 2017/2016 that’s when it got unusable for me.
BC1152 | @USER It’s a place to shit and piss in
BC1253 OFF? | being nice to people is not flirt u dumb f*ck.
BC1233 OFF? | make me a playlist of songs you wanna f*ck me to
BC1047 OFF? TIN?? | @USER K increased my volume to hear this shit and I WAS NOT ALONE
BC1406 OFF?? TIN?? | @USER The stupidity is immense, I’m sure their grandads will be spinning
| around in tha graves.
BC963 OFF?? TIN?? | @USER It’s an ugly weapon, but war is ugly business.
Table 4: Task B TIN-UNT category examples - gold standard indicates targeted
insult while Duluth system predicts untargeted.

Given the significant level of disagreement between the gold standard and
Duluth predictions we again carried out a qualitative analysis of
misclassifications. We focused on those cases where the gold standard said an
offensive tweet was a targeted insult, and where the Duluth system classified
it as untargeted. We refer to this as TIN-UNT.

Table 4 shows ten randomly selected examples from Task B. The first five shown
seem to be correct as offensive targeted insults. However, we feel that there
are questions about the other five. We do not find them obviously offensive,
and we are uncertain that there is really anything being targeted in the last
three instances (TIN??). We observed a similar distribution of question in a
total of 10 random trials of 10 instances each.

###  5.3 Task C

For Task C, the input is a targeted insult and the task is to decide if the
target is an individual (IND), group (GRP) or other (OTHER). As can be seen in
Table 5 the Duluth classifier’s precision was relatively high for the IND
category, and overall the classifier exceeded the majority baseline.

| IND | GRP | OTH |  | P | R | F1
---|---|---|---|---|---|---|---
IND | 537 | 69 | 47 | 653 | .822 | .926 | .871
GRP | 36 | 113 | 21 | 170 | .665 | .595 | .628
OTH | 7 | 8 | 12 | 27 | .444 | .150 | .224
| 580 | 190 | 80 | 850 |  |  |
Table 5: Task C Duluth : Logistic Regression, accuracy = .779, majority
classifier = .682

In this task the most common confusion between the gold standard and Duluth
predictions is that of tweets where the gold standard considers them as
targeting a group (GRP) but the system predicts individual (IND). As we did
for previous tasks we took several random trials (in this case only three
given the smaller size of this class) and we show one of those trials in Table
6.

BC518 | that whole ‘hot girl’ shit is just an excuse for b**ches to be they true selves
---|---
BC668 | @USER i didnt see that what the f*ck ig stans are a disgusting breed
BC836 GRP?? | @USER @USER @USER Welp, I’m done here. F*ck off nazi.
BC1714 GRP?? | @USER 238 followers tells me you’re the dumb F-u k making up Shiit
BC1803 OFF? | Muted this idiot and his bandwagon of fools , awon oloshi alayi ni ironu
BC1347 OFF?? | Skinny dudes the worst to cuddle with. Might as well fall asleep on a pile of remotes.
BC198 OFF?? | I like my men slightly clingy. Like be ambitious and motivated but once you’re
| off work be on my ass.
BC374 OFF?? | the f*ck is a bootycall i hate you all
BC1490? OFF?? | I hate a lazy b**ch , and b**ch isnt gender specific
BC1739 OFF?? | You can’t make a n**ga loyal that’s just some shit that can’t be taught
BC659 OFF?? | i was at 422 followers and b**ches thought it’d be cool to unfollow me lmao
| what the f*ck okay
Table 6: Task C GRP-IND category examples - gold standard indicates targeted
insult against group while Duluth system predicts against individual.

Here we have significant questions about the majority of the tweets in the
GRP-IND class. Only the first two instances appear to clearly be targeted
insults against a group. Thereafter we have two instances where an individual
is the target, and then six examples where is is unclear if the tweet is even
offensive in the first place. It appears that much of the confusion may result
over the use of the N-word and the B-word, as has been discussed for previous
tasks. We suspect that the use of these terms may automatically make them
offensive and targeted against a group. However, this is perhaps too coarse of
a view and overlooks a great deal of nuance.

##  6 Discussion

The qualitative analysis described above suggests that any use of profanity
often resulted in a tweet being labeled as offensive. This had a negative
effect on the downstream tasks which took such tweets and tried to determine
if the offense was targeted, and who was the target.

We wonder if the use of .5 as the boundary between offensive or not, targeted
or not, etc. may have been too lenient. By contrast, the Duluth system used a
cutoff of .8. Our manual inspection suggested that tweets that scored above .8
tended to be somewhat harsher and more offensive than those with lower scores,
although this is more of an intuition at this point rather than a conclusive
finding.

Table 7 and 8 (both in the Appendix) show the distribution of scores in the
training data for Tasks A, B and C. These tables show the significance of the
choice of .8 versus .5 as the category cutoff in that the total number of
tweets that were considered offensive or targeted was much smaller with the .8
cutoff. Whether the tweets further down the tail are consistently more
offensive is an interesting question for future work.

##  7 Ethical Considerations

Identifying offensive language is a problem without a clear definition. The
challenge is that a particular statement may or may not be offensive depending
on the context in which it is used, and that depends on the nature of the
source and target of such a statement. To further complicate matters,
offensive statements may also be true statements. A totalitarian leader may be
deeply offended by statements detailing human rights abuses committed by their
regime, or a celebrity may be offended that a recent criminal charge is widely
reported. There are also legitimately unclear boundaries. There are people who
may be genuinely offended for religious or cultural reasons by any use of
profanity. Since their preferences are genuine, should that be the standard
that offensive language detection relies on? If so, the problem reduces to the
simpler task of identifying profanity, which would then unfairly classify
common everyday even friendly uses of profanity as offensive and subject it to
flagging, removal, or other sanctions.

It seems clear that we can’t make blanket assumptions about what is offensive.
Instead, we need to be very specific as to what our boundaries are for a
particular corpus or task. In particular, we should be mindful of who is the
target of such a language (as was done in both [Zampieri et al., 2019b] and
[Zampieri et al., 2020]), but we should also consider the source, and the
context in which the language occurs. On a broader scale, we should reflect on
who holds power [Barabas et al., 2020], and who gains and loses power if a
statement is flagged as offensive.

While evaluation scores are important to advancing progress in NLP, there are
some serious problems with making decisions about how well a problem is being
solved simply by looking at such measures. We believe that in–depth
qualitative analyses of mismatches between gold standard data and system
predictions must be carried out in addition to providing more quantitative
results.

State of the art methods for many NLP problems have been shown to reach high
levels of accuracy simply by learning spurious patterns in the training data
without making a dent in the underlying problem we’d like to solve (e.g.,
[Niven and Kao, 2019]). We must acknowledge the possibility of this occurring
in offensive language and hate speech detection and carry out qualitative
analyses of our results in addition to the more typical quantitative ones.

The danger of simply focusing on high accuracy is that we can be lulled into a
false sense of success. In OffensEval–2020, 59 teams reached a macro F-1 score
of .90 or better. Does this mean the problem of offensive language detection
has been solved? We do not believe this is the case, nor do we think that any
of the task participants would draw this conclusion. However, an outsider
viewing such results might wrongly conclude that these methods and models are
ready for use in the wild. This could result in a flawed offensive language
detector being deployed which would over or under identify problematic
language, most likely to the disadvantage of already marginalized populations.

##  8 Conclusion

This is the second year a Duluth system participated in OffensEval. Last year
in OffensEval–2019 we took a relatively simple Machine Learning and rule-based
approach and carried out an analysis of the results from the task. Our
findings [Pedersen, 2019] were similar to this year, where the Duluth system
did not rank particularly high. Also, we observed both in 2019 and 2020 that
there appear to be some limitations in the gold standard annotations. In
particular, in both years it seemed like there were quite a few false
positives in the gold data, where tweets were labeled as offensive in Task A
and potentially targeting in Task B when in fact they were not. It is not
surprising that the 2019 OLID data and 2020 SOLID data would show similar
characteristics, since OLID was the seed used for the distant supervision that
created SOLID.

In general it would appear virtually impossible to reliably annotate data
without some background knowledge about the participants in the dialogue, as
well as larger cultural contexts that might be at work (e.g, [Patton et al.,
2019, Frey et al., 2020]). That said we believe that annotated corpora is an
important resource for this problem, and we need to continue to refine our
processes for the creation of the same. In the creation of our own corpora we
are working to develop Data Statements [Bender and Friedman, 2018] and plan to
incorporate domain experts in the annotation process.

## References

  * [Barabas et al., 2020]  Chelsea Barabas, Colin Doyle, JB Rubinovitz, and Karthik Dinakar.  2020\.  Studying up: Reorienting the study of algorithmic fairness around issues of power.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, page 167–176, New York, NY, USA. Association for Computing Machinery.
  * [Bender and Friedman, 2018]  Emily M. Bender and Batya Friedman.  2018\.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  Transactions of the Association for Computational Linguistics, 6:587–604.
  * [Fortuna and Nunes, 2018]  Paula Fortuna and Sérgio Nunes.  2018\.  A survey on automatic detection of hate speech in text.  ACM Comput. Surv., 51(4):85:1–85:30, July.
  * [Frey et al., 2020]  W. R. Frey, D. U. Patton, M. B. Gaskell, and K. A. McGregor.  2020\.  Artificial intelligence and inclusion: Formerly gang-involved youth as domain experts for analyzing unstructured twitter data.  Social Science Computer Review, 38(1):42–56.
  * [Jin and Pedersen, 2018]  Shuning Jin and Ted Pedersen.  2018\.  Duluth UROP at SemEval-2018 task 2: Multilingual emoji prediction with ensemble learning and oversampling.  In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 482–485, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * [Niven and Kao, 2019]  Timothy Niven and Hung-Yu Kao.  2019\.  Probing neural network comprehension of natural language arguments.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy, July. Association for Computational Linguistics.
  * [Patton et al., 2019]  D.U. Patton, P. Blandfort, W.R. Frey, S. Karaman, and M. Gaskell.  2019\.  Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators.  In Proceedings of the Hawaii International Conference on Systems Sciences (HICSS-52), Honolulu, Hawaii.
  * [Pedersen, 2019]  Ted Pedersen.  2019\.  Duluth at SemEval-2019 task 6: Lexical approaches to identify and categorize offensive tweets.  In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 593–599, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics.
  * [Pedregosa et al., 2011]  F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  2011\.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830.
  * [Rosenthal et al., 2020]  Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov.  2020\.  A Large-Scale Semi-Supervised Dataset for Offensive Language Identification.  In arxiv.
  * [Sap et al., 2019]  Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith.  2019\.  The risk of racial bias in hate speech detection.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy, July. Association for Computational Linguistics.
  * [Schmidt and Wiegand, 2017]  Anna Schmidt and Michael Wiegand.  2017\.  A survey on hate speech detection using natural language processing.  In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, April. Association for Computational Linguistics.
  * [Waseem et al., 2018]  Zeerak Waseem, James Thorne, and Joachim Bingel.  2018\.  Bridging the gaps: Multi task learning for domain transfer of hate speech detection.  In Jennifer Golbeck, editor, Online Harassment, pages 29–55. Springer International Publishing, Cham.
  * [Wiegand et al., 2019]  Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer.  2019\.  Detection of Abusive Language: the Problem of Biased Datasets.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 602–608, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * [Zampieri et al., 2019a]  Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  2019a.  Predicting the Type and Target of Offensive Posts in Social Media.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1415–1420.
  * [Zampieri et al., 2019b]  Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  2019b.  SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval).  In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval).
  * [Zampieri et al., 2020]  Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin.  2020\.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020).  In Proceedings of SemEval.

##  Appendix A Appendix : Supplementary Tables

| Task A | TaskB
---|---|---
range | percent | count | percent | count
0 - .1 | 0.001 | 11,385 | 0.000 | 2
.1 - .2 | 0.284 | 2,580,150 | 0.103 | 19,479
.2 - .3 | 0.351 | 3,189,564 | 0.409 | 77,312
.3 - .4 | 0.135 | 1,229,725 | 0.155 | 29,212
.4 - .5 | 0.069 | 629,455 | 0.125 | 23,546
.5 - .6 | 0.044 | 404,462 | 0.114 | 21,599
.6 - .7 | 0.037 | 338,467 | 0.045 | 8,533
.7 - .8 | 0.038 | 349,121 | 0.043 | 8,174
.8 - .9 | 0.039 | 351,169 | 0.006 | 1,118
.9 - 1.0 | 0.001 | 5,642 | 0.000 | 0
| 1.000 | 9.089,140 | 1.000 | 188,975
Table 7: Training data score distribution Task A. Duluth system interpreted scores $\geq$ .8 as offensive (OFF), for Task B scores $\geq$ .2 indicated untargeted (UNT). Note that .5 was used as the cutoff for both tasks in the gold data. | IND | GRP | OTH
---|---|---|---
range | percent | count | percent | count | percent | count
0 - .1 | 0.004 | 703 | 0.027 | 5,143 | 0.018 | 3,382
.1 - .2 | 0.042 | 7,871 | 0.462 | 87,276 | 0.342 | 64,700
.2 - .3 | 0.059 | 11,205 | 0.257 | 48,498 | 0.339 | 64,051
.3 - .4 | 0.082 | 15,563 | 0.114 | 21,581 | 0.223 | 42,171
.4 - .5 | 0.141 | 26,643 | 0.067 | 12,650 | 0.061 | 11,599
.5 - .6 | 0.186 | 35,064 | 0.037 | 6,971 | 0.013 | 2,499
.6 - .7 | 0.195 | 36,942 | 0.023 | 4,345 | 0.003 | 513
.7 - .8 | 0.249 | 47,113 | 0.012 | 2,300 | 0.000 | 55
.8 - .9 | 0.042 | 7,848 | 0.001 | 210 | 0.000 | 4
.9 - 1.0 | 0.000 | 22 | 0.000 | 0 | 0.000 | 0
| 1.000 | 188,974 | 1.000 | 188,974 | 1.000 | 188,974
Table 8: Training data score distribution Task C, higher scores indicate targeted individual (IND), group (GRP), or other (OTH). Duluth system interpreted scores $\geq$ .8 as IND, GRP, or OTH whereas gold data used .5 as the cutoff. A3321 | are an ugly i hate straight your hetero frick fake
---|---
A3410 | Literally everyone wants a piece of my girlfriends ass. I need to treat her better smh.
A2257 | @USER Get the gains or die alone b**ch boy.
A2793 | @USER What a liar. White terrorists are his followers.
A406 | @USER f*ck off Deb. The planet is dying and you don’t care
A171 | WAHT THE F**CK THEY ARE COMING TO LA f*ck THIS SHUT
| IM GOING TO WORK
A945 OFF?? | imagine thinking the death of a 20 year old man with an unborn child is funny …
| some people really are sick and pathetic
A217 OFF?? | @USER You mean because you’re a coward
A3285 OFF?? | I wanna do the hot wing challenge shit look so good
A3456 OFF?? | Guys who start off the conversation sexually, you suck that’s just a turn off bye
Table 9: Task A OFF-OFF category example : gold standard and Duluth system prediction agree on OFFensive. Those indicated with OFF?? may in fact be NOT. A2402 | @USER Idk, maybe they suddenly found out what a
---|---
| great person you are and decided to stan
A1113 | @USER it’s not about chara, bby
A1271 | I remember when RBG broke her ribs MOST of what I saw
| from both sides of the isle were well wishes and hopes she would feel better
soon.
A562 | @USER It was great
A1890 | @USER what’s up and thanks for the follow @USER says hello :)
A1813 | @USER @USER That would have made your other life easier.
A3501 | Great day to bless up
A1084 | Rest easy, Toni Morrison. You were the best of us.
A2228 | Click and Learn Some Valuable information!
A1738 | @USER you know the ones who aren’t appreciated bro it’s okay
Table 10: Task A NOT-NOT category examples : gold standard and Duluth system prediction agree on NOT offensive. Our analysis supports that these are NOT. A2193 | @USER @USER U call the whole Pastor a liar?
---|---
A1437 | @USER @USER @USER That sucks!
A996 | @USER UGH! That sucks
A427 | Stop dissecting these sick manifestos, they do not point to anything but a deranged,
| mentally ill mind.
A1230 | my friend thinks my dad is a liar when really I’m a liar and he just helped me lie
A3818 | @USER @USER @USER The devil is a liar
A1884 | Wet pants. What the f*ck
A1082 | Fat, ugly and sick of it
A2060 | *at the zoo* these fursonas suck
A1480 NOT? | conspiracy theory : billie eilish farted on live for y’all weird ass grown men
| to stop sexualizing her
Table 11: Task A NOT-OFF category examples - gold standard indicates NOT
offensive while Duluth system predicts OFFensive. Our analysis suggests gold
standard is correct except possibly for case indicated by NOT?.


# EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools
for Comprehensive Affective Analysis

Zhiwei Liu1, Kailai Yang1, Tianlin Zhang1, Qianqian Xie1111Corresponding
author. Qianqian is now affiliated with Yale University. The work was done
while she was at The University of Manchester., Zeping Yu1, Sophia Ananiadou1
1The University of Manchester
{zhiwei.liu-2, kailai.yang, tianlin.zhang,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Sentiment analysis and emotion detection are important research topics in
natural language processing (NLP) and benefit many downstream tasks. With the
widespread application of large language models (LLMs), researchers have
started exploring the application of LLMs based on instruction-tuning in the
field of sentiment analysis. However, these models only focus on single
aspects of affective classification tasks (e.g. sentimental polarity or
categorical emotions), and overlook the regression tasks (e.g. sentiment
strength or emotion intensity), which leads to poor performance in downstream
tasks. The main reason is the lack of comprehensive affective instruction
tuning datasets and evaluation benchmarks, which cover various affective
classification and regression tasks. Moreover, although emotional information
is useful for downstream tasks, existing downstream datasets lack high-quality
and comprehensive affective annotations. In this paper, we propose EmoLLMs,
the first series of open-sourced instruction-following LLMs for comprehensive
affective analysis based on fine-tuning various LLMs with instruction data,
the first multi-task affective analysis instruction dataset (AAID) with 234K
data samples based on 3 classification tasks and 2 regression tasks to support
LLM instruction tuning, and a comprehensive affective evaluation benchmark
(AEB) with 8 regression tasks and 6 classification tasks from various sources
and domains to test the generalization ability of LLMs. We propose a series of
EmoLLMs by fine-tuning LLMs with AAID to solve various affective instruction
tasks. We compare our model with a variety of LLMs and sentiment analysis
tools on AEB, where our models outperform all other open-sourced LLMs and
sentiment analysis tools, and surpass ChatGPT and GPT-4 in most tasks, which
shows that the series of EmoLLMs achieve the ChatGPT-level and GPT-4-level
generalization capabilities on affective analysis tasks, and demonstrates our
models can be used as affective annotation tools. The project will be
available at https://github.com/lzw108/EmoLLMs/.

##  1 Introduction

Emotions and sentiments play a crucial role in shaping our lives. Our words
and actions serve as indicators of our emotional states Liu (2020). Leveraging
natural language processing (NLP) techniques such as Emotion Detection (ED)
and Sentiment Analysis (SA), we can delve into the analysis of human
interactions, enabling us to comprehend people’s emotional responses toward
particular subjects Hakak et al. (2017). Specifically, SA tasks typically
involve predicting the polarity (usually positive, negative, or neutral),
along with the strength of this tone Thelwall et al. (2010), and emotion
detection tasks often involve classifying data into fine-grained emotion
categories (e.g. Ekman Ekman (1992), Plutchik Plutchik (1980)) or predicting
the intensity of emotions Xie et al. (2018). These affective information are
proven as useful features for many downstream tasks, including mental health
analysis Zhang et al. (2023b), misinformation detection Liu et al. (2023), and
empathetic dialogue systems Ma et al. (2020).

Pre-trained language models (PLMs) such as BERT Devlin et al. (2018) and
RoBERTa Liu et al. (2019), have shown excellent performance in classification
tasks. Many studies have applied them to sentiment analysis or emotion
detection tasks Bello et al. (2023); Liao et al. (2021); Yin et al. (2020).
However, these PLMs are limited by the scale of model parameters and the
training corpus, resulting in a lack of comprehensive understanding and
generalization ability for complex tasks Zhang et al. (2023a), which restricts
the effectiveness of affective analysis, especially in affective regression
tasks Zhang et al. (2020). Compared to PLMs, LLMs have the characteristic of
having an enormous parameter size, typically reaching hundreds of billions or
more, which gives them stronger generalization abilities in downstream tasks
and enables them to handle tasks with intricate complexities Zhang et al.
(2023c). Many researchers have started exploring the application of LLMs in
the field of sentiment analysis, which achieved excellent performance by fine-
tuning open-source LLMs on sentiment analysis tasks Zhang et al. (2023a, c);
Lei et al. (2023). However, these studies only focus on sentiment/emotion
classification tasks and overlook regression tasks (e.g. sentiment strength,
emotion intensity), which provide more fine-grained affective features Akhtar
et al. (2020) and are proven useful in many scenarios Zhang et al. (2021);
Qureshi et al. (2020); Consoli et al. (2022). The major reason is the lack of
a comprehensive instruction-based sentiment analysis dataset and evaluation
benchmark.

Moreover, though emotional information is proven useful for downstream tasks,
existing downstream datasets lack emotion-related resources such as
sentiment/emotion labels. Therefore, many works use affective analysis tools
(e.g. VADER Hutto and Gilbert (2014a),
TextBlob222https://textblob.readthedocs.io/) to provide sentiment annotations.
For example, in Govindasamy and Palanichamy (2021), the authors utilized the
TextBlob library to calculate sentiment scores and fed them into a depression
detection classifier. Additionally, some studies employ transfer learning
methods, by applying models trained on other sentiment analysis or emotion-
labeled datasets to automatically annotate the emotions expressed in
downstream task datasets Chan et al. (2023); Dong et al. (2022). However,
these tools or methods can only annotate one aspect of sentiment analysis
tasks, resulting in limited coverage of emotional features.

To address the above issues, we propose a suite of LLMs, instruction-tuning
datasets, and an evaluation benchmark for multi-task affective analysis. We
first construct the multi-task affective analysis instruction dataset (AAID)
with 234K data samples to support LLM instruction tuning, which is based on
SemEval-2018 Task1: Affect in Tweet Mohammad et al. (2018); Mohammad and
Kiritchenko (2018), including five tasks: emotion intensity regression,
ordinal classification of emotion intensity, sentiment strength regression,
sentiment classification, and multi-label emotion classification. Based on the
AAID dataset, we propose a series of emotional large language models
(EmoLLMs), the first open-sourced instruction-following LLMs for comprehensive
affective analysis, by performing multi-task instruction tuning on LLMs. To
evaluate the performance and generalizability of EmoLLMs, we also construct an
affective generalization testing instruction benchmark (AEB) based on 14
affective analysis datasets collected from various platforms and sources,
which include 8 regression tasks and 6 classification tasks.

Based on AEB, we evaluate EmoLLMs, a variety of open-sourced LLMs, close-
sourced LLMs (i.e. ChatGPT and GPT-4), and several sentiment analysis tools.
The experimental results indicate that the series of EmoLLMs overtake all
other open-sourced LLMs, and sentiment analysis tools, and exceed ChatGPT and
GPT-4 in 7 regression tasks and 4 classification tasks. These results
demonstrate that EmoLLMs achieve a comparable capability with ChatGPT and
GPT-4 in most affective analysis tasks. EmoLLMs can serve as comprehensive
affective annotation tools for annotating data from different platforms and
sources.

Our main contributions are as follows:

  * •

We build AAID, the first multi-task affective analysis instruction tuning
data, and AEB, the first affective generalization testing instruction
benchmark.

  * •

We introduce a series of EmoLLMs, the first open-source instruction following
LLMs for comprehensive affective analysis.

  * •

We compare EmoLLMs with other LLMs on AEB. Additionally, we conduct a
comprehensive analysis of the affective analysis capabilities of ChatGPT and
GPT-4. Our models achieve SOTA performance on the AEB dataset compared to
other open-sourced LLMs and present ChatGPT-level and GPT-4-level
generalization capabilities, establishing their potential as effective tools
for affective annotation.

The structure of this paper is as follows: Section 2 introduces the related
work about sentiment analysis models and open-sourced LLMs. Section 3
introduces the proposed method. Specifically, Section 3.1 introduce the task
definition. Section 3.2 presents the construction process of AAID and AEB.
Section 3.3 introduces the training process of EmoLLMs. Section 4 presents the
experiment results on AEB and analyses the performance of each model. Section
5 concludes this paper by summarizing our findings.

##  2 Related Work

###  2.1 Affective analysis model

There have been various affective analysis tools proposed, such as VADER Hutto
and Gilbert (2014a), and TextBlob. Although these tools are convenient to use,
their effectiveness in sentiment analysis is not ideal He et al. (2022). In
recent years, many studies have focused on fine-tuning PLMs to enhance their
capabilities in the field of sentiment analysis. Bello et al. (2023) combines
BERT with other deep learning models (e.g. CNN, RNN, LSTM) to improve the
ability of the model in short and simple text sentiment analysis. Liao et al.
(2021) proposed a multi-task model based on RoBERTa for aspect-category
sentiment analysis. Yin et al. (2020) proposed the SentiBERT model, which
focuses on the field of sentiment analysis. SentiBERT integrates a recursive
constituency tree based on BERT to better capture compositional sentiment
semantics. Recently, numerous studies have embarked on investigating the
utilization of LLMs in sentiment analysis, resulting in remarkable performance
gains in sentiment analysis tasks. Zhang et al. (2023a) proposed a retrieval-
augmented LLMs for financial sentiment analysis which, leveraging additional
context from external sources, is able to outperform LLM baselines such as
ChatGPT and LLaMA by margins of between 15% and 48%. Similarly, Lei et al.
(2023) also used a simple yet effective retrieval module to enhance the
emotion recognition capability of LLM in dialogue. Zhang et al. (2023c)
developed a context and emotion knowledge-tuned LLM, namely DialogueLLM,
obtained by fine-tuning LLM with multimodal (i.e., texts and videos) emotional
dialogues, which achieved SOTA results on three emotion recognition in
conversations (ERC) datasets. However, these PLMs and LLMs only focus on
individual aspects of affective analysis, lacking the ability to predict
sentiment strength and emotion intensity.

###  2.2 Open Sourced Large Language Models

Although ChatGPT and GPT-4 have shown excellent performance in various fields,
their closed-source availability affects the progress of scientific research.
Therefore, numerous studies are dedicated to democratizing LLMs, such as the
LLaMA series Touvron et al. (2023a, b), OPT series Zhang et al. (2022), BLOOM
series Workshop et al. (2022), and Falcon Penedo et al. (2023). Based on the
open-source LLMs, many efforts have been made to develop models with
instruction-following capabilities like ChatGPT by training on extensive
instruction-tuning datasets (e.g.
Alpaca333https://crfm.stanford.edu/2023/03/13/alpaca.html and the
Vicuna444https://lmsys.org/blog/2023-03-30-vicuna/). Recently, there has been
a lot of domain-specific work aimed at improving the performance of LLM in
specific domains by training on domain-specific instruction datasets. Such as
FinMA Xie et al. (2023) in the finance domain, MentalLLaMA Yang et al. (2023)
in the mental health domain, TimeLlaMA Yuan et al. (2023) used for temporal
reasoning, and ExTES-LLaMA Zheng et al. (2023) in emotional support chatbots.
Our work is the first open-sourced LLM series for comprehensive multitask
affective analysis.

##  3 Methods

The goal of this work is to evaluate and enhance the comprehensive and complex
affective analysis capabilities of LLMs. To achieve this objective, we build
the first affective analysis instruction dataset (AAID) to support LLMs tuning
for comprehensive affective analysis tasks. We propose EmoLLMs, a series of
emotional LLMs by fine-tuning LLMs based on AAID. Furthermore, we construct a
comprehensive affective evaluation benchmark to test the generalization
ability of LLMs.

###  3.1 Task Definition

Similar to Yang et al. (2023) in handling mental health analysis tasks, we
also approach affective analysis as a generative task, where a generative
model (i.e., an autoregressive language model $P_{\phi}(y|x)$ parameterized by
pre-trained weights $\phi$) is employed as the foundation, which is unlike
previous discriminative and regression models. This model is capable of
simultaneously addressing $N$ affective analysis tasks, such as sentiment
polarity and strength prediction, emotion classification and intensity
prediction. Each task t is represented by a subset of training context-target
pairs: $D_{t}={(q_{i}^{t},r_{i}^{t})}_{i={1,2,...N_{t}}}$, where $q$ is a
token sequence containing the task description, target text, and query, and
$r$ is another sequence containing the query answer (i.e., classification
result or regression result). All subsets are combined into a training
dataset: $D$. The model is optimized based on this merged data, aiming to
maximize the conditional language modeling objective to enhance the accuracy
of predictions.

###  3.2 Instruction Tuning Data Building

We build the instruction dataset based on the SemEval-2018 Task 1: Affect in
Tweets, which includes a series of highly annotated sentiment analysis
subtasks Mohammad et al. (2018); Mohammad and Kiritchenko (2018).

####  3.2.1 3.2.1 Raw Data

SemEval 2018 Task1 contains five subtasks: 1. emotion intensity regression
(EI-reg), 2. ordinal classification of emotion intensity (EI-oc), 3. valence
(sentiment) regression (V-reg), 4. ordinal classification of valence
(sentiment) (V-oc), and 5. emotion classification (E-c).

EI-reg: Given a tweet and an emotion E (anger, fear, joy, sadness), determine
the intensity of E that best represents the mental state of the tweeter—a
real-valued score between 0 (least E) and 1 (most E);

EI-oc: Given a tweet and an emotion E (anger, fear, joy, sadness), classify
the tweet into one of four ordinal classes (0: no E can be inferred. 1: low
amount of E can be inferred. 2: moderate amount of E can be inferred. 3: high
amount of E can be inferred) of intensity of E that best represents the mental
state of the tweeter;

V-reg: Given a tweet, determine the intensity of sentiment or valence (V) that
best represents the mental state of the tweeter—a real-valued score between 0
(most negative) and 1 (most positive);

V-oc: Given a tweet, classify it into one of seven ordinal classes (from -3:
very negative to 3: very positive), corresponding to various levels of
positive and negative sentiment intensity, that best represents the mental
state of the tweeter;

E-c: Given a tweet, classify it as ‘neutral or no emotion’ or as one, or more,
of eleven given emotions (anger, anticipation, disgust, fear, joy, love,
optimism, pessimism, sadness, surprise, trust) that best represent the mental
state of the tweeter.

Table 1: Statistics of the data. Raw denotes the raw data from SemEval-2018 Task 1: Affect in Tweets. Instruction denotes the converted instruction data based on Raw. Task |  Raw (Train/Dev) |  Instruction (Train/Dev) |  Source
---|---|---|---
EI-reg, EI-oc |  |  |
anger |  1701/388 |  17010/3880 |  Twitter
fear |  2252/389 |  22520/3890 |  Twitter
joy |  1616/290 |  16160/2900 |  Twitter
sadness |  1533/397 |  15330/3970 |  Twitter
V-reg, V-oc |  1181/449 |  11810/4490 |  Twitter
E-c |  6838/886 |  68380/8860 |  Twitter

####  3.2.2 3.2.2 AAID: Affective Analysis Instruction Dataset

Figure 1: An overview of multi-task instruction tuning of EmoLLaMA for
multiple affective analysis tasks.

We construct the instruction dataset based on the raw data. Due to the limited
quantity of the original dataset, we utilize 10 different instructions for
each task to augment the training set and validation set. The data statistics
are presented in Table 1. Specifically, we build instruction-tuning samples
based on some templates. Table 2 describes the specific instruction templates
for each task, and Figure 1 provides corresponding examples (Taking EmoLLaMA
as the example and each task selects one [task prompt] as an example). [task
prompt] describes the instructions for each specific task. The word ’Tweet’
can be adjusted based on the actual task. The [input text] refers to the
content of the raw data. The final [output] should be adjusted based on the
specific task to provide sentiment classification, sentiment strength, emotion
classification, or emotion intensity.

Table 2: Templates for constructing prompts for instruction dataset. [task prompt] denotes task instruction, [input text] is from the raw data, [emotion] can be anger, fear, joy, sadness, [output] is the output from LLM. Task |  Prompt Template
---|---
EI-reg |  Task: [task prompt] Tweet: [input text] Emotion E: [emotion] Intensity score: [output]
EI-oc |  Task: [task prompt] Tweet: [input text] Emotion E: [emotion] Intensity class: [output]
V-reg |  Task: [task prompt] Tweet: [input text] Intensity score: [output]
V-oc |  Task: [task prompt] Tweet: [input text] Intensity class: [output]
E-c |  Task: [task prompt] Tweet: [input text] This tweet contains emotions: [output]

####  3.2.3 3.2.3 AEB: Affective Evaluation Benchmark

Table 3: Statistics of AEB. ”R” denotes the regression task, followed by the intensity range. ”SC” denotes the sentiment classification task and ”EC” denotes the emotion classification task, followed by the number of categories. Dataset | Size | Type | Source | Dataset | Size | Type | Source | Dataset | Size | Type | Source
---|---|---|---|---|---|---|---|---|---|---|---
EI-reg | 4068 | R[0,1] | Twitter | V-Amazon | 1000 | R [-4,4] | Amazon | SST | 2210 | R [0,1] | Movie reviews
EI-oc | 4068 | EC(4) | Twitter | V-Movies | 1000 | R [-4,4] | Movies reviews | SST-5 | 2210 | SC (5) | Movie reviews
V-reg | 937 | R[0,1] | Twitter | V-NYT | 1000 | R [-4,4] | New York Times | TDT | 692 | SC (3) | Twitter
V-oc | 937 | SC(7) | Twitter | V-Tweet | 1000 | R [-4,4] | Twitter | GoEmotion | 5427 | EC (7) | Reddit
E-c | 3259 | EC(11) | Twitter | EmoBank | 1000 | R [1,5] | News, blogs. etc. |  |  |  |

We first collect the test data from SemEval-2018 Task 1: Affect in Tweets. To
test the robustness of our model, a random instruction from the ten
instructions used in train augment is selected for each instance in the test
set. We also collect additional sentiment analysis or emotion detection
datasets from various sources and domains to test the generalizability of our
model. We construct the AEB following the template format provided in Table 2.
Except for the four datasets from VADER, all other datasets utilize the
original test dataset. Table 3 shows the statistic details.

Datasets used in Valence Aware Dictionary for sEntiment Reasoning (VADER)
Hutto and Gilbert (2014b): There are four datasets from different social media
platforms with sentiment intensity (Valence) scores within [-4,4]: V-Amazon
(Amazon reviews snippets), V-Movies (Movies reviews snippets, collected from
rotten.tomatoes.com), V-NYT (New York Times editorial snippets), V-Tweet
(Tweets). We randomly sampled 1000 instances from each dataset for
generalizability testing.

EmoBank Buechel and Hahn (2022, 2017): This dataset was collected from News,
blogs, fictions, letters etc. and contains three dimensions, which were
manually annotated with emotion according to the psychological Valence-
Arousal-Dominance scheme with scores within [1,5].

Stanford Sentiment Treebank (SST) Socher et al. (2013): It is collected from
movie reviews, which is the first corpus with fully labeled parse trees,
allowing for a comprehensive analysis of the compositionality of sentiment in
language. In SST555https://huggingface.co/datasets/sst, each sentence is
assigned a floating-point label that indicates the degree of positive
sentiment, ranging from 0.0 to 1.0. while in
SST5666https://huggingface.co/datasets/SetFit/sst5, each sentence is annotated
with five labels: very positive, positive, neutral, negative, very negative.

Target Dependent Twitter Sentiment Classification (TDT) Dong et al. (2014): It
is a Twitter sentiment classification dataset collected from post comments for
the celebrities, products, and companies, which is annotated manually with
three labels (negative, neutral, positive). To facilitate our generalizability
testing, we restored the entities that were masked in the original data,
creating a standard sentence-level sentiment analysis dataset.

GoEmotion Demszky et al. (2020): It is a multi-label classification dataset
collected from Reddit comments, which consists of 28 emotion labels, including
the neutral. However, the original dataset with 28 emotion labels is
imbalanced. To mitigate this issue, we select the ”Ekman” option from the
dataset provided by the authors, which consists of 7 emotion labels, including
the neutral.

Since the first five datasets are collected from the same sources as the AAID,
the remaining data comes from different platforms and sources, we divide AEB
into two parts for comparison. The former is referred to as AEB-1, used to
test the training effectiveness of the models. The latter is called AEB-2,
which is suitable for testing the generalization ability of models.

###  3.3 EmoLLMs

We build EmoLLMs by fine-tuning various LLMs based on AAID. We train three
EmoLLaMA models based on LLaMA2 Touvron et al. (2023b): EmoLLaMA-7B, EmoLLaMA-
chat-7B, EmoLLaMA-chat-13B by fine-tuning LLaMA-7B, LLaMA-chat-7B, LLaMA-
chat-13B, where LLaMA-chat-7B and LLaMA-chat-13B are the first open-source
LLMs tuned with reinforcement learning from human feedback (RLHF) Stiennon et
al. (2020). We also train EmoOPT, and EmoBLOOM based on OPT-13B Zhang et al.
(2022) and BLOOM-7B Workshop et al. (2022)) respectively. All models are
trained for three epochs based on AdamW optimizer Loshchilov and Hutter
(2017), utilizing early stopping techniques Dodge et al. (2020) to prevent
overfitting, and leveraging DeepSpeed Rasley et al. (2020); Yao et al. (2023)
to reduce memory usage. We set the batch size to 256. The initial learning
rate is set to 1e-6 with a warm-up ratio of 5%, and the maximum model input
length is set to 2048. All models are trained on two Nvidia Tesla A100 GPUs,
each with 80GB of memory.

##  4 Evaluation Benchmark

###  4.1 Base Models

PLMs: Sentiment analysis and emotion detection are typically regarded as
classification tasks, while intensity prediction is considered a regression
task. We select some commonly used PLMs as baseline models, which can only
fine-tuned on a single task, including BERT, RoBERTa, and one domain-specific
pre-trained model (i.e. SentiBERT Yin et al. (2020)). We add a fully connected
neural layer to each model, which is used for classification or regression.
For EI-reg and V-reg tasks, we utilize the mean squared error (MSE) loss
function. For EI-oc and V-oc tasks, we use cross-entropy loss. For multi-label
task E-c, we adopt binary cross-entropy with logits loss.

Zero-shot/few-shot methods (LLMs without fine-tuning): With the emergence of
LLMs, zero-shot and few-shot learning have become effective approaches for
solving numerous tasks. We select Falcon-7b-instruct Penedo et al. (2023),
Vicuna-13b-v1.5, LLaMA-chat-7B and LLaMA-chat-13B to perform zero-shot
prompting on the instruction dataset. In addition, we employ zero-shot and
few-shot prompting methodologies with the closed-source LLM ChatGPT and GPT-4.
We select at least one piece of data for each emotion category or label
category to serve as few-shot prompts.

Emotion-based instruction-tuning methods: In addition to the EmoLLMs series
models, we also fine-tuned BART Lewis et al. (2019), T5 Raffel et al. (2020)
using the same instructional dataset as baseline models to further evaluate
the effectiveness of our models.

###  4.2 Evaluation methods

For AEB-1, we use the official evaluation
metric777https://competitions.codalab.org/competitions/17751, Pearson
correlation coefficient (pcc), as the evaluation metric for EI-reg, EI-oc,
V-reg, and V-oc and use accuracy, micro-F1 (mi-F1), macro-F1 (ma-F1) for E-c.
Additionally, the official evaluation also incorporates secondary evaluation
metrics. For the regression tasks, they also use pearson correlation for a
subset of the test set that includes only those tweets with intensity score
greater or equal to 0.5. For the ordinal classification tasks, they also use
pearson correlation for a subset of the test set that includes only those
tweets with intensity classes low X, moderate X, or high X (where X is an
emotion), and use weighted quadratic kappa on the full test set, and adopt
weighted quadratic kappa on the some-emotion subset of the test set. Due to
space limitations, we do not list the secondary evaluation, and its
conclusions are consistent with the primary evaluation results.

For AEB-2, we apply accuracy, and macro-F1 as evaluation metrics for affective
classification tasks. For regression tasks, we use the Pearson correlation
coefficient (pcc) Cohen et al. (2009) as the evaluation metric.

Table 4: Evaluation results on AEB-1. Some results are referenced from Yin et al. (2020); Zhang et al. (2020); Htait and Azzopardi (2021). ”FS” denotes few-shot method. Unmarked LLMs all adopt zero-shot method. ”ave” denotes macro-average. ”acc” denotes accuracy. ”mi-F1” denotes micro-F1. ”ma-F1” denotes macro-F1. The evaluation metric for the first four tasks is the Pearson correlation coefficient. The first line is the score of the top 1 on the SemEval-2018 Task1 leaderboard. model | EI-reg | EI-oc |  V-reg |  V-oc | E-c
---|---|---|---|---|---
|  ave |  anger |  fear |  joy |  sadness |  ave |  anger |  fear |  joy |  sadness |  valence |  valence |  acc |  mi-F1 |  ma-F1
Leaderboard(1) |  0.799 |  0.827 |  0.779 |  0.792 |  0.798 |  0.695 |  0.706 |  0.637 |  0.720 |  0.717 |  0.873 |  0.856 |  0.609 |  0.724 |  0.592
| PLMs
BERT-base |  0.785 |  0.800 |  0.781 |  0.783 |  0.742 |  0.683 |  0.698 |  0.656 |  0.712 |  0.665 |  0.840 |  0.805 |  0.567 |  0.718 |  0.568
RoBERTa-base |  0.717 |  0.670 |  0.736 |  0.769 |  0.694 |  0.664 |  - |  - |  - |  - |  0.845 |  0.772 |  0.563 |  0.721 |  0.536
SentiBERT |  0.722 |  0.724 |  0.740 |  0.731 |  0.691 |  0.665 |  - |  - |  - |  - |  0.835 |  0.763 |  0.535 |  0.700 |  0.522
| Zero-shot/few-shot methods
Falcon |  0.114 |  0.147 |  0.082 |  0.095 |  0.131 |  0.033 |  0.022 |  0.017 |  0.031 |  0.061 |  0.135 |  0.189 |  0.190 |  0.318 |  0.253
Vicuna |  0.281 |  0.307 |  0.257 |  0.260 |  0.299 |  0.214 |  0.238 |  0.193 |  0.186 |  0.241 |  0.298 |  0.579 |  0.220 |  0.359 |  0.253
LLaMA-7B-chat |  0.194 |  0.176 |  0.257 |  0.097 |  0.247 |  0.120 |  0.112 |  0.138 |  0.115 |  0.114 |  0.094 |  0.497 |  0.257 |  0.414 |  0.286
LLaMA-13B-chat |  0.488 |  0.524 |  0.506 |  0.398 |  0.526 |  0.194 |  0.262 |  0.178 |  0.119 |  0.216 |  0.312 |  0.568 |  0.274 |  0.424 |  0.302
ChatGPT |  0.599 |  0.637 |  0.573 |  0.569 |  0.618 |  0.455 |  0.500 |  0.428 |  0.363 |  0.529 |  0.637 |  0.748 |  0.382 |  0.546 |  0.429
ChatGPT-FS |  0.550 |  0.572 |  0.482 |  0.587 |  0.560 |  0.473 |  0.502 |  0.410 |  0.407 |  0.573 |  0.739 |  0.791 |  0.413 |  0.563 |  0.466
GPT-4 |  0.656 |  0.699 |  0.575 |  0.686 |  0.667 |  0.620 |  0.656 |  0.579 |  0.618 |  0.629 |  0.811 |  0.788 |  0.444 |  0.572 |  0.497
GPT-4-FS |  0.679 |  0.704 |  0.654 |  0.679 |  0.678 |  0.562 |  0.623 |  0.523 |  0.515 |  0.585 |  0.825 |  0.793 |  0.460 |  0.582 |  0.515
| Emotion-based instruction-tuning methods
EmoBART |  0.795 |  0.798 |  0.803 |  0.795 |  0.782 |  0.725 |  0.705 |  0.742 |  0.723 |  0.729 |  0.851 |  0.835 |  0.528 |  0.686 |  0.548
EmoT5 |  0.783 |  0.785 |  0.797 |  0.798 |  0.751 |  0.717 |  0.703 |  0.733 |  0.726 |  0.707 |  0.852 |  0.836 |  0.559 |  0.712 |  0.568
EmoOPT |  0.825 |  0.827 |  0.830 |  0.837 |  0.805 |  0.753 |  0.739 |  0.751 |  0.762 |  0.759 |  0.887 |  0.843 |  0.532 |  0.680 |  0.550
EmoBLOOM |  0.791 |  0.802 |  0.797 |  0.790 |  0.776 |  0.732 |  0.725 |  0.717 |  0.746 |  0.740 |  0.857 |  0.822 |  0.528 |  0.683 |  0.552
EmoLLaMA-7B |  0.822 |  0.819 |  0.821 |  0.837 |  0.809 |  0.743 |  0.738 |  0.722 |  0.768 |  0.745 |  0.879 |  0.843 |  0.545 |  0.695 |  0.563
EmoLLaMA-chat-7B |  0.824 |  0.825 |  0.830 |  0.832 |  0.810 |  0.751 |  0.748 |  0.754 |  0.764 |  0.739 |  0.876 |  0.827 |  0.534 |  0.693 |  0.540
EmoLLaMA-chat-13B |  0.831 |  0.827 |  0.835 |  0.843 |  0.817 |  0.763 |  0.755 |  0.764 |  0.777 |  0.755 |  0.886 |  0.860 |  0.537 |  0.696 |  0.545

###  4.3 Results

####  4.3.1 (1) Results on AEB-1

The evaluation results on AEB-1 are shown in Table 4. The first line is the
score of the top 1 on the SemEval-2018 Task1 leaderboard.

Comparison between EmoLLMs and PLMs, Zero-shot/few-shot methods: For EmoLLMs,
we chose EmoLLaMA-chat-13B, which shows the best overall performance, to
compare with other categories. The results in Table 4 show that EmoLLaMA-
chat-13B outperforms all other LLMs and surpasses the top ranking in the first
four tasks of AEB-1. For the complex tasks EI-reg and EI-oc, EmoLLaMA-chat-13B
shows high improvement compared to top 1, with respective increases of 3.2%
(EmoLLaMA:0.831, top1:0.799) and 6.8% (EmoLLaMA:0.763, top1:0.695). For
independent raw task fine-tuning methods, although these PLMs are trained on
extensive datasets and fine-tuned separately for each task, the results do not
surpass the original top 1 scores. The findings demonstrate that general PLMs
are more prone to overlooking important information compared to LLMs when
dealing with affective regression tasks and fine-grained sentiment
classification tasks. For zero-shot/few-shot methods, we can observe that this
category of methods performs poorly compared to other fine-tuning approaches,
especially in the EI-reg and EI-oc tasks. This indicates that the LLMs without
fine-tuning struggle to handle the issue of emotion intensity effectively (We
also test BART, T5, OPT and BLOOM in zero-shot and few-shot methods, but their
response is highly irrelevant).

Comparison between EmoLLMs: We can observe from Table 4 that EmoLLMs all
perform well compared with LLMs without fine-tuning. EmoT5 performs the best
on the emotion classification task E-c (ma-F1) (EmoT5:0.568, EmoLLaMA:0.545),
but it does not perform as well as other models on regression tasks (e.g. EI-
reg(ave): EmoT5:0.783, EmoLLaMA:0.831). Although EmoOPT slightly outperforms
EmoLLaMA in a few regression tasks (e.g. V-reg: EmoOPT:0.887, EmoLLaMA:0.886),
it still lags behind EmoLLaMA in most tasks.

In conclusion, our proposed instruction-tuning strategy for sentiment analysis
tasks outperforms PLMs and all LLMs without fine-tuning, achieving the best
comprehensive performance. Compared to other instruction-tuned EmoLLMs,
EmoLLaMA demonstrates a more comprehensive and integrated capability in
affective analysis.

####  4.3.2 (2) Results on AEB-2

Table 5: Evaluation results on AEB-2 dataset. All evaluation results are based on the zero-shot approach. ’pcc’ denotes Pearson correlation coefficient. ’V’ denotes Valence. ’V-A’ denotes V-Amazon. ’V-M’ denotes V-Movies. ’V-T’ denotes V-Tweet. ’A’ denotes arousal. ’D’ denotes dominance. ’acc’ denotes accuracy. model | V-A | V-M | V-NYT | V-T | SST | EmoBank | GoEmotion | SST5 | TDT
---|---|---|---|---|---|---|---|---|---
| pcc | pcc | pcc | pcc | pcc | V-pcc | A-pcc | D-pcc | acc | ma-F1 | acc | ma-F1 | acc | ma-F1
VADER | 0.565 | 0.446 | 0.464 | 0.862 | 0.450 | - | - | - | - | - | - | - | 0.510 | 0.266
TextBlob | 0.490 | 0.372 | 0.288 | 0.666 | 0.408 | - | - | - | - | - | - | - | 0.434 | 0.435
Falcon | 0.492 | 0.530 | 0.340 | 0.449 | 0.205 | 0.092 | 0.059 | 0.009 | 0.168 | 0.223 | 0.231 | 0.179 | 0.355 | 0.285
Vicuna | 0.634 | 0.592 | 0.320 | 0.580 | 0.733 | 0.184 | 0.140 | 0.002 | 0.250 | 0.307 | 0.293 | 0.253 | 0.312 | 0.269
LLaMA-chat-13B | 0.348 | 0.479 | -0.011 | 0.300 | 0.811 | 0.237 | 0.248 | -0.036 | 0.278 | 0.337 | 0.346 | 0.281 | 0.436 | 0.437
ChatGPT | 0.601 | 0.709 | 0.419 | 0.560 | 0.854 | 0.554 | 0.320 | -0.121 | 0.342 | 0.407 | 0.500 | 0.397 | 0.552 | 0.559
GPT4 | 0.616 | 0.727 | 0.510 | 0.778 | 0.872 | 0.723 | 0.364 | 0.193 | 0.327 | 0.401 | 0.543 | 0.504 | 0.532 | 0.538
EmoBART | 0.770 | 0.661 | 0.650 | 0.853 | 0.634 | 0.670 | 0.059 | 0.101 | 0.318 | 0.366 | 0.341 | 0.323 | 0.529 | 0.535
EmoT5 | 0.838 | 0.728 | 0.668 | 0.889 | 0.724 | 0.714 | 0.239 | 0.064 | 0.327 | 0.374 | 0.373 | 0.376 | 0.578 | 0.583
EmoOPT | 0.883 | 0.842 | 0.767 | 0.904 | 0.815 | 0.713 | 0.120 | 0.247 | 0.287 | 0.331 | 0.286 | 0.255 | 0.573 | 0.528
EmoBLOOM | 0.848 | 0.852 | 0.649 | 0.856 | 0.807 | 0.663 | 0.106 | 0.243 | 0.302 | 0.348 | 0.409 | 0.353 | 0.546 | 0.551
EmoLLaMA-7B | 0.866 | 0.838 | 0.732 | 0.902 | 0.819 | 0.717 | 0.219 | 0.209 | 0.330 | 0.366 | 0.296 | 0.260 | 0.599 | 0.579
EmoLLaMA-chat-7B | 0.885 | 0.835 | 0.797 | 0.910 | 0.822 | 0.728 | 0.192 | 0.226 | 0.371 | 0.392 | 0.400 | 0.362 | 0.554 | 0.554
EmoLLaMA-chat-13B | 0.868 | 0.815 | 0.780 | 0.906 | 0.797 | 0.726 | 0.332 | 0.218 | 0.350 | 0.369 | 0.412 | 0.399 | 0.574 | 0.578

In order to evaluate the generalizability of EmoLLMs, we execute experiments
on the AEB-2 that are not included in the training process (detailed
descriptions can be found in Table 3). All models apply zero-shot method. We
compare the series of EmoLLMs with ChatGPT, GPT4, several open-source LLMs
(i.e. LLaMA-chat, Falcon, and Vicuna) and several sentiment analysis tools
(i.e. VADER, TextBlob). Table 5 presents the experiment results. For EmoLLMs,
it is worth noting that, since we use labels ranging from 0 to 1 when fine-
tuning the model on the regression dataset, we also use the range of 0 to 1
for predictions during the generalization testing of regression tasks.
Afterward, we map these predictions to the corresponding range of the data.

Comparision between EmoLLMs and LLMs without fine-tuning: We still choose
EmoLLaMA as the representative for EmoLLMs. From Table 5, we can see that
EmoLLaMA series outperform ChatGPT, GPT-4, and LLMs without fine-tuning in
most regression tasks. In the first four regression tasks, EmoLLaMA overtakes
GPT-4 by over 10%, Although EmoLLaMA performs less well than ChatGPT and GPT-4
in SST and Emobank-Arousal, the difference is less than 5%. For classification
tasks, EmoLLaMA series performs better than ChatGPT and GPT-4 in the TDT task.
In the GoEmotion, the performance of EmoLLaMA is within a 5% difference
compared to ChatGPT and GPT-4. In SST5 tasks, GPT-4 performs exceptionally
well (acc:0.543, ma-F1:0.504), as we can see that ChatGPT, GPT-4, both
outperform other models in SST5 and SST tasks. The possible reason is that the
SST dataset is popular, and LLMs have been exposed to similar corpora during
pre-training, which enables them to perform better using zero-shot methods.

Comparision between EmoLLMs: Table 5 shows that all instruction-tuning LLMs
perform well on AEB-2 and have good transferability except EmoBART and EmoT5.
EmoBART and EmoT5 perform similarly to their performance on the AEB-1 dataset,
showing poor performance in regression tasks. Interestingly, EmoLLaMA-chat-7B
performs the best in most tasks of the AEB-2 and even outperforms EmoLLaMA-
chat-13B in most regression tasks. One possible reason is that models with a
larger number of parameters tend to overfit during fine-tuning, which can
subsequently affect their general performance ability.

It is worth noting that, in AEB-2 dataset, only TDT and V-Tweet are sourced
from Twitter, while the others are collected from different platforms and
domains. Although EmoLLMs’ training data is only sourced from Twitter, it
performs well on other platforms and domains, which demonstrates its excellent
transferability. The results also show that the performance of the current
sentiment analysis tools (i.e. VADER, TextBlob) is significantly inferior to
that of EmoLLMs. Overall, the experiment results on AEB-2 illustrate EmoLLMs
series achieves ChatGPT-level and GPT-4-level general capabilities (especially
EmoLLaMA) and can be used as emotion annotation tools.

####  4.3.3 (3) Analysis of Chatgpt and GPT-4

On the AEB-1 dataset, Table 4 shows that GPT-4 and GPT-4-FS perform best in
zero-shot/few-shot methods, followed by ChatGPT and ChatGPT-FS. This
illustrates the current open-sourced LLMs still have a big gap with ChatGPT
and GPT-4 in complex tasks (e.g. gaps between GPT-4 and Vicuna: EI-reg(ave):
0.375, EI-oc(ave): 0.426, V-reg: 0.406, V-oc: 0.513, E-c(macro-F1): 0.244). An
interesting phenomenon is that in most tasks, ChatGPT and GPT-4 perform better
with few-shot method than zero-shot. However, in EI-reg, ChatGPT’s zero-shot
method outperforms few-shot, while in EI-oc, GPT-4’s zero-shot performs better
than few-shot. A possible reason is that the EI-reg and EI-oc tasks are more
complex, and few-shot learning requires careful design in order to improve the
model’s performance. For different models, there may be different
understandings of few-shot examples. Therefore, for complex tasks, it is
necessary to design targeted few-shot examples specifically for different
LLMs.

On the AEB-2 dataset, GPT-4 and ChatGPT also perform better than other open-
sourced LLMs without fine-tuning in most tasks. However, compared to the
performance on the AEB-1 dataset, the performance gap between ChatGPT, GPT-4,
and other LLMs without fine-tuning becomes smaller in several tasks (e.g. gaps
between GPT-4 and Vicuna: V-A: -0.018, V-M: 0.135, and GoEmotion(acc):0.077).
One possible reason is that these tasks are simpler compared to the tasks in
AEB-1. This further demonstrates that ChatGPT and GPT-4 are more adept at
handling complex tasks compared to other open-source LLMs.

To sum up, there is still a certain gap between the current open-source LLMs
and ChatGPT, GPT-4 in affective analysis tasks. Currently, we can only surpass
ChatGPT and GPT-4 by fine-tuning on specific tasks.

##  5 Conclusion

In this paper, we propose EmoLLMs, a series of comprehensive affective
analysis models and annotation tools. We also construct a multi-task affective
analysis instruction dataset (AAID) and an affective evaluation benchmark
(AEB). We conduct a comprehensive analysis of the performance of EmoLLMs, as
well as a variety of LLMs on the AEB benchmark. The results indicate that
EmoLLMs perform exceptionally well in both affective analysis regression tasks
and classification tasks, achieving SOTA compared to the other open-sourced
LLMs, and EmoLLMs exhibit strong transferability, as it has achieved the
generalization capabilities of ChatGPT and GPT-4 in various unseen affective
analysis tasks. The results also show that there is still a certain gap
between the current open-sourced LLMs and ChatGPT, GPT-4 in specific domains.
An ideal solution to address the issue is the instruction-tuning strategy
employed in this article, which can greatly enhance the performance of LLMs in
a specific domain and surpass ChatGPT and GPT-4 in most tasks.

In the future, we will introduce more datasets from different platforms,
domains, modalities, and languages into instruction-tuning data to further
enhance the capabilities of EmoLLMs.

## Acknowledgments

The code in this project is based on BELLE code BELLEGroup (2023); Ji et al.
(2023); Wen et al. (2023). The EmoLLaMA picture in Figure 1 was generated by
PIXLR888https://pixlr.com/image-generator/. This work is supported by the
computational shared facility at the University of Manchester and the scholar
award from the Department of Computer Science at the University of Manchester.
This work is supported by the project JPNP20006 from New Energy and Industrial
Technology Development Organization (NEDO).

## References

  * Akhtar et al. [2020] Md Shad Akhtar, Asif Ekbal, and Erik Cambria.  How intense are you? predicting intensities of emotions and sentiments using stacked ensemble [application notes].  IEEE Computational Intelligence Magazine, 15(1):64–75, 2020.
  * BELLEGroup [2023] BELLEGroup.  Belle: Be everyone’s large language model engine.  https://github.com/LianjiaTech/BELLE, 2023.
  * Bello et al. [2023] Abayomi Bello, Sin-Chun Ng, and Man-Fai Leung.  A bert framework to sentiment analysis of tweets.  Sensors, 23(1):506, 2023.
  * Buechel and Hahn [2017] Sven Buechel and Udo Hahn.  Readers vs. writers vs. texts: Coping with different perspectives of text understanding in emotion annotation.  In Proceedings of the 11th linguistic annotation workshop, pages 1–12, 2017.
  * Buechel and Hahn [2022] Sven Buechel and Udo Hahn.  Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis.  arXiv preprint arXiv:2205.01996, 2022.
  * Chan et al. [2023] Jireh Yi-Le Chan, Khean Thye Bea, Steven Mun Hong Leow, Seuk Wai Phoong, and Wai Khuen Cheng.  State of the art: a review of sentiment analysis based on sequential transfer learning.  Artificial Intelligence Review, 56(1):749–780, 2023.
  * Cohen et al. [2009] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen.  Pearson correlation coefficient.  Noise reduction in speech processing, pages 1–4, 2009.
  * Consoli et al. [2022] Sergio Consoli, Luca Barbaglia, and Sebastiano Manzan.  Fine-grained, aspect-based sentiment analysis on economic and financial lexicon.  Knowledge-Based Systems, 247:108781, 2022.
  * Demszky et al. [2020] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi.  Goemotions: A dataset of fine-grained emotions.  arXiv preprint arXiv:2005.00547, 2020.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Dodge et al. [2020] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  arXiv preprint arXiv:2002.06305, 2020.
  * Dong et al. [2014] Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu.  Adaptive recursive neural network for target-dependent twitter sentiment classification.  In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers), pages 49–54, 2014.
  * Dong et al. [2022] Diwen Dong, Fuqiang Lin, Guowei Li, and Bo Liu.  Sentiment-aware fake news detection on social media with hypergraph attention networks.  In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2174–2180. IEEE, 2022.
  * Ekman [1992] Paul Ekman.  An argument for basic emotions.  Cognition & emotion, 6(3-4):169–200, 1992.
  * Govindasamy and Palanichamy [2021] Kuhaneswaran AL Govindasamy and Naveen Palanichamy.  Depression detection using machine learning techniques on twitter data.  In 2021 5th international conference on intelligent computing and control systems (ICICCS), pages 960–966. IEEE, 2021.
  * Hakak et al. [2017] Nida Manzoor Hakak, Mohsin Mohd, Mahira Kirmani, and Mudasir Mohd.  Emotion analysis: A survey.  In 2017 International Conference on Computer, Communications and Electronics (Comptelix), pages 397–402, 2017.
  * He et al. [2022] Lu He, Tingjue Yin, and Kai Zheng.  They may not work! an evaluation of eleven sentiment analysis tools on seven social media datasets.  Journal of Biomedical Informatics, 132:104142, 2022.
  * Htait and Azzopardi [2021] Amal Htait and Leif Azzopardi.  Sentiment intensity prediction using neural word embeddings.  In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, pages 93–102, 2021.
  * Hutto and Gilbert [2014a] Clayton Hutto and Eric Gilbert.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216–225, 2014.
  * Hutto and Gilbert [2014b] Clayton Hutto and Eric Gilbert.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216–225, 2014.
  * Ji et al. [2023] Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li.  Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases.  arXiv preprint arXiv:2303.14742, 2023.
  * Lei et al. [2023] Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang.  Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework.  arXiv preprint arXiv:2309.11911, 2023.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * Liao et al. [2021] Wenxiong Liao, Bi Zeng, Xiuwen Yin, and Pengfei Wei.  An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta.  Applied Intelligence, 51:3522–3533, 2021.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * Liu et al. [2023] Zhiwei Liu, Tianlin Zhang, Kailai Yang, Paul Thompson, Zeping Yu, and Sophia Ananiadou.  Emotion detection for misinformation: A review.  arXiv preprint arXiv:2311.00671, 2023.
  * Liu [2020] Bing Liu.  Sentiment analysis: Mining opinions, sentiments, and emotions.  Cambridge university press, 2020.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * Ma et al. [2020] Yukun Ma, Khanh Linh Nguyen, Frank Z Xing, and Erik Cambria.  A survey on empathetic dialogue systems.  Information Fusion, 64:50–70, 2020.
  * Mohammad and Kiritchenko [2018] Saif Mohammad and Svetlana Kiritchenko.  Understanding emotions: A dataset of tweets to study interactions between affect categories.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).
  * Mohammad et al. [2018] Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko.  Semeval-2018 task 1: Affect in tweets.  In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17, 2018.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  arXiv preprint arXiv:2306.01116, 2023.
  * Plutchik [1980] Robert Plutchik.  A general psychoevolutionary theory of emotion.  In Theories of emotion, pages 3–33. Elsevier, 1980.
  * Qureshi et al. [2020] Syed Arbaaz Qureshi, Gael Dias, Mohammed Hasanuzzaman, and Sriparna Saha.  Improving depression level estimation by concurrently learning emotion intensity.  IEEE Computational Intelligence Magazine, 15(3):47–59, 2020.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  * Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  * Thelwall et al. [2010] Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas.  Sentiment strength detection in short informal text.  Journal of the American society for information science and technology, 61(12):2544–2558, 2010.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * Wen et al. [2023] Cheng Wen, Xianghui Sun, Shuaijiang Zhao, Xiaoquan Fang, Liangyu Chen, and Wei Zou.  Chathome: Development and evaluation of a domain-specific language model for home renovation.  arXiv preprint arXiv:2307.15290, 2023.
  * Workshop et al. [2022] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * Xie et al. [2018] Hongliang Xie, Shi Feng, Daling Wang, and Yifei Zhang.  A novel attention based cnn model for emotion intensity prediction.  In Natural Language Processing and Chinese Computing: 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26–30, 2018, Proceedings, Part I 7, pages 365–377. Springer, 2018.
  * Xie et al. [2023] Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang.  Pixiu: A large language model, instruction data and evaluation benchmark for finance.  arXiv preprint arXiv:2306.05443, 2023.
  * Yang et al. [2023] Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, and Sophia Ananiadou.  Mentalllama: Interpretable mental health analysis on social media with large language models.  arXiv preprint arXiv:2309.13567, 2023.
  * Yao et al. [2023] Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al.  Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales.  arXiv preprint arXiv:2308.01320, 2023.
  * Yin et al. [2020] Da Yin, Tao Meng, and Kai-Wei Chang.  Sentibert: A transferable transformer-based architecture for compositional sentiment semantics.  arXiv preprint arXiv:2005.04114, 2020.
  * Yuan et al. [2023] Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou.  Back to the future: Towards explainable temporal reasoning with large language models.  arXiv preprint arXiv:2310.01074, 2023.
  * Zhang et al. [2020] Linrui Zhang, Hsin-Lun Huang, Yang Yu, and Dan Moldovan.  Affect in tweets: A transfer learning approach.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1511–1516, 2020.
  * Zhang et al. [2021] Xueyao Zhang, Juan Cao, Xirong Li, Qiang Sheng, Lei Zhong, and Kai Shu.  Mining dual emotion for fake news detection.  In Proceedings of the web conference 2021, pages 3465–3476, 2021.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * Zhang et al. [2023a] Boyu Zhang, Hongyang Yang, Tianyu Zhou, Muhammad Ali Babar, and Xiao-Yang Liu.  Enhancing financial sentiment analysis via retrieval augmented large language models.  In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 349–356, 2023.
  * Zhang et al. [2023b] Tianlin Zhang, Kailai Yang, Shaoxiong Ji, and Sophia Ananiadou.  Emotion fusion for mental illness detection from social media: A survey.  Information Fusion, 92:231–246, 2023.
  * Zhang et al. [2023c] Yazhou Zhang, Mengyao Wang, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin.  Dialoguellm: Context and emotion knowledge-tuned llama models for emotion recognition in conversations.  arXiv preprint arXiv:2310.11374, 2023.
  * Zheng et al. [2023] Zhonghua Zheng, Lizi Liao, Yang Deng, and Liqiang Nie.  Building emotional support chatbots in the era of llms.  arXiv preprint arXiv:2308.11584, 2023.