# When Simple Becomes Challenging: Adversarial Attacks based on Text
Simplification Corpora

# Simpler becomes Harder:
Is LLMs Behavior Consistent on Normal and Simplified Corpora?

# Simpler becomes Harder:
Do LLMs Maintain a Coherent Behavior on Simplified Corpora?

# Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?

# Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified
Corpora?

###### Abstract

Text simplification seeks to improve readability while retaining the original
content and meaning. Our study investigates whether pre-trained classifiers
also maintain such coherence by comparing their predictions on both original
and simplified inputs. We conduct experiments using 11 pre-trained models,
including BERT and OpenAI’s GPT 3.5, across six datasets spanning three
languages. Additionally, we conduct a detailed analysis of the correlation
between prediction change rates and simplification types/strengths. Our
findings reveal alarming inconsistencies across all languages and models. If
not promptly addressed, simplified inputs can be easily exploited to craft
zero-iteration model-agnostic adversarial attacks with success rates of up to
50%.

Keywords: text simplification, model robustness, model consistency

Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified
Corpora?

Miriam Anschütz, Edoardo Mosca, Georg Groh
---
Technical University of Munich
School of Computation, Information and Technology
{miriam.anschuetz<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Automatic text simplification (ATS) is a popular natural language processing
task that creates texts in plain language, preserving the original message of
the source text. Plain or simplified language is a version of the English
language with reduced text complexity and uses only well-known vocabulary.
This simplified version aims to increase accessibility and, thus, gives people
with learning impairments or reading difficulties access to information on the
internet (Martin et al., 2022). While simplifications must alter some text
features to reduce its overall complexity, they should still preserve the
original source’s content. Indeed, content coherency between the original
source and simplified output is a core element of text simplification,
spanning across various aspects (sentiment, emotion, topic, etc.) (Saggion and
Hirst, 2017). For instance, if a strong sentiment or emotion, e.g., anger
about something, is conveyed in the original text, this emotion should also be
perceivable in the simplified version as well.

In line with this thought, this paper investigates whether models also exhibit
this coherent behavior and assesses pre-trained classifiers and recent large
language models (LLM) like GPT3.5 on original and simplified texts. For this,
we exploit text simplification corpora across different languages, let the
models classify content-related features such as the addressed topic, emotion,
or sentiment, and analyze potential variations in these labels.

Figure 1: Manually created sentence pair. The simplified version simplifies
the word “conference” but preserves the meaning and neutral sentiment of the
original sentence. However, a pre-trained emotion classifier behaves
incoherent and predicts a different label for the simplified sentence.

Our results show that models change their predictions for up to 50% of the
samples, depending on the language and task used. Figure 1 shows an example of
an incoherent model behavior on a manually created sample. The simplified
version replaces “conference” with “science meeting”. Apart from that, both
versions convey the same message and sentiment. Nevertheless, a pre-trained
emotion classifier assigns different labels to the two samples. By using
human-created or human-aligned simplification corpora, we can ensure that our
benchmark samples are natural and that humans consider them meaning-preserving
and valid alterations.

Therefore, our contribution can be summarized as follows:

  * •

We compile a strict selection of human-created or human-aligned simplification
datasets as model consistency benchmarks. This ensures the naturalness and
correctness of our benchmark samples.

  * •

We test pre-trained classifiers for their prediction consistency on normal and
simplified language, covering multiple tasks across different languages to
give extensive insight into the models’ shortcomings.

  * •

Results show concerning discrepancies between the model behavior on normal and
simplified inputs. If not addressed, these discrepancies can be easily
exploited to produce zero-iteration model-agnostic adversarial attacks with a
success rate of up to 50%.

## 2\. Background and related work

Previous work by Elazar et al. (2021) defined model consistency as follows:
Given two equivalent paraphrases, a consistent model creates non-contradictory
predictions for both of them. In their study, they probed language models with
cloze-phrases and evaluated whether the predictions for the original and a
paraphrase prompt were similar. The models produced contradicting predictions
for 39%-51% of the samples, depending on the model. We extend the consistency
definition by Elazar et al. (2021) to classification tasks and deem a model
consistent if it assigns the same label to both versions.

Our work is inspired by model robustness checks with adversarial attacks. To
create an adversarial attack, a pre-trained classification model $f$, text
samples $x_{i}$, and model predictions $f(x_{i})=y_{i}$ are given. Then, the
attacker tries to find adversarial samples $x^{\prime}$ that fool the model to
change its prediction compared to the original sample, hence
$f(x_{i}^{\prime})=y_{i}^{\prime}\neq y_{i}$. The changes to the original
sample $x$ should be minimal and not change its ground truth. In addition,
humans should perceive the alteration as valid and natural (Qi et al., 2021).
If a model changes its prediction for the sample $x^{\prime}$, it is
considered sensitive to the specific adversarial attack.

Approaches to adversarial attacks can be classified based on their knowledge
of the target model. In a white-box scenario, the attacker has full access to
the model and can optimize perturbations using its output and gradients to
find adversarial samples. Black-box attacks, on the other hand, only have
access to predictions without further model information (Yoo et al., 2020).
Blind or zero-iteration attacks lack any model feedback and can only apply one
perturbation step to deceive the model. This study adopts a zero-iteration
setting to assess model sensitivity to text simplification perturbations.

Similar to our objective, Van et al. (2021) examined how NLI models change
their predictions when the samples are pre-processed by an automatic
simplification model and observed a performance drop of up to 50%. We extend
this research to further tasks and languages. Another study investigated
whether models are robust to text-style transfer attacks. Qi et al. (2021)
utilized a pre-trained style transfer model to transfer common datasets for
sentiment or topic prediction, e.g., into Twitter, bible, or poetry language
style. In most cases, at least one style adaption yielded the model to alter
its prediction. However, a human survey found that many transfers altered the
ground truth of samples, indicating that changing predictions was appropriate
behavior. To avoid unnatural paraphrases that change the samples’ ground
truth, we rely on human-supervised datasets for model coherence checks instead
of generating adversarial samples with pre-trained simplification models.

## 3\. Methodology

This paper examines whether models maintain a consistent behavior when dealing
with normal and simplified texts. For this, we investigate whether and to what
extent human-created simplification datasets can lead to output changes in
pre-trained models published to the Hugging Face model hub (Wolf et al.,
2020). We assume that although the altered text style is perceptible to
humans, it is still a natural and meaning-preserving paraphrase of the
original. Simplifications are targeted toward people with lower reading
understanding capabilities. As such, the simplifications should reduce the
complexity of the text but still preserve the topic or sentiment conveyed in
the original texts. We expect simplification corpora, used to train ATS
systems, to reflect this aspect.

In our experiments, we test various classification tasks covering topic,
emotion, fake news/toxicity, and sentiment prediction. For all of them, we
expect the original and simplified text to have the same content-related
features and, thus, get the same labels from the classifiers. The specific
tasks vary across languages, depending on the availability of pre-trained
models for the respective language. In the following, we introduce the models
under test, our selected datasets, and their pre-processing.

### 3.1. Models

Prediction task |  Model | #Classes | Domain | Benchmark
---|---|---|---|---
English |
Topic |  bert-agnews (Lee, 2023) | 4 | news | 94%*
Sentiment |  bert-base-multilingual-uncased-sentiment (NLPTown, 2020) | 5 | online | 67%*
Emotion |  emotion-english-distilroberta-base (Hartmann, 2022) | 7 | diverse | 62%*
Fake news |  roberta-fake-news-classification (Benyamina, 2022) | 2 | news | 100%*
German |
Topic |  bert-base-german-cased-gnad10 (Lai-King, 2021) | 9 | news | 68%*
Sentiment |  german-news-sentiment-bert (Lüdke et al., 2021) | 3 | news | 96%*
Toxicity |  distilbert-base-german-cased-toxic-comments (Buschmeier, 2022) | 2 | social media | 78%*
Italian |
Topic |  it5-topic-classification-tag-it (Papucci, 2022) | 10 | online | 57%
Sentiment |  feel-it-italian-sentiment (Bianchi et al., 2021) | 2 | twitter | 84%*
Emotion |  feel-it-italian-emotion (Bianchi et al., 2021) | 4 | Twitter | 73%*
Table 1: Pre-trained classifiers used to perform different classification
tasks. The classifiers vary in their number of classes and training data
domain.
* performance copied from the Hugging Face model page

Table 1 shows the pre-trained classification models we selected from the
Hugging Face model hub (Wolf et al., 2020). Not all classification tasks are
suited for our experiments as some text features in the simplified versions,
for example, the complexity, are altered on purpose. Therefore, we selected
only content-related prediction tasks like topic or sentiment classification.
Furthermore, we picked models with varying numbers of classes to experiment
with different task difficulties and tried to reflect the datasets’ domains to
avoid misclassifications due to domain mismatch. However, we could not always
find models with matching domains and, thus, preferred a mismatching domain
over skipping the task in the respective language.

As expected, the English language has the highest availability of pre-trained
models, and we found models with matching domains for all four tasks. For
German, we found in-domain classifiers to predict the sentiment and topic but
not for fake news or emotion prediction. To counteract the low availability of
suited models, we included a toxicity classifier that detects toxic comments
on social media data. Finally, for Italian, we could only find an in-domain
topic classifier and had to include an out-of-domain emotion and sentiment
classifier.

As shown in Table 1, the models show different performances on relevant
benchmark datasets. For most of the model evaluations, we used the test set
accuracies reported in the respective model pages on the hub. Only for the
Italian topic classifier, no score was reported. Therefore, we evaluated the
model on the TAG-it test set (Cimino et al., 2020) ourselves.

### 3.2. Datasets

A simplification corpus contains texts in standard language aligned with their
simplified version. These texts can be sentences or paragraphs, and the
simplifications span from replacing single words to completely rewriting the
text.

While there exists a collection of simplification corpora across many
languages (Ryan et al., 2023), we had specific requirements for our benchmark
datasets. First, we selected only datasets where humans created the texts or
alignments to avoid label changes due to misalignments. Second, a paragraph
can address different topics and have multiple sentiments, resulting in
ambiguous classifications. Hence, we only selected sentence-level datasets
where the simplifications had a large enough overlap with the original
version. In addition, we restricted ourselves to corpora with multiple levels
of simplifications to examine whether the strength of the simplification had
an impact on the prediction change rates. Moreover, the availability of pre-
trained classification models limited the language diversity. Finally, we
compiled a benchmark dataset for investigations in English, German, and
Italian.

Dataset |  Domain |  #Simplification levels | #Samples
---|---|---|---
English
Newsela EN (Xu et al., 2015) |  news | 4 |  61k
German
TextComplexityDE (Naderi et al., 2019) |  wikipedia | 2* | 249
DEplain (Stodden et al., 2023) |  online | 2 | 1.846
Italian
Simpitiki (Tonelli et al., 2016) |  wikipedia | 1 | 1.163
AdminIT (Miliani et al., 2022) |  wikipedia, government | 2* | 736
Terence/Teacher (Brunato et al., 2015) |  online, literature | 2* | 1.146
Table 2: Simplification datasets used to retrieve adversarial data with their
number of simplification levels and covered domains. Datasets with the number
of simplification levels marked with a ∗ differ from their original version.

Table 2 shows the datasets we selected for your study. For English, we used
Newsela (Xu et al., 2015), the gold standard for English simplification
(Martin et al., 2022): This dataset was created by language experts and
consists of sentences from news articles that were simplified into four
different levels, where V1 is the mildest and V4 the strongest simplification.

For German, we selected multiple datasets. The TextComplexityDE dataset by
Naderi et al. (2019) consists of sentences from Wikipedia and their simplified
versions. The simplifications were obtained from native speakers and are
annotated by their simplification strength. Initially, the dataset has three
simplification levels. However, we discarded the sample annotated with “could
not be simplified”, resulting in the simplification strengths slightly
simplified and strongly simplified. The second corpus, DEplain (Stodden et
al., 2023), is compiled from online articles with document- and sentence-level
alignments. We picked the test split of the sentence-level dataset. The
samples in the dataset were aligned manually and were annotated by their CEFR
language level (Council of Europe, 2001). The original samples are at level B2
or C2 and simplified into A1 or A2. To match the simplification levels of the
TextComplexityDE data, samples with an original language level of B2 are
considered slight simplifications, while samples with original level C2 are
strong simplifications.

We investigate three different simplification corpora for Italian. Simpitiki
(Tonelli et al., 2016) uses the edit history of Italian Wikipedia to select
edits with the annotation “simplification”. The authors manually labeled the
samples by their simplification operation and removed non-simplified versions.
The second corpus, AdminIT (Miliani et al., 2022), contains a subset of the
Simpitiki data and sentences from Italian municipality homepages. The samples
are categorized into three simplification strategies: samples with the label
OP show only a single simplification operation like sentence split or lexical
substitution, while other samples were either manually rewritten (label RS) or
manually aligned based on simplified documents (label RD). We grouped the
later categories together, yielding two levels of simplifications present in
the corpus: single simplification operation and more complex rewritings. The
final dataset consists of two subcorpora, the Terence and the Teacher corpus
(Brunato et al., 2015). Terence is created from books for children that were
simplified manually. In contrast, the Teacher corpus contains
original/simplified texts from educational homepages for teachers. Both
datasets were manually annotated by their simplification operations. We
divided the total number of annotations per text by the number of sentences
and grouped the samples into two simplification levels, one with one or fewer
simplification operations per sentence and one with multiple.

## 4\. Evaluation

(a) English

(b) German

(c) Italian

Figure 2: Prediction change rates across different languages and tasks, sorted
by the simplification strength of the samples. All classifiers show more
deviating predictions the stronger the simplification strength. Overall, the
English models are least coherent.

For each sample in the text simplification corpora, we obtained each
classifier’s prediction for the original and the simplified text and compared
their predictions. For our consistency analysis, we do not evaluate whether
any of the classifiers predicts a wrong label. We expect the classifiers to
label all samples the same, whether they are the original or simplified
versions. If the predictions deviate, we consider the respective classifier
inconsistent with these samples. We then counted the number of samples with
deviating predictions and compared the counts to the full dataset size to
obtain the prediction change rate (PCR) for each classifier. Figure 2 shows
the PCRs for all classifiers across different languages. We observe rates
around 20% on average and up to 50% for English Newsela. These change rates
are high, especially considering that the simplified samples are created
without any knowledge of the models and can thus be considered model-agnostic
adversarial samples. They are used for all classifiers simultaneously. Other
works that limited the number of model-specific changes to the sample achieved
only prediction change rates below 10% (Yoo et al., 2020, Fig. 2).

A common trend across all languages is that the model is more likely to change
its prediction the stronger the simplification is—i.e., the more
simplification operations are performed. Comparing the topic classifiers among
all languages (green curves in Figure 2), the English classifier has four
classes with an accuracy of 94% and is sensitive to simplification in more
than 30% of the time, while the Italian topic classifier has ten classes with
an accuracy of only 57% that only shows deviating predictions for 10% of the
samples. This indicates the number of labels and the classifier’s performance
seems to have little impact on the prediction change rate. Similarly, the
Italian models have a strong domain mismatch (Wikipedia data vs. Twitter-
trained models), while the English data and most of the models are from the
news domain. Nevertheless, the English models are more easily affected by
simplified inputs. Overall, the human-created or human-aligned samples in the
different simplification corpora evoke an alarming amount of prediction
changes.

In the following sections, we discuss further experiments to investigate
factors influencing classifiers behavior (sections 4.1 \- 4.4) as well as
examine whether s.o.t.a. LLMs like OpenAI’s GPT models (OpenAI, 2023) are also
sensitive to simplifications (section 4.5).

### 4.1. Edit distances

As stated before, the prediction change rates increase with a higher level of
simplification. An obvious assumption would be that this is due to increasing
differences between the original and simplified samples, especially since
higher-level simplifications sometimes remove parts of the original
information. To verify this assumption, we calculated the Levenshtein
distances, normalized by the original sample’s lengths, between the original
and simplified versions using the Python
Levenshtein111https://rapidfuzz.github.io/Levenshtein/levenshtein.html#distance
package. In Figure 3, these ratios are correlated with the number of
classifiers that changed their prediction for the respective sample. Some
samples have a normalized distance larger than $1$, e.g., when an explanation
is added in the simplified version. Among all languages, the German samples
have the highest ratios. For all languages, the samples with no prediction
change have the smallest normalized distances. However, the average distance
for samples with two or more classifiers with prediction changes stays the
same or even decreases. Overall, we observe a Spearman correlation of $0.34$
for English, $0.35$ for German, and $0.34$ for Italian between the normalized
Levenshtein distance and the number of classifiers with changing
predictions222p-values are $0.0$ (en), $8.5\mathrm{e}{-65}$ (de), and
$1.4\mathrm{e}{-85}$ (it); calculated with SciPy:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html.

Figure 3: Number of classifiers with changing predictions per sample and their
Levenshtein distances between the original and simplified sentences. The
distances were normalized by the sample’s lengths.

Therefore, only a weak correlation exists between the normalized distance and
coherency of the models. This indicates that the simplification operation and
the choice of vocabulary to simplify the samples are more relevant to the
classifiers than the pure number of edit operations or parts removed from the
original sentence.

### 4.2. Reducing task complexities

Figure 4: Predictions change rates for tasks with reduced number of classes.
The reduced tasks have a better performance but are still susceptible to
simplifications.

Fine-grained emotion and sentiment prediction tasks can be difficult and may
induce some ambiguity for the models. The English emotion and sentiment
classifiers have seven and five classes, respectively. To control for these
potential ambiguities, we reduced the number of classes and, thus, the task
complexities. For the emotion task, we summarized all negative emotions, like
anger, disgust, fear, sadness, and surprise, into a negative class. The only
positive class, joy, and the neutral class were kept, resulting in a three-
class classification task with which we can detect emotion flips. For the
sentiment task, we calculated the top-2 accuracy. That is, a prediction was
only considered as a deviating prediction if the difference was at least two
steps (e.g., strongly negative vs. negative is no prediction change, but
strongly negative to neutral is). Figure 4 shows how the classifier
predictions change for these reduced tasks compared to the original tasks. The
prediction change rate drops by five percentage points for the emotion tasks
and more than 20 percentage points for the sentiment task. The rate is still
higher with increased simplification strength. We conclude that the difficulty
and ambiguity of the classification task can influence the prediction change
rate. Nevertheless, even with the reduced tasks, the classifiers still perform
inconsistently with simplified versions.

### 4.3. Simplification operations

We further investigated which simplification operations especially tempt the
classifiers to change their predictions. For this, we utilized the Italian
Simplitiki corpus (Tonelli et al., 2016). The samples in this corpus are
annotated by the operation performed to obtain the simplified version. These
operations can be on the word level, such as deletion or replacement of single
words, or on the phrase level, for example, splitting a sentence into two or
transforming the verbal voice. Samples with word-level changes are closer to
their original than phrase-level ones. Therefore, we expected the word-level
operations to result in lower change rates. However, Figure 5 shows that they
are on par or lead to even more prediction changes than the phrase-level
simplifications. Word substitution is a combination of word deletion and
insertion and, as such, has the highest PCR of all word-level perturbations.
As such, replacing a word with its synonym has been used as word-level
adversarial attacks before (Chiang and Lee, 2023). On the phrase level,
merging two sentences does not affect the sentiment and topic classifiers.
Yet, the topic classifier is sensitive to the information order, exhibiting
the highest PCR of almost 20

Figure 5: Prediction change rate of different simplification operations as
annotated in the Italian Simpitiki corpus (Tonelli et al., 2016).

### 4.4. Masking named entities

Named entities (NE) can strongly impact the sentiment or topic of a phrase as,
sometimes, they only occur in a particular context. In simplified language,
NEs are sometimes generalized or removed. To test which influence named
entities have on the models’ prediction consistency, we compared the
predictions when masking the named entities.

We utilized Spacy (Honnibal et al., 2020) to detect named entities in our
original and simplified phrases. We searched for tokens with a tag in this
list: [’EVENT’, ’GPE’, ’LANGUAGE’, ’LAW’, ’LOC’, ’NORP’, ’ORG’, ’PERSON’,
’PRODUCT’, ’WORK_OF_ART’]. If such a token was found, we masked it by
replacing it with the placeholder “NAME”. With this, the NE in the aligned
pairs are the same and do not impact the classification outcome.

Figure 6: Comparison of prediction change rates of masked (solid) and unmasked
(dashed) named entities on the English Newsela corpus. It depends on the task
if NE masking increases or decreases the PCR.

In Figure 6, we compare the performances of the classifiers on the named-
entity-masked and unmasked data. For the fake news and emotion classification
tasks, masking the named entities reduces the prediction change rates. In
contrast, for the topic prediction task, the masking even increases the change
rates, probably because the sentences become more ambiguous. Therefore, it
depends on the classifier and its task whether the possibly changing NEs in
the simplifications impact its consistency.

### 4.5. ChatGPT

OpenAI’s GPT models (OpenAI, 2023) are among the NLP models displaying the
strongest capabilities in terms of generalization and performance. Hence, we
also examined whether these models are sensitive to our simplifications. We
used the OpenAI API to query the gpt-3.5-turbo model and predicted samples
from the Newsela dataset in a one-shot manner. We prompted each sample
individually to avoid biases due to previously seen non-simplified versions.
We probed the same tasks with the same labels as the English models from the
Hugging Face model hub described above. In addition, we asked the model to
return a dictionary with all predictions simultaneously and set the
temperature parameter to zero. The full prompt can be found in Figure 7. Due
to the long processing time of an API request and financial limitations, we
restricted ourselves to English and only classified the first 1000 samples per
simplification level.

⬇

{"role": "system", "content": "You are an assistant designed to label news
texts. Users will paste in a string of text and you will respond with labels
you’ve extracted from the text as a JSON object. The topic must be one of
world, sports, buisiness or sci/tech. The sentiment is on a scale from 1 to 5
stars. Fake news can be true or false. Emotion can be one of anger, disgust,
fear, joy, neutral, sadness or surprise"},

{"role": "system", "name": "example_user", "content": "Predict the sentence:
Even a big first-day jump in shares of Google (GOOG) couldn’t quiet debate
over whether the Internet search engine’s contentious auction was a hit or a
flop."},

{"role": "system", "name": "example_assistant", "content": "{

’topic’: ’sci/tech’,

’sentiment’: ’2 stars’,

’fake_news’: False,

’emotion’: ’sadness’}"},

{"role":"user", "content": f"Predict the sentence: {sentence}"}

Figure 7: OpenAI system description for our one-shot classification approach.

Figure 8 shows the prediction change rates on this subset. We compare
different simplification levels and tasks for the ChatGPT model and our
Hugging Face classifiers. The classification change rates for the fake news
detection task decrease significantly compared to the model by Benyamina
(2022). The PCRs for the other models decrease slightly, and the emotion
classification is almost on par with the Hugging Face classifier in the strong
simplifications. As we have seen before with the task-specific models, the
changes increase with stronger simplification levels. Therefore, even ChatGPT
is not robust to text simplification and makes incoherent predictions.

Figure 8: Comparison of Hugging Face classifiers with OpenAI’s GPT 3.5.
Especially for the emotion and the sentiment task, GPT is only slightly more
robust than the smaller, task-specific models.

## 5\. Discussion

This paper shows that even s.o.t.a models like GPT3.5 are sensitive towards
text simplification as a special form of text style transfer and produce
incoherent predictions for texts and their simplified versions. We observe
this behavior on human-generated or human-aligned text simplification
datasets. That means that our samples were validated by humans before and,
thus, should be natural, grammatically correct, and meaning-preserving.
However, previous work by Devaraj et al. (2022) has shown that even human-
curated simplification datasets can contain factual errors. While pre-
processing and filtering those datasets seems promising for text
simplification training (Ma et al., 2022), we worked with the original dataset
versions. Therefore, it is possible that some of the labels change due to
factual errors in the datasets. Unfortunately, such investigations only exist
for English corpora. Assuming that the corpora in other languages are of
higher factuality, this could explain why the English classifiers showed the
highest prediction change rates even though English is usually the best-
resourced language with top performance.

We tried to select well-documented classifiers with high accuracy rates.
Still, the classifiers show varying performance on relevant benchmark datasets
and, thus, potentially produce misclassifications. While our experiments show
that the language with the strongest domain mismatches and weakest
classifiers, Italian, has the lowest prediction change rates, we can not
guarantee that the models do not misclassify some of the samples.
Nevertheless, for our investigation, the actual label is not important as we
are only interested in whether the labels for the original and simplified
versions change. Even if the initial classification is wrong, consistent
models should still produce the same label for all simplifications.

We observe alarmingly high prediction change rates across all languages and
even OpenAI’s GPT-3.5 model. This suggests that the pre-training data of these
models lack samples in simplified language and, thus, that there is still only
little information available in plain language. Increasing internet
accessibility and, hence, increasing the amount of data in simplified language
is the most promising approach to improving plain language understanding in
language models. If such incoherence is not addressed, our findings
empirically show that simplification can easily be exploited as a zero-
iteration model-agnostic adversarial attack. Attackers would only need to
simplify any input text to achieve success rates of up to 50% with little
effort.

## 6\. Conclusion

In this paper, we have investigated how coherent models perform on simplified
texts. We have shown that different classifiers across multiple languages
struggle with plain language samples and exhibit incoherent behavior. Such
incoherency seems to affect also s.o.t.a LLMs like OpenAI’s GPT3.5, which are
not robust to simplifications from our benchmark datasets. We exploited human-
created or human-aligned text simplification corpora to ensure natural and
meaning-preserving samples. In this setting, we have observed prediction
change rates of more than 40%, indicating a severe lack of plain language
understanding in pre-trained language models.

In future studies, we aim to expand our experiments to include more languages
and settings involving automatically generated simplifications. Additionally,
we believe that improving model coherence on simplified inputs can be achieved
through human-preference tuning techniques—such as RLHF and DPO—and we
encourage researchers to explore this direction further.

## Ethical considerations

We investigated whether model predictions change between original and
simplified versions of the same text. Our findings can be valuable for
identifying models’ shortcomings and improving their robustness. However, our
results can potentially be misused as adversarial attacks, and as such, they
can threaten applications based on large language models. However, in our
approach, we only re-use existing corpora and do not craft a new adversarial
threat.

Especially for people with reading difficulties, the availability of
information in plain language is crucial. Our experiments demonstrate that
pre-trained language models are sensitive to plain language and that
simplified samples are still underrepresented in pre-training corpora. This
can cause severe problems when these people use LLM applications such as
ChatGPT. Hence, we ask content creators and data scientists to remember the
need for plain language and provide resources accordingly.

## Data/Code availability statement

Our work utilizes existing text simplification corpora. These corpora are
publicly available except for the English Newsela corpus (Xu et al., 2015). In
addition, we publish our experiment code and links to these corpora at
https://github.com/MiriUll/LLM-consistency-simplification.

## 7\. Bibliographical References

##

  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Are synonym substitution attacks really synonym substitution attacks? In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1853–1878, Toronto, Canada. Association for Computational Linguistics.
  * Cimino et al. (2020) Andrea Cimino, Felice Dell’Orletta, and Malvina Nissim. 2020.  Tag-it@ evalita 2020: Overview of the topic, age, and gender prediction task for italian.  _Evaluation Campaign of Natural Language Processing and Speech Tools for Italian_.
  * Council of Europe (2001) Council for Cultural Co-operation. Education Committee. Modern Languages Division Council of Europe. 2001.  _Common European framework of reference for languages: Learning, teaching, assessment_.  Cambridge University Press.
  * Devaraj et al. (2022) Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022.  Evaluating factuality in text simplification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
  * Dyrmishi et al. (2023) Salijona Dyrmishi, Salah Ghamizi, and Maxime Cordy. 2023.  How do humans perceive adversarial text? a reality check on the validity and naturalness of word-based adversarial attacks.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8822–8836, Toronto, Canada. Association for Computational Linguistics.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Lin et al. (2021) Jieyu Lin, Jiajie Zou, and Nai Ding. 2021.  Using adversarial attacks to reveal the statistical bias in machine reading comprehension models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 333–342, Online. Association for Computational Linguistics.
  * Ma et al. (2022) Yuan Ma, Sandaru Seneviratne, and Elena Daskalaki. 2022.  Improving text simplification with factuality error detection.  In _Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)_ , pages 173–178, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
  * Martin et al. (2022) Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022.  MUSS: Multilingual unsupervised sentence simplification by mining paraphrases.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 1651–1664, Marseille, France. European Language Resources Association.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Qi et al. (2021) Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021\.  Mind the style of text! adversarial and backdoor attacks based on text style transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4569–4580, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ryan et al. (2023) Michael Ryan, Tarek Naous, and Wei Xu. 2023.  Revisiting non-English text simplification: A unified multilingual benchmark.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4898–4927, Toronto, Canada. Association for Computational Linguistics.
  * Saggion and Hirst (2017) Horacio Saggion and Graeme Hirst. 2017.  _Automatic text simplification_ , volume 32.  Springer.
  * Van et al. (2021) Hoang Van, Zheng Tang, and Mihai Surdeanu. 2021.  How may I help you? using neural text simplification to improve downstream NLP tasks.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4074–4080, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Verma et al. (2023) Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, and Adam Poliak. 2023.  Evaluating paraphrastic robustness in textual entailment models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 880–892, Toronto, Canada. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yoo et al. (2020) Jin Yong Yoo, John Morris, Eli Lifland, and Yanjun Qi. 2020.  Searching for a search method: Benchmarking search algorithms for generating NLP adversarial examples.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 323–332, Online. Association for Computational Linguistics.

## 8\. Language Resource References

##

  * Benyamina (2022) Benyamina, Hamza. 2022.  _RoBERTa Fake news classification_.  PID https://huggingface.co/hamzab/roberta-fake-news-classification.
  * Bianchi et al. (2021) Federico Bianchi, Debora Nozza, and Dirk Hovy. 2021.  "FEEL-IT: Emotion and Sentiment Classification for the Italian Language".  In _Proceedings of the 11th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_. Association for Computational Linguistics.
  * Brunato et al. (2015) Dominique Brunato, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2015.  Design and annotation of the first Italian corpus for text simplification.  In _Proceedings of the 9th Linguistic Annotation Workshop_ , pages 31–41, Denver, Colorado, USA. Association for Computational Linguistics.
  * Buschmeier (2022) Konstantin Buschmeier. 2022.  _German Toxic Comment Classification_.  PID https://huggingface.co/ml6team/distilbert-base-german-cased-toxic-comments.
  * Hartmann (2022) Hartmann, Jochen. 2022.  _Emotion English DistilRoBERTa-base_.  PID https://huggingface.co/j-hartmann/emotion-english-distilroberta-base/.
  * Lai-King (2021) Mathieu Lai-King. 2021.  _German BERT for News Classification_.  PID https://huggingface.co/Mathking/bert-base-german-cased-gnad10.
  * Lee (2023) Lee, Jiaqi. 2023.  _BERT AGnews_.  PID https://huggingface.co/JiaqiLee/bert-agnews.
  * Lüdke et al. (2021) Simon Lüdke and Josephine Grau and Martin Drawitsch. 2021.  _German sentiment BERT finetuned on news data_.  PID https://huggingface.co/mdraw/german-news-sentiment-bert.
  * Miliani et al. (2022) Martina Miliani, Serena Auriemma, Fernando Alva-Manchego, and Alessandro Lenci. 2022\.  Neural readability pairwise ranking for sentences in Italian administrative language.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 849–866, Online only. Association for Computational Linguistics.
  * Naderi et al. (2019) Babak Naderi, Salar Mohtaj, Kaspar Ensikat, and Sebastian Möller. 2019.  Subjective assessment of text complexity: A dataset for german language.
  * NLPTown (2020) NLPTown. 2020.  _bert-base-multilingual-uncased-sentiment_.  PID https://doi.org/10.57967/hf/1515.
  * Papucci (2022) Michele Papucci. 2022.  _it5-topic-classification-tag-it_.  PID https://huggingface.co/mpapucci/it5-topic-classification-tag-it.
  * Stodden et al. (2023) Regina Stodden, Omar Momen, and Laura Kallmeyer. 2023.  DEplain: A German parallel corpus with intralingual translations into plain language for sentence and document simplification.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16441–16463, Toronto, Canada. Association for Computational Linguistics.
  * Tonelli et al. (2016) Sara Tonelli, Alessio Palmero Aprosio, and Francesca Saltori. 2016.  Simpitiki: a simplification corpus for italian.  In _CLiC-it/EVALITA_ , pages 4333–4338.
  * Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015.  Problems in Current Text Simplification Research: New Data Can Help.  _Transactions of the Association for Computational Linguistics_ , 3:283–297.


# Eliminating Sentiment Bias for Aspect-Level Sentiment Classification with
Unsupervised Opinion Extraction

Bo Wang1, Tao Shen2, Guodong Long2, Tianyi Zhou3,4, Yi Chang1,5,✉
1School of Artificial Intelligence, Jilin University
2Australian AI Institute, School of CS, FEIT, University of Technology Sydney
3University of Washington, Seattle; 4University of Maryland, College Park
5International Center of Future Science, Jilin University
bowang19<EMAIL_ADDRESS>
tianyizh<EMAIL_ADDRESS>
###### Abstract

Aspect-level sentiment classification (ALSC) aims at identifying the sentiment
polarity of a specified aspect in a sentence. ALSC is a practical setting in
aspect-based sentiment analysis due to no opinion term labeling needed, but it
fails to interpret why a sentiment polarity is derived for the aspect. To
address this problem, recent works fine-tune pre-trained Transformer encoders
for ALSC to extract an aspect-centric dependency tree that can locate the
opinion words. However, the induced opinion words only provide an intuitive
cue far below human-level interpretability. Besides, the pre-trained encoder
tends to internalize an aspect’s intrinsic sentiment, causing sentiment bias
and thus affecting model performance. In this paper, we propose a span-based
anti-bias aspect representation learning framework. It first eliminates the
sentiment bias in the aspect embedding by adversarial learning against
aspects’ prior sentiment. Then, it aligns the distilled opinion candidates
with the aspect by span-based dependency modeling to highlight the
interpretable opinion terms. Our method achieves new state-of-the-art
performance on five benchmarks, with the capability of unsupervised opinion
extraction.

##  1 Introduction

Aspect-based sentiment analysis (ABSA) (Jiang et al., 2011) aims to determine
sentiment polarity w.r.t. a specified aspect term in a piece of text. For
example, in “The food is tasty, but the service is terrible”, the sentiment
towards aspect term (AT) “food” is positive according to the opinion term (OT)
“tasty”, while the sentiment towards “service” is negative according to
“terrible”. The most comprehensive setting of ABSA is aspect sentiment triplet
extraction (ASTE) (Peng et al., 2020) consisting of a series of subtasks,
i.e., aspect extraction, aspect-level sentiment classification and opinion
extraction. Thereby, given a piece of text, ASTE can produce a set of triples,
i.e., (Aspect Term, Sentiment, Opinion Term), to describe sentiment with
details of What, How and Why, so it enjoys full interpretability. Continuing
the above example, ASTE can generate triples like (food, Positive, tasty).
However, the human annotation on opinion terms is much more label-intensive
than traditional sentiment analysis task.

Therefore, by following many recent works, we target the practical subtask of
ASTE, called aspect-level sentiment classification (ALSC). It predicts a
three-categorical sentiment (i.e., positive, neutral or negative) of a given
aspect term in a sentence. Most recent works capture the modification relation
between aspect and opinion terms in an implicit manner, which is usually
achieved by integrating graph neural networks (GNNs) over dependency parsing
tree into text representation learning (Zhang et al., 2019a; Sun et al.,
2019b; Tang et al., 2020; Wang et al., 2020). Further, the performance can be
significantly boosted when incorporating pre-trained Transformer encoder,
e.g., BERT (Devlin et al., 2019) in a fine-tuning paradigm (Sun et al., 2019a;
Tang et al., 2020; Wang et al., 2020; Chen et al., 2020). Although these works
achieve excellent results even close to humans, they cannot derive the
interpretability to explain why an aspect is associated with the polarity
prediction.

Luckily, pre-trained Transformer encoder can also be used to explain
linguistic knowledge underlying the given text via dependency probing (Clark
et al., 2019; Wu et al., 2020). This has been exploited by Dai et al. (2021)
to reveal that, after fine-tuning the pre-trained encoder on ALSC, an aspect-
centric dependency tree can be induced to highlight the modifier of an aspect.
Intuitively, the highlighted modifier is viewed as an opinion word of the
corresponding aspect, which thus, to some extent, brings the interpretability
back.

Nevertheless, compared with the span-formatted opinion terms in ASTE, the
opinion word illustrated by the induced dependency tree can only provide an
intuitive, noisy, word-level sentiment cue but is far from the human-level
interpretability as in opinion extraction. What’s worse, as verified by Huang
et al. (2020), the Transformer pre-trained on large-scale raw corpora tends to
internalize terms’ intrinsic attributes, so it causes the problem of sentiment
bias when generating text given a particular term prompt. In ALSC scenario, we
found that sentiment bias also exists and affects a model to determine the
sentiment of an aspect term regardless of its contextual information (e.g.,
opinion terms). The bias is especially obvious for the aspects that can imply
strong sentiment themselves. For example, for “There’s candlelight and music”,
a model based on pre-trained Transformer is likely to mis-classify the
sentiment towards “music” as positive, whereas the oracle label is neutral.

In this work, based on the pre-trained Transformer encoder to ensure state-of-
the-art ALSC performance, we aim to eliminate sentiment bias in the ALSC
scenario while equip the model with human-level aspect-opinion
interpretability.

To this end, we propose a Span-based Anti-bias aspect Representation Learning
(SARL) framework for ALSC with unsupervised opinion extraction. First, instead
of widely feeding a concatenation of sentence and aspect into a pre-trained
Transformer, we adopt a span-level paradigm (Hu et al., 2019; Zhao et al.,
2020) which focuses on deriving span representation of an aspect term. Then,
we propose an anti-bias aspect encoding module to eliminate the sentiment bias
existing in aspect representations, which is achieved by an adversarial
learning against the aspect’s prior sentiment indicated in SentiWordNet (Esuli
and Sebastiani, 2006). Next, built upon sentiment-agnostic aspect
representation from the above encoder, we propose an aspect-opinion dependency
alignment module to capture explicit modifications from opinion term
candidates to the targeted aspect, and integrate the modifications into aspect
representation via gating. The integrated representation is lastly passed into
a neural classifier for sentiment prediction.

For pairwise aspect-opinion alignment via a span-based model, our work share a
high-level inspiration with SpanMlt (Zhao et al., 2020) but differs in that,
SpanMlt targets fully-supervised aspect and opinion terms extraction whereas
ours leverages the alignment to empower aspect-opinion interpretability even
without opinion supervisions.

Our main contributions are as follows:

  * •

We propose a span-based aspect encoding module to alleviate the sentiment bias
problem from which pre-trained Transformers suffer.

  * •

By a dependency aligner, our model can derive human-level aspect-opinion
interpretability w/o opinion extraction supervisions.

  * •

We achieve new state-of-the-art results on 5 ALSC datasets with extensive
analyses and present metrics to measure the interpretability.

##  2 Methodology

This section begins with a task definition of ALSC. Then, we present our Span-
based Anti-bias aspect Representation Learning (SARL) framework (as in Figure
1) consisting of an adversarial anti-bias aspect encoder (§2.1) and a
distilled aspect-opinion dependency aligner (§2.2). Lastly, we detail the
training and inference of the proposed model (§2.3), including unsupervised
opinion extraction.

Figure 1: An overview of proposed Span-based Anti-bias aspect Representation
Learning (SARL) framework.

#### Task Definition.

Given a sentence with $n$ words, ${\bm{x}}=[x_{1},\dots,x_{n}]$, ALSC aims to
predict a three-categorical sentiment from $\\{Positive$, $Neutral$,
$Negative\\}$ of an aspect term, $a$, where $a$ is a span of ${\bm{x}}$, from
$s^{a}$ and to $e^{a}$, i.e., ${\bm{a}}=x_{s^{a}:e^{a}}$.

###  2.1 Adversarial Anti-bias Aspect Encoder

As suggested by Devlin et al. (2019) and verified by many works, a common
practice to tackle pair inputs is feeding the pre-trained Transformer with a
concatenation of them. Hence, many ALSC works (Jiang et al., 2020; Hou et al.,
2021) feed a concatenation of the sentence ${\bm{x}}$ and an aspect ${\bm{a}}$
into the encoder. Nonetheless, considering that every aspect is a word span of
the sentence, it is natural to employ a span-level paradigm for aspect
representation, which has been proven as effective as the concatenation models
(Zhou et al., 2019; Zhao et al., 2020). Also, a span-level model is more
efficient as it predicts the sentiment for all aspect terms in a single feed-
forward process.

#### Span-level Aspect Representation.

We thereby adopt the span-level paradigm to generate aspect representation.
Formally, a Transformer is fed with a sentence without additional aspect
terms,

$\displaystyle{\bm{H}}=\operatorname*{Transformer-
Enc}({\bm{x}};\theta^{(ptm)})$ (1)

where, ${\bm{H}}=[{\bm{h}}_{1},\dots,{\bm{h}}_{n}]\in\mathbb{R}^{d\times n}$
denotes contextualized representations corresponding to all the words. Then,
given span $[s^{a},e^{a}]$ of each aspect term ${\bm{a}}$, we derive its
representation by

$\displaystyle{\bm{c}}^{a}\\!=\\![{\bm{h}}_{s}^{a};\\!{\bm{h}}_{e}^{a};\\!\operatorname*{Attn-
Pool}({\bm{H}}_{s^{a}:e^{a}};\\!\theta^{(aa)})],$ (2)

where $[;]$ is vector concatenation, $\operatorname*{Attn-Pool}(\cdot)$ is
attention pooling to generate sequence-level embedding with multi-layer
perceptron (MLP)-derived weights (Lin et al., 2017; Liu et al., 2016). And
${\bm{c}}^{a}\in\mathbb{R}^{3d}$ is the resulting span-level aspect
representation. In the remainder, we omit the aspect indicator, $a$, if no
confusion caused. But, as briefed in §1, sentiment bias will occur since the
Transformer encoders pre-trained on large-scale text corpora incline to
internalize the sentiment polarity of aspect terms. Hence, directly applying a
classifier to the aspect representation ${\bm{c}}$ for its sentiment
prediction will be affected.

#### Adversarial Anti-Bias Module.

A promising way is leveraging adversarial learning to eliminate the sentiment
bias of a particular aspect. Open questions still remain about how to define
the sentiment bias as discriminator’s objective and how to fool the
discriminator for our anti-bias purpose. As an answer to the first answer, we
resort to external sentiment knowledge, SentiWordNet (Esuli and Sebastiani,
2006), for its prior three-categorical sentiment for common lexicons and
phrases. Given an aspect ${\bm{a}}$ from a sentence, we can easily obtain its
prior sentiment polarity $y^{(pr)}$ by querying SentiWordNet. The prior
sentiment polarity is the intrinsic attribute of an aspect, which thus can be
viewed as its sentiment bias. Hence, we train a sentiment bias discriminator
towards $y^{(pr)}$ to exploit bias information underlying the aspect
representation. Formally, we present an MLP-based neural classifier upon
${\bm{c}}$ as the discriminator, i.e.,

$\displaystyle{\bm{p}}^{(pr)}=\mathrm{softmax}(\operatorname*{MLP}({\bm{c}}^{a};\theta^{(dis)}))\in\mathbb{R}^{3}.$
(3)

Next, training loss of this discriminator is

$\displaystyle{\mathcal{L}}^{(dis)}_{\theta^{(dis)}}=-\sum\nolimits_{\mathcal{D}}\sum\nolimits_{{\mathcal{A}}}\log{\bm{p}}^{(pr)}_{[\hat{y}=y^{(pr)}]},$
(4)

where ${\mathcal{D}}$ denotes ALSC dataset with sentence-level samples, where
${\mathcal{A}}$ denotes all aspects in a sentence, and
${\bm{p}}^{(pr)}_{[\hat{y}=y^{(pr)}]}$ denotes fetching the probability value
corresponding to prior sentiment $y^{(pr)}$.

On the other side, to eliminate sentiment bias, the span-level aspect encoder
aims to fool the discriminator towards generating neutral-sentiment
representation. The adversarial loss is written as

$\displaystyle{\mathcal{L}}^{(adv)}_{\theta^{(ptm)},\theta^{(aa)}}=-\sum\nolimits_{\mathcal{D}}\sum\nolimits_{{\mathcal{A}}}\log{\bm{p}}^{(pr)}_{[\hat{y}=\text{Neutral}]}.$
(5)

With the adversarial learning between ${\mathcal{L}}^{(dis)}$ and
${\mathcal{L}}^{(adv)}$, the aspect representation ${\bm{c}}$ can escape from
sentiment bias and become sentiment-agnostic.

###  2.2 Aspect-Opinion Dependency Aligner

As opinion term is also a span of words, it is intuitive to use a span-based
alignment model (Lee et al., 2017) for explicit modification from opinion
terms to the aspect. This model is first proposed for co-reference resolution
to align an entity mention with its antecedent, but usually requires full
supervisions. For example, it has been adapted to extract pair-wise aspect and
opinion terms (Zhao et al., 2020) in ABSA, and the supervisions include both
opinion terms and aspect-opinion associations but none of them is available in
ALSC.

Fortunately, built upon the sentiment-agnostic aspect representation derived
from the above encoder, the span-based alignment model is likely to
automatically learn the dependency between opinion and aspect spans, which is
guided by the aspect’s sentiment label. To this end, we present an aspect-
opinion dependency aligner w/o supervisions.

#### Opinion Term Candidates.

Similar with Lee et al. (2017) and Zhao et al. (2020), we first enumerate all
the possible spans as candidates of opinion terms. Given the sentence
${\bm{x}}=[x_{1},\dots,x_{n}]$, an opinion term candidate ${\bm{o}}$ is a span
of words from $s^{o}$ and to $e^{o}$, i.e., ${\bm{o}}=x_{s^{o}:e^{o}}$. A set
of all the opinion term candidates can be written as

$\displaystyle{\mathcal{O}}=\\{x_{s^{o}:e^{o}}|$ $\displaystyle 1\leq
s^{o}\leq e^{o}\leq n\wedge e^{o}-s^{o}\leq l$
$\displaystyle\wedge(e^{o}<s^{a}\vee s^{o}>e^{a})\\},$ (6)

where the last condition means no overlap between an arbitrary oracle aspect
term and its opinion candidates, and $m\coloneqq|{\mathcal{O}}|$. Then, we can
easily derive the representation of each candidate ${\bm{o}}\in{\mathcal{O}}$
given the contextualized embedding ${\bm{H}}$ by the encoder in Eq.(1). We
adopt the same scheme to generate span-level opinion embedding as in Eq.(2),
i.e.,

$\displaystyle{\bm{u}}^{o}\\!=\\![{\bm{h}}_{s^{o}};\\!{\bm{h}}_{e^{o}};\\!\operatorname*{Attn-
Pool}({\bm{H}}_{s^{o}:e^{o}};\\!\theta^{(ao)})],$ (7)

where the resulting vector ${\bm{u}}^{o}\in\mathbb{R}^{3d}$ represents the
candidate ${\bm{o}}$. Thus, we can get a set of candidate representations,
${\bm{U}}=\\{{\bm{u}}^{o}\\}_{\forall
o\in{\mathcal{O}}}\in\mathbb{R}^{3d\times m}$.

Following previous models (Lee et al., 2017; Zhao et al., 2020), two kinds of
scores are generated for the dependency between opinions and aspects: (1) a
sentiment mention score determines if a span candidate is an opinion mention;
and (2) an aspect-opinion alignment score estimates the modification relation
in an aspect-opinion pair.

#### Distilled Sentiment Mention Scoring.

Most previous span-based alignment models employ an one-dim-out neural module
(e.g., MLP with $\operatorname*{Sigmoid}$) to determine the confidence of a
mention but require full supervision for accurate predictions. Considering
such supervision is unavailable in ALSC, we consequently weaken the
supervision from “whether a mention is a gold opinion term” to “whether a
mention expresses a sentiment polarity”. The weak supervision can be readily
obtained from a well-trained phrase-level sentiment classification model via
distilling model (Hinton et al., 2015). Hence, we first employ an MLP-based
classifier built upon an opinion term candidate ${\bm{u}}^{o}$ to derive a
three-categorical sentiment distribution,

$\displaystyle{\bm{p}}^{(ms)}_{o}$
$\displaystyle=\mathrm{softmax}(\operatorname*{MLP}({\bm{u}}^{o};\theta^{(ms)}))\\!\in\\!\mathbb{R}^{3};$
(8)

and a phrase-level sentiment classification model is

$\displaystyle\bar{\bm{p}}^{(pl)}_{o}=\operatorname*{Senti-
Model}({\bm{o}};\theta^{(pl)}),$ (9)

which is also based on a pre-trained Transformer and trained on a popular
phrase-level sentiment analysis dataset, Stanford Sentiment Treebank (Socher
et al., 2013). Then, we can define a soft loss of sentiment distillation based
on Kullback–Leibler (KL) divergence, i.e.,

$\displaystyle{\mathcal{L}}^{(kl)}=\operatorname*{KL}(\bar{\bm{p}}^{(pl)}_{o}||{\bm{p}}^{(ms)}_{o}).$
(10)

Last, we use the non-neutral probability as the confidence of a mention
expressing sentiment polarity, i.e., the sentiment mention score,

$\displaystyle r^{o}$ $\displaystyle\coloneqq
1-{\bm{p}}^{(ms)}_{o}{[\hat{y}=\text{Neutral}]},~{}\forall{\bm{o}}\in{\mathcal{O}},$
(11) $\displaystyle{\bm{r}}$
$\displaystyle=\\{r^{o}\\}_{{\bm{o}}\in{\mathcal{O}}}\in\mathbb{R}^{m}.$ (12)

#### Aspect-Opinion Dependency Modeling.

Besides the mention score, we use an MLP to estimate the alignment between the
aspect and opinion candidates. First, we obtain a relationship representation
by an interactive concatenation (Reimers and Gurevych, 2019) of their span-
level embeddings,

$\displaystyle{\bm{q}}^{o}=[{\bm{c}};{\bm{u}}^{o};{\bm{c}}\odot{\bm{u}}^{o};{\bm{z}}^{o}],~{}\forall{\bm{o}}\in{\mathcal{O}},$
(13)

where “$\odot$” is Hadamard product and ${\bm{z}}$ is a learnable relative-
position embedding indicating their distance over the syntactic dependency
parsing tree of the sentence. Again, please note we omitted the superscript
$a$ in the equation for simplification. Next, ${\bm{q}}^{o}$ is passed into an
MLP-based scorer to calculate pairwise alignment score, i.e.,

$\displaystyle\tilde{f}^{o}$
$\displaystyle=\operatorname*{MLP}({\bm{q}}^{o};\theta^{(as)})\in\mathbb{R},~{}\forall{\bm{o}}\in{\mathcal{O}},$
(14) $\displaystyle\tilde{\bm{f}}$
$\displaystyle=\\{f^{o}\\}_{{\bm{o}}\in{\mathcal{O}}}\in\mathbb{R}^{m}.$ (15)

Then, we apply $\mathrm{softmax}$ to $\tilde{\bm{f}}$ for normalized alignment
scores, which are subsequently weighted by the corresponding sentiment mention
scores ${\bm{r}}$ in Eq.(12) to derive the dependency scores between the
aspect and all opinion candidates, i.e.,

$\displaystyle{\bm{f}}=\mathrm{softmax}(\tilde{\bm{f}})\odot{\bm{r}}.$ (16)

#### “Dummy” Opinion Term.

However, a special scenario has been often ignored when capturing the aspect-
opinion dependency or alignment. That is, an aspect term may not correspond to
any opinion (statistically, $>65\%$ of neutral-labeled aspects in benchmarks
w/o opinion). To remedy this, we adopt a concept of “dummy span” that
indicates no opinion term in the sentence towards an aspect. We can imply
aspect-“dummy” dependency score by considering the neutral-sentiment
probability of the aligned opinion term candidates, i.e.,

$\displaystyle f^{(d)}$
$\displaystyle=\delta\sum(\mathrm{softmax}(\tilde{\bm{f}})\odot(\bm{1}-{\bm{r}})),$
(17)

where $\delta$ denotes a re-scaling hyperparameter. So we rewrite the
normalized dependency scores as

$\displaystyle{\bm{f}}\leftarrow[{\bm{f}};f^{(d)}]\in\mathbb{R}^{m+1}.$ (18)

It is also essential to integrate the aligned opinion term candidates
(including the dummy opinion) into the sentiment-agnostic aspect
representation ${\bm{c}}$. Specifically, we first obtain opinion
representation by an attention-like operation, i.e.,

$\displaystyle{\bm{u}}=[{\bm{U}},{\bm{c}}]\cdot{\bm{f}},$ (19)

where “$\cdot$” denotes matrix multiplication and, as following Lee et al.
(2017), the aspect representation itself corresponds to the dummy opinion,
i.e., $[{\bm{U}},{\bm{c}}]\in\mathbb{R}^{d\times(m+1)}$. Lastly, we integrate
the opinion representation to the aspect one by gating,

$\displaystyle{\bm{g}}$
$\displaystyle=\operatorname*{Sigmoid}(\operatorname*{MLP}([{\bm{c}};{\bm{u}}];\theta^{(gm)}))\in\mathbb{R}^{3d},$
(20) $\displaystyle{\bm{v}}$
$\displaystyle={\bm{g}}\cdot{\bm{c}}+(\bm{1}-{\bm{g}})\cdot{\bm{u}}.$ (21)

As a result, ${\bm{v}}$ stands for opinion-enrich aspect representation and is
ready for sentiment classification.

Dataset | Positive | Neutral | Negative
---|---|---|---
Train | Test | Train | Test | Train | Test
Laptop14 | 994 | 341 | 464 | 169 | 870 | 128
Rest14 | 2164 | 728 | 637 | 196 | 807 | 196
Rest15 | 912 | 326 | 36 | 34 | 256 | 182
Rest16 | 1240 | 469 | 69 | 30 | 439 | 117
Twitter | 1561 | 173 | 3127 | 346 | 1560 | 173
Table 1: Summary statistics of five benchmark datasets.

###  2.3 Model Training and Inference

#### Aspect-level Sentiment Classification.

On top of ${\bm{v}}$, we define a neural classifier for the final three-
categorical sentiment prediction as

$\displaystyle{\bm{p}}^{(sc)}=\mathrm{softmax}(\operatorname*{MLP}({\bm{v}};\theta^{(sc)}))\in\mathbb{R}^{3}.$
(22)

And the training loss of ALSC task is written as

$\displaystyle{\mathcal{L}}^{(sc)}=-\sum\nolimits_{\mathcal{D}}\sum\nolimits_{{\mathcal{A}}}\log{\bm{p}}^{(sc)}_{[\hat{y}=y^{(sc)}]},$
(23)

where $y^{(sc)}$ denotes the oracle label of an asepct.

#### Training and Inference.

Besides the discriminator loss ${\mathcal{L}}^{(dis)}$ in Eq.(4), we train the
learnable parameters in our proposed SARL model towards a linear combination
of the other three losses, i.e.,

$\displaystyle{\mathcal{L}}^{(alsc)}={\mathcal{L}}^{(sc)}+\beta{\mathcal{L}}^{(adv)}+\gamma{\mathcal{L}}^{(kl)}.$
(24)

We also set a hyper-parameter $\alpha$ to control the proportion of
discriminator learning (i.e., ${\mathcal{L}}^{(dis)}$) against the ALSC model
learning (i.e., ${\mathcal{L}}^{(alsc)}$). The inference procedure can be
simply written as

$\displaystyle y^{*}=\arg\max{\bm{p}}^{(sc)}.$ (25)

#### Unsupervised Opinion Extraction.

A well-trained SARL model is equipped with the capability to extract opinion
term(s) for an aspect, based on its intermediate variable, i.e.,

$\displaystyle{\bm{o}}^{*}=\arg\max\nolimits_{\\{o,\text{dummy}\\}}{\bm{f}},$
(26)

where ${\bm{f}}\in\mathbb{R}^{m+1}$ from Eq.(16) denotes aspect-opinion
dependency scores, including the last dim for dummy opinion, i.e., an aspect
w/o any opinion in the sentence. Thereby, ${\bm{o}}^{*}$ is the extracted
opinion term for a specified aspect term in the sentence, and it is worth
mentioning again that the opinion extraction is learned in an unsupervised
manner.

##  3 Experiment

Dataset | #AT | #OT | #$\theta$ | Dataset | #AT | #OT | #$a^{neu/o}$
---|---|---|---|---|---|---|---
Laptop14 | 444 | 527 | 112 | Rest14 | 832 | 974 | 137
Table 2: Summary statistics of unsupervised opinion extraction test set. #$a^{neu/o}$ is the number of neutral-sentiment aspects without corresponding opinion terms.  LR ($\times 10^{-5}$) | size | Laptop14 | Rest14 | Rest15 | Rest16 | Twitter
---|---|---|---|---|---|---
lr-$\theta^{(ptm)}$ | base | 1.3 | 2 | 2 | 2 | 2
large | 1 | 1 | 1 | 1 | 2
lr-others | base | 20 | 1 | 20 | 20 | 20
large | 20 | 1 | 1 | 20 | 1
Table 3: Settings of learning rate. “others” includes $\theta^{(aa)}$, $\theta^{(dis)}$, $\theta^{(ao)}$, $\theta^{(ms)}$, $\theta^{(as)}$, $\theta^{(gm)}$ and $\theta^{(sc)}$. Embedding | Method | Laptop14 | Rest14 | Rest15 | Rest16 | Twitter
---|---|---|---|---|---|---
Accu | Ma-F1 | Accu | Ma-F1 | Accu | Ma-F1 | Accu | Ma-F1 | Accu | Ma-F1
Static | ASGCN | 75.55 | 71.01 | 80.86 | 72.19 | 79.89 | 61.89 | 88.99 | 67.48 | 72.15 | 70.40
Embedding | CDT | 77.19 | 72.99 | 82.30 | 74.02 | - | - | 85.58 | 69.93 | 74.66 | 73.66
BiGCN | 74.59 | 71.84 | 81.97 | 73.48 | 81.16 | 64.79 | 88.96 | 70.84 | 74.16 | 73.35
BERT${}_{\text{base}}$ | DGEDT | 79.80 | 75.60 | 86.30 | 80.00 | 84.00 | 71.00 | 91.90 | 79.00 | 77.90 | 75.40
RGAT | 78.21 | 74.07 | 86.60 | 81.35 | - | - | - | - | 76.15 | 74.88
kumaGCN | 81.98 | 78.81 | 86.43 | 80.30 | 80.69 | 65.99 | 89.39 | 73.19 | 77.89 | 77.03
RoBERTa${}_{\text{base}}$ |  ${\dagger}$ASGCN-FT-RoBERTa | 83.33 | 80.32 | 86.87 | 80.59 | - | - | - | - | 76.10 | 75.07
${\dagger}$PWCN-FT-RoBERTa | 84.01 | 81.08 | 87.35 | 80.85 | - | - | - | - | 77.02 | 75.52
${\dagger}$RGAT-FT-RoBERTa | 83.33 | 79.95 | 87.52 | 81.29 | - | - | - | - | 75.81 | 74.91
${\dagger}$MLP | 83.78 | 80.73 | 87.37 | 80.96 | - | - | - | - | 77.17 | 76.20
SARL (ours) | 85.42 | 82.97 | 88.21 | 82.44 | 88.19 | 73.83 | 94.62 | 81.92 | 78.03 | 76.97
RoBERTa${}_{\text{large}}$ |  SARL (ours) | 85.74 | 82.97 | 90.45 | 85.34 | 91.88 | 78.88 | 95.76 | 84.29 | 78.32 | 77.32
Table 4: ALSC results on the five datasets. ${\dagger}$Numbers are from Dai et
al. (2021), and others are from the original papers, i.e., ASGCN (Zhang et
al., 2019a), CDT (Sun et al., 2019b), BiGCN (Zhang and Qian, 2020), DGEDT
(Tang et al., 2020), RGAT (Wang et al., 2020), kumaGCN (Chen et al., 2020),
PWCN-FT-RoBERTa (Zhang et al., 2019b).

#### Datasets.

For ALSC task, we evaluate our model on five datasets111The source code and
datasets are available at https://github.com/wangbo9719/SARL_ABSA, whose
statistics are listed in Table 1, including (i) Laptop14 (SemEval-2014T4)
(Pontiki et al., 2014) with laptop reviews, (ii) Rest14 (SemEval-2014T4),
Rest15 (SemEval-2015T12) and Rest16 (SemEval-2016T5) (Pontiki et al., 2014,
2015, 2016) with restaurant reviews, and (iii) Twitter (Mitchell et al., 2013)
with tweets. Following most competitors including the models compared in Table
4, we do not split training set.

To evaluate unsupervised opinion extraction, we employ the test set from Xu et
al. (2020) where annotations of opinion terms222The sentences in this dataset
cannot completely match the standard ALSC dataset so we only use the overlap
part. are from Fan et al. (2019). The statistics are listed in Table 2.

Super- | Method | Laptop14 | Rest14
---|---|---|---
vision | @1 | @3 | @1 | @3
EM | P | R | F | EM | P | R | F | EM | P | R | F | EM | P | R | F
✓ | SpanMlt | - | - | - | 80.6 | - | - | - | - | - | - | - | 84.0 | - | - | - | -
Peng et al. (2020) | - | 81.8 | 84.8 | 83.2 | - | - | - | - | - | 76.9 | 75.3 | 76.0 | - | - | - | -
✗ | Induced Tree | 8.5 | 29.4 | 20.6 | 22.6 | - | - | - | - | 10.6 | 30.2 | 21.8 | 23.5 | - | - | - | -
SARL (ours) | 42.3 | 62.2 | 68.7 | 62.8 | 60.5 | 76.3 | 81.4 | 77.1 | 49.4 | 65.9 | 67.1 | 64.8 | 76.0 | 86.0 | 88.2 | 86.1
w/o distillation | 16.3 | 37.5 | 32.1 | 31.8 | 27.5 | 52.8 | 50.7 | 47.8 | 19.0 | 43.9 | 45.3 | 41.1 | 34.8 | 65.3 | 68.1 | 62.8
w/o sentiment score | 15.0 | 32.7 | 29.7 | 28.6 | 27.7 | 54.2 | 51.9 | 49.4 | 30.9 | 52.5 | 50.5 | 48.9 | 52.1 | 73.7 | 74.1 | 71.2
Table 5: Unsupervised opinion extraction results (%) on Laptop14/Rest14. The
resulting numbers of SpanMlt (Zhao et al., 2020) and Peng et al. (2020) are
from their original papers which the metrics under @1 are the standard
metrics. The results of induced tree (Dai et al., 2021) are calculated by
regarding the aspects’ sub-nodes in the tree as its extracted opinions.

#### Training Setups.

We use a mini-batch Stochastic Gradient Descent (SGD) to minimize the loss
functions, with Adam optimizer, $10\%$ warm-up, and a linear decay of the
learning rate. To initialize the Transformer, we alternate between pre-trained
RoBERTa${}_{\text{base}}$ and ${}_{\text{large}}$. We set batch size $=16$ and
max sequence length $=64$ based on experience, and conduct grid searches for
the other hyperparameters. Then, we set the max width of candidate span
$l=15$, the number of training epochs $=7$ for Twitter and $10$ for other
datasets, the training proportion $\alpha=1/3$ for Laptop14, $0.2$ for Rest14
and Rest15, $0.1$ for Rest16, the loss weight in Eq.(24) $\beta=0.05$ for
Laptop14 and Rest16 and $0.1$ for the remains, $\gamma=1$, and $\delta=1/m$ in
Eq.(17). The learning rates are listed in Table 3.

###  3.1 Overall Performance

ALSC results of competitive approaches and our SARL on the five benchmarks are
shown in Table 4. Following prior works, we adopt accuracy (Accu) and macro-F1
(Ma-F1) to evaluate the performance, and the results of SARL are the best
values from ten runs. It is observed that our proposed SARL achieves state-of-
the-art performance on all these datasets. Compared to static embedding-based
methods, the methods with pre-trained Transformer gain better results. In the
same embedding genre RoBERTa${}_{\text{base}}$, SARL outperforms others by an
average of 1% on accuracy and 1.3% on macro-F1. Furthermore, the
RoBERTa${}_{\text{large}}$-based SARL derives more significant progress.

###  3.2 Sentiment Bias Elimination

Figure 2: Sentiment bias analysis on five datasets. The “Baseline” in right denotes the span-based Transformer baseline. Suppose an aspects set ${\mathcal{S}}=\\{a|(y^{(pr)}\neq\textit{Neutral})\wedge(y^{(pr)}\neq y^{(sc)})\\}$ denotes all potential aspects who will be misclassified due to the sentiment bias, the vertical axis in right represents the proportion of ${|{\mathcal{Q}}|}/{|{\mathcal{S}}|}$, where ${\mathcal{Q}}=\\{a|a\in{\mathcal{S}}\wedge y^{*}=y^{(pr)}\neq y^{(sc)}\\}$. And the number on the top of bin is $|{\mathcal{Q}}|$.  Figure 3: ALSC results of SARL, and “SARL with neutral reinforce”, i.e., w/ ${\mathcal{L}}^{(adv)}_{\theta^{(dis)},\theta^{(aa)},\theta^{(ptm)}}$ while w/o ${\mathcal{L}}^{(dis)}$.  Laptop14 | Rest14
---|---
Hits@1 | Hits@3 | Hits@1 | Hits@3
30.36 | 56.25 | 45.99 | 61.31
Table 6: The Hits@N results about dummy opinions.

#### Bias Statistics.

As in Figure 2 (left), the sentiment bias is common since aspects with bias
polarity account for nearly 50% in most datasets.

#### The Success of Adversarial Learning.

To explore the effectiveness of our adversarial learning, we first train a
model without adversarial, “w/o adv” for short (please refer to below ablation
study for more). Then, based on this trained model, we add a sentiment bias
detector that has the same architecture as the discriminator to detect the
bias existing in aspects representations. Here, we use the ratio of neutral
predictions to measure the accuracy of detector since the smaller ratio means
more bias detected. Compared to 98.75% by full SARL model, the detector’s
ratio 69.64% is much smaller, which proves both necessity and success of
adversarial learning. Further, as in Figure 3, we train a model w/o adv but
with neutral reinforce and there are obvious decreases compared with the full
SARL.

Method | Laptop14 | Rest14
---|---|---
Accu | Ma-F1 | Accu | Ma-F1
Full model | 85.74 | 82.97 | 90.45 | 85.34
w/o adv | 83.70 | 80.55 | 89.46 | 82.16
w/o AT-OT aligner | 84.64 | 81.75 | 89.28 | 83.51
w/o all | 83.86 | 80.53 | 89.02 | 82.59
Table 7: Ablation results of ALSC. The full model denotes our proposed SARL.
“w/o adv” removes anti-bias module in§2.1. “w/o AT-OT aligner” removes the
proposed module in §2.2. “w/o all” degrades our SARL model to Transformer-
based span-level ALSC (i.e., classification based on Eq.(2)).

#### Results of Elimination.

Comparing to a span-level baseline that simply feeds the aspect representation
${\bm{c}}$ in Eq.(2) into an MLP to get the sentiment predictions, ours
alleviates the problem as shown in Figure 2 (right). A possible reason is that
the aspects in Laptop14 (e.g., screen and size) merely have a little sentiment
in themselves so that the bias labels from SentiWordNet are noisier.

Index | Example | “w/o all” | SARL | Top-3 candidates
---|---|---|---|---
1 | After dinner I heard [music]neu playing and | pos | neu | dummy; playing;
discovered that there is a lounge downstairs. | playing and discovered
2 | Desserts include [flan]neu and sopaipillas. | pos | neu | dummy; and; include
3 | How is this [place]neg still open? | pos | neg | open; still open; dummy
4 | This place has beautiful [sushi]pos, | pos | pos | delicious; delicious CHEAP;
and it’s delicious CHEAP. | beautiful
5 | The wait [staff]pos was loud and inconsiderate. | neg | neg | loud; was loud;
loud and inconsiderate
6 | The sauce is excellent (very fresh) with | pos | pos | excellent; excellent (very fresh);
$\text{[dabs of real mozzarella]}_{neu}$. | fresh
7 | 15% gratuity automatically added to the [bill]neg. | neu | neu | dummy; automatically; 15
8 | The [quality of the meat]neg was on par | pos | neu | dummy; par; was on par
with your local grocery store.
Table 8: Case study for sentiment bias (row 1-3), unsupervised opinion
extraction (row 4-5) and error analysis (6-8).

###  3.3 Unsupervised Opinion Extraction

#### Explicit Opinion Extraction.

To measure the model’s ability of opinions extraction w/o supervision, we
present the following novel metrics including the top-N-based Exact Match
(EM@N), Precision (P@N), Recall (R@N) and F1-Score (F@N). Specifically, EM@N
denotes the gold opinion term appears in the top-N opinion term candidates.
And the Precision@N, Recall@N and F1@N are also employed to describe the
maximum char-level overlap. As shown in Table 5, SARL achieves promising
performance on unsupervised opinion extraction. Without any opinion annotation
data, the top-3 based metrics achieve similar or even better performance
compared with the models under opinion supervision. Further, comparing with
Induced Tree, SARL achieves far better performance.

#### Dummy Opinion Extraction.

To measure the dummy opinion ranking about neutral-sentiment aspects without
opinion terms, we apply Hits@N that stands for the ratio of such aspects that
the dummy opinion is ranked in top-N. As shown in Table 6, the performance is
adequate to support the purpose of introducing dummy opinions.

###  3.4 Ablation Study

To explore each module’s contribution, we conduct an extensive ablation study.
For ALSC, (1) The results of “model w/o adv” are slightly higher than the
“model w/o all”, which indicates that the AT-OT aligner is helpful for
classification due to our opinion terms integration; and (2) The “model w/o
aligner” achieves sub-optimal results and completely loses the ability of
opinion extraction. In addition, the component dropping also severely affects
the performance of opinion extraction as in Table 5. The extraction
performance decrease of SARL “w/o distillation” is larger than “w/o sentiment
score”. A potential reason is that the sentiment scorer performs poorly
without any supervision and thus generates incorrect sentiment scores.

###  3.5 Case Study

#### Does SARL eliminate sentiment bias?

As the top-3 rows in Table 8, those aspects with intrinsic positive bias are
mis-classified by our span-level baseline but correctly classified by SARL,
which verifies the effectiveness of SARL.

#### How does SARL obtain interpretability from unsupervised opinion
extraction?

As listed in Table 8, SARL can exactly extract the opinion terms of the
targeted aspect so provide promising interpretability for sentiment prediction
under the unsupervised opinion setting. In addition, for the neutral aspect
without explicit opinion terms in a sentence, the dummy opinion always ranks
first like the $2^{nd}$ row in Table 8, which explains the reason for neutral
prediction.

#### Error Analysis.

To analyze the limitation of ALSC models including ours, we investigate all
the examples mis-classified by SARL on Rest14, and summarize two main
problems: (1) The major (up to 66%) problem is neutral-related mis-classifying
because a neutral aspect term is affected by the polarity words associated
with other aspects (e.g., $6^{rd}$ row). The other problem ($23\%$) refers to
that it is infeasible to determine aspects’ sentiment without commonsense
knowledge (e.g., $7^{\text{th}}$ row) or additional information (e.g.,
$8^{\text{th}}$ row). In summary, how to accurately classify the aspect
without explicit opinion is still an open problem.

##  4 Related Work

#### Aspect-Level Sentiment Classification.

ALSC relies heavily on modification relations between aspect term and opinion
term in a sentence, so recent progresses mainly fall into modeling the
relations by applying graph neutral networks to dependency parsing tree (Sun
et al., 2019b; Zhang et al., 2019a; Wang et al., 2020). Despite their
effectiveness, they lack interpretability to the sentiment prediction. For
further boosted, many methods (Sun et al., 2019a; Hu et al., 2019; Chen et
al., 2020; Mao et al., 2021; Dai et al., 2021) introduce the pre-trained
Transformer, which also brings a little interpretability due to the highlight
modification between aspect and opinion terms derived from potential syntax
knowledge (Wu et al., 2020; Dai et al., 2021). However, the derived
interpretability is far from human-level. Furthermore, the Transformer tends
to internalize terms’ intrinsic sentiment bias, which is harmful to ALSC.

#### Aspect-Opinion Alignment.

Fan et al. (2019) first defined the aspect-oriented opinion extraction task in
which the aspect terms are given in advance. Later, Zhao et al. (2020)
proposed the aspect-opinion co-extraction task. More recently, Peng et al.
(2020) proposed the aspect sentiment triplet extraction and further
explorations have conducted by Xu et al. (2020) and Mao et al. (2021). These
models extract the opinion terms for aspects and thus provide explicit
interpretability. However, they require manually labeled opinions data for
training, which is much more expensive than three-categorical labeling in
ALSC.

##  5 Conclusion

In this work, we propose SARL framework for ALSC. Specifically, we first
present an adversarial anti-bias aspect encoder to eliminate sentiment bias in
aspects and then propose an aspect-opinion dependency aligner to
unsupervisedly extract opinions. The experiments on 5 benchmarks can greatly
support our motivations and empirical results show state-of-the-art
performance with interpretability.

## Acknowledgement

This work is supported by the National Natural Science Foundation of China
(No.61976102, No.U19A2065).

## References

  * Chen et al. (2020) Chenhua Chen, Zhiyang Teng, and Yue Zhang. 2020.  Inducing target-specific latent structures for aspect sentiment classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 5596–5607. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of bert’s attention.  _CoRR_ , abs/1906.04341.
  * Dai et al. (2021) Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, and Xipeng Qiu. 2021.  Does syntax matter? A strong baseline for aspect-based sentiment analysis with roberta.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 1816–1829. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Esuli and Sebastiani (2006) Andrea Esuli and Fabrizio Sebastiani. 2006.  SENTIWORDNET: A publicly available lexical resource for opinion mining.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy, May 22-28, 2006_ , pages 417–422. European Language Resources Association (ELRA).
  * Fan et al. (2019) Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019.  Target-oriented opinion words extraction with target-fused neural sequence labeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2509–2518. Association for Computational Linguistics.
  * Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  _CoRR_ , abs/1503.02531.
  * Hou et al. (2021) Xiaochen Hou, Peng Qi, Guangtao Wang, Rex Ying, Jing Huang, Xiaodong He, and Bowen Zhou. 2021.  Graph ensemble learning over multiple dependency trees for aspect-level sentiment classification.  _CoRR_ , abs/2103.11794.
  * Hu et al. (2019) Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019.  Open-domain targeted sentiment analysis via span-based extraction and classification.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 537–546. Association for Computational Linguistics.
  * Huang et al. (2020) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020.  Reducing sentiment bias in language models via counterfactual evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_ , pages 65–83. Association for Computational Linguistics.
  * Jiang et al. (2020) Bin Jiang, Jing Hou, Wanyue Zhou, Chao Yang, Shihan Wang, and Liang Pang. 2020.  Metnet: A mutual enhanced transformation network for aspect-based sentiment analysis.  In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 162–172. International Committee on Computational Linguistics.
  * Jiang et al. (2011) Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011.  Target-dependent twitter sentiment classification.  In _The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA_ , pages 151–160. The Association for Computer Linguistics.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pages 188–197. Association for Computational Linguistics.
  * Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017.  A structured self-attentive sentence embedding.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Liu et al. (2016) Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016.  Learning natural language inference using bidirectional LSTM model and inner-attention.  _CoRR_ , abs/1605.09090.
  * Mao et al. (2021) Yue Mao, Yi Shen, Chao Yu, and Longjun Cai. 2021.  A joint training dual-mrc framework for aspect based sentiment analysis.  _CoRR_ , abs/2101.00816.
  * Mitchell et al. (2013) Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013\.  Open domain targeted sentiment.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1643–1654. ACL.
  * Peng et al. (2020) Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020.  Knowing what, how and why: A near complete solution for aspect-based sentiment analysis.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8600–8607. AAAI Press.
  * Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia V. Loukachevitch, Evgeniy V. Kotelnikov, Núria Bel, Salud María Jiménez Zafra, and Gülsen Eryigit. 2016.  Semeval-2016 task 5: Aspect based sentiment analysis.  In _Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_ , pages 19–30. The Association for Computer Linguistics.
  * Pontiki et al. (2015) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015.  Semeval-2015 task 12: Aspect based sentiment analysis.  In _Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015_ , pages 486–495. The Association for Computer Linguistics.
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  Semeval-2014 task 4: Aspect based sentiment analysis.  In _Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014_ , pages 27–35. The Association for Computer Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3980–3990. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1631–1642. ACL.
  * Sun et al. (2019a) Chi Sun, Luyao Huang, and Xipeng Qiu. 2019a.  Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 380–385. Association for Computational Linguistics.
  * Sun et al. (2019b) Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2019b.  Aspect-level sentiment analysis via convolution over dependency tree.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 5678–5687. Association for Computational Linguistics.
  * Tang et al. (2020) Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. 2020.  Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 6578–6588. Association for Computational Linguistics.
  * Wang et al. (2020) Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020.  Relational graph attention network for aspect-based sentiment analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 3229–3238. Association for Computational Linguistics.
  * Wu et al. (2020) Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020.  Perturbed masking: Parameter-free probing for analyzing and interpreting BERT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4166–4176. Association for Computational Linguistics.
  * Xu et al. (2020) Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020.  Position-aware tagging for aspect sentiment triplet extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 2339–2349. Association for Computational Linguistics.
  * Zhang et al. (2019a) Chen Zhang, Qiuchi Li, and Dawei Song. 2019a.  Aspect-based sentiment classification with aspect-specific graph convolutional networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 4567–4577. Association for Computational Linguistics.
  * Zhang et al. (2019b) Chen Zhang, Qiuchi Li, and Dawei Song. 2019b.  Syntax-aware aspect-level sentiment classification with proximity-weighted convolution network.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , pages 1145–1148. ACM.
  * Zhang and Qian (2020) Mi Zhang and Tieyun Qian. 2020.  Convolution over hierarchical syntactic and lexical graphs for aspect level sentiment analysis.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 3540–3549. Association for Computational Linguistics.
  * Zhao et al. (2020) He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020.  Spanmlt: A span-based multi-task learning framework for pair-wise aspect and opinion terms extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 3239–3248. Association for Computational Linguistics.
  * Zhou et al. (2019) Yan Zhou, Longtao Huang, Tao Guo, Jizhong Han, and Songlin Hu. 2019.  A span-based joint model for opinion target extraction and target sentiment classification.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 5485–5491. ijcai.org.