# How Conservative are Language Models?
Adapting to the Introduction of Gender-Neutral Pronouns

First Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Stephanie Brandl, Ruixiang Cui, Anders Søgaard
University of Copenhagen, Denmark
{brandl, rc<EMAIL_ADDRESS>
###### Abstract

Gender-neutral pronouns have recently been introduced in many languages to a)
include non-binary people and b) as a generic singular. Recent results from
psycholinguistics suggest that gender-neutral pronouns (in Swedish) are not
associated with human processing difficulties. This, we show, is in sharp
contrast with automated processing. We show that gender-neutral pronouns in
Danish, English, and Swedish are associated with higher perplexity, more
dispersed attention patterns, and worse downstream performance. We argue that
such conservativity in language models may limit widespread adoption of
gender-neutral pronouns and must therefore be resolved.

How Conservative are Language Models?
Adapting to the Introduction of Gender-Neutral Pronouns

Stephanie Brandl, Ruixiang Cui, Anders Søgaard University of Copenhagen,
Denmark {brandl, rc<EMAIL_ADDRESS>

##  1 Introduction

Many linguistic scholars have observed how technology in general has altered
the course of language evolution Kristiansen et al. (2011); Abbasi (2020),
e.g., through the influence of social media conventions. Language
technologies, in particular, have also been argued to have such effects, e.g.,
by reducing the pressure to acquire multiple languages.

Gender-neutral pronouns is not an entirely modern concept. In 1912, Ella Flag
Young, then superintendent of the Chicago public-school system, said the
following to a room full of school principals: "The English language is in
need of a personal pronoun of the third person, singular number, that will
indicate both sexes and will thus eliminate our present awkwardness of
speech." The use of gender-neutral pronouns has become much more popular in
recent years Gustafsson Sendén et al. (2021). In 2013, a gender-neutral
pronoun was politically introduced in Swedish Gustafsson Sendén et al. (2015)
which can be used for both, people identifying outside the gender dichotomy
and as a generic pronoun where information about gender is either unavailable
or irrelevant.

In a recently recorded eye-tracking study, Vergoossen et al. (2020a) found no
evidence that native speakers of Swedish find it harder to process gender-
neutral pronouns than gendered pronouns, an argument often brought up by
opponents of gender-inclusive language Speyer and Schleef (2019); Vergoossen
et al. (2020b). In combination with their increasing popularity, this suggests
gender-neutral pronouns have been or will be widely and fully adapted over
time Gustafsson Sendén et al. (2015, 2021). However, since language technology
has the potential to alter the course of language evolution, we want to make
sure that our NLP models do not become a bottleneck for this positive
development.

### Contribution

We extract stimuli from a Swedish eye-tracking study that has shown no
increase in processing cost in humans for the gender-neutral pronoun hen
compared to gendered pronouns. We translate those stimuli into English and
Danish and compare model perplexity across gendered and gender-neutral
pronouns for all three languages. Furthermore, we systematically investigate
performance differences across pronouns in downstream tasks, namely natural
language inference (NLI) and coreference resolution. Across the board, we find
that NLP models, unlike humans, are challenged by gender-neutral pronouns,
incurring significantly higher losses when gendered pronouns are replaced with
their gender-neutral alternatives. We argue this is a problem the NLP
community must take seriously.111Our code is available at
github.com/stephaniebrandl/gender-neutral-pronouns

##  2 Model perplexity and attention

In this section we introduce a Swedish eye-tracking study and explain how we
adapt this study to investigate gender-neutral pronouns in language models.

| en | da | sv
---|---|---|---
| she/he | they | xe | hun/han | de | høn | hon/han | hen
perplexity | $1$ | $1.49$ | $2.37$ | $1$ | $1.21$ | $3.35$ | $1$ | $1.8$
correlation | 0.12 | 0.26 | 0.32 | -0.14 | 0.03 | -0.1 | 0.19 | 0.09
0.28 | 0.33 | 0.49 | 0.13 | 0.17 | 0.21 | 0.65 | 0.72
0.28 | 0.33 | 0.49 | 0.13 | 0.17 | 0.22 | 0.65 | 0.72
Table 1: Perplexity scores across pronouns and languages for the eye-tracking
stimuli. Correlation between attention flow and perplexity are listed row-wise
for layers 1, 6 and 12.

### Humans and _hen_

Vergoossen et al. (2020a) recently recorded a Swedish eye-tracking study to
test the hypothesis whether the gender neutral pronoun hen has a higher
processing cost during pronoun resolution than gendered pronouns. Participants
were reading sentence pairs where the first sentence contained a noun
referring to a person and the second sentence contained a pronoun referring to
that person either with a gendered pronoun or hen, for example:

_70-åringen dammsög golvet i vardagsrummet. Han/Hen skulle få besök på
kvällen._

_The 70-year-old vacuumed the living room floor. He/They would have visitors
in the evening._

It has recently been shown that attention flow, in contrast to attention
itself, correlates with human fixation patterns in task-specific reading
Eberle et al. (2022). We applied a similar analysis pipeline here and
extracted all 384 sentence pairs and fed them into the uncased Swedish BERT
model.222huggingface.co/af-ai-center/bert-base-swedish-uncased We calculate
perplexity values for each sentence pair over word probabilities as given by
BERT with the formula proposed by Wang et al. (2019). Furthermore, we
calculate attention flow Abnar and Zuidema (2020) propagated from layers 1, 6
and 12 and extract attention flow values assigned to the pronoun with respect
to the entity. Attention flow considers the attention matrices as a graph,
where tokens are represented as nodes and attention scores as edges between
consecutive layers. The edge values, i.e., attention scores, define the
maximal flow possible between a pair of nodes.

We consider different parameters of human fixation which we assume might be
influenced by a change in pronouns, in particular during pronoun resolution,
i.e., first and total fixation time on the pronoun and fixation time after the
first fixation on the noun. For both attention flow and perplexity, however we
could not find any meaningful correlation to those parameters. One reason for
that might be that the dataset only contains fixations for the two entities,
i.e., pronoun and noun, which makes data comparably sparse and impossible to
extract complete reading patterns.

### Language models and gender-neutral pronouns

We therefore focus on the model-based data alone in order to understand how
well language models can deal with gender-neutral pronouns. For this, we
consider perplexity values on sentence-level and calculate rank-based Spearman
correlation between perplexity and attention flow for the aforementioned
layers. Perplexity has been treated as an indicator for model surprisal and
language model quality Goodkind and Bicknell (2018) thus we argue that it
serves as a reasonable indicator for processing difficulty.

With this analysis, we can see if a) gender-neutral pronouns cause a higher
sentence perplexity, i.e., a higher _surprisal_ and if b) a possible higher
surprisal is connected to higher attention flow values on the pronoun with
respect to the entity.

We furthermore translate the sentence pairs into English and Danish where we
use two sets of gender-neutral pronouns: 3rd person plural (hence: they/de)
which are used in both languages as gender-neutral pronouns Miltersen (2020)
and _neopronouns_ (xe for English Hekanaho (2020) and høn for
Danish).333information.dk/kultur/hen-hoen For the translation, we use the
Google Translate API for Python and manually correct sentences such that
semantics agree with the original sentences in Swedish. We apply the same
experiments to those translated datasets with uncased Danish
BERT444huggingface.co/Maltehb/danish-bert-botxo and uncased English BERT555
huggingface.co/bert-base-uncased.

### Results

We show results on perplexity and correlations in Table 1 for Danish, English
and Swedish. Perplexity values for the datasets with gendered pronouns are set
to $1$ and we show relative increase for gender-neutral pronouns within a
language since perplexity values have been shown to not be comparable across
languages Mielke et al. (2019); Roh et al. (2020). There we can see that
perplexity scores for sentences with gender-neutral pronouns are significantly
higher (Wilcoxon signed-rank test resulted in p-values $<0.01$ for all pair-
wise comparisons).

For the correlation between perplexity and attention flow on the Swedish
sentence pairs, we can see a clear development between the first layer where
there is no correlation ($p>0.05$) for gender-neutral hen and very low
correlation for gendered pronouns which changes for the other layers where
correlations for hen are even higher ($\rho=0.72$) than for gendered pronouns
($\rho=0.65$). This suggests that there is some development across layers that
is stronger for hen than for gendered pronouns. Furthermore, we see a similar
evolvement for correlations across layers in English but a much weaker
correlation for Danish.

To investigate those effects across layers further, we look at word embeddings
for all Swedish pronouns from all 12 layers in BERT and compute pair-wise
cosine similarity including the Swedish word for book (bok) as a baseline
where we expect no specific relation to pronouns. In Figure 1, we see less
similarity between hen and the other pronouns in the first layer. This changes
for layer 6 and 12 where word representations seem to be more similar and the
three 3rd person pronouns hen, han, hon get closer to each other. This is in
line with the literature where it has been found that single attention heads
perform better on pronoun resolution than others. In particular middle and
deeper layers have shown stronger attention weights between coreferential
elements Vaswani et al. (2017); Webster et al. (2018); Clark et al. (2019).
Given that we do not consider individual heads or layers but the entire
attention graph it is not surprising that we also see those effects in the top
layer as has been shown in the original paper Abnar and Zuidema (2020).

Figure 1: Pair-wise cosine similarity between word representations of all pronouns and the Swedish word bok (book) as a baseline for different layers of BERT. We see that gender-neutral _hen_ grows from being an outsider (similar to _bok_) in the 1st layer into the cluster of gendered 3rd person pronouns _hon/han_ across layers. | en | da | sv
---|---|---|---
| orig. | they | xe | orig. | de | høn | orig. | de | hen
mBERT | 83.33 | 83.23 | 81.82 | 71.15 | 71.24 | 69.72 | 71.91 | 71.14 | 71.06
XLM-R | 95.13 | 94.81 | 94.05 | 80.19 | 79.18 | 75.48 | 78.79 | 78.5 | 78.58
Table 2: Accuracy [in %] on NLI for English, Danish and Swedish for both
models mBERT and XLM-R. Accuracies are calculated on the subset of sentences
that contain relevant pronouns (924 for en and 2339 for da/sv). The first
column for each language shows the accuracy on the original data, second and
third columns show accuracies for respective gender-neutral pronouns. Please
note, the total number of label flips in both directions for different
pronouns is higher than the performance difference for all pair-wise
comparisons. A baseline analysis where we exchanged punctuation ("." for "!")
yields similar deviations from the original dataset than the changing
pronouns.

##  3 Downstream Tasks

We also perform downstream task experiments on natural language inference and
coreference resolution for both gendered and gender-neutral pronouns to
investigate to what extent gender-neutral pronouns influence the performance.

### Natural Language Inference

Natural Language Inference (NLI) is commonly framed as a classification task,
which tests a model’s ability to understand entailment and contradiction
(Bowman et al., 2015). Despite high accuracies achieved by SOTA models, we are
yet to know whether they succeed in combating gender bias, especially in
cross-lingual settings. We apply two multilingual models
mBERT666multi_cased_L-12_H-768_A-12 (Devlin et al., 2019) and XLM-R777xlm-
roberta-large (Conneau et al., 2020) with cross-lingual fine-tuning, i.e., we
fine-tune on English and apply both models also on Danish and Swedish.
Therefore, mBERT was fine-tuned on the English MNLI train split and evaluated
on XNLI. For XLM-R, we apply a model that has been fine-tuned on both MNLI and
ANLI (Nie et al., 2020)888huggingface.co/vicgalle/xlm-roberta-large-xnli-anli.
For English we test both models on the MNLI test split, for Danish and Swedish
we test on the extended XNLI corpus Singh et al. (2019), the manual
translation of the first 15000 sentences of the MNLI corpus (Williams et al.,
2018) from English into 15 languages.

### Coreference Resolution

We also run pronoun resolution experiments on the Winogender dataset Rudinger
et al. (2018) where all 720 English sentences include an occupation, a
participant and a pronoun. For each occupation, two similar sentences are
composed, one where the pronoun refers to the occupation and one where it
refers to the participant. Those sentences are then presented in versions with
different pronouns (female, male, singular they). For our experiments, we
compare performance for those pronouns and add a version for the gender-
neutral pronoun xe. We run experiments with NeuralCoref 4.0 in
SpaCy.999github.com/huggingface/neuralcoref. Lauscher et al. (2022) conduct
similar experiments in English where all pronouns are exchanged for their POS
tag, in contrast to our experiments where we only exchange gendered pronouns
and replace them with gender-neutral pronouns.

For Danish, we apply the recently published coreference model Barrett et al.
(2021) to both the corresponding test set from the Dacoref dataset and a
gender-neutralized version where we exchange gendered pronouns hun/han for
either høn or singular de.101010So far, no Swedish coreference model has been
published, we therefore leave this analysis for future work.

##  4 Results

### Natural Language Inference

Accuracies for all languages and both models are displayed in Table 2. We
overall see a very small drop in performance for the datasets with gender
neutral pronouns compared to the original sentences. For mBERT we see
differences of $0.09-1.51\%$, for XLM-R the drop is slightly higher with
$0.21-4.71\%$. We see the biggest difference for the Danish pronoun høn in
comparison to the original dataset.

### Coreference Resolution

| she | he | they | xe
---|---|---|---|---
acc in % | 42.92 | 43.75 | 27.92 | 0
Table 3: Results for the pronoun resolution task on the English Winogender dataset. | orig. | de | høn
---|---|---|---
F1-score | 0.64 | 0.63 | 0.62
Prec. | 0.70 | 0.69 | 0.69
Recall | 0.59 | 0.57 | 0.56
Table 4: Results for the Danish coreference resolution task. Pronouns in the
original dataset (orig.) have been exchanged for singular de and gender-
neutral høn.

Table 3 shows accuracies on the English Winogender corpus for all four
pronouns. We see a clear drop in performance from gendered pronouns (she, he)
to both gender-neutral pronouns (they, xe). For xe, the model was not able to
perform coreference resolution at all. In most cases it was not even
recognized as part of a cluster and in the rare cases where it was, it was
clustered with the wrong tokens. Please note that since this dataset is not
labelled we are only classifying if the pronoun has been clustered with the
correct entity.

Results on the Danish Coref corpus, where we are able to perform a more
extensive coreference resolution task are displayed in Table 4. We were able
to replicate results from Barrett et al. (2021) (the first column orig.) and
see small drops in performance for singular de and høn.

##  5 Related Work

More eye-tracking studies have been conducted investigating the influence in
processing cost for both gender-neutral pronouns and the generic male pronoun.
Irmen (2007) and Redl et al. (2021) find male biases when using generic male
pronouns in Dutch and generic role nouns in German. The authors of Sanford and
Filik (2007) found a clear processing cost when using singular they in
English, however their stimuli did not include any investigation of how
(anti-)stereotypes influence this processing cost and is thus only in parts
comparable to other studies. English datasets have been proposed to
investigate gender bias in pronoun resolution but have not reported on
performance differences between gendered and gender-neutral pronouns Rudinger
et al. (2018); Zhao et al. (2018); Webster et al. (2018). Sun et al. (2021)
propose a rewriting task where data is transferred from gendered to gender-
neutral pronouns to train more inclusive language models. Cao and Daumé III
(2020) and Dev et al. (2021) discuss the necessity of including non-binary
pronouns into NLP research (see also Stanczak and Augenstein (2021)).

##  6 Discussion

With this paper we provide a first study on how well language can handle
gender-neutral pronouns in Danish, English and Swedish for various tasks. We
observe an increase in perplexity for gender-neutral pronouns and correlations
between perplexity on sentence level and attention flow on the pronoun, in
particular for English and Swedish that gets stronger across layers. This
indicates that language models indeed struggle with the use of gender-neutral
pronouns, even with singular _they_ , which has been used for many years as
gender-neutral Saguy and Williams (2022). The reason for this most likely lies
in the sparse representation of gender-neutral pronouns in the training data
and the fact that language models, once they are trained and published usually
are not updated Bender et al. (2021). However, Transformer models pre-trained
on subword units have been shown to be robust with respect to word frequency
Sennrich et al. (2016) and thus should be able to process unfamiliar gender-
neutral pronouns. At the same time, we observe that word representations of
all Swedish 3rd person pronouns grow closer in middle and top layers (see
Figure 1) which suggests that relevant information is also learned for gender-
neutral _hen_.

For NLI, we only see a small drop in performance when exchanging gendered
pronouns for gender-neutral pronouns which is in the same range as a baseline
analysis where we exchange punctuation ("!" for "."), except for Danish _høn_.
We argue that classification in NLI probably does not heavily rely on
individual pronouns in most cases. In stark contrast to pronoun resolution
where we see a very clear drop in performance for English when applying
singular _they_ in comparison to both female and male pronouns, again this is
surprising since in theory language models should have seen training samples
where singular _they_ has been used. The small drop in performance for Danish
coreference resolution might be because this dataset does not solely focus on
pronoun resolution, though further investigation is needed here. We strongly
argue that more needs to be done to adapt language models to a more gender
inclusive language, initiatives like the rewriting task as proposed by Sun et
al. (2021) need to be implemented and extended.

## Acknowledgements

This work was partially funded by the Platform Intelligence in News project,
which is supported by Innovation Fund Denmark via the Grand Solutions program.
We thank Vinit Ravishankar and Jonas Lotz for fruitful discussions and Daniel
Hershcovich and Yova Kementchedjhieva for proof-reading and valuable inputs on
the manuscript. We also thank Kellie Webster for her valuable input on gender-
neutral pronouns.

## References

  * Abbasi (2020) Irum Abbasi. 2020.  The influence of technology on english language and literature.  _English Language Teaching_ , 13:1.
  * Abnar and Zuidema (2020) Samira Abnar and Willem Zuidema. 2020.  Quantifying attention flow in transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4190–4197, Online. Association for Computational Linguistics.
  * Barrett et al. (2021) Maria Barrett, Hieu Lam, Martin Wu, Ophélie Lacroix, Barbara Plank, and Anders Søgaard. 2021.  Resources and evaluations for Danish entity resolution.  In _Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference_ , pages 63–69, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 610–623.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Cao and Daumé III (2020) Yang Trista Cao and Hal Daumé III. 2020.  Toward gender-inclusive coreference resolution.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4568–4595, Online. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Dev et al. (2021) Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021.  Harms of gender exclusivity and challenges in non-binary representation in language technologies.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1968–1994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eberle et al. (2022) Oliver Eberle, Stephanie Brandl, Jonas Pilot, and Anders Søgaard. 2022.  Do transformer models show similar attention patterns to task-specific human gaze? To appear at ACL 2022.
  * Goodkind and Bicknell (2018) Adam Goodkind and Klinton Bicknell. 2018.  Predictive power of word surprisal for reading times is a linear function of language model quality.  In _Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018)_ , pages 10–18, Salt Lake City, Utah. Association for Computational Linguistics.
  * Gustafsson Sendén et al. (2015) Marie Gustafsson Sendén, Emma A Bäck, and Anna Lindqvist. 2015.  Introducing a gender-neutral pronoun in a natural gender language: the influence of time on attitudes and behavior.  _Frontiers in Psychology_ , 6:893.
  * Gustafsson Sendén et al. (2021) Marie Gustafsson Sendén, Emma Renström, and Anna Lindqvist. 2021.  Pronouns beyond the binary: The change of attitudes and use over time.  _Gender & Society_, 35(4):588–615.
  * Hekanaho (2020) Laura Hekanaho. 2020.  _Generic and Nonbinary Pronouns : Usage, Acceptability and Attitudes_.  Ph.D. thesis, University of Helsink, Helsinki, Finland.
  * Irmen (2007) Lisa Irmen. 2007.  What’s in a (role) name? formal and conceptual aspects of comprehending personal nouns.  _Journal of Psycholinguistic Research_ , 36(6):431–456.
  * Kristiansen et al. (2011) Tore Kristiansen, Nikolas Coupland, Barbara Soukup, Sylvia Moosmüller, Frans Gregersen, Peter Garrett, Charlotte Selleck, Pirkko Nuolijärvi, Johanna Vaattovaara, Jan-Ola Ingemar Östman, Leila Mattfolk, Philipp Stoeckle, Christoph Hare Svenstrup, Stephen Pax Leonard, Kristján Árnason, Tadhg Ó Hifearnáin, Noel Ó Murchadha, Loreta Vaicekauskienė, Stefan Grondelaers, Roeland van Hout, Helge Sandøy, Mats Thelander, Elen Robert, Jannis Androutsopoulos, Peter Auer, Helmut Spiekermann, Allan Bell, Dirk Speelman, and Jane Stuart-Smith. 2011.  Language change and digital media: A review of conceptions and evidence.  In _Standard languages and language standards in a changing Europe_.
  * Lauscher et al. (2022) Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022.  Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender.  _arXiv preprint arXiv:2202.11923_.
  * Mielke et al. (2019) Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019\.  What kind of language is hard to language-model? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4975–4989, Florence, Italy. Association for Computational Linguistics.
  * Miltersen (2020) Ehm Hjorth Miltersen. 2020.  Singular de and its referential use in talk-in-interaction.  _Scandinavian Studies in Language_ , 11(2):37–37.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Redl et al. (2021) Theresa Redl, Stefan L Frank, Peter De Swart, and Helen De Hoop. 2021.  The male bias of a generically-intended masculine pronoun: Evidence from eye-tracking and sentence evaluation.  _PloS one_ , 16(4):e0249309.
  * Roh et al. (2020) Jihyeon Roh, Sang-Hoon Oh, and Soo-Young Lee. 2020.  Unigram-normalized perplexity as a language model performance measure with different vocabulary sizes.  _arXiv preprint arXiv:2011.13220_.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics.
  * Saguy and Williams (2022) Abigail C Saguy and Juliet A Williams. 2022.  A little word that means a lot: A reassessment of singular they in a new era of gender politics.  _Gender & Society_, 36(1):5–31.
  * Sanford and Filik (2007) Anthony J Sanford and Ruth Filik. 2007.  “they” as a gender-unspecified singular pronoun: Eye tracking reveals a processing cost.  _Quarterly Journal of Experimental Psychology_ , 60(2):171–178.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Singh et al. (2019) Jasdeep Singh, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  BERT is Not an Interlingua and the Bias of Tokenization.  _The Workshop on Deep Learning for Low-Resource NLP at EMNLP 2019_.
  * Speyer and Schleef (2019) Lydia Gabriela Speyer and Erik Schleef. 2019.  Processing ‘gender-neutral’pronouns: A self-paced reading study of learners of english.  _Applied Linguistics_ , 40(5):793–815.
  * Stanczak and Augenstein (2021) Karolina Stanczak and Isabelle Augenstein. 2021.  A survey on gender bias in natural language processing.  _arXiv preprint arXiv:2112.14168_.
  * Sun et al. (2021) Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021\.  They, them, theirs: Rewriting with gender-neutral english.  _arXiv preprint arXiv:2102.06788_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_.
  * Vergoossen et al. (2020a) Hellen P Vergoossen, Philip Pärnamets, Emma A Renström, and Marie Gustafsson Sendén. 2020a.  Are new gender-neutral pronouns difficult to process in reading? the case of hen in swedish.  _Frontiers in psychology_ , 11:2967.
  * Vergoossen et al. (2020b) Hellen Petronella Vergoossen, Emma Aurora Renström, Anna Lindqvist, and Marie Gustafsson Sendén. 2020b.  Four dimensions of criticism against gender-fair language.  _Sex Roles_ , 83(5):328–337.
  * Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019.  Does it make sense? and why? a pilot study for sense making and explanation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4020–4026, Florence, Italy. Association for Computational Linguistics.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20.


capbtabboxtable[][]

# Exploring Generalization Ability of Pretrained Language Models on Arithmetic
and Logical Reasoning

Cunxiang Wang$\spadesuit$$\clubsuit$, Boyuan Zheng$\clubsuit$$\diamondsuit$,
Yuchen Niu$\clubsuit$$\flat$ and Yue Zhang$\clubsuit$$\heartsuit$
$\spadesuit$Zhejiang University, China
$\clubsuit$School of Engineering, Westlake University, China
$\heartsuit$Institute of Advanced Technology, Westlake Institute for Advanced
Study, China
$\diamondsuit$Johns Hopkins University
$\flat$Imperial College London
{wangcunxiang, zhangyue, zhengboyuan<EMAIL_ADDRESS>
The corresponding author

###### Abstract

To quantitatively and intuitively explore the generalization ability of pre-
trained language models (PLMs), we have designed several tasks of arithmetic
and logical reasoning. We both analyse how well PLMs generalize when the test
data is in the same distribution as the train data and when it is different,
for the latter analysis, we have also designed a cross-distribution test set
other than the in-distribution test set. We conduct experiments on one of the
most advanced and publicly released generative PLM - BART. Our research finds
that the PLMs can easily generalize when the distribution is the same,
however, it is still difficult for them to generalize out of the distribution.

##  1 Introduction

Neural networks have shown strong capabilities in a range of NLP tasks
(Sutskever et al., 2014; Vaswani et al., 2017). Recently, pretrained language
models (PLMs) have achieved significantly levels of performance gains on many
benchmark datasets (Devlin et al., 2019; Lewis et al., 2020a; Radford et al.,
2019). Recently, some work shows that neural networks are lack of
generalization ability in mathematical and logical reasoning (Nogueira et al.,
2021; Madsen and alexander rosenberg johansen, 2019). This can lead to more
understanding of the limitation of existing models and motivate future work.
However, no work has been done to quantitatively or intuitively explore the
conditions under which PLMs can generalize, in terms of whether PLMs can
understand the internal mathematical rules and logical rules. The example of
mathematical rules is shown in Figure 1. We suppose that if the model can
effectively learn the underlying rules of Addition and Subtraction when giving
sufficient training data, it can generalize to all two-number addition and
subtraction calculation.

To this end, we conduct quantitative insights by designing a series of tasks
for simple mathematical operations and logical reasoning, which includes
numbering, addition, subtraction, comparison, and symbolic logic. We construct
a set of corresponding datasets, where instances are in the form of text or
mathematical expressions. Some examples are shown in the next section. For
example, in the Addition task, ‘100 + 200’ is the question and ‘300’ is the
answer.

Figure 1: Example mathematical rules for Addition and Subtraction. If the
model can master these rules, we suppose it can generalize well on all two-
number addition and subtraction samples.

There are various types of generalization (Linzen, 2020; Lake and Baroni,
2018), such as question generalization on distribution differences between
training set and test set (Wallace et al., 2019), and answer generalization on
distribution differences between training set and test set (Nogueira et al.,
2021). For example, in the Addition task, if the question and answer numbers
in training data are of three-digit, but the question and answer numbers in
the testing data are of two- or four-digit, they are in different
distribution. To cover each type of generalization, we use different kinds of
tasks and corresponding dataset. For example, we use addition to test the
generalization on the question distribution differences data between training
and testing. In this task, the numbers in the training set and development
have three digits. However, the numbers in test set is set to consist of two,
three, and four digits.

We conduct experiments using BART (Lewis et al., 2020a) since they can
generate arbitrary text sequences and have been shown to achieve the state-of-
art results on numerous Natural Language Processing (NLP) tasks. For each
task, we fine-tune BART with training data, validate on the development set
and finally evaluate on the test set. We find that strong PLMs can address
simple generalization of the same answer distribution for counting, arithmetic
and logic tasks. But they cannot master the underlying rules of arithmetic
reasoning, for example, the model trained on 3-digit addition can handle the
addition expressions with 2-digit or 4-digit.

We will release all the code and data set for future study.

##  2 Task

We construct five tasks related to algebraic and logical reasoning, namely
Numbering, Addition, Subtraction, Comparison, Symbolic Logic. In order to test
the generalization ability of models on the data with the same distribution
and on the data with the different distribution, we create an in-distribution
dataset and a cross-distribution dataset for each task. The in-distribution
dataset contains train set, development set and test set that are in the same
distribution. The cross-distribution dataset only serves as the test set and
it is in the different distribution in contrast to the in-distribution
dataset. We believe that if the model can understand the underlying rules of
arithmetic and logical Reasoning, it can both generalize well on in-
distribution and cross-distribution test set.

Task |  Train Set |  Dev Set |  In- + Cross- Distribution Test Set |  In- + Cross- Distribution Dataset
---|---|---|---|---
Numbering - Counting | 3,744 | 468 | 468 + 2,030 | 4,680 + 2,030
Numbering - Listing | 3,744 | 468 | 468 | 4,680
Addition | 256,320 | 32,040 | 32,040 + 4,000 | 320,400 + 4,000
Subtraction | 256,320 | 32,040 | 32,040 + 4,000 | 320,400 + 4,000
Comparison | 648,000 | 81,000 | 81,000 + 5,600 | 810,000 + 5,600
Symbolic Logic | 40,000 | 5,000 | 5,000 + 2,200 | 50,000 + 2,200
Table 1: Data statistics of each task. For each task, we list the in-
distribution dataset and cross-distribution test set.

###  2.1 Numbering

Figure 2: The Numbering task has two subtasks, namely Counting and Listing.

This task comprises two symmetric subtasks, namely Counting and Listing.
Examples are shown in Figure 2. The Counting task asks the model to count the
number of characters in the input sequence. For example, ‘A A A A A A’ is a
sequence with length ‘6’. The Listing task asks the model to output a list
with a specific length and character. For example, the model receives a
command ‘Generate a list of 6 A’ and the result is ‘A A A A A A’.

###  2.2 Addition

(a) Description of Addition task.

(b) Description of Subtraction task.

(c) Description of Comparison task.

(d) Description of Symbolic Logic task.

The Addition task is the standard summation of two input numbers. In order to
make sure that all numbers are in the same distribution during training, we
use only the equations whose left-hand-side and right-hand-side are both three
digits in the in-distribution dataset. We also adopt two-digit and four-digit
numbers on both sides in cross-distribution test set to further test the
generalization ability of models. One example is shown in Figure 3(a).

###  2.3 Subtraction

The Subtraction task the standard tack(?) to subtract a subtrahend from a
minuend. In order to make sure that all numbers are in same distribution
during training, we use only equations whose left-hand-side and right-hand-
side are both three digits in the in-distribution dataset. We also adopt two-
digit and four-digit numbers on both sides in cross-distribution test set to
further test the generalization ability of models. A example of Subtraction
task is shown in Figure 3(b).

###  2.4 Comparison

The Comparison task is to determine which of the two numbers is greater or
smaller. In order to make sure all numbers are in same distribution during
training, we use only equations whose left-hand-side and right-hand-side are
both three digits in the in-distribution dataset. We also adopt two-digit and
four-digit numbers on both sides in cross-distribution test set to further
test the generalization ability of models. One example is shown in Figure
3(c).

###  2.5 Symbolic Logic

As shown in Figure 3(d), this task is to reason over symbolic logic
expressions. The input question expression consists of six basic components,
which are ‘0’, ‘1’, ‘$\&$’, ‘$|$’, ‘$\neg$’ and ‘$\to$’, representing FALSE,
TRUE, AND, OR, NOT and IMPLY, respectively. The output answer is either 0 or
1, which represent FALSE and TRUE, respectively. This task asks the model to
reason over the input logic expression and determine whether it is true or
false.

In order to make sure all expressions are in the same distribution during
training, we use only the expressions that contain 6 - 10 basic ‘0’ and ‘1’
components. For testing the generalization ability of models, we also adopt
the some expressions with 1 - 15 basic ‘0’ and ‘1’ components in the test set.

Different from the other tasks, we select a subset from the overall dataset to
serve as the in-distribution dataset because the data is large. We take only
10,000 of expressions with X basic components, where X is a number between 6 -
10, respectively. So, we end up with 50,000 samples in the in-distribution
dataset.

###  2.6 Metrics

We use Exact Match to compute accuracy for Numbering, Addition, Subtraction
and Comparison tasks. However, for the Symbolic Logic task, since the answer
distribution is unbalanced (84% answers are ‘1’), we use the F1 score as the
metric.

##  3 Experiments

In this section, we separate the generalization experiments to In-Distribution
Generalization experiments and Cross-Distribution experiments. In the former,
the testing data is in the same distribution with the training data. In the
latter, the testing data is in the different distribution from the training
data. We suppose that if the model can master the underlying rules of the
mathematical and logical reasoning, it should achieve 100% accuracy on both
In-Distribution Generalization experiments and Cross-Distribution experiments.

We have organized the details of in-distribution data and cross-distribution
data in this section. In addition, We also sorted out the examples of them and
put the examples in the Appendix Table 1.

###  3.1 Experimental Settings

We adopt BART (Lewis et al., 2020a) namely due to the following reasons.
First, it is a generative pretrained language model, which means that they can
generate arbitrary sequences of tokens. This is essential for the addition and
subtraction task. Second, it has achieved state-of-art results on numerous
tasks and they has received much research attention. Last, it has released
model checkpoints, thus it can be more standardized and more fair can evaluate
them.

For the BART (Lewis et al., 2020a) model, we conduct experiments on the
publicly released ‘BART-Large’ checkpoint
111https://huggingface.co/facebook/bart-large/tree/main. We insert spaces
between numbers while representing them in the data. For example, ‘111’ is
written as ‘1 1 1’ both in the question and answer. For the character sequence
in the Numbering task, we also insert spaces between the sequence, such as ‘A
A A’.

(a) The results of counting task.

(b) The results of listing task.

(c) The results of addition task.

(d) The results of subtraction task.

(e) The results of comparison task.

(f) The results of symbolic logic.

Figure 4: The in-distribution results on each task.

###  3.2 In-Distribution Generalization

In this subsection, we mainly explore models’ generalization ability on test
data which in the same distribution with train data. For the Counting subtask
of the Numbering task, each question is a sequence with 10-99 same character
which is one character among the alphabet; each answer is an integer between
10 and 99. For the Listing task, each question is a textual sequence ‘Generate
a list with X Y’, where X is an integer between 10 and 99 and Y is one
character among the alphabet; each answer is a sequence with 10-99 same
characters. For the Addition task, each question is an addition expression,
and the answer is a sum number. Each number in the question and answer is
three digits. For the Subtraction task, each question is an subtraction
expression , and the answer is a difference number. Each number in the
question and answer is of three-digit. For the Comparison task, each question
is made of two numbers and each answer is a single symbol which is either
‘$>$’ or ‘$<$’ or ‘$=$’. The numbers in the question are all of three-digit.
For the Symbolic Logic task, each question is a sequence with 5-10 basic ‘0’
and ‘1’ components; each answer is either 0 or 1.

For testing generalization ability on the same distributional data, we explore
how the number of training samples affects the generalization. For each task,
we extract subsets from the in-distribution train set and train on the
subsets, but keep the distribution of development set and test set the same.
Thus, we analyse how the number of training samples influences the
performance, which also indicate the generalization ability of models on the
data with the same distribution.

The in-distribution results on the Numbering task are shown in Figure 4(b)
Figure 4(a). For the Listing subtask, we find that the model’s generation
results are very unstable, which means that the outputs often contain other
tokens other than the needed character. For example, when the input is
‘Generate a list of 6 A’, the output can be ‘A A a Aa E T A’. When the
sequence length increases, this kind of disruption will be more likely to
occur. So, results are always around zero. We suppose this result is result
from the instability of the generative model itself, because we also observe
this situation from other generative models, such as T5 (Raffel et al., 2020).
So, we mainly analyse the Counting task rather the Listing task in the
following sections.

It can be seen that when the number of training samples increase, the
performance of Counting will also improve.

The in-distribution results on the Addition task are shown in Figure 4(c). We
can seen that when the number of training samples is 1600 (0.5% of the
dataset), the model can achieve 99% accuracy; even when the number of training
samples is reduced to 160 (0.05% of the dataset), the model can still achieve
around 40% accuracy. The in-distribution results on the Subtraction,
Comparison, Symbolic Logic task are shown in Figure 4(d), Figure 4(e) and
Figure 4(f), respectively. It can be seen from the figures that when the
number of training samples increase, the model can perform better in the in-
distribution test set. And when the training samples increase to several
hundreds, the model can achieve around 100% accuracy or F1, showing BART’s
ability on the in-distribution generalization. Thus, we are wondering whether
the model has truly learn the underlying rules of these tasks or they just use
some spurious correlations to solve these questions, so, we design cross-
distribution generalization test set to further explore the model’s
generalization ability in the following section.

###  3.3 Cross-Distribution Generalization

In this section, we analyse how models generalize (1) when test question
distribution is different from train question while the test answer
distribution is the same; (2) when test answer distribution is different from
the train answer while the test question distribution keeps the same; (3) when
the test question distribution and test answer distribution are both different
from train set. We have designed testing data for different types of cross-
distribution on each task and list examples of the testing data in this
section.

####  3.3.1 Varying Questions

In this part, we mainly talk about when the test question distribution is
different from the train question while the test answer distribution keeps the
same, how strong is the model’s generalization ability. So, we use the
Counting, Addition, Subtraction, Comparison, and Symbolic Logic tasks to
analyse. For the Counting task, we use the instances whose character is not in
letters of an alphabet while the number is still of two-digit. For example,
the question is ‘$@\ @\ @\ @\ @\ @\ @\ @\ @\ @$’ and the answer is ‘10’. For
the Addition task, we use the instances whose at least one added number is of
two-digit. But we make sure answers of selected equations are all of three-
digit. For example, the question is ‘50 + 170’, the answer is ‘220’. For the
Subtraction task, the situation is similar to the Addition task, we use the
instances whose at least one number is of four-digit. But we make sure answers
of selected instances are all of three-digit. For example, the question is
‘1000 - 500’, the answer is ‘500’. For the Comparison task, the situation is
also similar, we use the instances whose at least one number is of two-digit
or four-digit. For example, the question is ‘56 176’, the answer is ‘$<$’. For
the Symbolic Logic task, the situation is also similar, we use the instances
which has 1 - 5 or 11 -15 basic ‘0’ and ‘1’ components. For example, the
question is ‘$not\ 0\ and\ 1\ or\ 0$’, the answer is ‘1’.

####  3.3.2 Varying Answers

In this part, we mainly talk about when the test answer distribution is
different from the train answer while the test question distribution keeps the
same, how strong is the model’s generalization ability. As a result, we use
the Addition and Subtraction to analyse.

For the Addition task, we use the instances whose two numbers are of three-
digit while the answer is of four-digit. For example, the question is ‘500 +
600’, the answer is ‘1100’. For the Subtraction task, the situation is similar
to the Addition task, we use the instances whose two numbers are of three-
digit while the answer is of two-digit. For example, the question is ‘550 -
500’, the answer is ‘50’.

####  3.3.3 Varying Instances

In this part, we mainly talk about hen the test question distribution and test
answer distribution are both different from the train set, how strong is the
model’s generalization ability. So, we use the Counting, Addition and
Subtraction tasks to analyse.

For the Counting task, we use the instances whose character is not in letters
of an alphabet and number is not of two-digit. For example, the question is
‘$@\ @\ @\ @\ @\ @\ @\ @\ @$’ and the answer is ‘9’. For the Addition task, we
use the instances whose at least one number in question is of two- or four-
digit and the answer number is also of two- or four-digit. For example, the
question is ‘50 + 960’, the answer is ‘1010’. For the Subtraction task, the
situation is similar to the Addition task, we use the instances whose at least
one number is of two- or four-digit and the answer is also of two- or four-
digit. For example, the question is ‘1100 - 50’, the answer is ‘1050’.

Task |  Question Cross-Distribution |  Answer Cross-Distribution |  Instance Cross-Distribution
---|---|---|---
Counting | 316/320(98.8%) | / | 0/1710
Addition | 15/1,500 (1.0%) | 0/1,500 | 0/1,000
Subtraction | 13/1,500 (0.87%) | 0/1,500 | 1/1,000 (0.1%)
Comparison | 2,555/5,600(45.63%) | / | /
Symbolic Logic | 2,200/2,200 (100%) | / | /
Table 2: The performance of BART on cross-distribution test set. For each task
and different distribution type, we select the model checkpoint which has
achieved 100% accuracy/F1 on the corresponding in-distribution test set. Note
that the random result on Comparison is around 49.9%. Data samples that models
answer correctly on Addition and Subtraction task in the Cross-Distribution
experiment can be found in Appendix A[Some of the examples here are deleted
since they do not conform the fomat]

####  3.3.4 Analysis on Different Cross-Distributions

The model’s performance on the test set of different types of cross-
distributions is shown in Table 2. From the table, we can see that although
BART has achieved 100% accuracy on the in-distribution testing data, it fails
to generalize on the cross-distribution testing data of arithmetic reasoning
tasks.

Results of Counting and Symbolic Logic task on cross-distribution testing data
are quite high. However, for Counting task, all correct instances are the
instances which have different length but have the same character
distributions with the training data. In addition, the cross-distribution
testing data only have length difference from the training data. Thus, we can
conclude that the model is not sensitive to the length of question if the
basic components does not change. This conclusion is also consistent with the
result of (Clark et al., 2020). In addition, the results show that the model
is especially weak in generalizing to the instances with different answer
distributions.

To conclude, the model is still struggling on cross-distribution
generalization, especially the carrying and borrowing in Addition and
Subtraction tasks.

###  3.4 Case Study on GPT-3

GPT-3 (Brown et al., 2020) has received a lot of attention since it was born.
And it has shown strong abilities on every single NLP task as well as on
generalization. Thus, we also conduct some case study experiments of
arithmetic calculation on GPT-3 222The experiments are conducted on
https://beta.openai.com/examples/default-qa . But since the OpenAI has not
released the whole API, we cannot finetune the model or do large scale
experiments.. We find that GPT-3 can handle the addition and subtraction
calculation with 1,000 perfectly, but when the number increases, GPT-3 starts
to lose its ability, it can only get some very specific instances correctly. A
interesting case is that it can get correct result ‘9999999’ from ‘12345678 +
87654321’, however, when we give it ‘12345678 + 8765432’, it still answers
‘99999999’. We guess that the model does not have calculation ability, but
rather remembers some examples that have appeared before, since each
calculation with 1000 and ‘12345678 + 87654321’ may appear in the Internet for
many times while ‘12345678 + 8765432’ may not so frequently appear.

###  3.5 Overlap Analysis

We have also explored how overlaps influence models’ performance.

Following (Lewis et al., 2020b) and (Wang et al., 2021), if one test question
appears in the questions of train set, we call it as question overlap,
otherwise it is question non-overlap. Similarly, if one test answer appears in
the answers of train set, we call it as answer overlap, otherwise it is answer
non-overlap. If one test instance is both question overlap and answer overlap,
we call it is instance overlap, otherwise it is instance non-overlap.

We mainly use results of the Addition task to illustrate this problem.
However, for these two task, the question overlap is sightly different, if the
two numbers of one test question both appear in the numbers of train set, we
call it is question overlap, otherwise it is question non-overlap. Since one
answer of these two tasks only contain one number, the situation is same with
the original definition.

We choose the two results which using 1920 instances (0.6% for the dataset)
for training in the Addition task, because it has achieved 68% accuracy, which
means that the results have both correct and incorrect instances.

| Correct |  Incorrect
---|---|---
Overlap | 1,083 | 524
Non- Overlap | 20,858 | 9,575
(a) Instance Overlap

| Correct |  Incorrect
---|---|---
Overlap | 4,538 | 1,846
Non- Overlap | 12,403 | 8,253
(b) question Overlap

| Correct |  Incorrect
---|---|---
Overlap | 6,066 | 3,070
Non- Overlap | 15,873 | 7,029
(c) answer Overlap

Table 3: The overlap analysis.

The results are shown in Table 3. From the table, we can see that, unlike
results from (Lewis et al., 2020b) and (Wang et al., 2021), the overlap and
non-overlap do not influence the models’ performance.

##  4 Related Work

Some works have investigated in Mathematical problems in NLP (Dua et al.,
2019; Wang et al., 2017; Zhao et al., 2020). DROP (Dua et al., 2019) is a
reading comprehension dataset comprising several kinds of mathematical tasks,
such as Subtraction and Selection. However, all answers of its questions can
be directly or indirectly found in the corresponding passages. Math23L (Wang
et al., 2017) is simple math word problem dataset with 23k problems. Its
problem is of the simple English context format, along with the equation and
the answer. Ape210K (Zhao et al., 2020) is a Chinese simple math word problem
dataset with 210k questions. The questions are similar to Math23L’s questions.
The data are taken from some elementary school math word problems. These
datasets do not contain a generalization test set, the test set is in the same
distribution with the train set. In addition, the often used methods for these
datasets are first to predict the equations or expression for the question and
then to use calculation tool to get the result (Wang et al., 2017;
Wangperawong, 2018). However, our work concentrate on the generalization
ability of models. Thus, we have designed test set with different
distribution. In addition, we try to use the model to directly solve the
questions, aiming test model’s internal ability of understanding the deep
rules of arithmetic and logical reasoning.

Some works have researched on models’ the internal ability of solving
mathematical expressions. Wallace et al. (2019) has investigated that how will
different types of embedding, such as BERT (Devlin et al., 2019) and GloVe
(Pennington et al., 2014), affect the performance of the same NAQANet model
(Dua et al., 2019) on the same tasks including List Maximum, Decoding and
Addition. Besides, Wallace et al. (2019) also explores that how the the way
numbers are represented and the way to do tokenization affect the performance
of models. Geva et al. (2020) try to inject numerical reasoning skill by
adding a calculation module into the PLMs, which helps the performance on DROP
(Dua et al., 2019) dataset.

There are also some works research focusing on the generalization ability of
neural network models. Lake and Baroni (2018) research on the compositional
generalization skills of sequence-to-sequence models, such as LSTM (Hochreiter
and Schmidhuber, 1997) and GRU (Chung et al., 2014). Linzen (2020) explain
that the generalization test in machine learning (ML) is not very reasonable,
they put forward seven suggestions to better evaluate the generalization
ability of ML models. Lewis et al. (2020b) and Wang et al. (2021) find that
the PLMs cannot generalize well on Closed-book QA task (Roberts et al., 2020),
the model can handle the test instances which overlap with the train data,
however, they cannot solve the non-overlapped instances. McCoy et al. (2020)
find that even when the model’s architecture is set, the generalization
ability of the model is still influenced largely by the random luck, the
random initialized weights and other things. Clark et al. (2020) perform
Transformer-based models on simple logic reasoning test, and their results
show that the model can get quite promising results and the model is not
sensitive to the question length. Though Wang et al. (2020a) proves that
pretrained language models (Liu et al., 2019; Lan et al., 2020) can generalize
well on textual commonsense reasoning tasks (Wang et al., 2019), Wang et al.
(2020b) finds that transformer models (Bosselut et al., 2019) may not
generalize well on commonsense knowledge graph (Sap et al., 2019) reasoning.
Zhang et al. (2020) analyses the generalization ability on the relation
extraction task and find some specific problems can induce a significant
decline in model performance.

##  5 Conclusion

We have designed a series of tasks for evaluating BART on simple mathematical
operations and logic reasoning, which includes numbering, addition,
subtraction, comparison, and symbolic logic. We constructed a corresponding
in-distribution datasets, and also designed cross-distribution test set to
further evaluate the model’s generalization ability. If the model can
understand the underlying rules of these mathematical operations and logic
reasoning, it can generalize well on both in-distribution and cross-
distribution test set. Our experiments showed that BART can only generalize on
the in-distribution test set but cannot perform well on the cross-distribution
test set, showing that the most advanced PLM still cannot understand the
underlying rules of simple mathematical operations and logic reasoning.

## References

  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019.  COMET: Commonsense transformers for automatic knowledge graph construction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4762–4779, Florence, Italy. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chung et al. (2014) J. Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _ArXiv_ , abs/1412.3555.
  * Clark et al. (2020) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020.  Transformers as soft reasoners over language.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3882–3890. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Geva et al. (2020) Mor Geva, Ankit Gupta, and Jonathan Berant. 2020.  Injecting numerical reasoning skills into language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 946–958, Online. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Lake and Baroni (2018) B. Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _ICML_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  _ArXiv_ , abs/1909.11942.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2020b) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020b.  Question and answer test-train overlap in open-domain question answering datasets.
  * Linzen (2020) Tal Linzen. 2020.  How can we accelerate progress towards human-like linguistic generalization? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5210–5217, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Madsen and alexander rosenberg johansen (2019) Andreas Madsen and alexander rosenberg johansen. 2019.  Measuring arithmetic extrapolation performance.  _ArXiv_ , abs/1910.01888.
  * McCoy et al. (2020) R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020.  BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 217–227, Online. Association for Computational Linguistics.
  * Nogueira et al. (2021) Rodrigo Nogueira, Zhiying Jiang, and J. Li. 2021.  Investigating the limitations of the transformers with simple arithmetic tasks.  _ArXiv_ , abs/2102.13019.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  _ArXiv_ , abs/1811.00146.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  _arXiv preprint arXiv:1409.3215_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Wallace et al. (2019) Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019.  Do NLP models know numbers? probing numeracy in embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5307–5315, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2020a) Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiao-Dan Zhu, and Y. Zhang. 2020a.  Semeval-2020 task 4: Commonsense validation and explanation.  In _SEMEVAL_.
  * Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019.  Does it make sense? and why? a pilot study for sense making and explanation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4020–4026, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2021) Cunxiang Wang, Pai Liu, and Yue Zhang. 2021.  Can generative pre-trained language models serve as knowledge bases for closed-book QA? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3241–3251, Online. Association for Computational Linguistics.
  * Wang et al. (2020b) Cunxiang Wang, Jinhang Wu, Luxin Liu, and Yue Zhang. 2020b.  Commonsense knowledge graph reasoning by selection or generation? why?  _ArXiv_ , abs/2008.05925.
  * Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017.  Deep neural solver for math word problems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 845–854, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wangperawong (2018) A. Wangperawong. 2018.  Attending to mathematical language with transformers.  _ArXiv_ , abs/1812.02825.
  * Zhang et al. (2020) Ningyu Zhang, Luoqiu Li, Shumin Deng, H. Yu, Xu Cheng, W. Zhang, and Huajun Chen. 2020.  Can fine-tuning pre-trained models lead to perfect nlp? a study of the generalizability of relation extraction.  _ArXiv_ , abs/2009.06206.
  * Zhao et al. (2020) Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020.  Ape210k: A large-scale and template-rich dataset of math word problems.  _CoRR_ , abs/2009.11506.

Task |  | Training Data &
---
In-Distribution Test
Cross-Distribution Test
|  |  | Question
---
Cross-
Distribution
| Answer
---
Cross-
Distribution
| Instance
---
Cross-
Distribution
Counting |  | a sequence
---
of [A-Z, a-z]
with a
length of
10 to 99.
(B B B
B B B
B B B B)
| a sequence
---
of special
characters
with a
length of
10 to 99
(@ @ @
@ @ @
@ @ @ @)
|  | a sequence
---
of special
characters
with a
length of
1 to 9 or
100 to 1000.
(@ @)
Addition |  | 3-digit
---
addition.
(100 + 200
= 300)
| at least
---
one addend
is 2 digits.
(50 + 170
= 220)
| the answer
---
is 4 digits.
(500 + 600
= 1100)
| at least one
---
number is
2 or 4 digits.
(50 + 960
= 1010)
Subtraction |  | 3-digit
---
subtraction.
(200 - 100
= 100)
| at least
---
one number
is 4 digits,
but the answer
is still 3 digits.
(1000 - 500
= 500)
| the answer
---
is 2 digits.
(550 - 500
= 50)
| at least one
---
number is
2 or 4 digits.
(1100 - 50
= 1050)
Comparison |  | 3-digit
---
comparison.
(100 <200)
| at least
---
one number
is not 3-digit.
(100 <2000)
|
| Symbolic
---
Logic
| an equation
---
consists of
6 to 10
”0”s or ”1”s.
(¬ 0 & 1 — 0
& 1 — 1 — 0 is 1)
| an equation
---
consists of
1 to 5
or 11 to 15
”0”s or ”1”s.
(¬ 0 & 1 — 0 is 1)
|
Table 4: Examples of training data, In-distribution test data, three kinds of
cross-distribution tests. Note that the training data and in-distribution test
share the same distribution.

##  Appendix A Correct Cases of Cross-Distribution Experiments

###  A.1 Subtraction

1101-974=127
---
1070-955=115
1069-959=110
1111-991=120
190-11=179
222-99=123

###  A.2 Addition

75+653=728
---
77+849=926
73+432=505
42+668=710
82+886=968
80+874=954