# NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting
into Gender-Neutral Alternatives

Eva Vanmassenhove  Chris Emmery  Dimitar Shterionov
Department of Cognitive Science and Artificial Intelligence
Tilburg University
The Netherlands
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recent years have seen an increasing need for gender-neutral and inclusive
language. Within the field of NLP, there are various mono- and bilingual use
cases where gender inclusive language is appropriate, if not preferred due to
ambiguity or uncertainty in terms of the gender of referents. In this work, we
present a rule-based and a neural approach to gender-neutral rewriting for
English along with manually curated synthetic data (WinoBias+) and natural
data (OpenSubtitles and Reddit) benchmarks. A detailed manual and automatic
evaluation highlights how our NeuTral Rewriter, trained on data generated by
the rule-based approach, obtains word error rates (WER) below 0.18% on
synthetic, in-domain and out-domain test sets.

##  1 Introduction

Recent years have seen an increasing need for gender-neutral and inclusive
language. This need is reflected, among others, by a surge in the use of
singular they,111The pronoun ‘they’ was announced word of the year in 2019
according to Merriam Webster https://www.nytimes.com/2019/12/10/us/merriam-
webster-they-word-year.html currently endorsed as part of APA style as the
generic and gender-neutral pronoun. 222https://apastyle.apa.org/ Within the
field of Natural Language Processing (NLP), there are various monolingual and
bilingual use cases where gender neutral and inclusive language is
appropriate, if not preferred due to e.g. ambiguity in terms of the gender of
referents. Section 3 provides a short outline of potential NLP use cases.

To support these use cases, we present a rule-based and a neural approach to
gender-neutral rewriting along with manually curated benchmarks, both of which
we provide open-access/source.333https://github.com/anonymous-until-
publication/NeuTralRewriter First, a rule-based rewriter is implemented
leveraging hand-written rules and an automatic error correction tool. Next, a
neural rewriter is trained on output generated by the rule-based rewriter to
remove the need for extensive pre-processing and the reliance on
computationally expensive tools such as dependency parsers. Our manual and
automatic evaluation show how the neural rewriter clearly improves over the
rule-based approach with word error rates (WER) below 0.18% on synthetic, in-
domain and out-domain test sets.

The main contributions of our work can be summarized as follows: (i)
WinoBias+, an open-source manually curated extension of WinoBias (Zhao et al.,
2018a) providing neutral alternatives for 3,167 sentences as well as a
manually curated set of 1,000 natural sentences (domain: Reddit,
OpenSubtitles), (ii) open-source code for rule-based and neural neutral
rewriters which can convert (binary) gendered English sentences into their
gender neutral counterparts, (iii) a detailed manual and automatic evaluation
of errors made by the rule-based and neutral rewriter on synthetic and natural
data.

##  2 Related Work

Recent years have seen an increase in research on gender and gender bias
mitigation in NLP. While a relatively large body of research has focused on
debiasing word embeddings (e.g., Bolukbasi et al., 2016; Font and Costa-jussà,
2019; Zhao et al., 2018c), our work is related to the generation of gender
variants. We broadly distinguish between: (i) approaches that incorporate
additional (meta-) information during training/testing allowing for a
controlled generation of gender alternatives, and (ii) approaches that focus
on gender rewriting. The synopsis will focus specifically on research related
to the gender of human referents.

Within the field of Machine Translation (MT), Vanmassenhove and Hardmeier
(2018); Vanmassenhove et al. (2019), and Basta et al. (2020) incorporate meta-
information in the form of gender tags on the source side to enable gender
alternative target translations for ambiguous source sentences. Moryossef et
al. (2019) propose a black-box approach by appending gender information to the
target sentences using parataxis constructions at translation time. Bau et al.
(2019) describe work on controlling linguistic features (a.o. gender) in
Neural MT by identifying and (de)activating the relevant neurons. They show
that gender is the most difficult feature to control with a success rate of
21% using the top five identified neurons.

Lu et al. (2020) uses a Counterfactual Data Augmentation (CDA) technique to
augment data sets by creating gender alternative sentences to decrease gender
bias. Their approach consists of swapping gendered words with their
male/female counterparts (e.g. he$:$she, father$:$mother…). Their results
indicate that a CDA approach outperforms a simple word embedding debiasing
technique Bolukbasi et al. (2016). Habash et al. (2019) and Alhafni et al.
(2020) present gender-aware reinflection models for Arabic. Using an Arabic
sentence and a target gender, the desired gender alternative is generated by
re-inflecting the input.

It is worth noting that all the previously described approaches focus on
generating binary (female/male) gendered alternatives or translations, while
our work focuses on generating gender-neutral alternatives. As such, the work
that is most closely related to ours is Sun et al. (2021). Their work is
contemporaneous to our submission.444Currently in $arxiv$ pre-print. Sun et
al. (2021) present a rule-based and neural rewriter for the generation of
gender-neutral singular they sentences as well as an evaluation benchmark555We
contacted the authors to obtain their benchmark for comparison as it is
currently not open-source, but have not been able to obtain it yet. We will
nevertheless attempt to compare our result to theirs to the best of our
ability. of 500 parallel sentences (gendered and gender-neutral) from five
domains (Twitter, Reddit, movie quotes, jokes). Their rule-based and neural
rewriters are able to generate gender-neutral sentences with an error-rate
below 1% (0.63% and 0.99% respectively). In terms of resources, compared to
Sun et al. (2021), we provide larger synthetic and natural benchmarks. In
terms of performance, although complicated due to the lack of a publicly
available benchmark, our models are seemingly better with error-rates of 0.52
(rule-based) and 0.02 (neural) on the most comparable benchmark, i.e. Reddit
data.

##  3 Use Cases

Generating neutral alternatives for gendered sentences has applications for
various monolingual language generation tasks (e.g. automatic responses),
where (i) one does not want to assume the gender of the referents, or (ii) one
wants to present the user with various options. Similarly, in a bilingual
setting, more specifically for MT, a neutral rewriter allows for the
generation of gender neutral alternatives for genderless and gender-neutral
source languages (Hungarian, Turkish, Persian, Swahili…) or null-subject
source languages (Spanish, Chinese, Arabic, Bulgarian…). For illustration,
Example ( (1)) and ( (2)) demonstrate how gender-neutral alternatives can be
useful in bilingual settings. Example ( (1)) features a sentence in Armenian
using the epicene (gender-neutral) pronoun ‘Նա’ which can be either translated
into ‘he’, ‘she’ or singular ‘they’.

  * (1)

HY: Նա բացեց դուռը
EN: He/She/They opened the door.666The translation in bold is the only one
provided by Bing and Google Translate consulted on May 4, 2021.

Similarly, Example ( (2)) illustrates the possible translations of a null-
subject source in Spanish which can be translated as "works in a company".

  * (2)

ES: Trabaja en una empresa.
EN: He/She works in a company.777The translation in bold is the only one
provided by Bing, Google Translate and DeepL consulted on May 4, 2021.
EN: They work in a company.

As a pre-processing step, rewriting into neutral alternatives could be useful
to debias training data and thereby its embeddings (see a.o., Bolukbasi et
al., 2016; Li et al., 2018; Gonen and Goldberg, 2019) and/or to obfuscate
sensitive ‘gender’ features from real user data facing automatic profiling
systems Reddy and Knight (2016); Shetty et al. (2018); Emmery et al. (2021).

##  4 Methodology & Experimental Setup

###  4.1 Datasets

All data is preprocessed using the Moses (de)tokenizer Koehn et al. (2007).
Training (Reddit) and test sets (WinoBias+, OpenSubtitles, Reddit) contain a
balanced amount of the eight (binary) target pronouns/determiners: he, she,
her(s), his, him, him/herself.888For a set containing $X$ sentences, we
extracted at least $X/8$ sentences containing each form - a completely uniform
distribution was not achievable due to the fact that multiple
pronouns/determiners can be present in a single sentence.

##### Reddit

A set of 2,259,386 sentences (containing a total of 3M pronouns/determiners)
was randomly sampled from Pushshift’s Reddit snapshots (Baumgartner et al.,
2020, including all subreddits) for the period of July–December 2019. This set
we would later use for training our neural rewriter. Another set of 1,693
sentences (containing a total of 2K pronouns/determiners) extracted from
Reddit in the same way would later be used as a development set. There are no
overlaps between the two sets.

##### WinoBias+

an extension of the WinoBias benchmark, providing (manual) neutral
alternatives for its 3,167 synthetic sentences, and corrections (e.g. for
ungrammatical sentences999For example, the original WinoBias sentence “The
laborer handed the application to the editor because she want the job.” is
corrected into “The laborer handed the application to the editor because she
wanted the job.”) of the original dataset.

##### OpenSubtitles, Reddit test

additional sets of 1,000 (manually corrected) parallel sentences (500 for each
set). The entire cleaned and extended version of the corpus—WinoBias+— the
OpenSubtitles Lison and Tiedemann (2016), and Reddit benchmark is made
publicly available under a CC BY-SA
4.0101010https://creativecommons.org/licenses/by-sa/4.0/
license.111111https://github.com/vnmssnhv/NeuTralRewritter

###  4.2 Rule-Based Rewriter

The rule-based rewriter (RBR), consists of two main components: (i) a rule-
based pronoun rewriter, and (ii) an error-correction language model.

####  4.2.1 Rule-Based Pronoun Rewriter

Table 1 gives an overview of the binary forms and their gender-neutral
alternatives. While most mappings are one-to-one, ‘her’ can be either a
pronoun (e.g. ‘I gave it to her.’ $\rightarrow$ ‘I gave it to them.’) or a
possessive determiner (e.g. ‘It is her book.’ $\rightarrow$ ‘It is their
book’) and ‘his’ can be either a possessive determiner (‘It is his book.’
$\rightarrow$ ‘It is their book’) or an independent possessive pronoun (‘The
book is his.’ $\rightarrow$ ‘The book is theirs’). To disambiguate these
forms, the POS tagger and dependency parser from Stanza Qi et al. (2020) were
used.121212The ‘his’ ambiguity can only be resolved using a dependency parser
since the xpos and upos tags do not differ when ‘his’ is used as a independent
or dependent possessive.

binary | $\rightarrow$ | gender-neutral
---|---|---
he, she | $\rightarrow$ | they
him | $\rightarrow$ | them
her | $\rightarrow$ | them, their
his | $\rightarrow$ | their, theirs
hers | $\rightarrow$ | theirs
him/herself | $\rightarrow$ | themselves131313‘Themselves’ is preferred over ‘themself’ according the APA guidelines: https://apastyle.apa.org/style-grammar-guidelines/grammar/singular-they
Table 1: Mapping binary pronouns/determiners to their gender-neutral
alternatives.

Following the guidelines from the European Parliament for gender neutral
language
141414https://www.europarl.europa.eu/cmsdata/151780/GNL_Guidelines_EN.pdf, we
provide an option to change gendered English animate nouns (‘chair(wo)man’
$\rightarrow$ ‘chairperson’, ‘bar(wo)man’ $\rightarrow$ ‘bartender’…),
unnecessary feminine forms of animate nouns (e.g. ‘actress’ $\rightarrow$
‘actor’, ‘heroine’ $\rightarrow$ ‘hero’…), and generic uses of ‘man’ (e.g.
‘freshman’$\rightarrow$‘first-year student’, ‘man-made’ $\rightarrow$ ‘human-
made’…).151515The complete list of nouns included can be found in the
appendix.

####  4.2.2 Subject-Verb Agreement Correction

3rd person | $\rightarrow$ | plural
---|---|---
works | $\rightarrow$ | work
has | $\rightarrow$ | have
is | $\rightarrow$ | are
Table 2: Subject-verb agreement correction examples.

The nominative pronouns (he and she) can be replaced by they. However, if they
are in agreement with a simple present tense verb (or the verb ‘to be’ ) the
3rd person form/ending should be replaced by a plural one (see Table 2). To
address this, we used a Python wrapper for LanguageTool, an open-source
grammar, style and spell corrector. 161616 https://pypi.org/project/language-
tool-python/ We limited the correction to grammar mistakes to avoid additional
changes (e.g. insertion of commas, different word choices, removal of
whitespaces…).

###  4.3 Neural Rewriter

We trained a Transformer model Vaswani et al. (2017) using fairseq Ott et al.
(2019)—following the setup of Sun et al. (2021) for comparison. For training
we used the 2,259,386 Reddit sentences as source and their gender-neutral
alternatives as target; for validation we used the 1,693 Reddit sentences and
their neutral alternatives (see Section 4.1). The gender-neutral alternatives,
i.e. the target sides, are generated by applying the RBR on the original
dataset. All hyperparameters and their values are listed in the Appendix along
with the preprocessing and training commands and options.

##  5 Results & Discussion

Both rewriters were (manually) evaluated on synthetic (WinoBias+) and natural
(Reddit, OpenSubs) evaluation benchmarks.

###  5.1 Manual Evaluation

Table 3 presents a detailed overview of the errors per test set for the Rule-
Based and Neural approach. An overview and explanation of all error labels can
be found in the Appendix.

##### Rule-Based Approach

The errors can be divided broadly into “language model” (LM), “postag” (POS)
and “other” errors. WinoBias+ consists of 3167 sentences. Only 21 of the
synthetic sentences were rewritten incorrectly. Issues arose either due to
incorrect disambiguation (‘her’ $\rightarrow$ ‘them’ (pronoun) instead of
‘their’ (determiner)) or due to incorrect subject-verb agreement (SVA).

The RBR struggled more with the noisy, often ungrammatical, natural data from
OpenSubtitles and Reddit. The main issues observed are incorrect SVA,
additional corrections by the language tool (unrelated to gender neutrality,
e.g. cause $\rightarrow$ because) and incorrect disambiguation of
"‘s".171717e.g. He’s worked. $\rightarrow$ They are worked. instead of They
have worked.

Error Classes | Rule-Based | Neural
---|---|---
WB$+$ | OpenS | Red | WB$+$ | OpenS | Red
LM | SVA | 9 | 16 | 11 | 0 | 5 | 0
corr. | 0 | 0 | 11 | 0 | 0 | 0
‘s (has) | 0 | 1 | 7 | 0 | 1 | 0
space | 0 | 0 | 3 | 0 | 0 | 4
other | 0 | 0 | 3 | 0 | 0 | 0
POS | error | 12 | 0 | 3 | 0 | 0 | 0
source | 0 | 0 | 2 | 0 | 0 | 0
OTH. | cap. | 0 | 4 | 2 | 0 | 1 | 0
ungram. | 0 | 2 | 0 | 0 | 0 | 0
| rule | 0 | 1 | 1 | 0 | 1 | 0
| UNK | 0 | 0 | 0 | 0 | 0 | 2
# of errors | 21 | 24 | 43 | 0 | 8 | 6
Table 3: Error classification and counts on the WinoBias+, OpenSubtitles and
Reddit test set for the Rule-Based and Neural approach.

##### Neural Approach

Interestingly, and in contrast with the findings described in Sun et al.
(2021), our neural model trained on the rule-based generated training data,
outperforms the rule-based approach. The error analysis reveals that the
neural model resolves many of the longer distance SVA issues, the
disambiguation of “’s” and errors that occurred due to incorrect postags.

No errors were made on the synthetic WinoBias+ data. Errors on the in-domain
Reddit data were due to the removal of additional spaces (4 errors) or because
of an unknown character/emoji (2 errors). On the out-of-domain OpenSubtitles
set, we noted 8 errors the majority of which due to incorrect SVA (5 errors).

###  5.2 Automatic Evaluation

For comparison, we employed the same metric as Sun et al. (2021): WER. A
combination of the baseline WER (indicating the amount of changes needed in
order to change to gender-neutral alternatives), and the WER computed between
the correct neutral forms and the automatically generated forms provides
insights into the performance of both approaches.

WER (%) | WB$+$ | OpenS | Reddit | Sun et al. (2021)
---|---|---|---|---
BASE | 8.76 | 14.09 | 11.02 | 12.40
RBR | 0.06 | 0.45 | 0.52 | 0.63
NR | 0.00 | 0.18 | 0.02 | 0.99
Table 4: WER on the synthetic WinoBias+ (WB$+$) test set and natural Reddit
and OpenSubtitles benchmark vs WER obtained by Sun et al. (2021).

Given that Sun et al. (2021) use an evaluation benchmark of 500 sentences
consisting of Twitter, Reddit, jokes and movie quotes data, its performance is
probably most comparable to the scores we obtained on the Reddit set. Like the
manual evaluation, and in contrast with Sun et al. (2021), the automatic
evaluation (Table 4) confirms that our neural approach is able to generalize
over the rule-based generated data, outperforming it with error rates below
0.18% (0.0% (WB+), 0.18% (OpenSubtitles) and 0.02% (Reddit)). Furthermore,
these error rates are all substantially lower than those reported by Sun et
al. (2021). We hypothesize this is due to the better performance of the RBR
(confirmed as well by the automatic/manual evaluation) leading to better
source (gendered)–target (neutral) training data for the NMT model.

We ought to note that WER does not take into account the removal of
superfluous spaces (e.g. before the first character of a sentence, double
spaces instead of a single one). We only observed the removal of such spaces
by the neural rewriter on the Reddit data (see detailed manual analysis
presented in Table 3).

##  6 Conclusion

This paper presents a rule-based and a neural gender-neutral rewriter for
English. First, the rule-based approach was implemented, leveraging hand-
written rules and an automatic error correction tool. Using the RBR, we
generated a parallel gendered-to-neutral corpus on which an NMT system was
trained. The NMT model removes the need for computationally expensive pre-
processing steps and, according to the manual and automatic evaluation,
outperforms the RBR on synthetic, in-domain and out-domain benchmarks. Along
with our open-access/source code, we also provide three manually curated
benchmarks for neutral rewriting.

For now, the neutral rewriter is limited to English using ‘singular they’ and
recommendations for gender neutral writing specific to the English language.
It is, in theory, possible to extend this approach (or a similar one) to other
languages. However, so far, few languages have a crystallized approach when it
comes to gender-neutral pronouns and gender-neutral word endings.

In future work, we intend to explore potential applications of the neutral
rewriters (e.g. gender debiasing of corpora). We furthermore plan to extend
our work to gender-neutral rewriting targeting specific referents within a
sentence to accommodate the gender preferences of individual referents.

## Ethics statement

##### Neutral Rewriter Application

The Neutral Rewriter is intended to provide gender-neutral alternatives and
increase the inclusiveness of NLP/MT applications. The rewriter can
furthermore be used as a preprocessing step to obfuscate a potentially
sensitive gender attribute from training data.

At this stage, the rewriter works on a sentence-level and does not allow for
rewriting pronouns or determiners of specific referents. We followed the
guidelines of the European Parliament for gender neutral language and provide
an option to change gendered animate nouns, unnecessary feminine forms of
animate nouns and generic uses of the word ‘man’ based on non-exhaustive word
lists.

##### Datasets

We present three openly available English benchmarks: (i) WinoBias+, (ii)
OpenSubtitles and (iii) Reddit. (i) WinoBias+ consists of a curated and
extended version of the synthetic WinoBias Zhao et al. (2018b) dataset,
distributed under the MIT License.181818https://opensource.org/licenses/MIT
(ii) The open-source OpenSubtitles Lison and Tiedemann
(2016)191919http://www.opensubtitles.org/ data was used to randomly sample a
subset for the OpenSubtitles benchmark. OpenSubtitles is distributed under a
Creative Commons license.202020Attribution-Non Commercial 4.0 International
(iii) The Reddit dataset was collected through the third-party snapshots of
Reddit’s publicly available API at https://pushshift.io. It is subject to
Reddit’s own User Agreement and Privacy Policy and covers the _free and public
sharing of user data_.212121See https://www.redditinc.com/policies/user-
agreement and https://www.redditinc.com/policies/privacy-policy respectively.

The neutral alternatives for the three benchmarks were manually created by a
linguist. The curation rationale behind the selected datasets is summarized as
follows. WinoBias was selected as it is one of the few benchmarks for gender
bias in NLP. We extended it with gender-neutral alternatives. The natural
Reddit and OpenSubtitles dataset allowed us to verify the robustness of the
rewriters on more noisy and diverse data sets. The OpenSubtitles and Reddit
datasets contain variety in terms of language and English social dialects.
Training and test sets contain a balanced amount of the eight (binary) target
pronouns/determiners. For a set containing $X$ sentences, we extracted at
least $X/8$ sentences containing each form - a completely uniform distribution
was not achievable due to the fact that multiple pronouns/determiners can be
present in a single sentence.

##### Carbon statement

The neural model presented in this work has an ecological footprint equivalent
to 1.68kg of CO2 emissions.222222Contribution based on GPU power consumption
at training the NeuTral rewriter model. The training time, consumption and
carbon emission can be found in Table 5.

Elapsed | Avg. power | kWh | CO2
---|---|---|---
time (h) | draw |  | (kg)
6.2 | 147.37 | 2.64 | 1.68 $\pm$0.13
Table 5: Train time, consumption and carbon emissions related to the training
of the NeuTral rewriter.

## Acknowledgements

We would like to thank the reviewers for their insightful feedback and
comments.

## References

  * Alhafni et al. (2020) Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2020.  Gender-aware reinflection using linguistically enhanced neural models.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pages 139–150.
  * Basta et al. (2020) Christine Basta, Marta R Costa-jussà, and Noe Casas. 2020.  Extensive study on the underlying gender bias in contextualized word embeddings.  _Neural Computing and Applications_ , pages 1–14.
  * Bau et al. (2019) Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019.  Identifying and Controlling Important Neurons in Neural Machine Translation.  In _Proceedings of the Seventh International Conference on Learning Representations (ICLR)_ , New Orleans, USA.
  * Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020.  The pushshift reddit dataset.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 830–839.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.  In _Proceedings of Thirtieth Conference on Neural Information Processing Systems (NIPS)_ , pages 4349–4357, Barcelona, Spain.
  * Emmery et al. (2021) Chris Emmery, Ákos Kádár, and Grzegorz Chrupała. 2021.  Adversarial stylometry in the wild: Transferable lexical substitution attacks on author profiling.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2388–2402.
  * Font and Costa-jussà (2019) Joel Escudé Font and Marta R Costa-jussà. 2019.  Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques.  In __To appear in_ Proceedings of the 1st ACL Workshop on Gender Bias for Natural Language Processing_, Florence, Italy.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 609–614.
  * Habash et al. (2019) Nizar Habash, Houda Bouamor, and Christine Chung. 2019.  Automatic gender identification and reinflection in arabic.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 155–165.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions_ , pages 177–180.
  * Li et al. (2018) Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018.  Towards robust and privacy-preserving text representations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 25–30.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929.
  * Lu et al. (2020) Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020\.  Gender bias in neural natural language processing.  In _Logic, Language, and Security_ , pages 189–202. Springer.
  * Moryossef et al. (2019) Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019.  Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection.  In __TO APPEAR IN_ 1st ACL Workshop on Gender Bias for Natural Language Processing_, Florence, Italy.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108.
  * Reddy and Knight (2016) Sravana Reddy and Kevin Knight. 2016.  Obfuscating gender in social media writing.  In _Proceedings of the First Workshop on NLP and Computational Social Science_ , pages 17–26.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Shetty et al. (2018) Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018.  A4nt: author attribute anonymity by adversarial training of neural machine translation.  In _27th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 18)_, pages 1633–1650.
  * Sun et al. (2021) Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021\.  They, them, theirs: Rewriting with gender-neutral english.  _arXiv preprint arXiv:2102.06788_.
  * Vanmassenhove and Hardmeier (2018) Eva Vanmassenhove and Christian Hardmeier. 2018.  Europarl datasets with demographic speaker information.  In _Proceedings of the 21st Annual Conference of the European Associations for Machine Translation (EAMT)_ , Alicante, Spain.
  * Vanmassenhove et al. (2019) Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2019.  Getting gender right in neural machine translation.  _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3003–3008.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Zhao et al. (2018a) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20.
  * Zhao et al. (2018b) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018b.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhao et al. (2018c) Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018c.  Learning Gender-Neutral Word Embeddings.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4847–4853, Brussels, Belgium.

##  Appendix A Appendix

The appendix provides additional information on generation of gender-neutral
alternatives (Section A.1), the error labels and analysis (Section A.2) and
the training hyperparameters of the Neural Machine Translation model (Section
A.3.1).

###  A.1 Advanced Rewriter

The advanced rewriter includes rewriting of gender-marked job titles
(chairman, anchorman…), rewriting of unnecessary feminine forms (actress,
comedienne, waitress…), avoidance of construction using a generic form of
‘man’ (‘average man’, ’man and wife’…), and rewriting of titles (‘Mrs’ and
‘Miss’).

####  A.1.1 Gender-neutral alternatives for gender-marked job titles

chairman/woman | businessman/woman
---|---
chairman | $\rightarrow$ | chairperson | businessman | $\rightarrow$ | business person
chairmen | $\rightarrow$ | chairpeople | businessmen | $\rightarrow$ | business people
chairwoman | $\rightarrow$ | chairperson | businesswoman | $\rightarrow$ | business person
chairwomen | $\rightarrow$ | chairpeople | businesswomen | $\rightarrow$ | business people
anchorman/woman | postman/postwoman
anchorman | $\rightarrow$ | anchor | postman | $\rightarrow$ | mail carrier
anchormen | $\rightarrow$ | anchors | postmen | $\rightarrow$ | mail carriers
anchorwoman | $\rightarrow$ | anchor | postwoman | $\rightarrow$ | mail carrier
anchorwomen | $\rightarrow$ | anchors | postwomen | $\rightarrow$ | mail carriers
congresswoman/congressman | mailman/mailwoman
congressman | $\rightarrow$ | member of congress | mailman | $\rightarrow$ | mail carrier
congressmen | $\rightarrow$ | members of congress | mailmen | $\rightarrow$ | mail carriers
congresswoman | $\rightarrow$ | member of congress | mailwoman | $\rightarrow$ | mail carrier
congresswomen | $\rightarrow$ | members of congress | mailwomen | $\rightarrow$ | mail carriers
policeman/policewoman | salesman/saleswoman
policeman | $\rightarrow$ | police officer | salesman | $\rightarrow$ | salesperson
policemen | $\rightarrow$ | police officers | salesmen | $\rightarrow$ | salespersons
policewoman | $\rightarrow$ | police officer | saleswoman | $\rightarrow$ | salesperson
policewomen | $\rightarrow$ | police officers | saleswomen | $\rightarrow$ | salespersons
spokesman/woman | fireman/firewoman
spokesman | $\rightarrow$ | spokesperson | fireman | $\rightarrow$ | firefighter
spokesmen | $\rightarrow$ | spokespersons | firemen | $\rightarrow$ | firefighters
spokeswoman | $\rightarrow$ | spokesperson | firewoman | $\rightarrow$ | firefighter
spokeswomen | $\rightarrow$ | spokespersons | firewomen | $\rightarrow$ | firefighters
steward/stewardess | barman/barwoman
steward | $\rightarrow$ | flight attendant | barman | $\rightarrow$ | bartender
stewards | $\rightarrow$ | flight attendants | barmen | $\rightarrow$ | bartenders
stewardess | $\rightarrow$ | flight attendant | barwoman | $\rightarrow$ | bartender
stewardesses | $\rightarrow$ | flight attendants | barwomen | $\rightarrow$ | bartenders
headmaster/mistress | cleaning man/lady
headmaster | $\rightarrow$ | principal | cleaning man | $\rightarrow$ | cleaner
headmasters | $\rightarrow$ | principals | cleaning lady | $\rightarrow$ | cleaners
headmistress | $\rightarrow$ | principal | cleaning men | $\rightarrow$ | cleaner
headmistresses | $\rightarrow$ | principals | cleaning ladies | $\rightarrow$ | cleaners
foreman/forewoman |
foreman | $\rightarrow$ | supervisor |  |  |
foremen | $\rightarrow$ | supervisors |  |  |
forewoman | $\rightarrow$ | supervisor |  |  |
forewomen | $\rightarrow$ | supervisors |  |  |

Table 6: Gender-neutral alternatives for gender-marked job titles

####  A.1.2 Gender-neutral alternatives for unnecessary feminine forms

actress | usherette
---|---
actress | $\rightarrow$ | actor | usherette | $\rightarrow$ | usher
actresses | $\rightarrow$ | actors | usherettes | $\rightarrow$ | usher
heroine | authoress
heroine | $\rightarrow$ | hero | authoress | $\rightarrow$ | author
heroine | $\rightarrow$ | heroes | authoresses | $\rightarrow$ | authors
comedienne | mailman/mailwoman
comedienne | $\rightarrow$ | comedian | mailman | $\rightarrow$ | mail carrier
comediennes | $\rightarrow$ | comedians | mailwomen | $\rightarrow$ | mail carriers
executrix | boss lady
executrix | $\rightarrow$ | executor | boss lady | $\rightarrow$ | boss
executrices | $\rightarrow$ | executors | boss ladies | $\rightarrow$ | boss
executrixes | $\rightarrow$ | executor |  |  |
poetess | waitress
poetess | $\rightarrow$ | poet | waitress | $\rightarrow$ | waiter
poetesses | $\rightarrow$ | poets | waitresses | $\rightarrow$ | waiters

Table 7: Gender-neutral alternatives for unnecessary feminine forms

####  A.1.3 Gender-neutral alternatives for generic ‘man’

average man | layman
---|---
average man | $\rightarrow$ | average person | layman | $\rightarrow$ | layperson
average men | $\rightarrow$ | average people | laymen | $\rightarrow$ | laypeople
best man for the job | freshman
best man for the job | $\rightarrow$ | best person for the job | freshman | $\rightarrow$ | first-year student
best men for the job | $\rightarrow$ | best people for the job | freshmen | $\rightarrow$ | first-year students
mankind | man-made
mankind | $\rightarrow$ | humankind | man-made | $\rightarrow$ | human-made
workmanlike | man and wife
workmanline | $\rightarrow$ | skillful | man and wife | $\rightarrow$ | husband and wife

Table 8: Gender-neutral alternatives for generic ‘man’

###  A.2 Overview Error Analysis

####  A.2.1 Error Classification Rewriter

As explained in the paper, errors are divided into Language Model (LM) errors,
postag error (POS) and other errors (OTHER). Within these three error classes,
we identified multiple subclasses of LM, POS and OTHER errors. An explanation
of the labels used in our error analysis and paper can be found in Table 9.
Table 10 provides example input and output sentences.

Error Label |  Explanation
---|---
LM ’s |  Wrongly disambiguated the contracted form ‘s as a verb form of ‘to be’ instead of ‘to have’
LM space |  Space added or removed by rewriter
LM correction (corr.) |  Error correction done by rewriter (language tool) that is not related to gender-neutral rewriting
LM subject-verb agreement (SVA) |  Failure to make correct subject-verb agreement, usually due to long distance dependencies.
POS |  Wrong form of ‘they’ produced by rewriter due to incorrect postag
POS (source) |  Wrong form of ‘they’ produced by rewriter due to incorrect postag which is related to an ungrammatical/incorrect soure sentences
OTHER rule |  Some forms such as ‘hisn’s’ are not standard language and does not covered by our rules. Similarly written forms such as ‘hes’ for ‘he’s’ are not corrected by the rewriter
Other ungram. |  Ungrammatical input sentence leading to an ungrammatical output
Other UNK |  The Neural Rewriter outputs <UNK> for unknown characters (in our case "?", "!", "…", and emojis/special characters that did not appear in our Reddit training data)
Table 9: Error label explanation

Error Label |  Example | $\rightarrow$ |  Output RBR
---|---|---|---
LM (‘s) |  He‘s worked hard | $\rightarrow$ |  They are worked hard.
LM (space) |  … aren ’t… | $\rightarrow$ |  …aren’t…
LM (corr.) |  Bit pricey… | $\rightarrow$ |  A bit pricey…
LM (SVA) |  He works and works… | $\rightarrow$ |  They work and works…
POS |  He saw her run fast… | $\rightarrow$ |  They saw their run fast…
POS (source) |  …looked at her weird (she ‘s close.. | $\rightarrow$ |  … looked at their weird (they are close…
Basic rule |  She’s hisn‘s.. | $\rightarrow$ |  .They are hisn’s
Other |  Where’s herself. | $\rightarrow$ |  Where’s themselves.

Table 10: Examples Error Labels

###  A.3 Neural Rewriter

Our neural model is trained with the following options: `transformer-iwslt-en-
de` architecture with 4 attention heads and encoder and decoder embedding
dimensions equal to 512, encoder and decoder embedding dimensions for the FFN
equal to 1024, Adam learning optimizer Kingma and Ba (2015) with a learning
rate of 0.005 and inverse square-root schedule with 4 000 warmup steps, an
early stopping based on the improvement on the validation set with patience 5,
dropout of 0.3, joint byte-pair encoding Sennrich et al. (2016) with 32 000
operations, token-based batches with maximum size of 4096. For ease of
replicability we provide our complete preprocessing and training scripts in
Appendix.

####  A.3.1 Training Hyperparameters

    fairseq-preprocess --source-lang $SRC \
      --target-lang $TRG \
      --trainpref $ENGDIR/data/train.tc.bpe \
      --validpref $ENGDIR/data/dev.tc.bpe \
      --testpref $ENGDIR/data/test.tc.bpe \
      --destdir $ENGDIR/data/ready_to_train

    fairseq-train $ENGDIR/data/train_data \
      --arch transformer_iwslt_de_en \
      --lr 0.0005 --optimizer adam \
      --adam-betas ’(0.9, 0.98)’ \
      --max-tokens 4096 \
      --dropout 0.3 \
      --update-freq=1 \
      --lr-scheduler inverse_sqrt \
      --warmup-init-lr 1e-07 --min-lr 1e-09 \
      --warmup-updates 4000 \
      --save-dir $ENGDIR/model \
      --skip-invalid-size-inputs-valid-test \
      --patience 5

With `$ENGDIR` we indicate the path where the data folder and the model folder
are located.


11institutetext: Karlsruhe Institute of Technology (KIT), Institute for
Nuclear Physics, Postfach 3640, 76021 Karlsruhe, Germany 22institutetext:
Observatorio Pierre Auger, Av. San Martín Norte 304, 5613 Malargüe, Argentina
33institutetext: also at: Astrophysical Institute, Vrije Universiteit Brussel,
Pleinlaan 2, 1050 Brussels, Belgium

# Radio detection of cosmic rays with the Auger Engineering Radio Array

Tim Huege on behalf of the Pierre Auger Collaboration Full author list at
http://www.auger.org/archive/authors_2018_10.html1133<EMAIL_ADDRESS>22

###### Abstract

The Auger Engineering Radio Array (AERA) complements the Pierre Auger
Observatory with 150 radio-antenna stations measuring in the frequency range
from 30 to 80 MHz. With an instrumented area of 17 km2, the array constitutes
the largest cosmic-ray radio detector built to date, allowing us to do multi-
hybrid measurements of cosmic rays in the energy range of 1017 eV up to
several 1018 eV. We give an overview of AERA results and discuss the
significance of radio detection for the validation of the energy scale of
cosmic-ray detectors as well as for mass-composition measurements.

##  1 Introduction

The Pierre Auger Observatory AugerNIM2014  exploits hybrid detection of
extensive air showers measured at the same time with multiple detection
techniques for optimum determination of the energy and mass of cosmic ray
primaries. Following this spirit, its baseline surface detector (SD) and
fluorescence detectors (FD) have been complemented by an array of radio
detectors. This Auger Engineering Radio Array (AERA) was deployed in the
region where many enhancements, focused in particular at the detection of air
showers with energies in the EeV energy regime, have been set up.

In this article, we briefly review the characteristics and capabilities of the
radio detectors of the Pierre Auger Observatory and discuss how they
contribute valuable information in particular for the determination of the
energy scale of cosmic rays as well as mass composition studies.

##  2 The Auger Engineering Radio Array

When work on AERA began in 2008, the nature and characteristics of the radio
emission from extensive air showers were still largely unclear — unlike today,
where the emission physics and signal characteristics are very-well understood
HuegePLREP ; SchroederReview . Furthermore, many questions regarding the best
strategy for the measurement of these pulsed radio signals were still open.
The goal of AERA, illustrated by the explicit mentioning of “engineering” in
its name, was to explore different options for implementing a radio detector
that can be scaled to areas significantly larger than a km2.

These efforts resulted in the setup of a 17 km2 array of radio detectors. The
individual detector stations are _autonomous_ and can thus be freely spaced.
Many of the experimental challenges, for example related to power supply,
communications and data acquisition, were indeed related to this autonomous
design, which is a key and unique feature of AERA that differentiates it from
purely cabled setups for radio detection of cosmic rays.

Figure 1: Map of the Auger Engineering Radio Array. 150 radio-detector
stations cover a total area of 17 km2. Further explanations are given in the
text.

The layout of the array is depicted in Figure 1. The radio antennas are co-
located with the 750 m surface detector array, the Coihueco and HEAT
fluorescence telescopes, and the AMIGA underground muon detectors. This
combination of detectors at the same location maximizes the information
gathered on each individual air shower by multi-hybrid detection.

The antennas have been set up on a graded array, with grid constants of 144,
250, 375 and 750 meters, probing different energy and zenith angle ranges. Two
different types of antennas, Logarithmic Periodic Dipole Antennas (LPDAs)
AERAAntennaPaper2012 ; AERALPDA2017  and “butterfly antennas”
AERAAntennaPaper2012  are in use. Both measure in the frequency band from 30
to 80 MHz. Furthermore, two different types of digital electronics, both of
them capable of “internal” triggering on radio signals and one of them, in
addition, allowing data buffering for up to 8 seconds and thus exploitation of
external triggering by the SD and FD, have been developed and deployed. A
reference “beacon” transmitter is used to achieve time synchronization of the
autonomous detectors within two nanoseconds AERAAirplane .

AERA started taking data in 2011 and has been finalized in its current form in
2015. Its current dataset encompasses more than 10,000 extensive air showers
measured with both the SD and the radio antennas. In the following, we give a
concise overview of the most important results derived with AERA up to now.

##  3 Determination of the cosmic-ray energy

One of the key parameters to characterize cosmic-ray primaries is their
energy. With AERA, we have demonstrated that this energy can be determined
with a resolution of 17% or better AERAEnergyPRD . Many different approaches
exist to determine the cosmic-ray energy from radio measurements HuegePLREP .
In AERA, we determine the _energy fluence_ , i.e. the energy deposited in the
form of radio signals per unit area, at the locations of our radio detector
stations. Exploiting our knowledge of the radio-emission pattern, we can then
interpolate and finally integrate the energy fluence to determine the total
energy deposited on the ground in the form of radio signals, the _radiation
energy_.

After correction for the orientation with regard to the geomagnetic field,
this radiation energy is closely correlated with the energy in the
electromagnetic cascade of the air shower, following a quadratic scaling
because of the coherent nature of the radio emission. This is shown in Figure
2. As the radio emission undergoes no significant absorption or scattering in
the atmosphere and constitutes a calorimetric quantity, it is very well suited
for cross-calibration of cosmic-ray detectors via radio measurements
AERAEnergyPRL . Comparing the measured radiation energy to the one predicted
by first-principle Monte Carlo simulations Gottowik:2017wio  also provides the
opportunity for an independent cross check and validation of the energy scale
of the Pierre Auger Observatory EnergyScaleICRC2013 , which is based on
measurements of the fluorescence detectors.

Figure 2: After correction for the angle of incidence with respect to the
geomagnetic field $\alpha$, the energy deposited on the ground in the form of
radio signals, the _radiation energy_ , scales quadratically with the energy
of the primary cosmic ray. With AERA, an energy resolution of 17% has so far
been demonstrated AERAEnergyPRL  for events detected in at least 5 radio-
detector stations with signal.

##  4 Mass composition sensitivity

The depth of maximum $X_{\mathrm{max}}$ of an extensive air shower is a mass-
sensitive parameter that is directly accessible with fluorescence
measurements. Radio signals from air showers also have sensitivity to
$X_{\mathrm{max}}$ due to the forward-beamed nature of the emission as well as
its enhancement on a Cherenkov ring introduced by the non-unity refractive
index of the atmosphere HuegePLREP . Deeper showers thus produce a steeper
signal falloff with lateral distance from the impact point than shallower
showers. In addition, pulse shape and polarization information can be
exploited.

We have demonstrated that by comparison of the radio signals of measured air
showers with per-event Monte Carlo simulations, $X_{\mathrm{max}}$ can be
determined and is in agreement with the values measured for the same showers
with the fluorescence detectors HoltICRC2017 , see Figure 3. The combined FD-
radio resolution is approximately 45 g/cm2, i.e. the radio-only resolution is
of order 35-40 g/cm2. This is likely limited by the sparsity of the radio
array, as measurements with much denser radio arrays have yielded
$X_{\mathrm{max}}$ resolutions below 20 g/cm2 LOFARXmaxMethod2014 .

Figure 3: The depth of shower maximum $X_{\mathrm{max}}$ as determined with
AERA correlates well with the $X_{\mathrm{max}}$ value determined from the
Auger fluorescence detectors HoltICRC2017 . The combined FD-radio resolution
is approximately 45 g/cm2, and the radio-only resolution is of order 35-40
g/cm2.

Another option for mass composition studies with radio antennas lies in their
pure measurement of the energy in the electromagnetic component of an air
shower. Combined with a measurement of the muon content of air showers, for
example by the AMIGA underground muon detectors, this allows mass-composition
studies through a completely different method. This approach has particular
potential for inclined air showers, for which a classical measurement of the
electron number at the ground is no longer feasible (the electromagnetic
cascade dies out before reaching the ground due to the large atmospheric mass
overburden). As radio signals are not significantly attenuated in the
atmosphere, radio detection at these geometries allows the determination of
the energy in the electromagnetic cascade, while particle detectors such as
those of the SD will provide a clean measurement of the muonic cascade. The
sensitivity for mass-composition measurements arising from this
complementarity is illustrated in Figure 4.

Figure 4: A simulation study of the separation power between proton- and iron-
induced air showers shows that combined measurements of the radiation energy,
$S_{\mathrm{RD}}^{\rho_{\theta}}$, and the muon content at a lateral distance
of 600 meters, $\rho_{\mu}^{600}$, provide very good mass composition
sensitivity at all zenith angles HoltARENA2018 . In contrast, combined
measurements of the electron number $N_{e}$ and the muon content lose
separation power towards higher zenith angles, as the electromagnetic cascade
dies out before the shower reaches the ground. Detector effects are not
included in this study.

##  5 Radio measurements of inclined air showers

The radio emission from extensive air showers is strongly forward-beamed. For
showers with near-vertical incidence, for which the shower maximum is located
at a few kilometers above sea level, the emission thus illuminates a rather
small area on the ground, of order hundreds of meters in diameter
HuegeUHECR2014 . The area does not grow significantly with the energy of the
primary particle because of the very steep lateral falloff of the radio signal
and the even deeper depth of maximum for higher particle energies. As a
consequence, radio detector arrays aiming to measure near-vertical air showers
need a dense antenna spacing with a grid constant of at most a few hundred
meters. Scaling such arrays significantly beyond the size of AERA is thus not
cost-effective. For near-vertical showers, the energy reach is therefore
limited to below 1019 eV, simply because the instrumentable area is limited.

Figure 5: With increasing zenith angle, the farthest distance from the shower
axis at which a detectable radio signal was measured by AERA grows
significantly. At the highest zenith angles, detectable radio signals
illuminate areas larger than 100 km2.

In contrast, for inclined air showers, due to the vastly increased atmospheric
mass overburden, the shower maximum is typically dozens of kilometers away
from a ground-based antenna array. Consequently the emission is bound to be
distributed over a much larger area HuegeUHECR2014 . This has recently been
confirmed experimentally with AERA data Aab:2018ytv  as presented in Figure 5:
The distance of the radio detector station farthest from the air-shower axis
which still measured a clear radio pulse increases significantly with the air-
shower zenith angle. At the highest zenith angles, the lateral extent of the
radio-emission footprint can exceed two kilometers in the shower plane.
Projection to the ground then yields areas of more than $2$ km$\,\times\,2$
km$\,\times\sec(83^{\circ})\times\,\pi=103$ km2 illuminated with measurable
radio signals. This allows radio detection of inclined air showers with very
sparse radio arrays. In fact, we have demonstrated that a grid constant of 1.5
km, the same as that of the Auger SD, is sufficient for the detection of
inclined air showers Aab:2018ytv . For inclined air showers, radio detection
then provides a pure measurement of the electromagnetic component of the air
shower, while the SD measurements provide the pure muon content. As described
above, this opens the possibility for mass-composition studies of inclined air
showers.

Motivated by these findings, we will equip each water-Cherenkov detector of
the SD with one radio antenna as part of the ongoing AugerPrime upgrade
Aab:2016vlz . The details of this endeavor are described in a dedicated
article HorandelUHECR2018 .

##  6 Conclusions

The Auger Engineering Radio Array, an array of 150 autonomous radio detector
stations covering a total area of 17 km2, measures extensive air showers in
the energy range of 1017 eV to several 1018 eV via their radio emission in the
30 to 80 MHz band. AERA explored many different approaches for the
experimental design as well as several analysis strategies, and has made
important contributions to leading the radio detection technique from
pioneering prototypes to maturity.

In particular, we have demonstrated that the energy deposited in the form of
radio waves on the ground, the _radiation energy_ , can be measured accurately
and is a reliable estimator for the energy in the electromagnetic cascade of
an extensive air shower. This quantity has a well-defined physical meaning and
can be measured and compared among radio detectors worldwide. As such, it is a
very useful means to cross-calibrate the energy scale of different cosmic-ray
experiments, and can even be used to validate the energy scale on the basis of
first principle calculations.

Mass composition sensitivity is also present in radio signals. We have
successfully reconstructed the depth of shower maximum of air showers from
radio measurements with AERA, and they are in agreement with measurements from
the Auger fluorescence detector. Another way to determine the mass of primary
cosmic rays with the help of radio measurements lies in combining radio
detection with muon measurements, as are provided for near-vertical geometries
by the Auger AMIGA detectors and for inclined air showers by the Auger surface
detector.

Finally, we have established experimentally that inclined air showers
illuminate areas of dozens or even more than a hundred km2 with measurable
radio signals. Hence, arrays with grid constants of a kilometer or larger can
measure inclined air showers with reasonable antenna multiplicity. This allows
radio measurements up to ultra-high energies, and yields very complementary
information to that provided by the water-Cherenkov detectors of the SD.
Consequently, as part of the AugerPrime upgrade, we will deploy a radio
antenna on top of each water-Cherenkov detector to extend the composition-
sensitive measurements of AugerPrime to inclined air showers. At the same
time, we will continue to operate AERA, which is focused at energies around
the EeV scale.

## References

  * (1) A. Aab et al. (Pierre Auger Collaboration), Nucl. Instrum. Meth. A 798, 172 (2015)
  * (2) T. Huege, Physics Reports 620, 1 (2016)
  * (3) F. G. Schröder, Progress in Particle and Nuclear Physics 93, 1 (2017)
  * (4) P. Abreu et al. (Pierre Auger Collaboration), JINST 7, P10011 (2012)
  * (5) A. Aab et al. (Pierre Auger Collaboration), JINST 12, T10005 (2017)
  * (6) A. Aab et al. (Pierre Auger Collaboration), JINST 11, P01018 (2016)
  * (7) A. Aab et al. (Pierre Auger Collaboration), Phys. Rev. D 93, 122005 (2016)
  * (8) A. Aab et al. (Pierre Auger Collaboration), Phys. Rev. Lett. 116, 241101 (2016)
  * (9) M. Gottowik, C. Glaser, T. Huege, J. Rautenberg, Astropart. Phys. 103, 87 (2018)
  * (10) V. Verzi, for the Pierre Auger Collaboration, Proc. 33rd ICRC, Rio de Janeiro, Brazil, id 0928 (2013)
  * (11) E. M. Holt for the Pierre Auger Collaboration, Proc. 35th ICRC, Busan, Korea, PoS(ICRC2017)492
  * (12) S. Buitink et al. (LOFAR CR KSP), Phys. Rev. D 90, 082003 (2014)
  * (13) E. M. Holt for the Pierre Auger Collaboration, Proc. of the 2018 ARENA conference, EPJ WoC in press (2019)
  * (14) T. Huege, A. Haungs, Proc. of the UHECR2014 conference, Springdale, USA, JPS Conf. Proc. 09, 010018 (2016)
  * (15) A. Aab et al. (Pierre Auger Collaboration), JCAP 10, 026 (2018)
  * (16) A. Aab et al. (Pierre Auger Collaboration), ArXiv e-prints, 1604.03637 (2016)
  * (17) J. Hörandel for the Pierre Auger Collaboration, these proceedings