#  Towards Countering Essentialism through Social Bias Reasoning

Emily Allaway1,2 Nina Taneja1 Sarah-Jane Leslie3 Maarten Sap2,4
1Columbia University, USA
2Allen Institute for Artificial Intelligence, USA
3Princeton University, USA
4Carnegie Mellon University
<EMAIL_ADDRESS>
###### Abstract

Essentialist beliefs (i.e., believing that members of the same group are
fundamentally alike) play a central role in social stereotypes and can lead to
harm when left unchallenged. In our work, we conduct exploratory studies into
the task of countering essentialist beliefs (e.g., “liberals are stupid”).
Drawing on prior work from psychology and NLP, we construct five types of
counterstatements and conduct human studies on the effectiveness of these
different strategies. Our studies also investigate the role in choosing a
counterstatement of the level of explicitness with which an essentialist
belief is conveyed. We find that statements that broaden the scope of a
stereotype (e.g., to other groups, as in “conservatives can also be stupid”)
are the most popular countering strategy. We conclude with a discussion of
challenges and open questions for future work in this area (e.g., improving
factuality, studying community-specific variation) and we emphasize the
importance of work at the intersection of NLP and psychology.

Towards Countering Essentialism through Social Bias Reasoning

Emily Allaway1,2 Nina Taneja1 Sarah-Jane Leslie3 Maarten Sap2,4 1Columbia
University, USA 2Allen Institute for Artificial Intelligence, USA 3Princeton
University, USA 4Carnegie Mellon University<EMAIL_ADDRESS>

##  1 Introduction

Essentialism, i.e., the belief that members of the same group are
fundamentally alike, plays a crucial role in how prejudices and biases about
social and demographic groups are formed and expressed (Leslie, 2014). For
example, the statement “I speak English, I don’t speak libt*rd” implies the
belief that all “liberals are stupid.” If left unchallenged, statements with
such essentializing implications can cause harm by perpetuating and reifying
stereotypical beliefs about social groups Greenwald and Banaji (1995); Steele
(2011); Prentice and Miller (2007); Rhodes et al. (2012); Leshin et al.
(2021).

Figure 1:  Two prejudiced statements with their essentialist implications
(top), along with our five types of counterstatements automatically generated
with our method. According to our results, a preferred strategy is
highlighting that an implication applies to more than the targeted group (Lots
or Alts).

In this work, we investigate the task of combating essentialist statements and
beliefs through psychologically and linguistically informed counterstatement
generation. We examine these essentialist beliefs through the lens of generics
(Rhodes et al., 2012), i.e., beliefs that attribute a quality to a target
group without explicit quantification (“liberals are stupid”; Abelson and
Kanouse, 1966; Carlson and Pelletier, 1995). In the context of toxic or
hateful language, these generic beliefs can be both expressed directly or
conveyed through subtle implications Gelman (2003); Sap et al. (2020).

Automatically countering essentialism is challenging because it requires deep
psychological reasoning about the linguistic implications of statements – for
example, changing people’s beliefs about stereotypes only through
counterexamples is difficult Kunda and Oleson (1995). Therefore, we examine
five different strategies for combating essentializing stereotypes, combining
insights from psychology Foster-Hanson et al. (2016, 2019); Wodak et al.
(2015) and NLP Allaway et al. (2023). We craft five types of statements (see
Fig 1): broadening the scope of a stereotype by generalizing to “all people”
or an alternative group (Lots and Alt), providing direct counter-evidence
through specific individuals or groups (Dir-Ind and Dir-Grp), and simply
calling out the generalization (Tol). In contrast to prior studies on
countering hate-speech which use uncontrolled end-to-end generation approaches
(Qian et al., 2019; Zhu and Bhat, 2021; Chung et al., 2020, e.g.,), we
generate counterstatements by reasoning directly about the targeted group,
attributed quality, and linguistic expression of a stereotype.

Since our work provides a preliminary exploration of this task, we conduct
online studies in three settings where counterstatements are paired with
human-written implications from the Social Bias Frames Inference Corpus (SBIC)
Sap et al. (2020). In these settings, we explore variation in counterstatement
effectiveness when the beliefs are conveyed either implicitly, explicitly
without context, or as an explicit inference from provided context. We find
that challenging a stereotype by applying it broadly (e.g., to “lots of
people”; Lots and Alts; Figure 1) is generally the most preferred strategy. In
contrast, statements containing direct counter-evidence (e.g., Dir-Ind and
Dir-Grp; Figure 1) are the least popular. Additionally, we observe that the
most favored strategy varies depending on whether the stereotype is explicitly
presented to annotators (e.g., providing the essentialist belief in Figure 1)
or only conveyed implicitly (e.g., only providing the first statement in
Figure 1). For example, direct counter-evidence is more popular when the
stereotype is explicitly provided. Our results highlight the complexity of
countering essentialist beliefs and the importance of further investigation at
the intersection of NLP and psychology.

##  2  Automatically Countering Essentialism

We operationalize our counterstatement generation by focusing on the
expression of stereotypes through generics (§2.1). Inspired by work in
psychology and philosophy, we construct five types of counterstatements to a
stereotype (§2.2).

TEXT: RT @Vbomb20: Got these hoes on my dick like brad pitt |  GENERIC: Women are sex objects.
---|---
| Actually this is a generalization about women. +
---
(Dir-Grp) The following women are not sex objects: businesswomen, female
atheletes, and female movie stars.
(Dir-Ind) The following women are not sex objects: ellen degeneres, sarah
palin, and rachel maddow.
(Alt) Men can also be sex objects.
(Lots) Lots of people can be sex objects.
(Tol) All groups of people deserve tolerance.
TEXT: What’s black and doesn’t work? Half of London |  GENERIC: Black people don’t work
| Actually, this is a generalization about black people. +
---
(Dir-Grp) The following black people work: black businessmen, famous black
people, and black movie stars.
(Dir-Ind) The following black people work: barack obama, misty copeland, and
usain bolt.
(Alt) White folks may also not work.
(Lots) Lots of people don’t work.
(Tol) All groups of people deserve tolerance.
|  TEXT: How do you kill a thousand flies in one hit?
---
Slap an Ethiopian in the face.
GENERIC: Ethiopian people are dirty.
| Actually, this is a generalization about ethiopian people. +
---
(Dir-Grp) The following ethiopian people are not dirty: male atheletes, female
movie stars,
and people who practice judaism.
(Dir-Ind) The following ethiopian people are not dirty: kenyan marathon
runners, michael jackson,
and ryan reynolds.
|  TEXT: A muslim enters a building..
---
With 500 passengers and a plane
GENERIC: Muslims are terrorists.
| Actually, this is a generalization about muslims. +
---
(Dir-Grp) The following muslims are not terrorists: male muslim businessmen,
muslims businessmen,
and male muslim movie stars.
(Dir-Ind) The following muslims are not terrorists: adult muslim men, all
muslims, and malala yousafzai.
$\bm{\ldots}$

Table 1: Automatically generated counterstatements (§2.2) from our system. The
bottom two examples illustrate challenges with factuality in the Dir
counterstatements.

###  2.1 Stereotypes as Generics

Many negative stereotypes are expressed as generics; they generalize a
dangerous or harmful quality (e.g., being a drunkard) to an entire group
(e.g., Scots) based on the behavior of only a few individuals. Leslie (2008,
2017) termed such generics striking and argued that such generalizations are
based upon an assumption that all members of the group in question (e.g.,
Scots) are disposed to possess the dangerous or harmful quality. We argue that
many stereotypes can also be interpreted as asserting a quasi-unique
association between the group and quality. For example, “Scots are drunkards”
also implies that Scots are distinctly more likely than other groups (e.g.,
the English) to exhibit drunkenness. In our work, we assume that all
stereotypes under consideration are generics and have both interpretations.

Since generics are unquantified, they naturally allow for exceptions (i.e.,
counterexamples to the generic). While these exceptions may provide a relevant
source of counter-statements for a stereotype, some evidence from psychology
suggests that people are adept at maintaining their stereotyped beliefs in the
face of such specific exceptions (e.g., Kunda and Oleson, 1995). Therefore, we
experiment with a variety of different counter-statements.

###  2.2 Generating Counter-Speech

To generate counter-speech to stereotypes, we produce five types of outputs in
three broad categories (see Table 2). Since the stereotypes we consider are
expressed as generics (e.g., “Scots are drunkards”), they can be separated
into three components: a group (e.g., Scots), a relation (e.g., are), and a
quality (e.g., “drunkards”), which we use to construct the counter-speech.
Additionally, we prepend the sentence “Actually, this is a generalization
about GROUP” to each type of statement we generate, in order to contextualize
the statements as counter-speech.

#### Direct Exceptions (Dir)

Direct exceptions present subgroups or individuals that do not have the
quality specified in the generic, and thereby counter the striking or
extrapolating implications of the stereotype. For example, for “Scots are
drunkards”, the extrapolating implication is that “All Scots are drunkards”;
thus, direct exceptions would be either individual Scots (e.g., Ewan
McGregor111https://fherehab.com/learning/celebrities-who-dont-drink) or sub-
groups of Scots (e.g., Scottish babies) who are not drunkards. We follow
Allaway et al. (2023) who propose that these exceptions can be constructed
with the following template:

$\displaystyle\textsc{Group}(x)+\text{not
}\textit{relation}+\textsc{Quality}.$ (Dir)

We say that $\textsc{Group}(x)$ is satisfied if $x$ is either a specific
member of the group or a subgroup. We generate subtypes (i.e., subgroups and
specific group members) using GPT-3 (Brown et al., 2020). In particular, we
prompt GPT-3 with a list of subtypes for an example group not in our data and
query the model to produce subtypes for Group as the prompts completion. We
choose as our example group “men” (see Appendix A.1 for prompts). We then
construct exceptions following template Dir using each generated subtype. In
order to select the most truthful and relevant subtypes, we apply a truth
discriminator from Allaway et al. (2023) to each exception, and rank the
subtypes by the probability of being true and relevant. We construct the final
statements by combining the top three ranked subgroups into a single exception
((Dir-Grp) in Table 2) and combining the top three individuals into a single
exception ((Dir-Ind) in Table 2).

#### Broadening Exceptions (Alts)

Broadening exceptions challenge the quasi-unique implication of the generic by
attributing the quality in question to a different social group (e.g.,
“Americans can also be drunkards”). Allaway et al. (2023) propose that these
exceptions follow the template:

$\displaystyle{\nsim}\textsc{Group}(x)+\textit{relation}+\textsc{Quality}.$
(Alt)

where ${\nsim}\textsc{Group}$ indicates a contextually relevant alternative
group. For example, if Group = Scots, then a contextually relevant alternative
would be ${\nsim}\textsc{Group}$ = Americans. In our work, we define the
relevant alternative group ${\nsim}\textsc{Group}$ to be the perceived
oppressing group. For example, if the generic is “women are vain”, then “men”
would be the relevant alternative group ${\nsim}\textsc{Women}$ (i.e., the
oppressing group). To avoid generating stereotypes about the oppressing group,
we convert the relation into a hedged form (see Appendix A). For example, if
the relation is “are”, the hedged form of the relation would be “can be”.

(a) First choice.

(b) Second choice.

Figure 2: Percentage of annotators that selected each counterstatement type
(§2) across all three settings.

#### Broadening Universals (Lots)

In addition to broadening exceptions, we generate broadening universals, which
maximize the scope of the quality so that it includes people in general,
rather than any specific social group. That is, we generate statements
following:

$\displaystyle\text{Lots of people}+\textit{relation}+\textsc{Quality}.$
(Lots)

For example, “Lots of people are drunkards" is a broadening universal for the
stereotype “Scots are drunkards”. See (Lots) in Table 2. Similarly to the
statements following template Alt, we also hedge the relation in template
Lots.

#### Tolerance (Tol)

Finally, we include the denouncing statement, “All groups of people deserve
tolerance”, since denouncing is a common strategy in countering hate-speech
(e.g., Mathew et al., 2019; Qian et al., 2019; Ziegele et al., 2018). This
form of counter-speech does not depend on the details of the generic in
question and so is the same for all stereotypes. See (Tol) in Table 2.

##  3 Online Study

As a preliminary investigation into the task of generating counterstatements
to combat essentialism, we use posts with gold-annotated implications (§3.1)
to conduct an online experiment with crowdworkers (§3.2).

###  3.1 Essentialism Data

We use annotations provided in the SBIC (Sap et al., 2020) to obtain pairs
$(t,s)$ where $t$ is a text and $s$ is a stereotype implied by $t$ (i.e., an
essentialist implication that can be drawn from $t$). The $s$ in SBF are human
written and so to ensure the statements we consider are clear implications of
the text $t$, we use only instances where at least two out of the three human
annotators wrote the same stereotype verbatim. This results in a set of $227$
pairs, covering $25$ unique groups, where each $s_{i}$ can be clearly inferred
from $t_{i}$.

###  3.2 Study Setup

In order to investigate the effectiveness of different counter statements
(§2), we conduct three different human studies. In each study, we ask
annotators on Amazon Mechanical Turk to play the role of an online content
moderator or fact-checker whose job is to provide counterstatements to
expressed stereotypes. Each annotator is provided with a statement and a set
of machine-generated counterstatements and asked to select their first and
second choices. We also include an attention check to monitor annotation
quality, and collect information on how much annotators agree with the
provided statement and annotator demographic information. See full
instructions in Appendix B.

Our three human studies vary the statements provided to annotators: (1) post –
an original text $t$ from SBF, (2) stereo – the stereotype $s$ implied by a
text $t$, or (3) post + stereo – both $t$ and $s$. Note that for each pair
$(t,s)$ the counterstatements are always derived from $s$, regardless of
whether annotators are provided $s$ directly.

##  4 Empirical Results

Our results show clear differences in how often certain types of
counterstatements are preferred over others to combat essentialism (Figure 2).
We see that overall, the Lots counterstatements are the most popular for both
first and second choice. In addition, when considering broadening statements
grouped together (Lots and Alt), there is a clear preference for such
statements, compared to both the Tol and the direct exceptions. Despite the
lack of content in the Tol statements, these are the second most popular as
the first choice. Note, we choose not to conduct statistical tests because our
goal is not to find the single most effective countering strategy but rather
to study a range of strategies.

Of the generics-exceptions-based counterstatements, the direct exceptions Dir
are consistently the least preferred. We hypothesize that this is impacted by
the high portion of incorrect statements among the Dir type (Figure 3), as
well as the subjective nature of many stereotypes (e.g., in Table 2, being a
‘sex object’ is subjective). When considering only the statements not marked
as incorrect by annotators, we do not observe a change in relative popularity.
Therefore, future investigation is needed to understand the role of correct
individuals in counterstatements.

Figure 3: Percentage of counterstatements marked as incorrect for each
setting. Counterstatements are the same across settings, variation is due to
annotators.

In contrast, the broadening exceptions Alt rank second as the second-choice
and only 7% are marked as incorrect. We also note that in settings where the
stereotype is provided explicitly (stereo and stereo+post) the proportion of
Lots was higher (and Tol lower) for the first choice, and for the second
choice the proportion of Alt increased markedly. From this we observe that the
effectiveness of a countering strategy may depend on the explicitness of the
demonstrated bias. For example, generalizing the stereotype (Lots) may be less
effective when the stereotype is not explicitly identified (post setting).

Finally, we observe that when annotators agree with a statement, their
preference for Lots statements increases while the preference for Dir
counterstatements decreases (Fig. 4(a)). Annotator preference for Tol also
decreases. We also note that annotators more often endorse a belief when it is
stated explicitly, rather than implied by a text (Fig. 4(b)) These results
underscore the importance both of directly identifying an essentialist belief
from an implication and of reasoning about the implications of the stereotype
when countering real-world essentialist beliefs (i.e., from individuals who
endorse the belief).

(a) Percent of annotators that selected each counterstatement type as their
first choice.

(b) Percent of annotators who agreed with the statement.

Figure 4: Self-reported annotator agreement with the provided statement(s).

##  5 Discussion and Conclusion

Through our online studies, we find that broadening statements are the most
preferred type of counterstatement, while statements with direct counter-
evidence are consistently least preferred. In addition, we observe variation
across our three settings. Below, we discuss how are findings related to work
in psychology (§5.1) and content moderation (§5.2), and finally, outline
challenges, limitations, and future directions (§5.3).

###  5.1 Stereotypes and Psychology

Generic language, with its quasi-unique implications, readily conveys
essentialist beliefs. Indeed, psychological research shows that generic
language is a powerful mechanism by which social essentialist beliefs are
transmitted between people, and even across generations Rhodes et al. (2012);
Leshin et al. (2021). Such implications can have a profound impact on children
— e.g., girls as young as 6 years old have absorbed the stereotype that males
are more likely than females to be “really, really smart” Bian et al. (2017).
In order to challenge such essentialist beliefs, we argue that it is important
to consider the complexities of generics and associated inferences.

Through reasoning directly about the implications of generics, we construct
counterstatements that directly challenge essentialist implications. In
particular, our results highlight the value of broadening statements (Lots and
Alt), which counter the implication that a particular negative quality is
distinctive of a particular group (e.g., “Only women are vain”). This finding
is consistent with recent work in psychology, in particular Foster-Hanson et
al. (2019). These statements thereby challenge the cognitive value of the
stereotype as an information-processing short-cut Devine (1989), since the
wide applicability of the stereotyped quality may result in many incorrect
inferences (e.g., assuming someone is not vain because they are not a woman).

Furthermore, our results corroborate findings from psychology that individuals
who do not fit a stereotype are not viewed as invalidating that stereotype,
since they are categorized as special (e.g., Kunda and Oleson, 1995). In
particular, the consistently low preference for direct exception statements
comports with that finding (Dir-Ind and Dir-Grp). Although providing facts
(e.g., exceptional individuals) has been previously studied as a strategy to
counter hate-speech (e.g., Chung et al., 2019; Mathew et al., 2019), our work
specifically isolates the type of facts (i.e., direct counter-evidence versus
broadening statements) as a variable for investigation. As such, we can
observe that providing broadening facts is much more effective than counter-
evidence. This further highlights the importance of reasoning about the
specific implications of a text to counter essentialist beliefs.

###  5.2 Essentialism, Counter Hate-Speech, and Content Moderation

Although countering essentialism is similar in spirit to countering hate-
speech and content moderation, common strategies in the latter are often
inapplicable to countering essentialist beliefs. In content moderation,
discursive actions such as answering clarifying questions or providing
additional details are common Ziegele et al. (2018). However, since
essentialist beliefs are often conveyed implicitly (e.g., see statements in
Figure 1), discursive actions aimed at a text may not actually address its
essentialist implications. For example, the additional detail “libt∗rd is not
a real language” does not actually counter the implication that liberals are
stupid in Fig 1. Similarly, while humor, expressing affiliation with the
targeted group (e.g., “us Scots only having a wee cuppa tea”), and pointing
out hypocrisy or contradictions (e.g., “it needs to involve food to be a
meal”) are common when countering hate-speech Chung et al. (2019); Mathew et
al. (2019), they also do not address the essentialist beliefs implicit in a
text (e.g., that Scots are drunkards, Figure 1). As such, we argue that it is
important to investigate effective ways to counter essentialist implications,
as distinct from general counter-speech and content moderation.

(a) Racial demographics.

(b) Gender demographics.

Figure 5: Self-reported annotator demographics (percentage) across settings.

###  5.3 Limitations, Challenges, and Future Directions

Along with promising preliminary findings, our results highlighted several
limitations and challenges that should be tackled in future work.

#### Human-annotated implications

Since this work constitutes preliminary investigation on the promise of using
NLP tools for combating essentialism, we used a corpus of statements paired
with gold human-annotated implications. However, such annotations will not
always be available. Future work should examine whether our findings would
hold with machine-generated implications (e.g., using the neural model from
Sap et al., 2020), on various types of source domains and overtness levels
(e.g., the corpus of implicit toxicity from Hartvigsen et al., 2022).
Furthermore, future research could investigate how the quality and specificity
of the implications affects the counterstatement generation and effectiveness.

Group | Nb Examples
---|---
Black folks | 66
Women | 60
Muslim folks | 18
Jewish folks | 16
Asian folks | 15
Gay men | 7
Latino/Latina folks | 6
Liberals | 5
Feminists | 4
African folks | 3
Mentally disabled folks | 3
Indian folks | 3
Lesbian women | 3
Immigrants | 3
Ethiopian folks | 3
American folks | 2
Mexican folks | 2
Physically disabled folks | 1
Folks with mental illness/disorder | 1
Japanese folks | 1
Polish folks | 1
Arabic folks | 1
Italian folks | 1
Christian folks | 1
Native American/First Nation folks | 1

Table 2: Counts for number of examples per group. There are 227 examples total
across 25 unique groups.

#### Targeted group and annotator identity

Our studies are conducted on Amazon Mechanical Turk which can be lacking in
diversity among annotators. For example, the majority of annotators in our
study were white (Fig 5(a)) or male (Fig 5(b)). In contrast, targeted groups
are often not white or male (see Table 2). Since an annotator’s identity and
beliefs may impact their perceptions of how effective a counterstatement is
(as they do with perceptions of toxicity; Sap et al., 2022), homogeneity in
the annotator population limits our results. Additionally, how deeply rooted
an essentialist belief is for an annotator may impact what they consider
effective counterstatements. Our results, which show large variation in
annotator preference depending on whether they endorse a statement,
corroborate these findings. Therefore, future work should investigate more
diverse annotator pools or matching annotators to targeted groups, as well as
examining how annotator’s familiarity with essentialist beliefs and identities
affect their judgements.

Furthermore, prior work in countering hate-speech has show that effective
strategies can vary widely depending on the target group (Mathew et al., 2019;
Chung et al., 2019). In our work, we consider results aggregated across all
groups. However, community-specific investigations are an important future
step towards developing effective counter-statements.

#### Accuracy of generated exceptions

The selection of specific individuals for direct exceptions presents an
ongoing challenge, based on the high number of Dir-Ind marked incorrect. Since
language models often encode biases and stereotypes derived from training
corpora Sheng et al. (2019), they may have difficulty producing relevant
individuals who are not prototypical (i.e., they do not have a particular
stereotype). We illustrate incorrect individuals and subgroups in the bottom
two examples of Table 2. Additionally, as mentioned in §4, many stereotypes
are subjective (e.g., “women are vain”). Therefore, individuals who are
counterexamples to the stereotype may be judged differently by different
people (e.g., our system proposes that “taylor swift, sarah palin, and
scarlett johansson” are not vain). Producing accurate and relevant direct
exceptions to a stereotype is important for understanding the role of such
examples to counter essentialist beliefs.

Our results and discussion highlight the complexity of countering essentialist
beliefs. We propose that future work should improve the factuality of
counterstatements, particularly of direct counter-evidence, and consider both
variation in respondent demographics and community-specific needs. Therefore,
we argue that working at the intersection of NLP and psychology is crucial for
further investigations in this area.

##  6 Societal and Ethical Considerations

#### Annotation Considerations

Prior work has highlighted the potential harms to workers who are subjected to
offensive statements Roberts (2017); Steiger et al. (2021). To mitigate these,
we encourage annotators to reach out to the authors with concerns and
questions or to the Crisis Text Line.222https://www.crisistextline.org/
Additionally, our study design was approved by our ethics review board (IRB)
and workers earned a median wage of $10/h.

#### Risks of Generation

Since our system automatically generates counterstatements, there is potential
for misuse in several ways. First, our system can automatically and quickly
produce millions of counterstatements could therefore be used in a
distributed-denial-of-service attack. Second, by generating counterstatements
to stereotypes in text the original text remains available and so it may still
cause harm Ullmann and Tomalin (2019) and perpetuate essentialist beliefs.
Additionally, the automatic construction of counterstatements has the
potential to produce false statements and further harmful generalizations
(e.g., generalize a harmful stereotype to another marginalized group).
Considering these factors, it is important to jointly develop regulation
alongside AI technology to limit harms and misuse in deployment Crawford
(2021); Reich et al. (2021).

## Acknowledgements

We would like to thank the Beaker Team at AI2 for the compute infrastructure,
and the anonymous reviewers for their suggestions. This work is supported in
part by the National Science Foundation Graduate Research Fellowship under
Grant No. DGE-1644869. The views and conclusions contained herein are those of
the authors and should not be interpreted as necessarily representing the
official policies, either expressed or implied, of the NSF or the U.S.
Government. The U.S. Government is authorized to reproduce and distribute
reprints for governmental purposes notwithstanding any copyright annotation
therein.

## References

  * Abelson and Kanouse (1966) Robert P Abelson and David E Kanouse. 1966.  Subjective acceptance of verbal generalizations.  In _Cognitive consistency_ , pages 171–197. Elsevier.
  * Allaway et al. (2023) Emily Allaway, Jena D Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey, and Yejin Choi. 2023.  Penguins don’t fly: Reasoning about generics through instantiations and exceptions.  In _EACL_.
  * Anonymous (2020) Anonymous. 2020.  Paper title anonymized to preserve double-blindness.
  * Bian et al. (2017) Lin Bian, Sarah-Jane Leslie, and Andrei Cimpian. 2017.  Gender stereotypes about intellectual ability emerge early and influence children’s interests.  _Science_ , 355(6323):389–391.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Carlson and Pelletier (1995) Gregory N Carlson and Francis Jeffry Pelletier. 1995.  _The generic book_.  University of Chicago Press.
  * Chung et al. (2019) Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019\.  CONAN - COunter NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy. Association for Computational Linguistics.
  * Chung et al. (2020) Yi-Ling Chung, Serra Sinem Tekiroğlu, and Marco Guerini. 2020.  Italian counter narrative generation to fight online hate speech.  In _CLiC-it_.
  * Crawford (2021) Kate Crawford. 2021.  _Atlas of AI_.  Yale University Press.
  * Devine (1989) Patricia G. Devine. 1989.  Stereotypes and prejudice: Their automatic and controlled components.  _Journal of Personality and Social Psychology_ , 56:5–18.
  * Foster-Hanson et al. (2016) Emily Foster-Hanson, Sarah-Jane Leslie, and Marjorie Rhodes. 2016.  How does generic language elicit essentialist beliefs.  In _Proceedings of the 38th annual conference of the Cognitive Science Society_ , pages 1541–1546.
  * Foster-Hanson et al. (2019) Emily Foster-Hanson, Sarah-Jane Leslie, and Marjorie Rhodes. 2019.  Speaking of kinds: How correcting generic statements can shape children’s concepts.
  * Gelman (2003) Susan A Gelman. 2003.  _The essential child: Origins of essentialism in everyday thought_.  Oxford Cognitive Development.
  * Greenwald and Banaji (1995) Anthony G Greenwald and Mahzarin R Banaji. 1995.  Implicit social cognition: attitudes, self-esteem, and stereotypes.  _Psychological review_ , 102(1):4.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  Toxigen: Controlling language models to generate implied and adversarial toxicity.  In _ACL_.
  * Kunda and Oleson (1995) Ziva Kunda and Kathryn C Oleson. 1995.  Maintaining stereotypes in the face of disconfirmation: constructing grounds for subtyping deviants.  _Journal of personality and social psychology_ , 68(4):565.
  * Leshin et al. (2021) Rachel A Leshin, Sarah-Jane Leslie, and Marjorie Rhodes. 2021.  Does it matter how we speak about social kinds? a large, preregistered, online experimental study of how language shapes the development of essentialist beliefs.  _Child development_ , 92(4):e531–e547.
  * Leslie (2008) Sarah-Jane Leslie. 2008.  Generics: Cognition and acquisition.  _Philosophical Review_ , 117(1):1–47.
  * Leslie (2014) Sarah-Jane Leslie. 2014.  Carving up the social world with generics.  _Oxford studies in experimental philosophy_ , 1.
  * Leslie (2017) Sarah-Jane Leslie. 2017.  The original sin of cognition: Fear, prejudice, and generalization.  _The Journal of Philosophy_ , 114(8):393–421.
  * Mathew et al. (2019) Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019.  Thou shalt not hate: Countering online hate speech.  In _Proceedings of the international AAAI conference on web and social media_ , volume 13, pages 369–380.
  * Prentice and Miller (2007) Deborah A Prentice and Dale T Miller. 2007.  Psychological essentialism of human categories.  _Current directions in psychological science_ , 16(4):202–206.
  * Qian et al. (2019) Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019\.  A benchmark dataset for learning to intervene in online hate speech.  In _EMNLP_ , pages 4755–4764.
  * Reich et al. (2021) Rob Reich, Mehran Sahami, and Jeremy M Weinstein. 2021.  _System error: Where big tech went wrong and how we can reboot_.  Hodder & Stoughton.
  * Rhodes et al. (2012) Marjorie Rhodes, Sarah-Jane Leslie, and Christina M Tworek. 2012.  Cultural transmission of social essentialism.  _Proceedings of the National Academy of Sciences_ , 109(34):13526–13531.
  * Roberts (2017) Sarah T Roberts. 2017.  Social media’s silent filter.  _The Atlantic_.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020.  Social bias frames: Reasoning about social and power implications of language.  In _ACL_.
  * Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.  In _NAACL_.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412.
  * Steele (2011) Claude M Steele. 2011.  _Whistling Vivaldi: How stereotypes affect us and what we can do_.  WW Norton & Company.
  * Steiger et al. (2021) Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021.  The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support.  In _CHI_ , number Article 341 in CHI ’21.
  * Ullmann and Tomalin (2019) Stefanie Ullmann and Marcus Tomalin. 2019.  Quarantining online hate speech: technical and ethical perspectives.  _Ethics Inf. Technol._
  * Wodak et al. (2015) Daniel Wodak, Sarah-Jane Leslie, and Marjorie Rhodes. 2015.  What a loaded generalization: Generics and social cognition.  _Philosophy Compass_ , 10(9):625–635.
  * Zhu and Bhat (2021) Wanzheng Zhu and S. Bhat. 2021.  Generate, prune, select: A pipeline for counterspeech generation against online hate speech.  In _FINDINGS_.
  * Ziegele et al. (2018) Marc Ziegele, Pablo Jost, Marike Bormann, and Dominique Heinbach. 2018.  Journalistic counter-voices in comment sections: Patterns, determinants, and potential consequences of interactive moderation of uncivil user comments.

##  Appendix A Data Processing

To construct the hedged counterstatments, if the main verb is ‘is’ or ‘are’ we
convert it to ‘can also be’. For example ‘men are vain’ becomes ‘men can also
be vain’. If the main verb is ‘should’ we convert it to ‘should also’.
Otherwise, we insert ‘may also’ before the quality. For example, ‘men think
they know everything’ becomes ‘men may also think they know everything’.

We also note that the group names in Table 2 have been normalized. We will
include both the normalized and unnormalized names in the released data.

###  A.1 GPT-3 Generation

We access GPT-3 using the API from
OpenAI333https://beta.openai.com/docs/introduction. To obtain subtypes from
GPT-3 we use the ‘davinci’ model and top-$p$ sampling with $p=0.9$,
temperature 0.8 and maximum length $100$ tokens. The presence and frequency
penalties are both $0$. We kept the top 5 generations from GPT-3. We filter
out generations that are the same as the queried group. The prompts are shown
in Table 3. We randomized the order of the 5 examples in each prompt for every
group.

Counterstatement Type | Prompt
---|---
Dir-Grp |  | Consider the following groups of men:
---
1\. male students
2\. male authors
3\. male atheletes
4\. businessmen
5\. male movie stars
##
##
Consider the following groups of GROUP:
Dir-Ind |  | Consider the following groups of men:
---
1\. Barack Obama
2\. Sherlock Holmes
3\. Usain Bolt
4\. Ryan Reynolds
5\. Stephan Hawking
##
##
Consider the following groups of GROUP:
Table 3: Prompts for generating subtypes for GROUP from GPT3 (e.g.,
GROUP=women).

##  Appendix B Human Studies

For our user studies, we recruit annotators from Amazon Mechanical Turk who
were qualified for a toxicity explanation task from our previous work
Anonymous (2020).444Anonymized to preserve double-blindness of reviewing, will
be de-anonymized upon public release. Racial and gender breakdowns of our
annotator pool are in Figure 5. Annotators were paid $0.27 per task. For each
instance in each of the three settings we have 3 annotators. This study was
approved by our institution’s ethics board (IRB).

We show the detailed task instructions in Figure 6. An example of the task
setup is shown in Figure 7. Before choosing the most convincing counter
statements, annotators have the option to mark each statement as incorrect or
ungrammatical (Figure 7(b)). Note that before asking annotators to select
their second choice, we include an attention check (in Figure 7(c)). The
attention check was randomly set in each HIT. Annotations where the attention
check incorrect were discarded. As a result, we removed 3 annotations from the
post setting, 5 from the stereo setting, and 4 from post+stereo.

Figure 6: Detailed annotation instructions for human studies.

(a) Input presentation for post+stereo setting. The statement was removed for
the stereo setting and the stereotype was removed in the post setting.

(b) Example presentation. All five types of counter statements are listed in
the same manner.

(c) Annotation questions and attention check.

Figure 7: Details of the annotation task for human studies.

For each annotation, we also collected demographic information (Figure 8). The
demographic information is associated only with an annonymized annotator ID.
Additionally, before annotators select counter-statements, we ask annotators
to indicate their own belief in or agreement with the provided statement and
stereotype (Figure 9).

Figure 8: Demographic questionnaire in human studies.

Figure 9: Questions about stereotype belief of annotators.


# Detecting Tidal Features using Self-Supervised Representation Learning

Alice Desmons  Sarah Brough  Francois Lanusse

###### Abstract

Low surface brightness substructures around galaxies, known as tidal features,
are a valuable tool in the detection of past or ongoing galaxy mergers. Their
properties can answer questions about the progenitor galaxies involved in the
interactions. This paper presents promising results from a self-supervised
machine learning model, trained on data from the Ultradeep layer of the Hyper
Suprime-Cam Subaru Strategic Program optical imaging survey, designed to
automate the detection of tidal features. We find that self-supervised models
are capable of detecting tidal features and that our model outperforms
previous automated tidal feature detection methods, including a fully
supervised model. The previous state of the art method achieved 76%
completeness for 22% contamination, while our model achieves considerably
higher (96%) completeness for the same level of contamination.

Machine Learning, ICML

##  1 Introduction

The currently accepted model of the Universe, known as the Lambda Cold Dark
Matter ($\Lambda$CDM) Cosmological Model, postulates that galaxies evolve
through a process which is referred to as the ‘hierarchical merger model’,
wherein the growth of the universe’s highest-mass galaxies is dominated by
merging with lower-mass galaxies (e.g. Lacey and Cole, 1994; Cole et al.,
2000; Robotham et al., 2014; Martin et al., 2018). During the merging process,
the extreme gravitational forces involved cause stellar material to be pulled
out from the galaxies, forming diffuse non-uniform regions of stars in the
outskirts of the galaxies, known as tidal features. These tidal features
contain information about the merging history of the galaxy, and can thus be
used to study the galaxy evolution process.

In order to draw accurate and statistically robust conclusions about this
evolution process, we require a large sample of galaxies exhibiting tidal
features. One thing that makes this difficult is the extremely low surface
brightness of tidal features, which can easily reach $\mu_{r}\geq$ 27 mag
$\rm{arcsec}^{-2}$. With the next generation of wide-field optical imaging
surveys reaching new limiting depths, such as the Vera C Rubin Observatory’s
Legacy Survey of Space and Time (LSST; Ivezić et al., 2019) which is predicted
to reach $\mu_{r}\sim$ 30.1 mag $\rm{arcsec}^{-2}$ (Martin et al.,, 2022),
assembling a statistically significant sample of galaxies with tidal features
is becoming more feasible. One challenge associated with surveys like LSST,
due to commence in 2024 and run for 10 years, is the amount of data predicted
to be released, with LSST predicted to output over 500 petabytes of imaging
data including billions of galaxies (Ivezić et al.,, 2019). Current tidal
feature detection and classification is primarily achieved through visual
identification (e.g. Tal et al., 2009; Sheen et al., 2012; Atkinson et al.,
2013; Hood et al., 2018; Bílek et al., 2020; Martin et al., 2022), but this
amount of data is virtually impossible to classify visually by humans, even
using large community based projects such as Galaxy Zoo (Lintott et al.,,
2008; Darg et al.,, 2010), and hence we are in urgent need of a tool that can
automate this classification task and isolate galaxies with tidal features.

With the promising recent results of machine learning in galaxy classification
tasks (e.g. Hocking et al., 2018; Diaz et al., 2019; Pearson et al., 2019;
Snyder et al., 2019; Walmsley et al., 2019; Cavanagh and Bekki, 2020; Martin
et al., 2020), we turn to machine learning to construct a model which can take
galaxy images as input, convert them into representations - low-dimensional
maps which preserve the important information in the image - and output a
classification based on whether the galaxy possesses tidal features. We use a
recently developed machine learning method that is essentially a middle-point
between supervised and unsupervised learning, known as Self-Supervised machine
Learning (SSL; He et al., 2019; Chen et al., 2020a ; Chen et al., 2020b ; Chen
et al., 2020c ; Chen and He, 2020). Such models do not require labelled data
for the training of the encoder, which learns to transform images into
meaningful low-dimensional representations, but can perform classification
when paired with a linear classifier and a small labelled dataset. Instead of
labels, SSL models rely on augmentations to learn under which conditions the
output low-dimensional representations should be invariant. These types of
models have been successfully used for a variety of astronomical applications
(e.g. Hayat et al., 2021; Stein et al., 2022; Slijepcevic et al., 2022;
Walmsley et al., 2022; Wei et al., 2022; Ćiprijanović et al., 2023; Huertas-
Company et al., 2023; Slijepcevic et al., 2023) Compared to supervised models,
self-supervised models are also much easier to adapt to perform new tasks, and
apply to datasets from different astronomical surveys (Ćiprijanović et al.,,
2023) making this kind of model perfect for our goal of applying the tool
developed using HSC-SSP data to future LSST data.

##  2 Methods

###  2.1 Sample Selection

The dataset used for this work is sourced from the Ultradeep (UD) layer of the
HSC-SSP Public Data Release 2 (PDR2; Aihara et al., 2019) for deep galaxy
images. We use the Ultradeep field, which spans an area of $3.5$ deg2 and
reaches a surface brightness depth of $\mu_{r}\sim$ 28.0 mag arcsec-2 as it
reaches depths faint enough to detect tidal features.

We assemble an unlabelled dataset of $\sim$44,000 galaxies by parsing objects
in the HSC-SSP PDR2 database using an SQL search and only selecting objects
which have at least 3 exposures in each band and have i-band magnitudes 15 $<$
i $<$ 20 mag. We set a faint magnitude limit of 20 mag to ensure that objects
are bright enough for tidal features to be visible. We access the HSC-SSP
galaxy images using the ‘Unagi’ Python tool (Huang et al.,, 2019) which, given
a galaxy’s right ascension and declination, allows us to create multi-band
‘HSC cutout’ images of size 128 $\times$ 128 pixels, or 21 $\times$ 21
arcsecs, centred around each galaxy. Each cutout is downloaded in five
($g,~{}r,~{}i,~{}z,~{}y$) bands.

For the training of the linear classifier we require a small labelled dataset
of galaxies with and without tidal features. We use the HSC-SSP UD PDR2
dataset assembled by Desmons et al., (2023) composed of 211 galaxies with
tidal features and 641 galaxies without tidal features. These galaxies were
selected from a volume-limited sample from the cross-over between then Galaxy
and Mass Assembly survey (Driver et al.,, 2011) and HSC-SSP with spectroscopic
redshift limits 0.04 $\leq$ z $\leq$ 0.2 and stellar mass limits 9.50 $\leq$
log10($M_{\star}$/M⊙) $\leq$ 11.00 and have i-band magnitudes in the range
12.8 $<$ i $<$ 21.6 mag. To increase the size of our tidal feature training
sample we classified additional galaxies from our HSC-SSP PDR2 unlabelled
dataset of $\sim$ 44,000 objects, according to the classification scheme
outlined in Desmons et al., (2023). Our final labelled sample contains 760
galaxies, 380 with tidal features, labelled 1, and 380 without, labelled 0. We
split our labelled dataset set into training, validation, and testing datasets
composed of 600, 60, and 100 galaxies respectively.

###  2.2 Image Pre-processing and Augmentations

Before the images are augmented and fed through the model we apply a pre-
processing function to normalise the images. The augmentations we use for this
project are:

  * •

Orientation: We randomly flip the image across each axis (x and y) with 50%
probability.

  * •

Gaussian Noise: We sample a scalar from $\mathcal{U}$(1,3) and multiply it
with the median absolute deviation of each channel (calculated over 1000
training examples) to get a per-channel noise $\sigma_{c}$. We then introduce
Gaussian noise sampled from $\sigma_{c}~{}\times~{}\mathcal{N}$(0,1) for each
channel.

  * •

Jitter and Crop: For HSC-SSP images we crop the 128 $\times$ 128 pixel image
to the central 109 $\times$ 109 pixels before randomly cropping the image to
96 $\times$ 96 pixel. Random cropping means the image center is translated, or
‘jittered’, along each respective axis by $i$, $j$ pixels where $i$,
$j~{}\sim~{}\mathcal{U}$(-13,13) before cropping to the central 96 $\times$ 96
pixels.

###  2.3 Model Architecture

The model we utilise to perform classification of tidal feature candidates
consists of two components; a self-supervised model used for pre-training, and
a ‘fine-tuned’ model used for classification. All models described below are
built using the TensorFlow framework (Abadi et al.,, 2016).

####  2.3.1 The Self-Supervised Architecture

For our task of classifying tidal feature candidates we use a type of self-
supervised learning known as Nearest Neighbour Contrastive Learning of visual
Representations (NNCLR; Dwibedi et al., 2021). We closely follow Dwibedi et
al., (2021) in designing the architecture and training process for our model.
The model was compiled using the Adam optimiser (Kingma and Ba,, 2015) and
trained for 25 epochs on our unlabelled dataset of $\sim$ 44,000 HSC-SSP PDR2
galaxies.

####  2.3.2 The Fine-tuned Architecture

The fine-tuned model is a simple linear classifier which takes galaxy images
as input and converts them to representations using the pre-trained self-
supervised encoder. These representations are passed through a ‘Dense’ layer
with a sigmoid activation, which outputs a single number between 0 and 1. This
fine-tuned model was compiled using the Adam optimiser (Kingma and Ba,, 2015)
and a binary cross entropy loss. It was trained for 50 epochs using the
labelled training set of 600 HSC-SSP galaxies. Training was completed within
$\sim$ 1 minute using a single GPU.

####  2.3.3 The Supervised Architecture

To draw conclusions about the suitability of self-supervised models for the
detection and classification of tidal features, we compare our results with
those of a fully supervised model. We do not construct this model from
scratch, but instead use the published model designed by Pearson et al.,
(2019) to classify merging galaxies. The output layer was changed from two
neurons with softmax activation, to a single neuron with sigmoid activation.
The network was compiled using the Adam optimiser (Kingma and Ba,, 2015) with
the default learning rate and loss of the network was determined using binary
cross entropy. We additionally changed the input image dimension from 64
$\times$ 64 pixels with three colour channels to 96 $\times$ 96 pixels with
five colour channels to ensure extended tidal features remain visible. We
train this fully supervised network from scratch using the labelled training
set of 600 HSC-SSP galaxies.

###  2.4 Model Evaluation

To evaluate our model performance we use the true positive rate (also known as
recall or completeness) and false positive rate (also known as fall-out or
contamination). The true positive rate (TPR) ranges from 0 to 1 and is defined
as the fraction of galaxies correctly classified by the model as having tidal
features with respect to the total number of galaxies with tidal features. The
false positive rate (FPR) also ranges from 0 to 1 and is defined as the
fraction of galaxies incorrectly classified by the model as having tidal
features with respect to the total number of galaxies without tidal features.

In addition to using the TPR for a given FPR to evaluate our model, we also
use the area under the receiver operating characteristic (ROC) curve, or ROC
AUC, to evaluate performance.

##  3 Results

###  3.1 Self-Supervised vs. Supervised Performance

Figure 1 illustrates the testing set ROC AUC for a supervised and self-
supervised network as a function of the number of labels used in training for
our HSC-SSP dataset. Each point represents the ROC AUC averaged over ten runs
using the same training, validation, and testing sets for each run. We average
the ROC AUC over the 10 runs and remove outliers further than $3\sigma$ from
the mean. Our SSL model maintains high performance across all amounts of
labels used for training, having ROC AUC $=$ 0.911 $\pm$ 0.002 when training
on the maximum number of labels and only dropping to ROC AUC $=$ 0.89 $\pm$
0.01 when using only 50 labels for training. The supervised model also
maintains its performance regardless of label number, but only reaches ROC AUC
$=$ 0.867 $\pm$ 0.004 when training on the maximum number and ROC AUC $=$ 0.83
$\pm$ 0.01 when using 50 labels for training.

Figure 1: Average ROC AUC as a function of the number of HSC-SSP labels used
for training for a supervised (blue) and self-supervised (red) model. Each
point is an average of ten runs.

This figure not only shows that an SSL model can be used for the detection of
tidal features with good performance, but also that it performs consistently
better than the supervised network regardless of the number of training
labels. We also calculated the average TPR reached by the self-supervised
model on the testing set for a given FPR $=$ 0.2, averaging over 10 runs and
removing outliers. When training using 600 labels, the model reaches TPR $=$
0.94 $\pm$ 0.01, and this only drops to TPR $=$ 0.90 $\pm$ 0.01 when using a
mere 50 labels for training.

Figure 2: Results from a similarity search using a random galaxy with tidal
features as a query image, displayed on the left, alongside the top 24
galaxies with the highest similarity scores for each similarity search on the
right. The similarity score is displayed in the top left corner for each
image. The red outlines indicate images containing galaxies which would be
visually classified as hosting tidal features, regardless of whether this
galaxy is the central object in the image.
Figure 3: Left: 2D UMAP projection of the self-supervised representations.
Made by binning the space into $100~{}\times~{}100$ cells and randomly
selecting a sample from that cell to plot in the corresponding cell location.
Right: The same 2D UMAP projection without binning, coloured according the
scores assigned to each galaxy by the linear classifier.

###  3.2 Detection of Tidal Features

One advantage of self-supervised models over supervised models is the ability
to use just one labelled example to find examples of similar galaxies from the
full dataset. By using just one image from our labelled tidal feature dataset
as a query image, and the encoded 128-dimensional representations from the
self-supervised encoder, we can perform a similarity search that assigns high
similarity scores to images which have similar representations to the query
image. This is demonstrated in Figure 2 where we select a random galaxy with
tidal features from our training sample and perform a similarity search with
the 44,000 unlabelled HSC-SSP galaxies. In Figure 2 the query image is shown
on the right alongside the 24 galaxies which received the highest similarity
scores. This figure shows the power of self-supervised learning, where using
only a single labelled example, we can find a multitude of other tidal feature
candidates.

We can also visualise how the model organises the galaxy images in
representation space, by using Uniform Manifold Approximation and Projection
(UMAP; McInnes et al., 2018) which reduces the encoded representations to an
easier to visualise 2 dimensional projection. Figure 3 illustrates this 2D
projection, created by binning the space into $100~{}\times~{}100$ cells and
randomly selecting a sample from that cell to plot in the corresponding cell
location. We also enquire whether the scores given to galaxies by the linear
classifier are related to the galaxies’ positions in the UMAP projection, by
colouring the UMAP plot according the scores given to each galaxy by the
linear classifier, shown in the right panel of Figure 3. We find that the
majority of galaxies which were assigned a high classifier score, indicating a
high likelihood of tidal features, are located on the left side of the UMAP
projection plot. This reinforces the idea that the encoded representations
contain meaningful information about tidal features.

##  4 Discussion and Conclusions

In this work, we have shown that SSL models composed of a self-supervised
encoder and linear classifier can not only be used to detect galaxies with
tidal features, but can do so reaching both high completeness (TPR $=$ 0\. 94
$\pm$ 0.1) for low contamination (FPR $=$ 0.20) and high area under the ROC
curve (ROC AUC $=$ 0.91 $\pm$ 0.002). This means that such models can be used
to isolate the majority of galaxies with tidal features from a large sample of
galaxies, thus drastically reducing the amount of visual classification needed
to assemble a large sample of tidal features. One major advantage of this
model over other automated classification methods, is that this level of
performance can be reached using only 600 labelled training examples, and only
drops mildly when using a mere 50 labels for training maintaining ROC AUC $=$
0.89 $\pm$ 0.01 and TPR $=$ 0.90 $\pm$ 0.1 for FPR $=$ 0.2. This makes SSL
models easy to re-train on data from different surveys with minimal visual
classification needed. Following Stein et al., (2021), we emphasise the
usefulness of being able to perform a similarity search using just the self-
supervised encoder and one example of a galaxy with tidal features to find
other galaxies with tidal features from a dataset of tens of thousands of
galaxies.

The level of comparison that can be carried out with respect to the results
obtained here and other works is limited due to the scarcity of similar works.
There is only one study focusing on the detection of tidal features using
machine learning, namely the work of Walmsley et al., (2019) who used a
supervised network to identify galaxies with tidal features from the Wide
layer of the Canada-France-Hawaii Telescope Legacy Survey (Gwyn,, 2012).
Walmsley et al., (2019) found that their method outperformed other automated
methods of tidal feature detection, reaching 76% completeness (or TPR) and 22%
contamination (or FPR). Our SSL model, trained on 600 galaxies performs
considerably better, reaching a completeness of 96% for the same contamination
percentage. Most importantly, our model consistently outperforms a fully
supervised model trained on the same data, reaching ROC AUC = 0.911 $\pm$0.002
while the fully supervised model only reaches a maximum ROC AUC of 0.864 $\pm$
0.004.

The code use to create, train, validate, and test the SSML model, along with
instructions on loading and using the pre-trained model as well as training
the model using different data can be downloaded from
GitHub111https://github.com/LSSTISSC/Tidalsaurus.

## References

  * Abadi et al., (2016) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mane, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viegas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2016).  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems.  arXiv e-prints, page arXiv:1603.04467.
  * Aihara et al., (2019) Aihara, H., AlSayyad, Y., Ando, M., Armstrong, R., Bosch, J., Egami, E., Furusawa, H., Furusawa, J., Goulding, A., Harikane, Y., Hikage, C., Ho, P. T. P., Hsieh, B.-C., Huang, S., Ikeda, H., Imanishi, M., Ito, K., Iwata, I., Jaelani, A. T., Kakuma, R., Kawana, K., Kikuta, S., Kobayashi, U., Koike, M., Komiyama, Y., Li, X., Liang, Y., Lin, Y.-T., Luo, W., Lupton, R., Lust, N. B., MacArthur, L. A., Matsuoka, Y., Mineo, S., Miyatake, H., Miyazaki, S., More, S., Murata, R., Namiki, S. V., Nishizawa, A. J., Oguri, M., Okabe, N., Okamoto, S., Okura, Y., Ono, Y., Onodera, M., Onoue, M., Osato, K., Ouchi, M., Shibuya, T., Strauss, M. A., Sugiyama, N., Suto, Y., Takada, M., Takagi, Y., Takata, T., Takita, S., Tanaka, M., Terai, T., Toba, Y., Uchiyama, H., Utsumi, Y., Wang, S.-Y., Wang, W., and Yamada, Y. (2019).  Second data release of the Hyper Suprime-Cam Subaru Strategic Program.  PASJ, 71(6):114.
  * Atkinson et al., (2013) Atkinson, A. M., Abraham, R. G., and Ferguson, A. M. N. (2013).  Faint Tidal Features in Galaxies within the Canada-France-Hawaii Telescope Legacy Survey Wide Fields.  ApJ, 765(1):28.
  * Bílek et al., (2020) Bílek, M., Duc, P.-A., Cuillandre, J.-C., Gwyn, S., Cappellari, M., Bekaert, D. V., Bonfini, P., Bitsakis, T., Paudel, S., Krajnović, D., Durrell, P. R., and Marleau, F. (2020).  Census and classification of low-surface-brightness structures in nearby early-type galaxies from the MATLAS survey.  MNRAS, 498(2):2138–2166.
  * Cavanagh and Bekki, (2020) Cavanagh, M. K. and Bekki, K. (2020).  Bars formed in galaxy merging and their classification with deep learning.  A&A, 641:A77.
  * (6) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020a).  A Simple Framework for Contrastive Learning of Visual Representations.  arXiv e-prints, page arXiv:2002.05709.
  * (7) Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. (2020b).  Big Self-Supervised Models are Strong Semi-Supervised Learners.  arXiv e-prints, page arXiv:2006.10029.
  * (8) Chen, X., Fan, H., Girshick, R., and He, K. (2020c).  Improved Baselines with Momentum Contrastive Learning.  arXiv e-prints, page arXiv:2003.04297.
  * Chen and He, (2020) Chen, X. and He, K. (2020).  Exploring Simple Siamese Representation Learning.  arXiv e-prints, page arXiv:2011.10566.
  * Ćiprijanović et al., (2023) Ćiprijanović, A., Lewis, A., Pedro, K., Madireddy, S., Nord, B., Perdue, G. N., and Wild, S. M. (2023).  DeepAstroUDA: Semi-Supervised Universal Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection.  arXiv e-prints, page arXiv:2302.02005.
  * Cole et al., (2000) Cole, S., Lacey, C. G., Baugh, C. M., and Frenk, C. S. (2000).  Hierarchical galaxy formation.  MNRAS, 319(1):168–204.
  * Darg et al., (2010) Darg, D. W., Kaviraj, S., Lintott, C. J., Schawinski, K., Sarzi, M., Bamford, S., Silk, J., Proctor, R., Andreescu, D., Murray, P., Nichol, R. C., Raddick, M. J., Slosar, A., Szalay, A. S., Thomas, D., and Vandenberg, J. (2010).  Galaxy Zoo: the fraction of merging galaxies in the SDSS and their morphologies.  MNRAS, 401(2):1043–1056.
  * Desmons et al., (2023) Desmons, A., Brough, S., Martínez-Lombilla, C., De Propris, R., Holwerda, B., and López Sánchez, Á. R. (2023).  Galaxy and mass assembly (GAMA): Comparing visually and spectroscopically identified galaxy merger samples.  MNRAS.
  * Diaz et al., (2019) Diaz, J. D., Bekki, K., Forbes, D. A., Couch, W. J., Drinkwater, M. J., and Deeley, S. (2019).  Classifying the formation processes of S0 galaxies using Convolutional Neural Networks.  MNRAS, 486(4):4845–4862.
  * Driver et al., (2011) Driver, S. P., Hill, D. T., Kelvin, L. S., Robotham, A. S. G., Liske, J., Norberg, P., Baldry, I. K., Bamford, S. P., Hopkins, A. M., Loveday, J., Peacock, J. A., Andrae, E., Bland-Hawthorn, J., Brough, S., Brown, M. J. I., Cameron, E., Ching, J. H. Y., Colless, M., Conselice, C. J., Croom, S. M., Cross, N. J. G., de Propris, R., Dye, S., Drinkwater, M. J., Ellis, S., Graham, A. W., Grootes, M. W., Gunawardhana, M., Jones, D. H., van Kampen, E., Maraston, C., Nichol, R. C., Parkinson, H. R., Phillipps, S., Pimbblet, K., Popescu, C. C., Prescott, M., Roseboom, I. G., Sadler, E. M., Sansom, A. E., Sharp, R. G., Smith, D. J. B., Taylor, E., Thomas, D., Tuffs, R. J., Wijesinghe, D., Dunne, L., Frenk, C. S., Jarvis, M. J., Madore, B. F., Meyer, M. J., Seibert, M., Staveley-Smith, L., Sutherland, W. J., and Warren, S. J. (2011).  Galaxy and Mass Assembly (GAMA): survey diagnostics and core data release.  MNRAS, 413(2):971–995.
  * Dwibedi et al., (2021) Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman, A. (2021).  With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations.  arXiv e-prints, page arXiv:2104.14548.
  * Gwyn, (2012) Gwyn, S. D. J. (2012).  The canada-france-hawaii telescope legacy survey: Stacked images and catalogs.  AJ, 143(2).
  * Hayat et al., (2021) Hayat, M. A., Stein, G., Harrington, P., Lukić, Z., and Mustafa, M. (2021).  Self-supervised Representation Learning for Astronomical Images.  ApJl, 911(2):L33.
  * He et al., (2019) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2019).  Momentum Contrast for Unsupervised Visual Representation Learning.  arXiv e-prints, page arXiv:1911.05722.
  * Hocking et al., (2018) Hocking, A., Geach, J. E., Sun, Y., and Davey, N. (2018).  An automatic taxonomy of galaxy morphology using unsupervised machine learning.  MNRAS, 473(1):1108–1129.
  * Hood et al., (2018) Hood, C. E., Kannappan, S. J., Stark, D. V., Dell’Antonio, I. P., Moffett, A. J., Eckert, K. D., Norris, M. A., and Hendel, D. (2018).  The Origin of Faint Tidal Features around Galaxies in the RESOLVE Survey.  ApJ, 857(2):144.
  * Huang et al., (2019) Huang, S., Li, J.-X., Lanusse, F., and Bradshaw, C. (2019).  Unagi.  https://github.com/dr-guangtou/unagi.git.
  * Huertas-Company et al., (2023) Huertas-Company, M., Sarmiento, R., and Knapen, J. (2023).  A brief review of contrastive learning applied to astrophysics.  arXiv e-prints, page arXiv:2306.05528.
  * Ivezić et al., (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., Abel, B., Acosta, E., Allsman, R., Alonso, D., AlSayyad, Y., Anderson, S. F., Andrew, J., Angel, J. R. P., Angeli, G. Z., Ansari, R., Antilogus, P., Araujo, C., Armstrong, R., Arndt, K. T., Astier, P., Aubourg, É., Auza, N., Axelrod, T. S., Bard, D. J., Barr, J. D., Barrau, A., Bartlett, J. G., Bauer, A. E., Bauman, B. J., Baumont, S., Bechtol, E., Bechtol, K., Becker, A. C., Becla, J., Beldica, C., Bellavia, S., Bianco, F. B., Biswas, R., Blanc, G., Blazek, J., Blandford, R. D., Bloom, J. S., Bogart, J., Bond, T. W., Booth, M. T., Borgland, A. W., Borne, K., Bosch, J. F., Boutigny, D., Brackett, C. A., Bradshaw, A., Brandt, W. N., Brown, M. E., Bullock, J. S., Burchat, P., Burke, D. L., Cagnoli, G., Calabrese, D., Callahan, S., Callen, A. L., Carlin, J. L., Carlson, E. L., Chandrasekharan, S., Charles-Emerson, G., Chesley, S., Cheu, E. C., Chiang, H.-F., Chiang, J., Chirino, C., Chow, D., Ciardi, D. R., Claver, C. F., Cohen-Tanugi, J., Cockrum, J. J., Coles, R., Connolly, A. J., Cook, K. H., Cooray, A., Covey, K. R., Cribbs, C., Cui, W., Cutri, R., Daly, P. N., Daniel, S. F., Daruich, F., Daubard, G., Daues, G., Dawson, W., Delgado, F., Dellapenna, A., de Peyster, R., de Val-Borro, M., Digel, S. W., Doherty, P., Dubois, R., Dubois-Felsmann, G. P., Durech, J., Economou, F., Eifler, T., Eracleous, M., Emmons, B. L., Fausti Neto, A., Ferguson, H., Figueroa, E., Fisher-Levine, M., Focke, W., Foss, M. D., Frank, J., Freemon, M. D., Gangler, E., Gawiser, E., Geary, J. C., Gee, P., Geha, M., Gessner, C. J. B., Gibson, R. R., Gilmore, D. K., Glanzman, T., Glick, W., Goldina, T., Goldstein, D. A., Goodenow, I., Graham, M. L., Gressler, W. J., Gris, P., Guy, L. P., Guyonnet, A., Haller, G., Harris, R., Hascall, P. A., Haupt, J., Hernandez, F., Herrmann, S., Hileman, E., Hoblitt, J., Hodgson, J. A., Hogan, C., Howard, J. D., Huang, D., Huffer, M. E., Ingraham, P., Innes, W. R., Jacoby, S. H., Jain, B., Jammes, F., Jee, M. J., Jenness, T., Jernigan, G., Jevremović, D., Johns, K., Johnson, A. S., Johnson, M. W. G., Jones, R. L., Juramy-Gilles, C., Jurić, M., Kalirai, J. S., Kallivayalil, N. J., Kalmbach, B., Kantor, J. P., Karst, P., Kasliwal, M. M., Kelly, H., Kessler, R., Kinnison, V., Kirkby, D., Knox, L., Kotov, I. V., Krabbendam, V. L., Krughoff, K. S., Kubánek, P., Kuczewski, J., Kulkarni, S., Ku, J., Kurita, N. R., Lage, C. S., Lambert, R., Lange, T., Langton, J. B., Le Guillou, L., Levine, D., Liang, M., Lim, K.-T., Lintott, C. J., Long, K. E., Lopez, M., Lotz, P. J., Lupton, R. H., Lust, N. B., MacArthur, L. A., Mahabal, A., Mandelbaum, R., Markiewicz, T. W., Marsh, D. S., Marshall, P. J., Marshall, S., May, M., McKercher, R., McQueen, M., Meyers, J., Migliore, M., Miller, M., Mills, D. J., Miraval, C., Moeyens, J., Moolekamp, F. E., Monet, D. G., Moniez, M., Monkewitz, S., Montgomery, C., Morrison, C. B., Mueller, F., Muller, G. P., Muñoz Arancibia, F., Neill, D. R., Newbry, S. P., Nief, J.-Y., Nomerotski, A., Nordby, M., O’Connor, P., Oliver, J., Olivier, S. S., Olsen, K., O’Mullane, W., Ortiz, S., Osier, S., Owen, R. E., Pain, R., Palecek, P. E., Parejko, J. K., Parsons, J. B., Pease, N. M., Peterson, J. M., Peterson, J. R., Petravick, D. L., Libby Petrick, M. E., Petry, C. E., Pierfederici, F., Pietrowicz, S., Pike, R., Pinto, P. A., Plante, R., Plate, S., Plutchak, J. P., Price, P. A., Prouza, M., Radeka, V., Rajagopal, J., Rasmussen, A. P., Regnault, N., Reil, K. A., Reiss, D. J., Reuter, M. A., Ridgway, S. T., Riot, V. J., Ritz, S., Robinson, S., Roby, W., Roodman, A., Rosing, W., Roucelle, C., Rumore, M. R., Russo, S., Saha, A., Sassolas, B., Schalk, T. L., Schellart, P., Schindler, R. H., Schmidt, S., Schneider, D. P., Schneider, M. D., Schoening, W., Schumacher, G., Schwamb, M. E., Sebag, J., Selvy, B., Sembroski, G. H., Seppala, L. G., Serio, A., Serrano, E., Shaw, R. A., Shipsey, I., Sick, J., Silvestri, N., Slater, C. T., Smith, J. A., Smith, R. C., Sobhani, S., Soldahl, C., Storrie-Lombardi, L., Stover, E., Strauss, M. A., Street, R. A., Stubbs, C. W., Sullivan, I. S., Sweeney, D., Swinbank, J. D., Szalay, A., Takacs, P., Tether, S. A., Thaler, J. J., Thayer, J. G., Thomas, S., Thornton, A. J., Thukral, V., Tice, J., Trilling, D. E., Turri, M., Van Berg, R., Vanden Berk, D., Vetter, K., Virieux, F., Vucina, T., Wahl, W., Walkowicz, L., Walsh, B., Walter, C. W., Wang, D. L., Wang, S.-Y., Warner, M., Wiecha, O., Willman, B., Winters, S. E., Wittman, D., Wolff, S. C., Wood-Vasey, W. M., Wu, X., Xin, B., Yoachim, P., and Zhan, H. (2019).  LSST: From Science Drivers to Reference Design and Anticipated Data Products.  ApJ, 873(2):111.
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. (2015).  Adam: A Method for Stochastic Optimization.  3rd International Conference for Learning Representations, page arXiv:1412.6980.
  * Lacey and Cole, (1994) Lacey, C. and Cole, S. (1994).  Merger Rates in Hierarchical Models of Galaxy Formation - Part Two - Comparison with N-Body Simulations.  MNRAS, 271:676.
  * Lintott et al., (2008) Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., Murray, P., and Vandenberg, J. (2008).  Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey.  MNRAS, 389(3):1179–1189.
  * Martin et al., (2022) Martin, G., Bazkiaei, A. E., Spavone, M., Iodice, E., Mihos, J. C., Montes, M., Benavides, J. A., Brough, S., Carlin, J. L., Collins, C. A., Duc, P. A., Gómez, F. A., Galaz, G., Hernández-Toledo, H. M., Jackson, R. A., Kaviraj, S., Knapen, J. H., Martínez-Lombilla, C., McGee, S., O’Ryan, D., Prole, D. J., Rich, R. M., Román, J., Shah, E. A., Starkenburg, T. K., Watkins, A. E., Zaritsky, D., Pichon, C., Armus, L., Bianconi, M., Buitrago, F., Busá, I., Davis, F., Demarco, R., Desmons, A., García, P., Graham, A. W., Holwerda, B., Hon, D. S. H., Khalid, A., Klehammer, J., Klutse, D. Y., Lazar, I., Nair, P., Noakes-Kettel, E. A., Rutkowski, M., Saha, K., Sahu, N., Sola, E., Vázquez-Mata, J. A., Vera-Casanova, A., and Yoon, I. (2022).  Preparing for low surface brightness science with the Vera C. Rubin Observatory: Characterization of tidal features from mock images.  MNRAS, 513(1):1459–1487.
  * Martin et al., (2018) Martin, G., Kaviraj, S., Devriendt, J. E. G., Dubois, Y., and Pichon, C. (2018).  The role of mergers in driving morphological transformation over cosmic time.  MNRAS, 480(2):2266–2283.
  * Martin et al., (2020) Martin, G., Kaviraj, S., Hocking, A., Read, S. C., and Geach, J. E. (2020).  Galaxy morphological classification in deep-wide surveys via unsupervised machine learning.  MNRAS, 491(1):1408–1426.
  * McInnes et al., (2018) McInnes, L., Healy, J., and Melville, J. (2018).  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.  arXiv e-prints, page arXiv:1802.03426.
  * Pearson et al., (2019) Pearson, W. J., Wang, L., Trayford, J. W., Petrillo, C. E., and van der Tak, F. F. S. (2019).  Identifying galaxy mergers in observations and simulations with deep learning.  A&A, 626:A49.
  * Robotham et al., (2014) Robotham, A. S. G., Driver, S. P., Davies, L. J. M., Hopkins, A. M., Baldry, I. K., Agius, N. K., Bauer, A. E., Bland-Hawthorn, J., Brough, S., Brown, M. J. I., Cluver, M., De Propris, R., Drinkwater, M. J., Holwerda, B. W., Kelvin, L. S., Lara-Lopez, M. A., Liske, J., López-Sánchez, Á. R., Loveday, J., Mahajan, S., McNaught-Roberts, T., Moffett, A., Norberg, P., Obreschkow, D., Owers, M. S., Penny, S. J., Pimbblet, K., Prescott, M., Taylor, E. N., van Kampen, E., and Wilkins, S. M. (2014).  Galaxy And Mass Assembly (GAMA): galaxy close pairs, mergers and the future fate of stellar mass.  MNRAS, 444(4):3986–4008.
  * Sheen et al., (2012) Sheen, Y.-K., Yi, S. K., Ree, C. H., and Lee, J. (2012).  Post-merger Signatures of Red-sequence Galaxies in Rich Abell Clusters at z ¡~0.1.  ApJs, 202(1):8.
  * Slijepcevic et al., (2022) Slijepcevic, I. V., Scaife, A., Walmsley, M., and Bowles, M. R. (2022).  Learning useful representations for radio astronomy “in the wild” with contrastive learning.  In Machine Learning for Astrophysics, page 53.
  * Slijepcevic et al., (2023) Slijepcevic, I. V., Scaife, A. M. M., Walmsley, M., Bowles, M., Wong, O. I., Shabala, S. S., and White, S. V. (2023).  Radio Galaxy Zoo: Building a multi-purpose foundation model for radio astronomy with self-supervised learning.  arXiv e-prints, page arXiv:2305.16127.
  * Snyder et al., (2019) Snyder, G. F., Rodriguez-Gomez, V., Lotz, J. M., Torrey, P., Quirk, A. C. N., Hernquist, L., Vogelsberger, M., and Freeman, P. E. (2019).  Automated distant galaxy merger classifications from Space Telescope images using the Illustris simulation.  MNRAS, 486(3):3702–3720.
  * Stein et al., (2022) Stein, G., Blaum, J., Harrington, P., Medan, T., and Lukić, Z. (2022).  Mining for Strong Gravitational Lenses with Self-supervised Learning.  ApJ, 932(2):107.
  * Stein et al., (2021) Stein, G., Harrington, P., Blaum, J., Medan, T., and Lukic, Z. (2021).  Self-supervised similarity search for large scientific datasets.  arXiv e-prints, page arXiv:2110.13151.
  * Tal et al., (2009) Tal, T., van Dokkum, P. G., Nelan, J., and Bezanson, R. (2009).  The Frequency of Tidal Features Associated with Nearby Luminous Elliptical Galaxies From a Statistically Complete Sample.  AJ, 138(5):1417–1427.
  * Walmsley et al., (2019) Walmsley, M., Ferguson, A. M. N., Mann, R. G., and Lintott, C. J. (2019).  Identification of low surface brightness tidal features in galaxies using convolutional neural networks.  MNRAS, 483(3):2968–2982.
  * Walmsley et al., (2022) Walmsley, M., Slijepcevic, I., Bowles, M. R., and Scaife, A. (2022).  Toward Galaxy Foundation Models with Hybrid Contrastive Learning.  In Machine Learning for Astrophysics, page 29.
  * Wei et al., (2022) Wei, S., Li, Y., Lu, W., Li, N., Liang, B., Dai, W., and Zhang, Z. (2022).  Unsupervised Galaxy Morphological Visual Representation with Deep Contrastive Learning.  PASP, 134(1041):114508.