# Putting Natural in Natural Language Processing

Grzegorz Chrupała
Department of Cognitive Science and Artificial Intelligence
Tilburg University
<EMAIL_ADDRESS>
###### Abstract

Human language is firstly spoken and only secondarily written. Text, however,
is a very convenient and efficient representation of language, and modern
civilization has made it ubiquitous. Thus the field of NLP has overwhelmingly
focused on processing written rather than spoken language. Work on spoken
language, on the other hand, has been siloed off within the largely separate
speech processing community which has been inordinately preoccupied with
transcribing speech into text. Recent advances in deep learning have led to a
fortuitous convergence in methods between speech processing and mainstream
NLP. Arguably, the time is ripe for a unification of these two fields, and for
starting to take spoken language seriously as the primary mode of human
communication. Truly natural language processing could lead to better
integration with the rest of language science and could lead to systems which
are more data-efficient and more human-like, and which can communicate beyond
the textual modality.

##  1 Introduction

The ACL 2023 theme track urges the community to check the reality of the
progress in NLP. This position paper adopts an expansive interpretation of
this question. It is definitely worth inquiring into the apparent advances of
current NLP in their own terms. Here, however, I question these terms and
argue that our field has focused on only a limited subset of human language
which happens to be convenient to work with, and thus misses major aspects of
human communication.

###  1.1 Human Language is Primarily Spoken

Humans are an exceptional species in many ways, and out of these, human
language is one of the most salient. Unlike communication systems used by
other organisms, human language is open-ended, capable of expressing abstract
concepts, and of reference to events displaced in time and space. While the
capacity to acquire language is universal and largely innate (Darwin, 1874;
Pinker and Bloom, 1990) it also is culturally mediated and likely arose via
gene-culture co-evolution (Deacon, 1998; Richerson and Boyd, 2010).

One revolutionary technology which turbo-charged human language was writing,
which was invented a handful of times in the most recent few thousand years of
the human story (Fischer, 2003). Writing, followed by the printing press,
followed by the Internet, have made written text ubiquitous to the extent that
it is easy to forget that the primary and universal modality for most human
communication throughout history has been spoken.111I am using _spoken
language_ in the broad sense here, including both the oral and gestural
(signed) modes of expression, and opposing these to the written modality.

Even today many of the world’s languages do not have a standardized written
form. For those that do, the written modality originated as a compressed,
symbolic representation of the spoken form.

Children acquire a spoken language (and not infrequently two or more) within
the first few years of their life with no or little explicit instruction,
largely relying on weak, noisy supervision via social interaction and
perceptual grounding. In contrast, they require hundreds of hours of explicit
instruction and arduous conscious practice to learn to read and write, and
most are only able to learn the written modality a couple of years at best
after becoming fluent communicators in one or more spoken languages.

###  1.2 Reality check

Thus, arguably, the natural language for which we are biologically equipped is
spoken. Written language is a secondary development, which happens to be very
useful and widespread, but is nevertheless derivative of speech. This appears
to be the consensus view in linguistics going back at least a century (de
Saussure, 1916; Bloomfield, 1933).222However see Aaron and Joshi (2006) for a
dissenting view.

Given these facts, is then the field of Natural Language Processing (NLP) a
misnomer? Are we making less progress with getting machines to communicate via
human language than current advances with processing written text would have
us believe?

##  2 NLP is Written Language Processing

To anyone with experience reading, reviewing and publishing papers in NLP
conferences and journals (such the ACL conferences and TACL) it is evident
that the field is very strongly focused on processing written language. While
this is evident to practitioners, it is also largely tacit and implicit.

###  2.1 Unstated assumptions

The fact that a paper is concerned with written as opposed to spoken oral or
sign language is almost invariably assumed to be the default and not
explicitly stated. Furthermore, even if there is some interest in tackling a
dataset of originally spoken language (for example in much work on dialog and
child language acquisition), the usual approach is to use a written
transcription of this data rather than the actual audio. This is partly a
matter of convenience, but partly due to the assumption that the written form
of language is the canonical one while the audio modality is just a weird,
cumbersome encoding of it.

To some extent such an implicit belief also lurks in much work within the
speech community: the main thrust of speech research has always been on so
called Automatic Speech Recognition (ASR), by which is meant automatically
transcribing spoken language into a written form. Written text is treated as
an interface and an abstraction barrier between the field of speech processing
and NLP. In Sections 3 and 4 I address problems arising from the above
assumptions, as well as the challenges and opportunities we have once we
discard them. Firstly, however, it will be instructive to briefly quantify the
assertion that NLP is Written Language Processing. by looking at historical
publication patterns.

###  2.2 Publication patterns

Figure 1: The proportion of papers in the ACL anthology up to year 2022 which
mention the words speech, spoken or audio in the title, excluding those with
part(s)-of-speech or speech act(s).

Figure 1 shows the proportion of NLP papers explicitly mentioning speech-
related terms in their title over the years covered by the ACL anthology (1950
through 2022), which is a comprehensive database of NLP papers from a wide
variety of relevant conferences, workshops and
journals.333https://aclanthology.org/ The fraction of speech-focused NLP
papers varies quite a bit over the years, but mostly stays below 10%. There is
a large peak going to 20% in 1989, followed by three years with around 10% of
speech papers. A look at the underlying data reveals that the 1989 peak is
associated with the inclusion in the anthology of the proceedings of the
Speech and Natural Language Workshop (Hirshman, 1989) organized by the US
Defense Advanced Research Projects Agency (DARPA), and featuring 79 papers.
This workshop ran until 1992 and is thus largely responsible for the four-year
run of sizable representation of spoken language research in the ACL
anthology.

The overview of the last edition of this event notes the then ongoing
“paradigm shift in natural language processing towards empirical, corpus based
methods” (Marcus, 1992). It is likely that this shift in NLP methodology was
at least partly driven by this workshop, the associated DARPA program, and the
resulting increased interaction between researchers working on spoken and
written language.

In recent years (since 2010) the proportion of NLP papers explicitly
mentioning spoken language has resolutely stayed below 6%. While the major ACL
events typically include speech processing as a topic in their calls for
papers, as well as a track including the term speech in its name, such as
Speech and Multimodality, processing of spoken language it clearly a rather
minor concern of these conferences. Instead, speech work is published in
different venues organized by a separate speech processing community.

##  3 Spoken Language is Richer

While the primacy of the spoken modality as means of communication is the
consensus view in linguistics, Section 2.1 identifies unstated assumptions
among NLP practitioners which amount to the opposite view. Here I outline why
these assumptions contradicting the scientific view are not only incorrect but
also detrimental to progress on understanding and processing real human
language.

###  3.1 Key features of spoken language

Speech and writing are two different modalities with different affordances,
and there is no straightforward mapping between them. Some writing systems
such as those used for English, Arabic or Chinese do not even represent the
phonology of the spoken language in a direct way. More crucially, writing only
captures a small proportion of the information carried in the equivalent audio
signal. Writing discards most of the information falling within the general
category of paralinguistic phenomena, such as that related to speaker
identity, speaker emotional state and attitude; likewise, information conveyed
by speech tempo and amplitude, including most of suprasegmental phonology such
as intonation and rhythm is typically not present in writing. In addition to
the auditory signal, oral spoken language can also feature visual clues in the
form of accompanying gestures, facial expressions and body posture. Sign
languages rely on the visual channel exclusively, and in fact there are no
widely used writing systems for any of them (Grushkin, 2017). Unlike most
text, speech also typically contains a variable amount of channel noise
(Shannon, 1948) such as environmental sounds.

Natural spontaneous speech contains fillers, hesitations, false starts,
repairs and other disfluencies (Dinkar et al., 2023) which are usually edited
out in the written form of language. Even more critically, spontaneous speech
typically takes the form of a dialog between two or more participants. Dialog
is unlike common written genres: crucially it features turn-taking behavior
which is governed by complex and incompletely understood rules (Skantze,
2021). These features of natural dialog also mean that the traditional
cascaded approach of ASR followed by NLP faces serious limitations, not least
due to low ASR performance in this regime (Szymański et al., 2020), but also
due to its inherently interactive nature.

For all these reasons, spoken language is more informationally rich than
written language;444One exception to this general pattern is the presence of
two spatial dimensions in written language, and the role of 2D layout in
textual publications. the same factors also make it more variable, complex and
noisy, and consequently more challenging for automated processing (Shriberg,
2005). Thus any understanding of language as a human faculty gained via the
written modality does not necessarily generalize to the spoken modality. The
same is also the case about language applications: for example the successes
and shortcomings of state-of-the-art text chatbot systems (e.g. Stiennon et
al., 2020) are likely to be substantially different from those of spoken
dialog systems.

###  3.2 Challenges of speech

As an illustrative example, let us consider the effectiveness of self-
supervision: inducing representations of words and phrases from just listening
to speech or reading text. For text, this general family of methods has been
successful since around the time of Latent Semantic Analysis (Dumais, 2004),
and currently large written language models exhibit a constantly expanding
range of abilities (Wei et al., ). In contrast, self-supervision with spoken
language has met with a limited amount of success only in the last few years
(e.g. Baevski et al., 2020; Hsu et al., 2021), and these models as of now are
usually only fine-tuned on the task of ASR. One obvious difference is that
items such as words and morphemes are either explicitly delimited or easily
discovered in text, but finding them is an unsolved research problem in
speech, due to the inherent variability of this modality.

On the other hand, learning spoken language becomes much more tractable when
self-supervision is augmented with grounding in perception. The cross-modal
correlations, though unreliable and noisy, are often sufficient to
substantially facilitate the discovery and representation of words (Peng and
Harwath, 2022; Nikolaus et al., 2022) and syllables (Peng et al., 2023) in
spoken language. For written language, grounding in the visual modality has
also been found to help in some cases (e.g. Tan and Bansal, 2020) but it does
not appear crucial, as the dominance of text-only language models
demonstrates.

Since spoken language is richer in information content, it should in principle
be possible to exploit this extra signal for improving performance. One
obstacle to such developments is the increased variability and channel noise.
Perhaps less obviously, a second obstacle is that widely used benchmarks are
often designed in a way which obstructs obtaining such gains. For example the
2021 Zerospeech challenge (Dunbar et al., 2021) which aimed to benchmark
spoken language modeling, evaluates systems according to the following
criteria: phoneme discrimination, word recognition, syntactic acceptability
and correlation to human judgments of word similarities. None of these metrics
would benefit much from modeling speaker characteristics, speech tempo, pitch,
loudness or even suprasegmental phonology. Except for the first one, these
metrics would be very well suited for models trained exclusively on written
language. The combined effect of these two obstacles was evident in the
results of Zerospeech 2021 where written-language toplines, such as RoBERTa
(Liu et al., 2019), outperformed spoken language models on the latter three
metrics, often by large margins.

##  4 Unifying Speech Processing and NLP

As evident from the examples highlighted above, spoken language is in some
ways quite different from written language and presents a distinct set of
challenges and potentials. In order to understand how much progress the fields
of speech and NLP are making in understanding and implementing human language,
we need to take speech seriously qua language, not just a cumbersome modality,
and measure our progress accordingly.

###  4.1 Converging methodology

The time is ripe for a closer integration of the speech and NLP communities
and for a unified computational science of language. The set of methodologies
used in speech and text processing used to be quite distinct in the past.
Since the adoption of deep learning both fields have converged to a large
extent: currently the state-of-the-art models for both spoken and written
language rely on transformer architectures (Vaswani et al., 2017) self-trained
on large amounts of minimally preprocessed data, with optional fine-tuning.
The technical communication barriers across disciplinary boundaries are thus
much lower. The recent emergence of the concept of _textless NLP_ (Lakhotia et
al., 2021) exemplifies the potential of unifying these two fields.

###  4.2 Opportunities

The following paragraphs outline the most important benefits of making NLP
more natural, ranging from basic science to practical applications.

#### Modeling language acquisition.

An increased attention to spoken language within NLP has the potential to lead
to a more realistic understanding of how well our current methods can
replicate key human language abilities. Acquiring language under constraints
that human babies face is the big one. There is a large amount of work on
modeling human language acquisition which uses exclusively written data (at
best transcribed from the original audio). Hopefully by this point the reader
will be convinced that the relevance of this work to the actual issue under
consideration is highly questionable. We stand a much better chance of
figuring out human language acquisition if we refocus attention on spoken
language.

#### Data efficiency.

Linzen (2020) argues convincingly for language models which are human-like in
their data-efficiency and generalization capabilities. It is, however, unclear
whether these properties can even be properly evaluated via the medium of
written language. Since the informational density and the signal-to-noise
ratio in written vs spoken language are so very different, it makes little
sense to compare human children with language models trained on text.
Furthermore, the challenges of pure self-supervision may motivate us to take
seriously the impact of grounding in perception and interaction, which humans
use universally as a learning signal.

#### Unwritten languages.

Many modes of human communication lack standard written representation. These
range from major languages spoken by millions of people such as Hokkien (Mair,
2003), to small or non-standard language varieties, to sign languages.
Shifting the emphasis of NLP research from text to the primary, natural oral
and gestural modalities will benefit the communities using these varieties.

#### Spoken dialog systems.

Dingemanse and Liesenfeld (2022) argue that language technology needs to
transition from the text to talk, and provide a roadmap of how to harness
conversational corpora in diverse languages to effect such a transition.
Indeed, one of the most obvious benefits of spoken language NLP would be
dialog systems that do not need to rely on ASR and are able to exploit the
extra information lost when transcribing speech, enabling them to understand
humans better and interact with them in a more natural way.

#### Non-textual language data.

Finally, there is a large and increasing stream of non-textual language data
such as podcasts, audio chat channels and video clips. Processing such content
could also benefit from an end-to-end holistic treatment without the need of
going through the lossy conversion to text.

###  4.3 Recommendations

If you are an NLP practitioner and view spoken language as outside the scope
of your field, reconsider. Getting into speech processing does require
understanding its specifics, but it is not as technically daunting as it used
to. Conversely, if you are a speech researcher, consider that ASR and text-to-
speech is not all there is: we can get from sound to meaning and back without
going through the written word. Both fields would do well to consider the
whole of human language as their purview. Increased collaboration would
benefit both communities, and more importantly, would give us a chance of
making real progress towards understanding and simulating natural language.

##  5 Limitations

The main limitation of this paper is the one applying to any opinion piece: it
is subjective and personal, as the views of the authors are inherently limited
by their expertise and experience. More specifically, this paper argues for an
increased interaction between the speech and NLP communities, but the author
is more strongly embedded in the latter, and thus addresses this audience
primarily. Additionally, the short paper format imposes significant
constraints on the amount of nuance, detail and discussion of relevant
literature, and thus readers may find some of the claims to be less strongly
supported and less hedged than would be ideal, or proper in a longer treatment
of this topic.

## Acknowledgements

I would like to thank Hosein Mohebbi, Afra Alishahi, Mark Dingemanse, Tanvi
Dinkar, Piotr Szymański and three anonymous reviewers for their valuable
feedback on this paper.

## References

  * Aaron and Joshi (2006) P. G. Aaron and R. Malatesha Joshi. 2006.  Written language is as natural as spoken language: A biolinguistic perspective.  _Reading Psychology_ , 27(4):263–311.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460.
  * Bloomfield (1933) Leonard Bloomfield. 1933.  _Language_.  Henry Holt, New York.
  * Darwin (1874) Charles Darwin. 1874.  _The descent of man, and selection in relation to sex_.  D. Appleton and Company, New York.
  * de Saussure (1916) Ferdinand de Saussure. 1916.  _Cours de linguistique générale_.  Payot, Paris.
  * Deacon (1998) Terrence W Deacon. 1998.  _The Symbolic Species: The Co-evolution of Language and the Brain_.  WW Norton & Company.
  * Dingemanse and Liesenfeld (2022) Mark Dingemanse and Andreas Liesenfeld. 2022.  From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5614–5633, Dublin, Ireland. Association for Computational Linguistics.
  * Dinkar et al. (2023) Tanvi Dinkar, Chloé Clavel, and Ioana Vasilescu. 2023.  Fillers in spoken language understanding: Computational and psycholinguistic perspectives.  _Traitement Automatique des Langues_ , 63(3).
  * Dumais (2004) Susan T Dumais. 2004.  Latent semantic analysis.  _Annual Review of Information Science and Technology (ARIST)_ , 38:189–230.
  * Dunbar et al. (2021) Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux. 2021.  The Zero Resource Speech Challenge 2021: Spoken language modelling.
  * Fischer (2003) Steven Roger Fischer. 2003.  _History of writing_.  Reaktion books.
  * Grushkin (2017) Donald A Grushkin. 2017.  Writing signed languages: What for? What form?  _American annals of the deaf_ , 161(5):509–527.
  * Hirshman (1989) Lynette Hirshman. 1989.  Overview of the DARPA speech and natural language workshop.  In _Proceedings of the workshop on Speech and Natural Language_ , pages 1–2.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354.
  * Linzen (2020) Tal Linzen. 2020.  How can we accelerate progress towards human-like linguistic generalization? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5210–5217, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.
  * Mair (2003) Victor H Mair. 2003.  How to forget your mother tongue and remember your national language.
  * Marcus (1992) Mitchell P Marcus. 1992.  Overview of the fifth DARPA speech and natural language workshop.  In _Proceedings of the workshop on Speech and Natural Language_ , pages 3–4.
  * Nikolaus et al. (2022) Mitja Nikolaus, Afra Alishahi, and Grzegorz Chrupała. 2022.  Learning English with Peppa Pig.  _Transactions of the Association for Computational Linguistics_ , 10:922–936.
  * Peng and Harwath (2022) Puyuan Peng and David Harwath. 2022.  Word Discovery in Visually Grounded, Self-Supervised Speech Models.  In _Proc. Interspeech 2022_ , pages 2823–2827.
  * Peng et al. (2023) Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, and David Harwath. 2023.  Syllable discovery and cross-lingual generalization in a visually grounded, self-supervised speech mode.  In _Proc. Interspeech 2023_.
  * Pinker and Bloom (1990) Steven Pinker and Paul Bloom. 1990.  Natural language and natural selection.  _Behavioral and Brain Sciences_ , 13(4):707–727.
  * Richerson and Boyd (2010) Peter J Richerson and Robert Boyd. 2010.  Why possibly language evolved.  _Biolinguistics_ , 4(2-3):289–306.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Shriberg (2005) Elizabeth Shriberg. 2005.  Spontaneous speech: how people really talk and why engineers should care.  In _Proc. Interspeech 2005_ , pages 1781–1784.
  * Skantze (2021) Gabriel Skantze. 2021.  Turn-taking in conversational systems and human-robot interaction: A review.  _Computer Speech & Language_, 67:101178.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 3008–3021. Curran Associates, Inc.
  * Szymański et al. (2020) Piotr Szymański, Piotr Żelasko, Mikolaj Morzy, Adrian Szymczak, Marzena Żyła-Hoppe, Joanna Banaszczak, Lukasz Augustyniak, Jan Mizgajski, and Yishay Carmiel. 2020.  WER we are and WER we think we are.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3290–3295, Online. Association for Computational Linguistics.
  * Tan and Bansal (2020) Hao Tan and Mohit Bansal. 2020.  Vokenization: Improving language understanding with contextualized, visual-grounded supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2066–2080, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * (32) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.


# Generalization of Apollonius Circle

Ömer Avcı111Department of Electrical $\&$ Electronics Engineering, Boğaziçi
University, 34342, Bebek, Istanbul, Turkey Ömer Talip Akalın 222Department of
Computer Engineering, Boğaziçi University, 34342, Bebek, Istanbul, Turkey
Faruk Avcı 333Department of Computer Engineering, Istanbul Technical
University, 34485, Maslak, Istanbul, Turkey Halil Salih Orhan444Department of
Computer Engineering, Boğaziçi University, 34342, Bebek, Istanbul, Turkey

###### Abstract

Apollonius of Perga, showed that for two given points $A,B$ in the Euclidean
plane and a positive real number $k\neq 1$, geometric locus of the points $X$
that satisfies the equation $|XA|=k|XB|$ is a circle. This circle is called
Apollonius circle. In this paper we generalize the definition of the
Apollonius circle for two given circles $\Gamma_{1},\Gamma_{2}$ and we show
that geometric locus of the points $X$ with the ratio of the power with
respect to the circles $\Gamma_{1},\Gamma_{2}$ is constant, is also a circle.
Using this we generalize the definition of Apollonius Circle, and generalize
some results about Apollonius Circle.

##  1 Preliminaries

###### Theorem 1.1

(Apollonius Theorem) For points $A,B$ in the Euclidean plane, and a positive
real number $k\neq 1$, the points $X$ which satisfies the equation

$|XA|=k|BX|$

forms a circle. When $k=1$, they form the line perpendicular to $AB$ at the
middle point of $[AB]$ [1].

Figure 1: An example of the Apollonius Thorem for fixed points $A,B$ and $k=2$

###### Definition 1.1

For three different points $A,B,C$ in Euclidean plane such that $A$ is not on
the perpendicular bisector of the segment $[BC]$. We will use the notation
$K_{A}(B,C)$ for the circle which consists of points that holds the equation

$\frac{|XB|}{|XC|}=\frac{|AB|}{|AC|}$

and $M_{A}(B,C)$ for the center and $r_{A}(B,C)$ for the radius of that
circle. During this article, we will call the notation $K_{A}(B,C)$,
Apollonius Circle of point $A$ to the points $B,C$.

Figure 2: A brief explanation of the Definiton 1.1

###### Definition 1.2

For a point $A$ and a circle $\Gamma$ with center $O$ and radius $r$ in the
Euclidean plane. The power of the point A with respect to circle $\Gamma$ is
shown by $P_{\Gamma}(A)$ and it is equal to:

$P_{\Gamma}(A)=|OA|^{2}-r^{2}$

Figure 3: Power of point A to the circle C

###### Definition 1.3

For two circles $\Gamma_{1},\Gamma_{2}$ with the centers $O_{1},O_{2}$
respectively, in the Euclidean space the points with equal power to the
circles i.e. the points $X$ satisfying the equation

$P_{\Gamma_{1}}(X)=P_{\Gamma_{2}}(X)$

forms a line which is perpendicular to the line $O_{1}O_{2}$, and it is called
the radical axis of the circles $\Gamma_{1},\Gamma_{2}$ [3].

Figure 4: Radical Axis of the circles $C_{1},C_{2}$

###### Theorem 1.2

For a given scalene triangle $\Delta ABC$, the line $BC$ and the bisectors of
the angle $\angle BAC$ intersects at two points $X,Y$, and those points lies
on the $K_{A}(B,C)$. [2]

Figure 5: Graphical explanation of Theorem 1.2

###### Theorem 1.3

Let $O$ be the circumcenter of the a scalene triangle $\Delta ABC$ then, the
line $AO$ is a tangent to the circle $K_{A}(B,C)$ which also means that
$AO\perp AM_{A}(B,C)$.

Figure 6: Graphical explanation of Theorem 1.3

###### Theorem 1.4

Let $\Delta ABC$ be a scalane triangle. Then the points
$M_{A}(B,C),M_{B}(C,A),M_{C}(A,B)$ are collinear. This line is called Lemoine
line [4]. Also there exists two points $S_{1},S_{2}$ and they lie on all the
circles $K_{A}(B,C),K_{B}(C,A),K_{C}(A,B)$. If $O$ is the circumcenter of the
triangle $\Delta ABC$, then the points $S_{1},S_{2},O$ are collinear.

Figure 7: Graphical explanation of Theorem 1.4

##  2 Results

###### Theorem 2.1

(Generalized Apollonius Circle Theorem) Let $\Gamma_{1},\Gamma_{2}$ be two
circles given in the plane. Let $k$ be a real number. Then the geometric locus
of the points $X$ that satisfies the equation

$P_{\Gamma_{1}}(X)=kP_{\Gamma_{2}}(X)$ (2.1)

is a circle (possibly degenerate - line, single point or non-existent)

Proof: Let $O_{1},O_{2}$ be the centers and $r_{1},r_{2}$ be the radii of the
circles $\Gamma_{1},\Gamma_{2}$ respectively. Without loss of generality let
us choose our coordinate system such that:

$O_{1}=(0,0),\;O_{2}=(|O_{1}O_{2}|,0)$

Let $X=(a,b)$ be a point that satisfies the desired relationship. Then we
have:

$(a^{2}+b^{2}-r_{1}^{2})=k((a-|O_{1}O_{2}|)^{2}+b^{2}-r_{2}^{2})$

we can write it down as:

$(k-1)a^{2}-2ak|O_{1}O_{2}|+(k-1)b^{2}=-k|O_{1}O_{2}|^{2}+kr_{2}^{2}-r_{1}^{2}$

If $k=1$ this equations yields:

$-2a=-|O_{1}O_{2}|^{2}+r_{2}^{2}-r_{1}^{2}$

from this equation we can see that $a$ is constant but $b$ can be anything
therefore the locus points is a line that is perpendicular to $O_{1}O_{2}$ and
it is known as the radical axis. If $k\neq 1$ then we can manipulate the
equation we have into a circle equation such that:

$\Big{(}a-\frac{k}{k-1}|O_{1}O|\Big{)}^{2}+b^{2}=\frac{kr_{2}^{2}-r_{1}^{2}}{k-1}+\frac{k}{(k-1)^{2}}|O_{1}O_{2}|^{2}$
(2.2)

Now we can see that it is a circle equation with center
$M=(\frac{k}{k-1}|O_{1}O|,0)$ we can see that depending on the value of the
right hand side the geometric locus will be a circle($RHS>0$) or a single
point ($RHS=0$) or non-existent ($RHS<0$). We leave the calculation of when
the right hand side of the equation is non-negative for later. Note that if
the RHS of the equation is positive then the radius of the circle will be:

$r^{2}=\frac{kr_{2}^{2}-r_{1}^{2}}{k-1}+\frac{k}{(k-1)^{2}}|O_{1}O_{2}|^{2}$
(2.3)

Note that we can observe that the center $M$ lies on the line $O_{1}O_{2}$ and
it satisfies the relationship:

$\frac{\overrightarrow{O_{1}M}}{\overrightarrow{O_{2}M}}=k=\frac{P_{\Gamma_{1}}(X)}{P_{\Gamma_{2}}(X)}$
(2.4)

###### Theorem 2.2

(Possible values of the power ratio) Let $\Gamma_{1},\Gamma_{2}$ be two
intersecting circles given in the plane. Let $k$ be a real number. Then the
geometric locus of the points $X$ that satisfies the equation

$P_{\Gamma_{1}}(X)=kP_{\Gamma_{2}}(X)$

is a circle for the all $k\in\mathbb{R}-\\{1\\}$.

Proof: Let the intersection points of the $\Gamma_{1},\Gamma_{2}$ be $A,B$.
Then we have:

$P_{\Gamma_{1}}(A)=P_{\Gamma_{2}}(A)=P_{\Gamma_{1}}(B)=P_{\Gamma_{2}}(B)=0$

We have shown that the equation 2.2 is a circle equation. We also know that it
is not a line when $k\neq 1$. We can clearly see that the points $A,B$
satisfies the the equation 2.1, therefore there is at least two points
satisfying the circle equation 2.2 which means that it can not be a single
point or non-existent therefore it must be a circle.

###### Theorem 2.3

(Possible values of the power ratio) Let $\Gamma_{1},\Gamma_{2}$ be two
circles that are not intersecting at two different points. Let $k$ be a real
number. Then the geometric locus of the points $X$ that satisfies the equation

$P_{\Gamma_{1}}(X)=kP_{\Gamma_{2}}(X)$

is:

  1. (i)

A line when $k=1$

  2. (ii)

A single point if

$k=\frac{r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2}\pm\sqrt{(r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2})^{2}-4r_{1}^{2}r_{2}^{2}}}{2r_{2}^{2}}$

  3. (iii)

A circle if

$\frac{r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2}+\sqrt{(r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2})^{2}-4r_{1}^{2}r_{2}^{2}}}{2r_{2}^{2}}<k$

or

$k<\frac{r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2}-\sqrt{(r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2})^{2}-4r_{1}^{2}r_{2}^{2}}}{2r_{2}^{2}}$

  4. (iv)

Non-existent otherwise

Proof: In the equation 2.2 we stated that the shape of the geometric locus is
dependent of whether the right hand side is positive, zero or negative. We
stated that if it is positive the locus is a circle, if it is zero locus
consists of only a single point and if it is negative there is no point on the
locus. So let us find when it is positive or negative. If we make equal the
denominators they become $(k-1)^{2}$ which is positive so we need to control
nominator and it is equal to:

$(k-1)(kr_{2}^{2}-r_{1}^{2})+k|O_{1}O_{2}|^{2}=k^{2}r_{2}^{2}+k(|O_{1}O_{2}|^{2}-r_{1}^{2}-r_{2}^{2})+r_{1}^{2}$

We can see that it is a quadratic equation($k$ as a variable and $r_{1},r_{2}$
as constants) with positive leading coefficient. Then the equation is negative
between the roots and positive otherwise. So let us find the roots of the
equation. We find the roots as:

$k_{1,2}=\frac{r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2}\pm\sqrt{(r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2})^{2}-4r_{1}^{2}r_{2}^{2}}}{2r_{2}^{2}}$

We can show that real roots exists because the discriminant is positive. This
is because we assume that the circles $\Gamma_{1},\Gamma_{2}$ does not
intersect at two distinct points therefore we either have $|O_{1}O_{2}|\geq
r_{1}+r_{2}$ or $|O_{1}O_{2}|\leq|r_{1}-r_{2}|$. Then we have:

$(r_{1}+r_{2})^{2}\geq|O_{1}O_{2}|^{2}\Rightarrow
r_{1}^{2}+r_{2}^{2}+2r_{1}r_{2}-|O_{1}O_{2}|^{2}\geq 0$

and

$(r_{1}-r_{2})^{2}\geq|O_{1}O_{2}|^{2}\Rightarrow
r_{1}^{2}+r_{2}^{2}-2r_{1}r_{2}-|O_{1}O_{2}|^{2}\geq 0$

therefore

$(r_{1}^{2}+r_{2}^{2}-|O_{1}O_{2}|^{2})^{2}-4r_{1}^{2}r_{2}^{2}\geq 0$

so we are done.

###### Definition 2.1

Here we will generalize the Definition 1.1. Let $A$ be a point and
$\Gamma_{1},\Gamma_{2}$ be two circles in the plane. Then we know that the
geometric locus of the points $X$ satisfying the equation:

$\frac{P_{\Gamma_{1}}(X)}{P_{\Gamma_{2}}(X)}=\frac{P_{\Gamma_{1}}(A)}{P_{\Gamma_{2}}(A)}$

is a circle(possibly degenerate). We will show this circle with
$K_{A}(\Gamma_{1},\Gamma_{2})$ and we will call it Apollonius circle of point
$A$ to the circles $\Gamma_{1},\Gamma_{2}$. Also let us denote the center of
this circle as $M_{A}(\Gamma_{1},\Gamma_{2})$ and the radius of the circle as
$r_{A}(\Gamma_{1},\Gamma_{2})$.

Figure 8: An example of generalized Apollonius circle

###### Corollary 2.1

If there is a point $S$ that lies on both $\Gamma_{1}$ and $\Gamma_{2}$ then
$S$ also lies on the $K_{A}(\Gamma_{1},\Gamma_{2})$.

###### Corollary 2.2

If the circles $\Gamma_{1},\Gamma_{2}$ intersects at the points $S_{1},S_{2}$,
then $K_{A}(\Gamma_{1},\Gamma_{2})$ is equivalent to the circumcircle of the
triangle $\Delta S_{1}AS_{2}$.

###### Theorem 2.4

Let $\Gamma_{1},\Gamma_{2},\Gamma_{3}$ be three circles in the plane with the
centers $O_{1},O_{2},O_{3}$ and radii $r_{1},r_{2},r_{3}$ respectively. Then
the points $M_{O_{1}}(\Gamma_{2},\Gamma_{3})$,
$M_{O_{2}}(\Gamma_{3},\Gamma_{1})$ and $M_{O_{3}}(\Gamma_{1},\Gamma_{2})$ are
collinear if and only if the following equation holds:

$\sum_{i=1}^{3}|O_{i}O_{i+1}|^{2}|O_{i}O_{i+2}|^{2}(r_{i+1}^{2}-r_{i+2}^{2})=\sum_{i=1}^{3}r_{i+1}^{2}r_{i+2}^{2}(|O_{i}O_{i+1}|^{2}-|O_{i}O_{i+2}|^{2})$
(2.5)

where $O_{i+3}=O_{i}$ and $r_{i+3}=r_{i}$ for $i=1,2,3$.

Proof: In this theorem we present a generalization of the Lemoine line for the
generalized definition of the Apollonius circle. For convenience let us show
$M_{O_{i}}(\Gamma_{i+1},\Gamma_{i+2})$ with $M_{i}$ for $i=1,2,3$ (note that
$\Gamma_{i+3}=\Gamma_{i}$). In the proof of 2.1 we have shown that $M_{i}$
lies on the line $O_{i+1}O_{i+2}$ and from the Equation 2.4 we have:

$\frac{\overrightarrow{O_{i+1}M_{i}}}{\overrightarrow{O_{i+2}M_{i}}}=\frac{P_{\Gamma_{i+1}}(O_{i})}{P_{\Gamma_{i+2}}(O_{i})}$
(2.6)

Now from Menelaus Theorem we know that, $M_{1},M_{2},M_{3}$ are collinear if
and only if the following equation holds:

$\frac{\overrightarrow{O_{2}M_{1}}}{\overrightarrow{O_{3}M_{1}}}\frac{\overrightarrow{O_{3}M_{2}}}{\overrightarrow{O_{1}M_{2}}}\frac{\overrightarrow{O_{1}M_{3}}}{\overrightarrow{O_{2}M_{3}}}=1$

From the Equation 2.4 we can transform this equation into this:

$\frac{P_{\Gamma_{2}}(O_{1})}{P_{\Gamma_{3}}(O_{1})}\frac{P_{\Gamma_{3}}(O_{2})}{P_{\Gamma_{1}}(O_{2})}\frac{P_{\Gamma_{1}}(O_{3})}{P_{\Gamma_{2}}(O_{3})}=1$
(2.7)

and we also know that $P_{\Gamma_{j}}(O_{i})=|O_{i}O_{j}|^{2}-r_{j}^{2}$
therefore the equation transform into:

$\frac{\Big{(}|O_{1}O_{2}|^{2}-r_{2}^{2}\Big{)}}{\Big{(}|O_{1}O_{3}|^{2}-r_{3}^{2}\Big{)}}\frac{\Big{(}|O_{2}O_{3}|^{2}-r_{3}^{2}\Big{)}}{\Big{(}|O_{2}O_{1}|^{2}-r_{1}^{2}\Big{)}}\frac{\Big{(}|O_{3}O_{1}|^{2}-r_{1}^{2}\Big{)}}{\Big{(}|O_{3}O_{2}|^{2}-r_{2}^{2}\Big{)}}=1$
(2.8)

and if we rearrange this, we get the desired equation.

###### Corollary 2.3

Let $\Gamma_{1},\Gamma_{2},\Gamma_{3}$ be three circles in the plane with the
centers $O_{1},O_{2},O_{3}$ and radii $r_{1},r_{2},r_{3}$ respectively. If the
centers $O_{1},O_{2},O_{3}$ forms an equilateral triangle, or all three of the
radii of the circles $\Gamma_{1},\Gamma_{2},\Gamma_{3}$ are equal then the
points $M_{O_{1}}(\Gamma_{2},\Gamma_{3})$, $M_{O_{2}}(\Gamma_{3},\Gamma_{1})$
and $M_{O_{3}}(\Gamma_{1},\Gamma_{2})$ are collinear.

Proof: It is clear that if we have

$|O_{1}O_{2}|=|O_{2}O_{3}|=|O_{3}O_{1}|$

or

$r_{1}=r_{2}=r_{3}$

both sides of the equation 2.5 is equal to zero and therefore the equation
holds.

Figure 9: Theorem 2.4 for three equal-radius circles centered at $A,B,C$
Figure 10: Theorem 2.4 when centers of the circles form an equilateral
triangle

###### Theorem 2.5

Let $\Gamma_{1},\Gamma_{2},\Gamma_{3}$ be three circles in the plane with the
centers $O_{1},O_{2},O_{3}$ and radii $r_{1},r_{2},r_{3}$ respectively. Let
$O$ be the circumcenter of the triangle $\Delta O_{1}O_{2}O_{3}$. Assume that
the points $M_{O_{1}}(\Gamma_{2},\Gamma_{3})$,
$M_{O_{2}}(\Gamma_{3},\Gamma_{1})$ and $M_{O_{3}}(\Gamma_{1},\Gamma_{2})$ are
collinear. Let the line $l_{i}$ be the radical axis of the circles
$K_{O_{i+1}}(\Gamma_{i+2},\Gamma_{i+3})$ and
$K_{O_{i+2}}(\Gamma_{i+3},\Gamma_{i+1})$ for $i=1,2,3$. Then the lines
$l_{1},l_{2},l_{3}$ are the same line and this line passes through the point
$O$.

Proof: For convenience let us show the center
$M_{O_{i}}(\Gamma_{i+1},\Gamma_{i+2})$ with $M_{i}$ and the circles
$K_{O_{i}}(\Gamma_{i+1},\Gamma_{i+2})$ with $K_{i}$ and
$r_{O_{i}}(\Gamma_{i+1},\Gamma_{i+2})$ with $R_{i}$ for $i=1,2,3$. We know
that radical axis of two circles is perpendicular to the line passing through
the centers of the circles. And since $M_{1},M_{2},M_{3}$ are collinear then
all of the lines $l_{1},l_{2},l_{3}$ are perpendicular to the line passing
through $M_{1},M_{2},M_{3}$. Therefore $l_{1}\parallel l_{2}\parallel l_{3}$
which means that they coincide or never intersect. If we show that all of this
three lines pass through $O$ then we also show that they are the same line. It
is sufficient to show that $O\in l_{1}$. From the definition of radical axis
we know that:

$O\in l_{1}\Leftrightarrow P_{K_{2}}(O)=P_{K_{3}}(O)$

Let $N_{1},N_{2},N_{3}$ be the middle points of the
$[O_{2}O_{3}],[O_{3}O_{1}],[O_{1}O_{2}]$ respectively. Also let $r$ be the
radius of the circle $(O_{1}O_{2}O_{3})$. We know that $M_{2}\in O_{1}O_{3}$
and $ON_{2}\perp O_{1}O_{3}$, then from Pythagoras Theorem we have:

$|OM_{2}|^{2}=|M_{2}N_{2}|^{2}+|ON_{2}|^{2}$

and also we have:

$r^{2}=|OO_{1}|^{2}=|O_{1}N_{2}|^{2}+|ON_{2}|^{2}$

therefore:

$|ON_{2}|^{2}=r^{2}-\frac{1}{4}|O_{1}O_{3}|^{2}$

so we have:

$|OM_{2}|^{2}=|M_{2}N_{2}|^{2}+r^{2}-\frac{1}{4}|O_{1}O_{3}|^{2}$

then we will calculate $|M_{2}N_{2}|$, let us say:

$\frac{\overrightarrow{O_{1}M_{2}}}{\overrightarrow{O_{3}M_{2}}}=k$

From Equation 2.4 we know that:

$k=\frac{|O_{2}O_{1}|^{2}-r_{1}^{2}}{|O_{2}O_{3}|^{2}-r_{3}^{2}}$

Since $N_{2}$ is the middle point of $O_{1}O_{3}$, we have:

$\frac{\overrightarrow{O_{1}N_{2}}}{\overrightarrow{O_{3}N_{2}}}=-1$

By combining them we get:

$\frac{\overrightarrow{M_{2}N_{2}}}{\overrightarrow{O_{3}O_{1}}}=\frac{k+1}{2(k-1)}$

therefore we have:

$|M_{2}N_{2}|^{2}=\Big{(}\frac{k+1}{2(k-1)}\Big{)}^{2}$

and then we get:

$|OM_{2}|^{2}=|M_{2}N_{2}|^{2}-|O_{1}N_{2}|^{2}+|OO_{1}|^{2}=|O_{1}O_{3}|^{2}\frac{k}{(k-1)^{2}}+r^{2}$

From the definition of power we have:

$P_{K_{2}}(O)=|OM_{2}|^{2}-R_{2}^{2}$

From Equation 2.3 we know that:

$R_{2}^{2}=\frac{k}{(k-1)^{2}}|O_{1}O_{3}|^{2}+\frac{kr_{3}^{2}-r_{1}^{2}}{k-1}$

So by combining them we get:

$P_{K_{2}}(O)=r^{2}+\frac{kr_{3}^{2}-r_{1}^{2}}{k-1}$

and if we put
$k=\frac{|O_{2}O_{1}|^{2}-r_{1}^{2}}{|O_{2}O_{3}|^{2}-r_{3}^{2}}$, we finally
get:

$P_{K_{2}}(O)=r^{2}+\frac{(|O_{1}O_{2}|^{2}-r_{1}^{2})r_{3}^{2}-(|O_{2}O_{3}|^{2}-r_{3}^{2})r_{1}^{2}}{|O_{1}O_{2}|^{2}+r_{3}^{2}-|O_{2}O_{3}|^{2}-r_{1}^{2}}$

and then we can see that $P_{K_{2}}(O)=P_{K_{3}}(O)$ if and only if:

$\frac{(|O_{1}O_{2}|^{2}-r_{1}^{2})r_{3}^{2}-(|O_{2}O_{3}|^{2}-r_{3}^{2})r_{1}^{2}}{|O_{1}O_{2}|^{2}+r_{3}^{2}-|O_{2}O_{3}|^{2}-r_{1}^{2}}=\frac{(|O_{1}O_{3}|^{2}-r_{1}^{2})r_{2}^{2}-(|O_{2}O_{3}|^{2}-r_{2}^{2})r_{1}^{2}}{|O_{1}O_{3}|^{2}+r_{2}^{2}-|O_{2}O_{3}|^{2}-r_{1}^{2}}$

and if we rearrange the equation we can see that it is equivalent to the
equation 2.5.

## References

  * [1] Prasolov,V.(2001), Problems in Plane and Solıd Geometry-v.1 Plane Geometry, Moscow.
  * [2] Altshiller, N.(1915), On the Circles of Apollonius, The American Mathematical Monthly, Volume 22, No: 8, 261-263.
  * [3] Coxeter, H.S.M.(1967), Geometry Revisited, The Mathematical Association of America, Washington.
  * [4] Weisstein, E.(1999), Encyclopedia of Mathematics, CRC Press, Volume 1-2 , Florida.


In this section I outline three approaches to stance detection: supervised
classifiers, zero-shot NLI classifiers, and in-context learning. My goal is
not to provide step-by-step instructions, but to present the information
necessary for researchers to make informed decisions about their methods and
design. As a running demonstration, I classify approval for President Trump on
Twitter. I compiled multiple data sets collected with various parameters and
labeling techniques. The data sets and sample sizes are listed in the
appendix. As a summary performance metric, I use Matthew’s Correlation
Coefficient (MCC), an emerging standard for binary classification performance
due to its robustness relative to F1 (chicco2020advantages). MCC ranges from
-1 to 1 with 0 indicating no correlation between true class and estimated
class. I use a 70-30 train-test split and results shown are performance on the
testing data. Figure 1 provides a top-line comparison across approaches.

Figure 1: Comparison across methods of stance detection. Stance detection
approaches are largely comparable with the obvious exception of sentiment
dictionaries.

##  0.1 Supervised Classifiers

Supervised classifiers can achieve performance equivalent to humans on
entailment classification tasks like stance detection (superglue). Their
primary shortcoming is that they are task specific and require a lot of
training data relative to other methods. Thus, consider a supervised
classifier if your stance detection task is narrow in scope and can afford to
curate manually labeled training data. Because supervised text classification
is a well established method, I forego a basic introduction to their use and
instead focus on decisions of particular importance to entailment
classification.

###  0.1.1 Model Selection

Supervised classifiers can be divided into two types: machine learning
algorithms that classify documents as unordered ”bags-of-words” (e.g. logistic
regression), and large language models that use word embeddings to classify
semantic representations of text (e.g. BERT). Language models provide the best
starting point for most stance detection tasks for two reasons. First,
language models generally out-preform machine learning algorithms (e.g.
gonzalez2020comparing).Language models use semantic representations of
documents to classify them rather than word counts (mikolov2013efficient).
This provides a more theoretically robust foundation that is particularly
relevant for a task like stance detection. Second, language models are both
easier to implement and better maintain the integrity of data. Converting
documents to a bag-of-words requires text pre-processing that can increase the
labor for researchers and alter the outcome of analysis (denny2018text).
Language models work with unedited documents and reduce the decision points
that can influence results.

State-of-the-art models, typically a type of neural network known as a
transformer (vaswani2017attention), are used through transfer-learning. In
this process, a model is pre-trained on large data sets and then released to
the public so others can train the model on task-specific data. Thousands of
these pre-trained models are publicly available through Python’s Transformers
library (wolf-etal-2020-transformers), and many are pre-trained for specific
document types or common research tasks (e.g. bertweet). In practice, this
means large models can be trained by practitioners in a matter of minutes with
a desktop GPU or a free cloud service such as Google Colab. Software packages
like Weights and Biases further ease the training process by automatically
sweeping the hyper-parameter space (wandb).

While many researchers use BERT (devlin2018bert) and RoBERTa (liu2019roberta)
models as a default, Electra (clark2020electra) and DeBERTa (he2020deberta)
models are better starting points if there is not a model already pre-trained
for your task. Electra uses a more effective training procedure than BERT,
making it both faster and less prone to overfitting (clark2020electra). For
more complicated tasks, DeBERTa significantly improves upon BERT in speed and
learning capacity (he2020deberta).

###  0.1.2 Classifier Knowledge: Training Samples

Supervised classifiers offer significant control over information context via
classier knowledge. The most obvious vector to control this through is the
training data. Researchers should pay particular attention to how they sample
their training data and who their manual labelers are. A common practice is to
label training data with crowd-workers. joseph2017constance however, showed
that when crowd workers lack context both intercoder reliability and
classifier performance suffer. This can be somewhat ameliorated by providing
examples of correctly labeled text (e.g. kawintiranon2021knowledge) or a
contextual prompt (e.g. joseph2017constance). Occasionally, crowd sourcing may
not be appropriate because the amount of information needed to infer accurate
labels is more than people can reasonably internalize in a short period of
time. In such instances, training one or two expert coders may be a more
appropriate approach. Discordance between the context known to labelers and
the context needed for accurate labels places a ceiling on your classifier’s
potential by reducing certainty in training labels (miller2020active;
joseph2017constance).

###  0.1.3 Classifier Knowledge: Domain Adaptation

Some projects may need to stance classify documents that use particular
language, such as legal or medical documents. In such cases, domain adaptation
can enhance classifier knowledge. Domain adaptation involves training a model
for a generic language task, such as predicting missing words in a sentence,
on data relevant to the domain. This adapted model can then be fine tuned on
specific tasks for better performance in that domain. Many models that have
already been domain adapted are available via Python’s Transformer’s library
(e.g. chalkidis2020legal).

To illustrate, I trained three RoBERTa models with varying levels of domain
adaptation to classify my test set: RoBERTa base (no adaptation), BERTweet
(adapted to Twitter) and PoliBERTweet (adapted to political
Twitter)(kawintiranon2022polibertweet). For each model I used a random search
of the hyperparameter space to train thirty iterations. 111To minimize
degeneracy I used a Bayesian search of the learning rate and number of
training epochs to find a range in which the model was stable. The random
search was conducted in this parameter range. Three models failed to converge
across the 90 runs. These models are omitted, but including them does not
change results

Figure 2: RoBERTa, BERTweet, and PoliBERTweet share the same model
architecture but differ domain adaptation. Each model was trained 30 times
using a random search of a bounded hyperparameter space. PoliBERTweet
demonstrated consistently better performance due to its domain adaptation.

Results are shown in figure 2. There is weak evidence ($t=1.32$) BERTweet
(Mean MCC $=0.56$) performs better than RoBERTa base (Mean MCC $=0.53$).
PoliBERTweet, however, showed significant ($t=7.35$) improvement over both
models (Mean MCC $=0.65$). However, this provides only a partial view of how
domain adaptation affects classification. Some models that preform well on
testing data show evidence of over-fitting and thus fail to generalize well to
the entire data set. One sign of over-fitting is evaluation loss, a
measurement of error in classifying the testing data. Higher evaluation loss
can indicate the model will generalize poorly. Figure 3 plots MCC against the
evaluation loss. Models in the top left corner indicate high performance with
less evidence of over-fitting. Here, the advantage of domain adaptation is
more evident with PoliBERTweet showing better performance and lower evaluation
loss.

Figure 3: Thirty models of each type were trained using a random search of a
bounded hyperparameter space. Each iteration is plotted by overall performance
(MCC) and generalization to out of sample data (Evaluation Loss). The size
correlates with the number of training epochs. PoliBERTweet demonstrates both
better performance and generalization due to domain adaptation.

###  0.1.4 Validation

Out of sample performance is the primary method of validation for supervised
classifiers. Additionally, intercoder reliability between human coders can
provide insight as to what the performance ceiling for a model is. Researchers
should also demonstrate they have adequately searched the hyperparameter space
and that models converge to consistent performance. This provides evidence
that results are due to consistent measurement rather than the artifact of a
particular model.

Supervised Classifiers for Stance Detection Supervised classifiers provide
state-of-the-art performance and control over classifier knowledge. They are a
good fit for research questions that need to classify few stances and can
collect training data. Selecting a Model • Transformers generally preform best
and are simpler to implement. • Thousands of pre-trained models are publicly
available. If a model has not been pre-trained for your specific task, Electra
and DeBERTa are good starting points. Controlling Information Context • Ensure
manual labelers have sufficient context to accurately label documents. • If
using crowd workers, provide examples of correctly labeled data or context
prompts. • When possible, train a model adapted to your domain of interest.
Validation • Out of sample prediction is the primary test of validity. •
Measures of inter-coder reliability can estimate the model’s performance
ceiling. • Demonstrate a sweep of the hyperparameter space and convergence on
consistent performance.

##  0.2 Zero-Shot NLI Classifiers

NLI classifiers are transformers pre-trained specifically for recognizing
textual entailment. During training, the model is presented a statement and
hypothesis pair and is tasked to determine if the statement entails,
contradicts, or is unrelated/neutral to the hypothesis. The generalizability
of this task allows these models to classify data with no additional training
(also known as zero-shot classification) by pairing documents with hypotheses
and framing classification as an entailment problem (yin2019benchmarking). For
example, to identify tweets that support Trump I might pair each tweet with
the hypothesis “The author of this tweet supports Trump” and the model would
then determine if each tweet entails that hypothesis. An alternative approach
is to devise a set of hypothesis that represent potential labels (e.g.
“…supports Trump”, “…opposes Trump”, etc.) and then classify the document with
each hypothesis in the set. The hypothesis most likely to be entailed is the
document’s label.

A zero-shot model works off-the-shelf on an arbitrary number of stance
detection tasks without training a model for each task. In many contexts, a
zero-shot NLI classifier performs on par or better than supervised
classifiers. Their primary limitation is that the users have no control over
classifier knowledge other than selecting which model will be used. This
implies zero-shot classifiers may struggle to classify data that is less
contextually complete. Accordingly, A zero-shot NLI classifier is ideal for
stances that can be framed in well-defined dimensions such as support or
opposition.

###  0.2.1 Classifier Knowledge

The only way to control classifier knowledge in a zero-shot context is model
selection. A model that generalizes to arbitrary tasks on unseen data needs a
sophisticated understanding of language that only emerges in large models
(bhargava2021generalization). Thus, performance will generally increase with
model size. To demonstrate, I classified the testing set of my data using
DeBERTa models trained for NLI with 44 (small), 86 (base), and 304 (large)
million parameters (he2021debertav3). Results in figure 4 show that stance
detection on par with supervised classifiers emerges in the largest version of
the model tested. Transformers are generally released in sizes that range from
small to extra large. While base and small models are commonly used for
supervised classifiers, contemporary zero-shot NLI classifiers should not be
expected to work well at sizes smaller than large.

A second variable to consider is the data used for pre-training.
Unsurprisingly, pre-training a model on more examples of entailment-hypothesis
pairs results in more robust generalization (nie2019adversarial). To date,
multiple large data sets have been curated precisely for pre-training,
including the ANLI (nie2019adversarial), and WANLI (liu2022wanli) data sets.
Training data for NLI evolve as researchers strive to present harder
challenges and ensure models learn language in a generalizable way
nie2019adversarial. Models that incorporate both more and more recent NLI data
sets into their pre-training should have better performance.

Figure 4: Models that classify data in a zero-shot or few-shot context rely on
linguistic understanding that generalizes well to unseen data. For a task as
complicated as entailment classification, this behavior only emerges in large
language models.

###  0.2.2 Context Completeness

A simple and effective way to control context completeness is by matching
hypotheses to documents via key words. For example, to classify tweets against
the hypotheses “The author of this tweet supports Trump” I may limit my data
to tweets that contain the word “Trump”. This ensures a minimum threshold of
context completeness is met. To equate anti-immigration tweets with pro-Trump
tweets, I might simply match hypotheses about immigration with tweets that
contain immigration related key words, and code them as pro-Trump if they
entail anti-immigrant hypotheses. As shown in figure 5, appropriately pairing
the hypotheses with text samples can have a dramatic effect on performance.
Tweets that do not mention Trump are classified with much lower precision than
tweets that do.

A second consideration is how many hypotheses should be used to classify a
document. As a rule of thumb, present the model with enough labels to
reasonably capture the range of possible opinions expressed and avoid a set of
labels that create a false dilemma. The intuition behind this is simple.
Imagine you are given a data set of tweets about President Trump and must
assign each tweet a label from a set of two: “The author of this tweet
supports Trump”, “The author of this tweet opposes Trump”. Stance neutral
statements about President Trump pose a dilemma in which you must choose a
label that may not seem appropriate. To alleviate this you might reduce the
task to a binary classification in which you decide if a tweet does or does
not express support for Trump, or you might add a third possible label for
neutral statements about Trump.

To demonstrate, I replicated this thought experiment with my testing data. I
classify the data three times, once with support, oppose, and neutral
hypotheses; another time with support and oppose hypotheses; and finally with
only a support hypothesis. Each document was hand labeled as support, neutral,
or oppose. To understand where performance is affected by the hypothesis set I
use, I examine accuracy within each of the three label categories in addition
to MCC across all categories.

Set of Hypotheses | Support | Oppose | Neutral | MCC (All)
---|---|---|---|---
Support | 88% | 86% | 61% | 0.62
Support/Oppose | 85% | 88% | 60% | 0.60
Support/Oppose/Neutral | 84% | 89% | 67% | 0.63
Table 1: The set of hypotheses presented to an NLI classifier represents the
universe of labels it can assign a document. While it is impossible to know
which set of hypotheses will produce the most accurate classifications for
each data set, a set of hypotheses that provides a distinct label for each
possible stance and avoids false dilemmas is a good starting point.

Results in table 1 show the model is less accurate in classifying a stance
when it has no appropriate hypothesis for that stance. The support only set
performed worse than the other two on oppose statements, while the set that
included a neutral stance performed best on neutral statements. The set of
hypotheses that present a false dilemma between support and oppose performs
the worst on the holistic measurement, MCC. However, the overall performance
of the three models is comparable.

There is not a single way to select a set of hypotheses. Introducing a neutral
label will probably not improve performance if there are no neutral samples in
your data. Conversely, a data set that mixes news headlines with social media
comments may benefit significantly from a neutral stance. It is difficult to
know what the composition of unlabeled data is a-priori. Without strong
priors, researchers should present a set of hypotheses that are both
exhaustive (i.e. at least one distinct hypothesis for each stance) and
mutually exclusive. Support, oppose, and neutral hypotheses may be sufficient
in most cases.

###  0.2.3 Validation

Validation presents a particular challenge to zero-shot classifiers because
they do not have a large training and test set like supervised classifiers.
However, the basics are not fundamentally different from a supervised
classifier. First, researchers should be explicit in defining stances and how
the hypotheses used are valid representations of those stances. Consider
developing a code book if selectively matching hypotheses to documents based
on the document’s content.

Second, calculate the sample size needed to estimate performance within a
confidence interval and label some data. Performance can be estimated with a
5-10% margin of error at a 95% confidence level with a small fraction of the
data needed to train a classifier. For headlines, tweets, or news articles
this can often be accomplished in a few hours.

Finally, consider conducting sensitivity analysis by generating synonymous
hypotheses and classifying the data multiple times. Researchers should
demonstrate their results are not sensitive to a specific set of hypotheses.
Well trained NLI classifiers should produce consistent results for synonymous
phrases. As an example, I classified my test set with 10 different synonymous
hypotheses that varied from simple phrases (“Trump is good”) to complete
hypotheses (“The author of this tweet approves of Trump”).222Hypotheses are
located in the appendix Across all sets of hypotheses the average MCC was .60,
with a minimum value of 0.57 and a maximum value of 0.63.

Zero-Shot NLI Classifiers for Stance Detection Zero-shot NLI classifiers work
with no training data. They are ideal for applications where training data is
unavailable, or many stances need to be classified. Classifier Knowledge •
Larger models will generalize better. • Use models trained on more and more
recent data. Context Completeness • NLI classifiers perform better on data
with more context completeness. Sample or match hypotheses to samples based on
key words. • Hypotheses represent the universe of labels that a model can
assign. Present a set of hypotheses that is exhaustive and avoids false
dilemmas (e.g. support or oppose). • Phrase hypotheses in complete sentences
and plain English. Validation • Calculate the number of samples needed to
estimate performance within a confidence interval and label some data. •
Conduct basic sensitivity analysis by classify documents multiple times with
synonymous hypotheses. Estimate inter-coder reliability between the
classifications and show results are robust to different phrasings.

Figure 5: NLI classifiers and in-context learners benefit from documents that
are more context complete. If the subject of a stance is not mentioned in a
document, it is more difficult for the model to infer stance.

##  0.3 In-Context Learning

In-context learning is an emergent behavior from a class of large language
models known as generative transformers (e.g. GPT-3, brown2020language). In-
context learning refers to the capability of these models to learn new tasks
via plain English prompts that describe the task. It is called “in-context”
because the model is not trained on the task, but learns it during the
prompt’s forward pass through the network (brown2020language). This allows the
model to do zero-shot classification (only a description of the task is
provided) and few-shot classification (a description of the task and/or a few
labeled examples are provided). In-context learning frames classification as a
next word prediction task. A description of the task and/or labeled samples
are presented to the model in a structured format as show in figure 6, with
the last sample leaving the label blank. The model then predicts the next word
in the sequence, which is presumably the label for the sample.

The ability to provide the model with a descriptive prompt and examples of the
task make them potentially more adaptable than NLI zero-shot transformers. The
ability to classify documents with plain English prompts also makes them
particularly attractive for off-the-shelf use. However, these models are still
not well understood and can be costly to implement. This obviates some of
their apparent advantages over supervised and NLI classifiers.

Figure 6: A 3-shot prompt example for GPT-3. The prompt consists of a
description of the classification task and three examples of correctly labeled
data. The sequence is left incomplete by leaving the label for the final tweet
blank. The model classifies the data by predicting the word after “Stance:”.
Here the model predicted “oppose”.

###  0.3.1 Classifier Knowledge

As with NLI classifiers, in-context learners depend on large model sizes. To
test this, I classified my testing data set with both the largest (Davinci)
and second largest (Curie) versions of GPT-3 with 50 and 35 labeled examples
respectively, the maximum number of examples each model could consistently
process. I used the older GPT-3 model for this task, as opposed to GPT-3.5
Turbo and GPT-4, because Open AI has not released recent models in various
parameter sizes and GPT-3 allows me to test results when only the number of
parameters is varied.333Davinci’s performance was stable at 35 and 20 labeled
examples as well, indicating that the number of examples was not what
differentiated performance. Shown in figure 4, GPT-3 demonstrates similar
behavior to DeBERTa – only large model versions are capable of robust stance
detection. This is consistent with evaluations from GPT-3’s creators
(brown2020language), and commonly used entailment classification benchmarks
(superglue).

Unlike NLI classifiers, this need for large models presents a significant
challenge to contemporary generative transformers. GPT-3, GPT-3.5 Turbo, and
GPT-4 require hundreds to thousands of times the computing resources of BERT
like models (hu2021lora). While supervised and NLI classifiers are deployable
via local hardware or free cloud services like Google Colab or university
compute clusters, models on the scale of generative transformers require
cloud-based super computers. This, combined with the proprietary nature of
contemporary generative transformers, poses a significant challenge for model
version control and, thus, scientific replication. There may also be
significant cost associated with their use due to either costs associated with
proprietary APIs or simply paying for the electricity necessary to power the
model. Thus, while a smaller NLI model can feasibly classify an entire data
set on its own, generative transformers may be more useful in creating
training sets that smaller models can be trained on.

###  0.3.2 Context Completeness

Like NLI classifiers, in-context learners benefit from context completeness.
However, in-context learners allow greater control over the context
completeness via prompts and labeled examples. As shown in figure 5, GPT-3.5
Turbo and GPT-4 perform better than DeBERTa when the target is not mentioned.
However, performance is still significantly better when the target is
mentioned. As with NLI classifiers, key-word based sampling or prompt matching
will benefit.

Realizing the contextual benefits prompts can provide presents a challenge for
in-context classification.“Prompt engineering” has spawned a significant
literature on its own due to how sensitive generative models are to
differences in prompts. Minor changes such as the ordering of labeled
examples, formatting, or the frequency of words, can result in large
differences in classification. These instabilities do not diminish with model
size or the number of examples in a prompt (lu2021fantastically;
zhao2021calibrate).444Solutions have been proposed, such as automatic prompt
generation and searches (gao2020making). However, no consensus resolution has
been reached, nor are there accessible software implementations of proposed
solutions. Thus, there are no clear rules on how many examples are needed, how
prompts should be formatted, or what makes a good set of labeled examples.
Further, searching across many prompts to optimize performance is a
potentially expensive endeavor. Instead, I offer broad guidance and recommend
users consult the most recent literature in this developing field.

First, the temperature parameter should be set to zero for any classification
task. Temperature controls the ”randomness” of a model’s output. To predict
the next word, generative transformers use a softmax function so that the
joint probability of each token appearing next sums to one. A higher
temperature parameter flattens the distribution such that as temperature
$T_{\to\infty}$ the probability distribution for the predicted token
approaches a uniform distribution (holtzman2020curious; ackley1985learning). A
temperature of zero makes the model prediction as deterministic as possible so
that the most likely class is chosen and results are maximally reproducible.

Second, more examples is generally better. Increasing the number of labeled
examples in a prompt can, but does not necessarily, reduce variance in
performance. More examples also increases average performance with diminishing
returns zhao2021calibrate. When prompting with a small number of examples,
provide a plain English description of the task. Task descriptions provide
decreased benefit as the number of labeled samples used in a prompt increases,
but can have a substantial impact on prompts with few examples
(gao2020making).

Second, be cognizant that generative transformers exhibit bias towards the
majority and final labels in a prompt (zhao2021calibrate;
lu2021fantastically). Consider any information known about the balance of the
data, as well as the nature of the task. If there is no reason to suspect
class imbalance, a random sample of labeled examples may be an appropriate
prompt. If recall is paramount it may be appropriate to over-sample examples
from the class of interest. Avoid long sequences of similarly labeled examples
towards the end of the prompt.

###  0.3.3 Validation

Many of the same validation principles that apply to NLI classifiers apply to
in-context learners. A small sample of labeled data can be used to estimate
performance within a margin of error. Classifying documents multiple times
with prompts that perform similarly can demonstrate robust results. However,
the resource and labor expenses associated with these models is multiplied
during validation. Researchers may need labeled data to test several prompts,
plus labeled data for out of sample testing. Further, classifying the data set
with multiple permutations of the prompt for sensitivity analysis multiplies
the cost by $n$ permutations. In short, validation a significant challenge for
in-context learners. While they should follow similar principles to zero-shot
NLI classifiers, the realities makes it impractical to do so.

In-Context Learners for Stance Detection In-context learners are a promising
frontier, however their size and cost makes them currently impractical for
most stance detection applications. Selecting a Model • Larger models are
generally more capable entailment classifiers. Controlling Information Context
• In-context learners benefit from more context-complete data. • Including
more labeled examples in the prompt will generally yield better results. •
Minor differences in prompts can cause large differences in results. Multiple
permutations should be tested. • The temperature parameter should be set to
zero. Validation • Estimate performance within a margin of error on a small
sample of labeled data. • Conduct sensitivity analysis by classifying the data
multiple times with similar performing prompts and showing results are robust
across prompts.